Space-time decoupling sentiment analysis method and system based on multi-modal data

Abstract

The invention discloses a space-time decoupling sentiment analysis method and system based on multi-modal data, and relates to the technical field of multi-modal sentiment analys.The method comprises the steps that text, audio and video initial features are input into a space-time decoupling and language focusing fusion model to be processed, and corresponding feature extraction is conducted on the enhanced text, audio and video features; obtaining specific text, audio and video features; performing shared feature extraction on the enhanced text, audio and video features to obtain shared text, audio and video features; the specific text, audio and video features and the shared text, audio and video features are input into a language focusing attractor module for multi-level feature extraction, and low-level features, middle-level features and high-level features are obtained respectively; emotion prediction is carried out based on the low-level features, the middle-level features and the high-level features, emotion prediction results of the features of all the levels are fused, a final emotion prediction result is obtained, and decoupling and fusion in multi-modal emotion analysis are achieved.

Description

Technical Field

[0001] This invention relates to the field of multimodal sentiment analysis technology, and in particular to a spatiotemporal decoupling sentiment analysis method and system based on multimodal data. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] Multimodal Sentiment Analysis (MSA) in large models aims to capture sentiment tendencies by integrating multimodal information such as linguistic content, visual cues (e.g., facial expressions and gestures), and acoustic signals (e.g., prosodic variations and pitch fluctuations). Compared to unimodal sentiment analysis, MSA leverages complementary and interdependent information across different modalities to achieve a more comprehensive and nuanced understanding of human emotions. Multimodal data typically exhibits a hierarchical feature structure: low-level features represent raw sensory input, mid-level features capture sentiment indicators specific to a particular modality, and high-level features encode abstract sentiment representations across modalities. Effectively modeling these hierarchical features is crucial for achieving accurate and robust sentiment inference in multimodal scenarios. With the exponential growth of multimedia content on social media platforms, MSA has been widely applied in various fields, including emotion-aware chatbots, sentiment annotation of audiovisual content, and large-scale public opinion analysis.

[0004] Early multimodal analysis methods primarily employed traditional machine learning techniques, which heavily relied on handcrafted features extracted individually from each modality. Shallow classifiers such as Support Vector Machines (SVM) and Random Forests were used for feature fusion and sentiment prediction. However, these methods have limited representational capabilities and struggle to capture the complex and heterogeneous characteristics of multimodal signals. Furthermore, traditional fusion strategies, such as tensor fusion and gating mechanisms, often fail to effectively handle noise interference and struggle to remove redundant or irrelevant data when filtering cross-modal information. With the rise of deep learning, models such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and attention-based architectures have been widely applied in Multimodal Adaptive Analysis (MSA). These models enable end-to-end learning of multimodal representations and support more accurate cross-modal alignment. For example, LSTM-based architectures effectively capture temporal dependencies in sentiment signals, while Transformer-based models model cross-modal semantic associations through self-attention mechanisms. Although these methods have achieved significant performance improvements on benchmark datasets, they still face significant challenges. Specifically, many existing methods lack discriminative power when processing or fusing all modalities, failing to consider the differences in reliability or relevance of different modalities in specific contexts. This approach may amplify modality-specific noise, introduce redundant or conflicting information, and ultimately weaken the quality and robustness of sentiment representation. Therefore, improving the model's ability to selectively focus on valid cues and suppress irrelevant signals remains a core research topic in the field of multimodal adaptive analysis.

[0005] Existing multimodal sentiment analysis methods can be broadly classified into two categories: representation-oriented learning methods and fusion-oriented methods.

[0006] Representation-oriented learning methods focus on extracting discriminative and complementary features from a single modality to alleviate challenges such as modal heterogeneity and feature redundancy. Text-Enhanced Modal Representation Learning (TEMSA) models enhance intermodal consistency by embedding semantic information from linguistic modalities into non-linguistic modal features through a text-guided cross-modal attention mechanism. This model also employs a text-guided reconstruction module to decouple specific modal features, balancing the maintenance of intermodal consistency with modality-specific differences. The Multimodal Fusion Contrastive Learning Framework (CLMLF) combines label contrastive learning and data contrastive learning tasks to capture sentiment-related common features while reducing noise interference in multimodal data. Unimodal Feature Extraction Network (UFEN) effectively extracts rich contextual information within a single modality by combining multi-layer convolutional neural networks and self-attention mechanisms, significantly reducing redundant features before fusion. Currently widely used pre-trained models, such as BERT and 3D Convolutional Neural Networks (3DCNN), are used for unimodal feature extraction from text and video, respectively. For example, the Lightweight Fusion Multimodal Emotion Recognition Model (MMER-LMF) achieves emotion classification suitable for resource-constrained environments by combining BERT-based text embeddings with video emotion scores extracted by 3D CNNs. Despite these advances, many representation learning methods still fail to fully utilize the unimodal feature information in audio and video, and significant structural differences between different modalities can lead to redundancy and noise problems when directly mapping features.

[0007] In contrast, fusion-oriented approaches focus on designing effective cross-modal interaction mechanisms to address challenges such as asynchronous or misaligned multimodal sequences. Various methods introduce gating mechanisms, combining multi-head attention with residual connections to dynamically adjust the Transformer architecture, thereby suppressing irrelevant features and highlighting key modalities. The Cross-Block Transformer (CB-Transformer) employs an asymmetric fusion strategy, first integrating fine-grained audio-visual features before interacting with text features, effectively reducing redundant computation. It successfully solves the sequence misalignment problem by combining local learning modules and a global self-attention mechanism. The Multi-Level Self-Attention Fusion Model (MAFOT) uses optimal transport kernel embedding technology to map heterogeneous single-modal features to a shared subspace and designs multi-level fusion modules along horizontal, vertical, and channel dimensions to improve classification accuracy. The Multi-Task Fusion Network (MTFN) models multi-sequence alignment as a multi-task learning problem, effectively mitigating inconsistencies in emotional representations between different modalities by jointly optimizing intra-modal feature extraction and cross-modal complementary learning. However, fusion methods that over-rely on cross-modal associations are prone to misidentifying irrelevant features, thus affecting the processing efficiency of long sequence data. Existing representation learning methods often fail to fully utilize single feature information in audio and video modalities, and significant structural differences between different modalities often lead to modal redundancy and noise interference problems. On the other hand, many fusion-oriented methods rely excessively on cross-modal association mechanisms, which not only easily leads to the incorrect association of irrelevant features but also reduces the processing efficiency of long sequence data. Summary of the Invention

[0008] To overcome the shortcomings of the prior art, this invention provides a spatiotemporal decoupling sentiment analysis method and system based on multimodal data, which realizes decoupling and fusion in multimodal sentiment analysis and solves the problems of modal redundancy and cross-modal noise in multimodal sentiment analysis.

[0009] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: In a first aspect, the present invention provides a spatiotemporal decoupling sentiment analysis method based on multimodal data, including: The original text, audio, and video data to be tested are obtained, and after feature extraction, the initial features of the text, audio, and video are obtained. The initial features of text, audio, and video are input into a spatiotemporal decoupling and language-focused fusion model for processing to obtain the final sentiment prediction result. The construction process of the spatiotemporal decoupling and language-focused fusion model includes: The initial features of text, audio, and video are aligned and then enhanced to obtain the enhanced text, audio, and video features, respectively. Feature extraction is performed on the enhanced text, audio, and video features respectively to obtain specific text, audio, and video features; shared feature extraction is performed on the enhanced text, audio, and video features to obtain shared text, audio, and video features. Specific text, audio, and video features, as well as shared text, audio, and video features, are input into the language focusing attraction submodule for multi-level feature extraction, resulting in low-level features, mid-level features, and high-level features, respectively. Sentiment prediction is performed based on low-level features, mid-level features, and high-level features respectively, and the sentiment prediction results of each level of features are combined to obtain the final sentiment prediction result.

[0010] A further technical solution involves encoding and aligning the initial text features to obtain enhanced text features; aligning the initial audio and video features and inputting them into a spatiotemporal dual attention module for feature enhancement, resulting in enhanced audio and video features respectively.

[0011] In a further technical solution, the spatiotemporal dual attention module performs channel attention and temporal attention on the aligned initial audio features and initial video features to obtain spatially enhanced features and temporally enhanced features; the initial audio features and initial video features are combined with the spatially enhanced features and temporally enhanced features respectively to obtain enhanced audio features and enhanced video features respectively.

[0012] A further technical solution is that the language focusing attraction submodule performs self-attention calculation using specific text features as query vectors and specific audio features and specific video features as key vectors to obtain text-dominant specific features; and performs cross-attention calculation using shared text features as query vectors and shared audio features and shared video features as key vectors to obtain text-dominant shared features.

[0013] A further technical solution involves using the shared features dominated by the text as low-level features, and then processing them through a multilayer perceptron to obtain low-level feature sentiment prediction results.

[0014] A further technical solution is that the text-dominant specific features are processed by a self-attention mechanism to obtain mid-level features, and then processed by layer normalization and nonlinear projection to obtain mid-level feature sentiment prediction results; the text-dominant shared features are processed by a cross-attention mechanism to obtain high-level features, and then processed by layer normalization and linear projection to obtain high-level feature sentiment prediction results.

[0015] A further technical solution is to design a decoupling loss for the spatiotemporal decoupling and language-focused fusion model, including reconstruction loss, specific feature consistency loss, triplet loss, and orthogonal constraint loss.

[0016] Secondly, this invention provides a spatiotemporal decoupled sentiment analysis system based on multimodal data, comprising: The multimodal feature acquisition module is configured to: acquire the raw text, audio, and video data to be tested, and obtain the initial features of the text, audio, and video after feature extraction; The multimodal sentiment analysis module is configured to input initial features of text, audio, and video into a spatiotemporal decoupling and language-focused fusion model for processing to obtain the final sentiment prediction result. The construction process of the spatiotemporal decoupling and language-focused fusion model includes: The initial features of text, audio, and video are aligned and then enhanced to obtain the enhanced text, audio, and video features, respectively. Feature extraction is performed on the enhanced text, audio, and video features respectively to obtain specific text, audio, and video features; shared feature extraction is performed on the enhanced text, audio, and video features to obtain shared text, audio, and video features. Specific text, audio, and video features, as well as shared text, audio, and video features, are input into the language focusing attraction submodule for multi-level feature extraction, resulting in low-level features, mid-level features, and high-level features, respectively. Sentiment prediction is performed based on low-level features, mid-level features, and high-level features respectively, and the sentiment prediction results of each level of features are combined to obtain the final sentiment prediction result.

[0017] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the spatiotemporal decoupling sentiment analysis method based on multimodal data as described in the first aspect.

[0018] Fourthly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the spatiotemporal decoupling sentiment analysis method based on multimodal data as described in the first aspect.

[0019] The above one or more technical solutions have the following beneficial effects: To address the issues of modal redundancy and cross-modal noise in multimodal sentiment analysis, this invention proposes an innovative framework: a spatiotemporal decoupling and language-focused fusion framework. This framework decouples modality-specific representations from modality-shared representations through a dual-branch coding technique and employs a quadruple loss constraint mechanism to effectively suppress feature overlap and noise amplification during cross-modal fusion. Technically, the spatiotemporal decoupling and language-focused fusion framework introduces two core modules: first, a spatiotemporal dual attention (STA) module, which simultaneously models the intramodal spatial structure and temporal dynamics of audiovisual content to achieve audio-visual enhancement; and second, a language-focused attractor (LFA) module, which uses the language modality as semantic guidance to extract complementary sentiment cues from non-textual modalities. Furthermore, a multi-level fusion prediction mechanism is designed, which significantly improves sentiment feature extraction and recognition by progressively integrating feature information from different levels.

[0020] This invention focuses on decoupling and fusion in multimodal sentiment analysis. It adopts a spatiotemporal dual attention mechanism to achieve decoupling and constructs a multi-level fusion prediction framework to integrate the results. Attached Figure Description

[0021] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0022] Figure 1 This is a flowchart of a spatiotemporal decoupling sentiment analysis method based on multimodal data, according to an embodiment of the present invention. Detailed Implementation

[0023] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0024] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0025] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0026] Example 1 like Figure 1As shown in the figure, this embodiment discloses a spatiotemporal decoupling sentiment analysis method based on multimodal data, which includes the following steps: S1: Obtain the raw text, audio, and video data to be tested, and after feature extraction, obtain the initial features of the text, audio, and video. In this embodiment, following the framework of previous research, the input modalities (multimodal) are defined as language (L), visual (V), and audio (A). Indicates initial features, ; Indicates enhanced features, Let the initial modal features of the input be the initial features of the text. Initial features of the video Initial characteristics of audio .

[0027] S2: Input the initial features of text, audio and video into the spatiotemporal decoupling and language focusing fusion model for processing to obtain the final sentiment prediction result.

[0028] The spatiotemporal decoupling and language-focused fusion model (also known as the spatiotemporal decoupling and language-focused fusion framework) includes a spatiotemporal dual attention module, a language-focused attraction submodule, and a multi-level fusion prediction mechanism.

[0029] S201: After encoding and aligning the initial text features, the enhanced text features are obtained.

[0030] For initial text features After processing by the BERT encoder, the enhanced representation is obtained by alignment through a one-dimensional convolutional projection layer. That is, enhanced text features.

[0031] S202: Align the initial audio features and initial video features and input them into the spatiotemporal dual attention module for feature enhancement, resulting in enhanced audio features and enhanced video features respectively.

[0032] For initial audio features Initial features of the video Because their initial dimensions are inconsistent, they are first aligned using a one-dimensional convolutional projection layer. The formula for the modality's convolutional network is as follows:

[0033] in, Indicates the initial dimension. This represents the dimension after modal alignment.

[0034] Afterwards, in order to enhance and In terms of spatiotemporal representation, this invention framework introduces a Spatiotemporal Dual Attention (STA) module, which designs a Spatiotemporal Dual Attention (STA) mechanism. The Spatiotemporal Attention (STA) mechanism is a dual-path mechanism combining channel attention and temporal attention to simultaneously capture the features of data in both spatial and temporal dimensions. The Spatiotemporal Dual Attention (STA) module includes channel attention and temporal attention, aiming to process information in the spatial and temporal dimensions separately through a dual-path mechanism, thereby gaining a more comprehensive understanding of the spatiotemporal features of the input data. Channel attention primarily focuses on the spatial features of the input data. First, feature maps are extracted from the input data, containing information from different channels. Then, an attention mechanism is used to calculate the attention weight for each channel, multiplying the feature of each channel by its corresponding attention weight to obtain a weighted feature. This allows the model to focus more on important channel information while suppressing unimportant information. STA combines channel attention and spatiotemporal attention to enhance features in both spatial and temporal dimensions. Channel attention compresses the features of each channel into a scalar by performing global average pooling (GAP), and then obtains the weight of each channel through convolutional layers and activation functions. and Finally, these weights are applied to the original feature map. Therefore, , Enhanced attention through the channel :

[0035] in, Indicates spatial enhancement features, , This is the activation function.

[0036] Temporal attention obtains the weights for each time step by performing convolution operations on the features at each time step. and Finally, these weights are applied to the original feature map. Therefore, , Enhanced attention over time :

[0037] in, This indicates time-enhanced features.

[0038] Finally, the initial audio features Initial features of the video The joint representation is obtained by combining the spatiotemporal dual attention mechanism enhancement features (spatial enhancement features and temporal enhancement features). , : ,

[0039] in, Indicates the enhanced audio features, This indicates the enhanced video features.

[0040] S203: Input the enhanced text features, enhanced audio features, and enhanced video features into the corresponding modality-specific encoders for feature extraction to obtain specific text features, specific audio features, and specific video features; input the enhanced text features, enhanced audio features, and enhanced video features into the shared encoder for feature extraction to obtain shared text features, shared audio features, and shared video features.

[0041] To achieve feature decoupling, the spatiotemporal decoupling and language focusing fusion framework introduces a feature decoupling module, which designs a shared encoder. and three modality-specific encoders Modality-specific encoder This invention extracts fine-grained features unique to each modality. The features extracted by the modality-specific encoder are called specific features. Specific features contain only unique information specific to a single modality, such as intonation fluctuations in audio and lexical sentiment in text; these features are extracted by the shared encoder. Extracting semantic information shared across modalities is called shared features. These features include semantic information shared across modalities, such as the commonality of "positive emotion" in text, smiles, and cheerful tones.

[0042] The encoder employs a Transformer architecture, enabling the model to consider information from all positions within the sequence when processing sequential data, thus effectively capturing long-range dependencies. Separating the two types of features through independent encoders effectively avoids interference from redundant information between modalities. Simultaneously, a spatiotemporal attention mechanism is used to strengthen key spatiotemporal features, suppress noise, and improve the quality of decoupled features. Both encoders are implemented based on TransformerEncoder, sharing encoder pairs. Process and extract shared features Shared features It can be further subdivided into shared text features, shared audio features, and shared video features; specific encoders target... Process them separately to extract modality-specific features. : , .

[0043] The modality-specific encoders are three distinct TransformerEncoders, each with its own independently and randomly initialized set of parameters. The shared encoder is a separate TransformerEncoder whose parameters are not shared with the three modality-specific encoders; it is used to process data from all three modalities. Upon entering the TransformerEncoder, the data from each modality has already undergone modal alignment through a one-dimensional convolutional projection layer, placing it in the same dimensional space.

[0044] S204: Input specific text features, specific audio features, specific video features, shared text features, shared audio features, and shared video features into the language focusing attraction submodule to perform multi-level feature extraction, and obtain low-level features, medium-level features, and high-level features respectively; In this embodiment, a Language Focus Attractor (LFA) module is introduced into the spatiotemporal decoupling and language focus fusion framework. This module uses language modalities (i.e., text) as semantic guides to extract complementary emotional cues from non-text modalities (i.e., video and audio).

[0045] In the LFA module, both shared and specific features use language modality features, i.e., text features, as the core (Query), and video and audio features as Key-Value pairs, resulting in text-dominated features. Specifically, shared text features, shared audio features, and shared video features are processed by the LFA module to obtain text-dominated shared features; that is, the final feature representation is all transformed into text. Specific text features, specific audio features, and specific video features are processed by the LFA module to obtain text-dominated specific features.

[0046] Specific features of decoupling modes First, it is processed through position embedding and dropout, and then fed into the Transformer layer. The core operation in these layers is the multimodal cross-attention mechanism. The LFA module performs self-attention and cross-attention mechanisms respectively, all of which are centered on the Language modality as the Query. This setting allows language modalities to be derived from other modality-specific features, i.e. , This attracts supplementary information to achieve decoupling and refactoring. The corresponding key-value pair is defined as ( , Self-attention mechanism formula:

[0047] in, Representing specific text features The query vector, , Other modes ( The key-value vector of ) Represents the feature dimension, used to scale the attention score; , and This represents the linear projection matrix, which is obtained from the linear projection layer created by the attention module during TransformerEncoder initialization.

[0048] The cross-attention mechanism is represented as:

[0049] in, The high-order query vector represents the shared text features, which are derived from the shared text features. via projection matrix The transformation is used to capture cross-modal global dependencies; , A key-value vector representing shared video and audio features; , , This represents the linear projection matrix, obtained from the linear projection layer created by the cross-attention module during TransformerEncoder initialization.

[0050] In this embodiment, to address the loss of modal information during extraction and enhancement, the present invention designs a quadruple loss function based on Euclidean space and cosine similarity, including reconstruction loss, specific feature consistency loss, triplet loss, and orthogonal constraint loss, to achieve optimization of feature decoupling.

[0051] To make the enhanced features (features processed by a one-dimensional convolutional projection layer) closer to the original features. Design refactoring loss:

[0052] in, Indicates the reconstruction loss. Indicates decoder, This indicates feature splicing.

[0053] To ensure consistency between the enhanced specific features (features from video, audio, and language modalities enhanced by the spatiotemporal dual attention mechanism and processed by BERT, including fine-grained features) and the original specific features (features from the original input data that contain modal fine-grained features and differences, obtained through a specific encoder), a specific feature consistency loss is designed:

[0054] in, This represents the feature consistency loss.

[0055] Despite reconstruction loss Consistency loss with specific features While shared encoders facilitate decoupling, their effectiveness in achieving robust decoupling remains limited. Therefore, this invention designs a triplet loss to improve the performance of modal shared encoders. The triplet loss is defined as:

[0056] in, Indicates the loss of the triplet. Represents the total number of samples. Represents a set of shared features , Indicates and Positive sample features with corresponding semantic labels (same modality or cross modality) are used to bring similar features closer together; Indicates and Negative sample features with different semantic labels are used to distance discriminative features. Indicates the threshold parameter. express and distance, express and The distance.

[0057] To further refine the decoupling between the shared feature space and the specific feature space, an orthogonal constraint loss is introduced. To minimize redundancy and conflicts between shared and specific multimodal features:

[0058] in, This represents the orthogonal constraint loss. This represents norm operations, specifically the square of the 2-norm.

[0059] Finally, the four losses (regularization terms) are combined to form the decoupling loss, which is expressed as:

[0060] in, Indicates the decoupling loss. , , , This represents hyperparameters.

[0061] S205: Sentiment prediction is performed based on low-level features, medium-level features, and high-level features respectively, and the sentiment prediction results of each level of features are integrated to obtain the final sentiment prediction result.

[0062] In the feature fusion and prediction process, features at low, medium and high levels are extracted to achieve hierarchical prediction.

[0063] Utilizing shared features The process of making preliminary sentiment prediction is called low-level feature prediction. Low-level features are the initial part of LFA, that is, the most basic shared text features obtained from the initial LFA processing, i.e., text-dominated shared features. Directly fusing multimodal features may lead to different modalities conveying similar information. The Language Focus Attractor (LFA) module, through an attention mechanism, forces the language modality to be a query and the audiovisual modality to be a key-value pair. Through the Transformer encoder, it actively filters complementary information from other modalities, achieving cross-modal information complementarity, providing richer semantic information, and better capturing local information, which is crucial for fine-grained feature extraction in sentiment analysis. Preliminary sentiment prediction using low-level features can supervise the learning process of the shared space. This means that the model can better understand and utilize shared features, thereby improving the model's generalization ability. At the same time, the preliminary prediction of low-level features can serve as a supervisory signal, helping the model learn more useful feature representations, thus performing better in subsequent mid-to-high-level feature fusion.

[0064] Low-level features These are text-dominant shared features obtained after initial LFA processing. They can achieve preliminary sentiment prediction and provide rich semantic information for fine-grained extraction. The formula is as follows:

[0065] in, This represents the sentiment prediction results of low-level features. This represents a multilayer perceptron. Indicates low-level projection weights. This represents the low-level projection bias term.

[0066] Mid-level features are text-dominant features output by the LFA module that are dynamically fused with specific audio, video, and text features through a self-attention mechanism. This effectively captures fine-grained information within the features. Mid-level features are intermediate abstract features in multimodal data, falling between low-order original features and high-order global semantic features. Compared to low-level features, mid-level features are more semantically oriented and can directly associate with sentiment tags. Compared to high-level features, mid-level features retain more fine-grained details, avoiding information loss caused by global abstraction, and their dimensionality is usually lower than low-level features, reducing computational complexity. The mid-level feature representation is obtained through residual connections and layer normalization.

[0067] in, Indicates mid-level features, Indicates specific text features, Representation layer normalization.

[0068] Mid-level feature prediction results:

[0069] in, This represents the sentiment prediction results for mid-level features. , This represents two layers of nonlinear projection.

[0070] High-level features are obtained through LFA operations and cross-modal cross-attention mechanisms. In other words, the text-dominant shared features output by LFA are processed by the cross-attention mechanism to obtain high-level features, which can effectively capture the overall sentiment tendency in long sequences and model semantic dependencies. High-level features are global semantic representations formed after multi-layered abstraction of multimodal data. They can capture the global sentiment tendency of multimodal data and, based on the Transformer's self-attention mechanism, can effectively model semantic dependencies in long sequences. Based on the shared and specific features generated in the feature decoupling stage, the LFA module uses a hierarchical cross-modal cross-attention mechanism to make high-level features the core of global semantic alignment between modalities. The high-level feature representation is obtained through layer-normalized residual connections.

[0071] in, Indicates high-level features, This indicates shared text features.

[0072] High-level features preserve the global sentiment tendency of the language modality while incorporating higher-order semantic supplements from the audiovisual modality. These high-level features are projected through a higher-order feature projection function. Mapped to high-level feature sentiment prediction results .

[0073] After LFA (Low-Level Facilitation) enhances modal representation capabilities, low-, mid-, and high-level features can be integrated to generate the final sentiment prediction result, which is high-order fusion prediction. High-order fusion prediction allows the model to capture information at different levels, enabling low-level features to retain fine-grained details, mid-level features to strengthen local alignment, and high-level features to capture global sentiment semantics. This improves the model's generalization ability, allowing it to better adapt to different datasets and tasks. Furthermore, by integrating multi-scale information, the model can better handle redundancy and noise issues, improving its robustness in different scenarios. In addition, this integration of multi-scale information helps the model make more accurate decisions in complex sentiment analysis tasks and allows the model to utilize richer information during the decision-making process. Finally, the low-, mid-, and high-level sentiment prediction results are combined through a fully connected layer function. Obtain the final fusion prediction result. :

[0074] in, This indicates the fusion prediction result.

[0075] To optimize the loss of modal information in multi-level fusion, this invention designs a modal output loss mechanism. :

[0076] in, Indicates the number of samples. For multimodal sentiment analysis tags.

[0077] Meanwhile, consider modal sharing loss Modal-specific loss and modal output loss The hierarchical prediction loss, the total MSA hierarchical prediction loss is expressed as:

[0078] in, Indicates output loss. This represents learnable hyperparameters. This represents the loss function.

[0079] The total model loss is obtained by combining the decoupling loss and the hierarchical prediction loss:

[0080] in, This represents the total loss of the model.

[0081] In summary, the framework of this invention comprises two core modules: a feature decoupling module and a feature fusion module. The former focuses on feature enhancement and extraction, while the latter integrates Language Focusing Attractor (LFA), multi-level fusion, and feature prediction techniques. It mainly consists of three parts: feature extraction based on a spatiotemporal attention mechanism, modal feature decoupling and reconstruction, and multi-level fusion prediction based on a language focusing attractor. The model's advantages are: the spatiotemporal attention mechanism simultaneously captures channel and temporal dimension features, alleviating the problem of weak audiovisual modal representation; decoupling constraints reduce modal redundancy and enhance feature independence; and multi-level fusion prediction combined with residual connections improves model robustness.

[0082] The specific implementation process is as follows: First, the original features from text, audio, and video are input into the feature extraction module. Feature alignment is achieved through a BERT encoder and a one-dimensional convolutional layer, respectively. A spatiotemporal dual attention module enhances the representation of audio and video modalities. Next, the enhanced features are decoupled by separating a shared encoder from a specific encoder. To optimize decoupling, a quadruple loss function based on Euclidean space and cosine similarity is designed. Finally, the reconstructed features are processed in multiple layers, extracting low-order, mid-order, and high-order features respectively. The three types of features are combined to form the final modality representation, resulting in the sentiment prediction.

[0083] Experimental evaluation: This invention evaluates the method on two widely used multimodal sentiment datasets: CMU-MOSI and CMU-MOSEI. CMU-MOSI contains 2199 monologue video clips, with audio and visual features extracted at sampling rates of 12.5 Hz and 15 Hz, respectively. This dataset is divided into 1284 training samples, 229 validation samples, and 686 test samples. CMU-MOSEI is a larger dataset containing 22856 YouTube movie review clips. Audio features are extracted at a sampling rate of 20 Hz, and visual features at a sampling rate of 15 Hz. The data is divided into 16326 training samples, 1871 validation samples, and 4659 test samples. In both datasets, each clip is labeled with a sentiment rating from -3 to +3, reflecting a sentiment spectrum from strongly negative to strongly positive.

[0084] As shown in Table 1, the proposed method demonstrates superior performance across most evaluation metrics. Key findings are summarized as follows: Compared to decoupled feature methods such as MISA and MuLT, the proposed method, by strengthening language-dominant cues within the decoupled subspace, more effectively captures cross-modal dynamics and enhances multimodal representation capabilities. Unlike Transformer-based methods such as LMF, Self-MM, and ALMT, which rely on joint attention mechanisms for cross-modal interaction, the proposed method operates within a decoupled subspace. By integrating hierarchical prediction strategies of pre- and post-fusion features, it constructs more robust sentiment representations. On the CMU-MOSI dataset, the proposed method outperforms existing state-of-the-art models, including DLF, in binary classification accuracy and F1 score. On the larger and more diverse CMU-MOSEI dataset, the framework remains competitive, matching or even slightly outperforming recent leading methods on key metrics.

[0085] Table 1. Validation on the MOSI and MOSEI datasets.

[0086] Example 2 This embodiment discloses a spatiotemporal decoupled sentiment analysis system based on multimodal data, including: The multimodal feature acquisition module is configured to: acquire the raw text, audio, and video data to be tested, and obtain the initial features of the text, audio, and video after feature extraction; The multimodal sentiment analysis module is configured to input initial features of text, audio, and video into a spatiotemporal decoupling and language-focused fusion model for processing to obtain the final sentiment prediction result. The construction process of the spatiotemporal decoupling and language-focused fusion model includes: The initial features of text, audio, and video are aligned and then enhanced to obtain the enhanced text, audio, and video features, respectively. Feature extraction is performed on the enhanced text, audio, and video features respectively to obtain specific text, audio, and video features; shared feature extraction is performed on the enhanced text, audio, and video features to obtain shared text, audio, and video features. Specific text, audio, and video features, as well as shared text, audio, and video features, are input into the language focusing attraction submodule for multi-level feature extraction, resulting in low-level features, mid-level features, and high-level features, respectively. Sentiment prediction is performed based on low-level features, mid-level features, and high-level features respectively, and the sentiment prediction results of each level of features are combined to obtain the final sentiment prediction result.

[0087] Example 3 The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method of Embodiment 1.

[0088] Example 4 The purpose of this embodiment is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the method of Embodiment 1.

[0089] The steps and methods involved in the apparatuses of Embodiments 3 and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0090] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0091] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0092] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A spatiotemporal decoupling sentiment analysis method based on multimodal data, characterized in that, include: The original text, audio, and video data to be tested are obtained, and after feature extraction, the initial features of the text, audio, and video are obtained. The initial features of text, audio, and video are input into a spatiotemporal decoupling and language-focused fusion model for processing to obtain the final sentiment prediction result. The construction process of the spatiotemporal decoupling and language-focused fusion model includes: The initial features of text, audio, and video are aligned and then enhanced to obtain the enhanced text, audio, and video features, respectively. Feature extraction is performed on the enhanced text, audio, and video features respectively to obtain specific text, audio, and video features; shared feature extraction is performed on the enhanced text, audio, and video features to obtain shared text, audio, and video features. Specific text, audio, and video features, as well as shared text, audio, and video features, are input into the language focusing attraction submodule for multi-level feature extraction, resulting in low-level features, mid-level features, and high-level features, respectively. Sentiment prediction is performed based on low-level features, mid-level features, and high-level features respectively, and the sentiment prediction results of each level of features are combined to obtain the final sentiment prediction result.

2. The spatiotemporal decoupling sentiment analysis method based on multimodal data as described in claim 1, characterized in that, After encoding and aligning the initial text features, the enhanced text features are obtained. The initial audio and video features are aligned and then input into the spatiotemporal dual attention module for feature enhancement, resulting in enhanced audio and video features, respectively.

3. The spatiotemporal decoupling sentiment analysis method based on multimodal data as described in claim 2, characterized in that, The spatiotemporal dual attention module performs channel attention and temporal attention on the aligned initial audio features and initial video features to obtain spatially enhanced features and temporally enhanced features. The initial audio features and initial video features are combined with spatial enhancement features and temporal enhancement features, respectively, to obtain enhanced audio features and enhanced video features.

4. The spatiotemporal decoupling sentiment analysis method based on multimodal data as described in claim 1, characterized in that, The language focus attraction submodule performs self-attention calculation using specific text features as query vectors and specific audio and video features as key vectors to obtain text-dominant specific features; it also performs cross-attention calculation using shared text features as query vectors and shared audio and video features as key vectors to obtain text-dominant shared features.

5. The spatiotemporal decoupling sentiment analysis method based on multimodal data as described in claim 4, characterized in that, Using the shared features dominated by the text as low-level features, and after processing by a multilayer perceptron, the low-level feature sentiment prediction results are obtained.

6. The spatiotemporal decoupling sentiment analysis method based on multimodal data as described in claim 4, characterized in that, The text-dominant specific features are processed through a self-attention mechanism to obtain mid-level features, which are then processed through layer normalization and nonlinear projection to obtain mid-level feature sentiment prediction results. The text-dominant shared features are processed through a cross-attention mechanism to obtain high-level features, which are then processed through layer normalization and linear projection to obtain high-level feature sentiment prediction results.

7. The spatiotemporal decoupling sentiment analysis method based on multimodal data as described in claim 1, characterized in that, The spatiotemporal decoupling and language-focused fusion model is designed with decoupling loss, including reconstruction loss, specific feature consistency loss, triplet loss and orthogonal constraint loss.

8. A spatiotemporal decoupled sentiment analysis system based on multimodal data, characterized in that, include: The multimodal feature acquisition module is configured to: acquire the raw text, audio, and video data to be tested, and obtain the initial features of the text, audio, and video after feature extraction; The multimodal sentiment analysis module is configured to input initial features of text, audio, and video into a spatiotemporal decoupling and language-focused fusion model for processing to obtain the final sentiment prediction result. The construction process of the spatiotemporal decoupling and language-focused fusion model includes: The initial features of text, audio, and video are aligned and then enhanced to obtain the enhanced text, audio, and video features, respectively. Feature extraction is performed on the enhanced text, audio, and video features respectively to obtain specific text, audio, and video features; shared feature extraction is performed on the enhanced text, audio, and video features to obtain shared text, audio, and video features. Specific text, audio, and video features, as well as shared text, audio, and video features, are input into the language focusing attraction submodule for multi-level feature extraction, resulting in low-level features, mid-level features, and high-level features, respectively. Sentiment prediction is performed based on low-level features, mid-level features, and high-level features respectively, and the sentiment prediction results of each level of features are combined to obtain the final sentiment prediction result.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the spatiotemporal decoupling sentiment analysis method based on multimodal data as described in any one of claims 1-7.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the spatiotemporal decoupling sentiment analysis method based on multimodal data as described in any one of claims 1-7.

Images