A multi-modal sentiment analysis method based on lightweight hierarchical decoupled contrastive learning

Abstract

The present application relates to a kind of multi-modal sentiment analysis methods based on lightweight hierarchical decoupling contrast learning, adopt " first purification after alignment " paradigm, first by feature decoupling module, each modal representation is decomposed into mutually orthogonal shared feature and private feature, effectively separate emotional semantics and modal noise;Then design three-level hierarchical contrast learning goal, in single mode, double mode intermediate body and global fusion three granularity, progressively forced cross-modal semantic alignment;Finally, using lightweight Transform fusion module only in modal level embedding is carried out cross-modal interaction, greatly reduce the computing cost;The method of the present application can effectively solve the problem of modal heterogeneity, noise interference and computational efficiency in multi-modal sentiment analysis, while ensuring the accuracy is better than similar method, realize significant lightweight effect, especially suitable for practical engineering deployment in resource-constrained scenarios.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and multimodal information processing, and in particular to a multimodal sentiment analysis method based on lightweight hierarchical decoupling contrastive learning. Background Technology

[0002] Multimodal sentiment analysis aims to infer human emotions by jointly modeling linguistic, visual, and acoustic signals from user-generated videos. With the rapid development of short video and social media platforms, users express their emotions and opinions through various means such as voice, facial expressions, and text, making multimodal sentiment analysis widely applicable in areas such as public opinion monitoring, human-computer interaction, and intelligent customer service. Compared to unimodal methods that rely solely on text, multimodal sentiment analysis can utilize complementary cross-modal cues to eliminate ambiguity in underlying emotions; for example, it can identify rhetorical devices such as sarcasm through tone of voice and facial expressions.

[0003] However, existing multimodal sentiment analysis methods still face the following three technical challenges: First, modal heterogeneity leads to difficulties in semantic alignment. Linguistic data consists of discrete, semantically rich text tags, while visual and acoustic streams are continuous signals and are often entangled with task-independent environmental changes. This difference creates a significant distribution gap between modalities. Traditional fusion strategies, such as direct early splicing or indiscriminate cross-modal attention mechanisms, easily introduce modality-specific noise into the fused features, failing to effectively synchronize high-level semantics and leading to modality dominance—that is, the model over-relies on text and does not fully utilize non-linguistic cues.

[0004] Second, there is the entanglement between modality-specific information and task-irrelevant noise. In real-world, unconstrained environments, each modality contains sentiment-related private information as well as harmful interfering factors. Traditional fusion architectures typically aggregate these features through attention mechanisms or pooling operations without explicitly distinguishing between these two types of information, leading to environmental noise propagating into the joint representation space or discarding valuable complementary cues. Although methods have attempted to decouple modality-shared components from modality-private components in recent years, such decoupling constraints are usually imposed only at a single fusion stage and cannot maintain sufficient semantic consistency across deep fusion layers, from single-modality encoding through bimodal intermediates to global trimodal fusion.

[0005] Third, current high-performance architectures suffer from low computational efficiency. Existing mainstream multimodal sentiment analysis models heavily rely on deep Transformer stacking or high-capacity fusion modules. The computational complexity of standard cross-modal self-attention mechanisms is quadratic with the time series length, O(T). 2 This leads to excessive memory consumption and computational overhead, severely hindering actual deployment on resource-constrained edge devices. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a multimodal sentiment analysis method based on lightweight hierarchical decoupled contrastive learning. It adopts a "purify first, then align" paradigm to effectively solve the problems of modal heterogeneity, noise interference, and computational efficiency in multimodal sentiment analysis.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A multimodal sentiment analysis method based on lightweight hierarchical decoupling contrastive learning includes the following steps: Step S1: Obtain the text containing Visual Harmony A multimodal sentiment dataset with three modalities was prepared, and the data for each modality was preprocessed. Step S2: Use a modality-specific encoder to encode the single-modal features of the three modalities respectively, and obtain the representation vector of each modality; Step S3: Through the feature decoupling module, the representation vector of each mode is decomposed into mode-invariant shared features and mode-specific private features, and orthogonal constraint loss is used to ensure strict separation of the two types of features; Step S4: Apply a three-level hierarchical contrastive learning objective to the purified shared features to achieve progressive alignment of cross-modal semantics; Step S5: Use the lightweight Transformer fusion module to perform cross-modal interactive fusion of shared features to obtain a fused representation; Step S6: Concatenate the fused representation with the private features of the three modalities, and output the final sentiment analysis result through the prediction layer.

[0008] Furthermore, the multimodal sentiment dataset in step S1 contains aligned or unaligned language sequences. Visual sequences Harmony and acoustic sequence The time length of each modal sequence and input feature dimensions different.

[0009] Furthermore, the single-modal feature encoding in step S2 specifically includes: Step S2.1: For the text modality, a pre-trained BERT encoder is used to extract text representations, and the embeddings corresponding to the [CLS] tags are taken as sentence-level text representations. ,in , For text representation dimensions; Step S2.2: For visual and acoustic modalities, a bidirectional long short-term memory network (BiLSTM) is used for encoding. By concatenating the final hidden states in the forward and backward directions, modal-level representations are obtained. and ,in , These are the visual and acoustic representation dimensions, respectively.

[0010] Further, in step S2, the BERT encoder uses a BERT-base-uncased pre-trained model for the English dataset and a Chinese-BERT pre-trained model for the Chinese dataset; the BiLSTM encoder has a hidden layer dimension of 64, a layer count of 1, and a dropout rate of 0.25, and obtains visual representations by concatenating the forward and backward hidden states. Harmony and acoustic characterization .

[0011] Furthermore, the feature decoupling module in step S3 decouples each mode. Single-mode characterization Perform the following operations: Step S3.1: Utilize two lightweight MLP projection heads and ,Will Mapped to shared features respectively and private features ,Right now , ,in , For common feature dimensions; each MLP projection head consists of a linear transformation layer, a ReLU activation function, a Dropout regularization layer, and a linear transformation layer in sequence; Step S3.2: Calculate the difference loss Force orthogonality constraints between shared and private features: ; in and These are the shared and private representation matrices after zero-mean normalization, respectively. It is the Frobenius norm.

[0012] Furthermore, the three-level hierarchical contrastive learning in step S4 specifically includes: Step S4.1: First-level cross-modal alignment, performing pairwise semantic synchronization between purified shared features of different modalities, for modal pairs Each pair in the diagram is mapped to a low-dimensional space through its respective linear projection head and then... Normalization is performed, and a symmetric cross-entropy loss function is used for contrastive learning to obtain the first-level contrastive loss. Complete the alignment of basic semantics between modalities; Step S4.2: Second-level bimodal center alignment, constructing trimodal composite features by stitching and projecting shared representations. ,in As a linear projection layer, the trimodal composite features are compared and aligned with the shared features of text, acoustics, and vision, respectively, to obtain the second-level contrastive loss. ; Complete deep fusion of intermediate representation levels between modalities; Step S4.3: Third-level global fusion alignment. The fused representation output by the lightweight Transformer fusion module is symmetrically aligned with the purified shared features of each modality to obtain the third-level contrastive loss. Achieve alignment and fusion at the global semantic level; Finally, the three levels of contrastive loss are integrated to obtain the overall contrastive loss, which guides the training and optimization of the multimodal model. The overall contrastive loss is: .

[0013] Furthermore, in step S4, each level of contrastive learning employs the CLIP-style InfoNCE symmetric loss function, the calculation formula of which is: ; in , , Represents the cosine similarity matrix. Indicates positive and negative sample pairs. This represents the truth matrix of positive and negative sample pairs. This is a learnable temperature scaling parameter.

[0014] Furthermore, the specific process of the lightweight Transformer fusion module in step S5 includes: Step S5.1: Stack the shared features of the three modalities: ,in For the number of modes, For common feature dimensions; Step S5.2: Input the stacked features into a shallow Transformer encoder for cross-modal interaction: The Transformer encoder contains only a single Transformer coding layer, employs four attention heads and a 256-dimensional feedforward network, and has a Dropout rate of 0.1. Step S5.3: Perform mean pooling on the Transformer output to obtain the fused representation: ; The lightweight Transformer fusion module only Cross-modal attention is performed on each modal-level embedding.

[0015] Furthermore, the final prediction in step S6 specifically includes: Step S6.1: Concatenate the fused representation with the private features of the three modalities: ; Step S6.2: Output the sentiment prediction result through the multilayer perceptron prediction layer: The prediction layer consists of a linear transformation layer, a ReLU activation function, a Dropout regularization layer, and another linear transformation layer in sequence.

[0016] Furthermore, the overall optimization objective function of the method is: ; in The task loss function is calculated using the mean absolute error (MAE): ; and The weights for the difference loss and contrast loss are determined by grid search with a step size of 0.01. The Adam optimizer is used, and different learning rates are set for the BERT module and the main network. Early stopping and gradient pruning are used during training.

[0017] Compared with the prior art, the present invention has at least the following beneficial effects: This invention projects heterogeneous multimodal features into mutually orthogonal subspaces through a constraint-based feature decoupling module. By using mathematical orthogonal constraints, it separates sentiment-consistent shared semantics from modality-specific private cues, effectively preventing the leakage of task-irrelevant noise and improving feature purity.

[0018] This invention designs a progressive three-level hierarchical contrastive learning objective, which enforces robust semantic consistency at three granularities: single-modal, bimodal intermediate, and global fusion, bridging the gap in cross-modal semantic distribution and suppressing the modality dominance problem.

[0019] This invention introduces a lightweight Transformer fusion architecture that operates only on compressed modal-level embeddings, rather than redundant time series, significantly reducing computational complexity and the number of parameters.

[0020] This invention achieves a balance between feature purification, progressive alignment, and model efficiency. It enhances semantic consistency through hard decoupling and hierarchical comparison, and achieves a superior accuracy-efficiency balance through lightweight design. Experiments show that compared to the MulT model, this invention reduces the number of parameters by approximately 19 times and the floating-point computation by over 200 times; compared to Cube MLP, it achieves better accuracy with a 28% reduction in parameters and approximately 17 times the floating-point computation, achieving a good accuracy-efficiency balance in resource-constrained application scenarios. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the network structure of the multimodal sentiment analysis method in this embodiment of the invention.

[0022] Figure 2 This is a schematic diagram of the feature decoupling module in an embodiment of the present invention.

[0023] Figure 3 This is a t-SNE visualization of the decoupled features in an embodiment of the present invention.

[0024] Figure 4 This is a visualization of the t-SNE effect of sentiment clustering in an embodiment of the present invention. Detailed Implementation

[0025] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0026] This invention provides a multimodal sentiment analysis method based on lightweight hierarchical decoupling contrastive learning. It adopts a "purify first, then align" paradigm and achieves efficient and accurate multimodal sentiment analysis through three core technical steps: feature decoupling, hierarchical contrastive learning, and lightweight Transformer fusion. Specific embodiments are given below.

[0027] like Figure 1 As shown, the overall architecture of the multimodal sentiment analysis (L-HDCL) framework based on lightweight hierarchical decoupled contrastive learning proposed in this invention includes four core components: single-modal encoding, feature decoupling, multi-level contrastive learning, and lightweight Transformer fusion. The multimodal sentiment analysis method of this invention includes the following steps S1-S6.

[0028] Step S1: Obtain the text containing Visual Harmony A multimodal sentiment dataset with three modalities was prepared, and the data for each modality was preprocessed. The multimodal sentiment dataset contains language sequences. Visual sequences Harmony and acoustic sequence The time length of each modal sequence and input feature dimensions different.

[0029] Step S2: Use a modality-specific encoder to encode the single-modal features of the three modalities respectively, and obtain the representation vector of each modality.

[0030] For the text modality, the text encoder uses a pre-trained BERT-base-uncased model to extract the 768-dimensional embedding vectors corresponding to the [CLS] tags as sentence-level text representations. .

[0031] The visual and acoustic encoder employs a bidirectional LSTM network with a hidden layer dimension of [missing information]. With a layer count of 1 and a Dropout rate of 0.25, the visual representation is obtained by concatenating the final hidden states in the forward and backward directions. .

[0032] Step S3: Through the feature decoupling module, the representation vector of each mode is decomposed into mode-invariant shared features and mode-specific private features, and orthogonal constraint loss is used to ensure strict separation of the two types of features.

[0033] like Figure 2 As shown, for each mode Single-mode characterization Feature decoupling is achieved using two independent lightweight MLP projectors: Shared feature projection head It consists of the following layers: linear transformation layer → ReLU activation function → Dropout regularization → linear transformation layer. The private feature projection head structure is the same as the shared feature projection head.

[0034] Step S3.1: Project the image using two projector heads. Mapping to shared features and private features .

[0035] Step S3.2: Calculate the difference loss Force orthogonality constraints between shared and private features: ; in and These are the normalized shared and private representation matrices, respectively. Minimize the Frobenius norm. This makes the dot product between shared and private representations approach zero, thus achieving geometric orthogonality between the two subspaces.

[0036] Step S4: Apply a three-level hierarchical contrastive learning objective to the purified shared features to achieve progressive alignment of cross-modal semantics.

[0037] The three-level hierarchical contrastive learning in step S4 specifically includes: Step S4.1: First-level cross-modal alignment, performing pairwise semantic synchronization between purified shared features of different modalities, for modal pairs Each pair in the diagram is mapped to a low-dimensional space through its respective linear projection head and then... Normalization is performed, and a symmetric cross-entropy loss function is used for contrastive learning to obtain the first-level contrastive loss. Complete the alignment of basic semantic levels between modalities.

[0038] Step S4.2: Second-level bimodal center alignment, constructing trimodal composite features by stitching and projecting shared representations. ,in As a linear projection layer, the trimodal composite features are compared and aligned with the shared features of text, acoustics, and vision, respectively, to obtain the second-level contrastive loss. ; Complete the deep fusion of intermediate representation levels between modalities.

[0039] Step S4.3: Third-level global fusion alignment. The fused representation output by the lightweight Transformer fusion module is symmetrically aligned with the purified shared features of each modality to obtain the third-level contrastive loss. Achieve alignment and fusion at the global semantic level.

[0040] All levels of contrastive learning use the InfoNCE loss function, which is calculated as follows: ; in , , Represents the cosine similarity matrix. Indicates positive and negative sample pairs. This represents the truth matrix of positive and negative sample pairs. The learnable temperature scaling parameter is initialized to [value]. .

[0041] Finally, the contrastive losses of the above three levels are integrated to obtain the overall contrastive loss, which guides the training and optimization of the multimodal model. The overall contrastive loss is: .

[0042] Step S5: Use the lightweight Transformer fusion module to perform cross-modal interactive fusion of shared features to obtain a fused representation.

[0043] Preferably, the specific process of the lightweight Transformer fusion module in step S5 includes: Step S5.1: Stack shared features: ,in This represents the number of modes.

[0044] Step S5.2: Input the stacked features into a shallow Transformer encoder for cross-modal interaction: The Transformer encoder contains a single Transformer coding layer, employs four attention heads and a 256-dimensional feedforward network, and has a Dropout rate of 0.1.

[0045] Step S5.3: Perform mean pooling on the Transformer output to obtain the fused representation: ; The lightweight Transformer fusion module performs cross-modal attention only on M=3 modal-level embeddings, reducing computational complexity from Reduce to This significantly reduces computational overhead.

[0046] Step S6: Concatenate the fused representation with the private features of the three modalities, and output the final sentiment analysis result through the prediction layer.

[0047] Step S6.1: Concatenate the fused representation with the private features of the three modalities: .

[0048] Step S6.2: Output the sentiment prediction result through the multilayer perceptron prediction layer: The prediction layer consists of a linear transformation layer, a ReLU activation function, a Dropout regularization layer, and another linear transformation layer in sequence.

[0049] The overall optimization objective function of the multimodal sentiment analysis method in this embodiment is: ; in The task loss function is calculated using the mean absolute error (MAE): ; and These are difference loss and overall contrast loss, respectively. and The weights for the difference loss and the overall contrast loss are determined using a grid search with a step size of 0.01. The optimizer used is the Adam optimizer, and the learning rate of the BERT module is 5 × 10⁻⁶. -5 The learning rate of the main network is 1×10. -4 The weight decays to 1×10.-4 The batch size is 128; the maximum number of training rounds is 50; an early stopping strategy is adopted, with a patience value of 10 rounds; and the gradient pruning threshold is 1.0.

[0050] This embodiment was validated on three publicly available benchmark datasets: (1) CMU-MOSI dataset: Contains 2199 English video clips of opinions from 93 YouTube movie review videos, each clip is labeled with a continuous sentiment intensity score in the range [-3, +3]. The standard split contains 1284 training samples, 229 validation samples and 686 test samples.

[0051] (2) CMU-MOSEI dataset: Contains 23,454 annotated English video clips from 1,000 different speakers on 250 different topics. Each clip is labeled with a sentiment score in the range of [-3, +3]. The official split includes 16,326 training samples, 1,871 validation samples and 4,659 test samples.

[0052] (3) CH-SIMS dataset: It contains 2281 Chinese video clips collected from movies, TV series and variety shows. Each clip is labeled with a global multimodal sentiment label in the range of [-1,+1] and an independent unimodal sentiment label. The dataset is divided into 1368 training samples, 456 validation samples and 457 test samples.

[0053] Each sample in each dataset contains unaligned sequence data in three modalities: text, visual, and acoustic. The input feature dimension for the text modality is 768 dimensions (extracted by the BERT model), while the input feature dimensions for the visual and acoustic modalities vary depending on the dataset.

[0054] Evaluation Protocol: Sentiment prediction was constructed as a regression task, with key metrics including Mean Absolute Error (MAE) and Pearson Correlation Coefficient (Corr). Binary classification metrics (Acc-2 and F1 scores) were obtained by thresholding continuous sentiment scores at 0. For the dataset above, the standard Has0 / Non0 protocol was used: Has0 includes neutral samples in the evaluation, while Non0 excludes them.

[0055] The experimental environment for this invention is as follows: Python version 3.8 or higher, using the open-source Python deep learning framework PyTorch in conjunction with CUDA for training.

[0056] Training hardware requirements: GPU model is NVIDIA RTX 4090, with 24GB of video memory.

[0057] To comprehensively compare the performance of the models, the following evaluation metrics were used in the experiment: mean absolute error (MAE), correlation coefficient (Corr), binary classification accuracy (Acc-2), and F1 score.

[0058] The formula for calculating MAE is: .

[0059] Acc-2 performs evaluations under two protocols: Has0 (including zero-value samples) and Non0 (excluding zero-value samples), which classify non-negative / negative samples and positive / negative samples into two classes, respectively.

[0060] Tables 1-3 below present the experimental results on the three datasets respectively; it should be noted that in Tables 1-3, for the Acc-2 and F1 indicators, Has0 and Non0 scores are separated by " / ".

[0061] Table 1: Experimental Results of the CMU-MOSI Dataset As shown in Table 1, the L-HDCL framework proposed in this invention achieves state-of-the-art performance on the CMU-MOSI dataset for binary sentiment classification, achieving the highest Acc-2 and F1 scores under both Has0 and Non0 evaluation protocols (Has0Acc-2: 84.6%, Non0Acc-2: 86.9%). This result outperforms recent competitive benchmark models, including the efficient Transformer model EMT (85.0% for Non0Acc-2) and the adapter-based method MMA (86.4% for Non0Acc-2). Although MMA has a slight advantage in regression metrics (mean absolute error MAE and correlation coefficient Corr), the L-HDCL of this invention performs better in sentiment polarity determination. This confirms that the progressive hierarchical contrastive learning mechanism proposed in this invention can effectively enhance the semantic decision boundary in the latent space. Compared to the lightweight CubeMLP architecture, the L-HDCL model in this patent improves the Non0Acc-2 accuracy from 85.6% to 86.9% and reduces the MAE from 0.770 to 0.714, demonstrating its excellent characterization capabilities without relying on large-scale parameterization.

[0062] Table 2: Experimental Results of the CMU-MOSEI Dataset As shown in Table 2, in the larger and more complex CMU-MOSEI benchmark, our patented model (L-HDCL) achieved the best performance in Has0 partitioning (Acc-2: 83.8 / 85.6, F1: 83.7 / 85.5). In terms of relevance metrics, L-HDCL achieved a competitive score of 0.779, ranking second best, only 0.001 behind ConFEDE (0.780). Although EMT and ConFEDE have slightly higher Non0 accuracy, our patented model L-HDCL maintains a robust balance in both Has0 and Non0 scenarios, while its computational cost is lower than these benchmark models that rely heavily on Transformer architectures.

[0063] Table 3: Experimental Results of CH-SIMS Dataset As shown in Table 3, the patented model L-HDCL achieved state-of-the-art performance across all evaluation metrics. Compared to the competitive Self-MM benchmark model, L-HDCL reduced MAE from 42.50 to 41.72 and improved Acc-2 from 80.04% to 80.67%. This sustained advantage, demonstrated in both continuous regression and binary classification tasks, indicates that hierarchical contrast alignment effectively compensates for the inherent modal gaps in Chinese multimedia data. Chinese emotional expression often exhibits subtle modal differences, such as the contrast between implicit textual semantics and strong speech or facial expressions. L-HDCL's architecture addresses this issue: the contrast mechanism synchronizes the underlying consensus, while the decoupling module isolates language-independent expressive cues into a private subspace, preventing them from being masked during the final trimodal fusion process.

[0064] Furthermore, Figures 3-4 This demonstrates a visual analysis of the patented L-HDCL model on the MOSI dataset; Figure 3 The learned modal features were visualized, showing that shared and private embeddings occupy largely separate regions with very low overlap. This spatial separation validates the effectiveness of the soft orthogonality constraint, demonstrating that the proposed model effectively reduces redundancy between subspaces and establishes a clear shared space for cross-modal alignment.

[0065] Figure 4 This is a representation of the effect after hierarchical contrastive learning and final fusion, with colors encoded according to true emotional polarity (positive and negative). Centroids of each class are calculated in the projected 2D space, indicated by black crosses, and the Euclidean distance between the centroids is 8.26. This visualization demonstrates compact intra-class clustering and clear inter-class separation; this topological structure directly reflects the goal of hierarchical contrastive learning, which requires consistent semantic organization at multiple granularities.

[0066] In summary, the method of this invention can effectively solve the problems of modal heterogeneity, noise interference and computational efficiency in multimodal sentiment analysis. While ensuring that the accuracy is better than similar methods, it achieves a significant lightweight effect, which is especially suitable for practical engineering deployment in resource-constrained scenarios.

[0067] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A multimodal sentiment analysis method based on lightweight hierarchical decoupling contrastive learning, characterized in that: Includes the following steps: Step S1: Obtain the text containing Visual Harmony A multimodal sentiment dataset with three modalities was prepared, and the data for each modality was preprocessed. Step S2: Use a modality-specific encoder to encode the single-modal features of the three modalities respectively, and obtain the representation vector of each modality; Step S3: Through the feature decoupling module, the representation vector of each mode is decomposed into mode-invariant shared features and mode-specific private features, and orthogonal constraint loss is used to ensure strict separation of the two types of features; Step S4: Apply a three-level hierarchical contrastive learning objective to the purified shared features to achieve progressive alignment of cross-modal semantics; Step S5: Use the lightweight Transformer fusion module to perform cross-modal interactive fusion of shared features to obtain a fused representation; Step S6: Concatenate the fused representation with the private features of the three modalities, and output the final sentiment analysis result through the prediction layer.

2. The multimodal sentiment analysis method based on lightweight hierarchical decoupling contrastive learning according to claim 1, characterized in that: The multimodal sentiment dataset in step S1 contains aligned or unaligned language sequences. Visual sequences Harmony and acoustic sequence The time length of each modal sequence and input feature dimensions different.

3. The multimodal sentiment analysis method based on lightweight hierarchical decoupling contrastive learning according to claim 1, characterized in that: The single-modal feature encoding in step S2 specifically includes: Step S2.1: For the text modality, a pre-trained BERT encoder is used to extract text representations, and the embeddings corresponding to the [CLS] tags are taken as sentence-level text representations. ,in , For text representation dimensions; Step S2.2: For visual and acoustic modalities, a bidirectional long short-term memory network (BiLSTM) is used for encoding. By concatenating the final hidden states in the forward and backward directions, modal-level representations are obtained. and ,in , These are the visual and acoustic representation dimensions, respectively.

4. The multimodal sentiment analysis method based on lightweight hierarchical decoupling contrastive learning according to claim 3, characterized in that: In step S2, the BERT encoder uses a BERT-base-uncased pre-trained model for the English dataset and a Chinese-BERT pre-trained model for the Chinese dataset; the BiLSTM encoder has a hidden layer dimension of 64, a layer count of 1, and a dropout rate of 0.25, and obtains visual representations by concatenating the forward and backward hidden states. Harmony and acoustic characterization .

5. The multimodal sentiment analysis method based on lightweight hierarchical decoupling contrastive learning according to claim 1, characterized in that: The feature decoupling module in step S3 performs each mode Single-mode characterization Perform the following operations: Step S3.1: Utilize two lightweight MLP projection heads and ,Will Mapped to shared features respectively and private features ,Right now , ,in , For common feature dimensions; each MLP projection head consists of a linear transformation layer, a ReLU activation function, a Dropout regularization layer, and a linear transformation layer in sequence; Step S3.2: Calculate the difference loss Force orthogonality constraints between shared and private features: ； in and These are the shared and private representation matrices after zero-mean normalization, respectively. It is the Frobenius norm.

6. The multimodal sentiment analysis method based on lightweight hierarchical decoupling contrastive learning according to claim 1, characterized in that: The three-level hierarchical contrastive learning in step S4 specifically includes: Step S4.1: First-level cross-modal alignment, performing pairwise semantic synchronization between purified shared features of different modalities, for modal pairs Each pair in the diagram is mapped to a low-dimensional space through its respective linear projection head and then... Normalization is performed, and a symmetric cross-entropy loss function is used for contrastive learning to obtain the first-level contrastive loss. Complete the alignment of basic semantics between modalities; Step S4.2: Second-level bimodal center alignment, constructing trimodal composite features by stitching and projecting shared representations. ,in As a linear projection layer, the trimodal composite features are compared and aligned with the shared features of text, acoustics, and vision, respectively, to obtain the second-level contrastive loss. ; Complete deep fusion of intermediate representation levels between modalities; Step S4.3: Third-level global fusion alignment. The fused representation output by the lightweight Transformer fusion module is symmetrically aligned with the purified shared features of each modality to obtain the third-level contrastive loss. Achieve alignment and fusion at the global semantic level; Finally, the three levels of contrastive loss are integrated to obtain the overall contrastive loss, which guides the training and optimization of the multimodal model. The overall contrastive loss is: .

7. A multimodal sentiment analysis method based on lightweight hierarchical decoupling contrastive learning according to claim 6, characterized in that: In step S4, all levels of contrastive learning employ the CLIP-style InfoNCE symmetric loss function, calculated as follows: ; in , , Represents the cosine similarity matrix. Indicates positive and negative sample pairs. This represents the truth matrix of positive and negative sample pairs. This is a learnable temperature scaling parameter.

8. The multimodal sentiment analysis method based on lightweight hierarchical decoupling contrastive learning according to claim 1, characterized in that: The specific process of the lightweight Transformer fusion module in step S5 includes: Step S5.1: Stack the shared features of the three modalities: ,in For the number of modes, For common feature dimensions; Step S5.2: Input the stacked features into a shallow Transformer encoder for cross-modal interaction: The Transformer encoder contains only a single Transformer coding layer, employs four attention heads and a 256-dimensional feedforward network, and has a Dropout rate of 0.

1. Step S5.3: Perform mean pooling on the Transformer output to obtain the fused representation: ; The lightweight Transformer fusion module only Cross-modal attention is performed on each modal-level embedding.

9. A multimodal sentiment analysis method based on lightweight hierarchical decoupling contrastive learning according to claim 1, characterized in that: The final prediction in step S6 specifically includes: Step S6.1: Concatenate the fused representation with the private features of the three modalities: ; Step S6.2: Output the sentiment prediction result through the multilayer perceptron prediction layer: The prediction layer consists of a linear transformation layer, a ReLU activation function, a Dropout regularization layer, and another linear transformation layer in sequence.

10. A multimodal sentiment analysis method based on lightweight hierarchical decoupling contrastive learning according to claim 1, characterized in that: The overall optimization objective function of the method is: ; in The task loss function is calculated using the mean absolute error (MAE): ; and These are difference loss and contrast loss, respectively. and The weights for the difference loss and the contrast loss are determined by grid search with a step size of 0.01; the optimizer is the Adam optimizer, and different learning rates are set for the BERT module and the main network. Early stopping and gradient clipping are employed during training.

Images