A multi-flow chain perception-enhanced multi-modal aspect-level sentiment analysis method

Abstract

A kind of multi-flow chain type perception enhanced multimodal aspect-level sentiment analysis method is referred to as MCPE model, it is related to multimodal sentiment analysis technical field.The present application solves the problem of low accuracy of fine-grained sentiment analysis caused by the difference of information density within modal and the imbalance of information between modal in the prior art.The present application comprises: obtaining a multi-modal text-image pair to be analyzed, inputting the text-image pair into the trained MCPE model to obtain aspect words and their sentiment polarity;The MCPE model comprises: a feature extraction module, a chain enhancement module, a multi-flow interaction module and a classifier;Text features are extracted by BART and image features are extracted by Faster R-CNN in the feature extraction;The chain enhancement module suppresses image and text noise and enhances fine-grained semantics through IFE-TFE double-chain architecture;The multi-flow interaction module realizes the dynamic complementary fusion of text reasoning and visual evidence by using bidirectional cross-modal attention;The classifier outputs the analysis result based on the fusion features, and does not require external tools.The present application is suitable for sentiment analysis of social media comments.

Description

Technical Field

[0001] This invention relates to the field of multimodal sentiment analysis technology, specifically to a multi-stream chain-based perception enhancement multimodal aspect-level sentiment analysis method, abbreviated as MCPE model. Background Technology

[0002] Multimodal aspect-level sentiment analysis (MABSA) is a complex and challenging research field with significant applications in social media sentiment analysis. It aims to accurately capture and interpret user emotional responses from diverse information sources. Its core tasks fall into three categories: Multimodal Aspect-Level Terminology Extraction (MATE), Multimodal Aspect-Level Sentiment Classification (MASC), and Joint Multimodal Aspect-Level Sentiment Analysis (JMASA). MATE aims to identify and extract specific aspect terms from given multimodal data. MASC aims to determine the sentiment tendencies of these aspects. JMASA is a comprehensive task that simultaneously extracts and classifies aspects and their sentiments.

[0003] As a hot challenge in the field of multimodal learning, multimodal learning algorithms have received widespread attention. Previous sentiment analysis research mainly relied on the fusion of visual and textual information. In 2021, the concept of the JMASA task was first defined, and a joint learning method, JML, combining image-text relationship mechanisms, was proposed to evaluate the contribution of visual content to the extraction of aspect-sentiment pairs. Subsequently, in 2022, a task-specific visual language pre-training framework, VLP-MABSA, was proposed for MABSA. This is a unified multimodal encoder-decoder architecture used for all pre-training and downstream tasks. Recently, a multi-task learning architecture called CMMT was designed for the end-to-end MABSA task. In 2023, an aspect-oriented method, AoM, was proposed to detect aspect-related semantic and sentiment information. During the same period, cross-modal fine-grained alignment and fusion networks were proposed to improve the performance of visual language models in seamlessly integrating visual and linguistic information. Recently, to align images and text in the language modality at multiple granularities, the Atlantis training model-assisted discrimination was introduced, an innovative aesthetic-oriented approach for the JMASA task. While current methods have made impressive progress, they all overlook two key issues: intramodal information density differences and intermodal information imbalances. Summary of the Invention

[0004] To address the issues of varying intramodal information density and intermodal information imbalance in existing methods, this invention proposes a multimodal aspect-level sentiment analysis method based on multi-stream chain-based perception enhancement.

[0005] The specific process of a multi-stream chain-based perception enhancement multimodal aspect-level sentiment analysis method is as follows:

[0006] Obtain the multimodal text-image pairs to be analyzed, input the text-image pairs into the trained model MCPE, and output aspect words and their sentiment polarity classification results;

[0007] The trained MCPE model is obtained through the following methods:

[0008] Step 1: Construct a training set using a multimodal dataset labeled with aspect terms and sentiment polarity;

[0009] Step 2: Train the initial MCPE model using the training set to obtain the final MCPE model with optimized weights;

[0010] The MCPE model includes: a feature extraction module, a chained enhancement module, a multi-stream interaction module, and a classifier.

[0011] The feature extraction module uses a pre-trained model to extract image text features, specifically:

[0012] First, text features are extracted using the BART text encoder;

[0013] Then, the Faster R-CNN object detector is used to extract image region features;

[0014] The chain-based enhancement module includes a parallel text feature enhancement chain (TFE) and an image feature enhancement chain (IFE), which perform intra-modal noise suppression and fine-grained semantic enhancement on text features and image features, respectively.

[0015] The multi-stream interaction module achieves dynamic complementary fusion of text and visual features through a bidirectional cross-modal attention mechanism.

[0016] The classifier outputs aspect words and their sentiment classification results based on fused features.

[0017] Furthermore, the feature extractor includes: a pre-trained text encoder BART and a pre-trained visual encoder Faster R-CNN.

[0018] The pre-trained text encoder BART is a Transformer-based pre-trained language model that excels at natural language understanding and generation tasks, and can extract semantic features H from text. T ∈R n×d Where n is the length of the text sequence and d is the feature dimension, and then the context information is captured through an encoder-decoder structure;

[0019] The pre-trained visual encoder Faster R-CNN is a classic object detection model that can extract visual features V∈R from images. d×36 Candidate target regions are generated through a region proposal network, and features are then extracted from these regions for classification and localization.

[0020] Then H T And V parallel input chain enhancement module CEM.

[0021] Furthermore, the chain-based enhancement module CEM achieves intramodal feature optimization through an IFE and TFE dual-chain architecture;

[0022] A1. The Text Feature Enhancement (TFE) chain performs fine-grained feature enhancement on the original text features, specifically as follows:

[0023] Step 1: Analyze text features H T Perform self-attention H attn Calculation:

[0024]

[0025] Among them W Q W K W V ∈Rw ×d These are the learnable query, key, and value projection matrices, respectively. It is the scaling factor;

[0026] Step 2: H attn ∈R n×d Split into h heads to generate heterogeneous query projection Q i :

[0027]

[0028] Where d / h is the feature dimension of each head;

[0029] Step 3: Construct a shared key-value space and Specifically:

[0030]

[0031] in This is the key-value projection matrix;

[0032] Step 4: Perform CrossAttn i Calculation:

[0033]

[0034] Step 5: Integrate multi-head attention H through gating mechanisms cross And output:

[0035] H cross=σ(Γ)⊙(Concat(CrossAttn1,…,CrossAttn h W o )

[0036] Among them W O ∈R n×d The output projection matrix is ​​σ(Γ), which is the introduced learnable gating parameter, σ is the Sigmoid activation function, ⊙ represents element-wise multiplication, and Concat is the concatenation operation.

[0037] Step 6: To further balance the attention output with the original feature distribution and enhance the model's robustness, H... cross Layer normalization and feedforward network processing are performed sequentially to output an enhanced representation with global consistency.

[0038]

[0039] Where FFN stands for feedforward network operation and LayerNorm stands for layer normalization operation;

[0040] Step 7: To simulate the adversarial training mechanism, adversarial perturbation is introduced to force the model to maintain feature robustness under small noise interference, thereby enhancing the stability of feature representation during chain augmentation and obtaining the adversarial perturbation representation H. pert :

[0041]

[0042] Where Φ adv () represents the disturbance generator, and α is the disturbance intensity coefficient;

[0043] Step 8: To avoid gradient instability caused by differences in numerical ranges in subsequent operations, a learnable gating vector α is used to enhance the perturbation feature H. pert Compared with the original text features H T Perform weighted fusion to generate intermediate fusion features H res :

[0044] H res =LayerNorm(α⊙H pert +(1-α)⊙H T )

[0045] Where H T For text features, α is a learnable gating vector;

[0046] Step 9: Based on intermediate fusion feature H resThe mean μ and standard deviation σ are calculated, and global statistics are extracted to eliminate distributional differences between samples or levels. The normalized features are dynamically adjusted using a learnable scaling parameter γ and a translation parameter β to obtain the intermediate layer features.

[0047]

[0048] Step 10: Recursively fuse all intermediate layer features, explicitly model cross-level dependencies, and improve global semantic consistency through multi-scale feature aggregation, thereby obtaining enhanced text features H. fused :

[0049]

[0050] in Represents the features from layer 0 to the i-th layer;

[0051] A2. The Image Feature Enhancement (IFE) chain performs fine-grained feature enhancement on the original image features, specifically as follows:

[0052] Step 1: Calculate the spatial code s for the image region coordinates (x1, t1, x2, y2).

[0053]

[0054] Where W and H are the image width and height, respectively. A represents the area of ​​the target detection box;

[0055] Step 2: Generate a high-dimensional spatial code Es through nonlinear mapping, and then convert the generated high-dimensional spatial code E... s Spatial visual features are obtained by fusing the transpose transformation with the original visual feature V;

[0056] Step 3: Apply convolution and attention mechanisms to spatial visual features to obtain enhanced visual feature representations. The feature structure is then normalized, adjusted via linear projection, and redundancy in multi-head attention is eliminated. A dynamically gated Sigmoid activation function is used to generate weights σ(W). g ) Screening out important features V final :

[0057] V final =LayerNorm(GELU(V reorg W1)⊙σ(V reorg W g )+V norm1 )

[0058] Where V norm1 This represents the result of the first normalization of visual features, using V. reorg=V norm1 W reorg Perform feature recombination, W reorg It is a learnable weight matrix, and GELU is the activation function;

[0059] Step 4: Construct multi-scale pyramid feature V pyramid :

[0060]

[0061] Where AdaptiveMaxPool1d is a one-dimensional adaptive max pooling operation, ReLU is the activation function, T represents the transpose operation, and Conv1D is a one-dimensional convolution operation;

[0062] Step 5: Pool all pyramid features to a fixed length L=20, and then concatenate them along the feature dimensions to form a multi-scale fused visual feature V. fused :

[0063] V fused =Concat(AdaptiveMaxPool1d L＝20 (V pyramid ))

[0064] Furthermore, the specific process by which the multi-stream interaction module (MIM) achieves dynamic complementary fusion is as follows:

[0065] The first stage involves text-guided visual enhancement, using the enhanced text features as the query Q. text Visual features as key K vis Sum V vis The first stage uses an improved multi-head cross-modal attention mechanism to filter out the visual regions most relevant to the text. The second stage is visual-guided text enhancement, which uses the enhanced visual features output from the first stage as the query Q. vis Text features as key K text Sum V text Further optimize text features to align with visual context, and finally fuse them into a multimodal feature F with rich information; Step 1: First, use the chained enhanced text features H fused As a query (Q) text );

[0066] Secondly, the image features V after chain enhancement. fused As a key (K) vis ) and value (V) vis );

[0067] Finally, the visual features of the text guidance were calculated;

[0068] Step 2: First, use the visual features output from Step 1 as the query (Q).vis );

[0069] Secondly, the text feature H after chain-enhanced processing... fused As a key (K) text ) and value (V) text );

[0070] Finally, the text features of the visual guidance were calculated;

[0071] Step 3: Finally, the visual and text features calculated in Step 1 and Step 2 are fused into a multimodal feature F with rich information.

[0072] The multi-stream interaction module (MIM) performs noise filtering on the text image features after mutual interaction guidance, generating dual-channel pooled features F. pool :

[0073] F pool =Concat(GAP(F),GMP(F))

[0074] GAP stands for average pooling operation, and GMP stands for max pooling operation.

[0075] Furthermore, regarding F pool Redundant channels are dynamically suppressed through a noise filtering layer, resulting in the feature F after redundancy suppression. ′ :

[0076] F ′ =σ(MLP(NFL(F) pool )))⊙F

[0077] Where σ is the Sigmoid activation function, MLP is a multilayer perceptron, and NFL is a noise filtering layer.

[0078] Furthermore, the Lightweight Enhancement Layer (LEL) compresses the input features, concatenates them, and then applies a convolution operation. Finally, the original features are recalibrated through element-wise multiplication to obtain the integrated final output F″. This feature is input to a classifier composed of BartDecoders, which processes it and outputs the classification result.

[0079]

[0080] Where C is the number of channels in the input feature, F (c) It is the c-th channel of the input feature.

[0081] The beneficial effects of this invention are as follows:

[0082] The proposed MCPE model in this invention is used for multimodal aspect-level sentiment analysis. It achieves intramodal feature enhancement through the TFE-IFE dual-chain architecture of the Chain Enhancement Module (CEM): the text chain TFE suppresses noise and enhances semantic association through cross-head attention collaboration and multi-layer feature fusion, while the image chain IFE filters interference regions and focuses on key visual information using spatial coding and multi-scale fusion. This significantly improves the purity and fineness of intramodal features, laying a high-quality foundation for cross-modal interaction. This invention also designs a Multi-Stream Interaction Module (MIM), breaking through traditional modal feature interaction methods. It generates joint representations through the complementary fusion of textual abstract reasoning and concrete image evidence, solving the problems of different information densities within modalities and information imbalance between modalities, thus enhancing cross-modal semantic consistency. This invention also introduces a global enhancement mechanism in multi-stream interactions. This mechanism optimizes the overall features through noise filtering and a lightweight enhancement layer, enabling the MCPE model to significantly outperform existing methods on the JMASA, MASC, and MATE tasks on the Twitter-2015 and Twitter-2017 benchmark datasets without introducing additional tools. This improves the accuracy of aspect word extraction and sentiment classification, thereby obtaining better multimodal fine-grained sentiment analysis results. Attached Figure Description

[0083] Figure 1 This is a framework diagram of the MCPE proposed in this invention;

[0084] Figure 2 TFE framework diagram for text feature enhancement;

[0085] Figure 3 IFE framework diagram for image feature enhancement;

[0086] Figure 4 This is a framework diagram of the Multi-Stream Interaction Module (MIM). Detailed Implementation

[0087] The MCPE model mainly consists of four parts: a feature extraction module, a chained enhancement module (CEM), a multi-stream interaction module (MIM), and a classifier. The feature extraction module generates text features based on pre-trained BART, and extracts 36 region features using Faster R-CNN. The CEM module achieves intra-modal noise suppression and fine-grained semantic enhancement through a TFE-IFE dual-chain architecture. The MIM module achieves dynamic complementary fusion of text and visual information through a multi-head cross-modal attention mechanism. The classifier finally outputs the classification. This design solves the problems of intra-modal information density differences and inter-modal information imbalance, significantly improving the performance of multimodal fine-grained sentiment analysis without introducing additional pre-trained models. Figure 1 The overall architecture of the MCPE model is shown. The invention will then be described in conjunction with specific implementation methods.

[0088] Specific Implementation Method 1: Implement a multi-stream chain-based perception enhancement multimodal aspect-level sentiment analysis method, the specific process of which is as follows:

[0089] Obtain the text-image pair to be detected, input the text-image pair into the trained MCPE model, and obtain the detection result;

[0090] The trained MCPE model is obtained through the following methods:

[0091] Step 1: Use the original text and original images labeled with aspect terms and sentiment polarity to form a training dataset;

[0092] The training sets are the Twitter-2015 dataset and the Twitter-2017 dataset;

[0093] Step 2: Train the initial MCPE model using the training set to obtain the final MCPE model with optimized weights;

[0094] like Figure 1 As shown, the MCPE model includes: a feature extraction module, a chained enhancement module (CEM), a multi-stream interaction module (MIM), and a classifier;

[0095] The feature extraction module first uses the BART text encoder to extract text features H. T ∈R n×d Then, the object detector Faster R-CNN is used to extract image region features V∈R. d×36 The obtained text image features are then input into the chained enhancement module CEM;

[0096] Where n is the length of the text sequence and d is the feature dimension.

[0097] The chain-like enhancement module consists of a text feature enhancement chain and an image feature enhancement chain working together. First, it enhances the received text features H... T Cross-head attention collaboration, multi-level feature fusion, and adversarial perturbation enhancement are performed to ultimately output enhanced text features H. fused Then, the image feature enhancement chain performs spatial encoding, multi-scale convolutional filtering, and attention weighting on V, outputting fused and enhanced visual features V. fused ;

[0098] The multi-stream interaction module first uses the enhanced text features as the query Q. text Visual features as key K vis Sum V vis Then, the enhanced visual features output from the previous stage are used as the query Q. vis Text features as key K text Sum V textThe text features are mutually guided to optimize and align with the visual context, and finally the feature purity is improved through the multi-stream interaction module MIM.

[0099] The classifier generates the final aspect word sentiment classification result through the Bart encoder;

[0100] In this embodiment, the text encoder and image encoder are executed only once.

[0101] Specific Implementation Method Two: The chain-like enhancement module, through the synergistic action of the text feature enhancement chain and the image feature enhancement chain, is used to perform fine-grained feature enhancement on the received original text image features, specifically as follows:

[0102] A1, such as Figure 2 As shown, fine-grained feature enhancement of the original text features is performed through the Text Feature Enhancement Chain (TFE), specifically as follows:

[0103] Step 1: Analyze text features H T Perform self-attention calculation:

[0104]

[0105] Among them W Q W K W V ∈R n×d These are the learnable query, key, and value projection matrices, respectively. It is the scaling factor;

[0106] Step 2: H attn ∈R n×d Split into h heads to generate heterogeneous query projection Q i :

[0107]

[0108] Where d / h is the feature dimension of each head;

[0109] Step 3: Construct a shared key-value space and Specifically:

[0110]

[0111] in This is the key-value projection matrix;

[0112] Step 4: Perform CrossAttn i Calculation:

[0113]

[0114] Step 5: Integrate multi-head attention H through gating mechanisms cross And output:

[0115] H cross =σ(Γ)⊙(Concat(CrossAttn1,…,CrossAttn h W o )

[0116] Among them W O ∈R n×d The output projection matrix is ​​σ(Γ), which is the introduced learnable gating parameter, σ is the Sigmoid activation function, ⊙ represents element-wise multiplication, and Concat is the concatenation operation.

[0117] Step 6: To further balance the attention output with the original feature distribution and enhance the model's robustness, H... cross Layer normalization and feedforward network processing are performed sequentially to output an enhanced representation with global consistency.

[0118]

[0119] Where FFN stands for feedforward network operation and LayerNorm stands for layer normalization operation;

[0120] Step 7: To simulate the adversarial training mechanism, adversarial perturbation is introduced to force the model to maintain feature robustness under small noise interference, thereby enhancing the stability of feature representation during chain augmentation and obtaining the adversarial perturbation representation H. pert :

[0121]

[0122] Where Φ adv () represents the disturbance generator, and α is the disturbance intensity coefficient;

[0123] Step 8: To avoid gradient instability caused by differences in numerical ranges in subsequent operations, a learnable gating vector α is used to enhance the perturbation feature H. pert Compared with the original text features H T Perform weighted fusion to generate intermediate fusion features H res :

[0124] H res =LayerNorm(α⊙H pert +(1-α)⊙H T )

[0125] Where H T For text features, α is a learnable gating vector;

[0126] Step 9: Based on intermediate fusion feature H res The mean μ and standard deviation σ are calculated, and global statistics are extracted to eliminate distributional differences between samples or levels. The normalized features are dynamically adjusted using a learnable scaling parameter γ and a translation parameter β to obtain the intermediate layer features.

[0127]

[0128] Step 10: Recursively fuse all intermediate layer features, explicitly model cross-level dependencies, and improve global semantic consistency through multi-scale feature aggregation, thereby obtaining enhanced text features H. fused :

[0129]

[0130] in Represents the features from layer 0 to the i-th layer;

[0131] A2, such as Figure 3 As shown, fine-grained feature enhancement of the original image features is performed through the Image Feature Enhancement Chain (IFE), specifically as follows:

[0132] Step 1: Calculate the spatial code s for the image region coordinates (x1, y1, x2, y2).

[0133]

[0134] Where W and H are the width and height of the image, respectively, and A represents the area of ​​the target detection box;

[0135] Step 2: Generate a high-dimensional spatial code Es through nonlinear mapping, and then convert the generated high-dimensional spatial code E... s Spatial visual features are obtained by fusing the transpose transformation with the original visual feature V;

[0136] Step 3: Apply convolution and attention mechanisms to spatial visual features to obtain enhanced visual feature representations. The feature structure is then normalized, adjusted via linear projection, and redundancy in multi-head attention is eliminated. A dynamically gated Sigmoid activation function is used to generate weights σ(W). g ) Screening out important features V final :

[0137] V final =LayerNorm(GELU(V reorg W1)⊙σ(V reorg W g )+V norm1 )

[0138] Where V norm1 This represents the result of the first normalization of visual features, using V. reorg =V norm1 W reorg Perform feature recombination, W reorg It is a learnable weight matrix, and GELU is the activation function;

[0139] Step 4: Construct multi-scale pyramid feature V pyramid :

[0140]

[0141] Where AdaptiveMaxPool1d is a one-dimensional adaptive max pooling operation, ReLU is the activation function, T represents the transpose operation, and Conv1D is a one-dimensional convolution operation;

[0142] Step 5: Pool all pyramid features to a fixed length L=20, and then concatenate them along the feature dimensions to form a multi-scale fused visual feature V. fused :

[0143] V fused =Concat(AdaptiveMaxPool1d L＝20 (V pyramid ))

[0144] In this implementation, the chain-based enhancement module (CEM) adopts a dual-chain architecture. The text feature enhancement chain (TFE) significantly suppresses irrelevant noise in the text and enhances fine-grained sentiment semantic associations through cross-head attention collaboration and adversarial perturbation training. The image feature enhancement chain (IFE) utilizes spatial coding technology to transform geometric positions into semantic vectors, combined with multi-scale pyramid fusion, to filter visual background interference and focus on key regions. This dual-chain architecture is the first to solve the problem of intramodal information density differences—sparse distribution of sentiment words in text and low proportion of target regions in images—thus providing high-purity, strongly correlated feature representations for subsequent multimodal interactions, thereby laying the foundation for accurate cross-modal alignment.

[0145] Specific implementation method three: such as Figure 4 As shown, the multi-stream interaction module receives enhanced text features and image features. First, through a text-guided visual enhancement mechanism, the enhanced text features are used as the query Q. text Visual features as key K vis Sum V vis By using an improved multi-head cross-modal attention mechanism, the most relevant visual regions to the text are selected. Next, a visually guided text enhancement mechanism is used to incorporate the guided visual features as the query Q. vis Text features as key Ktext Sum V text Further optimization of text features to align with visual context, ultimately merging them into a multimodal feature F rich in information.

[0146] The multi-stream interaction module (MIM) performs noise filtering on the text image features after mutual interaction guidance, generating dual-channel pooled features F. pool :

[0147] F pool =Concat(GAP(F),GMP(F))

[0148] GAP stands for average pooling operation, and GMP stands for max pooling operation.

[0149] For F pool Redundant channels are dynamically suppressed through a noise filtering layer, resulting in the feature F after redundancy suppression. ′ :

[0150] F ′ =σ(MLP(NFL(F) pool )))⊙F

[0151] Where σ is the Sigmoid activation function, MLP is a multilayer perceptron, and NFL is a noise filtering layer.

[0152] The Lightweight Enhancement Layer (LEL) compresses the input features, concatenates them, and then applies a convolution operation. Finally, the original features are recalibrated through element-wise multiplication to obtain the integrated final feature F″. This feature is input to a classifier composed of Bart Decoders, which processes it and outputs the classification result.

[0153]

[0154] Where C is the number of channels in the input feature, F (c) It is the c-th channel of the input feature.

[0155] This invention innovatively constructs the MCPE model, truly solving the core problems of intramodal information density differences and intermodal information imbalance in multimodal aspect-level sentiment analysis. Existing methods typically rely on modality alignment or external pre-trained models, which, while improving performance, struggle to address intramodal noise interference and cross-modal semantic alignment issues. For example, traditional methods fail to simultaneously enhance key semantic information when filtering noise through gating mechanisms, and simple attention interactions are insufficient to achieve complementary synergy between abstract textual reasoning and concrete image evidence. This invention, for the first time, decouples chained feature enhancement from multimodal interaction, achieving fine-grained intramodal optimization through a dual-chain architecture. The text feature enhancement chain TFE employs cross-head attention collaboration and multi-level feature fusion to accurately purify semantic noise and strengthen sentiment association, while adversarial perturbation training enhances feature robustness. The image feature enhancement chain IFE combines spatial coding and multi-scale pyramid fusion to suppress irrelevant visual regions and focus on key targets. Building on this foundation, the Multi-Stream Interaction Module (MIM) breaks through the semantic gap of traditional splicing and fusion, constructs a dynamic evidence closed loop of "text → image → text", guides the bidirectional correction of text and vision with multi-head cross-modal attention, and filters residual noise after interaction through a global enhancement layer, ultimately achieving complementary fusion of text sentiment cues and image visual evidence.

[0156] Example: To verify the beneficial effects of the present invention, the following experiments were conducted:

[0157] All experiments in this embodiment were conducted using the Twitter2015 and Twitter2017 detection datasets, and the main metrics of the model were validated. For the multimodal aspect sentiment classification (MASC) task, the accuracy (Acc) and F1 score of the prediction results for all aspects were comprehensively evaluated. For the combined multimodal sentiment analysis (JMASA) and multimodal aspect word extraction (MATE) tasks, the Micro-F1 score, precision (P) and recall (R) were used as evaluation metrics.

[0158] The example uses VLP-MABSA as the baseline method, which is a unified multi-mode encoder-decoder architecture used for all pre-training and downstream tasks. The pre-training task is trained for 40 epochs (batch size 64, learning rate 5e-5), and the downstream task is fine-tuned for 35 epochs (batch size 16, learning rate 5e-5). The multi-task tradeoff coefficients λ1–λ5 are all set to 1 to balance the task contributions.

[0159] The experimental results of the MCPE model and the baseline model in tasks such as Joint Multimodal Sentiment Analysis (JMASA), Multimodal Aspect Sentiment Classification (MASC), and Multimodal Aspect Word Extraction (MATE) are shown in Tables 1, 2, and 3, respectively.

[0160] Table 1 Comparison of different models on the JMASA task

[0161]

[0162] As shown in Table 1, different models exhibit differentiated performance on the Twitter2015 and Twitter2017 datasets of the JMASA task, with MCPE showing a significant advantage. In the text-based category, SPAN, D-GCN, and BART have evolved iteratively, with precision (P), recall (R), and F1 score gradually improving on both datasets. For example, in the Twitter2015 dataset, P increased from 53.7 to 62.9, but there is still room for improvement overall. Compared with the multimodal category, OSCGA shows balanced performance but limited improvement, RpBERT has poor adaptability, and while UMT, JML, and VLP-MABSA have made progress, they have not reached the optimal level. Models such as M2DF, Atlantis, MCPL-VLP, and RNG have gradually increased their F1 scores during iterations, but they still lag behind MCPE. MCPE achieved breakthroughs on two datasets: Twitter2015 dataset (P = 68.72, R = 70.81, F1 = 69.74), representing a 1.14 improvement in F1 compared to the RNG model on the same dataset and a 5.84 improvement compared to the best text-based BART model; and Twitter2017 dataset (P = 69.75, R = 71.24, F1 = 70.49), representing a 0.29 improvement in F1 compared to the RNG model on the same dataset and a 5.09 improvement compared to the best text-based BART model. In summary, while there have been iterations from text-based to Multimodal models in the JMASA task, MCPE, with its more efficient text feature processing and interaction modeling, comprehensively leads in core metrics.

[0163] Table 2 Comparison of different models on the MATE task

[0164]

[0165] Table 2 shows that different models exhibit differentiated performance on the Twitter2015 and Twitter2017 datasets for the MATE task, with MCPE showing a significant advantage. RAN achieved P=80.5, R=81.5, and F1=81.0 on the Twitter2015 dataset, and P=90.7, R=90.7, and F1=90.0 on the Twitter2017 dataset, indicating a certain foundation but significant room for improvement. UMT performed relatively weakly overall, achieving only 79.7 F1 on the Twitter2015 dataset and 86.7 on the Twitter2017 dataset, suggesting room for improvement in its adaptability to the MATE task. Models such as OSCGA, JML, and VLP-MABSA gradually increased their F1 scores with iterations, reflecting a deeper understanding of the task, but still fell short of the optimal performance. Models such as M2DF and MCPL-VLP continuously improved their metrics during iterations, but still lag behind MCPE. MCPE achieved performance breakthroughs on two datasets: Twitter2015 dataset (P=85.74, R=88.41, F1=87.05), a 0.35 improvement over the VLP-MABSA model (which performed well on the same dataset) and a 0.75 improvement over the M2DF model; and Twitter2017 dataset (P=92.44, R=93.92, F1=93.17), a 1.47 improvement over the VLP-MABSA model and a 0.77 improvement over the M2DF model. In summary, while performance has improved across basic models like RAN and iterative models like M2DF and MCPL-VLP in the MATE task, MCPE, with its more efficient feature mining, interaction, and modeling, leads in precision, recall, and F1 score.

[0166] Table 3 Comparison of different models on the MASC task

[0167]

[0168] Table 3 shows that different models exhibit differentiated performance on the Twitter2015 and Twitter2017 datasets for the MASC task, with MCPE showing a significant advantage. TomBERT achieved an ACC of 77.2 and an F1 score of 71.8 on the Twitter2015 dataset, while its ACC was 70.5 and F1 score was 68.0 on the Twitter2017 dataset, indicating room for performance improvement. Models such as CapTBERT, JML, and VLP-MABSA showed gradual changes in their ACC and F1 scores with iterations, reflecting adjustments in their task suitability, but they had not yet reached their optimal levels. Models like M2DF, MGFN-SD, and MCPL continuously improved their metrics during iterations; for example, MCPL achieved an ACC of 79.3 and an F1 score of 74.9 on the Twitter2015 dataset, and an ACC of 75.1 and an F1 score of 74.07 on the Twitter2017 dataset, but still lagged behind MCPE. Models such as A2II, MPF1, and DPFN also performed well, but not as well as MCPE. MCPE achieved performance breakthroughs on two datasets: ACC 80.01 and F1 76.11 on the Twitter2015 dataset, representing an improvement of 0.71 in ACC and 1.21 in F1 compared to MCPL, which performed well on the same dataset; and ACC 75.27 and F1 74.04 on the Twitter2017 dataset, showing an improvement of 0.17 in ACC compared to MCPL on the same dataset, maintaining its overall advantage. In conclusion, while there have been performance advancements from basic models like TomBERT to iterative models like MCPL-VLP in MASC tasks, MCPE demonstrates strong competitiveness in ACC and F1 scores due to its more efficient processing and modeling of text features, making it well-suited to the text analysis requirements of MASC tasks.

[0169] The results of adding each component to the MCPE model on different tasks are shown in Tables 4, 5 and 6, respectively.

[0170] Table 4 shows the results of the MCPE model on the JMASA task after adding different components.

[0171]

[0172] Table 5. Results of the MCPE model with different components added to the MATE task.

[0173]

[0174] As shown in Table 4, in the JMASA task, from the Baseline to the MCPE method, the precision (P), recall (R), and F1 score mostly showed a gradual improvement trend. For example, on the Twitter2015 dataset, the Baseline F1 score was 66.6, while MCPE improved it to 69.75; on the Twitter2017 dataset, the Baseline F1 score was 68.0, while MCPE was 70.49, indicating that the model improvement effectively enhanced the task performance. Furthermore, adding combinations such as CEM and Interaction could improve the metrics to varying degrees, demonstrating the positive role of these modules in feature processing and interaction modeling. However, after integration and optimization, MCPE achieved the best F1 score and other metrics on both subsets of the dataset, making it a superior solution.

[0175] Table 5 shows that TFE performs significantly well on the text-intensive Twitter-2015 dataset, while IFE performs better on the image-based multi-object Twitter-2017 dataset. The introduction of the dual-chain architecture CEM simultaneously improves the metrics of both datasets, validating the rationality of the proposed dual-chain architecture. The interaction module further enhances the complementarity between tasks through fine-grained information fusion, resulting in improved F1 scores for all three tasks.

[0176] Table 6 shows the results of the MCPE model after adding different components to the MASC task.

[0177]

[0178] As can be seen from Table 6, in the MASC task, it is worth noting that the effect of introducing GlobalEnhancement on the basis of the integrated CEM module is weaker than the combination of CEM and Interaction; while the effect is optimal when CEM, Interaction and GlobalEnhancement work together, which illustrates the necessity of performing overall feature optimization again after modal feature interaction in the Multi-Stream Interaction Module (MIM).

[0179] Therefore, it can be seen that the MCPE model proposed in this invention, a multimodal aspect-level sentiment analysis model, achieves intramodal noise suppression and fine-grained semantic enhancement of text and image features through the IFE-TFE dual-chain architecture of the chained enhancement module CEM. The multi-stream interaction module MIM achieves dynamic complementary fusion of text and vision through a multi-head cross-modal attention mechanism. The addition of a global feature enhancement strategy to the details of the MIM module solves the problems of noise residue and local misalignment after interaction, playing an important role in solving the problems of different information densities within modalities and information imbalance between modalities.

Claims

1. A multi-stream chain-based perceptual enhancement multimodal aspect-level sentiment analysis method, characterized in that... The specific process of the method is as follows: Obtain the multimodal text-image pairs to be analyzed, input the text-image pairs into the trained model MCPE, and obtain the aspect words and their sentiment polarity classification results; The trained MCPE model is obtained through the following methods: Step 1: Construct a training set using a multimodal dataset labeled with aspect terms and sentiment polarity; Step 2: Train the initial MCPE model using the training set to obtain the final MCPE model with optimized weights; The MCPE model includes: a feature extraction module, a chained enhancement module, a multi-stream interaction module, and a classifier. The feature extraction module uses a pre-trained model to extract image text features, specifically: First, text features are extracted using the BART text encoder; Then, the Faster R-CNN object detector is used to extract image region features; The chained enhancement module includes a parallel text feature enhancement chain (TFE) and an image feature enhancement chain (IFE), which perform intra-modal noise suppression and fine-grained semantic enhancement on text features and image features, respectively. The text feature enhancement chain (TFE) performs fine-grained feature enhancement on the original text features, specifically: enhancing the text features... Self-attention The calculation; Split into h headers to generate heterogeneous query projections ; Construct a shared key-value space ; Perform cross-head attention The calculation; the fusion of multiple heads through gating mechanisms And output; to further balance the attention output with the original feature distribution and enhance the model's robustness, for Layer normalization and feedforward network processing are performed sequentially to output an enhanced representation with global consistency. To simulate the adversarial training mechanism, adversarial perturbation is introduced to force the model to maintain feature robustness under small noise interference, thereby enhancing the stability of feature representation during chain augmentation and obtaining the adversarial perturbation representation. To avoid gradient instability caused by differences in numerical ranges in subsequent operations, learnable gating vectors are used. Enhanced features of perturbation Features of the original text Perform weighted fusion to generate intermediate fusion features Based on intermediate fusion features Calculate the mean and standard deviation It eliminates distributional differences between samples or levels by extracting global statistics, and the normalized features are processed by learnable scaling parameters. Translation parameters Dynamic adjustment calculations are performed to obtain intermediate layer features. ; By recursively fusing features from all intermediate layers, explicitly modeling cross-level dependencies, and improving global semantic consistency through multi-scale feature aggregation, enhanced text features are obtained. ; The multi-stream interaction module achieves dynamic complementary fusion of text and visual features through a bidirectional cross-modal attention mechanism. The classifier outputs aspect words and their sentiment classification results based on fused features.

2. The multi-stream chain-based perceptual enhancement multimodal aspect-level sentiment analysis method according to claim 1, characterized in that: The feature extractor includes: a pre-trained text encoder BART and a pre-trained visual encoder Faster R-CNN. The pre-trained text encoder BART is a Transformer-based pre-trained language model that excels at natural language understanding and generation tasks, and can extract semantic features from text. ,in The length of the text sequence. The feature dimension is used, and then contextual information is captured through an encoder-decoder structure; The pre-trained visual encoder Faster R-CNN is a classic object detection model that can extract visual features from images. Candidate target regions are generated through a region proposal network, and features are then extracted from these regions for classification and localization. Then and Parallel input chain enhancement module CEM.

3. The multi-stream chain-based perceptual enhancement multimodal aspect-level sentiment analysis method according to claim 2, characterized in that: The chain enhancement module CEM achieves intramodal feature optimization through a TFE and IFE dual-chain architecture; A1. The Text Feature Enhancement (TFE) chain performs fine-grained feature enhancement on the original text features, specifically as follows: Step 1: Text Features Self-attention Calculation: in These are the learnable query, key, and value projection matrices, respectively. It is the scaling factor; Step 2: Split into h headers to generate heterogeneous query projections : Where d / h is the feature dimension of each head; Step 3: Construct a shared key-value space Specifically: in , This is the key-value projection matrix; Step 4: Perform cross-head attention Calculation: Step 5: Integrate multiple heads through a gating mechanism And output: in To output the projection matrix, For the introduction of learnable gating parameters, It is the Sigmoid activation function. This indicates element-wise multiplication. For splicing operations; Step 6: To further balance the attention output with the original feature distribution and enhance the model's robustness, [the following steps are taken]. Layer normalization and feedforward network processing are performed sequentially to output an enhanced representation with global consistency. : in For feedforward network operation, For layer normalization operation; Step 7: To simulate the adversarial training mechanism, adversarial perturbation is introduced to force the model to maintain feature robustness under small noise interference, thereby enhancing the stability of feature representation during chain augmentation and obtaining the adversarial perturbation representation. : in To counter the disturbance generator, This is the disturbance intensity coefficient; Step 8: To avoid gradient instability caused by differences in numerical ranges in subsequent operations, a learnable gating vector is used. Enhanced features of perturbation Features of the original text Perform weighted fusion to generate intermediate fusion features : in For text features, For learnable gated vectors; Step 9: Based on intermediate fusion features Calculate the mean and standard deviation It eliminates distributional differences between samples or levels by extracting global statistics, and the normalized features are processed by learnable scaling parameters. Translation parameters Dynamic adjustment calculations are performed to obtain intermediate layer features. : Step 10: Recursively fuse all intermediate layer features, explicitly model cross-level dependencies, and improve global semantic consistency through multi-scale feature aggregation, thereby obtaining enhanced text features. : in Represents from 0 to the Layer features; A2. The Image Feature Enhancement (IFE) chain performs fine-grained feature enhancement on the original image features, specifically as follows: Step 1: Set the coordinates of the image region ( The spatial code is obtained by performing calculations. : Where W and H are the width and height of the image, respectively, and A represents the area of ​​the target detection box; Step 2: Generate high-dimensional space encoding through nonlinear mapping Encode the generated high-dimensional space Transpose transformation and original visual features By merging, spatial visual features are obtained; Step 3: Apply convolution and attention mechanisms to spatial visual features to obtain enhanced visual feature representations. The features are then normalized, and their structure is adjusted through linear projection to eliminate redundancy in multi-head attention. Weights are generated using a dynamically gated Sigmoid activation function. Filter out important features : in This represents the result of the first normalization of visual features, using... Perform feature recombination, It is a learnable weight matrix. For activation functions; Step 4: Construct multi-scale pyramid features : in This is a one-dimensional adaptive max pooling operation. Let T be the activation function, and let T represent the transpose operation. Step 5: Pool all pyramid features to a fixed length L=20, and then concatenate them along the feature dimensions to form a multi-scale fused visual feature. : 。 4. The multi-stream chain-based perceptual enhancement multimodal aspect-level sentiment analysis method according to claim 3, characterized in that: The specific process by which the Multi-Stream Interaction Module (MIM) achieves dynamic complementary fusion is as follows: Step 1: First, use the chained enhanced text features As a query ; Secondly, the image features after chain enhancement. As a key Sum ; Finally, the visual features of the text guidance were calculated; Step 2: First, use the visual features output from Step 1 as the query. ; Secondly, the text features enhanced by chaining. As a key Sum ; Finally, the text features of the visual guidance were calculated; Step 3: Finally, the visual text features calculated in Step 1 and Step 2 are fused into a multimodal feature F with rich information.

5. The multi-stream chain-based perceptual enhancement multimodal aspect-level sentiment analysis method according to claim 4, characterized in that: The multi-stream interaction module (MIM) performs noise filtering on the text image features after mutual interaction guidance. : GAP stands for average pooling operation, and GMP stands for max pooling operation.

6. The multi-stream chain-based perceptual enhancement multimodal aspect-level sentiment analysis method according to claim 5, characterized in that: right Redundant channels are dynamically suppressed through a noise filtering layer to obtain the features after redundancy suppression. : in The Sigmoid activation function is used, MLP is a multilayer perceptron, and NFL is a noise filtering layer.

7. The multi-stream chain-based perceptual enhancement multimodal aspect-level sentiment analysis method according to claim 6, characterized in that: The Lightweight Enhancement Layer (LEL) compresses the input features, concatenates them, and then applies a convolution operation. Finally, the original features are recalibrated through element-wise multiplication to obtain the integrated final features. This feature is input into a classifier consisting of Bart Decoders, which processes it and outputs the classification result. Where C is the number of channels in the input feature. It is the c-th channel of the input feature.

Images