Prompt-based double-layer cross-modal distillation learning multi-modal aspect sentiment analysis method, system, electronic device and readable storage medium

Abstract

The present application relates to a multi-modal aspect sentiment analysis method, system, electronic device and readable storage medium based on prompt-based double-layer cross-modal distillation learning, and belongs to the technical field of multi-modal sentiment analysis. The existing methods have the problem of insufficient adaptability when facing various visual noise interference in practical applications. Most of the existing sentiment analysis models are designed specifically for a particular scene, and are difficult to adapt to different scenes such as visual modal missing. The present application focuses on distillation learning and aspect perception, and guides the model to focus on aspect information through prompts constructed by aspect words. By minimizing the distillation loss between its output and the current best multi-modal aspect sentiment analysis model, the multi-modal representation can be selectively learned effectively, and even in the case of missing visual information or visual noise interference, the sentiment polarity of the aspect can be accurately predicted. The present application shows extremely high accuracy and robustness in the sentiment analysis task containing noise vision.

Description

Technical Field

[0001] The present invention relates to a multimodal sentiment analysis method, system, electronic device, and readable storage medium based on prompt-based double-layer cross-modal distillation learning, and belongs to the technical field of multimodal sentiment analysis. Background Art

[0002] In the digital society, social media platforms have become an important channel for the public to express emotions and share opinions, generating a large amount of multimodal data (including text, images, etc.). However, traditional text-only aspect-level sentiment analysis can no longer meet the increasingly complex information needs. Multimodal aspect-based sentiment analysis (MABSA) significantly improves the accuracy and comprehensiveness of sentiment analysis by fusing text and image information, making up for the shortcomings of a single modality in fine-grained sentiment analysis. Although multimodal sentiment analysis can cope with multimodal social media environments such as images and text, the interference of noisy data in real social media environments is inevitable. Existing multimodal sentiment analysis methods are usually based on idealized datasets and are difficult to adapt to complex noisy scenarios. Therefore, how to perform accurate and robust sentiment judgment in the presence of noise interference has become an important research challenge.

[0003] Early research focused on designing effective models that capture the relationship between aspect information and sentence context in text. For example, Li et al. applied graph convolutional networks to extract syntactic dependency features from parsed syntactic dependency trees and obtained aspect-aware syntactic representations through attention mechanisms and aspect masks. More recently, MABSA research has focused on improving the alignment between visual and text. For example, CMMT proposed a multi-aspect and sentiment detection task for cross-modal interactive learning. HIMT and VLP-MABSA leverage object detection methods to extract objects from visual information and exclude background noise. However, most MABSA methods perform poorly with visual noise. To address this, Li et al. reconstructed missing semantic features through adaptive feature optimization and knowledge integration self-distillation. Han et al. designed a dynamic learner with an adjustable context sliding window to supplement missing modal features. However, these methods still only partially address the visual noise problem. Current multimodal aspect sentiment analysis methods still have several shortcomings: 1. These models typically infer aspect sentiment polarity in text-only or multimodal environments and struggle to adapt to mixed noise scenarios. 2. Due to the modal differences and information redundancy between images and text, it is difficult to accurately align aspect words with image and text information.

[0004] To address these challenges, we propose a multimodal aspect sentiment analysis method using two-layer cross-modal distillation learning based on hints, aiming to address the challenges of multi-scene adaptability and noise robustness in multimodal aspect sentiment analysis (MABSA). Cross-modal visual distillation is a potential approach to integrate aspect-specific visual details to improve MABSA performance across multiple scenes, thereby achieving multi-scene adaptability. By leveraging text-image aligned data to learn cross-modal distillation, this strategy effectively addresses the challenges of multi-scene adaptability and robust representation, and can also be extended to text-only scenes. However, training MABSA models and performing cross-modal distillation from scratch to achieve text-image semantic alignment is often challenging. Since these two tasks are closely linked, they may negatively impact each other, making it difficult to learn robust cross-modal distillation capabilities. Building on the latest MABSA models, a hint-guided distillation method can focus on cross-modal visual distillation for multiple scenes. This approach ensures that the existing text-image alignment in MABSA is not affected while simultaneously learning cross-modal visual distillation through hints. Inspired by this, we propose a cue-based two-layer distillation learning strategy, leveraging the latest MABSA model, to learn the ability to capture valuable visual information in noisy or even imageless scenes through cross-modal distillation.

[0005] Furthermore, in current research on fine-grained sentiment analysis, the core task is to determine the sentiment polarity of aspect terms in text and image information. Effectively focusing on aspect information can help models more accurately understand context and improve sentiment classification accuracy. Based on this, we constructed a two-layer aspect representation based on hints to enhance the model's focus on aspect semantics.

[0006] In summary, this paper proposes a multimodal aspect sentiment analysis method based on prompt-based two-layer cross-modal distillation learning. Summary of the Invention

[0007] The technical problem to be solved by the present invention is: the present invention provides a multimodal sentiment analysis method, system, electronic device, and readable storage medium based on prompt-based two-layer cross-modal distillation learning, so as to solve the problems of poor multi-scene adaptability and noise robustness in multimodal sentiment analysis. The present invention improves the accuracy of sentiment classification.

[0008] The technical solution of the present invention is: a multimodal aspect sentiment analysis method based on prompt-based two-layer cross-modal distillation learning, the method comprising:

[0009] Step 1, prompt generation: construct prompts through text templates with aspect words as the core, and regard the text templates as extensions of aspect words to form prompts;

[0010] Step 2: Embedding text and visual representations: Use a pre-trained language model (such as BERT) to generate embedded representations for the input text and prompts; use an existing multimodal aspect sentiment analysis model (DQPSA) to extract visual aspect representations for the input visual information and prompts;

[0011] Step 3: Constructing shallow and deep aspect representations: Using a shallow attention mechanism to compute shallow aspect representations of text and prompts; using a multi-layer Transformer to further extract deep aspect representations;

[0012] Step 4. Gating mechanism filters noise: By filtering out several tokens with the highest correlation between visual aspect representation, shallow aspect representation, deep aspect representation and CLS label, and integrating them, the noise is filtered out to filter out visual gating representation, shallow gating representation and deep gating representation respectively;

[0013] Step 5. Two-layer cross-modal distillation: Calculate the shallow distillation loss for the shallow gated representation and the visual gated representation; calculate the deep distillation loss for the deep gated representation and the visual gated representation;

[0014] Step 6. Model training and optimization: By jointly optimizing the classification loss, shallow distillation loss, and deep distillation loss, the model's sentiment classification capability in multimodal scenarios is trained.

[0015] Furthermore, in Step 1, constructing the prompt includes: constructing a prompt Q using a predefined text template containing a fixed context structure; used to enhance the model's attention to the semantics related to aspect words; the input sentence S and the generated prompt Q will serve as the basis for subsequent representation.

[0016] Furthermore, the Step 2 includes:

[0017] Step 2.1. For sentence S and prompt Q, get sentence embedding T through pre-trained language model BERT S and prompt embedded T Q ; Sentence embedding T S and prompt embedded T Q Expressed as:

[0018] T S =BERT emb (S) (1)

[0019] T Q =BERT emb (Q) (2)

[0020] Where T S ∈R n×d , T Q ∈Rm×d , n represents the sentence sequence length, m represents the prompt sequence length, and d is the dimension of the BERT embedding layer;

[0021] Step 2.2: Using the pre-trained PDQ module in the DQPSA model, extract the visual representation H by inputting the noise image V and the prompt Q. V ; Visual aspect representation H V Expressed as:

[0022] H V =PDQ_module(V,Q) (3)

[0023] Among them, H V ∈R m×d Representation of visual aspects extracted by the representation model.

[0024] Furthermore, the Step 3 includes:

[0025] Step 3.1, obtain shallow aspect representation: for a given sentence embedding T S and prompt embedded T Q , first calculate the attention score α through the interactive attention mechanism; the calculation formula of the attention score α is:

[0026] α=softmax(T Q W1T S T ) (4)

[0027] where W1∈R d×d is the shallow learning weight matrix, α∈R m×k ;

[0028] Based on this attention score, we obtain the hint-guided shallow aspect representation H L , shallow aspect representation H L Expressed as:

[0029] H L =relu(α)T S (5)

[0030] Among them, H L ∈R m×d It is a superficial aspect representation;

[0031] Step 3.2, obtain deep aspect representation: shallow aspect representation H through multi-layer Transformer structure L Further extraction is performed to calculate the deep aspect representation; the deep aspect representation is:

[0032] H H =TransformweEncoder(HL ) (6)

[0033] where H H ∈R m×d is the deep aspect representation.

[0034] Further, the Step4 includes:

[0035] Step4.1, for visual aspect representation H V , by calculating the cosine similarity between each token mark and the CLS mark, the top k token marks with the highest relevance are selected and integrated, thereby filtering noise and screening out key representations, wherein the CLS mark is the first feature in the visual aspect representation, i.e., the global representation;

[0036] wherein the similarity score calculation formula is as follows:

[0037]

[0038] wherein γ i represents the similarity score between the i-th token mark and the CLS mark ;

[0039] The top k tokens in the similarity score ranking are selected as new visual gating representations

[0040]

[0041] wherein, k is set as a gating coefficient and participates in training as a hyperparameter;

[0042] Step4.2, for shallow aspect representation H L and deep aspect representation H H , Step4.1 is also applied to improve the noise robustness of aspect representation, to generate shallow gating representation and deep gating representation

[0043] Further, the Step5 includes:

[0044] Step5.1, the deep gating representation shallow gating representation and visual gating representation processed by the gating mechanism are respectively converted into probability distribution: shallow gating representation deep gating representation visual gating representation probability distribution PL 、P H 、P V Expressed as:

[0045]

[0046]

[0047] Where τ is a temperature hyperparameter that controls the smoothness of softmax;

[0048] Step 5.2. Calculate the shallow distillation loss and deep distillation loss to measure the transfer learning effect of the model;

[0049] Shallow distillation loss L LV It is used to measure the KL divergence between the shallow gated representation and the visual aspect representation after the gated mechanism, and the shallow distillation loss L LV for:

[0050]

[0051] Deep distillation loss L HV It is used to measure the KL divergence between the deep gated representation and the visual aspect representation after the gating mechanism, and the deep distillation loss L HV for:

[0052]

[0053] Among them, P V j 、P L j 、P H j Represents P L 、P H 、P V The jth element in .

[0054] Furthermore, the Step 6 includes:

[0055] During the training phase, by minimizing the shallow distillation loss and the deep distillation loss L LV 、L HV , allowing the model to fully learn the visual representation; then the deep and shallow representations of the model are combined and sent to the classifier for sentiment prediction; the classifier processing process is as follows:

[0056] First, the deep and shallow aspect representations are added and pooled to obtain a fused representation:

[0057]

[0058] The fusion representation is then processed through the relu activation function, and the logarithmic probability distribution is calculated through the log_softmax function:

[0059] Y=log_softmax(w1relu(w0η)) (13)

[0060] in Represents the addition of two matrices, η∈R 1×d Represents the fusion representation finally fed into the classifier, Y∈R num Represents the prediction result, num represents the number of classifications, w0∈R d×0.5d , w1∈R 0.5d×num is the learned weight matrix;

[0061] The cross entropy loss between the predicted result Y and the true label M is used as the prediction loss:

[0062]

[0063] M i represents the i-th true label, Y i Represents the i-th prediction result;

[0064] Finally, the total loss is as follows:

[0065] L=λ pre L pre +λ LV L LV +λ HV L HV (15)

[0066] The three hyperparameters λ are pre ,λ LV ,λ HV Used to control the contribution rate of the three losses respectively;

[0067] During the training phase, the model accepts text, visual information, and prompts as input;

[0068] During the inference phase, the model only needs text and prompts to complete the sentiment classification task.

[0069] The present invention also provides a multimodal aspect sentiment analysis system based on prompt-based two-layer cross-modal distillation learning. The system includes: a module for executing the above-mentioned multimodal aspect sentiment analysis method based on prompt-based two-layer cross-modal distillation learning.

[0070] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored on the memory and runnable on the processor, wherein when the processor executes the program, the multimodal aspect sentiment analysis method based on prompt-based two-layer cross-modal distillation learning is implemented.

[0071] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned multimodal aspect sentiment analysis method based on prompt-based two-layer cross-modal distillation learning.

[0072] The beneficial effects of the present invention are:

[0073] 1. The proposed method of multimodal aspect sentiment analysis based on double-layer cross-modal distillation learning based on prompts has excellent performance in multimodal sentiment analysis;

[0074] 2. The proposed method exhibits strong noise robustness and scene adaptability in scenes such as visual noise;

[0075] 3. When using only text input during inference, the model first uses prompts to guide the model to focus on aspect word information, then effectively acquires multimodal information from the existing MABSA model through distillation learning, and uses a gating mechanism to filter redundant noise. The combination of these strategies enables the method of the present invention to achieve state-of-the-art performance.

[0076] 4. The method of the present invention solves the problem that existing methods are poorly robust or even inapplicable in scenarios with no vision or visual interference;

[0077] 5. The present invention demonstrates extremely high accuracy and robustness in sentiment analysis tasks involving noisy vision. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 It is a framework diagram of the method in the present invention. DETAILED DESCRIPTION

[0079] Example 1: The following method proposed in this example is implemented on three multimodal sentiment analysis datasets (Twitter2015, Twitter2017, and MASAD);

[0080] This paper uses three benchmark datasets, Twitter2015, Twitter2017, and MASAD, to perform sentiment analysis tasks. The two Twitter datasets collect user posts from 2014-2015 and 2016-2017, respectively. MASAD is a newly released large-scale multimodal dataset that covers 57 aspects in seven fields: food, goods, architecture, animals, humans, plants, and scenery. In the three datasets, each sample contains an aspect word, a sentence, and a corresponding image. In the three datasets, the sentiment polarity of aspect words in the Twitter dataset is divided into three categories: positive, neutral, and negative, while the sentiment polarity of aspect words in the MASAD dataset is only divided into two categories: positive and negative. The statistical information of these datasets is shown in Table 1.

[0081] Table 1 Dataset statistics

[0082]

[0083] like Figure 1 As shown, a multimodal aspect sentiment analysis method based on prompt-based two-layer cross-modal distillation learning, the method includes:

[0084] Step 1, prompt generation: construct prompts through text templates with aspect words as the core, and regard the text templates as extensions of aspect words to form prompts;

[0085] Furthermore, in Step 1, constructing the prompt includes: using a predefined text template containing a fixed context structure "sentiment of Aspect is [positive neutral negative]", where Aspect represents an aspect word, replacing Aspect with the aspect word in each sample in the data, thereby obtaining a prompt Q corresponding to each sample, which is used to enhance the model's attention to the semantics related to the aspect word; the input sentence S and the generated prompt Q will serve as the basis for subsequent representation.

[0086] Step 2: Embedding text and visual representations: Use a pre-trained language model (such as BERT) to generate embedded representations for the input text and prompts; use an existing multimodal aspect sentiment analysis model (DQPSA) to extract visual aspect representations for the input visual information and prompts;

[0087] Furthermore, the Step 2 includes:

[0088] Step 2.1. For sentence S and prompt Q, get sentence embedding T through pre-trained language model BERT S and prompt embedded T Q ; Sentence embedding T S and prompt embedded TQ is denoted as:

[0089] T S = BERT emb (S) (1)

[0090] T Q = BERT emb (Q) (2)

[0091] wherein T S ∈R n×d , T Q ∈R m×d , n represents the length of the sentence sequence, m represents the length of the prompt sequence, and d is the dimension of the BERT embedding layer;

[0092] Step2.2, using the pre-trained PDQ module in the DQPSA model, the visual aspect representation H V is extracted by inputting the noise image V and the prompt Q; V is denoted as:

[0093] H V = PDQ_module(V, Q) (3)

[0094] wherein H V ∈R m×d represents the visual aspect representation extracted by the model.

[0095] Step3, shallow and deep aspect representation construction: the shallow aspect representation of the text and the prompt is calculated by using the shallow attention mechanism; the deep aspect representation is further extracted by using the multi-layer Transformer;

[0096] Further, the Step3 includes:

[0097] Step3.1, obtaining the shallow aspect representation: for a given sentence embedding T S and prompt embedding T Q , first, the attention score a is calculated by the interaction attention mechanism; the attention score a is calculated by the formula:

[0098] a = softmax(T Q W1T S T ) (4)

[0099] wherein W1∈R d×d is the learning weight matrix of the shallow layer, and a∈R m×k ;

[0100] Based on the attention score, the shallow aspect representation H L under the guidance of the prompt is obtained, and the shallow aspect representation HL Expressed as:

[0101] H L =relu(α)T S (5)

[0102] Among them, H L ∈R m×d It is a superficial aspect representation;

[0103] Step 3.2, obtain deep aspect representation: shallow aspect representation H through multi-layer Transformer structure L Further extraction is performed to calculate the deep aspect representation; the deep aspect representation is:

[0104] H H =TransformerEncoder(H L ) (6)

[0105] Among them H H ∈R m×d Represents the deep aspects.

[0106] Step 4. Gating mechanism filters noise: By filtering out several tokens with the highest correlation between visual aspect representation, shallow aspect representation, deep aspect representation and CLS label, and integrating them, the noise is filtered out to filter out visual gating representation, shallow gating representation and deep gating representation respectively;

[0107] Furthermore, the Step 4 includes:

[0108] Step 4.1. Represent H in terms of vision V By calculating the cosine similarity between each token and the CLS tag, the k tokens with the highest correlation are selected and integrated to filter out the noise and select the key representations. Among them, the CLS tag is the first feature in the visual representation, that is, the global representation;

[0109] The similarity score calculation formula is as follows:

[0110]

[0111] Among them, γ i Indicates the i-th token With CLS markup The similarity score between them;

[0112] Select the top k tokens in the similarity score ranking as the new visual gating representation

[0113]

[0114] in, k is set as the gating coefficient and participates in training as a hyperparameter;

[0115] Step 4.2: Characterize H in terms of shallow aspects L and deep aspect representation H H , also apply the gating mechanism to perform Step 4.1 to improve the noise robustness of the aspect representation and generate a shallow gated representation after the gating mechanism is processed and deep gating representation

[0116] Step 5. Two-layer cross-modal distillation: Calculate the shallow distillation loss for the shallow gated representation and the visual gated representation; calculate the deep distillation loss for the deep gated representation and the visual gated representation;

[0117] Furthermore, the Step 5 includes:

[0118] Step 5.1: Deep gated representation processed by the gating mechanism Shallow gating representation and visual gating representations Converted into probability distributions: shallow gated representation Deep gated representation Visual gating representation The probability distribution P L 、P H 、P V Expressed as:

[0119]

[0120] Where τ is a temperature hyperparameter that controls the smoothness of softmax;

[0121] Step 5.2. Calculate the shallow distillation loss and deep distillation loss to measure the transfer learning effect of the model;

[0122] Shallow distillation loss L LV It is used to measure the KL divergence between the shallow gated representation and the visual aspect representation after the gated mechanism, and the shallow distillation loss L LV for:

[0123]

[0124] Deep distillation loss L HV It is used to measure the KL divergence between the deep gated representation and the visual aspect representation after the gating mechanism, and the deep distillation loss L HV for:

[0125]

[0126] Among them, P V j 、P L j 、P H j Represents P L 、P H 、P V The jth element in .

[0127] Step 6. Model training and optimization: By jointly optimizing classification loss, shallow distillation loss, and deep distillation loss, the model's sentiment classification capability in multimodal scenarios is trained.

[0128] Furthermore, the Step 6 includes:

[0129] During the training phase, by minimizing the shallow distillation loss and the deep distillation loss L LV 、L HV , allowing the model to fully learn the visual representation; then the deep and shallow representations of the model are combined and sent to the classifier for sentiment prediction; the classifier processing process is as follows:

[0130] First, the deep and shallow aspect representations are added and pooled to obtain a fused representation:

[0131]

[0132] The fusion representation is then processed through the relu activation function, and the logarithmic probability distribution is calculated through the log_softmax function:

[0133] Y=log_softmax(w1relu(w0η)) (13)

[0134] in Represents the addition of two matrices, η∈R 1×d Represents the fusion representation finally fed into the classifier, Y∈R num Represents the prediction result, num represents the number of classifications, w0∈R d×0.5d , w1∈R 0.5d×num is the learned weight matrix;

[0135] The cross entropy loss between the predicted result Y and the true label M is used as the prediction loss:

[0136]

[0137] M i represents the i-th true label, Y i Represents the i-th prediction result;

[0138] Finally, the total loss is as follows:

[0139] L=λ pre L pre +λ LV L LV +λ HV L HV (15)

[0140] The three hyperparameters λ are pre ,λ LV ,λ HV Used to control the contribution rate of the three losses respectively;

[0141] During the training phase, the model accepts text, visual information, and prompts as input;

[0142] During the inference phase, the model only needs text and prompts to complete the sentiment classification task.

[0143] The present invention also provides a multimodal aspect sentiment analysis system based on prompt-based two-layer cross-modal distillation learning, the system comprising:

[0144] a prompt generation module, which is used to construct prompts by using text templates with aspect words as the core, and regards the text templates as extensions of aspect words to form prompts;

[0145] A representation extraction module is used to generate embedded representations for the input text and prompts using a pre-trained language model (such as BERT); and to extract visual aspect representations for the input visual information and prompts using an existing multimodal aspect sentiment analysis model (DQPSA);

[0146] Shallow and deep aspect representation building blocks for computing shallow aspect representations of text and prompts using a shallow attention mechanism; and further extracting deep aspect representations using a multi-layer Transformer.

[0147] The gated representation filtering module is used to filter out the visual aspect representation, shallow aspect representation, deep aspect representation and several tokens with the highest correlation with the CLS label, and integrate them to filter out noise, thereby filtering out the visual gated representation, shallow gated representation and deep gated representation respectively;

[0148] Distillation loss calculation module, used to calculate shallow distillation loss for shallow gated representation and visual gated representation; deep distillation loss for deep gated representation and visual gated representation;

[0149] The sentiment classification module is used to train the model's sentiment classification capabilities in multimodal scenarios by jointly optimizing classification loss, shallow distillation loss, and deep distillation loss.

[0150] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored on the memory and runnable on the processor, wherein when the processor executes the program, the multimodal aspect sentiment analysis method based on prompt-based two-layer cross-modal distillation learning is implemented.

[0151] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned multimodal aspect sentiment analysis method based on prompt-based two-layer cross-modal distillation learning.

[0152] To verify the effectiveness of the proposed model, we compared the proposed model with other state-of-the-art sentiment analysis models, covering three datasets, including plain text and image-text scenarios, as follows:

[0153] In the plain text scenario:

[0154] ATAE-LSTM uses aspect-oriented attention mechanism to extract important information from sentences.

[0155] IAN combines the long short-term memory (LSTM) network and the attention mechanism to model the interaction between the target and the context, accurately capture key information and predict sentiment tendencies.

[0156] TNet uses convolutional neural networks (CNNs) to extract features and generate aspect-specific sentence representations while preserving the original contextual information.

[0157] MGAN utilizes multi-granularity attention networks to understand aspects.

[0158] BERT utilizes the pre-trained BERT model to capture the interaction between aspects and text.

[0159] DualGCN is a dual graph convolutional network model that considers the complementarity between syntactic structure and semantic association.

[0160] In the text-image multimodal scenario:

[0161] MIMN explores attention-based interactions between aspects, sentences, and related images via a multi-hop memory network.

[0162] TomBERT learns aspect-sensitive text features, matches aspect-image pairs to obtain visual features, and uses a self-attention strategy to capture multimodal interactions.

[0163] MMAP reveals hidden relationships among sentences, aspects, and images through multimodal interaction layers and adversarial training.

[0164] AMIFN improves the accuracy of fine-grained sentiment analysis by integrating attention mechanism and graph convolutional network to achieve multi-perspective interaction and fusion for specific aspects.

[0165] ESAFN divides sentences into left and right contexts, uses attention mechanisms to explore aspect-text and aspect-image interactions, and finally fuses these features through a bilinear layer for sentiment prediction.

[0166] Res-BERT combines the visual features extracted by ResNet and the hidden representations of BERT.

[0167] HIMT conducts aspect-text and aspect-image interactions and builds an auxiliary reconstruction module to eliminate the semantic differences between different modalities.

[0168] VLP-MABSA is a task-specific vision-language pre-training model that models aspect, viewpoint, and alignment by performing five specialized pre-training tasks.

[0169] KEF-TomBERT is a knowledge-enhanced framework that improves task performance by associating images with adjectives and nouns.

[0170] Evaluation metrics: Accuracy (Acc) and macro-average F1 score (F1) are used as evaluation metrics. Higher Acc and F1 values ​​indicate better performance.

[0171] The proposed method was first compared with other methods on the Twitter2015 and Twitter2017 benchmark datasets, both in the absence of images and in the presence of combined images and text. The results are shown in Table 2. The following conclusions were drawn: 1) In the absence of images and in the presence of combined images and text, the proposed method outperformed other SOTA methods; 2) In the pure text scenario, the proposed method significantly improved over all other methods, verifying the effectiveness of the proposed method in fine-grained sentiment recognition; 3) The proposed method also significantly outperformed other methods on both evaluation metrics of the two tasks, further demonstrating the effectiveness of the proposed method in sentiment analysis; 4) On both datasets, the proposed method also outperformed other pre-trained MABSA methods, demonstrating the effectiveness of the distillation mechanism in sentiment analysis.

[0172] Experimental results on public datasets such as twitter2015 and twitter2017 show that the cue-based two-layer cross-modal distillation learning method exhibits extremely high accuracy and robustness in sentiment analysis tasks involving noisy vision.

[0173] Table 2 compares the experimental results on the twitter2015 and twitter2017 datasets.

[0174]

[0175]

[0176] Then, on the MASAD dataset, the present application also carried out comparative experiments in two scenarios. Since other models obtained experimental results in seven fields of the dataset respectively, the present application took the average results of each model for comparison, and the results are shown in Table 3. The present application draws the following conclusions: 1) the method of the present application shows strong robustness and adaptability in the task of processing diversified data in different fields, which proves that the method of the present application can perform well in various sentiment analysis scenarios; 2) whether in the pure text or the image-text scenario, the method of the present application also shows significant advantages over most models in the dataset, which again proves the effectiveness and robustness of the method of the present application.

[0177] Table 3 is the comparison of experimental results on the MASAD dataset

[0178]

[0179] In addition, in order to evaluate the robustness of the method proposed in the present application, the present application carried out experiments in multiple visual noise scenarios, including: visual occlusion noise (VON), visual interference noise (VIN), image-text mismatch noise (ITMN) and visual missing noise (VMN). The experimental results are shown in Table 4.

[0180] 1) In all three datasets, the addition of VIN and VMN will cause a significant decline in sentiment analysis performance, because VIN will seriously damage the visual information, and then have a negative impact on the performance; 2) the performance of sentiment analysis in the VMN scenario is the worst, which may be due to the complete loss of visual information, causing the model to lose the most information; 3) the introduction of VON has a smaller impact on performance, which demonstrates the robustness of the method proposed in the present application. This also shows that the Gaussian white noise has less loss of visual information, because compared with other more destructive noises, the added visual noise has a lighter impact.

[0181] Table 4 is the comparison of experimental results under different visual noise

[0182]

[0183]

[0184] In order to further verify the effectiveness of the modules proposed in the present application, the present application carried out ablation experiments on three datasets. The experimental results are shown in Table 5.

[0185] Table 5 is the ablation study on three datasets

[0186]

[0187] w / o Prompt-Guided and w / o Gating represent the removal of the prompt-guided dual-layer aspect-guided representation and gating mechanism from the model proposed in this invention. w / o L HV and w / o L LV Indicates the removal of shallow distillation loss and deep distillation loss.

[0188] The present invention draws the following conclusions: 1) Removing the two distillation learning losses leads to a significant drop in performance on the three datasets. This demonstrates the effectiveness of the distillation mechanism in sentiment analysis, especially in improving the model's adaptability to complex scenarios; 2) After removing Prompt-Guided, the model's performance on all datasets has declined, indicating that the aspect perception module is crucial to the model's ability to focus information in fine-grained sentiment analysis; 3) After removing Gating, the model's performance also drops significantly, demonstrating the role of the gating mechanism in reducing the semantic gap between modalities and the influence of noise; 4) Removing L HV Afterwards, the model's distillation learning ability is incomplete and the model performance has declined, which proves that double-layer distillation learning can more effectively enhance the model's sentiment analysis ability. In summary, the ablation experiment results fully verify the importance of each module and loss function in the fine-grained sentiment analysis task.

[0189] The specific embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the scope of the present invention.

Claims

1. A multimodal aspect sentiment analysis method based on two-layer cross-modal distillation learning based on prompts, characterized by: The method comprises: Step 1, prompt generation: construct prompts through text templates with aspect words as the core, and regard the text templates as extensions of aspect words to form prompts; Step 2: Embedding text and visual representations: Use a pre-trained language model to generate embedded representations for the input text and prompts; use an existing multimodal sentiment analysis model to extract visual aspect representations for the input visual information and prompts; Step 3: Constructing shallow and deep aspect representations: Using a shallow attention mechanism to compute shallow aspect representations of text and prompts; using a multi-layer Transformer to further extract deep aspect representations; Step 4. Gating mechanism filters noise: By filtering out several tokens with the highest correlation between visual aspect representation, shallow aspect representation, deep aspect representation and CLS label, and integrating them, the noise is filtered out to filter out visual gating representation, shallow gating representation and deep gating representation respectively; Step 5. Two-layer cross-modal distillation: Calculate the shallow distillation loss for the shallow gated representation and the visual gated representation; calculate the deep distillation loss for the deep gated representation and the visual gated representation; Step 6: Model training and optimization: By jointly optimizing the classification loss, shallow distillation loss, and deep distillation loss, the model is trained to perform sentiment classification in multimodal scenarios. In Step 1, the prompt construction includes: constructing a prompt Q using a predefined text template containing a fixed context structure; used to enhance the model's attention to the semantics related to aspect words; the input sentence S and the generated prompt Q will serve as the basis for subsequent representation; Step 4 includes: Step 4.

1. Representation of visual aspects By calculating the cosine similarity between each token and the CLS tag, the k tokens with the highest correlation are selected and integrated to filter out the noise and select the key representations. Among them, the CLS tag is the first feature in the visual representation, that is, the global representation; The similarity score calculation formula is as follows: (1)； in, Indicates the Token With CLS markup The similarity score between them; Select the top similarity score tokens as new visual gating representations ; (2)； in, , Set as the gating coefficient, This is the dimension of the BERT embedding layer and is used as a hyperparameter in training; Step 4.2: Representation of shallow aspects and deep aspect representations , also apply the gating mechanism to execute Step 4.1 to generate the shallow gating representation after the gating mechanism is processed and deep gating representation .

2. The multimodal aspect sentiment analysis method based on prompt-based two-layer cross-modal distillation learning according to claim 1, characterized in that: Step 2 includes: Step 2.

1. For the sentence and tips , sentence embedding is obtained through the pre-trained language model BERT and ; Sentence Embedding and Expressed as: (3)； (4)； in , , Indicates the length of the sentence sequence, Indicates the length of the prompt sequence; Step 2.2: Use the pre-trained PDQ module in the DQPSA model to extract the visual aspect representation by inputting the noise image V and the prompt Q. ; Visual aspect representation Expressed as: (5)； in, Representation of visual aspects extracted by the representation model.

3. The multimodal aspect sentiment analysis method based on prompt-based two-layer cross-modal distillation learning according to claim 1, characterized in that: Step 3 includes: Step 3.

1. Obtain shallow aspect representation: For a given sentence embedding and , first calculate the attention score through the interactive attention mechanism ; Attention score The calculation formula is: (6)； in is the shallow learning weight matrix, , Indicates the length of the prompt sequence; Based on this attention score, we obtain a cue-guided shallow aspect representation. , shallow aspect representation Expressed as: (7)； in, It is a superficial aspect representation; Step 3.2, obtain deep aspect representation: shallow aspect representation through multi-layer Transformer structure Further extraction is performed to calculate the deep aspect representation; the deep aspect representation is: (8)； in Represents the deep aspects.

4. The multimodal aspect sentiment analysis method based on prompt-based two-layer cross-modal distillation learning according to claim 1, characterized in that: Step 5 includes: Step 5.1: Deep gated representation processed by the gating mechanism , shallow gated representation and visual gating representations Converted into probability distributions: shallow gated representation , deep gated representation , visual gating representation The probability distribution of Expressed as: ； (9)； ； in is the temperature hyperparameter, which is used to control the smoothness of softmax; Step 5.2, calculate the shallow distillation loss and deep distillation loss; Shallow distillation losses Used to measure the KL divergence between the shallow gated representation and the visual aspect representation after the gated mechanism, shallow distillation loss for: (10)； Deep distillation loss Used to measure the KL divergence between the deep gated representation and the visual aspect representation after the gated mechanism, deep distillation loss for: (11)； in 、 、 Respectively The elements.

5. The multimodal aspect sentiment analysis method based on prompt-based two-layer cross-modal distillation learning according to claim 1, characterized in that: Step 6 includes: During the training phase, by minimizing the shallow distillation loss and deep distillation losses , allowing the model to fully learn the visual representation; then the deep and shallow representations of the model are combined and sent to the classifier for sentiment prediction; the classifier processing process is as follows: First, we characterize the deep aspects and shallow aspect representations Add and pool to get the fusion representation: (12)； Then the fusion representation is processed by the relu activation function, and then The function calculates the logarithmic probability distribution: (13)； in represents the addition of two matrices, represents the fusion representation finally fed into the classifier, , Indicates the number of categories, 、 is the learned weight matrix; Will and the true label The cross entropy loss is used as the prediction loss: (14)； represents the i-th true label, Indicates the i-th ; Finally, the total loss is as follows: (15)； The three hyperparameters 、 Used to control the contribution rate of the three losses respectively; During the training phase, the model accepts text, visual information, and prompts as input; During the inference phase, the model only needs text and prompts to complete the sentiment classification task.

6. A multimodal aspect sentiment analysis system based on two-layer cross-modal distillation learning based on prompts, characterized by: The system includes: a module for performing the multimodal aspect sentiment analysis method based on prompt-based two-layer cross-modal distillation learning as described in any one of claims 1 to 5.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the multimodal aspect sentiment analysis method based on prompt-based two-layer cross-modal distillation learning as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the multimodal aspect sentiment analysis method based on prompt-based two-layer cross-modal distillation learning as described in any one of claims 1 to 5.

Images