Social media user sentiment analysis method based on iterative pseudo tag and knowledge enhancement

Abstract

The invention discloses a social media user sentiment analysis method based on iteration pseudo tags and knowledge enhancement. The method comprises the steps of obtaining a data set, dividing the data set, extracting multi-modal features, constructing a multi-modal aspect sentiment analysis network, training the multi-modal aspect sentiment analysis network, testing the multi-modal aspect sentiment analysis network and identifying aspect word sentiment polarity. According to the method, a multi-modal aspect word knowledge enhancement fusion network is adopted, and polarity classification of aspect words is achieved. In the network training process, the correlation between aspect words and images is judged through a semantic alignment network, external knowledge enhancement is performed on image contents by means of a knowledge enhancement network, and semantic expression is enhanced; through a multi-modal fusion layer, information of different modals is integrated, the technical problems that the correlation between aspect words and images is weak, and multi-modal information is difficult to fuse are solved, through comparison experiment verification, the method has the advantages of being high in accuracy, high in network robustness and the like, and the method can be used for the technical field of aspect-level sentiment analysis with image-text coexistence.

Description

Technical Field

[0001] This invention belongs to the field of multimodal data processing and artificial intelligence technology, specifically relating to a method for analyzing user sentiment on social media. Background Technology

[0002] User data on social media platforms typically contains multimodal information, including text and images, which collectively express users' sentiment tendencies. Traditional sentiment analysis methods primarily focus on a single modality (such as text), classifying sentiment by extracting shallow features like word frequency and syntactic structure. Single-modal analysis struggles to capture the semantic connections between text and images, leading to insufficient accuracy in sentiment prediction. In recent years, with the development of deep learning and multimodal learning, research has begun to utilize deep neural networks (such as convolutional neural networks and transformer models) to automatically extract deep features from images and text, and to attempt to improve sentiment analysis performance through cross-modal fusion. For example, methods based on pre-trained visual-language models (such as CLIP) can jointly encode image and text features to generate a unified semantic representation.

[0003] Existing methods still face the following technical problems:

[0004] In social media data, some text content (such as aspect terms) has a weak correlation with images, making it difficult to effectively integrate multimodal information.

[0005] The lack of external knowledge support for image content leads to insufficient semantic expression.

[0006] Data annotation is costly, and the generation of relevance and sentiment labels relies on manual labor, which limits the scalability of the model.

[0007] In the field of sentiment analysis technology, a pressing technical problem is to provide a method that can automatically generate relevant labels, enhance multimodal semantic expression, and improve the accuracy and robustness of sentiment analysis. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a social media user sentiment analysis method based on iterative pseudo-labels and knowledge enhancement that has high accuracy, high robustness and low annotation cost.

[0009] The technical solution adopted to solve the above technical problems consists of the following steps:

[0010] (1) Obtaining the dataset

[0011] from IJCAI2019_data.zip - Google DriveDownload the Twitter 2015 dataset to obtain multimodal data, totaling 3,502 tweets. Each tweet contains a text sentence, image, aspect words, and sentiment tags. After data cleaning, invalid image connections and data with missing sentiment tags were removed, resulting in a total of 3,502 images, 3,502 sentences, and 5,338 aspect words used as the experimental dataset.

[0012] (2) Split the dataset

[0013] The Twitter 2015 dataset was divided into a training set, a validation set, and a test set. The training set contained 2801 data points, the validation set contained 350 data points, and the test set contained 350 data points.

[0014] (3) Extracting multimodal features

[0015] 1) Extract semantic alignment module features

[0016] Image features v are extracted using the pre-trained model CLIP. i and aspect features t i,j , The features obtained by encoding the i-th image, d t t represents the dimension of the image embedding; i,j The feature obtained by encoding the j-th aspect word in the i-th data item. The dimension of the embedding of aspect terms.

[0017] 2) Extracting emotional module features in terms of fusion

[0018] Combine text x using the following formula i :

[0019] x i =s i [SEP]a i,1 ,a i,2 ,...,a i,j [SEP]k' i.j

[0020] Among them, s i For the sentence of the i-th data, a i,j For aspect words, k' i.j For knowledge text, [SEP] is the delimiter, and x is processed using the pre-trained model CLIP. i Encode to obtain combined text features Using the pre-trained model CLIP to target the term a i,j and knowledge text k' i.j Encode the aspect word features to obtain them. and knowledge text features The image is preprocessed and encoded using the ResNet152 model to obtain image features. d v This indicates the dimension of the image embedding.

[0021] (4) Constructing a multimodal sentiment analysis network

[0022] The multimodal sentiment analysis network is composed of a multimodal word knowledge enhancement fusion network and a sentiment classification network connected together.

[0023] The multimodal aspect word knowledge enhancement fusion network of the present invention is composed of a semantic alignment enhancement network and an aspect knowledge image fusion network connected in series.

[0024] The semantic alignment enhancement network of the present invention is composed of an image coding layer 1, a text coding layer 1, and a projection layer connected together, and the outputs of the image coding layer 1 and the text coding layer are connected to the projection layer.

[0025] The image encoding layer 1 of the present invention is composed of an embedding layer 1, a transform layer 1, a normalization layer 1, and a pooling layer 1 connected in series; the text encoding layer 1 is composed of an embedding layer 2, a transform layer 2, a normalization layer 2, and a pooling layer 2 connected in series.

[0026] The knowledge image fusion network of the present invention is composed of text encoding layer 2, text encoding layer 3, text encoding layer 4, image encoding layer 2, text knowledge fusion network, and multimodal fusion network connected together; the output ends of text encoding layer 2 and text encoding layer 3 are connected to the text knowledge fusion network, the output ends of text encoding layer 4 and image encoding layer 2 are connected to the multimodal fusion network, and the text knowledge fusion network and the multimodal fusion network are connected in series.

[0027] The text encoding layer 2 of the present invention is composed of an embedding layer 3, a Transformer layer 3, a normalization layer 3, and a pooling layer 3 connected in series.

[0028] The image coding layer 2 of the present invention is composed of a convolutional layer, a residual layer, and a pooling layer 4 connected in series.

[0029] The text knowledge fusion network of the present invention is composed of a normalization layer 4, a multi-head attention layer 1, a feedforward neural network 1, and a pooling layer 5 connected in series; the feedforward neural network 1 is composed of a linear layer 1, a linear rectifier layer 1, and a linear layer 2 connected in series.

[0030] The multimodal fusion network of the present invention is composed of a linear projection module and a normalization layer 5, a multi-head attention layer 2, a feedforward neural network 2, and a pooling layer 6 connected in series. The linear projection module is composed of a linear projection layer 1, a linear projection layer 2, and a linear projection layer 3 connected in parallel. The feedforward neural network 2 is composed of a linear layer 3, a linear rectification layer 2, a linear layer 4, a linear rectification layer 3, and a linear layer 5 connected in series.

[0031] The emotion classification network of this invention consists of a fully connected layer and a softmax layer connected in series.

[0032] (5) Training a multimodal sentiment analysis network

[0033] 1) Constructing the loss function

[0034] Construct the cross-entropy loss function L according to equation (1):

[0035]

[0036] Among them, y i,j Indicates sentiment label, y i,j Let c ∈ {0,1,2}, where 0 represents positive, 1 represents negative, and 2 represents neutral emotion. For the smoothed sentiment label, N c w is the number of samples in category c. c For resampling weights, w c ∈(0,1];β c β is the category adjustment factor. c ∈(0,1]; B is the number of data entries in the batch, B∈[4,12]; J is the number of valid aspect terms for the i-th data entry, J∈[1,6]; Let ∈ represent the sentiment prediction of words in the training set, where ∈ is the smoothing label parameter, and ∈ ∈ (0, 1).

[0037] 2) Training a multimodal sentiment analysis network

[0038] The training set was input into a multimodal aspect-based sentiment analysis network for training. The server was a GTX3090, the optimizer was AdamW, and adversarial training was performed using perturbation embeddings with fast gradient method. The initial learning rate was 0.00005, and it was adjusted by cosine annealing scheduling. Linear warm-up was used in the early stage, and cosine annealing scheduling was used in the later stage. Training was carried out until the loss function L converged.

[0039] (6) Testing multimodal sentiment analysis networks

[0040] The test set is fed into the trained multimodal aspect-of-sentiment analysis network for testing, and the output is social media user sentiment.

[0041] (7) Identifying social media user sentiment

[0042] Multimodal features are input into the trained multimodal aspect-based sentiment analysis network, and regression is performed using fully connected layers and softmax layers to obtain the probability values ​​for different sentiment categories: positive probability p pos p pos ∈(0,1], negative probability p neg p neg ∈(0,1], neutral probability p neu p neu ∈(0,1).

[0043] Predict the aspect sentiment category according to formula (2):

[0044]

[0045] p = (p pos ,p neg ,p neu )

[0046] in, This indicates that the emotional polarity corresponding to the element with the largest probability distribution value is selected as the result.

[0047] In the multimodal sentiment analysis network constructed in step (4) of this invention, the semantic alignment enhancement network construction method of this invention is as follows:

[0048] 1) Generate similarity scores and pseudo-labels

[0049] ① Determine the similarity of each pair of image-aspect words according to formula (3):

[0050]

[0051] Among them, v i ×t i,j Denotes the vector dot product, ||v i , ||t i,j || denotes the Euclidean norm of the vector's magnitude, Sim(v i ,t i,j )∈[-1,1].

[0052] ② Generate the initial pseudo-tags according to formula (4):

[0053]

[0054] Among them, l i,j ∈{1,0}, where 1 indicates correlation and 0 indicates no correlation, θ h As a positive threshold, θ h ∈(0,1), θ w For a negative threshold, θw ∈(0,1), and θ h +θ w ≤1; Pseudo-labeled dataset for:

[0055]

[0056] Among them, s i Let i be the sentence of the i-th data entry.

[0057] ③ Iterative optimization of the pre-trained model CLIP

[0058] The pre-trained model CLIP is optimized using the InfoNCE loss function to maximize positive sample similarity and minimize negative sample similarity.

[0059] Construct the lnfoNCE loss function according to equation (5):

[0060]

[0061] Where M is the number of data entries, determined according to the dataset; J is the number of aspect terms, J∈[1,6]; τ is the temperature parameter, τ∈(0,1]; the average similarity of positive samples is calculated according to equation (6).

[0062]

[0063] Among them, s i,j For positive samples, the similarity is N. pos This represents the number of positive samples.

[0064] Adjust the positive sample quantiles according to formula (7).

[0065]

[0066] Where, γ posw γ is the lower limit threshold for the average similarity of positive samples. posh lnfoNCE is the upper limit threshold for the average similarity of positive samples. pos Let δ be the loss component of the lnfoNCE for positive samples, t represent the t-th iteration, and δ pos The threshold for positive sample loss; q m q represents the minimum value of the quantile proportion. M The maximum value of the quantile proportion; Δ q The adjustment step size for the quantile proportion, ∈ s This is the threshold for the rate of change of loss.

[0067] Adjusting negative sample quantiles Methods and Adjustment of Positive Sample Quantiles The method is the same.

[0068] ④ Determine the final pseudo-label

[0069] Determine the final pseudo-label according to formula (8):

[0070]

[0071] Where T is the final iteration number, T takes values ​​from 2 to 10, β is the weight factor, β∈[0,1], Sim′(v i ,t i,j The similarity is obtained after optimizing the model.

[0072] 2) Knowledge Enhancement

[0073] ①Generate instructions

[0074] The large language model generates a fixed instruction template Instr based on the user's description, and fills in the sentence s when processing specific data. i and aspect words t i,j Generate instructions.

[0075] ② Generate knowledge text

[0076] Using a visual large language model to process pseudo-label datasets l in i,j =1, generate knowledge text k' i.j ,

[0077] For l i,j =0, knowledge text k' i.j equals sentence s i .

[0078] The knowledge of the present invention regarding the construction method of image fusion network is as follows:

[0079] 1) Constructing a text knowledge fusion network

[0080] Construct a text knowledge fusion network according to formula (9):

[0081]

[0082] d h =d t / H

[0083] in, For fusion features; W q W k W v It is a linear transformation matrix. d hFor a single-head dimension, H is the number of attention heads; α is the residual weight, α∈(0,1], FFN1 is a feedforward network, a two-layer fully connected network.

[0084] 2) Construct a multimodal fusion network

[0085] Construct a multimodal fusion network according to equation (10):

[0086]

[0087]

[0088] in, It is a linear transformation matrix. FFN2 is a feedforward network, a three-layer fully connected network; As a combination feature, For the image features after projection alignment, W img It is a linear transformation matrix.

[0089] This invention constructs a neural network combining a multimodal aspect-level word knowledge enhancement fusion network and a sentiment classification network to achieve polarity classification of aspect-level words. During neural network training, a semantic alignment network determines the correlation between aspect-level words and images. Simultaneously, a knowledge enhancement network enhances the semantic expression of relevant aspect-level words by applying external knowledge to the image content. The multimodal fusion network integrates information from different modalities, effectively addressing the technical problem of weak correlation between aspect-level words and images in some scenarios, making it difficult to effectively fuse multimodal information. Comparative experiments with existing multimodal aspect-level sentiment analysis methods demonstrate that this invention has advantages such as high accuracy, high robustness, and low annotation cost, and can be applied to the field of aspect-level sentiment analysis technology in the context of complex social media platforms where images and text coexist. Attached Figure Description

[0090] Figure 1 This is a flowchart of Embodiment 1 of the present invention.

[0091] Figure 2 This is a schematic diagram of the structure of a multimodal sentiment analysis network.

[0092] Figure 3 yes Figure 2 A schematic diagram of the structure of a multimodal vocabulary knowledge enhancement fusion network.

[0093] Figure 4 yes Figure 3 A schematic diagram of the semantic alignment enhancement network.

[0094] Figure 5 yes Figure 3A schematic diagram of the structure of a knowledge-based image fusion network.

[0095] Figure 6 yes Figure 5 A schematic diagram of the structure of Chinese text encoding layer 2.

[0096] Figure 7 yes Figure 5 A schematic diagram of the structure of the image coding layer 2.

[0097] Figure 8 yes Figure 5 A schematic diagram of the structure of the Chinese text knowledge fusion network.

[0098] Figure 9 yes Figure 5 A schematic diagram of the structure of a multimodal fusion network.

[0099] Figure 10 yes Figure 2 A schematic diagram of the structure of the sentiment classification network. Detailed Implementation

[0100] The present invention will be further described below with reference to the accompanying drawings and embodiments, but the present invention is not limited to the following embodiments.

[0101] Example 1

[0102] The social media user sentiment analysis method based on iterative pseudo-labeling and knowledge enhancement in this embodiment consists of the following steps (refer to...). Figure 1 ):

[0103] (1) Obtaining the dataset

[0104] from IJCAI2019_data.zip - Google Drive Download the Twitter 2015 dataset to obtain multimodal data, totaling 3,502 tweets. Each tweet contains a text sentence, image, aspect words, and sentiment tags. After data cleaning, invalid image connections and data with missing sentiment tags were removed, resulting in a total of 3,502 images, 3,502 sentences, and 5,338 aspect words used as the experimental dataset.

[0105] (2) Split the dataset

[0106] The Twitter 2015 dataset was divided into a training set, a validation set, and a test set. The training set contained 2801 data points, the validation set contained 350 data points, and the test set contained 350 data points.

[0107] (3) Extracting multimodal features

[0108] 1) Extract semantic alignment module features

[0109] Image features v are extracted using the pre-trained model CLIP. iand aspect features t i,j , The features obtained by encoding the i-th image, d t t represents the dimension of the image embedding; i,j The feature obtained by encoding the j-th aspect word in the i-th data item. The dimension of the embedding of aspect terms.

[0110] 2) Extracting emotional module features in terms of fusion

[0111] Combine text x using the following formula i :

[0112] x i =s i [SEP]a i,1 ,a i,2 ,...,a i,j [SEP]k' i.j

[0113] Among them, s i For the sentence of the i-th data, a i,j For aspect words, k' i.j For knowledge text, [SEP] is the delimiter, and x is processed using the pre-trained model CLIP. i Encode to obtain combined text features Using the pre-trained model CLIP to target the term a i,j and knowledge text k' i.j Encode the aspect word features to obtain them. and knowledge text features The image is preprocessed and encoded using the ResNet152 model to obtain image features. d v This indicates the dimension of the image embedding.

[0114] (4) Constructing a multimodal sentiment analysis network

[0115] Figure 2 A schematic diagram of the multimodal sentiment analysis network in this embodiment is provided. Figure 2 In this embodiment, the multimodal aspect sentiment analysis network is composed of a multimodal aspect word knowledge enhancement fusion network and a sentiment classification network connected in series.

[0116] Figure 3 Given Figure 2 A schematic diagram of the structure of a multimodal vocabulary knowledge enhancement fusion network. Figure 3 In this embodiment, the multimodal aspect word knowledge enhancement fusion network is composed of a semantic alignment enhancement network and an aspect knowledge image fusion network connected in series.

[0117] This invention employs a multimodal aspect word knowledge enhancement fusion network and a sentiment classification network to achieve polarity classification of aspect words. During neural network training, a semantic alignment network determines the correlation between aspect words and images, while a knowledge enhancement network enhances the semantic expression of relevant aspect words by applying external knowledge to the image content.

[0118] Figure 4 Given Figure 3 A schematic diagram of the semantic alignment enhancement network. Figure 4 In this embodiment, the semantic alignment enhancement network is composed of an image coding layer 1, a text coding layer 1, and a projection layer connected together. The outputs of the image coding layer 1 and the text coding layer are connected to the projection layer.

[0119] The image encoding layer 1 is composed of an embedding layer 1, a transform layer 1, a normalization layer 1, and a pooling layer 1 connected in series; the text encoding layer 1 is composed of an embedding layer 2, a transform layer 2, a normalization layer 2, and a pooling layer 2 connected in series.

[0120] The semantic alignment enhancement network construction method in this embodiment is as follows:

[0121] 1) Generate similarity scores and pseudo-labels

[0122] ① Determine the similarity of each pair of image-aspect words according to formula (3):

[0123]

[0124] Among them, v i ×t i,j Denotes the vector dot product, ||v i ||,||t i,j || denotes the Euclidean norm of the vector's magnitude, Sim(v i ,t i,j )∈[-1,1].

[0125] ② Generate the initial pseudo-tags according to formula (4):

[0126]

[0127] Among them, l i,j ∈{1,0}, where 1 indicates correlation and 0 indicates no correlation, θ h As a positive threshold, θ h ∈(0,1), θ in this embodiment h The value is 0.5, θ w For a negative threshold, θ w ∈(0,1), θ in this embodiment w The value is 0.5, and θ h+θ w ≤1; Pseudo-labeled dataset for:

[0128]

[0129] Among them, s i Let i be the sentence of the i-th data entry.

[0130] ③ Iterative optimization of the pre-trained model CLIP

[0131] The pre-trained model CLIP is optimized using the InfoNCE loss function to maximize positive sample similarity and minimize negative sample similarity.

[0132] Construct the lnfoNCE loss function according to equation (5):

[0133]

[0134] Where M is the number of data entries, which is determined according to the dataset; J is the number of aspect terms, J∈[1,6]; τ is the temperature parameter, τ∈(0,1], and the value of τ in this embodiment is 0.5.

[0135] Calculate the average similarity of positive samples according to formula (6).

[0136]

[0137] Among them, s i,j For positive samples, the similarity is N. pos This represents the number of positive samples.

[0138] Adjust the positive sample quantiles according to formula (7).

[0139]

[0140] Where, γ posw γ is the lower limit threshold for the average similarity of positive samples. posh lnfoNCE is the upper limit threshold for the average similarity of positive samples. pos Let δ be the loss component of the lnfoNCE for positive samples, t represent the t-th iteration, and δ pos The threshold for positive sample loss; q m q represents the minimum value of the quantile proportion. M The maximum value of the quantile proportion; Δ q The adjustment step size for the quantile proportion, ∈ s This is the threshold for the rate of change of loss.

[0141] Adjusting negative sample quantiles Methods and Adjustment of Positive Sample Quantiles The method is the same.

[0142] ④ Determine the final pseudo-label

[0143] Determine the final pseudo-label according to formula (8):

[0144]

[0145] Where T is the final iteration number, and T ranges from 2 to 10. In this embodiment, T is 6. β is the weighting factor, β∈[0,1], and in this embodiment, β is 0.5. Sim′(v i ,t i,j The similarity is obtained after optimizing the model.

[0146] 2) Knowledge Enhancement

[0147] ①Generate instructions

[0148] The large language model generates a fixed instruction template Instr based on the user's description, and fills in the sentence s when processing specific data. i and aspect words t i,j Generate instructions;

[0149] ② Generate knowledge text

[0150] Using a visual large language model to process pseudo-label datasets l in i,j =1, generate knowledge text k' i.j For l i,j =0, knowledge text k' i.j equals sentence s i .

[0151] Figure 5 Given Figure 3 A schematic diagram of the structure of a knowledge-based image fusion network. Figure 5 In this embodiment, the aspect knowledge image fusion network is composed of text encoding layer 2, text encoding layer 3, text encoding layer 4, image encoding layer 2, text knowledge fusion network, and multimodal fusion network connected together; the output ends of text encoding layer 2 and text encoding layer 3 are connected to the text knowledge fusion network, the output ends of text encoding layer 4 and image encoding layer 2 are connected to the multimodal fusion network, and the text knowledge fusion network and the multimodal fusion network are connected in series.

[0152] Because this invention employs a multimodal fusion network, it integrates and processes information from different modalities, effectively solving the technical problem that aspect words and images are weakly correlated in some scenarios, making it difficult to effectively fuse multimodal information. When the method of this invention is applied to aspect-level sentiment analysis in complex social media environments where images and text coexist, it achieves high accuracy, high robustness, and low annotation cost.

[0153] Figure 6 Given Figure 5 A schematic diagram of the structure of Chinese text encoding layer 2. Figure 6 In this embodiment, the text encoding layer 2 is composed of an embedding layer 3, a Transformer layer 3, a normalization layer 3, and a pooling layer 3 connected in series. Figure 7 Given Figure 5 A schematic diagram of the structure of the image coding layer 2.

[0154] exist Figure 7 In this embodiment, the image coding layer 2 is composed of a convolutional layer, a residual layer, and a pooling layer 4 connected in series.

[0155] Figure 8 Given Figure 5 A schematic diagram of the structure of a Chinese text knowledge fusion network. Figure 8 In this embodiment, the text knowledge fusion network is composed of a normalization layer 4, a multi-head attention layer 1, a feedforward neural network 1, and a pooling layer 5 connected in series; the feedforward neural network 1 is composed of a linear layer 1, a linear rectifier layer 1, and a linear layer 2 connected in series.

[0156] Figure 9 Given Figure 5 A schematic diagram of the structure of a multimodal fusion network. Figure 9 In this embodiment, the multimodal fusion network is composed of a linear projection module connected in series with a normalization layer 5, a multi-head attention layer 2, a feedforward neural network 2, and a pooling layer 6; the linear projection module is composed of a linear projection layer 1 connected in parallel with a linear projection layer 2 and a linear projection layer 3; the feedforward neural network 2 is composed of a linear layer 3 connected in series with a linear rectification layer 2, a linear layer 4, a linear rectification layer 3, and a linear layer 5.

[0157] The aspect knowledge image fusion network construction method in this embodiment is as follows:

[0158] 1) Constructing a text knowledge fusion network

[0159] Construct a text knowledge fusion network according to formula (9):

[0160]

[0161] d h =d t / H

[0162] in, For fusion features; W q W k W v It is a linear transformation matrix. d hH is the number of attention heads, H is the number of attention heads, α is the residual weight, α∈(0,1], and in this embodiment, the value of α is 0.5. FFN1 is a feedforward network, a two-layer fully connected network.

[0163] 2) Construct a multimodal fusion network

[0164] Construct a multimodal fusion network according to equation (10):

[0165]

[0166] in, It is a linear transformation matrix. FFN2 is a feedforward network, a three-layer fully connected network; As a combination feature, For the image features after projection alignment, W img It is a linear transformation matrix.

[0167] Figure 10 Given Figure 2 A schematic diagram of the structure of a sentiment classification network. Figure 10 In this embodiment, the sentiment classification network consists of a fully connected layer and a softmax layer connected in series.

[0168] (5) Training a multimodal sentiment analysis network

[0169] 1) Constructing the loss function

[0170] Construct the cross-entropy loss function L according to equation (1):

[0171]

[0172] Among them, y i,j Indicates sentiment label, y i,j Let c ∈ {0,1,2}, where 0 represents positive, 1 represents negative, and 2 represents neutral emotion. For the smoothed sentiment label, N c w is the number of samples in category c. c For resampling weights, w c ∈(0,1], w in this embodiment c The value is 0.5, β c β is the category adjustment factor. c ∈(0,1], β in this embodiment c The value is 0.5, B is the number of data entries in the batch, B∈[4,12]; in this embodiment, B is 8, and J is the number of valid aspect words of the i-th data, J∈[1,6]; The sentiment prediction of words in the training set is represented by ∈, which is the smoothing label parameter, ∈ ∈ (0, 1), and the value of ∈ in this embodiment is 0.5.

[0173] 2) Training a multimodal sentiment analysis network

[0174] The training set was input into a multimodal aspect-based sentiment analysis network for training. The server was a GTX3090, the optimizer was AdamW, and adversarial training was performed using perturbation embeddings with fast gradient method. The initial learning rate was 0.00005, and it was adjusted by cosine annealing scheduling. Linear warm-up was used in the early stage, and cosine annealing scheduling was used in the later stage. Training was carried out until the loss function L converged.

[0175] (6) Testing multimodal sentiment analysis networks

[0176] The test set is fed into the trained multimodal aspect-of-sentiment analysis network for testing, and the output is social media user sentiment.

[0177] (7) Identifying social media user sentiment

[0178] Multimodal features are input into the trained multimodal aspect-based sentiment analysis network, and regression is performed using fully connected layers and softmax layers to obtain the probability values ​​for different sentiment categories: positive probability p pos p pos ∈(0,1], negative probability p neg p neg ∈(0,1], neutral probability p neu p neu ∈(0,1).

[0179] Predict the aspect sentiment category according to formula (2):

[0180]

[0181] p = (p pos ,p neg ,p neu )

[0182] in, This indicates that the emotional polarity corresponding to the element with the largest probability distribution value is selected as the result.

[0183] Develop a social media user sentiment analysis method based on iterative pseudo-labels and knowledge enhancement.

[0184] Example 2

[0185] The social media user sentiment analysis method based on iterative pseudo-labels and knowledge enhancement in this embodiment consists of the following steps:

[0186] (1) Obtaining the dataset

[0187] The steps are the same as in Example 1.

[0188] (2) Split the dataset

[0189] The steps are the same as in Example 1.

[0190] (3) Extracting multimodal features

[0191] The steps are the same as in Example 1.

[0192] (4) Constructing a multimodal sentiment analysis network

[0193] The multimodal aspect-based sentiment analysis network structure in this embodiment is the same as that in Embodiment 1.

[0194] The semantic alignment enhancement network construction method in this embodiment is as follows:

[0195] 1) Generate similarity scores and pseudo-labels

[0196] ① Determine the similarity of each pair of image-aspect words according to formula (3):

[0197] The expression of equation (3) is the same as that in Example 1.

[0198] In equation (3), Sim(v) i ,t i,j )∈[-1,1], and other parameters, variables and their ranges are the same as in Example 1.

[0199] ② Generate the initial pseudo-tags according to formula (4):

[0200] The expression of equation (4) is the same as that in Example 1.

[0201] In equation (4), θ h As a positive threshold, θ h ∈(0,1), θ in this embodiment h The value is 0.1, θ w For a negative threshold, θ w ∈(0,1), θ in this embodiment w The value is 0.9, and other parameters, variables, and value ranges are the same as in Example 1. The other steps in this process are the same as in Example 1.

[0202] ③ Iterative optimization of the pre-trained model CLIP

[0203] The pre-trained model CLIP is optimized using the InfoNCE loss function to maximize positive sample similarity and minimize negative sample similarity.

[0204] Construct the lnfoNCE loss function according to equation (5):

[0205] The expression of equation (5) is the same as that in Example 1.

[0206] In equation (5), τ is a temperature parameter, τ∈(0,1], and the value of τ in this embodiment is 0.1; other parameters and variables and their value ranges are the same as in embodiment 1.

[0207] The other steps in this procedure are the same as in Example 1.

[0208] ④ Determine the final pseudo-label

[0209] Determine the final pseudo-label according to formula (8):

[0210] The expression of equation (8) is the same as that in Example 1.

[0211] In equation (8), T is the final iteration number, and T takes values ​​from 2 to 10. In this embodiment, T takes the value of 2. β is the weight factor, and β ∈ [0,1]. In this embodiment, β takes the value of 0. The meanings and value ranges of other parameters and variables are the same as those in embodiment 1.

[0212] The other steps in this procedure are the same as in Example 1.

[0213] The other steps are the same as in Example 1. The semantic alignment enhancement network construction method is now complete.

[0214] The aspect knowledge image fusion network construction method in this embodiment is as follows:

[0215] 1) Constructing a text knowledge fusion network

[0216] Construct a text knowledge fusion network according to formula (9):

[0217] The expression of equation (9) is the same as that in Example 1.

[0218] In equation (9), α is the residual weight, α∈(0,1]. In this embodiment, the value of α is 0.1. The meanings and value ranges of other parameters and variables are the same as those in embodiment 1.

[0219] The other steps in this step are the same as in Example 1. The semantic alignment enhancement network construction method is now complete.

[0220] (5) Training a multimodal sentiment analysis network

[0221] 1) Constructing the loss function

[0222] Construct the cross-entropy loss function L according to equation (1):

[0223] The expression of equation (1) is the same as that in Example 1.

[0224] In equation (1), w cFor resampling weights, w c ∈(0,1], w in this embodiment c The value is 0.1, β c β is the category adjustment factor. c ∈(0,1], β in this embodiment c The value is 0.1, B is the number of data entries in the batch, B∈[4,12], and the value of B in this embodiment is 4. J is the number of valid aspect words of the i-th data, J∈[1,6]. The value represents the sentiment prediction of words in the training set, ∈ is the smoothing label parameter, ∈ ∈ (0, 1); in this embodiment, ∈ takes the value of 0.1. The meanings and value ranges of other parameters and variables are the same as in Embodiment 1.

[0225] The other steps in this procedure are the same as in Example 1.

[0226] The other steps are the same as in Example 1. This completes the method for social media user sentiment analysis based on iterative pseudo-labels and knowledge enhancement.

[0227] Example 3

[0228] The social media user sentiment analysis method based on iterative pseudo-labels and knowledge enhancement in this embodiment consists of the following steps:

[0229] (1) Obtaining the dataset

[0230] The steps are the same as in Example 1.

[0231] (2) Split the dataset

[0232] The steps are the same as in Example 1.

[0233] (3) Extracting multimodal features

[0234] The steps are the same as in Example 1.

[0235] (4) Constructing a multimodal sentiment analysis network

[0236] The multimodal aspect-based sentiment analysis network structure in this embodiment is the same as that in Embodiment 1.

[0237] The semantic alignment enhancement network construction method in this embodiment is as follows:

[0238] 1) Generate similarity scores and pseudo-labels

[0239] ① Determine the similarity of each pair of image-aspect words according to formula (3):

[0240] The expression of equation (3) is the same as that in Example 1.

[0241] In equation (3), Sim(v) i ,ti,j )∈[-1,1], and other parameters, variables and their ranges are the same as in Example 1.

[0242] ② Generate the initial pseudo-tags according to formula (4):

[0243] The expression of equation (4) is the same as that in Example 1.

[0244] In equation (4), θ h As a positive threshold, θ h ∈(0,1), θ in this embodiment h The value is 0.9, θ w For a negative threshold, θ w ∈(0,1), θ in this embodiment w The value is 0.1, and other parameters, variables, and value ranges are the same as in Example 1. The other steps in this process are the same as in Example 1.

[0245] ③ Iterative optimization of the pre-trained model CLIP

[0246] The pre-trained model CLIP is optimized using the InfoNCE loss function to maximize positive sample similarity and minimize negative sample similarity.

[0247] Construct the lnfoNCE loss function according to equation (5):

[0248] The expression of equation (5) is the same as that in Example 1.

[0249] In equation (5), τ is a temperature parameter, τ∈(0,1], and the value of τ in this embodiment is 1; other parameters and variables and their value ranges are the same as in embodiment 1.

[0250] The other steps in this procedure are the same as in Example 1.

[0251] ④ Determine the final pseudo-label

[0252] Determine the final pseudo-label according to formula (8):

[0253] The expression of equation (8) is the same as that in Example 1.

[0254] In equation (8), T is the final iteration number, and T takes values ​​from 2 to 10. In this embodiment, T takes the value of 10. β is the weight factor, and β ∈ [0,1]. In this embodiment, β takes the value of 1. The meanings and value ranges of other parameters and variables are the same as those in embodiment 1.

[0255] The other steps in this procedure are the same as in Example 1.

[0256] The other steps are the same as in Example 1. The semantic alignment enhancement network construction method is now complete.

[0257] The aspect knowledge image fusion network construction method in this embodiment is as follows:

[0258] 1) Constructing a text knowledge fusion network

[0259] Construct a text knowledge fusion network according to formula (9):

[0260] The expression of equation (9) is the same as that in Example 1.

[0261] In equation (9), α is the residual weight, α∈(0,1]. In this embodiment, the value of α is 1. The meanings and value ranges of other parameters and variables are the same as those in embodiment 1.

[0262] The other steps in this step are the same as in Example 1. The semantic alignment enhancement network construction method is completed.

[0263] (5) Training a multimodal sentiment analysis network

[0264] 1) Constructing the loss function

[0265] Construct the cross-entropy loss function L according to equation (1):

[0266] The expression of equation (1) is the same as that in Example 1.

[0267] In equation (1), w c For resampling weights, w c ∈(0,1], w in this embodiment c The value of β is 1. c β is the category adjustment factor. c ∈(0,1], β in this embodiment c The value is 1, B is the number of data entries in the batch, B∈[4,12]; in this embodiment, the value of B is 12, and J is the number of valid aspect words of the i-th data, J∈[1,6]; The sentiment prediction for words in the training set is represented by ∈, which is the smoothing label parameter, ∈ ∈ (0, 1); in this embodiment, ∈ takes the value of 0.9. The meanings and value ranges of other parameters and variables are the same as in Embodiment 1.

[0268] The other steps in this procedure are the same as in Example 1.

[0269] The other steps are the same as in Example 1. This completes the method for social media user sentiment analysis based on iterative pseudo-labels and knowledge enhancement.

[0270] To verify the beneficial methods of this invention, the social media user sentiment analysis method based on iterative pseudo-labels and knowledge enhancement (hereinafter referred to as the method of this invention) in Example 1 of this invention was compared with the following experiments: Entity-sensitive attention and fusion network for entity-level multimodal sentiment classification (hereinafter referred to as Comparison Experiment 1), Adapting BERT for Target-Oriented Multimodal Sentiment Classification (hereinafter referred to as Comparison Experiment 2), Exploiting BERT for multimodal target sentiment classification through input space translation (hereinafter referred to as Comparison Experiment 3), Joint multi-modal aspect-sentiment analysis with auxiliary cross-modal relation detection (hereinafter referred to as Comparison Experiment 4), Vision-Language Pre-Training for Multimodal Aspect-Based Sentiment Analysis (hereinafter referred to as Comparison Experiment 5), Cross-Modal Multitask Transformer for End-to-End Multimodal Aspect-Based Sentiment Analysis (hereinafter referred to as Comparison Experiment 6), AoM: Detecting Aspect-oriented Information for Multimodal Aspect-Based Sentiment Analysis (hereinafter referred to as Comparison Experiment 7), and Aspects are Anchors: Towards Multimodal Aspect-based... A comparative experiment was conducted using Sentiment Analysis via Aspect-driven Alignment and Refinement (referred to as Comparative Experiment 8).

[0271] Calculate the F1 score according to formula (11):

[0272]

[0273] Among them, F1 c P represents the F1 score for category c; cP represents the precision for class c. c ∈[0,1];R c R represents the recall rate for category c. c ∈[0,1];TP c It is a true positive; FP c False positive; FN c It is a false negative.

[0274] Calculate the accuracy Acc according to formula (12):

[0275]

[0276]

[0277] Where Acc represents accuracy, acc i,j express Whether the prediction is accurate.

[0278] The experimental and calculation results are shown in Table 1.

[0279] Table 1. Experimental results of the present invention and comparative experiments.

[0280] Experimental methods Accuracy (%) F1 score (%) Comparative Experiment 1 73.4 67.4 Comparative Experiment 2 77.2 71.8 Comparative Experiment 3 78.0 73.2 Comparative Experiment 4 78.7 - Comparative Experiment 5 78.6 73.8 Comparative Experiment 6 77.9 - Comparative Experiment 7 80.2 75.9 Comparative Experiment 8 81.3 77.1 Method of the present invention 88.06 87.90

[0281] As shown in Table 1, in all experimental results, the average accuracy of the method of the present invention in the final five experiments was 88.06%, and the F1 score was 87.90%. The method of the present invention improves the accuracy by 14.66% and the F1 score by 20.5% compared to Comparative Experiment 1; improves the accuracy by 10.86% and the F1 score by 16.10% compared to Comparative Experiment 2; improves the accuracy by 4.60% and the F1 score by 14.70% compared to Comparative Experiment 3; improves the accuracy by 9.36% compared to Comparative Experiment 4; improves the accuracy by 9.46% and the F1 score by 14.1% compared to Comparative Experiment 5; improves the accuracy by 10.16% compared to Comparative Experiment 6; and improves the accuracy by 7% compared to Comparative Experiment 7. The accuracy and F1 score improved by 12.00%, which is 6.76% higher and 10.8% higher than those in Comparative Experiment 8. This method uses a semantic alignment network to determine the correlation between aspect words and images, and uses a knowledge enhancement network to enhance the semantic expression of the image content with external knowledge. Through a multimodal fusion layer, information from different modalities is integrated, which solves the technical problem of weak correlation between aspect words and images and difficulty in integrating multimodal information. Comparative experiments have verified that this method has the advantages of high accuracy and strong network robustness.

Claims

1. A social media user sentiment analysis method based on iterative pseudo-labeling and knowledge enhancement, characterized in that... It consists of the following steps: (1) Obtaining the dataset from IJCAI2019_data.zip - Google Drive Download the Twitter 2015 dataset to obtain multimodal data, totaling 3,502 tweets. Each tweet contains a text sentence, image, aspect words, and sentiment tags. After data cleaning, invalid image connections and data with missing sentiment tags were removed, resulting in a total of 3,502 images, 3,502 sentences, and 5,338 aspect words used as the experimental dataset. (2) Split the dataset The Twitter2015 dataset was divided into a training set, a validation set, and a test set. The training set contained 2801 data points, the validation set contained 350 data points, and the test set contained 350 data points. (3) Extracting multimodal features 1) Extract semantic alignment module features Image features v are extracted using the pre-trained model CLIP. i and aspect features t i,j , The features obtained by encoding the i-th image, d t t represents the dimension of the image embedding; i,j The feature obtained by encoding the j-th aspect word in the i-th data item. The dimension of aspect word embedding; 2) Extracting emotional module features in terms of fusion Combine text x using the following formula i : x i =s i [SEP]a i,1 ,a i,2 ,...,a i,j [SEP]k' i.j Among them, s i For the sentence of the i-th data, a i,j For aspect words, k' i.j For knowledge text, [SEP] is the delimiter, and x is processed using the pre-trained model CLIP. i Encode to obtain combined text features Using the pre-trained model CLIP to target the term a i,j and knowledge text k' i.j Encode the aspect word features to obtain them. and knowledge text features The image is preprocessed and encoded using the ResNet152 model to obtain image features. d v Indicates the dimension of the image embedding; (4) Constructing a multimodal sentiment analysis network The multimodal sentiment analysis network consists of a multimodal vocabulary knowledge enhancement fusion network and a sentiment classification network connected together; (5) Training a multimodal sentiment analysis network 1) Constructing the loss function Construct the cross-entropy loss function L according to equation (1): Among them, y i,j Indicates sentiment label, y i,j Let c ∈ {0,1,2}, where 0 represents positive, 1 represents negative, and 2 represents neutral emotion. For the smoothed sentiment label, N c w is the number of samples in category c. c For resampling weights, w c ∈(0,1];β c β is the category adjustment factor. c ∈(0,1]; B is the number of data entries in the batch, B∈[4,12]; J is the number of valid aspect terms for the i-th data entry, J∈[1,6]; Let ∈ represent the sentiment prediction of words in the training set, where ∈ is the smooth label parameter, and ∈ ∈ (0, 1); 2) Training a multimodal sentiment analysis network The training set was input into a multimodal aspect-based sentiment analysis network for training. The server was a GTX3090, the optimizer was AdamW, and adversarial training was performed using perturbation embedding with fast gradient method. The initial learning rate was 0.00005, and it was adjusted by cosine annealing scheduling. Linear warm-up was used in the early stage, and cosine annealing scheduling was used in the later stage. Training was carried out until the loss function L converged. (6) Testing multimodal sentiment analysis networks The test set is fed into the trained multimodal aspect-of-sentiment analysis network for testing, and the output is social media user sentiment. (7) Identifying social media user sentiment Multimodal features are input into the trained multimodal aspect-based sentiment analysis network, and regression is performed using fully connected layers and softmax layers to obtain the probability values ​​for different sentiment categories: positive probability p pos p pos ∈(0,1], negative probability p neg p neg ∈(0,1], neutral probability p neu p neu ∈(0,1]; Predict the aspect sentiment category according to formula (2): p=(p pos ,p neg ,p neu ) in, This indicates that the emotional polarity corresponding to the element with the largest probability distribution value is selected as the result.

2. The social media user sentiment analysis method based on iterative pseudo-labels and knowledge enhancement according to claim 1, characterized in that... In the multimodal aspect sentiment analysis network constructed in step (4), the multimodal aspect word knowledge enhancement fusion network is composed of a semantic alignment enhancement network and an aspect knowledge image fusion network connected in series.

3. The social media user sentiment analysis method based on iterative pseudo-labels and knowledge enhancement according to claim 2, characterized in that: The semantic alignment enhancement network consists of an image coding layer 1, a text coding layer 1, and a projection layer connected together, with the outputs of the image coding layer 1 and the text coding layer connected to the projection layer. The image encoding layer 1 is composed of an embedding layer 1, a transform layer 1, a normalization layer 1, and a pooling layer 1 connected in series; the text encoding layer 1 is composed of an embedding layer 2, a transform layer 2, a normalization layer 2, and a pooling layer 2 connected in series.

4. The social media user sentiment analysis method based on iterative pseudo-labels and knowledge enhancement according to claim 2, characterized in that: The aspect knowledge image fusion network is composed of text coding layer 2, text coding layer 3, text coding layer 4, image coding layer 2, text knowledge fusion network, and multimodal fusion network connected together; the output ends of text coding layer 2 and text coding layer 3 are connected to the text knowledge fusion network, the output ends of text coding layer 4 and image coding layer 2 are connected to the multimodal fusion network, and the text knowledge fusion network and the multimodal fusion network are connected in series.

5. The social media user sentiment analysis method based on iterative pseudo-labels and knowledge enhancement according to claim 4, characterized in that: The text encoding layer 2 is composed of an embedding layer 3, a Transformer layer 3, a normalization layer 3, and a pooling layer 3 connected in series.

6. The social media user sentiment analysis method based on iterative pseudo-labels and knowledge enhancement according to claim 4, characterized in that: The image coding layer 2 is composed of a convolutional layer, a residual layer, and a pooling layer 4 connected in series.

7. The social media user sentiment analysis method based on iterative pseudo-labels and knowledge enhancement according to claim 4, characterized in that: The text knowledge fusion network is composed of a normalization layer 4, a multi-head attention layer 1, a feedforward neural network 1, and a pooling layer 5 connected in series; the feedforward neural network 1 is composed of a linear layer 1, a linear rectifier layer 1, and a linear layer 2 connected in series.

8. The social media user sentiment analysis method based on iterative pseudo-labels and knowledge enhancement according to claim 4, characterized in that: The multimodal fusion network is composed of a linear projection module connected in series with a normalization layer 5, a multi-head attention layer 2, a feedforward neural network 2, and a pooling layer 6; the linear projection module is composed of a linear projection layer 1 connected in parallel with a linear projection layer 2 and a linear projection layer 3; the feedforward neural network 2 is composed of a linear layer 3 connected in series with a linear rectification layer 2, a linear layer 4, a linear rectification layer 3, and a linear layer 5.

9. The social media user sentiment analysis method based on iterative pseudo-labels and knowledge enhancement according to claim 2 or 3, characterized in that... In the multimodal sentiment analysis network constructed in step (4), the semantic alignment enhancement network construction method is as follows: 1) Generate similarity scores and pseudo-labels ① Determine the similarity of each pair of image-aspect words according to formula (3): Among them, v i ×t i,j Denotes the vector dot product, ||v i ||,||t i,j || denotes the Euclidean norm of the vector's magnitude, Sim(v i ,t i,j )∈[-1,1]: ② Generate the initial pseudo-tags according to formula (4): Among them, l i,j ∈{1,0}, where 1 indicates correlation and 0 indicates no correlation, θ h As a positive threshold, θ h ∈(0,1), θ w For a negative threshold, θ w ∈(0,1), and θ h +θ w ≤1; Pseudo-labeled dataset for: Among them, s i This is the sentence of the i-th data item; ③ Iterative optimization of the pre-trained model CLIP The pre-trained model CLIP is optimized using the InfoNCE loss function to maximize positive sample similarity and minimize negative sample similarity. Construct the lnfoNCE loss function according to equation (5): Where M is the number of data entries, determined according to the dataset; J is the number of aspect terms, J∈[1,6]; τ is the temperature parameter, τ∈(0,1]; the average similarity of positive samples is calculated according to equation (6). Among them, s i,j For positive samples, the similarity is N. pos This represents the number of positive samples. Adjust the positive sample quantiles according to formula (7). Where, γ posw γ is the lower limit threshold for the average similarity of positive samples. posh lnfoNCE is the upper limit threshold for the average similarity of positive samples. pos Let δ be the loss component of the lnfoNCE for positive samples, t represent the t-th iteration, and δ pos The threshold for positive sample loss; q m q represents the minimum value of the quantile proportion. M The maximum value of the quantile proportion; Δ q The adjustment step size for the quantile proportion, ∈ s The threshold for the rate of change of loss; Adjusting negative sample quantiles Methods and Adjustment of Positive Sample Quantiles The method is the same; ④ Determine the final pseudo-label Determine the final pseudo-label according to formula (8): Where T is the final iteration number, T takes values ​​from 2 to 10, β is the weight factor, β∈[0,1], Sim′(v i ,t i,j The similarity is obtained after optimizing the model. 2) Knowledge Enhancement ①Generate instructions The large language model generates a fixed instruction template Instr based on the user's description, and fills in the sentence s when processing specific data. i and aspect words t i,j Generate instructions; ② Generate knowledge text Using a visual large language model to process pseudo-label datasets l in i,j =1, generate knowledge text k' i.j , For l i,j =0, knowledge text k' i.j equals sentence s i .

10. The social media user sentiment analysis method based on iterative pseudo-labels and knowledge enhancement according to claim 2 or 4, characterized in that... The method for constructing the aspect knowledge image fusion network is as follows: 1) Constructing a text knowledge fusion network Construct a text knowledge fusion network according to formula (9): d h =d t / H in, For fusion features; W q W k W v It is a linear transformation matrix. d h For a single-head dimension, H is the number of attention heads; α is the residual weight, α∈(0,1], FFN1 is a feedforward network, a two-layer fully connected network; 2) Construct a multimodal fusion network Construct a multimodal fusion network according to equation (10): in, It is a linear transformation matrix. FFN2 is a feedforward network, a three-layer fully connected network; As a combination feature, For the image features after projection alignment, W img It is a linear transformation matrix.

Images