A social media multi-modal sentiment classification method based on prompt driving and contrast learning

Abstract

This invention proposes a cue-driven and contrastive learning-based multimodal sentiment classification method for social media, aiming to improve the accuracy of sentiment classification in social media content. By introducing cue information to optimize the text backbone structure, removing noise and redundant information, and combining visual data augmentation techniques, the model's ability to understand complex sentiment information is enhanced. Textual and visual features are extracted and fused using models such as ResNet-50, RoBERTa, and Transformer. A contrastive learning strategy deepens the connections between modalities, and finally, consistency loss and classification loss are used to optimize model performance. This invention demonstrates outstanding performance on MVSA-Single, MVSA-Multiple, and HFM datasets, providing a more accurate sentiment analysis tool for fields such as social media monitoring, customer service, and marketing strategies.

Description

Technical Field

[0001] This invention belongs to the technical field of natural language processing, and in particular relates to a method for multimodal sentiment classification of social media based on prompt-driven and contrastive learning. Background Technology

[0002] With the rapid development of information technology, social media has become an important platform for people to express their feelings, attitudes, and perspectives. People are no longer limited to single-modal text communication but use multiple modalities to express themselves comprehensively and from multiple angles. Sentiment classification aims to identify and classify the emotional tendencies contained in various modalities of information, categorizing them as positive, negative, or neutral. Early sentiment classification models mostly utilized single-modal data mining to extract sentiment information, which easily led to missing context and biased classification results, making it difficult to meet the needs of accurate and comprehensive sentiment recognition and classification. Multimodal sentiment classification, on the other hand, can integrate multiple modal information sources, more comprehensively capture emotional cues, and improve the accuracy of sentiment classification.

[0003] While existing multimodal sentiment classification methods have improved the accuracy of sentiment recognition to some extent, they still face several challenges. Firstly, textual modalities contain important sentiment cues, providing more detailed and accurate sentiment descriptions. However, textual data in social media often contains grammatical errors and redundant information, potentially introducing additional noise that affects the accurate expression of the text and may even mislead sentiment classification. This not only increases the complexity of model processing but may also reduce the accuracy of sentiment recognition. Existing methods often overlook this when processing textual modal information. Secondly, existing contrastive learning-based multimodal sentiment classification methods often focus only on simple feature matching when designing contrastive learning tasks. While this can improve the model's generalization ability to some extent, overly simplistic contrastive learning tasks may not fully explore and utilize the sentiment cues in multimodal information, thus limiting the model's performance in sentiment classification. Furthermore, neglecting contrastive learning between sentiment polarities may prevent the model from fully learning the subtle feature differences between samples of different sentiment polarities, thus affecting the model's accuracy and robustness in sentiment classification.

[0004] Therefore, this invention aims to address the problem of insufficient handling of errors and redundant information in text modalities and optimize the design of contrastive learning tasks to apply multimodal sentiment classification technology to fields such as social media monitoring, customer service, and marketing strategy formulation. In social media monitoring, this technology can analyze public sentiment towards specific brands, products, or events in real time, providing valuable market insights for businesses. In customer service, by analyzing information posted by customers, businesses can accurately identify customers' emotional states, thereby adjusting service strategies and improving customer satisfaction. In marketing strategy formulation, it helps businesses gain a deeper understanding of audiences' emotional responses to advertising content, providing a scientific basis for optimizing marketing strategies. Summary of the Invention

[0005] The purpose of this invention is to provide a social media multimodal sentiment classification method based on cue-driven and contrastive learning, in order to solve the problem of insufficient processing of text modal noise and redundant information when users post informal language, emoticons and other multimedia content on social media.

[0006] The technical solution adopted in this invention is a social media multimodal sentiment classification method based on cue-driven and contrastive learning, comprising the following steps:

[0007] Step S1: Obtain text and visual modal data samples from social media and preprocess the data samples;

[0008] Step S2: Construct the main text structure prompt information;

[0009] Step S3: Extract the modal feature sequences of the original text and the data-enhanced text, divide the text data into sentences, and add special identifiers [CLS] and [SEP] to mark the beginning and end of the sentences respectively;

[0010] Step S4: Extract visual feature sequences for the visual modality of each branch;

[0011] Step S5: Construct the network and input the fused features into an attention layer to deeply explore the potential connections between modalities;

[0012] Step S6: Compare and learn the original text-visual sample pairs after feature fusion and the text-visual sample pairs with added prompts with the data-enhanced text-visual sample pairs respectively.

[0013] Step S7: Calculate the consistency loss between the original sample pairs and the sample pairs with added prompt text, as well as the final classification loss, to further optimize the model;

[0014] Step S8: Obtain the sentiment classification result through a multimodal sentiment classifier.

[0015] Furthermore, step S1 specifically includes:

[0016] S11, divide the dataset into training set, validation set, and test set;

[0017] S12, Remove visual-text pairs with inconsistent labels;

[0018] S13. Data from different modalities are processed uniformly, and data augmentation techniques are used to expand the dataset. Synonym replacement, random insertion, and reverse translation data augmentation techniques are used to expand the text modal dataset, and cropping, color transformation, rotation, and chroma separation data augmentation techniques are used to expand the visual modal dataset.

[0019] Furthermore, in step S2, the specific formula for constructing the text backbone structure prompt information is as follows:

[0020] P = Llama(t1,t2,…,t) s )

[0021] Where P represents the text backbone structure prompt information constructed by the Llama3 model, t1, t2, ..., t s represents the original text slice, s represents the length of the original text, and Llama(·) represents the process of the Llama3 model generating prompt text information.

[0022] Furthermore, in step S3, the segmented text is converted into a list of IDs that the BERT model can recognize, which can be specifically represented as:

[0023] T origin =token_to_id([[CLS],t1,t2,…,t s ,[SEP]])

[0024] T aug =token_to_id([[CLS],a1,a2,…,a k ,[SEP]])

[0025] Among them, T origin T represents the preprocessed original text. aug Represents the preprocessed data augmented text, t1, t2, ..., t s This represents a slice of the original text, where s represents the length of the original text, and a, a2, ..., a k This represents the data-augmented text slice, k represents the length of the data-augmented text, and token_to_id(·) represents the operation of converting the segmented text into a list of IDs that the BERT model can recognize;

[0026] Then, the text modal feature sequence with added prompt information in step S2 is extracted, and the BERT word segmenter is used to segment the sentence. The original text and the prompt information are connected by a special identifier [SEP], and the beginning and end are marked by [CLS] and [SEP] respectively. The segmented text with added prompt information is converted into a list of IDs that the BERT model can recognize, which can be represented as follows:

[0027] T prompt =token_to_id([[CLS],t1,t2,…,t s [SEP],p1,p2,…,p m ,[SEP]])

[0028] Among them, T prompt This represents the preprocessed text with added prompts, t1, t2, ..., t s This represents a slice of the original text, where s represents the length of the original text, and p1, p2, ..., p m This represents a slice of the prompt text generated by the Llama3 model, where m represents the length of the prompt text, and token_to_id(·) represents the operation of converting the segmented text into a list of IDs that the BERT model can recognize;

[0029] Then T origin T aug and T prompt The input is fed into the BERT model for encoding to extract the feature sequences of the original text and the data-augmented text, as shown below:

[0030]

[0031] in, This represents the sequence of original text features extracted by BERT. This represents the text feature sequence after data augmentation extracted by BERT. This represents the text feature sequence with added prompts extracted by BERT, where BERT(·) denotes the feature extraction process using the BERT model, and T origin T represents the preprocessed original text. aug T represents preprocessed data augmentation text. prompt This refers to the text containing the added prompt information after preprocessing.

[0032] Furthermore, in step S4, the specific visual modality feature sequence is represented as follows:

[0033]

[0034]

[0035] Among them, I origin Indicates the original visual input, I aug ResNet(·) indicates enhancing visual input; ResNet(·) indicates using the ResNet-50 model for visual feature extraction. This represents the original visual feature sequence extracted by the ResNet model. This represents the enhanced visual feature sequence extracted by the ResNet model, and RoBERTa(·) represents the visual modality features further extracted using the RoBERTa model. This represents the final extracted sequence of original visual features. This represents the final extracted data after augmentation of the visual feature sequence.

[0036] Furthermore, in step S5, the specific formula for fusing and aligning cross-modal features and inputting the fused features into an attention layer to further explore the potential connections between modalities is as follows:

[0037]

[0038] Where Cat(·) represents the cascading operation, H represents the feature of the text-visual modality simple connection, H o H represents the simple connection features of the original sample pairs. a H represents the simple connection features of data-augmented sample pairs. p Let f represent the simple connection features of the sample pairs with added hints, and let f represent the fused features obtained after the Transformer. o f represents the original text-visual pair fusion features obtained through the Transformer. a f represents the text-visual pair fusion features obtained after data augmentation by the Transformer. p This represents the text-visual pair fusion features obtained by the Transformer that incorporate cue information, Q. f This represents the query vector obtained by f. K represents the key vector obtained from f. f The transpose of V f Let f represent the value vector obtained by f, and F represent the final fused features obtained by the attention layer. o F represents the fusion feature of the raw data obtained by the attention layer. a F represents the fusion feature of the data augmented by the attention layer. p This represents the fusion features of the data with added cues obtained from the attention layer. This represents the final extracted sequence of original visual features. This represents the final extracted visual feature sequence after data enhancement. This represents the sequence of original text features extracted by BERT. H represents the augmented text feature sequence extracted by BERT. t p This represents the text feature sequence with added prompts extracted by BERT, TF(·) represents the process of cross-modal feature fusion using the Transformer fusion network, and softmax(·) represents the normalization process.

[0039] Furthermore, in step S6, the specific representation of contrastive learning is as follows:

[0040]

[0041] Among them, F oa F represents the matrix product between the original data sample and the data-augmented data sample. o This represents the final extracted fusion features of the original text-visual pair. F represents the transpose of the final extracted fusion features of the original text-visual pair. pa F represents the matrix product between the data sample with added prompts and the data sample with data augmentation. p The text-visual fusion feature representing the final extracted text-visual pair with added prompts is shown, where τ represents the temperature parameter with a value of 0.07, and L... oacl L represents the contrast loss between the original data sample and the data-augmented sample. pacl y represents the contrast loss between the data samples with added prompts and the data-augmented samples. i =y j This represents samples with the same emotional polarity, y i ≠y j This represents samples with different emotional polarities, y i Let y represent the i-th data sample. j Let j represent the j-th data sample, Dot(·) denotes the dot product similarity calculation process, and n represent the number of samples in the batch. This represents the matrix product of the i-th original data sample and the data-augmented data sample. This represents the matrix product of the j-th original data sample and the data-augmented data sample. This represents the matrix product of the i-th data sample with added prompt information and the data augmentation data sample. This represents the matrix product of the j-th data sample with added prompt information and the data augmentation data sample.

[0042] Furthermore, in step S7, the specific steps for calculating the loss are as follows:

[0043] S71, the consistency loss between the original sample pairs and the samples with added prompt text is specifically expressed as:

[0044]

[0045] Among them, L opcl This represents the consistency loss between the original sample pairs and the sample pairs with added prompt text. This represents the fused feature of the i-th original sample to the final extracted features. denoted as the fusion feature extracted from the i-th sample with added prompt information, cos(·) represents the calculation of cosine similarity, n represents the number of samples in the batch, i represents the i-th sample, τ represents the temperature parameter with a value of 0.07, γ1 represents the hyperparameter coefficient in the mean square error calculation process with a value of 0.8, and γ2 represents the hyperparameter coefficient in the cosine similarity consistency loss calculation process with a value of 0.2.

[0046] S72, calculate the final classification loss, specifically expressed as:

[0047]

[0048] Among them, L CE CrossEntropyLoss(·) represents the classification loss, while CrossEntropyLoss(·) represents the cross-entropy loss. This represents the model's predicted output, and label represents the sample's true label.

[0049] S73, calculate the total loss, specifically as follows:

[0050] L total =L CE +λ1L opcl +λ2L oacl +λ3L pacl

[0051] Among them, L total L represents the total loss. CE L represents the classification loss. opcl L represents the consistency loss between the original sample pairs and the samples with added prompt text. oacl L represents the contrast loss between the original data sample and the data-augmented sample. pacl λ1 represents the hyperparameter coefficient of the contrast loss between the data sample with added prompt information and the data augmented sample, with a value of 0.4; λ2 represents the hyperparameter coefficient of the contrast loss between the data sample with added prompt information and the data augmented sample, with a value of 0.3; and λ3 represents the hyperparameter coefficient of the consistency loss between the original sample and the sample with added prompt text, with a value of 0.3.

[0052] Furthermore, in step S8, the sentiment classification result is specifically represented as follows:

[0053]

[0054] Where FC(·) represents a fully connected layer, and GeLU(·) represents an activation function. F represents the model's predicted output. o This represents the fusion features of the raw data obtained from the attention layer.

[0055] The beneficial effects of this invention are:

[0056] 1. This invention effectively reduces the impact of noise and redundant information in social media text on the accuracy of sentiment classification by introducing a prompt-driven mechanism and a contrastive learning strategy. This enables the model to more accurately capture sentiment cues in textual and visual modal information, thereby significantly improving the accuracy of sentiment classification.

[0057] 2. This invention fully considers the complementarity and correlation between multimodal data. By constructing contrastive loss and consistency loss functions, it promotes the effective fusion of features between different modalities, which not only enhances the model's ability to process complex emotional information, but also enables the model to more comprehensively understand the emotional expression in social media content. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a flowchart illustrating the multimodal emotion classification method in an embodiment of the present invention.

[0060] Figure 2 This is a schematic diagram of the structure of the multimodal emotion classification model in an embodiment of the present invention. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] Example 1

[0063] like Figures 1-2 As shown, embodiments of the present invention provide a method for multimodal sentiment classification on social media based on cue-driven and contrastive learning, the specific steps of which include:

[0064] Step S1: Obtain text and visual modal data samples from social media, denoted as T and I respectively, and preprocess the data samples;

[0065] S11, divide the training set, validation set and test set into three sets with a ratio of 8:1:1;

[0066] S12, remove visual-text pairs with inconsistent labels. Specifically, for the MVSA-Single dataset labeled by one annotator, if the text and visual modal labels represent positive and negative respectively, the sample pair is considered invalid and needs to be removed. For the MVSA-Multiple dataset labeled by three annotators, in the single modal label, the label is considered a valid label for the modality only when at least two of the three annotators have the same label. Thus, samples with inconsistent text and visual modal labels are removed to reduce the impact on the model.

[0067] S13. Data from different modalities are processed uniformly, and data augmentation techniques are used to expand the dataset. Synonym replacement, random insertion, and reverse translation data augmentation techniques are used to expand the text modal dataset; cropping, color transformation, rotation, and chroma separation data augmentation techniques are used to expand the visual modal dataset.

[0068] In this embodiment, the text and visual modal datasets related to social media include MVSA-Single, MVSA-Multiple, and HFM. The MVSA dataset is a three-class classification task (positive, neutral, and negative). MVSA-Single contains 5129 visual-text pairs, each labeled by one annotator. MVSA-Multiple contains 19600 visual-text pairs, each labeled by three annotators. HFM is a binary classification task dataset (positive and negative), containing 24635 visual-text pairs. To reduce the negative impact of erroneous samples, samples with inconsistent visual-text labels in the MVSA dataset were removed, and the dataset was then divided into training, validation, and test sets. The final dataset statistics are shown in Table 1.

[0069] Table 1 Statistical results for different datasets

[0070] Dataset category positive neutral negative total MVSA-Single 3 2683 470 1358 4511 MVSA-Multiple 3 11320 4408 1088 16816 HFM 2 10560 -- 14075 24635

[0071] Step S2: Construct the text backbone structure prompt information using the Llama3 model, specifically as follows:

[0072] P = Llama(t1,t2,…,t) s )

[0073] Where P represents the text backbone structure prompt information constructed by the Llama3 model, t1, t2, ..., t s represents the original text slice, s represents the length of the original text, and Llama(·) represents the process of the Llama3 model generating prompt text information.

[0074] By connecting P as a cue to the original text and forming a data sample pair branch, the model's understanding of the text is enhanced, mitigating the adverse effects of grammatical errors, interference, and redundant information. For example, the text "MT@explincolnshire@LostVillages Holloway,once abusyroad,atGainsthorpe#deserted#medieval village,Lincs" posted on social media contains redundant information such as "MT@explincolnshire" and "#deserted," which hinders the model's understanding of the text's correct meaning and may introduce grammatical errors. The Llama3 model extracts the text's core structure as "Hollowaywas once abusyroad." This generated core structure removes interference and redundant information from the original text and extracts the main idea, aiding the model's understanding of the text.

[0075] Step S3: Extract the modal feature sequences of the original text and the data-augmented text. Use the BERT tokenizer to segment the text data into sentences, and add special identifiers [CLS] and [SEP] to mark the beginning and end of the sentences respectively. Convert the segmented text into a list of IDs that the BERT model can recognize, specifically as follows:

[0076] T origin =token_to_id([[CLS],t1,t2,…,t s ,[SEP]])

[0077] T aug =token_to_id([[CLS],a1,a2,…,a k ,[SEP]])

[0078] Among them, T origin T represents the preprocessed original text. aug Represents the preprocessed data augmented text, t1, t2, ..., t s This represents a slice of the original text, where s represents the length of the original text, and a, a2, ..., ak This represents the text slice after data augmentation, k represents the length of the data-augmented text, and token_to_id(·) represents the operation of converting the segmented text into a list of IDs that the BERT model can recognize.

[0079] The text modal feature sequence with added prompt information in step S2 is extracted, and the BERT word segmenter is used to segment the sentence. The original text and the prompt information are connected by a special identifier [SEP], and the beginning and end are marked by [CLS] and [SEP] respectively. The segmented text with added prompt information is converted into a list of IDs that the BERT model can recognize, which can be represented as follows:

[0080] T prompt =token_to_id([[CLS],t1,t2,…,t s [SEP],p1,p2,…,p m ,[SEP]])

[0081] Among them, T prompt This represents the preprocessed text with added prompts, t1, t2, ..., t s This represents a slice of the original text, where s represents the length of the original text, and p1, p2, ..., p m This represents a slice of the prompt text generated by the Llama3 model, where m represents the length of the prompt text, and token_to_id(·) represents the operation of converting the segmented text into a list of IDs that the BERT model can recognize.

[0082] Then T origin T aug and T prompt The input is fed into the BERT model for encoding to extract the feature sequences of the original text and the data-augmented text, as shown below:

[0083]

[0084] in, This represents the sequence of original text features extracted by BERT. This represents the text feature sequence after data augmentation extracted by BERT. This represents the text feature sequence with added prompts extracted by BERT, where BERT(·) denotes the feature extraction process using the BERT model, and T origin T represents the preprocessed original text. aug T represents preprocessed data augmentation text. prompt This refers to the text containing the added prompt information after preprocessing.

[0085] Step S4: For each branch's visual modality, extract visual feature sequences using ResNet-50 and RoBERTa models. The specific visual modality feature sequences are represented as follows:

[0086]

[0087] Among them, I origin Indicates the original visual input, I aug ResNet(·) indicates enhancing visual input; ResNet(·) indicates using the ResNet-50 model for visual feature extraction. This represents the original visual feature sequence extracted by the ResNet model. This indicates the enhanced visual feature sequence extracted by the ResNet model; RoBERTa(·) indicates that visual modal features are further extracted using the RoBERTa model. This represents the final extracted sequence of original visual features. This represents the final extracted data after augmentation of the visual feature sequence.

[0088] Step S5: Construct a Transformer-based network to achieve the fusion and alignment of cross-modal features, and input the fused features into an attention layer to deeply explore the potential connections between modalities. Specifically, this is represented as follows:

[0089]

[0090] Where Cat(·) represents the cascading operation, H represents the feature of the text-visual modality simple connection, H o H represents the simple connection features of the original sample pairs. a H represents the simple connection features of data-augmented sample pairs. p Let f represent the simple connection features of the sample pairs with added hints, and let f represent the fused features obtained after the Transformer. o f represents the original text-visual pair fusion features obtained through the Transformer. a f represents the text-visual pair fusion features obtained after data augmentation by the Transformer. p This represents the text-visual pair fusion features obtained by the Transformer that incorporate cue information, Q. f This represents the query vector obtained by f. K represents the key vector obtained from f. f The transpose of V f Let f represent the value vector obtained by f, and F represent the final fused features obtained by the attention layer. o F represents the fusion feature of the raw data obtained by the attention layer.a F represents the fusion feature of the data augmented by the attention layer. p This represents the fusion features of the data with added cues obtained from the attention layer. This represents the final extracted sequence of original visual features. This represents the final extracted visual feature sequence after data enhancement. This represents the sequence of original text features extracted by BERT. This represents the text feature sequence after data augmentation extracted by BERT. This represents the text feature sequence with added prompts extracted by BERT, TF(·) represents the process of cross-modal feature fusion using the Transformer fusion network, and softmax(·) represents the normalization process.

[0091] Step S6: Compare and learn from the original text-visual sample pairs after feature fusion and the text-visual sample pairs with added prompts, respectively, with the data-enhanced text-visual sample pairs. By comparing positive and negative samples, the model learns an effective representation of the data, thereby improving the performance of the model in the sentiment classification task. This makes the distance between samples with the same sentiment polarity in the feature space become closer, while the distance between samples with different sentiment polarities becomes farther. Specifically, this is expressed as follows:

[0092]

[0093]

[0094] Among them, F oa F represents the matrix product between the original data sample and the data-augmented data sample. o This represents the final extracted fusion features of the original text-visual pair. F represents the transpose of the final extracted fusion features of the original text-visual pair. pa F represents the matrix product between the data sample with added prompts and the data sample with data augmentation. p The text-visual fusion feature representing the final extracted text-visual pair with added prompts is shown, where τ represents the temperature parameter with a value of 0.07, and L... oacl L represents the contrast loss between the original data sample and the data-augmented sample. pacl y represents the contrast loss between the data samples with added prompts and the data-augmented samples. i =y j This represents samples with the same emotional polarity, y i ≠y j This represents samples with different emotional polarities, y i Let y represent the i-th data sample.j Let j represent the j-th data sample, Dot(·) denotes the dot product similarity calculation process, and n represent the number of samples in the batch. This represents the matrix product of the i-th original data sample and the data-augmented data sample. This represents the matrix product of the j-th original data sample and the data-augmented data sample. This represents the matrix product of the i-th data sample with added prompt information and the data augmentation data sample. This represents the matrix product of the j-th data sample with added prompt information and the data augmentation data sample.

[0095] Step S7: Calculate the consistency loss between the original sample pairs and the sample pairs with added prompt text, as well as the final classification loss, to further optimize the model;

[0096] S71, the consistency loss between the original sample pairs and the samples with added prompt text is specifically expressed as:

[0097]

[0098] Among them, L opcl This represents the consistency loss between the original sample pairs and the sample pairs with added prompt text. This represents the fused feature of the i-th original sample to the final extracted features. Let represent the fusion feature extracted from the i-th sample with added prompt information, cos(·) represents the calculation of cosine similarity, n represents the number of samples in the batch, i represents the i-th sample, τ represents the temperature parameter with a value of 0.07, γ1 represents the hyperparameter coefficient in the mean square error calculation process with a value of 0.8, and γ2 represents the hyperparameter coefficient in the cosine similarity consistency loss calculation process with a value of 0.2.

[0099] S72, calculate the final classification loss, specifically expressed as:

[0100]

[0101] Among them, L CE CrossEntropyLoss(·) represents the classification loss, while CrossEntropyLoss(·) represents the cross-entropy loss. This represents the model's predicted output, and label represents the sample's true label.

[0102] S73, calculate the total loss, specifically as follows:

[0103] L total =L CE +λ1L opcl +λ2L oacl +λ3L pacl

[0104] Among them, L total L represents the total loss. CE L represents the classification loss. opcl L represents the consistency loss between the original sample pairs and the samples with added prompt text. oacl L represents the contrast loss between the original data sample and the data-augmented sample. pacl λ1 represents the hyperparameter coefficient of the contrast loss between the data sample with added prompt information and the data augmented sample, with a value of 0.4; λ2 represents the hyperparameter coefficient of the contrast loss between the data sample with added prompt information and the data augmented sample, with a value of 0.3; and λ3 represents the hyperparameter coefficient of the consistency loss between the original sample and the sample with added prompt text, with a value of 0.3.

[0105] Step S8: Obtain the sentiment classification result through a multimodal sentiment classifier, specifically as follows:

[0106]

[0107] Where FC(·) represents a fully connected layer, and GeLU(·) represents an activation function. F represents the model's predicted output. o This represents the fusion features of the raw data obtained from the attention layer.

[0108] Sentiment classification enables sentiment analysis of multimodal data in areas such as social media user comments, customer service, and marketing strategy development. For binary classification tasks, the result is either positive or negative; for triangular classification tasks, the result is either positive, negative, or neutral. Sentiment classification allows for a deeper understanding of user groups' emotional responses, thereby improving service satisfaction.

[0109] Experimental verification

[0110] To verify the effectiveness of the proposed model, comparative experiments were conducted on three multimodal sentiment classification datasets: MVSA-Single, MVSA-Multiple, and HFM. Tables 2 and 3 summarize the performance comparison of baseline methods on these three datasets. The performance of all baselines is derived from the open-source code and parameters given in the paper.

[0111] Table 2: Experimental results of each model on the MVSA dataset

[0112]

[0113]

[0114] Table 3: Experimental results of each model on the HFM dataset

[0115]

[0116] The results show that most multimodal sentiment classification models outperform unimodal models on all three datasets, indicating that multimodal data provides more effective information for sentiment classification tasks. Furthermore, this invention demonstrates strong competitiveness against a range of baseline models on the MVSA and HFM datasets. Specifically, as shown in Table 2, in the three-class sentiment classification task, this invention outperforms other methods in both ACC and F1 on the MVSA-Single dataset, achieving an accuracy improvement of 8.38% over MultiSentiNet and 2.89% over CLMLF. For the MVSA-Multiple dataset, the method achieves an accuracy improvement of 2.27% over the ITIN method and 6.93% over the MultiSentiNet method. As shown in Table 3, in the two-class multimodal sentiment classification task, this invention outperforms other baseline models, achieving improvements of 1.29% and 1.81% in ACC and F1 compared to CLMLF, respectively.

[0117] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0118] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for multimodal sentiment classification in social media based on cue-driven and contrastive learning, characterized in that, Includes the following steps: Step S1: Obtain text and visual modal data samples from social media and preprocess the data samples; Step S2: Construct the main text structure prompt information; Step S3: Extract the modal feature sequences of the original text and the data-enhanced text, divide the text data into sentences, and add special identifiers [CLS] and [SEP] to mark the beginning and end of the sentences respectively; Step S4: Extract visual feature sequences for the visual modality of each branch; Step S5: Construct the network and input the fused features into an attention layer to deeply explore the potential connections between modalities; Step S6: Compare and learn the original text-visual sample pairs after feature fusion and the text-visual sample pairs with added prompts with the data-enhanced text-visual sample pairs respectively. Step S7: Calculate the consistency loss between the original sample pairs and the sample pairs with added prompt text, as well as the final classification loss, to further optimize the model; Step S8: Obtain the sentiment classification result through a multimodal sentiment classifier; In step S2, the specific formula for constructing the text backbone structure prompt information is as follows: in, P This indicates the text backbone structure prompt information constructed using the Llama3 model. Represents a slice of the original text. Indicates the length of the original text. The process of the model generating prompt text information; In step S6, the specific representation of contrastive learning is as follows: in, This represents the matrix product between the original data sample and the data-augmented data sample. This represents the final extracted fusion features of the original text-visual pair. This represents the transpose of the final extracted fusion features of the original text-visual pair. This represents the matrix product between the data sample with added prompts and the data sample with data augmentation. This represents the final extracted text-visual fusion features that include the added prompt information. This represents the temperature parameter, with a value of 0.

07. This represents the contrast loss between the original data sample and the data-augmented sample. This represents the contrast loss between the data samples with added prompts and the data-augmented samples. This indicates samples with the same emotional polarity. This represents samples with different emotional polarities. Indicates the first One data sample, Indicates the first One data sample, This describes the process of calculating dot product similarity. Indicates the number of samples in the batch. Indicates the first The matrix product of the original data sample and the data-augmented data sample. Indicates the first The matrix product of the data sample with added prompts and the data sample with data augmentation. Indicates the first The matrix product of a data sample with added prompts and a data sample with data augmentation.

2. The social media multimodal sentiment classification method based on cue-driven and contrastive learning according to claim 1, characterized in that, Step S1 specifically includes: S11, divide the dataset into training set, validation set, and test set; S12, Remove visual-text pairs with inconsistent labels; S13. Data from different modalities are processed uniformly, and data augmentation techniques are used to expand the dataset. Synonym replacement, random insertion, and reverse translation data augmentation techniques are used to expand the text modal dataset, and cropping, color transformation, rotation, and chroma separation data augmentation techniques are used to expand the visual modal dataset.

3. The social media multimodal sentiment classification method based on cue-driven and contrastive learning according to claim 1, characterized in that, In step S3, the segmented text is converted into a list of IDs that the BERT model can recognize, specifically as follows: in, This represents the original text after preprocessing. This indicates text that has been preprocessed and augmented. Represents a slice of the original text. Indicates the length of the original text. This represents a text slice after data augmentation. Indicates the length of the augmented text. This represents the operation of converting the segmented text into a list of IDs that the BERT model can recognize; Then, the text modal feature sequence with added prompt information in step S2 is extracted, and the BERT word segmenter is used to segment the sentence. The original text and the prompt information are connected by a special identifier [SEP], and the beginning and end are marked by [CLS] and [SEP] respectively. The segmented text with added prompt information is converted into a list of IDs that the BERT model can recognize, specifically represented as follows: in, This refers to the pre-processed text containing the added prompt message. Represents a slice of the original text. Indicates the length of the original text. This represents a slice of the prompt text generated by the Llama3 model. Indicates the length of the prompt text. This represents the operation of converting the segmented text into a list of IDs that the BERT model can recognize; Furthermore The input is fed into the BERT model for encoding to extract the feature sequences of the original text and the data-augmented text, as shown below: in, This represents the sequence of original text features extracted by BERT. This represents the text feature sequence after data augmentation extracted by BERT. This represents the text feature sequence with added prompts extracted by BERT. This describes the process of feature extraction using the BERT model. This represents the original text after preprocessing. This indicates text that has been preprocessed and augmented. This refers to the text containing the added prompt information after preprocessing.

4. The social media multimodal sentiment classification method based on cue-driven and contrastive learning according to claim 1, characterized in that, In step S4, the specific visual modality feature sequence is represented as follows: in, Represents the original visual input. This indicates enhanced visual input; This indicates that the ResNet-50 model is used for visual feature extraction. This represents the original visual feature sequence extracted by the ResNet model. This represents the enhanced visual feature sequence extracted by the ResNet model. This indicates that the RoBERTa model is used to further extract visual modal features. This represents the final extracted sequence of original visual features. This represents the final extracted data after augmentation of the visual feature sequence.

5. The social media multimodal sentiment classification method based on cue-driven and contrastive learning according to claim 1, characterized in that, In step S5, the specific formula for fusing and aligning cross-modal features and inputting the fused features into an attention layer to further explore the potential connections between modalities is as follows: in, Indicates a cascading operation. Features representing simple text-visual modal connections Simple connection features representing the original sample pairs. This represents the simple connection features of data-augmented sample pairs. This represents the simple connection features of the sample pairs for which hints were added. This represents the fused features obtained through the Transformer. This represents the original text-visual pair fusion features obtained through the Transformer. This represents the text-visual pair fusion features obtained after data augmentation by the Transformer. This represents the text-visual pair fusion features obtained by the Transformer that incorporate the cue information. Indicates by The obtained query vector Indicates by Obtained key vector The transpose of , Indicates by The obtained value vector This represents the final fused features obtained further by the attention layer. This represents the fusion features of the raw data obtained from the attention layer. This represents the fusion features of the data after data augmentation obtained from the attention layer. This represents the fusion features of the data with added cues obtained from the attention layer. This represents the final extracted sequence of original visual features. This represents the final extracted visual feature sequence after data enhancement. This represents the sequence of original text features extracted by BERT. This represents the text feature sequence after data augmentation extracted by BERT. This represents the text feature sequence with added prompts extracted by BERT. This describes the process of cross-modal feature fusion using a Transformer fusion network. This indicates the normalization process.

6. The social media multimodal sentiment classification method based on cue-driven and contrastive learning according to claim 1, characterized in that, In step S7, the specific steps for calculating the loss are as follows: S71, the consistency loss between the original sample pairs and the samples with added prompt text is specifically expressed as: in, This represents the consistency loss between the original sample pairs and the sample pairs with added prompt text. Indicates the first The fusion features extracted from each original sample. Indicates the first The final extracted fusion features are obtained from samples with added prompt information. This indicates the calculation of cosine similarity. Indicates the number of samples in the batch. Indicates the first One sample, This represents the temperature parameter, with a value of 0.

07. This represents the hyperparameter coefficient used in the mean square error calculation process, with a value of 0.

8. The hyperparameter coefficient representing the cosine similarity consistency loss calculation process is 0.

2. S72, calculate the final classification loss, specifically expressed as: in, Represents classification loss. Represents cross-entropy loss, This represents the model's predicted output. Indicates the true label of the sample; S73, calculate the total loss, specifically as follows: in, Indicates the total loss. Represents classification loss. This represents the consistency loss between the original sample pairs and the samples with added prompt text. This represents the contrast loss between the original data sample and the data-augmented sample. This represents the contrast loss between the data samples with added prompts and the data-augmented samples. The hyperparameter coefficient representing the contrast loss between the original data sample and the data-augmented sample, with a value of 0.

4. The hyperparameter coefficient representing the contrast loss between the data sample with added prompts and the data-augmented sample is 0.

3. The hyperparameter coefficient representing the consistency loss between the original sample and the sample with added prompt text is 0.

3.

7. The social media multimodal sentiment classification method based on cue-driven and contrastive learning according to claim 1, characterized in that, In step S8, the sentiment classification result is specifically represented as follows: in, Indicates a fully connected layer. This represents the activation function. This represents the model's predicted output. This represents the fusion features of the raw data obtained from the attention layer.

Images