Image-Text Sentiment Classification Method and System Based on Cross-Modal Generative Adversarial Network

Abstract

The present invention provides a method and system for image-text sentiment classification based on a cross-modal generative adversarial network, which includes data set collection and preprocessing, that is, obtaining user-posted images and texts on a social media platform based on an input topic, and then removing invalid information in the text; data annotation, including pre-labeling, and then recognizing the text in the image and storing it in the data set after verification; setting up the GANSA model, in which image-text modal feature extraction, text feature extraction, and OCR feature extraction are first performed, and then image-text feature generation is carried out; model training and image-text sentiment classification, including training the GANSA model in an end-to-end manner, using the cross-entropy loss function, and then inputting the image-text pair to be analyzed into the trained GANSA model to obtain the sentiment analysis result of the image-text pair. The present invention supports image-text sentiment classification tasks on multiple social media platforms, optimizes feature extraction and feature fusion of image-text modalities, and has more accurate sentiment prediction.

Description

Technical Field

[0001] The present invention belongs to the field of graphic and text sentiment recognition in the cross - field of natural language processing and vision. More specifically, it relates to a graphic and text sentiment classification scheme based on a cross - modal generative adversarial network. Specifically, a generative adversarial network is used to optimize the generation of multi - modal features, and a gated fusion module is used for graphic and text feature fusion, so as to judge the graphic and text sentiment state of the technical solution. Background Technique

[0002] With the continuous innovation of social media and social ways, compared with traditional text - based social expressions, users tend to post their views and comments on events, products, and life in the form of text and images on social media. These messages are rich in emotional color and tendentiousness. For social media, the emotions expressed in a single modality are incomplete and limited, while multi - modality can often convey more accurate and rich emotional information and reveal information that may be hidden in the text. The consistency and complementarity existing between these single - modality data can effectively explain the associated representation within the multi - modality data, and can further enhance the model's expression ability and stability, and improve the performance of emotion task analysis. Compared with text or image emotion analysis, multi - modality emotion analysis often involves multi - modality feature learning, multi - modality alignment, and multi - modality fusion techniques. Feature learning of modalities is generally extracted through pre - trained models or deep neural networks. In terms of alignment, a multi - modality transformer (MuIT) can be introduced to solve the modality misalignment problem in an end - to - end manner. The mainstream fusion methods include feature fusion, decision fusion, and consistency regression fusion.

[0003] In social media, users tend to use emojis rather than real photos to express emotions because emojis can convey more vivid and direct emotions, and they can also supplement or strengthen the emotions expressed by the corresponding text. Due to the lack of context in the blogs posted by users, sentiment analysis in social media is challenging. The multi-modal sentiment analysis task based on emojis is challenging in three aspects: the inherent multi-modality of emojis, the significant differences between emojis, and the complex multi-modal emotion fusion. First, emojis are inherently multi-modal because they embed text, and the same emoji with different emoji texts may have significant emotional differences. Second, emojis are diverse in style, which makes it difficult for the model to learn a robust representation of emojis according to different distributions. In contrast, traditional images are mainly composed of portraits, landscapes, or food, etc., with little style variation. Finally, the emotion fusion of text and emojis is complex. Previous methods considered that text plays a dominant role, ignoring the case where the image itself is multi-modal, or obtained image and text features by applying deep neural networks separately, and used a multi-layer perceptron (MLP) to combine the modal information for inferring the user's potential emotional state. Such processing is too simple and not flexible enough. To optimize the representation of text and image features, it is meaningful for the overall text-image emotion analysis to obtain the generated features of the target modality with the supplement of the auxiliary modality through a generative adversarial network, and fully explore the complementarity between text and image modalities through gated fusion.

[0004] So far, researchers at home and abroad have proposed a series of cross-modal emotion prediction methods, which generally follow the "extract first and then fuse" paradigm. That is, first extract the single-modal features of text and visual content separately, and then use different fusion strategies to aggregate these features for emotion prediction. Through the retrieval of existing patents and related technologies, the existing methods related to cross-modal emotion prediction are as follows:

[0005] [1] The method disclosed in "A Text-Image Cross-Modal Sentiment Classification Method Based on Compact Bilinear Fusion" [P]. CN107066583A by Chen Duansheng, Wu Qiong, Wu Yangyang, Lei Qing, and Zhang Hongbo first extracts the text features and image features of multi-modal information separately, and then uses a bilinear fusion method to fuse the text features and image features to obtain multi-modal features. Finally, the multi-modal features are input into the fully connected layer to complete sentiment classification.

[0006] [2] The method disclosed in "A Social Media Sentiment Analysis Method and System Based on Multi-Modal Feature Fusion" [P]. CN112508077A by Geng Yushui, Zhang Kang, Zhao Jing, Liu Jianxin, and Li Wenxiao extracts text features using a denoising autoencoder, extracts image features using a variational autoencoder, and finally performs cross-modal fusion of the text features and image features for sentiment classification.

[0007] It can be seen that although the existing cross-modal sentiment analysis methods have achieved good results, there are still the following deficiencies: (1) Most methods fail to consider the case where the image itself is multimodal and ignore the sentiment information contained in the text in the image.

[0008] (2) Most methods ignore the correlation between modalities. A single modality may have semantic ambiguity, which will lead to poor performance in multimodal sentiment prediction.

[0009] (3) Most methods only use simple addition and concatenation to fuse different modality features, making it difficult to model the complex semantic interactions between cross-modalities. Summary of the Invention

[0010] The existing image-text sentiment classification methods fail to fully consider the problem that the image itself is multimodal. To solve the above deficiencies in the existing technology, the present invention provides a method capable of optimizing image-text feature representation and image-text modality fusion for sentiment classification.

[0011] To achieve the above object, the present invention proposes an image-text sentiment classification method based on a cross-modal generative adversarial network, including the following steps:

[0012] Step 1, dataset collection and preprocessing, including obtaining the images and texts published by users based on the input topic on the social media platform, and then removing the invalid information in the text;

[0013] Step 2, data annotation, including pre-labeling, and then recognizing the text in the image and storing it in the dataset after verification;

[0014] Step 3, setting up the GANSA model, where the GANSA model is an image-text sentiment classification model.

[0015] In the GANSA model, image-text modality feature extraction is first performed. The image-text modality features include text features, OCR features, and image features.

[0016] Text feature extraction and OCR feature extraction include converting the text and OCR in the preprocessed dataset into word sequences and inputting them into the RoBERTa model to obtain text word encodings; then passing the text word encodings through a bidirectional gated recurrent unit to obtain text features and OCR features with context information; the RoBERTa model is a text pre-training model.

[0017] Image feature extraction includes normalizing the images in the preprocessed dataset and inputting them into the Fast R-CNN model to obtain image features; the Fast R-CNN model is an image pre-training model.

[0018] Then, generate text-image features, including inputting text features and image features into the GAN module in the GANSA model to enhance the feature representation of the text-image modality; then, fuse the text-image features through the GF module in the GANSA model and input them into the fully connected layer and the Softmax function to obtain the final sentiment prediction value; the GAN module is a generative adversarial network module; the GF module is a gated fusion module;

[0019] Step 4, model training and text-image sentiment classification, including training the GANSA model in an end-to-end manner, using the cross-entropy loss function, and then inputting the text-image pair to be analyzed into the trained GANSA model to obtain the sentiment analysis result of the text-image pair.

[0020] Moreover, after concatenating the text published by the user, convert it into a word sequence X = [x1, x2,..., x N , where x i is the sum of the word and position embeddings, N is the maximum length of the sequence, input the word sequence X into the RoBERTa model for encoding, and take the output of the last layer of the encoder as the text word encoding V z ; then input the text word encoding into a bidirectional gated recurrent unit to obtain the text feature V with context information T ; the OCR feature extraction method is the same as that of the text feature extraction, and the OCR feature is denoted as V O .

[0021] Moreover, uniformly adjust the image I to a preset size and normalize it, and then use the pre-trained Fast R-CNN model on the Visual Genoome dataset to extract the visual feature V of each image I .

[0022] Moreover, the realization of text-image feature generation, text-image feature fusion, and sentiment prediction is as follows:

[0023] 1) Construct an auxiliary fusion module to retain information from each auxiliary modality. For the features V1 and V2 of a given auxiliary modality, first concatenate them to obtain the input vector V in , then apply a linear transformation to the input vector to obtain the auxiliary fusion vector V F , and reduce the dimension of the auxiliary fusion vector.

[0024] 2) Obtain the meme auxiliary feature V through the auxiliary fusion module for the image feature and the text feature in the meme IO ; then use the text feature as the input feature of the target modality through the generator, and add a certain amount of noise to obtain the generated feature V of the text modality TG , and the discriminator will identify the source of the input feature; finally, train the network in an adversarial manner, and use the meme auxiliary feature VIO Labeled as real samples, generate feature V TG Labeled as generated samples, and calculate the loss function L T ;

[0025] 3) Similarly to 2), use the generator of the generative adversarial network to obtain the generated feature V of the image modality with the supplement of the auxiliary modality IG ;

[0026] 4) Obtain the gating weight matrix g of the text-image modality through the Sigmoid activation function for the text-generated feature in 2) and the image-generated feature in 3); then, use the gating weight matrix to obtain the fused feature V of the text-image modality M ;

[0027] Then input the fused feature vector of the text and image into the fully connected layer and the Softmax function to obtain the final sentiment prediction value

[0028] Moreover, according to the sentiment prediction value and the true label y, train the GANSA model through the cross-entropy loss function;

[0029] The formula of the cross-entropy loss function is:

[0030]

[0031] where Loss is the value of the loss function

[0032] On the other hand, the present invention provides a text-image sentiment classification system based on a cross-modal generative adversarial network for implementing a text-image sentiment classification method based on a cross-modal generative adversarial network as described above

[0033] Moreover, it includes the following modules

[0034] The first module is used for dataset collection and preprocessing, including obtaining the text-images published by users based on the input topic on the social media platform, and then removing the invalid information in the text

[0035] The second module is used for data annotation, including pre-labeling, and then recognizing the text in the image and storing it in the dataset after verification

[0036] The third module is used for setting the GANSA model. The GANSA model is a text-image sentiment classification model. In the GANSA model, first perform text-image modality feature extraction. The text-image modality features include text features, OCR features, and image features

[0037] Text feature extraction and OCR feature extraction, including converting the text and OCR in the preprocessed dataset into word sequences and inputting them into the RoBERTa model to obtain text character encodings; then passing the text character encodings through a bidirectional gated recurrent unit to obtain text features and OCR features with context information; the RoBERTa model is a text pre-trained model;

[0038] Image feature extraction, including normalizing the images in the preprocessed dataset and inputting them into the Fast R-CNN model to obtain image features; the Fast R-CNN model is an image pre-trained model;

[0039] Then, cross-modal feature generation is performed, including inputting the text features and image features into the GAN module in the GANSA model to enhance the feature representation of the cross-modal; then, through the GF module in the GANSA model, the cross-modal features are fused and input into a fully connected layer and a Softmax function to obtain the final sentiment prediction value; the GAN module is a generative adversarial network module; the GF module is a gated fusion module;

[0040] The fourth module is used for model training and cross-modal sentiment classification, including training the GANSA model in an end-to-end manner, using the cross-entropy loss function, and then inputting the to-be-analyzed cross-modal pair into the trained GANSA model to obtain the sentiment analysis result of the cross-modal pair.

[0041] Or, it includes a processor and a memory. The memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a cross-modal sentiment classification method based on a generative adversarial network as described in any one of the above.

[0042] Or, it includes a readable storage medium. A computer program is stored on the readable storage medium, and when the computer program is executed, a cross-modal sentiment classification method based on a generative adversarial network as described in any one of the above is implemented.

[0043] The present invention is applicable to sentiment prediction of cross-modal published by users on social media. By introducing a generative adversarial network, the feature representation of the cross-modal is enhanced with the supplement of the auxiliary modality, and gated fusion is used to further explore the complementarity between the cross-modal, making the cross-modal sentiment classification more accurate.

[0044] Compared with the prior art, the present patent mainly optimizes two points:

[0045] The first is the inherent multi-modality of the image itself. When the image is an emoji, the image features and text features in the image are fully extracted

[0046] The second is to reduce the noise caused by semantic ambiguity. An auxiliary fusion module is used to help the generative adversarial network generate text features and image features.

[0047] Finally, a gating fusion mechanism is introduced to measure the complementary relationship between information in different modalities, and the feature information of the text and image modalities is effectively fused.

[0048] The solution of the present invention is simple and convenient to implement, and has strong practicability. It solves the problems of low practicability and inconvenience in actual application existing in the related technologies, can improve the user experience, and has important market value. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flowchart of an embodiment of the present invention;

[0050] Figure 2 is a framework diagram of the AF module in the GANSA model of an embodiment of the present invention;

[0051] Figure 3 is a framework diagram of the GAN module in the GANSA model of an embodiment of the present invention.

[0052] Figure 4 is a specific application example diagram of an embodiment of the present invention.

[0053] Figure 5 is a sample diagram of the dataset and annotation tool of an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0054] The technical solution of the present invention will be specifically described below in conjunction with the drawings and embodiments.

[0055] The present invention discloses a text-image sentiment classification based on a cross-modal generative adversarial network, which solves the problem that mainstream models cannot directly process or have poor effects when the image itself is multi-modal; collects a social media text-image sentiment dataset; builds an annotation tool to construct a text-image sentiment dataset; performs model training, including training the GANSA model using the dataset; when extracting features, uses a bidirectional gated recurrent unit to obtain text features and OCR features with context, and uses a region detection network to obtain image visual features; uses the GAN module to optimize the feature representation of the target modality with the supplement of the auxiliary modality; when performing feature fusion, uses a gating fusion module to fully explore the complementarity between the text and image modalities and learn potential sentiment representations for overall text-image sentiment prediction. The present invention supports text-image sentiment classification tasks on multiple social media platforms, optimizes the feature extraction and feature fusion of text and image modalities in the case of common text-image sentiment classification dealing with multi-modal images, and the sentiment prediction is more accurate.

[0056] First, the technical terms of the present invention are explained and described:

[0057] AF: AF is the abbreviation of Auxiliary Fusion, which means auxiliary fusion;

[0058] GAN: GAN is the abbreviation of Generative Adversarial Network, which means Generative Adversarial Network;

[0059] GF: GF is the abbreviation of Gated Fusion, which means Gated Fusion

[0060] RoBERTa: The full name of RoBERTa is A Robustly Optimized BERT Pretraining Approach, which is a pre-trained language representation model. Based on the language masking strategy of BERT, the key hyperparameters in BERT are modified, and the masked language model (MLM) that dynamically changes the training data is adopted, so as to generate deep bidirectional language representations.

[0061] Fast R-CNN: The full name of Fast R-CNN is Fast Region-based Convolutional Network, which is a visual representation model of a pre-trained model. As an object detector, by extracting multiple detection targets with the highest probability in each image, they are used as visual features.

[0062] Softmax: The Softmax function is one of the most widely used activation functions, which transforms the prediction results of the model onto the exponential function and then normalizes the transformed results.

[0063] See Figure 1 , the embodiment of the present invention provides a text-image sentiment classification method based on dual attention, including the following steps:

[0064] Step 1. First, collect and preprocess the dataset, and construct a text-image sentiment dataset. The implementation method preferably adopted in the embodiment specifically includes the following sub-steps:

[0065] Step 1.1: Obtain the text and images published by users on the social media platform with the theme of "worker", and then remove invalid information such as illegal characters, web links, and advertisements in the text;

[0066] In the embodiment, web crawler technology is used to collect the text and image data published by users on Weibo with keywords, and the HTML data is structured to extract the text and images.

[0067] Step 1.2: Perform data preprocessing to remove invalid characters such as illegal characters and web links in the text; <>

[0068] Step 2. Then, build a labeling platform and label the sorted data.

[0069] SeeFigure 5 , the preferred implementation method in the embodiment is as follows:

[0070] Step 2.1: It is preferably recommended to use the VUE front-end framework to build a visual annotation platform. Specifically, texts and images can be uploaded locally in advance.

[0071] Step 2.2: Data annotation. According to the pre-established annotation specifications, whether the text and image are valid, the overall sentiment label of the text and image, the emoticon text converted from the emoticon, and the clue text with a specific sentiment tendency are marked in sequence as a complete text and image sentiment pair. Specifically, the standard can be carried out in advance, and the prepared text and image sentiment pairs are input.

[0072] Step 2.3: Image text recognition. Use open-source OCR to recognize the text in the image and store it in the data set after verification.

[0073] Specifically, the overall sentiment of the text and image is annotated from three aspects. First, if the sentiment tendency of the text is consistent with that of the image, the overall sentiment is the same as the sentiment tendency of the text. Second, if the sentiment tendency of the text is inconsistent with that of the image, the overall sentiment tendency is considered according to the actual situation. Finally, if there is a sentiment turn in the text, the overall sentiment is judged according to the image sentiment. Specifically, reference can be made to Figure 5 the data set and annotation tool examples shown.

[0074] Step 3. Set the GANSA model, and the GANSA model is a text and image sentiment classification model.

[0075] 1) First, text and image modality feature extraction is performed in the GANSA model. The text and image modality features include text features, OCR features, and image features.

[0076] When training the model, the organized data set can be used for text and image modality feature extraction. The preferred recommended solution is:

[0077] Text feature extraction and OCR feature extraction, including converting the text and OCR in the pre-processed data set into word sequences and inputting them into the RoBERTa model to obtain text character encodings; then passing the text character encodings through a bidirectional gated recurrent unit to obtain text features and OCR features with context information; the RoBERTa model is a text pre-training model;

[0078] Image feature extraction, including normalizing the images in the pre-processed data set and inputting them into the Fast R-CNN model to obtain image features; the Fast R-CNN model is an image pre-training model;

[0079] In this embodiment, the text and image sentiment data set uses a pre-trained model for feature extraction, including the following steps:

[0080] Step 3.1: Text feature extraction. Convert the text into a word sequence and input it into the RoBERTa model to obtain text word encodings. Then, pass the text word encodings through a bidirectional gated recurrent unit to obtain text features with context information.

[0081] Step 3.2: OCR feature extraction. Convert the text in the image into a word sequence and input it into the RoBERTa model to obtain word encodings. Then, pass the word encodings through a bidirectional gated recurrent unit to obtain OCR features with context information.

[0082] Step 3.3: Image feature extraction. Normalize the image and input it into the Fast R-CNN model to obtain image features.

[0083] The specific feature extraction methods of the preferred suggestions are as follows:

[0084] Concatenate the text published by the user and convert it into a word sequence X = [x1, x2,..., x N , where x i is the sum of the word and position embeddings, i is the sequence label, N is the maximum length of the sequence. Input the word sequence X into the RoBERTa model for encoding, and take the output of the last layer of the encoder as the text word encoding V z ; Then, input the obtained text word encoding into a bidirectional gated recurrent unit to obtain text features V T with context information; The OCR feature extraction method is the same as that of the text feature extraction, and the obtained OCR feature is denoted as V O . The RoBERTa model is a prior art and will not be elaborated in this invention.

[0085] The extraction formula for the text feature V T is:

[0086] V Z = RoBERTa(X) = [v1, v2,..., v n (1)

[0087]

[0088]

[0089]

[0090] V T = BiGRU(V Z ) = [h1, h2,..., h n (5)

[0091] where RoBERTa() represents the RoBERTa pre-trained model, [v1, v2,..., vn represents text auto - encoding, v i represents the character encoding of the i - th word, represents the forward GRU, represents the backward GRU, represents the output of the hidden layer in the forward GRU, represents the output of the hidden layer in the backward GRU, h i represents the output of the hidden layer of the bidirectional GRU, BiGRU() represents the bidirectional GRU.

[0092] Uniformly adjust the image I to a size of 224 * 224 and normalize it, then use the pre - trained Fast R - CNN model on the Visual Genoome dataset to extract the visual features V of each image I ; The Fast R - CNN is a prior art and will not be elaborated in the present invention;

[0093] The extraction formula of the described image features is:

[0094] V I = Fast R - CNN(I) (6)

[0095] Wherein, Fast R - CNN() represents the Fast R - CNN pre - trained model.

[0096] 2) Then perform text - image feature generation, including inputting the text features and image features into the GAN module in the GANSA model to enhance the feature representation of the text - image modality; then fuse the text - image features through the GF module in the GANSA model and input them into the fully - connected layer and the Softmax function to obtain the final sentiment prediction value; the GAN module is a generative adversarial network module; the GF module is a gated fusion module;

[0097] The embodiment uses the GAN module for text - image feature generation, and the steps are as follows:

[0098] 1) Obtain the image auxiliary feature and the text auxiliary feature respectively through the auxiliary fusion module for the two auxiliary modalities.

[0099] 2) Input the image auxiliary feature and the text features into the generative adversarial network; train the text feature generator and the discriminator to obtain the text - generated features with the interaction of the image auxiliary features;

[0100] 3) Input the text auxiliary feature and the image features into the generative adversarial network; train the image feature generator and the discriminator to obtain the image - generated features with the interaction of the text auxiliary features.

[0101] See Figure 2 and Figure 3, the specific way to generate the recommended graphic and text features is as follows:

[0102] Construct an auxiliary fusion module to retain information from each auxiliary modality. For the features V1 and V2 of a given auxiliary modality, first concatenate them to obtain the input vector V in , and then apply a linear transformation to the input vector to obtain the auxiliary fusion vector V F , and reduce the dimension of the auxiliary fusion vector.

[0103]

[0104] V F = Linear(V in ) (8) where, represents feature concatenation, and Linear() represents a linear layer.

[0105] Obtain the emoji auxiliary feature V through the auxiliary fusion module for the image feature and the text feature in the emoji IO ; then use the text feature as the input feature of the target modality through the generator, and add a certain amount of noise to obtain the generated feature V of the text modality TG , and the discriminator will identify the source of the input feature. Finally, train the network in an adversarial manner, mark the emoji auxiliary feature V IO as a real sample, mark the generated feature V TG as a generated sample, and calculate the loss function L T

[0106] V IO = AF(V I , V O ) (9)

[0107] V TG = G(V T ) (10)

[0108]

[0109] where AF() is the auxiliary fusion module, G() is the generator in the generative adversarial network, D() is the discriminator in the generative adversarial network, and respectively represent the probabilities of the auxiliary modality and the generated modality.

[0110] Similarly, use the generator of the generative adversarial network to obtain the generated feature V of the image modality with the supplement of the auxiliary modality IG

[0111] V TO = AF(V T , V O ) (12)

[0112] V IG = G(V I ) (13)

[0113] where AF() is the auxiliary fusion module, and the text auxiliary feature V TO is marked as the real sample, and the generated feature V IG is marked as the generated sample.

[0114] The embodiment uses the GF module for graphic and text feature fusion and sentiment prediction, and the steps are as follows:

[0115] 1) Obtain the gating weight matrix of the graphic and text modalities by passing the generated text features and image features through the Sigmoid activation function.

[0116] 2) Use the gating weight matrix to obtain the fusion features of the graphic and text modalities.

[0117] 3) Input the fusion feature vector of the graphic and text into the fully connected layer and the Softmax function for graphic and text sentiment prediction.

[0118] The processing process of the preferred GF module is as follows:

[0119] Obtain the gating weight matrix g of the graphic and text modalities by passing the text generation feature in 2) and the image generation feature in 3) through the Sigmoid activation function; then, use the gating weight matrix to obtain the fusion feature V M

[0120] g = Sigmoid(W I V IG + W T V TG ) (14)

[0121] V M = g ⊙ V TG + (1 - g) ⊙ V IG (15)

[0122] where W I , W T are both trainable weight matrices, Sigmoid() is the activation function, and ⊙ represents matrix multiplication.

[0123] Then input the fusion feature vector of the graphic and text into the fully connected layer and the Softmax function to obtain the final sentiment prediction value;

[0124] F = Linear(W f V M ) (16)

[0125] p = softmax(Wp F) (17)

[0126] Among them, W f and W p are both trainable weight matrices.

[0127] Step 4. Model training and image-text sentiment classification, including training the GANSA model in an end-to-end manner, using the cross-entropy loss function, and then inputting the image-text pair to be analyzed into the trained GANSA model to obtain the sentiment analysis result of the image-text pair.

[0128] The preferred cross-entropy loss function is as follows:

[0129] According to the sentiment prediction value and the true label y, the GANSA model is trained through the cross-entropy loss function;

[0130] The formula of the cross-entropy loss function is:

[0131]

[0132] Among them, Loss is the value of the loss function.

[0133] In specific implementation, the method proposed by the technical solution of the present invention can be automatically run by those skilled in the art using computer software technology. The system device for implementing the method, such as a computer-readable storage medium storing the corresponding computer program of the technical solution of the present invention and a computer device including running the corresponding computer program, should also be within the protection scope of the present invention. For example, a corresponding system can be provided to implement image-text sentiment data collection and screening, and model training in separate modules.

[0134] Image-text sentiment data collection and screening adopt a semi-artificial annotation method. First, the machine is used to collect, structure, and screen the image-text data. Then, the artificial annotation method is used to supplement information for the data, and the image-text sentiment dataset is obtained after cleaning the data.

[0135] The model is trained using the image-text sentiment dataset. For the application example of this embodiment, refer to Appendix Figure 4 , first, the text in the image is recognized as "It's just sand in my eyes" through OCR, and then the text "[Dizzy] My sleep is really tailor-made for office workers. I only slept for more than 4 hours. No matter how late I go to bed, I can wake up at 7 o'clock. I really can't fall asleep when I think about something at night" respectively pass through the RoBERTa pre-trained model to obtain word encodings, and then the word encoding representations are input into a bidirectional gated recurrent unit to obtain text feature representations and OCR feature representations with context semantic information; for the preprocessed image, it is input into the Fast R-CNN pre-trained model to obtain the image representation;

[0136] Secondly, for the feature optimization module, enhanced feature representations are learned from two perspectives. The first is from the text. Through a generative adversarial network, text-generated features are obtained with the supplement of image auxiliary features. The second is from the vision. Through a generative adversarial network, image-generated features are obtained with the supplement of text auxiliary features. In the feature fusion module, the complementary relationship between text-generated features and image-generated features is learned through gated fusion to obtain text-image fusion features. Finally, the fused features pass through a fully connected layer and a Softmax activation function to predict the sentiment result.

[0137] In some possible embodiments, a text-image sentiment classification system based on a cross-modal generative adversarial network is provided, including the following modules.

[0138] The first module is used for dataset collection and preprocessing, including obtaining text-images published by users based on the input topic on a social media platform, and then removing invalid information in the text.

[0139] The second module is used for data annotation, including pre-labeling, and then recognizing the text in the image and storing it in the dataset after verification.

[0140] The third module is used to set up the GANSA model. The GANSA model is a text-image sentiment classification model. In the GANSA model, text-image modal feature extraction is first performed. The text-image modal features include text features, OCR features, and image features.

[0141] Text feature extraction and OCR feature extraction include converting the text and OCR in the preprocessed dataset into word sequences and inputting them into the RoBERTa model to obtain text word encodings. Then, the text word encodings pass through a bidirectional gated recurrent unit to obtain text features and OCR features with context information. The RoBERTa model is a text pre-training model.

[0142] Image feature extraction includes normalizing the images in the preprocessed dataset and inputting them into the Fast R-CNN model to obtain image features. The Fast R-CNN model is an image pre-training model.

[0143] Then, text-image feature generation is performed, including inputting the text features and image features into the GAN module in the GANSA model to enhance the feature representation of the text-image modality. Then, the text-image features are fused through the GF module in the GANSA model and input into a fully connected layer and a Softmax function to obtain the final sentiment prediction value. The GAN module is a generative adversarial network module. The GF module is a gated fusion module.

[0144] The fourth module is used for model training and image-text sentiment classification, including training the GANSA model in an end-to-end manner, adopting a cross-entropy loss function, and then inputting the image-text pair to be analyzed into the trained GANSA model to obtain the sentiment analysis result of the image-text pair.

[0145] In some possible embodiments, a system for image-text sentiment classification based on a cross-modal generative adversarial network is provided, including a processor and a memory. The memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the method for image-text sentiment classification based on a cross-modal generative adversarial network as described above.

[0146] In some possible embodiments, a readable storage medium is provided, on which a computer program is stored.

[0147] The specific embodiments described herein are merely illustrative of the spirit of the present invention. Those skilled in the art to which the present invention pertains can make various modifications or supplements to the described specific embodiments or use similar methods for substitution, but will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A text-image sentiment classification method based on a cross-modal generative adversarial network, characterized in that, It includes the following steps: Step 1, dataset collection and preprocessing, including obtaining the text and images published by users based on the input topic on the social media platform, and then removing the invalid information in the text; Step 2, data annotation, including pre-labeling, and then recognizing the text in the image and storing it in the dataset after verification; Step 3, setting up the GANSA model, where the GANSA model is a text-image sentiment classification model, In the GANSA model, text-image modality feature extraction is first performed, and the text-image modality features include text features, OCR features, and image features, Text feature extraction and OCR feature extraction, including converting the text and OCR in the preprocessed dataset into word sequences and inputting them into the RoBERTa model to obtain text character encodings; then passing the text character encodings through a bidirectional gated recurrent unit to obtain text features and OCR features with context information; The RoBERTa model is a text pre-training model; Image feature extraction, including normalizing the images in the preprocessed dataset and inputting them into the Fast R-CNN model to obtain image features; the Fast R-CNN model is an image pre-training model; Then text-image feature generation is performed, including inputting the text features and image features into the GAN module in the GANSA model to enhance the feature representation of the text-image modality; then fusing the text-image features through the GF module in the GANSA model and inputting them into the fully connected layer and the Softmax function to obtain the final sentiment prediction value; the GAN module is a generative adversarial network module; the GF module is a gated fusion module; Step 4, model training and text-image sentiment classification, including training the GANSA model in an end-to-end manner, using the cross-entropy loss function, and then inputting the text-image pair to be analyzed into the trained GANSA model to obtain the sentiment analysis result of the text-image pair; The implementation of text-image feature generation and text-image feature fusion and sentiment prediction is as follows: 1) Construct an auxiliary fusion module to retain information from each auxiliary modality. For the features V1 and V2 of a given auxiliary modality, first concatenate them to obtain the input vector V in , and then apply a linear transformation to the input vector to obtain the auxiliary fusion vector V F , and reduce the dimension of the auxiliary fusion vector; 2) The image features and the text features in the emoji are used to obtain the emoji auxiliary feature V through the auxiliary fusion module IO ; Then, the text features are used as the input features of the target modality through the generator, and noise is added to obtain the generated feature V of the text modality TG , and the discriminator will identify the source of the input features; Finally, the network is trained in an adversarial manner, and the emoji auxiliary feature V IO is marked as the real sample, and the generated feature V TG is marked as the generated sample, and the loss function L T is calculated; 3) Similarly to 2), the generator of the generative adversarial network obtains the generated feature V of the image modality with the supplement of the auxiliary modality. IG ; 4) Obtain the gated weight matrix g of the text and image modalities by passing the text generation features in 2) and the image generation features in 3) through the Sigmoid activation function; then, use the gated weight matrix to obtain the fused feature V of the text and image modalities M ; Then, the fused feature vector of the text and image is input into the fully connected layer and the Softmax function to obtain the final sentiment prediction value; according to the sentiment prediction value and the true label y, the GANSA model is trained through the cross-entropy loss function; The formula of the cross-entropy loss function is: where Loss is the value of the loss function 2. The method for image-text sentiment classification based on a cross-modal generative adversarial network according to claim 1, characterized in that: Concatenate the text published by the user and convert it into a word sequence X = [x1, x2, …, x N , where x i is the sum of the word and position embeddings, N is the maximum length of the sequence. Input the word sequence X into the RoBERTa model for encoding, and take the output of the last layer of the encoder as the text word encoding V z ; then input the text word encoding into a bidirectional gated recurrent unit to obtain the text feature V T with context information; the OCR feature extraction is the same as the text feature extraction method, and the OCR feature is denoted as V O .

3. The method for classifying image-text sentiment based on a cross-modal generative adversarial network according to claim 1, wherein: The image I is uniformly adjusted to a preset size and normalized, and then the visual features V of each image are extracted using the Fast R-CNN model pre-trained on the Visual Genoome dataset I .

4. A text-image sentiment classification system based on a cross-modal generative adversarial network, characterized in that: It is used to implement a text-image sentiment classification method based on a cross-modal generative adversarial network as described in any one of claims 1-3.

5. The cross-modal generative adversarial network-based image-text sentiment classification system according to claim 4, characterized in that: It includes the following modules, The first module is used for dataset collection and preprocessing, including obtaining the text and images published by users based on the input topic on the social media platform, and then removing the invalid information in the text; The second module is used for data annotation, including pre-labeling, and then recognizing the text in the image and storing it in the dataset after verification; The third module is used for setting up the GANSA model, where the GANSA model is a text-image sentiment classification model, In the GANSA model, text-image modality feature extraction is first performed, and the text-image modality features include text features, OCR features, and image features, Text feature extraction and OCR feature extraction, including converting the text and OCR in the preprocessed dataset into word sequences and inputting them into the RoBERTa model to obtain text character encodings; then passing the text character encodings through a bidirectional gated recurrent unit to obtain text features and OCR features with context information; The RoBERTa model is a text pre-training model; Image feature extraction, including normalizing the images in the preprocessed dataset and inputting them into the Fast R-CNN model to obtain image features; the Fast R-CNN model is an image pre-training model; Then, cross-modal feature generation is performed, including inputting text features and image features into the GAN module in the GANSA model to enhance the feature representation of the cross-modal; then, the text and image features are fused through the GF module in the GANSA model and input into the fully connected layer and the Softmax function to obtain the final sentiment prediction value; the GAN module is a generative adversarial network module; the GF module is a gated fusion module; The fourth module is used for model training and cross-modal sentiment classification, including training the GANSA model in an end-to-end manner, using the cross-entropy loss function, and then inputting the text-image pair to be analyzed into the trained GANSA model to obtain the sentiment analysis result of the text-image pair.

6. The cross-modal generative adversarial network-based image-text sentiment classification system according to claim 4, characterized in that: It includes a processor and a memory. The memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a cross-modal sentiment classification method based on a generative adversarial network as described in any one of claims 1-3.

7. The image-text sentiment classification system based on a cross-modal generative adversarial network according to claim 4, characterized in that: It includes a readable storage medium, on which a computer program is stored. When the computer program is executed, it implements a cross-modal sentiment classification method based on a generative adversarial network as described in any one of claims 1-3.

Images