A cross-modal positive and negative semantic classification method based on text sentiment and image content perception

Abstract

The present invention discloses a cross-modal positive and negative semantic classification method based on text emotion and image content perception, which relates to the technical field of cross-modal information fusion analysis, and is characterized by comprising the following steps: S1: mining and analyzing social media information; S2: constructing a multi-modal positive and negative semantic classification architecture, including text and visual feature extractors, comment feature extractors and graph feature extractors; S3: multi-modal feature fusion; using a common attention mechanism to capture mutual information between different modalities, and enhancing cross-modal features by learning attention weights between different modal features; S4: modality alignment. The present invention effectively combines text, vision, comment and social graph features to improve positive and negative semantic classification.

Description

Technical Field

[0001] The present invention relates to the technical field of cross-modal information fusion analysis, and in particular to a cross-modal positive and negative semantic classification method based on text sentiment and image content perception. Background Art

[0002] With the rapid development of social media, a large amount of negative or unverified information has emerged and spread, affecting online social network users and causing a huge impact on offline society. The creators of negative information often spread information for various purposes, which not only affects people's normal lives, but also has a significant negative impact on society. Since a large number of users post social media information every day, automatic identification of negative information is conducive to providing early preventive measures to minimize the negative impact of negative information.

[0003] Deep learning plays an important role in positive and negative semantic classification. It can automatically and effectively learn feature vectors containing deep semantic information from the text and pictures of social media information. Recurrent neural networks (RNNs) represented by long short-term memory (LSTM) and gated recurrent units (GRU) and their various variants can effectively capture the temporal relationship between the text of each social media information in the negative information propagation chain and learn hidden features from social media information. Through structures such as convolutional layers, pooling layers, and fully connected layers, convolutional neural networks that perform convolution and pooling operations on input data, extract features, and perform learning and classification can obtain key features and their high-level interactions from negative information. For example, existing technologies introduce recurrent neural networks (RNNs) to capture potential temporal semantic information, learn potential features from related texts, and successfully defeat positive and negative semantic classification models that use manual features; high-level interactions and key features of social media information are obtained using convolutional neural networks, and good performance is shown in the process of positive and negative semantic classification. Recently, many researchers believe that the positive and negative semantic classification tasks involve very complex situations. Therefore, we cannot only consider the text and visual information contained in social media information, but also need to consider the complex social background information and integrate the social background information into the positive and negative semantic classification model to assist detection. For example, graphical models such as graph attention networks (GAT) and graph convolutional networks (GCN) are used to aggregate adjacent node information to obtain better positive and negative semantic classification node representations. In addition, considering privacy issues or data crawling limitations, the available social graph data lacks reliable connections between entities. Therefore, it is very necessary to supplement the potential links on the social graph to achieve more accurate detection. With the help of graphical models, connected instances can exchange information to facilitate learning from each other.

[0004] However, these studies have a common limitation, that is, they only consider the text and visual information and social background information of social media, but do not consider the comment information of social media. The page structure of social media information not only has text information, but also includes likes, reposts and a large amount of comment text information. Comments contain attitudes towards social media information, and this kind of deep feature neural network is often difficult to capture. Therefore, the comment sentiment polarity can be regarded as an extremely important feature and added to the positive and negative semantic classification model.

[0005] Negative information usually creates more sensational plots, thereby achieving rapid spread. The comments on social media information contain a large amount of questioning and refuting text information, which will greatly increase the proportion of negative sentiment words in the comments. Sentiment analysis is usually regarded as a text classification problem. Generally speaking, there are more positive sentiment words in the comments on positive information, that is, there are more positive comments; there are more negative sentiment words in the comments on negative information, that is, there are more negative comments. Therefore, it is very necessary to add the sentiment polarity of the comments to the positive and negative semantic classification model.

[0006] The traditional positive and negative semantic classification model has the following shortcomings:

[0007] Traditional positive and negative semantic classification models mostly focus only on text, images and other content, ignoring social graph features and a large amount of comment information in social media information. The previous positive and negative semantic classification methods did not fully integrate modalities, resulting in insufficient learning between features. Since the directly collected comment text contains a lot of spam, if it is not cleaned up, it will seriously interfere with the experimental results. Summary of the invention

[0008] The technical problem to be solved by the present invention is to provide a cross-modal positive and negative semantic classification method based on text sentiment and image content perception, effectively combining text, vision, comments and social graph features to improve the model and extract four types of features. In order to produce better social graph features, the graph topology and aggregation process are improved on the basis of GAT. Then, modal interaction and alignment are performed to achieve better multimodal fusion. Finally, the enhanced multimodal features are connected for classification.

[0009] The present invention adopts the following technical solutions to achieve the invention objectives:

[0010] A cross-modal positive and negative semantic classification method based on text sentiment and image content perception, characterized by comprising the following steps:

[0011] S1: Mining and analyzing social media information;

[0012] S2: Construct a multimodal positive and negative semantic classification detection architecture, including text and visual feature extractors, comment feature extractors, and graph feature extractors;

[0013] S3: Multimodal feature fusion; use the common attention mechanism to capture the mutual information between different modalities and enhance cross-modal features by learning the attention weights between different modal features;

[0014] S4: Modality Alignment: We introduce modality alignment to improve the representations learned in each modality by enhancing text features to make them close to enhanced graphic features.

[0015] S5: Fusion of multimodal features;

[0016] S6: Adversarial training classification.

[0017] As a further limitation of the technical solution, the specific steps of S1 are:

[0018] S11: Set is a set of social media information with text and pictures. For each set of social media information , and ,in: , and Respectively indicate the release of the social media information Text, images and users, express Comments;

[0019] Each comment All by the corresponding users published;

[0020] S12: To represent the user’s behavior on social media, a graph is constructed. G :

[0021] ;

[0022] in: It is a collection of nodes, including user nodes, comment nodes, and content nodes of social media information;

[0023] It is the adjacency matrix between nodes, which is used to describe the relationship between nodes, including the release, comment and forwarding of social media information;

[0024] is the set of edges;

[0025] S13: Define positive and negative semantic classification as a binary classification task. represents the category label, where Indicates negative information, otherwise , learning function To predict social media information .

[0026] As a further limitation of the present technical solution, the workflow of the text and visual feature extractor is to extract the semantic features of a sentence using a convolutional neural network and a pooling layer:

[0027] S211: For each set of social media information , whose text are padded or truncated to have the same number of tokens, i.e. , which is expressed as ,in: , is the word embedding dimension, for No. word embeddings of words;

[0028] S212: In the word embedding matrix Apply convolutional layers to get feature maps ,in is the size of the convolution kernel, let ;

[0029] S213: Using the maximum pooling, we get , using different receptive fields of Filters are used to obtain semantic features of different granularities;

[0030] S214: Connect the outputs of all filters to form Overall text features:

[0031] (1).

[0032] As a further limitation of the present technical solution, the workflow of the comment feature extractor is to use the pre-trained model ResNet50 trained on the ImageNet database to extract image Features:

[0033] S221: Extract the output of the penultimate layer of ResNet50 and represent it as ;

[0034] S222: It obtains the final visual features with the same dimension as the text features through a fully connected layer, namely:

[0035] (2)

[0036] in: is the weight matrix of the fully connected layer;

[0037] is the sigmoid activation function.

[0038] As a further limitation of this technical solution, the workflow of the comment feature extractor is as follows:

[0039] S231: The comment sentiment value is divided into 5 star ratings, namely 1, 2, 3, 4, and 5. When the star rating is 1, 2, and 3, the comment is judged as a negative comment; when the star rating is 4 and 5, the comment is judged as a positive comment, that is:

[0040] (3)

[0041] in: For roberta-base-finetuned-chinanews-chinese model;

[0042] For comments Emotional star rating;

[0043] S232: The label of the negative comment is set to 0, and the label of the positive comment is set to 1;

[0044] (4)

[0045] In this way, each comment gets a corresponding label according to the corresponding sentiment score;

[0046] S233: Convert these tags into vectors to obtain the final comment features of social media information, namely:

[0047] (5)

[0048] in: is the number of comments corresponding to each social media message.

[0049] As a further limitation of the present technical solution, the workflow of the graph feature extractor is as follows:

[0050] Social graph data lacks reliable connections between entities due to privacy issues or data crawling limitations. Therefore, it is necessary to infer hidden connections between nodes. According to network homogeneity, similar nodes are more likely to be connected to each other than dissimilar nodes. Therefore, the feature similarity between different nodes is calculated, and the connection between nodes with high similarity is inferred. Specifically:

[0051] S241: Define the node embedding matrix as , There are three types of nodes in the paper. The sentence vectors are used as the initial embeddings of the text and comment nodes of the social media information, and the average of the text node embeddings of the social media information posted by the user is used as the initial user embedding. and calculate and The correlation coefficient between ,Right now:

[0052] (6)

[0053] in: and for and Node embedding of

[0054] If the similarity is greater than 0.5, there is a potential edge between them, that is:

[0055] (7)

[0056] S242: Infer the original adjacency matrix of potential edge pairs Enhancement, Expressed as Elements of express and There is an edge between them, otherwise , then the enhanced adjacency matrix Elements in is defined as:

[0057] (8);

[0058] S243: Use graph attention network to capture social graph structure information, use signature attention-based graph attention network to capture multi-faceted correlations, for nodes and its neighbor nodes ;

[0059] First, compute the nodes and The attention weight set of each node in ,Right now:

[0060] (9)

[0061] Among them: || represents the serial operation;

[0062] and represents a learnable parameter;

[0063] and Representation Node and , and ;

[0064] is the activation function;

[0065] Then, the attention weights are normalized using the softmax function and the softmax function is used to calculate and The normalized weight of is:

[0066] (10);

[0067] S244: In order to capture the positive and negative relationships between nodes, we use and To obtain the weighted sum of adjacent node features, the two vectors are then concatenated together and passed through a fully connected layer to obtain the final node feature. The node features of can be obtained, namely

[0068] (11);

[0069] in: is the weight matrix of the fully connected layer;

[0070] is the activation function;

[0071] for The feature matrix of

[0072] S245: For each node, update its embedding according to equation (8) to obtain the updated node embedding matrix ,in: is the number of nodes, and then a multi-head attention mechanism is used to capture features from different angles. Finally, the updated node embeddings of each head are connected together as the overall graph feature:

[0073] (12)

[0074] in: represents the number of attention points, then social media information Graph features of locations Corresponds to No. List.

[0075] As a further limitation of the technical solution, the specific steps of S3 are:

[0076] S31: Use multi-head self-attention mechanism to enhance intra-modal feature representation, for text features , respectively use , and To calculate its query matrix, key matrix and value matrix, where: is a linear transformation, and then, a multi-head self-attention feature of the text modality is generated:

[0077] (13)

[0078] in: Expressed as an attention head;

[0079] is the linear transformation of the output;

[0080] right , and Perform the same operation to obtain the corresponding features , and ;

[0081] S32: Generate enhanced multimodal features using cross-modal attention mechanism. Specifically, To perform text-visual cross-modal attention, we first perform the same operation as the multi-head self-attention mechanism above, but replace Replace with , and get the query matrix ,Will Replace with , get the key matrix Sum Matrix , and then the cross-modal enhancement features are obtained :

[0082] (14)

[0083] in: is the linear transformation of the output;

[0084] represents text features enhanced by visual features based on text visual relevance;

[0085] Based on the cross-modal joint attention process, by exchanging the roles of text features and visual features, the enhanced visual features can be obtained through formula (11): .

[0086] As a further limitation of the technical solution, the specific steps of S4 are:

[0087] S41: For social media information , and enhance its graph features and enhanced text features Transform to the same modal feature space, that is:

[0088] (15)

[0089] in: and is a learnable parameter;

[0090] S42: Use MSE loss for modality alignment to reduce and Distance between:

[0091] (16)

[0092] in: is the total number of social media messages;

[0093] Get finely aligned text features and graphic features , used for multimodal fusion.

[0094] As a further limitation of the present technical solution, the specific steps of S5 are:

[0095] S51: Again and These four modal features perform the above cross-modal attention mechanism, and finally obtain 12 cross-modal enhanced features: ;

[0096] S52: Then concatenate them as the final multimodal features:

[0097] (17).

[0098] As a further limitation of the present technical solution, the specific steps of S6 are:

[0099] S61: The multimodal features finally obtained from social media information are input into the fully connected layer to predict whether it is negative information:

[0100] (18)

[0101] in: Indicates prediction is the probability of negative information;

[0102] S62: Then use the cross entropy loss function:

[0103] (19)

[0104] Represents prediction is the probability of negative information;

[0105] The final losses are as follows:

[0106] (20)

[0107] in: and A factor used to balance the two losses.

[0108] Compared with the prior art, the advantages and positive effects of the present invention are: the present invention jointly models the text information, visual information, social background information and comment sentiment information of social media information for the first time. In order to effectively combine the features of these four modalities, a cross-modal collaborative attention mechanism between two sets of features is used to capture the mutual information between different modalities. A self-supervised loss is introduced to align the source-post representations learned from two different views (i.e., text visual view and social graph view), aiming to improve the representation learning under each view. Through the above enhanced cross-modal fusion, the performance of multimodal positive and negative semantic classification detection can be improved. The present invention proposes a multimodal positive and negative semantic classification scheme based on sentiment analysis of social media comments. First, a strategy for sentiment analysis of social media comments is proposed, that is, to mine the sentiment words contained in the comment text and extract features to discriminate the sentiment tendency of the comment text. The sentiment analysis of social media can mine the sentiment difference between positive text and negative text as the feature of positive and negative semantic classification. In addition, the text features, image features and social graph features of social media information are extracted by a feature extractor. Finally, the text features, image features, social graph features and comment text features of the four modalities of social media information are used to obtain the enhanced features between each two modalities using a cross-modal co-attention mechanism. Then all the enhanced modal features are integrated into the positive and negative semantic classification model, further improving the performance of the positive and negative semantic classification model. BRIEF DESCRIPTION OF THE DRAWINGS

[0109] Figure 1 It is the overall structure diagram of the present invention. DETAILED DESCRIPTION

[0110] A specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings, but it should be understood that the protection scope of the present invention is not limited by the specific implementation.

[0111] The present invention comprises the following steps:

[0112] S1: Mining and analyzing social media information.

[0113] The specific steps of S1 are:

[0114] S11: Set is a set of social media information with text and pictures. For each set of social media information , and ,in: , and Respectively indicate the release of the social media information Text, images and users, express Comments;

[0115] Each comment All by the corresponding users published;

[0116] S12: To represent the user behavior on social media, a graph is created. G :

[0117] ;

[0118] in: It is a collection of nodes, including user nodes, comment nodes, and content nodes of social media information;

[0119] It is the adjacency matrix between nodes, which is used to describe the relationship between nodes, including the release, comment and forwarding of social media information;

[0120] is the set of edges;

[0121] S13: Define positive and negative semantic classification as a binary classification task. represents the category label, where Indicates negative information, otherwise , learning function To predict tweets .

[0122] S2: Construct a multimodal positive and negative semantic classification architecture, including text and visual feature extractors, comment feature extractors, and graph feature extractors.

[0123] The workflow of the text and visual feature extractor is to extract the semantic features of sentences using convolutional neural networks (CNNs) and pooling layers:

[0124] S211: For each set of social media information , whose text are padded or truncated to have the same number of tokens, i.e. , which is expressed as ,in: , is the word embedding dimension, for No. word embeddings of words;

[0125] S212: In the word embedding matrix Apply the convolution layer to get the feature map ,in is the size of the convolution kernel, let ;

[0126] S213: Using the maximum pooling, we get , using different receptive fields of Filters are used to obtain semantic features of different granularities;

[0127] S214: Connect the outputs of all filters to form Overall text features:

[0128] (1).

[0129] The workflow of the comment feature extractor is to extract images using the pre-trained model ResNet50 trained on the ImageNet database. Features:

[0130] S221: Extract the output of the penultimate layer of ResNet50 and represent it as ;

[0131] S222: It obtains the final visual features with the same dimension as the text features through a fully connected layer, namely:

[0132] (2)

[0133] in: is the weight matrix of the fully connected layer;

[0134] is the sigmoid activation function.

[0135] The workflow of the review feature extractor is:

[0136] The corresponding comment information is extracted from social media information, but the directly collected network text usually contains a lot of junk information, which will seriously interfere with the experimental results if it is not cleaned up. The data is full of a large number of numbers, network characters, emoticons, etc. These characters do not affect the emotional judgment of the comments. Therefore, the present invention performs text preprocessing operations to filter out non-Chinese characters.

[0137] Comment sentiment analysis is to mine sentiment words in comments and extract features to identify the sentiment tendency of the text.

[0138] The overall sentiment tendency of social media comments is not easy to obtain, and the sentiment tendency of each comment should be distinguished. Regardless of the intensity of the sentiment, the text will be divided into positive and negative categories. The present invention is based on the roberta-base-finetuned-chinanews-chinese model for annotation.

[0139] S231: The comment sentiment value is divided into 5 star ratings, namely 1, 2, 3, 4, and 5. When the star rating is 1, 2, and 3, the comment is judged as a negative comment; when the star rating is 4 and 5, the comment is judged as a positive comment, that is:

[0140] (3)

[0141] in: For roberta-base-finetuned-chinanews-chinese model;

[0142] For comments Emotional star rating;

[0143] S232: The label of the negative comment is set to 0, and the label of the positive comment is set to 1;

[0144] (4)

[0145] In this way, each comment gets a corresponding label according to the corresponding sentiment score;

[0146] S233: Convert these tags into vectors to obtain the final comment features of social media information, namely:

[0147] (5)

[0148] in: is the number of comments corresponding to each social media message.

[0149] The workflow of the graph feature extractor is:

[0150] Social graph data lacks reliable connections between entities due to privacy issues or data crawling limitations, so it is necessary to infer hidden connections between nodes. According to network homogeneity, similar nodes are more likely to be connected to each other than dissimilar nodes. Therefore, the feature similarity between different nodes is calculated, and the connection between nodes with high similarity is inferred. Specifically:

[0151] S241: Define the node embedding matrix as , There are three types of nodes in the paper. The sentence vectors are used as the initial embeddings of the text and comment nodes of the social media information, and the average of the text node embeddings of the social media information posted by the user is used as the initial user embedding. and calculate and The correlation coefficient between ,Right now:

[0152] (6)

[0153] in: and for and Node embedding of

[0154] If the similarity is greater than 0.5, there is a potential edge between them, that is:

[0155] (7)

[0156] S242: Infer the original adjacency matrix of potential edge pairs Enhancement, Expressed as Elements of express and There is an edge between them, otherwise , then the enhanced adjacency matrix Elements in is defined as:

[0157] (8);

[0158] S243: Use Graph Attention Network (GAT) to capture social graph structure information. Different from the traditional Graph Attention Network (GAT), the signed attention mechanism is used to capture the positive and negative correlations between adjacent nodes to obtain better graph features. Use the Graph Attention Network based on Signed Attention (Signed GAT) to capture multi-faceted correlations. The key to GAT is the aggregation of neighborhood information. For nodes and its neighbor nodes ;

[0159] First, compute the nodes and The attention weight set of each node in ,Right now:

[0160] (9)

[0161] Among them: || represents the serial operation;

[0162] and represents a learnable parameter;

[0163] and Representation Node and , and ;

[0164] is the activation function;

[0165] Then, the softmax function is used to normalize the attention weights. exist and It may be a negative value between nodes, which becomes very small after using the softmax function. In fact, the attention weights between nodes contain potential positive and negative relationships, which will be ignored when using the softmax function directly. , get the weight between it and its neighboring nodes After normalization by the softmax function, the weight is . It can be seen that the node corresponding to the value "-0.9" in the weight vector contributes the least to the output. However, "-0.9" may mean that the two node vectors are in opposite directions. This larger negative relationship may also be beneficial to positive and negative semantic classification. For example, it may reflect disguised behavior, such as a communicator purchasing some honest users as comments against positive information, whose node vectors are inherently negatively correlated. Unfortunately, existing positive and negative semantic classification models ignore this negative correlation. Therefore, the present invention designs a graph attention network based on signed attention, namely signed GAT. It uses signed attention to reflect the positive and negative relationships between nodes. Specifically, for nodes , we compare it with the attention weights of the neighboring nodes The inversion of .

[0166] Then, the softmax function is used to calculate and The normalized weight is:

[0167] (10);

[0168] S244: In order to capture the positive and negative relationships between nodes, we use and to obtain the weighted sum of adjacent node features, then concatenate the two vectors together and pass through a fully connected layer to obtain the final node features, for example, The node features of can be obtained, namely

[0169] (11);

[0170] in: is the weight matrix of the fully connected layer;

[0171] is the activation function;

[0172] for The feature matrix of

[0173] S245: For each node, update its embedding according to equation (8) to obtain the updated node embedding matrix ,in: is the number of nodes, and then a multi-head attention mechanism is used to capture features from different angles. Finally, the updated node embeddings of each head are connected together as the overall graph feature:

[0174] (12)

[0175] in: represents the number of attention points, then social media information Graph features of locations Corresponds to No. List.

[0176] S3: Multimodal feature fusion; use the common attention mechanism to capture the mutual information between different modalities and enhance the cross-modal features by learning the attention weights between different modal features.

[0177] The specific steps of S3 are:

[0178] S31: Use multi-head self-attention mechanism to enhance intra-modality feature representation, e.g., for text features , respectively use , and To calculate its query matrix, key matrix and value matrix, where: is a linear transformation, and then, a multi-head self-attention feature of the text modality is generated:

[0179] (13)

[0180] in: Expressed as an attention head;

[0181] is the linear transformation of the output;

[0182] right , and Perform the same operation to obtain the corresponding features , and ;

[0183] S32: Generate enhanced multimodal features using cross-modal attention mechanism. Specifically, To perform text-visual cross-modal attention, we first perform the same operation as the multi-head self-attention mechanism above, but replace Replace with , get the query matrix ,Will Replace with , get the key matrix Sum Matrix , and then the cross-modal enhancement features are obtained :

[0184] (14)

[0185] in: is the linear transformation of the output;

[0186] represents text features enhanced by visual features based on text visual relevance;

[0187] Based on the cross-modal joint attention process, by exchanging the roles of text features and visual features, the enhanced visual features can be obtained through formula (11): .

[0188] S4: Modality alignment: Based on the cross-modal attention mechanism, enhanced text features and visual features can be obtained, and vice versa. However, for social media information, its different forms of expression should have an intrinsic connection. This correspondence is not included in the joint attention mechanism. Therefore, modality alignment is introduced to improve the representation learned in each modality by enhancing text features and making them close to enhanced graphic features.

[0189] The specific steps of S4 are:

[0190] S41: For social media information , and enhance its graph features and enhanced text features Transform to the same modal feature space, that is:

[0191] (15)

[0192] in: and is a learnable parameter;

[0193] S42: Use MSE loss for modality alignment to reduce and Distance between:

[0194] (16)

[0195] in: is the total number of social media messages;

[0196] Get finely aligned text features and graphic features , used for multimodal fusion.

[0197] S5: Fusion of multimodal features.

[0198] The specific steps of S5 are:

[0199] S51: Again and These four modal features perform the above cross-modal attention mechanism, and finally obtain 12 cross-modal enhanced features: ;

[0200] S52: Then concatenate them as the final multimodal features:

[0201] (17).

[0202] S6: Adversarial training classification.

[0203] The specific steps of S6 are:

[0204] S61: Social media information The final multimodal features are input into the fully connected layer to predict Is it negative information?

[0205] (18)

[0206] in: Represents prediction is the probability of negative information;

[0207] S62: Then use the cross entropy loss function:

[0208] (19)

[0209] Represents prediction is the probability of negative information;

[0210] The final losses are as follows:

[0211] (20)

[0212] in: and A factor used to balance the two losses.

[0213] The above disclosure is only a specific embodiment of the present invention, but the present invention is not limited thereto, and any changes that can be conceived by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. A cross-modal positive and negative semantic classification method based on text sentiment and image content perception, characterized in that: The following steps are involved: S1: Mining and analyzing social media information; S2: Construct a multimodal positive and negative semantic classification architecture, including text feature extractor, visual feature extractor, comment feature extractor and graph feature extractor; S3: Multimodal feature fusion; Use the co-attention mechanism to capture the mutual information between different modalities and enhance cross-modal features by learning the attention weights between different modal features; S4: Modality alignment; Introducing modality alignment to improve the representations learned in each modality by augmenting text features to make them close to augmented graphic features; S5: Fusion of multimodal features; S6: Adversarial training classification; The specific steps of S3 are: S31: Use multi-head self-attention mechanism to enhance intra-modal feature representation, for text features , respectively use , and To calculate its query matrix, key matrix and value matrix, where: is a linear transformation, and then, a multi-head self-attention feature of the text modality is generated: (13) in: Expressed as an attention head; is the linear transformation of the output; right , and Perform the same operation to obtain the corresponding features , and ; S32: Generate enhanced multimodal features using cross-modal attention mechanism. Specifically, To perform text-visual cross-modal attention, we first perform the same operation as the multi-head self-attention mechanism above, but replace Replace with , and get the query matrix ,Will Replace with , get the key matrix Sum Matrix , and then the cross-modal enhancement features are obtained : （14） in: is the linear transformation of the output; represents text features enhanced by visual features based on text visual relevance; Based on the cross-modal joint attention process, by exchanging the roles of text features and visual features, the enhanced visual features can be obtained through formula (11): ; The specific steps of S4 are: S41: For social media information , and enhance its graph features and enhanced text features Transform to the same modal feature space, that is: （15） in: and is a learnable parameter; S42: Use MSE loss for modality alignment to reduce and Distance between: （16） in: is the total number of social media messages; Get finely aligned text features and graphic features , used for multimodal fusion.

2. The cross-modal positive and negative semantic classification method based on text emotion and image content perception according to claim 1 is characterized by: The specific steps of S1 are: S11: Set is a set of social media information with text and pictures. For each set of social media information , and ,in: , and Respectively indicate the release of the social media information Text, images and users, express Comments; Each comment All by the corresponding users published; S12: To represent the user’s behavior on social media, a graph is constructed. G : ； in: It is a collection of nodes, including user nodes, comment nodes, and content nodes of social media information; It is the adjacency matrix between nodes, which is used to describe the relationship between nodes, including the release, comment and forwarding of social media information; is the set of edges; S13: Define positive and negative semantic classification as a binary classification task. represents the category label, where Indicates negative information, otherwise , learning function To predict social media information .

3. The cross-modal positive and negative semantic classification method based on text emotion and image content perception according to claim 2 is characterized by: The workflow of the text feature extractor is to extract the semantic features of sentences using convolutional neural networks and pooling layers: S211: For each set of social media information , whose text are padded or truncated to have the same number of tokens, i.e. , which is expressed as ,in: , is the word embedding dimension, for No. word embeddings of words; S212: In the word embedding matrix Apply the convolution layer to get the feature map ,in is the size of the convolution kernel, let ; S213: Using the maximum pooling, we get , using different receptive fields of Filters are used to obtain semantic features of different granularities; S214: Connect the outputs of all filters to form Overall text features: （1）。 4. The cross-modal positive and negative semantic classification method based on text emotion and image content perception according to claim 3 is characterized by: The workflow of the visual feature extractor is to extract images using the pre-trained model ResNet50 trained on the ImageNet database Features: S221: Extract the output of the penultimate layer of ResNet50 and represent it as ; S222: It obtains the final visual features with the same dimension as the text features through a fully connected layer, namely: （2） in: is the weight matrix of the fully connected layer; is the sigmoid activation function.

5. The cross-modal positive and negative semantic classification method based on text emotion and image content perception according to claim 4 is characterized by: The workflow of the review feature extractor is: S231: The comment sentiment value is divided into 5 star ratings, namely 1, 2, 3, 4, and 5. When the star rating is 1, 2, and 3, the comment is judged as a negative comment; when the star rating is 4 and 5, the comment is judged as a positive comment, that is: （3） in: for the roberta-base-finetuned-chinanews-chinese model; For comments Emotional star rating; S232: The label of the negative comment is set to 0, and the label of the positive comment is set to 1; （4） In this way, each comment gets a corresponding label according to the corresponding sentiment score; S233: Convert these tags into vectors to obtain the final comment features of social media information, namely: （5） in: is the number of comments corresponding to each social media message.

6. The cross-modal positive and negative semantic classification method based on text emotion and image content perception according to claim 5 is characterized by: The workflow of the graph feature extractor is: Social graph data lacks reliable connections between entities due to privacy issues or data crawling limitations. Therefore, it is necessary to infer hidden connections between nodes. According to network homogeneity, similar nodes are more likely to be connected to each other than dissimilar nodes. Therefore, the feature similarity between different nodes is calculated, and the connection between nodes with high similarity is inferred. Specifically: S241: Define the node embedding matrix as , There are three types of nodes in the paper. The sentence vectors are used as the initial embeddings of the text and comment nodes of the social media information, and the average of the text node embeddings of the social media information posted by the user is used as the initial user embedding. and calculate and The correlation coefficient between ,Right now: （6） in: and for and Node embedding of If the similarity is greater than 0.5, there is a potential edge between them, that is: （7） S242: Infer the original adjacency matrix of potential edge pairs Enhancement, Expressed as Elements of express and There is an edge between them, otherwise , then the enhanced adjacency matrix Elements in is defined as: （8）； S243: Use graph attention network to capture social graph structure information, use signature attention-based graph attention network to capture multi-faceted correlations, for nodes and its neighbor nodes ; First, compute the nodes and The attention weight set of each node in ,Right now: （9） Among them: || represents the serial operation; and represents a learnable parameter; and Representation Node and , and ; is the activation function; Then, the attention weights are normalized using the softmax function and the softmax function is used to calculate and The normalized weight of is: （10）； S244: In order to capture the positive and negative relationships between nodes, we use and To obtain the weighted sum of adjacent node features, the two vectors are then concatenated together and passed through a fully connected layer to obtain the final node feature. The node features of can be obtained, namely （11）； in: is the weight matrix of the fully connected layer; is the activation function; for The feature matrix of S245: For each node, update its embedding according to formula (8) to obtain the updated node embedding matrix ,in: is the number of nodes, and then a multi-head attention mechanism is used to capture features from different angles. Finally, the updated node embeddings of each head are connected together as the overall graph feature: （12） in: represents the number of attention points, then social media information Graph features of locations Corresponds to No. List.

7. The cross-modal positive and negative semantic classification method based on text emotion and image content perception according to claim 6 is characterized by: The specific steps of S5 are: S51: Again and These four modal features perform the above cross-modal attention mechanism, and finally obtain 12 cross-modal enhanced features: ; S52: Then concatenate them as the final multimodal features: （17）。 8. The cross-modal positive and negative semantic classification method based on text emotion and image content perception according to claim 7 is characterized by: The specific steps of S6 are: S61: The multimodal features finally obtained from social media information are input into the fully connected layer to predict whether it is negative information: （18） in: Indicates prediction is the probability of negative information; S62: Then use the cross entropy loss function: （19） Indicates prediction is the probability of negative information; The final losses are as follows: （20） in: and A factor used to balance the two losses.

Images