News public opinion analysis method based on multi-modal graph representation learning

Abstract

The invention discloses a news public opinion analysis method based on multi-modal graph representation learning, and belongs to the field of multi-modal deep learning. Constructing a graph structure from the text and image features by adopting a graph topology extraction and enhancement module; a cross-modal multi-scale feature fusion module is adopted to align and integrate features of texts and images in a multi-scale manner, and the cross-modal multi-scale feature fusion module comprises an ASPP and a global attention mechanism module; a final expansion mode feature, a single-mode text mode feature and an image mode feature are obtained through an image topology extraction and enhancement module and a cross-mode multi-scale feature fusion module and are input into a classifier together to obtain respective probabilities, and finally a news public opinion classification result is obtained through weighted fusion. The performance of the DCPNet on a multi-mode data set is superior to that of a traditional single-mode method and other existing multi-mode irony detection methods, and the huge potential of the DCPNet in practical application is proved.

Description

Technical Field

[0001] The present invention relates to the field of multimodal deep learning, and particularly to a news public opinion analysis method based on multimodal graph representation learning. Background Art

[0002] In the era of highly fragmented information dissemination, news public opinion analysis has become an important tool in multiple fields such as government decision-making, enterprise brand management, and social governance. By analyzing news reports, social media comments, and online discussions, public opinion analysis can perceive the trend of public sentiment in real time, identify potential risks, and provide data support for response strategies. However, with the evolution of information dissemination methods, the influence of ironic content in the public opinion field has been increasing, posing severe challenges to traditional sentiment analysis and opinion mining.

[0003] Irony is an important part of human communication, often conveying ironic or implicit intentions through expressions that are opposite to the literal meaning. Its subtle and multi-dimensional nature makes irony detection a highly challenging task in the field of public opinion analysis. The widespread use of social media platforms has further exacerbated the complexity of irony detection because users increasingly combine text and visual content to convey delicate emotional information. This change highlights the increasingly important position of multimodal irony detection (MSD), which aims to analyze text and visual cues to accurately identify ironic content.

[0004] Multimodal irony detection (MSD) faces a series of challenges. One of them is to reveal the implicit intention behind ironic expressions. Different from sentiment analysis, which directly conveys emotions, irony often manifests as a contradiction between the literal meaning and the expected emotional tone. In a multimodal context, the nuances of irony require the system to capture and understand the complex relationship between text and visual information. In recent years, certain progress has been made in the research on multimodal irony detection. Early studies explored the combination of text and visual features for irony detection. Some subsequent developments introduced attention mechanisms and graph neural network-based methods to better model cross-modal relationships. However, some existing methods using bimodal encoders, such as BERT (Bidirectional Encoder Representations from Transformers) for text feature extraction and ViT (Vision Transformer) for image feature extraction, still have difficulty fully capturing the complex interactions between modalities, making it difficult for them to distinguish true ironic cues from irrelevant signals.

[0005] To address the above challenges, the present invention uses a Graph Convolutional Network to separately extract fine-grained graph structure information from text and image modalities, and combines multi-scale feature fusion and attention mechanisms to improve the accuracy of cross-modal information integration. These advanced neural network architectures enable the model to handle complex relationships from different modalities in the multi-modal irony detection task, thereby effectively improving the performance of the model.

[0006] The present invention introduces a news public opinion analysis method based on multi-modal graph representation learning, which uses a graph neural network to extract and capture information within the modality, and realizes the capture of cross-modal feature information through a multi-scale feature fusion strategy and an attention mechanism, thereby completing multi-modal news public opinion analysis. Summary of the Invention

[0007] The present invention proposes a novel multi-modal irony detection framework - Dual-Channel Cross-Modal Perception Network (DCPNet), aiming to address the challenges faced by multi-modal irony detection by effectively integrating single-modal and cross-modal features, and combining enhanced structural information. To capture the rich topological relationships in each modality, we introduce a graph topology extraction and enhancement module, which uses CLIP (Contrastive Language-Image Pre-Training) to align text and image modality features, and constructs a graph structure from text and image features to promote deeper semantic representations. In addition, we also propose a cross-modal multi-scale feature fusion module, aiming to ensure the capture of comprehensive context information by multi-scale alignment and integration of text and image features. To optimize the feature fusion process and improve the accuracy of irony detection, we introduce an attention mechanism to allocate appropriate weights to text and visual features.

[0008] A news public opinion analysis method based on multi-modal graph representation learning, and the steps of using DCPNet to implement multi-modal irony detection are as follows:

[0009] Step 1. Construct a graph structure from text and image features using the graph topology extraction and enhancement module.

[0010] Step 1.1. The graph data is first preliminarily processed by the multi-modal pre-training model CLIP to align the image and text into a shared semantic space. The specific method is as follows:

[0011] For a given text-image pair (T, V), where T represents the text content and V represents the corresponding image, given a sample set S = {T i , V i}, CLIP will generate output encodings for the text and image, and then obtain the hidden encoding representation of the image-text pair, as follows:

[0012]

[0013] where n and m respectively represent the text T i and the image V i sequence lengths, S t 、S v respectively represent the lengths of the text and image encoding features, and the obtained t CLS and v CLS are special tokens required by the encoder.

[0014] The hidden encoded representation of the image-text pair is segmented to build a graph structure. Specifically, for a single image, it is segmented into image patches of the same size, and each patch serves as a node. For the text, it is finely segmented by words.

[0015] Step 1.2: Treat the segmented text patches and image patches as graph nodes, apply a fully connected connection between the nodes for feature aggregation, and construct a graph structure. Input the graph structure into a feed-forward neural network for further feature extraction and aggregation. Then input the result into the graph neural network GCN. In GCN: For each text-image pair (T i , V i ), the adjacency matrix A t or A v of the single-modal graph and the corresponding node representation T i or V i are input into multiple layers of GCN to obtain the feature representations G t and G v of single-modal topological feature aggregation. For each graph convolution operation, in the l-th GCN layer, each node is updated according to the hidden representation of its neighborhood in the cross-modal graph, defined as follows:

[0016]

[0017] where is the normalized symmetric adjacency matrix of A t or A v , D is the degree matrix, G l-1 represents the hidden graph representation evolved from the previous layer of GCN, W l , b l are trainable parameters, A t is the graph adjacency matrix composed of text patches, and A v ]>is the graph adjacency matrix composed of image patches.

[0018] Step 2: Use a cross-modal multi-scale feature fusion module to multi-scale align and integrate the features of text and images. The cross-modal multi-scale feature fusion module includes ASPP and a global attention mechanism module;

[0019] Use ASPP (Atrous Spatial Pyramid Pooling). For the text and image encodings after the initial processing of CLIP, adapt to different semantic levels within the text and images by applying convolutions with different dilation rates. The formula is expressed as:

[0020]

[0021] where j = 1, 2, 3, 4, 5, K j represents the convolutional kernel of the dilated convolution, d j represents different dilation rates, and Y j represents the feature representation after the aggregation of features from different dilated convolutional layers. After the dilated convolution, apply a fully connected layer to further extract complete context information from the images and text:

[0022] F MSPP = concat(Y1, Y2, Y3, Y4, Y5)

[0023] where Y j is the feature map of different scale layers, representing the global semantic information of the image or text.

[0024] The cross-modal multi-scale feature fusion module advocates implementing a single-layer multi-head attention mechanism, defined as follows:

[0025]

[0026] where f Q (·), f K (·) and f V (·) represent linear feed-forward neural networks, ‖·‖2 represents the L2 norm, D attn represents the score of each head in the multi-head attention mechanism, d represents the dimension size of Q and K, Z attn represents the feature weights after fusion, h represents the number of heads in the multi-head attention mechanism, and the outputs from multiple attention heads are averaged to generate the final output. Then, fuse the initial input F MSPP with the attention-based representation Z attn as shown in the following formula:

[0027] Z out = FFN((1 - α)Z attn + αF MSPP ),

[0028] Among them, α is a hyperparameter used to adjust the attention-based representation Z attn and the initial representation F MSPP FFN(·) represents a shallow feedforward neural network.

[0029] Step 3: The final cross-modal feature Z is obtained through the graph topology extraction and enhancement module and the cross-modal multi-scale feature fusion module. out and unimodal text modality feature G t and image modality features G v , input into the classifier together to obtain their respective probabilities p fuse , p t and p v , and finally the weighted fusion is performed to obtain the final classification result of multimodal sarcasm detection.

[0030] Furthermore, in order to train the model, we use the cross entropy loss function to calculate the loss of each modality, and finally get a total loss value, which is the fusion feature Z out With the text modality feature G t Together with the probability of the image modality feature Gv, it is input into the classifier for prediction and the loss is calculated:

[0031] L fuse =CE(p fuse ,y),

[0032] L t =CE(p t ,y),

[0033] L v =CE(p v ,y).

[0034] Y represents the true label of the data;

[0035] The final predicted probability is given by the following formula:

[0036] L total =L fuse +L t +L v .

[0037] The present invention proposes an innovative framework for multimodal irony detection, which can effectively capture structured clues from text and images. In the structural branch of the model, a graph topology extraction and enhancement module is designed to focus on key entities and their relationships, thereby enhancing the model's ability to detect irony based on entity-level features. To model the inconsistencies between different modalities, a multi-scale cross-modal feature fusion module is adopted, which aligns and integrates text and image features at multiple levels. In addition, an attention mechanism is introduced, which dynamically adjusts weights according to the contribution of each modality in the task. This mechanism enables the model to flexibly cope with the changes in the importance of text and images in different scenarios, thereby enhancing the robustness and generalization ability of the model. Experimental results further show that DCPNet outperforms traditional unimodal methods and other existing multimodal irony detection methods on multimodal datasets, which helps to more accurately identify ironic comments, restore the real public opinion situation, and provide a more credible reference for public opinion decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is the overall framework diagram of the present invention.

[0039] Figure 2 It is the diagram of the graph topology extraction and enhancement module.

[0040] Figure 3 It is the diagram of the cross-modal multi-scale feature fusion module. DETAILED DESCRIPTION OF THE INVENTION

[0041] The following further describes the specific implementation manners of the present invention in conjunction with the drawings and technical solutions.

[0042] A news public opinion analysis method based on multimodal graph representation learning, and the steps of using DCPNet to implement multimodal irony detection are specifically as follows:

[0043] Step 1. Construct a graph structure from text and image features by using the graph topology extraction and enhancement module.

[0044] Step 1.1. The graph data is first preliminarily processed by the multimodal pre-trained model CLIP to align the image and text into a shared semantic space. This enables the model to better understand and align the semantic relationships between the image and text. CLIP has achieved extensive success in multiple downstream tasks, including OCR (Optical Character Recognition), geolocation, and action recognition, etc. The specific implementation is as follows:

[0045] For a given text-image pair (T, V), where T represents the text content and V represents the corresponding image, given a sample set S = {T i , V i}, CLIP generates output encodings for the text and the image, and then obtains the hidden encoding representation of the image-text pair, as follows:

[0046]

[0047] where n and m represent the sequence lengths of the text T i and the image V i respectively, S t , S v represent the lengths of the text and image encoding features respectively, and the obtained t CLS and v CLS are special tokens required by the encoder.

[0048] To better extract the topological structure features of the two modalities, the hidden encoding representation of the image-text pair is segmented to build a graph structure. Specifically, for a single image, it is segmented into image patches of the same size, and each patch serves as a node. For the text, it is segmented at the word level with fine granularity.

[0049] Step 1.2. Subsequently, to better explore the internal relationships and topological structures within different modalities and enhance the representational ability of features, the text and image features are constructed into a graph structure. Each feature is regarded as a node in the graph, and the relationships between nodes are formed through full connectivity. In this way, the topological structure information between nodes is effectively captured. The segmented text patches and image patches are regarded as graph nodes. To capture rich relationships between nodes, reduce information loss, and improve efficiency, a fully connected layer is applied between nodes for feature aggregation to construct the graph structure. The graph structure is input into a feed-forward neural network for further feature extraction and aggregation. Then the result is input into the graph neural network GCN. In the GCN: In this fully connected graph structure, all nodes can exchange information with each other, thus realizing fast and comprehensive information propagation between nodes. For each text-image pair (T i , V i ), a graph architecture is explored to extract key semantic clues by aggregating the correlations between nodes in the single-modal graphs. Specifically, the adjacency matrix A t or A v of the single-modal graph and the corresponding node representations T i or V i are input into multiple layers of GCN to obtain the feature representations G t and G v of the single-modal topological feature aggregation. For each graph convolutional operation, in the l-th GCN layer, each node is updated according to the hidden representations of its neighborhood in the cross-modal graph, defined as follows:

[0050]

[0051] Among them is A t or A v is the normalized symmetric adjacency matrix, D is the degree matrix, and G l-1 represents the hidden graph representation evolved from the previous layer of GCN, and W l , b l are trainable parameters, A t is the graph adjacency matrix composed of text blocks, and A v is the graph adjacency matrix composed of image blocks.

[0052] Step 2: Use a cross-modal multi-scale feature fusion module to multi-scale align and integrate the features of text and images. The cross-modal multi-scale feature fusion module includes ASPP and a global attention mechanism module;

[0053] To achieve the fusion of multi-modal features, the present invention designs a method for multi-scale feature fusion. The features of images and texts usually show obvious scale differences. Images usually contain rich spatial details, while texts present sequential language structures. To effectively capture multi-scale context information from these two modalities, a module using dilated convolutions for cross-modal multi-scale feature extraction and enhancement is designed. This method allows the convolutional kernel to extract features of different scales of images or texts, thereby increasing the receptive field without increasing the computational cost.

[0054] In image processing, dilated convolutions can capture context information of various scales through multi-scale convolutions using different dilation rates. This method is also applicable to the text modality, especially when dealing with long texts or texts rich in semantic information. Using ASPP (Atrous Spatial Pyramid Pooling), for the text and image encodings after the initial processing of CLIP, convolutions with different dilation rates are applied to adapt to different semantic levels within the text and images. The formula is expressed as:

[0055]

[0056] where j = 1, 2, 3, 4, 5, and K j represents the convolutional kernel of the dilated convolution, d j represents different dilation rates, and Y j represents the feature representation after the aggregation of features of different dilated convolutional layers. Through this method, the model can effectively perform multi-scale feature extraction on images and texts, providing rich context information for subsequent cross-modal fusion. Although images and texts contain local and global information respectively, they also have many complementary aspects. After the dilated convolution, we apply a fully connected layer to further extract complete context information from images and texts:

[0057] FMSPP = concat(Y1, Y2, Y3, Y4, Y5)

[0058] Among them, Y j is the feature map of different scale layers, representing the global semantic information of the image or text.

[0059] To optimize computing resources and improve the efficiency during the training process of the multi-modal irony detection graph data, the cross-modal multi-scale feature fusion module advocates implementing a single-layer multi-head attention mechanism, which is defined as follows:

[0060]

[0061] Among them, f Q (·), f K (·) and f V (·) represent linear feed-forward neural networks, ‖·‖2 represents the L2 norm, D attn represents the score of each head in the multi-head attention mechanism, d represents the dimension size of Q and K, Z attn represents the feature weight after fusion, h represents the number of heads in the multi-head attention mechanism, and the outputs from multiple attention heads are averaged to generate the final output. Then, the initial input F MSPP is fused with the attention-based representation Z attn as shown in the following formula:

[0062] Z out = FFN((1 - α)Z attn + αF MSPP ),

[0063] Among them, α is a hyperparameter used to adjust the balance between the attention-based representation Z attn and the initial representation F MSPP . FFN(·) represents a shallow feed-forward neural network.

[0064] Step 3: Through the graph topology extraction and enhancement module and the cross-modal multi-scale feature fusion module, the final cross-modal feature Z out and the single-modal text modality feature G t and the image modality feature G v are input into the classifier to obtain their respective probabilities p fuse , p t and p v , and finally weighted fusion is performed to obtain the classification result of the final multi-modal irony detection.

[0065] Furthermore, to train the model, we use the cross-entropy loss function to calculate the loss of each modality and finally obtain a total loss value. The fused feature Z out and the text modality feature Gt Input the probability of the image modality feature Gv into the classifier for prediction and calculate the loss:

[0066] L fuse = CE(p fuse , y),

[0067] L t = CE(p t , y),

[0068] L v = CE(p v , y).

[0069] Y represents the true label of the data;

[0070] The final predicted probability is given by the following formula:

[0071] L total = L fuse + L t + L v .

[0072] The experiment of the present invention uses the MMSD2.0 dataset for experimental verification. The MMSD2.0 dataset contains paired English tweets and images, and the goal is to determine whether the tweets and their corresponding images convey ironic or non-ironic information.

[0073] Table 1 shows the experimental results of multimodal irony detection

[0074]

[0075] In the evaluation of this method, we adopt accuracy (Acc), precision (P), recall (R), and F1 value (F1) as performance evaluation metrics. These metrics help to comprehensively evaluate the overall effect of classification. Accuracy measures the proportion of correctly classified instances, while the F1 value is the harmonic mean of precision and recall, which is particularly suitable for class-imbalanced datasets.

[0076] We use clip-vit-base-patch32 as the backbone network of the model. The model is optimized using the Adam optimizer, and the learning rate is set to 5e-4. The batch size is set to 32, the training is carried out for 20 epochs, and an early stopping strategy is adopted with a patience value of 5 to prevent overfitting.

[0077] For the text modality, we embed each word as a 512-dimensional vector. For the image modality, we embed each visual region block as a 768-dimensional vector, and the resolution of the image region block is 32. In the preliminary experiments, we determined that using two layers of GCN achieved the best results. In addition, we set the dropout ratio to 0.1 to reduce the risk of overfitting.

[0078] The operating device system of the present invention is Ubuntu 16.04, the CPU model is Intel Xeon CPU E5-2650v4@2.20GHz, and the GPU used is NVIDIA GeForce TITAN V with a video memory of 12G.

Claims

1. A news public opinion analysis method based on multi-modal graph representation learning, characterized in that, The steps to implement multi-modal sarcasm detection using DCPNet are as follows: Step 1: Use the graph topology extraction and enhancement module to construct a graph structure from text and image features; Step 2: Use the cross-modal multi-scale feature fusion module to multi-scale align and integrate the features of text and images. The cross-modal multi-scale feature fusion module includes the ASPP and global attention mechanism modules; Step 3: Through the graph topology extraction and enhancement module and the cross-modal multi-scale feature fusion module, the final cross-modal features, single-modal text modal features, and image modal features are obtained, and are input into the classifier to obtain their respective probabilities, and finally weighted fusion is performed to obtain the classification result of the final multi-modal sarcasm detection.

2. The news public opinion analysis method based on multi-modal graph representation learning according to claim 1, wherein The specific content of the above Step 1 is as follows: Step 1.1: The graph data is first preliminarily processed by the multi-modal pre-training model CLIP to align the image and text into a shared semantic space; the specific method is as follows: For a given text-image pair (T, V), where T represents the text content and V represents the corresponding image, given a sample set S = {T i , V i}, CLIP generates output encodings for the text and the image, and then obtains the hidden encoding representation of the image-text pair, as follows: T i = {t1, t2, …, t n , t CLS} ∈ R n×St V i = {v CLS , v1, v2, …, v m} ∈ R m×Sv where n and m respectively represent the text T i and the image V i sequence lengths, S t 、S v respectively represent the lengths of the text and image encoding features, and the obtained t CLS and v CLS are special tokens required by the encoder; The hidden encoding representation of the image-text pair is segmented to establish a graph structure; specifically, for a single image, it is segmented into image patches of the same size, and each patch is used as a node. For text, it is finely segmented by word; Step 1.2: Treat the segmented text blocks and image blocks as graph nodes, apply full connections between the nodes for feature aggregation, and construct a graph structure; input the graph structure into a feed-forward neural network for further feature extraction and aggregation; then input the result into a graph neural network GCN, where in the GCN: for each text-image pair (T i , V i ), the adjacency matrix A t or A v and the corresponding node representation T i or V i are input into multiple layers of GCN to obtain the feature representations G t and G v of single-modal topological feature aggregation; for each graph convolutional operation, in the l-th GCN layer, each node is updated according to the hidden representation of its neighborhood in the cross-modal graph, defined as follows: Among them is A t or A v is the normalized symmetric adjacency matrix, D is the degree matrix, G l-1 represents the hidden graph representation evolved from the upper layer of GCN, W l , b l are trainable parameters, A t is the graph adjacency matrix composed of text blocks, A v is the graph adjacency matrix composed of image blocks.

3. A news public opinion analysis method based on multi-modal graph representation learning according to claim 1, characterized in that The specific content of the above Step 2 is as follows: Using ASPP, for the text and image encodings after the preliminary processing of CLIP, convolutions with different dilation rates are applied to adapt to different semantic levels within the text and image; the formula is expressed as: where j = 1, 2, 3, 4, 5, K j represents the convolutional kernel of the dilated convolution, d j represents different dilation rates, Y j represents the feature representation after the aggregation of the features of different dilated convolutional layers; after the dilated convolution, a fully connected layer is applied to further extract complete context information from the image and text: F MSPP = concat(Y1, Y2, Y3, Y4, Y5) Among which Y j is the feature map of different scale layers, representing the global semantic information of the image or text; The cross-modal multi-scale feature fusion module advocates implementing a single-layer multi-head attention mechanism, which is defined as follows: where f Q (·), f K (·) and f V (·) represent linear feedforward neural networks, ‖·‖2 represents the L2 norm, D attn represents the score of each head in the multi-head attention mechanism, d represents the dimension size of Q and K, Z attn represents the fused feature weights, h represents the number of heads in the multi-head attention mechanism, and the outputs from multiple attention heads are averaged to generate the final output; then, the initial input F MSPP is fused with the attention-based representation Z attn as shown in the following formula: Z out = FFN((1 - α)Z attn + αF MSPP ), where α is a hyperparameter used to adjust the balance between the attention-based representation Z attn and the initial representation F MSPP ; FFN(·) represents a shallow feed-forward neural network.

4. A news public opinion analysis method based on multi-modal graph representation learning according to claim 1, characterized in that The specific steps of Step 3 are as follows: The final cross-modal feature Z is obtained through the graph topology extraction and enhancement module and the cross-modal multi-scale feature fusion module. out and the unimodal text modality feature G t and the image modality feature G v are jointly input into the classifier to obtain their respective probabilities p fuse , p t and p v , and finally, weighted fusion is performed to obtain the classification result of the final multi-modal sarcasm detection.

5. The news public opinion analysis method based on multi-modal graph representation learning according to claim 1, wherein, To train the model, the cross-entropy loss function is used to calculate the loss of each modality, and finally a total loss value is obtained. The fused feature Z out is input into the classifier together with the probability of the text modality feature G t and the image modality feature Gv for prediction, and the loss is calculated: L fuse = CE(p fuse , y), L t = CE(p t , y), L v = CE(p v , y). Y represents the true label of the data; The final predicted probability is given by the following formula: L total = L fuse + L t + L v 。

Images