Multi-modal social media popularity prediction method based on time perception hypergraph learning

Abstract

The invention belongs to the technical field of information propagation, and discloses a time perception hypergraph learning-based multi-modal social media popularity prediction method, which comprises the following steps of: firstly, searching first K instances related to a target UGC (User Generated Content) by adopting a method taking time as a core; then, by utilizing UGC attributes, connecting all related instances together to form a hypergraph of the target UGC; secondly, a time-aware guided hypergraph Transform is researched and developed to capture correlation between the interior and cross-modality, and a time encoder is used to embed time information into an information fusion process; and finally, predicting the popularity of the target UGC by using a model comprising two layers of feedforward neural networks. According to the method, modeling is carried out on the time relation of target user generated content (UGC), target representation is enhanced through multi-modal hypergraph aggregation, and the method aims at guiding self-adaptive hypergraph construction through aggregation time perception information so as to assist multi-modal popularity prediction.

Description

Technical Field

[0001] The present invention belongs to the field of information dissemination technology, and relates to multimodal social media popularity prediction, and in particular to multimodal learning and popularity prediction in deep learning, and is a hypergraph representation learning and time-aware method. Background Art

[0002] With the widespread dissemination of multimodal user-generated content (UGC), such as videos, images, and text, on online social media platforms, how to understand, model, and recommend UGC has become an important but challenging problem. Multimodal social media popularity prediction (MSMPP) aims to infer the number of future interactions between users and UGC by learning and aggregating multimodal content. This is very beneficial for helping users filter information and improving various applications such as recommendation and rumor detection. Existing MSMPP research can be summarized into two categories: (1) feature-based methods emphasize the design and integration of hand-crafted UGC features; (2) deep learning-based methods utilize an end-to-end framework to capture more comprehensive multimodal representations. Despite their success, the usability of existing works is limited by the following aspects: (1) existing works process individual UGC independently to learn its representation without considering the mutual relationship between different UGCs. Auxiliary information present in related UGCs is also ignored to help the reasoning of UGCs. For example, different UGCs posted by the same user may interact with similar user groups and generate similar UGC popularity. (2) existing works ignore the temporal dependencies between UGCs. Summary of the invention

[0003] The purpose of the present invention is to address the above technical problems existing in the prior art and propose a framework for a multimodal social media popularity prediction method based on time-aware hypergraph learning. First, in order to preserve the temporal correlation between different UGCs, we retrieve the top K instances related to the target UGC from a time-centric perspective. Subsequently, using UGC attributes, we connect all relevant instances to construct a hypergraph of the target UGC. Second, we design a time-aware hypergraph Transformer to capture the correlation between internal and cross-modalities, while using a temporal encoder to inject temporal information into the information mixing process. This dual attention ensures fine-grained and aligned UGC representations that are crucial for multimodal popularity prediction. Finally, we use a 2-layer feedforward neural network to predict the popularity of the target UGC.

[0004] Based on the above invention ideas, the present invention provides a multimodal social media popularity prediction method based on time-aware hypergraph learning, which includes the following steps:

[0005] S1 uses a hypergraph neural network to perform network embedding on UGC according to the publishing time, and obtains the hypergraph representation embedding of UGC, that is, the embedding vector of user-generated content that contains temporal relationship and multimodal embedding representation information.

[0006] S2 uses Transformer to model the temporal information and multimodal information in the hypergraph to integrate temporal information and the correlation between modalities to obtain the final embedding vector representation of each UGC. For MSMPP, the main challenge lies in how to jointly integrate temporal information and model internal and cross-modal correlations. A time-aware hypergraph Transformer is designed, which integrates a temporal encoder to inject temporal information into the multimodal mixing process of the hypergraph Transformer. For each UGC and its surrounding hyperedges, as well as each feature of its corresponding node, we obtain expressive representations through our designed hypergraph Transformer.

[0007] S3 predicts popularity based on the embedded representation of UGC.

[0008] The above step S1 includes the following sub-steps:

[0009] S11 Given Target and release time t, we retrieve the top K instances related to it in the UGC sequence S before time t from the perspective of the time center and construct a time context sequence P t .P t It can be represented by a ternary sequence: Where K represents the instance count.

[0010] S12 in obtaining P t After that, the obtained instance set P t Convert to target instance Adaptive Hypergraph Each data instance forms a node, namely The attributes of the instances (i.e., posting user, category, topic) construct hyperedges that represent the relationships between instances.

[0011] The above step S2 includes the following sub-steps:

[0012] The S21 temporal encoder (T-encoder) (Xu et al. 2019) is used to model continuous temporal information, mapping scalar timestamps to d T dimensional vector space. This process can be summarized as: in are learnable parameters.

[0013] The S22 Hypergraph Transformer learns intra- and cross-modal correlations through a multimodal mixing process, including node-to-hyperedge and hyperedge-to-node. For node-to-hyperedge, we extract visual (text) information from the visual (text) hypergraph to reduce modality heterogeneity. The process from node to k-th hyperedge is defined as: where v and t represent the visual and textual modalities respectively. represents the feature of the kth hyperedge. Here H is the number of attention heads. Represent the query, key, and value vectors of node i and hyperedge k, respectively, and are calculated by linear transformation and slicing: is the embedding matrix of K hyperedges. Indicates all The key and value transformation of each header. / N1 and p / Indicates the start and end index of the hth slice.

[0014] S23 To mine cross-modal correlations and reduce the impact of modal heterogeneity, we implemented a gating mechanism to control the flow of information from one modality to another. The cross-modal interaction process can be summarized as follows: Specifically, the gating process can be summarized as:

[0015]

[0016] S24 uses a concatenation operation to fuse the H head-specific representations of hyperedge k: Similarly, Can be replaced by To obtain In the process of hyperedge to node, we propagate information from hyperedge to node through a similar but opposite process. Eventually, we will obtain the representation of node i and

[0017] The final popularity prediction of S3 is completed by a two-layer feedforward neural network:

[0018]

[0019] in, and are the parameters of the prediction network. is the concatenation operation. In addition, we use mean squared error (MSE) as the optimization loss to train the parameters of the model.

[0020] Compared with the existing technology, the present invention has the following beneficial effects:

[0021] 1. This paper explores modeling the temporal relationship of target user-generated content (UGC) and enhances the representation of the target through multimodal hypergraph aggregation, aiming to lead the adaptive hypergraph construction by aggregating time-aware information to assist multimodal popularity prediction.

[0022] 2. We propose a guided hypergraph transformer to extend information aggregation to multimodal information mixture. We design intra-modal propagation and inter-modal propagation to capture intra-modal and inter-modal correlations, respectively, to obtain fine-grained aligned user-generated content (UGC) representations. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Schematic diagram of the process of multimodal social media popularity prediction based on time-aware hypergraph learning.

[0024] Figure 2 Schematic diagram of the guided hypergraph transformer.

[0025] Explanation of terms

[0026] Hypergraph: Hypergraphs extend the traditional graph structure, allowing us to connect three or more nodes through hyperedges. Hypergraphs provide a natural way to represent complex high-order associations and have been used in a variety of downstream graph tasks, visual question answering (VQA), image classification, recommendation systems, and document summarization. Specifically, HGNN is the first work to design a hypergraph convolutional network and introduce graph convolution to hypergraphs. It adopts an information aggregation paradigm, first aggregating node features to update hyperedge features, and then updating node features again. Its theoretical basis can be found in the literature [Feng, Yifan, et al. "Hypergraph neural networks." Proceedings of the AAAI conference on artificial intelligence. Vol. 33. No. 01. 2019.]. DHGNN proposes a dynamic hypergraph neural network consisting of two modules: hypergraph construction and hypergraph convolution. Its theoretical basis can be found in the literature [Jiang, Jianwen, et al. "DynamicHypergraph Neural Networks." IJCAI. 2019.].

[0027] Transformer: It was originally proposed by Vaswani et al. in 2017 for natural language processing tasks such as machine translation. Its theoretical basis can be found in the literature [Vaswani, Ashish, et al. "Attention is all you need." Advances in neural information processing systems 30 (2017).]. It introduces a self-attention mechanism, replacing the recurrent neural network (RNN) and convolutional neural network (CNN) previously used on sequence data. Transformer has been widely used in various fields. The success of Transformer is attributed to its ability to process long sequences, parallel computing, and high scalability. Many variants and extended Transformer architectures have also emerged to meet the needs of different application fields. DETAILED DESCRIPTION

[0028] The present invention is further described in conjunction with the accompanying drawings.

[0029] Example 1

[0030] The goal of this invention is to solve the existing technical problems and propose a framework for a multimodal social media popularity prediction method based on time-aware hypergraph learning, such as Figure 1 As shown in Figure 1. First, in order to maintain the temporal correlation between different UGCs, we adopt a time-centric approach to retrieve the top K instances related to the target UGC. Then, by leveraging UGC attributes, we connect all related instances together to form a hypergraph of the target UGC. Second, we develop a time-aware guided hypergraph Transformer, as shown in Figure 1. Figure 2 As shown in Figure 2, we aim to capture the correlations between intra- and cross-modal contexts, and use a temporal encoder to embed temporal information into the information fusion process. This dual focus ensures a detailed and coordinated UGC representation that is critical for multimodal popularity prediction. Finally, we apply a model consisting of a two-layer feedforward neural network to predict the popularity of the target UGC.

[0031] Based on the above analysis, as shown in the attached Figure 2 As shown, the multimodal social media popularity prediction method based on time-aware hypergraph learning provided in this embodiment includes the following steps:

[0032] S1 Through the hypergraph neural network, we can embed user-generated content (UGC) in the network, fuse the temporal information with the multimodal embedding information, and obtain the hypergraph representation embedding of UGC. This embedding vector contains information about the temporal relationship and multimodal representation.

[0033] S2By using Transformer, we can model the temporal information and multimodal information in the hypergraph to fuse the temporal information and cross-modal correlation, and finally obtain the embedding vector representation of each UGC. In multimodal social media popularity prediction (MSMPP), the main challenge is how to effectively integrate temporal information and model internal and cross-modal correlation. We designed a time-aware hypergraph Transformer, which integrates a temporal encoder to incorporate temporal information into the multimodal fusion process of the hypergraph Transformer. For each UGC and its surrounding hyperedges and the features of each node, we obtain expressive representations through our designed hypergraph Transformer. This ensures that each UGC has a comprehensive information embedding.

[0034] S3 Based on the embedded representations of UGC, we can make popularity predictions. These embedded representations contain rich information, including temporal correlation and the integration of multimodal information, allowing us to more accurately predict the popularity of social media content. These embedded representations can be input into a properly designed model, such as a feedforward neural network, to predict the popularity of each UGC. This approach is expected to improve the accuracy and efficiency of multimodal social media popularity prediction.

[0035] The above steps S1-S3 are explained in detail below.

[0036] In the above step S1, for a given target and release time t, we adopt a time-centric approach to retrieve the top K instances related to the target UGC in the UGC sequence S before time t and construct a temporal context sequence P t . Temporal context sequence P t It can be represented by a sequence of triplets: Here K represents the instance count.

[0037] After obtaining the temporal context sequence P t Then, we transform these instances into target instances Adaptive Hypergraph Each data instance forms a node, namely Target instance The attributes of instances (including posting users, categories, topics, etc.) are used to construct hyperedges that represent the relationships between instances.

[0038] The time encoder (T-encoder) is used to model continuous time information and map the scalar timestamp to a d T dimensional vector space. This process can be summarized as: in is a learnable parameter. This process encodes the time information into a d T dimensional vector that includes a combination of sine and cosine functions to capture periodicity and patterns in time.

[0039] In step S2, the Hypergraph Transformer learns the intra- and cross-modal correlations through a multimodal mixing process, including node-to-hyperedge and hyperedge-to-node. For node-to-hyperedge, we extract visual (text) information from the visual (text) hypergraph to reduce modality heterogeneity. The process from node to k-th hyperedge can be defined as follows:

[0040] For visual modality:

[0041]

[0042] For text modal:

[0043]

[0044] Where v and t represent the visual and textual modalities, respectively. Here represents the feature of the kth hyperedge, is the number of attention heads. Represent the query, key, and value vectors of node i and hyperedge k, respectively, and are calculated through linear transformation and slicing as follows:

[0045]

[0046] here, is the embedding matrix of k hyperedges, Indicates all The key and value transformation of each header, p / N1 and p / Indicates the start and end index of the hth slice.

[0047] In order to mine cross-modal correlations and reduce the impact of modal heterogeneity, we introduce a gating mechanism to control the flow of information from one modality to another. The cross-modal interaction process can be summarized as follows:

[0048] For the information flow from vision to text, it is expressed as

[0049]

[0050] Specifically, the gating process can be summarized as follows:

[0051]

[0052] In this process, λ is a gating parameter used to control the information flow between two different modalities. This mechanism helps to balance the influence of different modalities to reduce the impact of modality heterogeneity and better mine cross-modal correlations.

[0053] In order to integrate the hyperedge k To represent a specific header, we use the concatenation operation as follows:

[0054] For text mode, it is represented as

[0055]

[0056] Similarly, for the visual modality, you can replace To obtain

[0057] In the process of hyperedge to node, we propagate information from hyperedge to node through a similar but opposite process. Eventually, we will obtain the representation of node i and This process helps to integrate the information of hyperedges at the node level to obtain the final representation of the node.

[0058] The popularity prediction in the final step is done by a two-layer feed-forward neural network as shown below:

[0059]

[0060] in, and are the parameters of the prediction network, where ⊕ represents the connection operation.

[0061] In addition, we use the mean squared error (MSE) as the optimization loss to train the parameters of the model so that the predicted results are as close as possible to the actual popularity. The goal of this model is to minimize the mean squared error between the predicted values ​​and the actual values ​​to improve the accuracy of popularity prediction.

[0062] Application Examples

[0063] The multimodal social media popularity prediction method based on time-aware hypergraph learning provided by the embodiment is used in two different real data sets (SMPD and ICIP, the first data set is from reference [Bo Wu, Wen-Huang Cheng, Peiye Liu, Bei Liu, Zhaoyang Zeng, and Jiebo Luo. 2019. SMP Challenge: An Overview of Social Media Prediction Challenge 2019. In ACM MM.], and the second data set is from reference [Alessandro Ortis, Giovanni Maria Farinella, and Sebastiano Battiato. 2019. Predic-

[0064] tion of social image popularity dynamics.In Image Analysis andProcessing–ICIAP

[0065] 2019:20th International Conference,Trento,Italy,September 9–13,2019,Proceedings,Part II 20.Springer,572–582]. In two of the datasets, the ratio of sample data in the training set to the test set is 8:2.

[0066] At the same time, the multimodal social media popularity prediction method based on time-aware hypergraph learning provided by the present invention is compared with 5 different baseline models (SVR, Hyfea, DTCN, UHAN, MHF), and MSE, MAE, and SRC are used as our evaluation indicators (the smaller the MSE and MAE values, the better the prediction effect, and the larger the SRC value, the better the prediction effect). The prediction results are shown in Table 1.

[0067] Table 1: The effect of popularity prediction on the application example dataset

[0068]

[0069]

[0070] The remaining methods in the table are described as follows:

[0071] SVR: Use support vector regression for popularity prediction, which only considers the visual information of UGC. For our task, we also further incorporate text features into the model through TF-IDF and word embedding (i.e. GloVe). In addition, we tried different combinations of feature types and recorded the best results. [Aditya Khosla, Atish Das Sarma, and Raffay Hamid. 2014. What makes an image popular? In WWW. 867–876.]

[0072] ·Hyfea: Hyfea is a feature engineering model that designs handcrafted features and then selects a model with good performance. Hyfea uses multimodal features such as images, categories, spatiotemporal information, user profiles, and tags to predict popularity based on CatBoost.

Xin Lai, Yihong Zhang, and Wei Zhang. 2020. Hyfea: winning solution to socialmedia popularity prediction for multimedia grand challenge 2020. In ACM MM. 4565–4569.

[0073] DTCN: focuses on sequential popularity prediction and integrates visual features and user attributes. It considers the neighboring temporal context and periodic temporal context in social media popularity prediction through the temporal attention mechanism.

Bo Wu, Wen-Huang Cheng, Yongdong Zhang, Qiushi Huang, Jintao Li, and Tao Mei. 2017. Sequential prediction of social media popularity with deep temporal context networks. arXiv preprint arXiv:1712.04443(2017).

[0074] UHAN: It designs a user-guided hierarchical attention network to merge the textual and visual features of UGC for popularity prediction through the guidance of user embeddings. [Wei Zhang, Wen Wang, Jun Wang, and Hongyuan Zha. 2018. User-guided hierarchical attention network for multi-modal social image popularity prediction. In WWW. 1277–1286.]

[0075] ·MHF: A state-of-the-art model for social media popularity prediction. It builds a hierarchical fusion framework for learning multimodal features (i.e., images and text and their related attributes) for image popularity prediction. MHF uses a residual network to extract image features and generate image-attribute features. Then, it obtains text features by merging image-attribute features, GloVe embeddings, and the output features of the LSTM model. Finally, MHF merges the above features into a feature vector for popularity prediction.

Jing Wang, Shuo Yang, Hui Zhao, and Yue Yang. 2023. Social media popularity prediction with multimodal hierarchical fusion model. Computer Speech & Language 80 (2023), 101490.

[0076] It can be seen from the experimental results in Table 1 that the multimodal social media popularity prediction method based on time-aware hypergraph learning provided by the present invention can significantly improve the accuracy of popularity prediction compared with other baseline models. The r model we proposed shows significant improvement in multimodal social media popularity prediction (MSMPP) performance compared with the baseline model. In addition, these results also verify our motivation to introduce neighborhood knowledge in UGC reasoning. The improvement in model performance can be attributed to the fact that we effectively capture the correlation within and between modalities through the hypergraph Transformer method. This shows that our model can better integrate information when processing multimodal social media data, thereby improving the accuracy of popularity prediction.

[0077] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A multimodal social media popularity prediction method based on time-aware hypergraph learning, characterized in that: The following steps are involved: S1 uses a hypergraph neural network to perform network embedding on UGC according to the release time, and obtains the hypergraph representation embedding of UGC, that is, the embedding vector of user-generated content containing temporal relationship and multimodal embedding representation information; S2 uses Transformer to model the temporal information and multimodal information in the hypergraph to integrate temporal information and the correlation between modalities to obtain the final embedding vector representation of each UGC; designs a time-aware hypergraph Transformer that integrates a temporal encoder to inject temporal information into the multimodal mixing process of the hypergraph Transformer; for each UGC and its surrounding hyperedges, as well as each feature of its corresponding node, an expressive representation is obtained through the designed hypergraph Transformer; S3 predicts popularity based on the embedded representation of UGC.

2. The multimodal social media popularity prediction method based on time-aware hypergraph learning according to claim 1 is characterized in that: Step S1 includes the following sub-steps: S11 Given target UGCC t and release time t, retrieve the top K instances related to it in the UGC sequence S before time t from the perspective of the time center, and construct a time context sequence P t ;P t It can be represented by a ternary sequence: Where K represents the instance count; S12 in obtaining P t After that, the obtained instance set P t Convert to target instance Adaptive Hypergraph Each data instance forms a node, namely The attributes of the instances (i.e., posting user, category, topic) construct hyperedges that represent the relationships between instances.

3. The multimodal social media popularity prediction method based on time-aware hypergraph learning according to claim 1 is characterized in that: Step S2 includes the following sub-steps: The S21 temporal encoder (T-encoder) (Xu et al. 2019) is used to model continuous temporal information, mapping scalar timestamps to d T dimensional vector space; this process can be summarized as: in is a learnable parameter; The S22 hypergraph transformer learns internal and cross-modal correlations through a multimodal mixing process, including node-to-hyperedge and hyperedge-to-node. For node-to-hyperedge, visual (text) information is extracted from the visual (text) hypergraph to reduce modal heterogeneity. The process from node to k-th hyperedge is defined as: Where v and t represent visual and textual modalities, respectively; represents the characteristics of the kth hyperedge; Here H is the number of attention heads; Represent the query, key, and value vectors of node i and hyperedge k, respectively, and are calculated by linear transformation and slicing: is the embedding matrix of K hyperedges; Indicates all The key and value transformation of each header; h-1 and p h Indicates the start and end index of the hth slice; S23 mines cross-modal correlations and reduces the impact of modal heterogeneity. A gating mechanism is implemented to control the flow of information from one modality to another. The cross-modal interaction process can be summarized as follows: Specifically, the gating process can be summarized as: S24 uses a concatenation operation to fuse the H head-specific representations of hyperedge k: Similarly, Can be replaced by To obtain In the process from hyperedge to node, information is propagated from hyperedge to node through a similar but opposite process; eventually, the representation of node i is obtained and 4. The multimodal social media popularity prediction method based on time-aware hypergraph learning according to claim 1 is characterized in that: In step S3, the final popularity prediction is completed by a two-layer feedforward neural network: in, and are the parameters of the prediction network; It is a connection operation.

5. The multimodal social media popularity prediction method based on time-aware hypergraph learning according to claim 1 is characterized in that: The mean squared error (MSE) is used as the optimization loss to train the parameters of the model.

Images