A time-aware hypergraph learning based method for predicting popularity of multi-modal social media streams
Through the time-aware hypergraph learning method, combined with the hypergraph neural network and Transformer model, the problems of the relationship and time dependency between UGC in multimodal social media popularity prediction are solved, and more accurate popularity prediction is achieved.
Patent Information
- Application Number
- CN202411628881.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2023-11-14
- Filing Date
- 2024-11-14
- Publication Date
- 2025-10-14
- Estimated Expiration
- 2044-11-14
AI Technical Summary
Existing multimodal social media popularity prediction methods fail to effectively consider the relationships between different UGCs and ignore the temporal dependencies and auxiliary information between UGCs, resulting in insufficient prediction accuracy.
A time-aware hypergraph learning method is adopted to embed UGC through a hypergraph neural network. The Transformer model is used to integrate temporal information and multimodal correlation. A time-aware hypergraph Transformer is designed to capture internal and cross-modal correlations, and popularity prediction is performed through a feedforward neural network.
The accuracy and efficiency of multimodal social media popularity prediction are improved, which can better integrate time and modality information and enhance the granularity and alignment of prediction.
Smart Images
Figure CN119692524B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of information dissemination, and relates to multi-modal social media popularity prediction, in particular to multi-modal learning and popularity prediction in deep learning, which is a hypergraph representation and time-aware method. BACKGROUND
[0002] With the widespread dissemination of multi-modal user-generated content (UGC), such as videos, images, and texts, on online social media platforms, how to understand, model, and recommend UGC has become an important but challenging problem. Multi-modal social media popularity prediction (MSMPP) aims to infer the future number of interactions between users and UGC by learning and aggregating multi-modal content. This is very beneficial to help users filter information and improve various applications such as recommendation, rumor detection, etc. Existing MSMPP research can be summarized into two categories: (1) feature-based methods emphasize the design and integration of hand-crafted UGC features; (2) deep learning-based methods utilize end-to-end frameworks to capture more comprehensive multi-modal representations. Although they have achieved success, the availability of current work is limited by the following aspects: (1) existing work independently processes individual UGC to learn its representation, without considering the inter-relationship between different UGC. Auxiliary information existing in related UGC is also ignored to help the reasoning of UGC. For example, different UGC posted by the same user can interact with similar user groups and produce similar UGC popularity. (2) existing work ignores the temporal dependency between UGC. SUMMARY
[0003] The purpose of the present application is to propose a framework of multi-modal social media popularity prediction method based on time-aware hypergraph learning for the above technical problems existing in the prior art. First, in order to preserve the temporal correlation between different UGC, we retrieve the top K instances related to the target UGC from a time-centric perspective. Subsequently, using UGC attributes, we connect all related instances to construct the hypergraph of the target UGC. Second, we design a time-aware hypergraph Transformer to capture the correlation within and across modalities, while using a temporal encoder to inject temporal information into the information mixing process. This dual focus ensures a fine-grained and aligned UGC representation that is crucial for multi-modal popularity prediction. Finally, we use a 2-layer feedforward neural network to predict the popularity of the target UGC.
[0004] Based on the above inventive ideas, the present invention provides a multimodal social media popularity prediction method based on time-aware hypergraph learning, which includes the following steps:
[0005] S1 uses a hypergraph neural network to perform network embedding on UGC according to the release time, and obtains the hypergraph representation embedding of UGC, that is, the embedding vector of user-generated content that contains temporal relationship and multimodal embedding representation information.
[0006] S2 uses Transformer to model the temporal and multimodal information in the hypergraph to integrate temporal information and the correlation between modalities to obtain the final embedding vector representation of each UGC. For MSMPP, the main challenge lies in how to jointly integrate temporal information and model internal and cross-modal correlations. A time-aware hypergraph transformer is designed, which integrates a temporal encoder to inject temporal information into the multimodal mixing process of the hypergraph transformer. For each UGC and its surrounding hyperedges, as well as each feature of its corresponding node, we obtain expressive representations through our designed hypergraph transformer.
[0007] S3 predicts popularity based on the embedded representation of UGC.
[0008] The above step S1 includes the following sub-steps:
[0009] S11 given target and release time t, we retrieve the top K instances related to it in the UGC sequence S before time t from the perspective of the time center and construct a time context sequence P t .P t It can be represented by a ternary sequence: Where K represents the instance count.
[0010] S12 in obtaining P t After that, the obtained instance set P t Convert to target instance Adaptive Hypergraph Each data instance forms a node, i.e. The attributes of the instances (i.e., posting users, categories, topics) construct hyperedges that represent the relationships between instances.
[0011] The above step S2 includes the following sub-steps:
[0012] The S21 temporal encoder (T-encoder) (Xu et al. 2019) is used to model continuous temporal information, mapping scalar timestamps to d T dimensional vector space. This process can be summarized as follows: where are learnable parameters.
[0013] S22 The S22 Hypergraph Transformer learns intra- and cross-modal correlations through a multi-modal mixing process, including node-to-hyperedge and hyperedge-to-node. For node-to-hyperedge, we extract visual (textual) information from the visual (textual) hypergraph to reduce modality heterogeneity. The node-to-kth hyperedge process is defined as: where v and t represent the visual and textual modalities, respectively. denotes the feature of the kth hyperedge. Here H is the number of attention heads. denote the query, key, and value vectors of node i and hyperedge k, respectively, which are computed by linear transformations and slicing: is the embedding matrix of K hyperedges. denotes the key and value transformations of all heads. p / N1 and p / denote the start and end indices of the hth slice.
[0014] S23 To mine cross-modal correlations and reduce the impact of modality heterogeneity, we implement a gating mechanism to control the flow of information from one modality to another. The cross-modal interaction process can be summarized as follows: Specifically, the gating process can be summarized as:
[0015]
[0016] S24 The H head-specific representations of hyperedge k are fused using a concatenation operation: Similarly, can be replaced with to obtain In the hyperedge-to-node process, we propagate information from hyperedges to nodes through a similar but reversed process. Finally, we will obtain the representation of node i and
[0017] S3 The final popularity prediction is done by a two-layer feedforward neural network:
[0018]
[0019] where and are the parameters of the prediction network. is the concatenation operation. In addition, we use the mean squared error (MSE) as the optimization loss to train the parameters of the model.
[0020] Compared with the prior art, the present application has the following beneficial effects:
[0021] 1、 The present application explores the modeling of the time relationship of target user generated content (UGC), and enhances the representation of the target through multi-modal hypergraph aggregation, aiming to lead adaptive hypergraph construction by aggregating time-aware information, to assist in multi-modal popularity prediction.
[0022] 2、 The present application proposes a guided hypergraph Transformer for expanding information aggregation to multi-modal information mixing. We designed intra-modal propagation and inter-modal propagation to capture intra-modal and inter-modal correlations respectively, to obtain fine-grained aligned user generated content (UGC) representation. BRIEF DESCRIPTION OF DRAWINGS
[0023] Figure 1 Flowchart for multi-modal social media popularity prediction based on time-aware hypergraph learning.
[0024] Figure 2 Guided hypergraph transformer schematic diagram.
[0025] TERMS EXPLANATION
[0026] Hypergraph (hypergraph): Hypergraph extends the traditional graph structure, allowing us to connect three or more nodes through hyperedges. Hypergraph provides a natural way to represent complex high-order correlations and is used in various downstream graph tasks, visual question answering (VQA), image classification, recommendation systems and document summarization, etc. Specifically, HGNN is the first work to design hypergraph convolutional networks and introduce graph convolution to hypergraph. It adopts the information aggregation paradigm, which first aggregates node features to update hyperedge features, and then updates node features again. Its theoretical basis can be referred to the literature
Feng, Yifan, et al. "Hypergraph neural networks." Proceedings of the AAAI conference on artificial intelligence. Vol. 33. No. 01. 2019.
Jiang, Jianwen, et al. "Dynamic Hypergraph Neural Networks." IJCAI. 2019.
[0027] Transformer: Initially proposed by Vaswani et al. in 2017 for natural language processing tasks such as machine translation, its theoretical foundation can be referred to in the literature
Vaswani, Ashish, et al. "Attention is all you need." Advances in neural information processing systems 30 (2017).
[0028] The present application is further described in conjunction with the accompanying drawings.
[0029] Embodiment 1
[0030] The present application aims to solve the existing technical problems and proposes a framework for predicting the popularity of multi-modal social media based on time-aware hypergraph learning, as shown in Figure 1 First, in order to maintain the time correlation between different UGCs, we use a time-centered approach to retrieve the top K instances related to the target UGC. Subsequently, by utilizing UGC attributes, we connect all related instances together to form a hypergraph of the target UGC. Second, we develop a time-aware guided hypergraph Transformer, as shown in Figure 2 , aiming to capture the correlation within and across modalities, and use a time encoder to embed time information into the information fusion process. This dual attention ensures a detailed and coordinated UGC representation that is crucial for multi-modal popularity prediction. Finally, we use a model containing two layers of feedforward neural networks to predict the popularity of the target UGC.
[0031] Based on the above analysis, as shown in the accompanying Figure 2 , the multi-modal social media popularity prediction method based on time-aware hypergraph learning provided in this embodiment includes the following steps:
[0032] S1 Through the hypergraph neural network, we can perform network embedding on user-generated content (UGC), fuse time information with multi-modal embedding information, and obtain a hypergraph representation embedding of UGC. This embedding vector contains information about time relationships and multi-modal representation.
[0033] S2 By using Transformer, we can model the temporal information and multimodal information in the hypergraph to fuse temporal information and cross-modal correlation, and finally obtain an embedded vector representation for each UGC. In multimodal social media popularity prediction (MSMPP), the main challenge lies in how to effectively integrate temporal information and model internal and cross-modal correlation. We designed a time-aware hypergraph Transformer, which integrates a temporal encoder to incorporate temporal information into the multimodal fusion process of the hypergraph Transformer. For each UGC and its surrounding hyperedges, as well as the features of each node, we obtain expressive representations through our designed hypergraph Transformer. This ensures that each UGC has a comprehensive information embedding.
[0034] S3 uses embedded representations of user-generated content (UGC) to predict its popularity. These embeddings contain rich information, including temporal correlation and the integration of multimodal information, enabling more accurate prediction of the popularity of social media content. These embeddings can be input into a suitably designed model, such as a feedforward neural network, to predict the popularity of each piece of user-generated content. This approach is expected to improve the accuracy and efficiency of multimodal social media popularity prediction.
[0035] The above steps S1-S3 are explained in detail below.
[0036] In the above step S1, for a given target and release time t, we adopt a time-centric approach to retrieve the top K instances related to the target UGC in the UGC sequence S before time t and construct a time context sequence P t . Temporal context sequence P t It can be represented by a sequence of triples: Here K represents the instance count.
[0037] After obtaining the temporal context sequence P t Then, we convert these instances into target instances Adaptive Hypergraph Each data instance forms a node, i.e. Target instance The attributes of instances (including posting users, categories, topics, etc.) are used to construct hyperedges that represent the relationships between instances.
[0038] The time encoder (T-encoder) is used to model continuous time information and map the scalar timestamp to a d T dimensional vector space. This process can be summarized as: in is a learnable parameter. This process encodes the time information into a d T dimensional vectors that include combinations of sine and cosine functions to capture periodicity and patterns in time.
[0039] In step S2, the Hypergraph Transformer learns the internal and cross-modal correlations through a multimodal mixing process, including node-to-hyperedge and hyperedge-to-node. For node-to-hyperedge, we extract visual (text) information from the visual (text) hypergraph to reduce modality heterogeneity. The process from node to k-th hyperedge can be defined as follows:
[0040] For visual modality:
[0041]
[0042] For text modal:
[0043]
[0044] Where v and t represent visual and textual modalities respectively. represents the feature of the kth hyperedge, is the number of attention heads. Denote the query, key, and value vectors of node i and hyperedge k, respectively, and are calculated by linear transformation and slicing as follows:
[0045]
[0046] here, is the embedding matrix of k hyperedges, Indicates all The key and value transformation of each header, p / N1 and p / Indicates the start and end index of the hth slice.
[0047] In order to exploit cross-modal correlations and reduce the impact of modal heterogeneity, we introduce a gating mechanism to control the flow of information from one modality to another. The cross-modal interaction process can be summarized as follows:
[0048] For the information flow from vision to text, it is expressed as
[0049]
[0050] Specifically, the gating process can be summarized as follows:
[0051]
[0052] In this process, λ is a gating parameter used to control the information flow between two different modalities. This mechanism helps balance the influence of different modalities to reduce the impact of modality heterogeneity and better mine cross-modal correlations.
[0053] In order to integrate the hyperedge k To get a specific representation of a header, we use the concatenation operation as follows:
[0054] For text mode, it is represented as
[0055]
[0056] Similarly, for the visual modality, we can replace To obtain
[0057] In the process of hyperedge to node, we propagate information from hyperedge to node through a similar but reverse process. Eventually, we will obtain the representation of node i and This process helps to integrate the information of hyperedges at the node level to obtain the final representation of the node.
[0058] The popularity prediction in the final step is done by a two-layer feed-forward neural network as shown below:
[0059]
[0060] in, and are the parameters of the prediction network, where ⊕ represents the connection operation.
[0061] In addition, we use the mean squared error (MSE) as the optimization loss to train the model parameters so that the predicted results are as close as possible to the actual popularity. The goal of this model is to minimize the mean squared error between the predicted values and the actual values to improve the accuracy of popularity prediction.
[0062] Application Examples
[0063] The prediction effect of the time-aware hypergraph learning-based multimodal social media popularity prediction method provided by the embodiment is applied to two different real data sets (SMPD and ICIP, the first data set is from reference document
Bo Wu, Wen-Huang Cheng, Peiye Liu, Bei Liu, Zhaoyang Zeng, and Jiebo Luo. 2019. SMP Challenge: An Overview of Social Media Prediction Challenge 2019. In ACM MM.
Alessandro Ortis, Giovanni Maria Farinella, and Sebastiano Battiato. 2019. Prediction of social image popularity dynamics. In Image Analysis and Processing–ICIAP 2019: 20th International Conference, Trento, Italy, September 9–13, 2019, Proceedings, Part II 20. Springer, 572–582
[0064]
[0065] The ratio of sample data in the training set to the test set in the two data sets is 8:2.
[0066] Meanwhile, the time-aware hypergraph learning-based multimodal social media popularity prediction method provided by the embodiment is compared with five different baseline models (SVR, Hyfea, DTCN, UHAN and MHF), and MSE, MAE and SRC are used as evaluation indexes (the smaller the values of MSE and MAE, the better the prediction effect, and the larger the value of SRC, the better the prediction effect), and the prediction results are shown in Table 1.
[0067] Table 1: Effect of popularity prediction on application example data set
[0068]
[0069]
[0070] The remaining methods in the table are introduced as follows:
[0071] SVR: Uses support vector regression for popularity prediction, which only considers the visual information of UGC. For our task, we further incorporate text features into the model through TF-IDF and word embeddings (i.e., GloVe). In addition, we try different combinations of feature types and record the best results. [Aditya Khosla, Atish Das Sarma, and Raffay Hamid. 2014. What makes an image popular? In WWW. 867–876.]
[0072] Hyfea: Hyfea is a feature engineering model that designs handcrafted features and then selects a well-performing model. Hyfea uses multimodal features such as images, categories, spatiotemporal information, user profiles, and tags to perform popularity prediction based on CatBoost. [Xin Lai, Yihong Zhang, and Wei Zhang. 2020. Hyfea: winning solution to social media popularity prediction for multimedia grandchallenge 2020. In ACM MM. 4565–4569.]
[0073] DTCN: Focuses on sequential popularity prediction and integrates visual features and user attributes. It considers both neighboring and periodic temporal context in social media popularity prediction through a temporal attention mechanism. [Bo Wu, Wen-Huang Cheng, Yongdong Zhang, Qiushi Huang, Jintao Li, and Tao Mei. 2017. Sequential prediction of social media popularity with deep temporal context networks. arXiv preprint arXiv:1712.04443(2017).]
[0074] • UHAN: It designs a user-guided hierarchical attention network that combines UGC's textual and visual features together through user-embedded guidance for popularity prediction.
Wei Zhang, Wen Wang, Jun Wang, and Hongyuan Zha. 2018. User-guided hierarchical attention network for multi-modal social image popularity prediction. In WWW. 1277-1286.
[0075] • MHF: The latest model for social media popularity prediction. It builds a hierarchical fusion framework for learning multi-modal features (i.e., images and texts and their related attributes) for image popularity prediction. MHF uses a residual network to extract image features and generate image-attribute features. Then, it obtains text features by merging image-attribute features, GloVe embeddings, and the output features of an LSTM model. Finally, MHF merges the above features into a feature vector for popularity prediction.
Jing Wang, Shuo Yang, Hui Zhao, and Yue Yang. 2023. Social media popularity prediction with multimodal hierarchical fusion model. Computer Speech & Language 80 (2023), 101490.
[0076] From the experimental results in Table 1, it can be seen that the multi-modal social media popularity prediction method based on time-aware hypergraph learning provided by the present application can greatly improve the accuracy of popularity prediction compared to other baseline models. The r model we proposed shows significant improvement in the performance of multi-modal social media popularity prediction (MSMPP) compared to the baseline models. In addition, these results also verify our motivation for introducing neighborhood knowledge in UGC reasoning. The improvement in model performance can be attributed to our effective capture of intra-modal and inter-modal correlations through the hypergraph Transformer method. This indicates that our model can better integrate information when processing multi-modal social media data, thereby improving the accuracy of popularity prediction.
[0077] Those skilled in the art will appreciate that the embodiments described herein are presented for purposes of illustration and that the inventive principles are not limited to these particular embodiments. Other variations and modifications can be made to the embodiments without departing from the spirit and scope of the inventive principles.
Claims
1. A multimodal social media popularity prediction method based on time-aware hypergraph learning, characterized by: The following steps are involved: S1 uses a hypergraph neural network to perform network embedding on UGC based on the publishing time, and obtains the hypergraph representation embedding of UGC, that is, the embedding vector of user-generated content that contains temporal relationship and multimodal embedding representation information; S2 uses Transformer to model temporal and multimodal information in the hypergraph, integrating temporal information and the correlation between modalities to obtain the final embedding vector representation of each UGC. A time-aware hypergraph transformer is designed that integrates a temporal encoder to inject temporal information into the multimodal mixing process of the hypergraph transformer. For each UGC, its surrounding hyperedges, and each feature of its corresponding node, the designed hypergraph transformer obtains an expressive representation. This includes the following sub-steps: The S21 time encoder (T-encoder) is used to model continuous time information and map scalar timestamps to d T dimensional vector space; this process can be summarized as: in is a learnable parameter; The S22 hypergraph transformer learns internal and cross-modal correlations through a multimodal mixing process, including node-to-hyperedge and hyperedge-to-node. For node-to-hyperedge, visual-textual information is extracted from the visual-textual hypergraph to reduce modal heterogeneity. The process from node to k-th hyperedge is defined as: Where v and t represent visual and textual modalities respectively; represents the feature of the kth hyperedge; Here H is the number of attention heads; Denote the query, key, and value vectors of node i and hyperedge k, respectively, and are computed by linear transformation and slicing: is the embedding matrix of K hyperedges; Indicates all The key and value transformation of each header; h-1 and p h Indicates the start and end index of the hth slice; S23 exploits cross-modal correlations and reduces the impact of modal heterogeneity. A gating mechanism is implemented to control the flow of information from one modality to another. The cross-modal interaction process can be summarized as follows: Specifically, the gating process can be summarized as: S24 uses a concatenation operation to fuse the H head-specific representations of hyperedge k: Similarly, Can be replaced by To obtain In the process of hyperedge to node, information is propagated from hyperedge to node through a similar but opposite process; eventually, the representation of node i is obtained and S3 predicts popularity based on the embedded representation of UGC.
2. The multimodal social media popularity prediction method based on time-aware hypergraph learning according to claim 1 is characterized in that: Step S1 includes the following sub-steps: S11 given target and release time t, retrieve the top K instances related to it in the UGC sequence S before time t from the perspective of the time center, and construct a time context sequence P t ;P t It can be represented by a ternary sequence: Where K represents the instance count; S12 in obtaining P t After that, the obtained instance set P t Convert to target instance Adaptive Hypergraph Each data instance forms a node, i.e. The attributes of construct hyperedges that represent the relationships between instances.
3. The multimodal social media popularity prediction method based on time-aware hypergraph learning according to claim 1 is characterized in that: In step S3, the final popularity prediction is completed by a two-layer feedforward neural network: in, and are the parameters of the prediction network; It is a join operation.
4. The multimodal social media popularity prediction method based on time-aware hypergraph learning according to claim 1 is characterized in that: The mean squared error (MSE) is used as the optimization loss to train the model parameters.