A Micro-video Popularity Prediction Method Based on Graph Hint Enhancement

Abstract

This invention discloses a micro-video popularity prediction method based on graph cue enhancement, belonging to the field of machine learning technology. It primarily utilizes a generative retrieval system (PVL) driven by a large multimodal model to create context-rich retrieval queries and execute retrievals, forming a dynamic micro-video support set that effectively reflects evolutionary patterns. Based on this, a graph-based cue generator generates evolutionary cues by capturing the temporal and relational structure of the support set. These cues represent the latest trend dynamics and can serve as a few examples to guide pre-trained visual language models. By integrating these evolutionary cues, PVLs can more accurately simulate changing popularity trends, thereby producing stronger and more predictive representations, thus enhancing popularity prediction.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and more specifically to a method for predicting the popularity of micro-videos based on graph cues. Background Technology

[0002] The rapid rise of micro-video sharing platforms such as TikTok, YouTube, and Kuaishou has transformed the way content is created and consumed, attracting widespread attention from academia and industry. This shift has made Micro-Video Popularity Prediction (MVPP) a valuable research problem with applications in personalized recommendation, trend analysis, and platform optimization. MVPP is a research problem in the multimodal field, and this invention primarily focuses on prediction using the MicroLens micro-video dataset. Currently, methods developed for MVPP can be broadly categorized into two types:

[0003] (1) Manual methods: popularity prediction relies on manual feature extraction. The feasibility of these methods depends on objective expert knowledge and representative features.

[0004] (2) PVL-based methods: With the rise of pre-trained visual language models (PVLs), researchers have turned to using these models to simulate higher-order cross-modal correlations to enhance MVPP.

[0005] Despite the progress made in current research, they have overlooked the dynamic and evolving nature of micro-videos. This dynamic characteristic poses a significant challenge to existing PVL-based methods, which rely primarily on pre-trained, static knowledge stored in pre-trained models, making them ill-suited to emerging and rapidly changing trends.

[0006] Therefore, it is necessary to propose a micro-video popularity prediction method based on graph prompt enhancement to solve the above problems. Summary of the Invention

[0007] The purpose of this invention is to solve the problems of inaccurate popularity prediction and increased prediction costs caused by frequent data updates due to real-time dynamic changes in micro-video platform data.

[0008] To achieve the above objectives, the present invention specifically adopts the following technical solution:

[0009] A method for predicting the popularity of micro-videos based on graph cues includes the following steps:

[0010] S1 represents the collection of micro-videos from an online video sharing platform as... Each micro-video content Defined as ,in, This refers to the content of the micro-video. This indicates its textual description, and This indicates its popularity score;

[0011] S2, Design a prompt template Utilizing micro-video content To guide LMM Generate comprehensive text descriptions for micro-videos ;

[0012] ;

[0013] S3, using a pre-trained text encoder to generate text descriptions Mapping to text embedding Then embed the text. Used as a retrieval vector, the first k micro-videos are retrieved, forming the support set for the i-th micro-video. The final set of support sets for the N micro-videos is represented as... ;

[0014] S4 uses a graph-based cue generation module that leverages a graph attention network to adjust frozen PVLs with cueing to reflect evolving information. Each target micro-video is treated as a core node in the graph attention network, while retrieved instances serve as neighboring nodes of the core nodes.

[0015] S5, using a graph attention network to derive text prompts. and visual cues Global suggestions;

[0016] S6, using text prompts and visual cues To adjust the frozen PVLs, fine-tune the PVLs, and obtain the text features and visual features of each layer of the PVLs.

[0017] S7, define the text features and visual features of the last layer of PVLs as follows: and Text features and visual features The features are concatenated and fed into a transformer-based predictor to generate a popularity score. ;

[0018] ;

[0019] Where: [,] indicates concatenation.

[0020] Furthermore, the retrieval based on cosine similarity in S3 follows the mechanism as follows:

[0021] ;

[0022] in: It is a database supporting micro-video sets, which retrieves the top k micro-video content. and text description .

[0023] Furthermore, in step S4, for the text portion, a pre-trained text encoder is used to describe the target text. and retrieved text description Labeling and projecting into their respective embeddings are defined as follows:

[0024] ;

[0025] in: This represents a pre-trained text encoder. It is the embedded dimension. and The sequence lengths of the target text description and the retrieved text description, respectively;

[0026] Similarly, for the visual part, the definition is:

[0027] ;

[0028] in: This represents the pre-trained visual encoder.

[0029] Furthermore, in S5, self-attention is first performed between the representation of the target micro-video and the representation of the retrieved instance, defined as follows:

[0030] ;

[0031] This is used to calculate the attention coefficient:

[0032] ;

[0033] in: It is a learnable linear transformation;

[0034] They are normalized using the softmax function:

[0035] ;

[0036] Final text prompt for the target micro-video Calculate the weighted sum of the features of the retrieved instances:

[0037] ;

[0038] in: It is a non-linear activation function;

[0039] Similarly, applied to visual representation. and To obtain visual cues :

[0040] .

[0041] Furthermore, in S6, let and These represent the pre-trained language model and the visual model, respectively. Each of the first Transformer layer, text prompt To be inserted into the target text In the middle, visual cues Inserted into the target vision middle;

[0042] ;

[0043] ;

[0044] in: and Is it input to the number The hidden representations of the textual and visual modalities of the layer.

[0045] Furthermore, in S6 and S7, during the training process, the parameters in the PVLs are frozen, and the mean squared error loss (MSE) is used to optimize the parameters in the graph-based prompt generation module and predictor.

[0046] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0047] 1. This invention proposes EvoPrompt, a high-efficiency cue-enhanced micro-video popularity prediction framework. It utilizes a generative retrieval system driven by a large multimodal model to retrieve and construct a dynamic micro-video support set that effectively reflects evolutionary patterns. Then, a graph-based cue generator produces evolutionary cues by capturing the temporal and relational structure of the support set to guide PVLs in generating stronger and more predictive representations.

[0048] 2. This invention proposes an LMM-driven generative retrieval tool for micro-video retrieval. This retrieval tool utilizes knowledge stored in the LMM to generate descriptive text representations of micro-videos for retrieval queries, thereby improving the accuracy and relevance of the retrieval.

[0049] 3. This invention designs an evolution-aware cue mechanism for PVLs, which realizes real-time cue adjustment of PVLs by updating the micro-video support set. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the micro-video popularity prediction model based on graph cues enhancement according to the present invention. Detailed Implementation

[0051] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0052] MVPP is an abbreviation for Micro-Video Popularity Prediction, which refers to a class of tasks that automatically predict the future popularity (such as views, likes, etc.) of micro-videos on short video platforms. The task definition can be found in the reference [Jiayi Xie, et al. “A Multimodal Variational Encoder-Decoder Framework for Micro-video Popularity Prediction.” The Web Conference.2020.].

[0053] LMM is an abbreviation for Large Multimodal Model, which is a deep learning model architecture capable of simultaneously processing and understanding multiple modalities of data (such as images, text, and audio). Its implementation method can be found in the reference [Jiayang Wu, et al. “Multimodal large language models: A survey.” IEEE International Conference on Big Data. 2023.].

[0054] PVLs are an abbreviation for Pre-trained Vision-Language models, which are model frameworks that learn cross-modal representations and alignment capabilities between vision and language through joint pre-training on large-scale image and text data. The implementation method can be found in the reference [Yifan Du, et al. “A Survey of Vision-Language Pre-Trained Models.” International Joint Conference on Artificial Intelligence. 2022.].

[0055] GAT is an abbreviation for Graph Attention Network, a neural network method that introduces an attention mechanism to graph-structured data, dynamically assigning different attention weights based on the importance of neighboring nodes. Its implementation method can be found in the reference [Veličković, Petar, et al. "Graph Attention Networks." International Conference on Learning Representations. 2018.].

[0056] Transformer is short for Transformer Architecture, a deep learning model framework based on self-attention mechanisms, widely used in natural language processing and multimodal modeling tasks. Its implementation method can be found in the reference [Vaswani A., et al. “Attention Is All You Need,” Advances in Neural Information Processing Systems. 2017.].

[0057] Please see Figure 1 A method for predicting the popularity of micro-videos based on graph cues enhancement.

[0058] The main purpose of micro-video popularity prediction algorithms is to analyze the visual and textual modalities of micro-video datasets to determine the popularity of micro-videos, thereby assisting micro-video platforms in personalized recommendations, trend analysis, and platform optimization.

[0059] The framework of the micro-video popularity prediction model is as follows: Figure 1 As shown, it includes the following steps:

[0060] Data preprocessing. This invention was tested and evaluated on a dataset collected on a real-world online micro-video platform (MicroLens), with each data record formatted as (text, video, comment, publisher, popularity, etc.).

[0061] First, check the dataset for duplicate micro-videos. If duplicates are found, keep only one set. Next, check the input data for missing values. If missing values ​​are found, delete all rows with missing values. For the micro-videos, convert them into individual frames using uniform sampling or keyframe extraction. Specifically, extract 10 keyframe images from the micro-videos in the MicroLens dataset. These keyframes and their corresponding text are used as the original input, and the micro-video popularity is used as the label.

[0062] S1 represents the collection of micro-videos from an online video sharing platform as... Each micro-video content Defined as ,in, This refers to the content of the micro-video. This indicates its textual description, and This indicates its popularity score.

[0063] Dataset partitioning and data retrieval. First, a training set, validation set, and test set are generated in an 8:1:1 ratio. During the retrieval phase, this invention uses ViT-B / 32 CLIP as the image encoder and AngleE as the text encoder. All parameters of these models are frozen during the preprocessing of video frames and text. The micro-video database does not contain data from the test set. After extracting features from all micro-video frames and text in the database, the cosine similarity score between the target micro-video and all samples in the video database is calculated. Similarly, the cosine similarity of the text is calculated. Then, the similarity scores are sorted from high to low to enable retrieval of the top K instances most similar to the query data.

[0064] S2, Design a prompt template Utilizing micro-video content To guide LMM (For example, Qwen, Gemma, etc.) Generate comprehensive text descriptions for micro-videos ;

[0065] ;

[0066] S3, using a pre-trained text encoder to generate text descriptions Mapping to text embedding Then embed the text. Used as a retrieval vector, the first k micro-videos are retrieved, forming the support set for the i-th micro-video. The final set of support sets for the N micro-videos is represented as... ;

[0067] In S3, retrieval based on cosine similarity works as follows:

[0068] ;

[0069] in: It is a database supporting micro-video sets, which retrieves the top k micro-video content. and text description .

[0070] S4 uses a graph-based cue generation module that leverages a graph attention network (GAT) to adjust frozen PVLs with cueing to reflect evolving information. Each target micro-video is treated as a core node in the graph attention network (GAT), while retrieved instances serve as neighboring nodes of the core nodes.

[0071] In S4, for the text portion, a pre-trained text encoder is used to describe the target text. and retrieved text description Labeling and projecting into their respective embeddings are defined as follows:

[0072] ;

[0073] in: This represents a pre-trained text encoder. It is the embedded dimension. and The sequence lengths of the target text description and the retrieved text description, respectively;

[0074] Similarly, for the visual part, the definition is:

[0075] ;

[0076] in: This represents the pre-trained visual encoder.

[0077] This process is performed once, and the retrieval results and similarity scores for each query target are recorded and stored.

[0078] Model fine-tuning. In the model fine-tuning stage, this invention uses the ViT-B / 32 CLIP model as the visual encoder and the AngloE model as the text encoder.

[0079] Specifically, visual cues and textual cues are constructed for the target micro-video data and the related micro-video data obtained through retrieval, respectively, to guide the model to better capture key information related to the popularity of micro-videos.

[0080] S5 uses a graph attention network (GAT) to derive text prompts. and visual cues Global suggestions;

[0081] In S5, self-attention is first performed between the representation of the target micro-video and the representation of the retrieved instances, defined as follows:

[0082] ;

[0083] This is used to calculate the attention coefficient:

[0084] ;

[0085] in: These are learnable linear transformations; these coefficients represent the importance of the retrieved instances to the target micro-video. To ensure the comparability of attention coefficients across different instances,

[0086] They are normalized using the softmax function:

[0087] ;

[0088] Final text prompt for the target micro-video Calculate the weighted sum of the features of the retrieved instances:

[0089] ;

[0090] in: It is a non-linear activation function;

[0091] Similarly, applied to visual representation. and To obtain visual cues :

[0092] .

[0093] Subsequently, using the constructed cue information, the visual encoder and text encoder are jointly fine-tuned to better adapt to the specific feature distribution and cross-modal semantic space in the micro-video popularity prediction task.

[0094] S6, using text prompts and visual cues To adjust the frozen PVLs, fine-tune the PVLs, and obtain the text features and visual features of each layer of the PVLs.

[0095] S6 let and These represent the pre-trained language model and the visual model, respectively. Each of the first Transformer layer, text prompt To be inserted into the target text In the middle, visual cues Inserted into the target vision middle;

[0096] ;

[0097] ;

[0098] in: and Is it input to the number The hidden representations of the textual and visual modalities of the layer.

[0099] The fine-tuned visual encoder and text encoder extract video and text features respectively. These features are then fed into the downstream Transformer layer for cross-modal fusion and modeling. Finally, a regression head predicts the fused features and outputs the corresponding micro-video popularity score.

[0100] S7, define the text features and visual features of the last layer of PVLs as follows: and Text features and visual features The features are concatenated and fed into a transformer-based predictor to generate a popularity score. ;

[0101] ;

[0102] Where: [,] indicates concatenation.

[0103] In S6 and S7, the parameters in PVLs are frozen during training, and mean squared error loss (MSE) is used to optimize the parameters in the graph-based cue generation module and predictor.

[0104] This fine-tuning strategy not only enhances the expressive power of cross-modal features, but also effectively strengthens the model's ability to model and predict popularity drivers.

[0105] The relevant parameters in the model are updated. During model fine-tuning training, this invention focuses on the efficient updating and optimization of prompts to improve the expressive power and prediction accuracy of feature embeddings. Specifically, as model training continues, the system automatically updates the prompt parameters through backpropagation, while keeping the main parameters of the original visual encoder and text encoder frozen, updating and optimizing only the prompt-related parameters. This lightweight prompt fine-tuning mechanism can effectively improve the model's adaptability to specific tasks, while avoiding the risk of overfitting or computational overhead caused by large-scale parameter updates. For the optimizer selection, this invention uses the AdamW optimizer to update the prompt parameters using gradients. The AdamW optimizer has strong robustness and is suitable for deep learning scenarios with sparse or noisy gradients, enabling stable and efficient model optimization. Finally, through reasonable hyperparameter tuning, the fine-tuned model with optimal performance on the validation set is obtained for accurate prediction of micro-video popularity.

[0106] Save the trained model and record the test set metrics. After training is complete, input the test set into the model to obtain the corresponding evaluation metric values, and evaluate the effectiveness of the model.

[0107] As shown in Table 1, this example uses the real-world MicroLens dataset for the experiment.

[0108] Table 1. Relevant information on experimental data of this invention.

[0109]

[0110] The micro-video popularity prediction method based on graph prompts (referred to as the EvoPrompt method) of this invention was tested on the aforementioned dataset, along with existing methods such as MMRA and MQMC. In this dataset, the ratio of training, validation, and test sets was 8:1:1. The test results are shown in Table 2, and three metrics—nMSE, MAE, and SRC—were used for evaluation.

[0111] nMSE (normalized mean squared error) is an evaluation metric used in regression tasks to measure the error between the predicted result and the true label. It is mainly calculated by normalizing the mean squared error (MSE) and combining it with the variance of the true label to eliminate the influence of different data scales on error calculation. The lower the value, the better.

[0112] MAE (Mean Absolute Error) is a metric used in regression tasks to measure the difference between the predicted results and the true labels. It mainly evaluates the overall prediction bias of the model by calculating the average absolute error between the predicted value and the true value. The lower the value, the better.

[0113] Spearman's Rank Correlation (SRC) is a metric used in regression tasks to measure the consistency between predicted results and true labels in terms of ranking. It mainly evaluates the model's performance in ranking or trend prediction by calculating the Spearman rank correlation coefficient between the predicted results and the true results. A higher value is better.

[0114] Table 2. Performance of popularity prediction on real datasets

[0115]

[0116] As shown in Table 2, the experimental results of the proposed method on authoritative micro-video datasets outperform existing mainstream baseline methods, fully validating the effectiveness and advancement of the proposed method. Specifically, the proposed method, EvoPrompt, achieved the best performance and demonstrated significant performance improvements across all regression evaluation metrics. Compared to the most competitive comparative method, EvoPrompt achieved average performance improvements of 14.01%, 6.49%, and 7.38% in nMSE, MAE, and SRC metrics, respectively. These results fully demonstrate that effective modeling and information integration of multimodal content in micro-videos through LMM can significantly improve the retrieval effect of micro-videos, thereby constructing a high-quality support set for personalized PVLs, generating evolutionary hints with stronger representational capabilities, and effectively promoting the performance improvement of micro-video popularity prediction tasks.

[0117] Therefore, this invention effectively improves the accuracy and robustness of micro-video popularity prediction by constructing evolutionary cues to deeply optimize multimodal data of micro-videos. First, this invention retrieves relevant micro-video data and constructs visual and textual cues, fully utilizing external supporting information to compensate for deficiencies in the target video feature representation, thereby promoting the richness and diversity of multimodal features. Simultaneously, this invention employs a cue-driven joint fine-tuning mechanism, updating only the cue parameters, which reduces the computational and storage costs of model fine-tuning while accurately adapting to the feature distribution and semantic features of the micro-video popularity prediction task. Finally, this invention further inputs the cued-optimized visual and textual features into the downstream Transformer regression module to achieve accurate prediction of video popularity.

[0118] The TMALL model is a transformational multimodal learning model designed to find the optimal latent common space, unifying and preserving information from different modalities to better represent micro-videos. Its implementation process is described in the reference [Jingyuan Chen, et al. "Micro Tells Macro: Predicting the Popularity of Micro-Videos via a Transductive Model." International Conference on Multimedia, 2016.].

[0119] MASSL model: This is a model framework that uses feature extraction techniques and a variational autoencoder framework to predict the popularity of online micro-videos. Its implementation process is referenced in [Zhuoran Zhang, et al. "Multi-modal Variational Auto-Encoder Model for Micro-video Popularity Prediction." International Conference on Communication and Information Processing, 2022.].

[0120] CBAN model: This is a model that uses a bipolar attention mechanism to fuse modal information through the direct and inverse semantic relationships between visual and textual information, thereby classifying multimodal data. Its implementation process is referenced in [Tsun-hin Cheung and Kin-man Lam. "Crossmodal bipolar attention for multimodal classification on social media." Neurocomputing, 2022.].

[0121] JAB Model: This is an interpretable attention-based model used to evaluate the impact of post titles on their popularity, while controlling for post publication time. Its implementation can be found in the reference [Weissburg, Evan, Arya Kumar, and Paramveer S. Dhillon. "Judging a book by its cover: Predicting the marginal impact of title on Reddit post popularity." International AAAI Conference on Web and Social Media, 2022.].

[0122] MQMC model: This is a novel approach called Multi-queue Momentum Contrast Network for bidirectional retrieval, involving both unimodal feature and multimodal instance representation learning. Its implementation can be found in the reference [Yali Du, et al. "Multi-queueMomentum Contrast for Microvideo-Product Retrieval." International Conference on Web Search and Data Mining, 2023.].

[0123] MMRA model: This is a multimodal retrieval model for predicting the popularity of micro-videos, which can improve prediction accuracy by using relevant retrieved information. Its implementation process can be found in the reference [Ting Zhong, et al. "Predicting Micro-video Popularity via Multi-modal Retrieval Augmentation." International Conference on Research and Development in Information Retrieval, 2024.].

[0124] BLIP model: This is a multimodal model that uses self-guided learning for image-text pre-training, aiming to build high-quality image-text alignment features to improve cross-modal understanding and generation capabilities. Its implementation process is referenced in the literature: [Junnan Li, et al. "BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation." International Conference on Machine Learning, 2022.].

[0125] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. The scope of patent protection of the present invention shall be determined by the claims. Similarly, any equivalent structural changes made based on the content of the present invention's specification shall also be included within the scope of protection of the present invention.

Claims

1. A method for predicting the popularity of micro-videos based on graph cues enhancement, characterized in that, Includes the following steps: S1 represents the collection of micro-videos from an online video sharing platform as... Each micro-video content Defined as ,in, This refers to the content of the micro-video. This indicates its textual description, and This indicates its popularity score; S2, Design a prompt template Utilizing micro-video content To guide LMM Generate comprehensive text descriptions for micro-videos ; ； S3, using a pre-trained text encoder to generate text descriptions Mapping to text embedding Then embed the text. Used as a retrieval vector, the first k micro-videos are retrieved, forming the support set for the i-th micro-video. The final set of support sets for the N micro-videos is represented as... ; S4 uses a graph-based cue generation module that leverages a graph attention network to adjust frozen pre-trained visual language models (PVLs) with cueing to reflect evolving information. Each target micro-video is treated as a core node in the graph attention network, while retrieved instances serve as neighboring nodes of the core nodes. S5, using a graph attention network to derive text prompts. and visual cues Global suggestions; S6, using text prompts and visual cues To adjust the frozen PVLs, fine-tune the PVLs, and obtain the text features and visual features of each layer of the PVLs. S7, define the text features and visual features of the last layer of PVLs as follows: and Text features and visual features The features are concatenated and fed into a transformer-based predictor to generate a popularity score. ; ； Where: [,] indicates concatenation.

2. The method for predicting the popularity of micro-videos based on graph cues enhancement according to claim 1, characterized in that: The retrieval based on cosine similarity in S3 follows the mechanism as follows: ； in: It is a database supporting micro-video sets, which retrieves the top k micro-video content. and text description .

3. The method for predicting the popularity of micro-videos based on graph cues enhancement according to claim 2, characterized in that: In step S4, for the text portion, a pre-trained text encoder is used to describe the target text. and retrieved text descriptions Labeling and projecting into their respective embeddings are defined as follows: ； in: This represents a pre-trained text encoder. It is the embedded dimension. and The sequence lengths of the target text description and the retrieved text description, respectively; Similarly, for the visual part, the definition is: ； in: This represents the pre-trained visual encoder.

4. The method for predicting the popularity of micro-videos based on graph cues enhancement according to claim 3, characterized in that: In step S5, self-attention is first performed between the representation of the target micro-video and the representation of the retrieved instances, defined as follows: ； This is used to calculate the attention coefficient: ； in: It is a learnable linear transformation; They are normalized using the softmax function: ； Final text prompt for the target micro-video Calculate the weighted sum of the features of the retrieved instances: ； in: It is a non-linear activation function; Similarly, applied to visual representation. and To obtain visual cues : 。 5. The method for predicting the popularity of micro-videos based on graph cues enhancement according to claim 4, characterized in that: In S6, let and These represent the pre-trained language model and the visual model, respectively. Each of the first Transformer layer, text prompt To be inserted into the target text In the middle, visual cues Inserted into the target vision middle; ； ； in: and Is it input to the number The hidden representations of the textual and visual modalities of the layer.

6. The method for predicting the popularity of micro-videos based on graph cues enhancement according to claim 5, characterized in that: In S6 and S7, during the training process, the parameters in the PVLs are frozen, and the mean squared error loss (MSE) is used to optimize the parameters in the graph-based prompt generation module and predictor.

Images