Micro-video popularity prediction method based on graph prompt enhancement

Abstract

The invention discloses a micro-video popularity prediction method based on graph prompt enhancement, relates to the technical field of machine learning, and mainly utilizes a generation retriever driven by a large-scale multi-modal model to create retrieval query with rich context and execute retrieval so as to form a dynamic micro-video support set which effectively reflects an evolution mode. On this basis, the graph-based prompter generates evolutionary prompts by capturing time and relational structures in the support set. The prompts represent the latest trend dynamics, and can be used as a few examples for guiding the pre-training visual language model. By integrating the evolution prompts, the PVLs can more accurately simulate the constantly changing popularity trend, so that stronger and more predictive representation is generated, and the purpose of enhancing popularity prediction is achieved.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and more particularly to a micro-video popularity prediction method based on graph prompt enhancement. Background Art

[0002] The rapid rise of micro-video sharing platforms such as TikTok, YouTube, and Kuaishou has changed the way content is created and consumed, attracting widespread attention from academia and industry. This shift has made micro-video popularity prediction (MVPP) a valuable research problem with applications in personalized recommendations, trend analysis, and platform optimization. MVPP is a research problem in the multimodal field, and this paper mainly predicts it based on the MicroLens micro-video dataset. Currently, the methods developed for MVPP can be roughly divided into two categories:

[0003] (1) Handcrafted methods: They rely on handcrafted feature extraction for popularity prediction. The feasibility of these methods depends on objective expert knowledge and representative features.

[0004] (2) PVLs-based methods: With the rise of pre-trained visual-language models (PVLs), researchers have turned to using these models to simulate high-order cross-modal correlations to enhance MVPP.

[0005] Despite the progress made in current research, they have ignored the dynamic and evolving nature of micro-videos. This dynamic nature poses a significant challenge to existing PVLs-based methods, which mainly rely on pre-trained static knowledge stored in pre-trained models, making them unable to adapt to emerging and rapidly changing popular trends.

[0006] Therefore, it is necessary to propose a micro-video popularity prediction method based on graph prompt enhancement to solve the above problems. Summary of the Invention

[0007] The purpose of the present invention is to solve the problem that the real-time dynamic changes of micro-video platform data lead to frequent data updates, resulting in inaccurate popularity prediction and increased prediction costs.

[0008] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0009] A micro-video popularity prediction method based on image prompt enhancement includes the following steps:

[0010] S1, the micro video set from the online video sharing platform is represented as V = {V1, ..., V N}, each micro video content V i Defined as V i =(v i, t i ,y i ), where v i Indicates the content of micro video, t i Indicates its text description, and y i represents its popularity score;

[0011] S2, design a prompt template Leverage micro-video content i To guide Generate comprehensive text description of micro-videos

[0012]

[0013] S3, uses the pre-trained text encoder to generate text descriptions Mapping to text embedding ε i Then embed the text into ε i is used as a retrieval vector to retrieve the first k micro-videos, forming a support set for the i-th micro-video. The final set of N micro-video support sets is expressed as

[0014] S4 uses a graph-based hint generation module that leverages a graph attention network to adjust hints on frozen PVLs to reflect evolving information. Each target micro-video is considered a core node in the graph attention network, and the retrieved instances are treated as neighbor nodes of the core node.

[0015] S5, deriving textual hints using graph attention networks and visual cues Global prompts;

[0016] S6, using text prompts and visual cues To adjust the frozen PVLs, fine-tune the PVLs, and obtain the text features and visual features of each layer of PVLs;

[0017] S7, define the text features and visual features of the last layer of PVLs as E t and E v , the text feature E t and visual features E v Concatenate and feed the combined features into a transformer-based predictor to produce a popularity score

[0018]

[0019] Among them: [,] means serial operation.

[0020] Furthermore, the retrieval in S3 is based on cosine similarity, and the mechanism is as follows:

[0021]

[0022] in: It is a database of micro-video support set, and obtains the first k micro-video contents retrieved and text description

[0023] Furthermore, in the S4, for the text part, a pre-trained text encoder is used to convert the target text description t i and the retrieved text description are labeled and projected into their respective embeddings, defined as:

[0024]

[0025] Where: δ(·) represents the pre-trained text encoder, d is the embedding dimension, N T and N R The length of the sequences that refer to the target text description and the retrieved text description respectively;

[0026] Similarly, for the visual part, define:

[0027]

[0028] Where: Ω(·) represents the pre-trained visual encoder.

[0029] Furthermore, in S5, first, self-attention is performed between the representation of the target micro-video and the representation of the retrieved instance, and it is defined as follows:

[0030]

[0031] The attention coefficient is calculated from this:

[0032]

[0033] Where: W is a learnable linear transformation;

[0034] Normalize them using the softmax function:

[0035]

[0036] Final text prompt of the target micro-video Computed as the weighted sum of the retrieved instance features:

[0037]

[0038] Where: σ is a nonlinear activation function;

[0039] Same as above, applied to visual representation and Get visual cues from this

[0040]

[0041] Furthermore, in S6, let Ψ and Φ represent the pre-trained language model and vision model respectively. For each l-th transformer layer in Ψ, the text prompt Will be inserted into the target text Visual cues Insert into target visual middle;

[0042]

[0043]

[0044] in: and are the hidden representations of the textual and visual modalities input to the lth layer.

[0045] Furthermore, in S6 and S7, during the training process, the parameters in PVLs are frozen, and the mean square error loss MSE is used to optimize the parameters in the graph-based prompt generation module and the predictor.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1. This paper proposes EvoPrompt, an efficient prompt-enhanced micro-video popularity prediction framework. It leverages a large multimodal model-driven generative retriever to construct a dynamic micro-video support set that effectively reflects its evolving patterns. A graph-based prompter then generates evolving prompts by capturing the temporal and relational structure of the support set, guiding PVLs to produce stronger and more predictive representations.

[0048] 2. The present invention proposes an LMM-driven generative retriever for micro-video retrieval, which utilizes the knowledge stored in LMM to generate descriptive text representations of micro-videos used as retrieval queries, thereby improving the accuracy and relevance of retrieval.

[0049] 3. The present invention designs an evolutionary perception prompt mechanism for PVLs, which realizes real-time prompt adjustment of PVLs by updating the micro-video support set. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 The figure is a flow chart of the micro-video popularity prediction model based on enhanced graphical prompts according to the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0052] MVPP stands for Micro-Video Popularity Prediction, which refers to a task that automatically predicts the future popularity (e.g., views, likes, etc.) of micro-videos on short video platforms. The task definition is referenced in [Jiayi Xie, et al. "A Multimodal Variational Encoder-Decoder Framework for Micro Video Popularity Prediction." The WebConference, 2020.]

[0053] LMM stands for Large Multimodal Model, a deep learning model architecture that can simultaneously process and understand data from multiple modalities (such as images, text, and audio). Its implementation is described in [Jiayang Wu, et al. "Multimodal large language models: A survey." IEEE International Conference on Big Data, 2023].

[0054] PVLs stands for Pre-trained Vision-Language models, a framework that uses large-scale image and text data for joint pre-training to learn cross-modal representation and alignment between vision and language. For its implementation, see Yifan Du, et al. "A Survey of Vision-Language Pre-Trained Models." International Joint Conference on Artificial Intelligence, 2022.

[0055] GAT is the abbreviation of Graph Attention Network, which means "Graph Attention Network", a neural network method that introduces an attention mechanism on graph structured data, which can dynamically assign different attention weights according to the importance of neighboring nodes. Its implementation method is referenced in [ Petar, et al.″Graph Attention Networks.″International Conference on Learning Representations.2018.】

[0056] Transformer, short for Transformer Architecture, is a deep learning model framework based on the self-attention mechanism, widely used in natural language processing and multimodal modeling tasks. Its implementation is referenced in [Vaswani A., et al. "Attention Is Al1 You Need," Advances in Neural Information Processing Systems, 2017].

[0057] See also Figure 1 ,A micro-video popularity prediction method based on graph hint enhancement,

[0058] The main purpose of the micro-video popularity prediction algorithm is to obtain the popularity of micro-videos by analyzing the visual modality and textual modality of micro-video datasets, so as to help the personalized recommendation, development trend analysis and platform optimization of micro-video platforms.

[0059] The framework of the micro-video popularity prediction model is as follows Figure 1 As shown, it includes the following steps:

[0060] Data preprocessing: The model is tested and evaluated on a dataset collected from a real-world micro-video online platform (MicroLens), where each data record is formatted as (text, video, comment, publisher, and popularity, etc.).

[0061] First, check the dataset for duplicate micro-videos. If so, retain one set. Then check the input data for missing values. If so, delete all rows with missing values. For micro-videos, convert the original micro-videos into single frames using uniform sampling or keyframe extraction. Specifically, extract 10 keyframe images from the micro-videos in the MicroLens dataset. These keyframes and their corresponding text serve as the raw input, and the micro-video popularity is used as the label.

[0062] S1, represents the micro-video set from the online video sharing platform as Each micro video content V i Defined as V i =(v i , t i ,y i ), where v i Indicates the content of micro video, t i Indicates its text description, and y i Indicates its popularity score.

[0063] Dataset division and data retrieval. First, generate a training set, a validation set, and a test set in a ratio of 8:1:1. In the retrieval stage, the present invention uses ViT-B / 32CLIP as the image encoder and AnglE as the text encoder. When preprocessing video frames and text, all parameters of these models are frozen. The micro-video database does not contain the data content of the test set. After extracting features from all micro-video frames and texts in the database, the cosine similarity score between the target micro-video and all samples in the video database is calculated. Similarly, the cosine similarity of the text is calculated. Then, the similarity scores are sorted from high to low, so that the top K instances most similar to the query data can be retrieved.

[0064] S2, design a prompt template Leverage micro-video content i To guide (e.g. Qwen, Gemma, etc.) Generate comprehensive text descriptions of micro-videos

[0065]

[0066] S3, uses the pre-trained text encoder to generate text descriptions Mapping to text embedding ε i Then embed the text into ε i is used as a retrieval vector to retrieve the first k micro-videos, forming a support set for the i-th micro-video. The final set of N micro-video support sets is expressed as

[0067] S3 is searched based on cosine similarity. The mechanism is as follows:

[0068]

[0069] in: It is a database of micro-video support set, and obtains the first k micro-video contents retrieved and text description

[0070] S4 uses a graph-based prompt generation module that utilizes a graph attention network (GAT) to adjust the frozen PVLs using prompts to reflect the evolving information. Each target micro-video is regarded as a core node in the graph attention network (GAT), and the retrieved instances are regarded as neighbor nodes of the core node.

[0071] For the text part in S4, the pre-trained text encoder is used to convert the target text description t i and the retrieved text description are labeled and projected into their respective embeddings, defined as:

[0072]

[0073] Where: δ(·) represents the pre-trained text encoder, d is the embedding dimension, N T and N R The length of the sequences that refer to the target text description and the retrieved text description respectively;

[0074] Similarly, for the visual part, define:

[0075]

[0076] Where: Ω(-) represents the pre-trained visual encoder.

[0077] This process is performed once, and the retrieval results and similarity scores for each query target are recorded and stored.

[0078] Model fine-tuning: During the model fine-tuning phase, the present invention uses the ViT-B / 32CLIP model as the visual encoder and the AnglE model as the text encoder.

[0079] Specifically, visual cues and textual cues are constructed for the target micro-video data and the related micro-video data obtained through retrieval, respectively, to guide the model to better capture key information related to the popularity of the micro-video.

[0080] S5, deriving textual hints using Graph Attention Network (GAT) and visual cues Global prompts;

[0081] In S5, first, self-attention is performed between the representation of the target micro-video and the representation of the retrieved instance, defining:

[0082]

[0083] The attention coefficient is calculated from this:

[0084]

[0085] Where: W is a learnable linear transformation; these coefficients represent the importance of the retrieved instance to the target micro-video. In order to ensure that the attention coefficients of different instances are comparable,

[0086] Normalize them using the softmax function:

[0087]

[0088] Final text prompt of the target micro-video Computed as the weighted sum of the retrieved instance features:

[0089]

[0090] Where: σ is a nonlinear activation function;

[0091] Same as above, applied to visual representation and Get visual cues from this

[0092]

[0093] Subsequently, the constructed hint information is used to jointly fine-tune the visual encoder and text encoder respectively, so that they can better adapt to the specific feature distribution and cross-modal semantic space in the micro-video popularity prediction task.

[0094] S6, using text prompts and visual cues To adjust the frozen PVLs, fine-tune the PVLs, and obtain the text features and visual features of each layer of PVLs;

[0095] In S6, let Ψ and Φ denote the pre-trained language model and vision model respectively. For each lth transformer layer in Ψ, the text prompt Will be inserted into the target text Visual cues Insert into target visual middle;

[0096]

[0097] in: and are the hidden representations of the textual and visual modalities input to the lth layer.

[0098] The fine-tuned visual encoder and text encoder extract video and text features, respectively. These features are then fed into the downstream Transformer layer for cross-modal fusion and modeling. Finally, the regression head predicts the fused features and outputs the corresponding micro-video popularity score.

[0099] S7, define the text features and visual features of the last layer of PVLs as E t and E v , the text feature E t and visual features E v Concatenate and feed the combined features into a transformer-based predictor to produce a popularity score

[0100]

[0101] Among them: [,] means serial operation.

[0102] In S6 and S7, during the training process, the parameters in PVLs are frozen, and the mean squared error loss MSE is used to optimize the parameters in the graph-based prompt generation module and predictor.

[0103] This fine-tuning strategy can not only improve the expressive power of cross-modal features, but also effectively enhance the model's ability to model and predict popularity drivers.

[0104] Update the relevant parameters in the model. During the model fine-tuning training process, the present invention mainly focuses on the efficient update and optimization of prompts to improve the expressive power and prediction accuracy of feature embedding. Specifically, as the model training continues, the system automatically updates the prompt parameters through the back-propagation mechanism, while keeping the main parameters of the original visual encoder and text encoder frozen, and only updates and optimizes the prompt-related parameters. This lightweight prompt fine-tuning mechanism can effectively improve the model's adaptability to specific tasks, while avoiding the risk of overfitting or computational overhead caused by large-scale parameter updates. In terms of the choice of optimizer, the present invention uses the AdamW optimizer to perform gradient updates on the prompt parameters. The AdamW optimizer has strong robustness and is suitable for deep learning scenarios with sparse gradients or noisy gradients, and can optimize the model stably and efficiently. Finally, through reasonable hyperparameter tuning, a fine-tuning model with the best performance on the validation set is obtained for accurate prediction of micro-video popularity.

[0105] Save the trained model and record the indicator results of the test set. After the training is completed, input the test set into the model to obtain the corresponding evaluation indicator values and evaluate the effectiveness of the model.

[0106] As shown in Table 1, this example uses the real-world MicroLens dataset as an example for experiments.

[0107] Table 1 Relevant information of experimental data of the present invention

[0108] Dataset Video User Train Valid Test Language MicroLens 19738 100000 15790 1974 1974 Chinese

[0109] The micro-video popularity prediction method enhanced by graphical prompts (EvoPrompt) presented in this paper was tested against existing methods such as MMRA and MQMC on the aforementioned dataset. The training, validation, and test sets had a sample data ratio of 8:1:1. The test results are shown in Table 2, using nMSE, MAE, and SRC metrics for evaluation.

[0110] nMSE (normalized Mean Squared Error) is an evaluation metric used in regression tasks to measure the error between the predicted result and the true label. It is mainly normalized by calculating the mean square error (MSE) and combining it with the variance of the true label to eliminate the influence of different data scales on the error calculation. The lower the value, the better.

[0111] MAE (Mean Absolute Error) is an indicator used in regression tasks to measure the difference between the predicted results and the true labels. It mainly calculates the average absolute error between the predicted value and the true value to evaluate the overall prediction bias of the model. The lower the value, the better.

[0112] SRC (Spearman's Rank Correlation) is an indicator used in regression tasks to measure the ranking consistency between the predicted results and the true labels. It mainly evaluates the effectiveness of the model in ranking or trend prediction by calculating the Spearman rank correlation coefficient between the predicted results and the true results. The higher the value, the better.

[0113] Table 2 The effect of popularity prediction on real datasets

[0114]

[0115] From the experimental results in Table 2, it can be seen that the experimental results of the method of the present invention on the authoritative micro-video dataset are better than the existing mainstream baseline methods, which fully verifies the effectiveness and advancement of the proposed method. Specifically, the method of the present invention, EvoPrompt, has achieved the best performance and achieved significant performance improvements in various regression evaluation indicators. Compared with the most competitive comparison method, the average performance improvement of EvoPrompt in nMSE, MAE and SRC indicators reached 14.01%, 6.49% and 7.38% respectively. The above results fully demonstrate that the effective modeling and information integration of the multimodal content of micro-videos through LMM can significantly improve the retrieval effect of micro-videos, thereby constructing a high-quality support set for personalized PVLs, generating evolutionary prompts with stronger representation capabilities, and effectively promoting the performance improvement of micro-video popularity prediction tasks.

[0116] It can be seen that the present invention deeply optimizes micro-video multimodal data by constructing evolutionary prompts, effectively improving the accuracy and robustness of micro-video popularity prediction. The present invention first constructs visual and textual prompts by retrieving relevant micro-video data, making full use of external support information, and making up for the shortcomings of target video feature representation, thereby promoting the richness and diversity of multimodal features. At the same time, the present invention utilizes a prompt-driven joint fine-tuning mechanism to only update the prompt parameters, which not only reduces the computational and storage costs of model fine-tuning, but also can accurately adapt to the feature distribution and semantic features of the micro-video popularity prediction task. Finally, the present invention further inputs the visual and textual features after prompt optimization into the downstream Transformer regression module to achieve accurate prediction of video popularity.

[0117] TMALL model: This is a transductive multimodal learning model that aims to find the optimal latent common space, unifying and preserving information from different modalities to better represent micro-videos. Its implementation is referenced in [Jingyuan Chen, et al. "Micro Tells Macro: Predicting the Popularity of Micro-Videos via a Transductive Model." International Conference on Multimedia, 2016].

[0118] MASSL model: This model uses feature extraction techniques and a variational autoencoder framework to predict the popularity of online micro-videos. Its implementation is referenced in [Zhuoran Zhang, et al. "Multi-modalVariational Auto-Encoder Model for Micro-video Popularity Prediction." International Conference on Communication and Information Processing, 2022].

[0119] CBAN model: This model uses a bipolar attention mechanism to fuse modal information through direct and inverse semantic relationships between visual and textual information to classify multimodal data. Its implementation is referenced in [Tsun-hin Cheung and Kin-man Lam. "Crossmodal bipolar attention for multimodal classification on social media." Neurocomputing, 2022.].

[0120] JAB model: This is an interpretable attention-based model for estimating the impact of a post's title on its popularity while controlling for the post's time of publication. Its implementation is based on [Weissburg, Evan, Arya Kumar, and Paramveer S. Dhillon. "Judging a book by its cover: Predicting the marginal impact of title on Reddit post popularity." International AAAIConference on Web and Social Media, 2022.].

[0121] MQMC model: This is a new approach called Multi-queue Momentum Contrast Network for bidirectional retrieval, which incorporates both unimodal features and multimodal instance representation learning. Its implementation is described in [Yali Du, et al. "Multi-queue Momentum Contrast for Microvideo-Product Retrieval." International Conference on Web Search and Data Mining, 2023].

[0122] MMRA model: This is a multimodal retrieval micro-video popularity prediction model that uses retrieved information to improve prediction accuracy. Its implementation is referenced in [Ting Zhong, et al. "Predicting Micro-video Popularity via Multi-modal Retrieval Augmentation." International Conference on Research and Development in Information Retrieval, 2024].

[0123] BLIP model: This multimodal model uses bootstrapped learning for image-text pre-training. It aims to build high-quality image-text alignment features to improve cross-modal understanding and generation. Its implementation is described in [Junnan Li, et al. "BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation." International Conference on Machine Learning, 2022].

[0124] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. The scope of patent protection of the present invention shall be based on the claims. Any equivalent structural changes made using the contents of the description of the present invention shall also be included in the scope of protection of the present invention.

Claims

1. A micro-video popularity prediction method based on graph prompt enhancement, characterized in that: The following steps are involved: S1, represents the micro-video set from the online video sharing platform as Each micro video content V i Defined as V i =(v i , t i ,y i ), where v i Indicates the content of micro video, t i Indicates its text description, and y i represents its popularity score; S2, design a prompt template Leverage micro-video content i To guide LMM Generate comprehensive text description of micro-videos S3, uses the pre-trained text encoder to generate text descriptions Mapping to text embedding ε i Then embed the text into ε i is used as a retrieval vector to retrieve the first k micro-videos, forming a support set for the i-th micro-video. The final set of N micro-video support sets is expressed as S4 uses a graph-based hint generation module that leverages a graph attention network to adjust hints on frozen PVLs to reflect evolving information. Each target micro-video is considered a core node in the graph attention network, and the retrieved instances are treated as neighbor nodes of the core node. S5, deriving textual hints using graph attention networks and visual cues Global prompts; S6, using text prompts and visual cues To adjust the frozen PVLs, fine-tune the PVLs, and obtain the text features and visual features of each layer of PVLs; S7, define the text features and visual features of the last layer of PVLs as E t and E v , the text feature E t and visual features E v Concatenate and feed the combined features into a transformer-based predictor to produce a popularity score Among them: [,] means serial operation.

2. The micro-video popularity prediction method based on enhanced graphical prompts according to claim 1, characterized in that: The retrieval in S3 is based on cosine similarity, and the mechanism is as follows: in: It is a database of micro-video support set, and obtains the first k micro-video contents retrieved and text description 3. The micro-video popularity prediction method based on enhanced graphical prompts according to claim 2, characterized in that: In S4, for the text part, a pre-trained text encoder is used to convert the target text description t i and the retrieved text description are labeled and projected into their respective embeddings, defined as: Where: δ(·) represents the pre-trained text encoder, d is the embedding dimension, N T and N R The length of the sequences that refer to the target text description and the retrieved text description respectively; Similarly, for the visual part, define: Where: Ω(·) represents the pre-trained visual encoder.

4. The micro-video popularity prediction method based on enhanced graphical prompts according to claim 3 is characterized by: In S5, first, self-attention is performed between the representation of the target micro-video and the representation of the retrieved instance, and the definition is: The attention coefficient is calculated from this: Where: W is a learnable linear transformation; Normalize them using the softmax function: Final text prompt of the target micro-video Computed as the weighted sum of the retrieved instance features: Where: σ is a nonlinear activation function; Same as above, applied to visual representation and Get visual cues from this 5. The micro-video popularity prediction method based on enhanced graphical prompts according to claim 4, characterized in that: In S6, let Ψ and Φ represent the pre-trained language model and vision model respectively. For each lth transformer layer in Ψ, the text prompt Will be inserted into the target text Visual cues Insert into target visual middle; in: and are the hidden representations of the textual and visual modalities input to the lth layer.

6. The micro-video popularity prediction method based on graphical hint enhancement according to claim 5, characterized in that: In S6 and S7, during the training process, the parameters in PVLs are frozen, and the mean square error loss MSE is used to optimize the parameters in the graph-based prompt generation module and predictor.

Images