Short video popularity prediction method based on multi-modal retrieval enhancement

Abstract

The invention belongs to the technical field of information propagation, and discloses a short video popularity prediction method based on multi-modal retrieval enhancement, which comprises the following steps: extracting key frames from a video to be predicted, and obtaining the image description of each key frame; cascading the original text information of the to-be-predicted video with the image description of the key frame of the to-be-predicted video to form a text prompt of the to-be-predicted video; inputting a text prompt of a to-be-predicted video into the large language model to obtain a retrieval vector of the to-be-predicted video; retrieving a plurality of related videos from the retrieval pool; performing feature extraction on the video to be predicted and the related video obtained by retrieval by using a pre-trained large model, and respectively extracting text and visual features; performing cross-modal feature interaction on the to-be-predicted video and the related video to obtain a corresponding mode, and performing interaction to obtain a retrieval enhancement vector; and predicting the popularity of the short video according to the modality of the video to be predicted, the modality of the related video and the retrieval enhancement vector. According to the method, retrieval enhancement is introduced, so that the prediction accuracy is greatly improved.

Description

Technical Field

[0001] The present invention belongs to the field of information dissemination technology, and relates to a video popularity prediction method based on deep learning, and specifically to a short video popularity prediction method based on multi-modal retrieval enhancement, which mainly uses multi-modal retrieval enhancement technology (Multi-modal RetrievalAugmentation), generates image descriptions (Image Caption) of key frames of videos to be predicted through large language models (LLMs), and concatenates the generated descriptions with video text to form text prompts (TextPrompt), and then provides the text prompts to the large language model to generate retrieval vectors (RetrievalVector) of the videos to be predicted. Finally, the retrieval vectors are used to retrieve several related videos in the video retrieval pool, and the information of these related videos is used to improve the accuracy of popularity prediction of the videos to be predicted. Background Art

[0002] With the popularity of online services, short video platforms such as TikTok and Instagram are constantly competing for traffic, attracting the attention of millions of users to actively create, share and spread short videos. The booming short video platform has spawned a crucial field, namely short video popularity prediction (Micro-video Popularity Prediction, MVPP). The main goal of short video popularity prediction is to predict the ratings of short videos and various user interactions (likes, reposts, comments) within a specific time range in the future. From the statistics of the number of views of different short videos, it can be seen that some short videos have attracted the attention of many users, but some videos are almost ignored. Predicting in advance whether a short video can be watched by many users can provide great help for practical problems such as advertising, commercial marketing and traffic distribution of short video platforms. Therefore, this research field has attracted great attention from scholars and experts.

[0003] Short video popularity prediction methods can be divided into two categories: (1) Feature Engineering Method, which mainly predicts popularity by artificially constructing multimodal features based on short videos. However, this method of artificially constructing features relies heavily on expert knowledge and has high requirements on the quality of the original features; (2) Deep Learning Method. Researchers use different neural networks to effectively model short video multimodal data, capture and learn cross-modal information, and thus better use limited data quality to predict short video popularity.

[0004] However, existing research mainly focuses on modeling the information of a single video, ignoring the potential collaborative information between short videos. Therefore, this limitation motivates the exploration of knowledge retrieval and explores methods to assist in predicting the popularity of a short video by allowing the model to explicitly access videos related to the short video. Summary of the invention

[0005] The purpose of the present invention is to address the problems existing in the above-mentioned prior art and provide a short video popularity prediction method based on multimodal retrieval enhancement. The retrieval enhancement technology is used to enhance the popularity prediction of the predicted video by retrieving videos related to the target short video, rather than relying solely on modeling the information of the target short video itself to infer the popularity of the video.

[0006] The idea of ​​the present invention is to generate image descriptions of key frames of the video to be predicted through a large language model, and concatenate the generated descriptions with the video text to form text prompts, and then provide the text prompts to the large language model to generate a retrieval vector for the video to be predicted. Finally, the retrieval vector is used to retrieve several related videos in the video retrieval pool, and the information of these related videos and the information of the video to be predicted is used to predict the popularity of the video to be predicted.

[0007] Based on the above invention ideas, the present invention proposes a short video popularity prediction method based on multimodal retrieval enhancement, which specifically includes the following steps:

[0008] S1, extract key frames from the video to be predicted. Specifically, for the i-th short video V to be predicted i , evenly collect L frames of pictures as its key frames.

[0009] S2, generate an image description of each key frame of the video to be predicted through a large language model (such as BLIP). Specifically, for the short video V i , the large language model generates image descriptions of its L frames:

[0010] S3, concatenate the original text information of the video to be predicted (such as the title of the video, the description of the video) with the image description of its key frame to form a text prompt of the video to be predicted. Specifically, for the short video V i , and its text prompt is: Where T i The original text information of the video.

[0011] S4, input the text prompt of the video to be predicted into the large language model (such as UAE-Large) to obtain the corresponding semantic vector, which is the retrieval vector of the video to be predicted. i , through this step to obtain its retrieval vector r i .

[0012] S5, calculating the similarity between the retrieval vector of the video to be predicted and the retrieval vector of the video to be matched in the retrieval pool, and obtaining several videos related to the video to be predicted. i , its related videos can be obtained by the following formula:

[0013]

[0014] in, It represents the video retrieval pool (Memory Bank), which calculates the other videos in the pool and the video to be predicted V i The inner product of the retrieval vector is taken, and the top K videos with the largest value are taken as the retrieval results. Each related video contains several key frames.

[0015] S6, using the pretrained large model to extract features from the video to be predicted and the retrieved related videos, extracting textual and visual features respectively. Specifically, for the video to be predicted V i , use a large visual model (such as ViT) to extract features from its key frames and obtain visual features: where d ％ is the dimension of visual features; use a large language model (such as AngLE) to extract features from its text and obtain text features: Where n represents the number of words in the text description, d + is the dimension of text features; features are extracted from the key frames of each related video in the same way.

[0016] S7, based on the extracted text features and visual features of the video to be predicted, perform cross-modal feature interaction on the video to be predicted to obtain the modality of the video to be predicted. In short videos, sometimes the text and visual information of the video are inconsistent or even opposite. In order to better adapt to the situation where the two are sometimes consistent and sometimes inconsistent, a bipolar attention mechanism is used to model cross-modal feature interaction:

[0017]

[0018] Where α and β are hyperparameters, They represent weight matrices respectively.

[0019] The above two equations model the visually guided text cross-modal features from the forward and reverse directions respectively, where the hyperparameters α and β are used to adjust the ratio of attention in the two directions. These are the modeling results of the forward and reverse attention mechanisms. Visual cross-modal features guided by text modality: The same can be obtained.

[0020]

[0021] Where α and β are hyperparameters, They represent weight matrices respectively.

[0022] Then, the original text modality content and the text-guided visual cross-modal features are aggregated together through a feed-forward network (FFN) to form a cross-modal representation of the text modality. Similarly, the cross-modal representation of the visual modality can be obtained:

[0023]

[0024] Where W1 and W2 represent weight matrices respectively.

[0025] Then, through an attention pooling layer, the text and visual cross-modal of all key frames are aggregated separately to obtain the final text and visual modality modeling results of the video to be predicted: i and V i .

[0026] S8, based on the extracted text features and visual features of the related videos, performs cross-modal feature interaction on the related videos, and aggregates the modalities of all related videos to obtain the modalities of the related videos; at the same time, obtains the label enhancement vectors of all related videos.

[0027] First, the similarity values ​​of the K related videos calculated in S5 are used to aggregate the text and visual modal features of the K videos through an attention mechanism, as shown below:

[0028]

[0029] in, This is the value of the similarity calculated in S5, indicating the retrieved related video b X The similarity between the i-th video to be predicted is normalized to obtain its corresponding attention weight; and Respectively represent related videos b X The key frames are aggregated with the visual and text features extracted by the large visual model and large language model to obtain the final representation of the retrieved video: and Then, the visual and text information after aggregation of K related videos is subjected to the same cross-modal modeling according to the operation in step S7 to obtain the modeling results of text and visual modalities: and Then, through an Attentive Pooling Layer, the text and visual cross-modalities of all key frames of the relevant video are aggregated separately to obtain the final text and visual modality modeling results of the relevant video: and

[0030] Then, the tag values ​​of the retrieved related videos are passed through a linear layer to obtain a vector with the same dimension as the modal feature, and then similarity aggregation is used in the same way to obtain a tag-enhanced vector

[0031]

[0032] S9, interactively operate the modes of the video to be predicted, the modes of the related videos, and the label enhancement vectors one by one, and concatenate the results of all operations to obtain the retrieval enhancement vector:

[0033]

[0034] Where inter(·) represents the inner product operation.

[0035] S10, predicting the popularity of the short video based on the mode of the video to be predicted, the mode of the related videos and the retrieval enhancement vector. i The features of , the retrieved features (including the modality and label enhancement vectors of related videos), and the retrieval enhancement vector concatenation: The spliced ​​vector is input into a decoder to output the popularity of the short video. The decoder used in the present invention is a multi-layer perceptron (MLP).

[0036] Thus, the present invention realizes a short video popularity prediction method based on multimodal retrieval enhancement. Compared with the prior art, the present invention has the following benefits:

[0037] 1) The core of the present invention is based on retrieval enhancement, which is not involved in traditional prediction methods. Therefore, it opens up a feasible way for introducing retrieval enhancement into short video popularity prediction;

[0038] 2) By adding information about several videos similar to the video to be predicted, the popularity of the video to be predicted can be predicted more accurately. This is especially true for cold-start users, whose video popularity fluctuates greatly. By retrieving related videos to enhance the prediction, the prediction accuracy can be greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic flow chart of the short video popularity prediction method based on multimodal retrieval enhancement provided by the present invention.

[0040] Figure 2 This is a principle block diagram of the short video popularity prediction method based on multimodal retrieval enhancement provided by the present invention.

[0041] Figure 3 The effect of the search part of the present invention is demonstrated.

[0042] Figure 4 The figure compares the video popularity prediction accuracy of cold-start users with other methods; wherein Ground Truth represents the real popularity; MMRA represents the popularity obtained by the method of the present invention; MMRA w / o Retrieval represents MMRA without the short video retrieval module, in which case MMRA is implemented by steps S1-S3, S6, S7, and S10 in the method of the present invention, and the related video processing process and related video features and retrieval enhancement vector in steps S6 and S10 are partially deleted; CBAN represents the popularity obtained by using the CBAN model.

[0043] Explanation of terms

[0044] BLIP refers to a model or method based on Bidirectional Language-Image Pretraining, which aims to process language and image data simultaneously, for example, to improve the ability of image understanding and language understanding through joint learning or multimodal representation learning.

[0045] UAE-Large is universal English sentence embedding, which is a large language model pre-trained on English text based on Bert’s language pre-training.

[0046] ViT is the Visual Attention Transformer, a model architecture based on the self-attention mechanism for image classification and other computer vision tasks, which converts image data into sequence input and processes it using the Transformer model.

[0047] AngLE stands for Angle-optimized Text Embeddings, a new SOTA model for semantic text similarity. DETAILED DESCRIPTION

[0048] The present invention is further described below in conjunction with the accompanying drawings.

[0049] like Figure 1 As shown, the short video popularity prediction method based on multimodal retrieval enhancement provided by the present invention includes the following steps:

[0050] S1, extract key frames from the predicted video;

[0051] S2, generates an image description of each key frame of the video to be predicted through a large language model;

[0052] S3, concatenating the original text information of the video to be predicted with the image description of its key frame to form a text prompt of the video to be predicted;

[0053] S4, inputting the text prompt of the video to be predicted into the large language model to obtain a corresponding semantic vector, which is the retrieval vector of the video to be predicted;

[0054] S5, calculating the similarity between the retrieval vector of the video to be predicted and the retrieval vector of the video to be matched in the retrieval pool, and obtaining several videos related to the video to be predicted;

[0055] S6, using the pre-trained large model to extract features from the predicted video and the retrieved related videos, extracting text and visual features respectively;

[0056] S7, performing cross-modal feature interaction on the video to be predicted based on the extracted text features and visual features of the video to be predicted, to obtain the modality of the video to be predicted;

[0057] S8, based on the extracted text features and visual features of the relevant videos, perform cross-modal feature interaction on the relevant videos, aggregate the modalities of all relevant videos, and obtain the modalities of the relevant videos; at the same time, obtain the label enhancement vectors of all relevant videos;

[0058] S9, interactively operate the modes of the video to be predicted, the modes of the related videos, and the label enhancement vectors in pairs, and concatenate the results of all operations to obtain the retrieval enhancement vector;

[0059] S10, predicting the popularity of the short video based on the mode of the video to be predicted, the mode of related videos and the retrieval enhancement vector.

[0060] The specific operations of steps S1-S10 are as described above and will not be explained in detail here.

[0061] The method of the present invention can be Figure 2 It is implemented by the model consisting of the short video retrieval module and the multimodal interaction & prediction module shown in FIG.

[0062] (1) Short video retrieval module: The input to this module is a short video (i.e., the video to be predicted) and its title. First, extract the key frames of the video to be predicted according to the above step S1; then, obtain the image description of the key frames of the video to be predicted through the large language model BLIP according to step S2; then, concatenate the image description with the title of the video to be predicted according to step S3 to form a text prompt of the video to be predicted; and extract the semantic information through the large model UAE according to step S4 to obtain the retrieval vector; then, obtain the relevant K videos in the retrieval pool through the retrieval vector according to step S5.

[0063] (2) Multimodal interaction & prediction module: After obtaining K related videos of the video to be predicted from module (1), they are first preprocessed by the large model ViT and AngLE according to step S6, and then the two-modal representations of the video to be predicted and the retrieved related videos are obtained through cross-modal bidirectional interaction according to steps S7 and S8; at the same time, the popularity label value of the retrieved related video is encoded through a linear layer to obtain a new label representation (i.e., label enhancement vector). Finally, according to step S9, the modal representation (including the modality of the video to be predicted and the modality of the related videos) and the label representation are interacted through retrieval enhancement to obtain a retrieval enhancement vector. Then, according to step S10, the features of the video to be predicted, the retrieved features, and the retrieval enhancement vector are spliced ​​and input into the multi-layer perceptron to obtain the final popularity value.

[0064] Application Examples

[0065] This application example uses a real short video dataset provided by MicroLens, which consists of 19,738 independent short videos watched by 100,000 users from multiple Chinese short video platforms. The dataset provides an open source access version. The link of the open source version is: https: / / github.com / westlake-repl / MicroLens. The collection process and more details can be found in the literature [Yongxin Ni, Yu Cheng, Xiangyan Liu, Junchen Fu, Youhua Li, Xiangnan He, Yongfeng Zhang, and Fajie Yuan. 2023. A Content-Driven Micro-Video Recommendation Dataset at Scale. arXiv preprint arXiv: 2309.15379 (2023)]. The data set was divided into training set, validation set and test set according to the ratio of 8:1:1 by using the traditional data set division method; the model used in the present invention was trained and parameter optimized by using the conventional training method in the field according to the training set and validation set; and the effects of the method of the present invention (MMRA) and some other related methods were tested by using the test set, and the evaluation indicators were nMSE, MAE and SRC. The experimental results are compared in Table 1.

[0066] Table 1: Popularity prediction results of various methods on the MicroLens dataset

[0067]

[0068] Among them, the smaller the values ​​of nMSE and MAE, the larger the SRC value, which means the better the performance of the model.

[0069] The remaining methods in the table are described as follows:

[0070] SVR: The full name is Support Vector Regression. It is a machine learning method. The core idea is to find a hyperplane that minimizes the distance from all data to the hyperplane. Its implementation process in the field of short video popularity can be found in the literature [Aditya Khosla, Atish Das Sarma, and Raffay Hamid. 2014. What makes an image popular? . In Proceedings of the ACM Web Conference (WWW). 867–876].

[0071] HyFea: The full name is Hybrid Feature Based Approach. It carefully considers multiple feature types and adopts a tree-based ensemble learning method: CatBoost. Its implementation process can be found in the literature [Xin Lai, Yihong Zhang, and Wei Zhang. 2020. HyFea: Winning solution to social media popularity prediction for multimedia grand challenge 2020. In ACM International Conference on Multimedia (MM). 4565–4569].

[0072] MFTM: The full name is Multi-modal Feature Transformation Module. It uses LightGBM and TabNet as the model framework to effectively capture the intricate semantic relationship between different modal features and user-specific data. Its implementation process can be found in the literature [Chih-Chung Hsu, Chia-Ming Lee, Xiu-Yu Hou, and Chi-Han Tsai. 2023. GradientBoost Tree Network based on Extensive Feature Analysis for Popularity Prediction of Social Posts. In ACM International Conference on Multimedia (MM). 9451–9455].

[0073] CLSTM: The full name is Contextual Long Short-Term Memory. It is a long short-term memory network based on contextual learning. Its implementation process can be found in the literature [Shalini Ghosh, Oriol Vinyals, Brian Strope, Scott Roy, Tom Dean, and Larry Heck. 2016. Contextual lstm (clstm) models for largescale nlp tasks. arXiv preprint arXiv: 1602.06291 (2016)].

[0074] TMALL: The full name is Transductive Multi-modal Learning Approach, which assumes that there is an optimal common space that can maintain the original intrinsic characteristics of short videos in the original space. At the same time, short videos with different popularity can be better separated in this optimal common space compared with short videos of each single mode. The implementation process can be referred to the literature [Jingyuan Chen, Xuemeng Song, Liqiang Nie, Xiang Wang, Hanwang Zhang, and Tat-Seng Chua. 2016. Micro tells macro: Predicting the popularity of micro-videos via a transductive model. In ACM International Conference on Multimedia (MM). 898–907].

[0075] MASSL: The full name is Multi-modal Based VAE Regression Model. It is a deep learning model based on variational encoder. Its implementation process can be found in the literature [Zhuoran Zhang, Shibiao Xu, Li Guo, and Wenke Lian. 2022. Multi-modal Variational Auto-Encoder Model for Micro-video Popularity Prediction. In Proceedings of the International Conference on Communication and Information Processing (ICCIP). 9–16].

[0076] CBAN: The full name is Crossmodal Bipolar Attention. It uses a bidirectional attention mechanism to better capture the consistency and inconsistency of text and visual modalities and model them separately. Its implementation process can be found in the literature [Tsun-hin Cheung and Kin-man Lam.2022.Crossmodal bipolar attentionformultimodal classification on social media.Neurocomputing 514(2022),1–12].

[0077] HMMVED: The full name is Hierarchical Multimodal Variational Encoder-Decoder, which comprehensively utilizes user information and micro video content in a hierarchical manner to predict the popularity of short videos. Specifically, the multimodal variational encoder-decoder framework encodes the input modality into a low-dimensional random embedding and decodes the popularity of the short video from it. The implementation process can be found in the literature [JiayiXie, Yaochen Zhu, and Zhenzhong Chen. 2023. Micro-Video Popularity Prediction Via Multimodal Variational Information Bottleneck. IEEE Transactions on Multimedia (TMM) 25 (2023), 24–37].

[0078] It can be seen from the experimental results in Table 1 that the short video popularity prediction based on multimodal retrieval enhancement proposed in the present invention has a higher prediction accuracy than other methods.

[0079] also, Figure 3 The contents of the top three related videos with the highest similarity to the target video retrieved in steps S1-S5 of the method of the present invention are displayed (the target video and related videos are both from the MicroLens dataset). Figure 3 It can be seen that through steps S1-S5, related videos with high similarity to the target video can be obtained, thereby proving the effectiveness of the retrieval method provided by the present invention.

[0080] Figure 4 The popularity prediction results of five cold start users (users who have only posted one video) from the MicroLens dataset using different methods are given. As can be seen from the figure, the popularity obtained by the method of the present invention is closest to the real popularity. The present invention can accurately predict the popularity of cold start user videos.

[0081] It can be seen that the present invention utilizes the information of several related videos obtained by retrieval to enhance the method of predicting the popularity of the video to be predicted, thereby achieving a prediction effect that is superior to the existing methods.

[0082] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A short video popularity prediction method based on multimodal retrieval enhancement, characterized in that: The following steps are involved: S1, extract key frames from the predicted video; S2, generates an image description of each key frame of the video to be predicted through a large language model; S3, concatenating the original text information of the video to be predicted with the image description of its key frame to form a text prompt of the video to be predicted; S4, inputting the text prompt of the video to be predicted into the large language model to obtain a corresponding semantic vector, which is the retrieval vector of the video to be predicted; S5, calculating the similarity between the retrieval vector of the video to be predicted and the retrieval vector of the video to be matched in the retrieval pool, and obtaining several videos related to the video to be predicted; S6, using the pre-trained large model to extract features from the predicted video and the retrieved related videos, extracting text and visual features respectively; S7, performing cross-modal feature interaction on the video to be predicted based on the extracted text features and visual features of the video to be predicted, to obtain the modality of the video to be predicted; S8, based on the extracted text features and visual features of the relevant videos, perform cross-modal feature interaction on the relevant videos, aggregate the modalities of all relevant videos, and obtain the modalities of the relevant videos; at the same time, obtain the label enhancement vectors of all relevant videos; S9, interactively operate the modes of the video to be predicted, the modes of the related videos, and the label enhancement vectors in pairs, and concatenate the results of all operations to obtain the retrieval enhancement vector; S10, predicting the popularity of the short video based on the mode of the video to be predicted, the mode of related videos and the retrieval enhancement vector.

2. The short video popularity prediction method based on multimodal retrieval enhancement according to claim 1 is characterized in that: In step S5, for the short video V i , its related videos can be obtained by the following formula: in, represents the video retrieval pool, r i Represents the retrieval vector of the video to be predicted, r b Represents the retrieval vector of the video to be matched.

3. The short video popularity prediction method based on multimodal retrieval enhancement according to claim 1 is characterized in that: In step S7, the text cross-modal features guided by the visual feature modality are: In the formula, the hyperparameters α and β are used to adjust the ratio of attention in the two directions. Respectively represent the modeling results of the forward and reverse attention mechanisms; The visual cross-modal features guided by the text modality are as follows: Respectively represent the modeling results of the forward and reverse attention mechanisms; Then, the original text modality content and the text-guided visual cross-modal features are aggregated together through a feed-forward network to form a cross-modal representation of the text modality as shown in formula (6): Similarly, we can obtain cross-modal representation of visual modality As shown in formula (7); Then, through an attention pooling layer, the text and visual cross-modal of all key frames are aggregated separately to obtain the final text and visual modality modeling results of the video to be predicted: T i and V i .

4. The short video popularity prediction method based on multimodal retrieval enhancement according to claim 3 is characterized in that: In step S8, first, the similarity values ​​of the K related videos calculated in S5 are used to aggregate the text and visual cross-modal features of the K videos through an attention mechanism, as shown below: in This is the value of the similarity calculated in S5, indicating the retrieved related video b J The similarity between the i-th video to be predicted is normalized to obtain its corresponding attention weight; through aggregation, the final representation of the related video is obtained: and Then, the visual and text information after aggregation of K related videos is subjected to the same cross-modal modeling according to the operation in step S7 to obtain the modeling results of text and visual modalities: and Then, through an attention pooling layer, the text and visual cross-modalities of all key frames of the relevant video are aggregated separately to obtain the final text and visual modality modeling results of the relevant video: and 5. The short video popularity prediction method based on multimodal retrieval enhancement according to claim 4 is characterized in that: The tag value of the retrieved video is passed through a linear layer to obtain a vector with the same dimension as the modal feature, and then similarity aggregation is used in the same way to obtain a tag-enhanced vector 6. The short video popularity prediction method based on multimodal retrieval enhancement according to claim 5 is characterized in that: In step S9, the search enhancement vector is represented as follows: Where inter(·) represents the inner product operation.

7. The short video popularity prediction method based on multimodal retrieval enhancement according to claim 1 is characterized in that: The mode of the video to be predicted, the mode of related videos, and the retrieval enhancement vector are concatenated, and the concatenated vector is input into the decoder to output the popularity of the short video.

8. The short video popularity prediction method based on multimodal retrieval enhancement according to claim 7 is characterized in that: The concatenated vector is represented as: Where V i 、T i Respectively represent the modeling results of the final text and visual modalities of the video to be predicted, Represent the modeling results of the final text and visual modalities of the relevant videos, respectively. represents the label enhancement vector of the related videos of all related videos, and I represents the retrieval enhancement vector.

9. The short video popularity prediction method based on multimodal retrieval enhancement according to claim 7 is characterized in that: The decoder is a multi-layer perceptron.

Images