Video popularity prediction method and device based on multi-modal data

Abstract

The invention discloses a video popularity prediction method and device based on multi-modal data, and relates to the field of video content analysis. According to the method, image, audio and text modal features are extracted through a visual encoder, an audio feature extractor and a text encoder respectively, the weight of each modal is adjusted through a context gating mechanism, the modal features and user behavior embedding based on Mamba Hawkes Process (MHP) modeling are used as input together, the input is sent to a forward attention fusion module, and the forward attention fusion module is used for performing forward attention fusion. And the relevance between the key modal features is enhanced in a self-adaptive manner. In order to improve the modeling precision, sequence modeling is carried out aiming at the time dependence and diffusion effect of user behaviors, so that behavior information serves as a static condition to participate in prediction and serves as a dynamic driving signal to influence the feature selection and fusion process. And the multi-source features are fused under a forward attention mechanism and then are input into a predictor, and a video popularity result is output. According to the method, the feature representation capability is effectively enhanced, the accuracy is high, and the robustness is strong.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and multi-modal content understanding, in particular to a video popularity prediction method and device based on multi-modal data. The method can be used for content popularity analysis and propagation trend prediction in short video platforms, improving the accuracy and real-time performance of popularity estimation. BACKGROUND

[0002] With the rapid development of short video platforms, the continuous generation of massive video content brings new challenges to information acquisition and personalized recommendation. During the process of watching videos, platform users usually have interactive behaviors such as likes, comments, and forwards, which are not only a direct reflection of video popularity, but also constitute potential propagation paths. Popularity prediction, as an important part of the recommendation system, is of great significance to improving content exposure efficiency, optimizing recommendation ranking, and enhancing user experience. Traditional popularity prediction methods mostly rely on single-modal content features of videos (such as cover images, title keywords, etc.) or simple statistical behavior data (such as historical click volume, likes, etc.), which are difficult to fully capture the complex factors affecting video propagation. On the one hand, short videos contain multiple information modalities such as images, audio, and text, and a single modality cannot fully express the overall semantics of the video; on the other hand, user behavior has time dependence and group propagation characteristics, and existing methods lack dynamic modeling capabilities. In addition, although some methods attempt to input multi-modal features, they fail to effectively model the importance difference between modalities and their coupling relationship with user behavior, resulting in lagging and biased prediction results, which limits their practical application in high-precision and strong-generalization recommendation systems.

[0003] Therefore, in order to overcome the above problems, an intelligent prediction method that can fully integrate multi-modal video content features and model user behavior is needed to improve the accurate perception of video popularity trends and provide support for platform content distribution optimization, user interest modeling, and public opinion monitoring. For this purpose, we designed a popularity prediction method based on multi-modal feature fusion and user behavior time modeling, which can effectively improve the accuracy and real-time performance of video popularity estimation and is suitable for various scenarios such as short video recommendation and content popularity monitoring, with broad application prospects. SUMMARY

[0004] To solve the above problems, the present application proposes a video popularity prediction method and device based on multi-modal data, which fully utilizes the multi-modal features of videos and user behavior features, aiming to realize more accurate and efficient video popularity judgment and recommendation assistance.

[0005] The present application is implemented by the following technical solutions:

[0006] A video popularity prediction method based on multi-modal data, comprising the following steps:

[0007] Step S1, a text description of the input video is generated using a large language model, and then image frames, audio streams and text descriptions of the input video are extracted to obtain visual, audio and text modal features, respectively. The visual features are extracted by a ResNet-50 network, the audio features are encoded using MFCC parameters, and the text features are semantically embedded using CLIP. The three types of modal features jointly constitute a multi-modal input stream;

[0008] Step S2, a context gating mechanism is used to weight and adjust each type of modal feature. The key information is enhanced and the redundant features are suppressed through learnable gating parameters, and the weighted modal representation is output;

[0009] Step S3, user historical behavior sequence data (such as likes, forwards, comments, etc.) are obtained, and Mamba Hawkes Process (MHP) is used to time model the behavior sequence to capture the timing dependence and propagation mechanism between behaviors. The modeling results are input into a Transformer structure for deep encoding, and the user behavior embedding representation is output;

[0010] Step S4, the weighted multi-modal features and user behavior embedding are spliced and fused, input into a forward attention mechanism module, and the importance relationship between modalities is enhanced through a learnable attention matrix. The fused context expression is input into a feedforward neural network to output the popularity score prediction result of the video.

[0011] Further, in step S1, K extracted image frames F = {F i 1 ,F i 2 ,...,F i K} are encoded according to the formula to obtain frame-level visual feature embedding, and the visual modal representation is obtained where F i j represents the jth image, d v represents the dimension of each image embedding, and the video audio stream is divided into an audio segment set W = {w1, w2,..., w K} according to the timestamps of the sampled frames, where w j represents the audio segment aligned with the jth image. The audio features are extracted according to the formula where d a represents the dimension of the audio feature vector, and a text description T i of each video is generated using a large language model. The text information is encoded according to the formula to output word-level embedding ​where d t represents the embedding dimension of each text token, and the word-level embedding is transformed by a linear layer and a ReLU activation function, and mapped to a unified spatial dimension d to obtain the text modality feature

[0012] Further, in step S1, a global average pooling layer is used to process the features of the visual, audio and text modalities respectively to obtain the final multi-modal feature flow

[0013] Further, step S2 specifically includes the following steps:

[0014] Step S21, unify the three types of modality features into where L m represents the time step of modality m, and d is the unified embedding dimension;

[0015] Step S22, design a set of learnable gating structures for each modality, according to the formula learn its key information weight through mapping and nonlinear activation, where is the gating weight and bias of the modality, and σ(·) is the Sigmoid activation function, which is applied to each position, is the gating weight matrix, which outputs the importance of each time step and dimension;

[0016] Step S23, according to the formula perform element-wise multiplication of the gating weight and the original feature to realize information enhancement and inhibition, where ⊙ represents Hadamard multiplication (element-wise multiplication), is the weighted modality feature.

[0017] Further, in step S3, the user behavior sequence includes the interaction behaviors of the user on the target video on the platform, such as likes, comments, forwards, etc., and the behavior sequence is organized into multi-dimensional behavior trajectory data in chronological order.

[0018] Further, in step S3, the user behavior sequence H i = {(a1,t1),(a2,t2),...,(a n ,t n )} according to the formula h k-1 = MHP(h k ,a k ,t k ), k = 1, 2,..., n, establishes a behavior time modeling mechanism to depict the propagation dependency and long-term time sequence influence between behavior events, where represents the feature of the kth behavior (such as like, comment, forward, etc. type encoding), tk ∈R represents the timestamp of the event, denoted as ∈R. To output the hidden state sequence, d h For the hidden state dimension of Mamba, then... The input is fed into the Transformer encoder, which uses a multi-layer self-attention mechanism to model the semantic structure and dynamic patterns between behaviors, and outputs a fixed-dimensional user behavior embedding vector for subsequent fusion with multimodal features.

[0019] Furthermore, in step S4, the fusion module includes a modality fusion path and a behavior enhancement path. The modality fusion path concatenates the weighted multimodal features and strengthens the semantic dependencies between different modalities through a positive attention mechanism, thereby enhancing the contribution weight of key modalities in the context representation. The behavior enhancement path injects user behavior embeddings into the fusion process and incorporates user behavior interests and preferences for the current content into the multimodal feature representation through a cross-modal attention interaction mechanism. This serves as the input to the subsequent feedforward neural network for generating the popularity prediction results of the video.

[0020] Furthermore, in step S4, the mean squared error (MSE) is used as the optimization objective. This loss function is expressed by the formula... Calculate the predicted popularity value With real label y i The squared difference between the parameters guides the model to continuously optimize parameters during training, improving prediction accuracy. Training adopts an end-to-end approach, with all modules jointly optimizing until convergence.

[0021] Furthermore, both the training and test samples are from the MicroLens-100K dataset.

[0022] A method and apparatus for predicting video popularity based on multimodal data, characterized in that the apparatus includes a processor and a memory, wherein the memory stores program instructions, and the processor calls the program instructions stored in the memory to cause the apparatus to execute the steps of the method described above.

[0023] The present invention has the following beneficial effects:

[0024] 1. This invention utilizes a combined structure of multimodal feature extraction and context gating mechanism to achieve dynamic enhancement of key modal information in videos, obtaining more discriminative multi-source feature representations, and providing a solid foundation for subsequent popularity prediction;

[0025] 2. This invention introduces a combined structure of Mamba Hawkes Process and Transformer to collaboratively model the time dependence and propagation effect of user behavior, effectively capturing the implicit influence of behavior sequences on the evolution of video popularity and improving the model's ability to perceive behavioral dynamics.

[0026] 3. This invention differs from traditional popularity prediction methods that rely solely on video content or static statistical data. It innovatively constructs a positive attention mechanism network that integrates visual, audio, text, and user behavior, thereby enhancing intermodal correlation and semantic information interaction perception, providing a more robust and efficient solution for popularity prediction tasks. Attached Figure Description

[0027] The present invention will now be described in further detail with reference to the accompanying drawings.

[0028] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0029] Example 1

[0030] like Figure 1 As shown, a video popularity prediction method based on multimodal data includes the following steps:

[0031] Step 101: Perform modal feature extraction on the input short video. Input the video's image frames into a ResNet-50 image encoder to extract visual features, and input the corresponding audio into an MFCC feature extractor to generate spectral features; simultaneously, input the video description text into a CLIP text encoder to extract semantic embeddings. These visual, audio, and text features together constitute a multimodal input feature stream.

[0032] Step 102: Use a context gating mechanism to adjust the features of the three modalities separately. This mechanism calculates the importance weights for each modality through a learnable gating function, thereby enhancing key semantic features, suppressing redundant information, and outputting a weighted modal feature representation.

[0033] Step 103: Perform time modeling on user historical behavior sequences (such as likes, comments, and reposts). First, the Mamba Hawkes Process (MHP) is used to capture the time dependency and propagation effect of user behavior. Then, the behavior event sequence is input into a Transformer structure for deep encoding to obtain a dynamic user behavior embedding representation.

[0034] Step 104: Concatenate the context-weighted multi-modal features and user behavior embeddings and input into a forward attention mechanism module. This module models the key associations between different modal features through a learnable attention matrix, outputs the fused unified context feature representation and sends it to a feedforward neural network for prediction calculation. The network outputs the popularity score corresponding to the video, which is used for subsequent recommendation sorting or trend judgment.

[0035] Embodiment 2

[0036] The scheme in Embodiment 1 is further introduced below in combination with calculation formulas and examples, as described below:

[0037] 201: This step is mainly used to extract multi-modal feature representations of the input video.

[0038] Specifically, the extracted K image frames F = {F i 1 ,F i 2 ,...,F i K} are processed according to the formula to extract frame-level visual feature embeddings and obtain visual modal representation where F i j represents the jth image, d v represents the dimension of each image embedding, and the video audio stream is divided into an audio segment set W = {w1, w2, …, w K} according to the timestamps of the sampled frames, where w j represents the audio segment aligned with the jth image, and the audio feature is extracted according to the formula where d a represents the dimension of the audio feature vector, and a large language model is used to generate the text description T i of each video, which is encoded according to the formula to output word-level embeddings where d t represents the embedding dimension of each text token, and the word-level embeddings are mapped to a unified space dimension d through a linear transformation layer and a ReLU activation function to obtain text modal features After that, global average pooling layers are used to process the features of the visual, audio, and text modalities respectively to obtain the final multi-modal feature stream

[0039] 202: This step uses a context gating mechanism to adjust and enhance the features of the three types of modalities.

[0040] ​Specifically, first, the three types of modal features are unified as where L m represents the time step of the modal m, and d is the unified embedding dimension. Then, a set of learnable gating structures is designed for each type of modal. The gating structures have learnability and can automatically adjust the feature dimension and the attention degree on the time step for each modal during the training process, thereby improving the discriminability of the feature representation. The gating structure acts on the modal feature sequence through mapping and an activation function, and is used to automatically mine key information regions and suppress redundant or invalid features. According to the formula the key information weight is learned through mapping and nonlinear activation, where is the gating weight matrix and bias term of the modal, and σ(·) is a Sigmoid activation function acting on each position to compress the weight to the interval (0, 1), is the gating weight matrix, and the importance score at each time step and dimension is output;

[0041] Finally, according to the formula that is, the feature strength is adjusted at each time step and each dimension, respectively, and the obtained gating weight matrix is multiplied element by element with the original modal feature to obtain the weighted modal feature representation, thereby realizing information enhancement and suppression, where represents Hadamard multiplication (element-wise multiplication), is the weighted modal feature. The result of the weighting operation enables the model to dynamically highlight information highly related to the prediction target while suppressing background or ambiguous features, thereby effectively enhancing the expression ability of each modal feature before fusion.

[0042] 203: This step includes the construction of a user behavior sequence, the establishment of a time modeling mechanism, and the generation of a high-dimensional behavior representation, to enhance the understanding of user individual preferences by the model and improve the accuracy of video popularity prediction.

[0043] Specifically, the user behavior sequence refers to the interactive behaviors of a user on a target video on a video platform, including but not limited to likes, comments, and reposts. The behavior sequence is arranged in chronological order to form a multi-dimensional behavior trajectory data reflecting the dynamic interest of the user. Let the interactive sequence of a user in a time period be H i = {(a1, t1), (a2, t2), …, (a n , t n )}, where represents the feature of the kth behavior (such as like, comment, and repost type code), t k ∈ R represents the timestamp of the behavior, and the behavior trajectory is converted into a fixed-dimensional behavior representation sequence where x idenotes the embedding feature of the i-th behavior, d h denotes the embedding dimension of the behavior. To model the temporal dependence and propagation mechanism between behaviors, the present application introduces Mamba Hawkes Process (MHP) as the core of behavior modeling. This module integrates the self-exciting propagation structure in Hawkes process and the state modeling capability in Mamba model, simulating the trigger chain and time memory effect between user behaviors. The MHP model outputs a time-aware behavior representation sequence h k = MHP(h k-1 , a k , t k ), k = 1, 2, …, n, which is then input into the Transformer encoder structure to further model the semantic dependence, attention degree and long-distance dynamic pattern between behaviors through multiple layers of self-attention mechanism. The Transformer module outputs the final global behavior representation vector is the behavior embedding representation of the user for the target video, which can be used as global behavior background information and fused with the aforementioned visual, audio and text modal features for subsequent video popularity score prediction.

[0044] 204: This step mainly realizes the effective fusion of multi-modal features and user behavior representation, and models the importance relationship between different modalities by means of forward attention mechanism to generate a fused feature representation with global context expression capability, thereby providing a unified representation basis for subsequent popularity prediction.

[0045] Specifically, the weighted modal feature representation output by step 202 is concatenated with the user behavior embedding representation obtained in step 203 to form a multi-modal fusion input sequence where L = L v + L a + L t + 1 denotes the length of the fused sequence. To enhance the connection between different modalities, the present application introduces a forward attention mechanism module (Forward Attention Fusion Module) to automatically mine the key dependency relationship between modalities through a learnable attention weight matrix, thereby improving the sensitivity of the model to popularity influencing factors. Based on the multi-head attention structure, this module focuses attention on the high correlation area to output the fused context representation X c = MHA(X f , X f , X f ​), wherein the attention operation includes query (Q), key (K), and value (V) being completed within the same sequence, and belongs to the category of self-attention mechanism (Self-Attention). In order to further improve the nonlinear modeling capability of feature expression, the present application will fuse the representation X c into a feedforward neural network (FFN), and the feature conversion is realized through two fully connected layers and a nonlinear activation function, and finally the popularity score or classification prediction result y = FFN(X c ) of the target video is output.

[0046] In order to realize accurate prediction of the video popularity, a supervised learning loss function based on regression target is designed in this step. After the model completes the multi-modal feature fusion and user behavior modeling, a continuous popularity prediction value is finally output, which represents the predicted heat score of the i-th video. In order to measure the difference between the predicted value and the real popularity score y i , a mean squared error loss function (Mean Squared Error, MSE) is used as the optimization target, and the squared difference between the predicted popularity value and the real label y i is calculated according to to guide the model to continuously optimize the parameters in the training process and improve the prediction accuracy. Wherein N represents the total number of training samples. The mean squared error can smooth the penalty of prediction deviation, and has a stronger optimization guide for the prediction results with larger errors, so as to improve the fitting ability of the overall model. By taking this loss function as the target, the model modules (including modal extraction, context gating, user behavior modeling, attention fusion, and feedforward prediction network) are jointly trained in an end-to-end manner. During the training process, all parameters are continuously optimized through the gradient back propagation mechanism until convergence is achieved on the validation set or the preset accuracy is reached.

[0047] Embodiment 3

[0048] The Microlens-100k short video dataset is used, which contains about 1 billion user interaction records, 34 million users and 1 million short videos. It also includes various original video information modes, such as title, cover image, audio and full-length video. Randomly select 80% of the short videos as the training set, and the rest as the test set, extract the multi-modal video features and user behavior features from them, and use them to verify the effectiveness of the present application for short video popularity prediction.

[0049] Embodiment 4

[0050] The method and device for predicting the popularity of a video based on multi-modal data comprise a processor and a memory, the memory stores program instructions, and the processor invokes the program instructions in the memory to make the device execute the following method steps:

[0051] The multi-modal feature acquisition module is used to extract image frames, audio streams and text descriptions in the input short video respectively to form original features of three modalities of vision, audio and text. The visual modality features are extracted by a ResNet-50 image encoder to retain rich spatial structure and texture information; the audio modality features are extracted by an MFCC encoder to retain sound rhythm and energy distribution; and the text modality features are encoded by a CLIP to extract context semantic information, thereby realizing multi-level and multi-dimensional content feature expression.

[0052] The context gating weighting module is used to perform context gating processing on the multi-modal features. A learnable gating network is used to dynamically adjust the importance of the modality features according to the global context, enhance the attention to key modality features, suppress redundant interference information, and form weighted visual, audio and text modality fusion representations.

[0053] The user behavior modeling module constructs a user behavior time modeling network. First, the time dependence and propagation effect of the user historical behavior are modeled by a Mamba Hawkes process, and then the behavior sequence is deeply encoded by a Transformer structure to obtain user behavior embedding features containing time sequence dynamics. The features are used to reflect the user engagement, influence and potential influence on the video propagation trend.

[0054] The feature fusion and enhancement module concatenates the weighted multi-modal content features and the user behavior embedding features, and inputs them into a forward attention fusion module. The learnable attention matrix is used to adaptively enhance the semantic correlation and task relevance between modalities to obtain joint feature expression under the global context.

[0055] The popularity prediction module constructs a feedforward neural network as a predictor to perform multi-layer nonlinear mapping on the fused joint features to output the popularity score of the corresponding video.

[0056] The above description is only a preferred embodiment of the present application, and therefore cannot limit the scope of the present application. Any equivalent changes and modifications made according to the scope and content of the present application should still be within the scope of the present application.

Claims

1. A video popularity prediction method based on multimodal data, characterized in that: The method includes the following steps: Step S1: Generate text descriptions of the input video using a large language model. Then, extract visual, audio, and textual modal features from the image frames, audio stream, and text descriptions of the input video, respectively. Visual features are extracted using a ResNet-50 network, audio features are encoded using MFCC parameters, and textual features are semantically embedded using CLIP. The three types of modal features together constitute a multimodal input stream. Step S2: Use a context gating mechanism to weight and adjust the features of each modality. Enhance key information and suppress redundant features through learnable gating parameters, and output the weighted modality representation. Step S3: Obtain user historical behavior sequence data (such as likes, reposts, comments, etc.), use Mamba HawkesProcess (MHP) to perform time modeling on the behavior sequence, capture the temporal dependencies and propagation mechanisms between behaviors, and then input the modeling results into the Transformer structure for deep encoding, outputting the user behavior embedded representation. Step S4: The weighted multimodal features and user behavior are embedded and fused together, and then input into the positive attention mechanism module. The importance relationship between the modalities is enhanced through the learnable attention matrix. The fused contextual expression is obtained and fed into the feedforward neural network to output the popularity score prediction result of the video.

2. The video popularity prediction method based on multimodal data according to claim 1, characterized in that: In step S1, the extracted K-frame image frames F = {F i 1 ,F i 2 ,...,F i K According to the formula Frame-level visual feature embedding is extracted to obtain visual modality representation. Where F i j Let d represent the j-th frame image. v This represents the dimension of each image embedding. Based on the timestamps of the sampled frames, the video audio stream is divided into a set of audio segments W = {w1, w2, ..., w...} aligned with the image frames. K }, where w j This represents the audio segment aligned with the j-th frame image, according to the formula... Extracting audio features Where d a The dimension of the audio feature vector is represented by T, and a text description T for each video is generated using a large language model. i According to the formula Encode text information and output word-level embeddings Where d t This represents the embedding dimension of each text token, and the word-level embeddings are processed through a linear transformation layer W. t ∈R dt×d Then, a ReLU activation function is applied, which is mapped to a uniform spatial dimension d to obtain text modal features.

3. The video popularity prediction method based on multimodal data according to claim 2, characterized in that: In step S1, a global average pooling layer is used to process the features of the visual, audio, and text modalities respectively, to obtain the final multimodal feature stream.

4. The video popularity prediction method based on multimodal data according to claim 3, characterized in that: Step S2 specifically includes the following steps: Step S21: Unify the three types of modal features into Where L m The time step of modality m is represented by d, and the uniform embedding dimension is d. Step S22: Design a set of learnable gating structures for each mode, according to the formula... Its key information weights are learned through mapping and nonlinear activation, where Here, σ(·) represents the gating weights and biases for this mode, and σ(·) is the Sigmoid activation function applied at each position. Given a gated weight matrix, output the importance of each time step and dimension; Step S23: According to the formula The gating weights are multiplied element-wise with the original features to achieve information enhancement and suppression, where ⊙ represents Hadamard multiplication (element-wise multiplication). These are the weighted modal features.

5. The video popularity prediction method based on multimodal data according to claim 4, characterized in that: In step S3: the user behavior sequence includes user interaction behaviors such as liking, commenting, and forwarding the target video on the platform. This behavior sequence is organized in chronological order to form multidimensional behavior trajectory data.

6. The video popularity prediction method based on multimodal data according to claim 5, characterized in that: In step S3, the user behavior sequence H i ={(a1,t1),(a2,t2),...,(a n ,t n According to the formula h k =MHP(h k-1 ,a k ,t k (k = 1, 2, ..., n) A behavioral temporal modeling mechanism is established to characterize the propagation dependency and long-term temporal impact between behavioral events, where... t represents the feature of the k-th action (such as the type encoding of likes, comments, reposts, etc.). k ∈R represents the timestamp of the event, denoted as ∈R. To output the hidden state sequence, d h For the hidden state dimension of Mamba, then... The input is fed into the Transformer encoder, which uses a multi-layer self-attention mechanism to model the semantic structure and dynamic patterns between behaviors, and outputs a fixed-dimensional user behavior embedding vector for subsequent fusion with multimodal features.

7. The video popularity prediction method based on multimodal data according to claim 6, characterized in that: In step S4, the fusion module includes a modality fusion path and a behavior enhancement path. The modality fusion path concatenates the weighted multimodal features and strengthens the semantic dependencies between different modalities through a positive attention mechanism, thereby enhancing the contribution weight of key modalities in the context representation. The behavior enhancement path injects user behavior embeddings into the fusion process and incorporates user behavior interests and preferences for the current content into the multimodal feature representation through a cross-modal attention interaction mechanism. This serves as the input to the subsequent feedforward neural network for generating the popularity prediction results of the video.

8. The video popularity prediction method based on multimodal data according to claim 6, characterized in that: In step S4, the mean squared error (MSE) is used as the optimization objective. This loss function is expressed by the formula... Calculate the predicted popularity value With real label y i The squared difference between the parameters guides the model to continuously optimize parameters during training, improving prediction accuracy. Training is conducted end-to-end, with all modules jointly optimized until convergence.

9. A video popularity prediction method based on multimodal data according to any one of claims 1 to 8, characterized in that: Both the training and test samples are from the MicroLens-100K dataset.

10. A video popularity prediction device based on multimodal data, characterized in that: It includes a processor and memory. The memory stores program instructions, and the processor calls the program instructions in memory to implement the following modules: The multimodal feature acquisition module extracts image frames, audio streams, and text descriptions from the input short video to form raw features for visual, audio, and text modalities. Visual modal features are extracted using a ResNet-50 image encoder, preserving rich spatial structure and texture information; audio modal features are extracted using an MFCC encoder to extract spectral features, preserving sound rhythm and energy distribution; and text modal features are encoded using CLIP to extract contextual semantic information, achieving multi-level and multi-dimensional content feature representation. Context-gated weighted module: This module performs context-gated processing on the aforementioned multimodal features. It utilizes a learnable gating network to dynamically adjust the importance of each modal feature based on the global context, enhancing focus on key modal features, suppressing redundant interference information, and forming a weighted fusion representation of visual, audio, and text modalities. User behavior modeling module: Constructs a user behavior temporal modeling network. First, it models the temporal dependence and propagation effect of historical user behavior using the Mamba Hawkes Process. Then, it combines a Transformer structure to perform deep encoding on the behavior sequence, obtaining user behavior embedding features that contain temporal dynamics. These features are used to reflect user engagement, influence, and their potential impact on video dissemination trends. Feature fusion and enhancement module: The weighted multimodal content features and user behavior embedding features are concatenated and input into the positive attention fusion module. The semantic correlation and task relevance between modalities are adaptively enhanced through a learnable attention matrix to obtain joint feature representation in the global context. Popularity prediction module: Constructs a feedforward neural network as a predictor, performs multi-layer nonlinear mapping on the fused joint features, and outputs the popularity score of the corresponding video.

Images