User engagement prediction method based on distillation multimodal retrieval enhancement

Abstract

The application discloses a user participation degree prediction method based on distillation multimodal retrieval enhancement, and belongs to the technical field of social media data mining and user behavior analysis.The application introduces a self-enhanced distillation module to evaluate the relevance of UGC, and selects top-K relevant UGC in combination with a selected retriever, so that irrelevant information is effectively avoided from interfering with relevant UGC, and the noise caused by irrelevant artifacts is filtered out when the relevant UGC is retrieved, so that the original feature representation of the relevant UGC is maintained.A heterogeneous graph construction module enhances the interaction representation capability between UGC through multi-relation modeling, and can more accurately optimize the prediction result in user participation degree prediction.The application can better balance the information retrieval between relevant UGC and irrelevant UGC, avoid excessive spread of irrelevant information, and improve the performance of the model in user participation degree prediction.In particular, when complex multimodal data containing a large amount of irrelevant UGC is processed, the mechanism of the application can effectively enhance the distinguishability of relevant UGC and improve the prediction accuracy.

Description

Technical Field

[0001] This invention belongs to the field of social media data mining and user behavior analysis technology, and is applied to user engagement prediction and content recommendation on social media platforms. In particular, it relates to a user engagement prediction method based on distillation multimodal retrieval enhancement. Background Technology

[0002] With the rapid development of social media platforms, User Engagement Prediction (UEP) has become a key technology for applications such as recommendation systems, online marketing, and content moderation. The goal of UEP is to predict user engagement with specific content by analyzing multimodal data (text, images, audio, and video), thereby enabling accurate content recommendations and optimized traffic scheduling.

[0003] Traditional user engagement prediction methods primarily rely on single-modal feature learning. Methods such as HMMVED and UHAN utilize only single-modal features and lack in-depth cross-modal insights. These single-modal methods ignore the complex intermodal relationships in multimodal social media data, resulting in limited prediction accuracy.

[0004] In recent years, multimodal learning architectures have made significant progress in user engagement prediction. For example, methods such as CLSTM and DLBA have designed neural network architectures capable of handling complex multimodal data; BLIP and MASSL have enhanced multimodal feature extraction capabilities by introducing attention mechanisms or hierarchical processing. However, these methods make the learning of user-generated content (UGC) relatively independent, ignoring the interaction between the target UGC and context-related UGC.

[0005] Recently, retrieval-oriented methods have demonstrated significant potential in user engagement prediction. By retrieving context-relevant UGC from databases, these methods can capture the relationship between queries and their surrounding content. For example, retrieval-oriented models such as NIPA and MMRA incorporate state-of-the-art retrieval-enhanced generative techniques from large language models. These models aim to retrieve relevant UGC from memory or databases to improve the representation quality of the target UGC.

[0006] Despite significant progress, existing retrieval methods still face two key challenges:

[0007] 1. Context mismatch problem: Existing retrieval strategies mainly calculate the semantic similarity between different modalities, but fail to capture key features related to various UGCs (such as the publishing users and topics), often resulting in the retrieval of irrelevant instances. For example, when the query only focuses on UGC photos, the model may retrieve many UGCs with similar photos, but their associated users and topics may be significantly different from the query.

[0008] 2. Irrelevant Retrieval Artifacts: Not all retrieved UGC is truly relevant to the query UGC. The effectiveness of a retrieval largely depends on the robustness of the retrieval algorithm, which may not always guarantee high-quality results. Therefore, noisy or irrelevant UGC is often introduced, which can dilute the retrieval process and negatively impact prediction accuracy. Summary of the Invention

[0009] This invention proposes a user engagement prediction method based on distilled multimodal retrieval enhancement, aiming to solve the problems of context mismatch and irrelevant retrieval artifacts in existing methods within a multimodal UGC environment. This method explores distilled multimodal representation through information bottleneck technology, enhances UGC relevance retrieval, improves the quality of retrieved UGC, and thus achieves more accurate user engagement prediction. The specific technical problems existing in the prior art are as follows:

[0010] Traditional user engagement prediction methods primarily rely on single-modal feature learning, neglecting the complex intermodal relationships in multimodal social media data, thus limiting prediction accuracy. This reliance on a single modality prevents the model from capturing interactions between multiple modalities such as text, images, audio, and video, resulting in significant prediction bias in complex UGC environments.

[0011] Multimodal learning architectures make UGC learning relatively independent, ignoring the interaction between target UGC and context-related UGC, leading to context mismatch issues. The retrieved UGC is often irrelevant to the query, introducing noise or irrelevant artifacts, diluting the retrieval process, and negatively impacting prediction accuracy.

[0012] Existing retrieval-oriented methods fail to capture key features relevant to various UGC (such as posting users and topics), often resulting in the retrieval of irrelevant instances. This irrelevant retrieval artifact not only exacerbates noise interference but also weakens the model's robustness in high-noise or information-deficient scenarios, significantly increasing prediction errors.

[0013] To solve the above-mentioned technical problems, the specific technical solution of the present invention is as follows:

[0014] A user engagement prediction method based on distillation multimodal retrieval enhancement includes the following steps:

[0015] Step 1: Construct a retrieval target query based on the fusion of user-generated content (UGC) with visual modality, text modality, and additional descriptive information. Input each UGC representation into the UGC feature encoder and obtain a retrieval distillation vector that minimizes redundant information through information bottleneck constraints.

[0016] Step 2: The target query and candidate UGCs after distillation are scored using a ranking function, and the top-K with the highest search scores are selected as the final candidate UGCs;

[0017] Step 3: The final candidate UGCs are used as nodes, and the publishing topic, time, and publisher are used as edge types to construct a heterogeneous graph. The node representations are learned through a heterogeneous convolutional network. Finally, the predicted multimodal UGC participation is obtained through a fully connected layer, and the prediction results are optimized using a loss function.

[0018] Step 1 is described in detail as follows:

[0019] Step 1.1: Multimodal UGC query construction; First, a pre-trained image-to-text model is used to extract semantic features of the visual modality, and these features are combined with the text modality to obtain merged text; Entities and actions are extracted from the merged text and integrated with additional descriptive information to obtain the target query;

[0020] Step 1.2: Self-enhancing distillation mechanism; representing each UGC. The input is fed into the UGC feature encoder, which is an encoder composed of a three-layer fully connected network. It processes the input UGC representation sequentially, and this process requires information bottleneck constraints to obtain the most informative representation. Constraint loss It consists of two mutual information terms:

[0021]

[0022] in, It is a balance trade-off coefficient. The parameters representing the UGC feature encoder, The original multimodal representation set representing all UGC, i.e. ; For participation tags; after distillation, the target UGC query is obtained. and distillation candidate UGC ;

[0023] Maximize Reconstruction The distribution of the UGC feature encoder is denoted as: Assuming Following an isotropic Gaussian distribution, it is represented as ,in, It is the mean of this distribution. It is the variance vector of this distribution. Given the covariance matrix of this distribution, transform this mutual information term and derive its lower bound:

[0024]

[0025] in, Participation tags Marginal probability distribution, Indicates label entropy, Indicates the joint distribution The expectation, this boundary term is included in the loss function of the pre-trained distillation model;

[0026] Minimize regularization Regularization term The upper bound is:

[0027]

[0028] in, and Represent and Marginal probability distribution, express and The joint probability distribution, Indicates a given hour The true conditional probability distribution Indicates the joint distribution Expectations Calculate the KL divergence.

[0029] Step 2 is described in detail below:

[0030] A simplified BM25 sorting function is used for retrieving calculated scores and sorting selections:

[0031]

[0032] in Target UGC query Feature dimensions in It is an adjustable hyperparameter. Target UGC query In feature dimension The features represented on, Candidate UGC In feature dimension Feature representation on, IDF yes Inverse frequency in UGC; Jaccard correlation is defined as follows:

[0033]

[0034] Where, when two vectors When they are the same, It equals 1 if it is equal to 1, otherwise it is 0.

[0035] After calculating the retrieval scores of the target UGC and candidate UGCs, the top-K candidate UGCs with the highest retrieval scores are selected, denoted as _____. .

[0036] Step 3 is as follows:

[0037] Step 3.1: Heterogeneous UGC Graph Construction; The top-K UGC content obtained from the selected search is divided into two types of nodes: visual nodes. and text nodes The entire UGC content of the target is treated as a single user node. Three types of relationships are designed by combining additional descriptive information: topic relationship, time relationship, and publisher relationship.

[0038] Step 3.2: Design a heterogeneous convolutional network to learn node representations by considering different types of nodes and their various relationships; for each node , Define the heterogeneous convolutional network for the set of nodes. The representation of nodes in the layer is updated, and the update rules are derived from neighboring nodes. Aggregate information, Given a set of neighboring nodes, consider the edges connecting them. Type:

[0039]

[0040] in, , It is the first Layer nodes The representation of u and These are the node-specific weight matrix and the matrix corresponding to the edge type. The weight matrix represents the node. and The relationship between them A set of edge types, It is a non-linear activation function, ReLU.

[0041] Step 3.3: The representations learned from the heterogeneous graph are passed through two fully connected layers to predict the engagement level of each target UGC. We introduce mean square error as the optimization loss function.

[0042] Compared to existing technologies, the main advantage of this invention lies in its ability to effectively reduce the negative impact of irrelevant information on relevant UGC by finely controlling the propagation of irrelevant retrieval artifacts. While existing retrieval-oriented methods (such as NIPA and MMRA) can retrieve contextual UGC, they often suffer from excessive propagation of irrelevant information between UGCs when processing multimodal UGCs, causing irrelevant artifacts to interfere with the features of relevant UGCs. The introduction of these irrelevant artifacts dilutes the retrieval process and reduces prediction accuracy.

[0043] This invention introduces a self-enhancing distillation module to evaluate the relevance of UGC, and combines this with a selective retrieval tool to choose the top-K relevant UGC. This effectively avoids interference from irrelevant information on relevant UGC, ensuring that noise from irrelevant artifacts is filtered out during retrieval, thus preserving its original feature representation. Simultaneously, the heterogeneous graph construction module enhances the interactive representation capabilities between UGC through multi-relationship modeling. In this way, the features of relevant UGC are effectively protected, while irrelevant UGC is effectively excluded, thereby enabling more accurate optimization of prediction results in user engagement prediction.

[0044] Compared to traditional methods, this invention better balances information retrieval between relevant and irrelevant UGC, avoiding the excessive diffusion of irrelevant information and improving the model's performance in user engagement prediction. Especially when dealing with complex multimodal data containing a large amount of irrelevant UGC, this mechanism effectively enhances the distinguishability of relevant UGC and improves prediction accuracy. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the method framework of the present invention. Detailed Implementation

[0046] This invention proposes a user engagement prediction method based on distilled multimodal retrieval enhancement, aiming to solve the problems of context mismatch and irrelevant retrieval artifacts in existing methods in multimodal UGC environments. This method explores distilled multimodal representation through information bottleneck technology, enhances UGC relevance retrieval, improves the quality of retrieved UGC, and thus achieves more accurate user engagement prediction.

[0047] like Figure 1As shown, this invention proposes the DRUEP (Distilled Retrieval knowledge-augmentedgraph for enhancing User Engagement Prediction) framework, which consists of three key components: Self-boosting Distillation, Curated Retriever, and Heterogeneous Graph Contriving. First, we construct a retrieval query and apply an information bottleneck method to effectively distill all user-generated content (UGC). Then, we retrieve the top K most relevant UGCs based on a relevance ranking algorithm. Finally, we construct a heterogeneous graph using the distilled information and the top K UGCs to further enhance user engagement prediction. The following is a detailed description of the main modules:

[0048] Step 1: Construct and distill UGC retrieval queries based on the self-enhancing distillation module.

[0049] Step 1.1, Multimodal UGC Query Construction

[0050] Before the retrieval process, a retrieval target query needs to be constructed for each multimodal UGC. Search target query The design includes three aspects: visual modal information Text modal information Additional description information This includes the topic, publication time, and user ID.

[0051] First, a pre-trained image-to-text model is used to extract visual modalities. Semantic features and their relationship with text content The combined text is then used to obtain the merged text. Next, entities and actions are extracted from the merged text to further enrich the query content and contextual information. Finally, this information is integrated with additional descriptive information to ensure the target query... It possesses completeness and contextual relevance during the retrieval process.

[0052] Step 1.2, Self-enhanced distillation mechanism

[0053] Each UGC and target query are uniformly represented as Input to UGC feature encoder In this process, the retrieval distillation vector with minimal redundant information is obtained:

[0054]

[0055] Among them, UGC feature encoder An encoder consisting of a three-layer fully connected network sequentially processes the input UGC representation. It is a potential variable containing the most informative content of UGC. These are the weight matrix and bias vector, where l represents the layer number. Simultaneously, this process requires constraints related to information bottlenecks to obtain the most informative representation. Constraint loss It consists of two mutual information terms:

[0056]

[0057] in, Indicates mutual information, This represents the mutual information between Z and H under a distribution defined by parameter φ. It is a balance trade-off coefficient. Represents UGC feature encoder The parameters, The original multimodal representation set representing all UGC, i.e. By adjusting The value of can ensure From engagement tags Learn the most information from the original multimodal representations of all UGC, while drawing as much information as possible from the original collection of UGC representations. Irrelevant information is removed. After distillation, the target UGC query and candidate UGCs are obtained. Based on the information bottleneck assumption, the multimodal information of each UGC is evaluated by distillation to determine whether to retain key features. This method can enhance the sensitivity of the model in user engagement prediction, especially when dealing with UGCs with complex multimodal structures, and can effectively identify perturbations caused by irrelevant retrieval.

[0058] Simplified parameters This indicates that maximizing reconstruction The distribution of the previously proposed UGC feature encoder is denoted as: As An approximation. Assume... Following an isotropic Gaussian distribution, it is represented as ,in, It is the mean of this distribution. It is the variance vector of this distribution. This is the covariance matrix of the distribution, calculated by the UGC feature encoder. Then, this mutual information term is formally transformed and its lower bound is derived:

[0059]

[0060] in, Participation tags Marginal probability distribution, Indicates label entropy, Indicates the joint distribution The expected value. Due to the inherent nonnegativity of cross-entropy, the derived inequality holds, and this boundary term is included in the loss function of the pre-trained distillation model.

[0061] Minimize regularization Its regularization term The upper bound can be:

[0062]

[0063]

[0064]

[0065] in, and Represent and Marginal probability distribution, express and The joint probability distribution, Indicates a given hour The true conditional probability distribution Indicates the joint distribution The expectation. The upper bound of the derivation is approximately posterior. and prior The Kullback-Leibler (KL) divergence between them is calculated, assuming both are Gaussian distributions. Calculate the KL divergence.

[0066] During the distillation process, DRUEP maximizes the retention of relevant information and minimizes the propagation of irrelevant information through information bottleneck constraints. Maximizing reconstruction ensures that the most information is learned from engagement labels, while minimizing regularization removes irrelevant information from UGC representations to prevent noise interference. This mechanism effectively reduces information pollution caused by irrelevant UGC, thereby improving the accuracy of user engagement prediction.

[0067] Step 2. Use the Selected Searcher module to filter candidate UGC.

[0068] and These are the distilled query and candidate UGC, respectively, obtained from the self-enhancing distillation module to retrieve the distillation target UGC query. and distillation candidate UGC Subsequently, a simplified BM25 ranking function was proposed for retrieval calculation of scores and ranking selection:

[0069]

[0070] in Target UGC query Feature dimensions in It is an adjustable hyperparameter. Target UGC query In feature dimension The features represented on, Candidate UGC In feature dimension The features represented on, Measurement and Feature correlation between them, IDF yes Inverse frequency in UGC. Jaccard correlation is defined as follows:

[0071]

[0072] Where, when two vectors When they are the same, Equals 1 if true, otherwise 0. This metric is used based on sharing. and unique element Quantifying the correlation between two vectors involves actually measuring the degree to which documents share similar content based on their terminology.

[0073] After calculating the retrieval scores of the target UGC and candidate UGCs, the top-K candidate UGCs with the highest retrieval scores are selected. These top-K UGCs are denoted as... This represents the most relevant UGC based on the adopted scoring mechanism. The refined retrieval module ensures that the retrieved UGC is highly relevant to the query, reducing noise interference and improving the robustness of predictions.

[0074] Step 3. Heterogeneous Graph Construction Module

[0075] In user engagement prediction, traditional isomorphic graphs have limitations in capturing the complex and diverse relationships between entities because they only represent a single type of node and edge. However, heterogeneous graphs allow for the representation of multiple types of nodes (such as visual and textual modalities) and edges (such as topics and user relationships), providing a richer and more detailed structure.

[0076] 3.1 Construction of Heterogeneous UGC Graphs;

[0077] Node entity classification: The top-K UGC content obtained from the selected retrieval is divided into two types of nodes: visual nodes. and text nodes The entire UGC content of the target is treated as a single user node. .

[0078] Multiple Relationship Construction: Three types of relationships are designed by fully combining additional descriptive information: (1) VCV represents nodes of the same type that share the same topic relationship; (2) VTV represents nodes of the same type that are published on the same day; (3) VUV represents nodes of the same type that share the same social publisher.

[0079] 3.2 Graph Representation Learning;

[0080] Given the three types of heterogeneous nodes and three types of edges defined above, a heterogeneous convolutional network is designed to learn node representations by considering different types of nodes and their various relationships. For each node... , Define the heterogeneous convolutional network for the set of nodes. The representation of nodes in the layer is updated, and the update rules are derived from neighboring nodes. Aggregate information, Given a set of neighboring nodes, consider the edges connecting them. Type:

[0081]

[0082] in, , It is the first Layer nodes The representation of u and These are the node-specific weight matrix and the matrix corresponding to the edge type. The weight matrix represents the node. and The relationship between them A set of edge types, It is a non-linear activation function, ReLU.

[0083] 3.3 Optimization Objectives;

[0084] The task is to predict multimodal UGC participation. Therefore, two fully connected (FC) layers are proposed to learn the representation from the heterogeneous graph to predict the engagement level of each target UGC. We introduce mean squared error as the optimization loss function:

[0085]

[0086] in, This represents the set of node representations learned from the heterogeneous graph. Indicates a fully connected layer. Representing the number of samples of all target UGC in the dataset, heterogeneous graphs provide a more comprehensive framework for predicting UGC engagement by incorporating diverse relationships, thereby improving accuracy.

[0087] In this embodiment, the method was validated on three large-scale real-world social media datasets containing multimodal UGC: ICIP, SMPD, and Instagram. Basic statistical information for these datasets is shown in the table below:

[0088] Dataset ICIP SMPD Instagram UGC Quantity 20,337 305,613 297,865 Number of users 17,302 38,312 33,935 Average participation 200.78 493.14 4,694.26 Average text length 27.68 91.75 442.78

[0089] To ensure the reliability of the evaluation results, each dataset is divided into a training set, a validation set, and a test set in an 8:1:1 ratio. Each dataset contains the following information:

[0090] 1. Multimodal content data: UGC data containing multiple modalities such as text descriptions, image content, and audio information, reflecting the diversity and complexity of social media content.

[0091] 2. User attribute information: Static features such as user ID, posting time, and geographic location are obtained through public APIs to assist in context modeling and similarity calculation.

[0092] 3. Engagement tags: Real engagement data for each UGC, including the number of likes, comments, and shares, serves as a supervisory signal for model training and evaluation.

[0093] The comparison methods include feature engineering methods (SVR, HyFea, MFTM), deep learning methods (CLSTM, HMMVED, DLBA, MASSL, BLIP, CBAN), and retrieval base methods (NIPA, MMRA). The evaluation metrics are MSE, MAE, and SRC.

[0094] Experimental results show that the method of this invention achieves optimal performance on all three datasets, significantly reducing MSE and MAE while improving SRC. In scenarios with high noise levels and severe information loss, the method of this invention reduces prediction error by an average of 7% to 15% compared to the existing best methods.

[0095] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A user engagement prediction method based on distillation multimodal retrieval enhancement, characterized in that, Includes the following steps: Step 1: Construct a retrieval target query based on the fusion of user-generated content (UGC) with visual modality, text modality, and additional descriptive information. Input each UGC representation into the UGC feature encoder and obtain a retrieval distillation vector that minimizes redundant information through information bottleneck constraints. Step 2: The target query and candidate UGCs after distillation are scored using a ranking function, and the top-K with the highest search scores are selected as the final candidate UGCs; Step 3: The final candidate UGCs are used as nodes, and the publishing topic, time, and publisher are used as edge types to construct a heterogeneous graph. The node representations are learned through a heterogeneous convolutional network. Finally, the predicted multimodal UGC participation is obtained through a fully connected layer, and the prediction results are optimized using a loss function. Step 1 is described in detail as follows: Step 1.1: Multimodal UGC query construction; First, a pre-trained image-to-text model is used to extract semantic features of the visual modality, and these features are combined with the text modality to obtain merged text; Entities and actions are extracted from the merged text and integrated with additional descriptive information to obtain the target query; Step 1.2: Self-enhancing distillation mechanism; Each UGC and target query are input into the UGC feature encoder, which is an encoder composed of a three-layer fully connected network. The encoder sequentially processes the input UGC representations. This process requires information bottleneck constraints to obtain the most informative representation. Constraint loss It consists of two mutual information terms: in, Indicates mutual information, It is a balance trade-off coefficient. The parameters representing the UGC feature encoder, The original multimodal representation set representing all UGC, i.e. ; For participation tags; after distillation, the target UGC query is obtained. and distillation candidate UGC ; Maximize Reconstruction The distribution of the UGC feature encoder is denoted as: Assuming Following an isotropic Gaussian distribution, it is represented as ,in, It is the mean of this distribution. It is the variance vector of this distribution. Given the covariance matrix of this distribution, transform this mutual information term and derive its lower bound: in, Participation tags Marginal probability distribution, Indicates label entropy, Indicates the joint distribution The expectation, this boundary term is included in the loss function of the pre-trained distillation model; Minimize regularization Regularization term The upper bound is: in, and Represent and Marginal probability distribution, express and The joint probability distribution, Indicates a given hour The true conditional probability distribution Indicates the joint distribution Expectations Calculate the KL divergence.

2. The user engagement prediction method based on distillation multimodal retrieval enhancement according to claim 1, characterized in that, Step 2 is described in detail below: A simplified BM25 sorting function is used for retrieving calculated scores and sorting selections: in Target UGC query Feature dimensions in It is an adjustable hyperparameter. Target UGC query In feature dimension The features represented on, Candidate UGC In feature dimension Feature representation on, IDF yes Inverse frequency in UGC; Jaccard correlation is defined as follows: Where, when two vectors When they are the same, It equals 1 if it is equal to 1, otherwise it is 0. After calculating the retrieval scores of the target UGC and candidate UGCs, the top-K candidate UGCs with the highest retrieval scores are selected, denoted as _____. .

3. The user engagement prediction method based on distillation multimodal retrieval enhancement according to claim 2, characterized in that, Step 3 is as follows: Step 3.1: Heterogeneous UGC Graph Construction; The top-K UGC content obtained from the selected search is divided into two types of nodes: visual nodes. and text nodes The entire UGC content of the target is treated as a single user node. Three types of relationships are designed by combining additional descriptive information: topic relationship, time relationship, and publisher relationship. Step 3.2: Design a heterogeneous convolutional network to learn node representations by considering different types of nodes and their various relationships; for each node , Define the heterogeneous convolutional network for the set of nodes. The representation of nodes in the layer is updated, and the update rules are derived from neighboring nodes. Aggregate information, Given a set of neighboring nodes, consider the edges connecting them. Type: in, , It is the first Layer nodes The representation of u and These are the node-specific weight matrix and the matrix corresponding to the edge type. The weight matrix represents the node. and The relationship between them A set of edge types, It is a non-linear activation function, ReLU. Step 3.3: The representations learned from the heterogeneous graph are passed through two fully connected layers to predict the engagement level of each target UGC. We introduce mean square error as the optimization loss function.

Images