A User Preference Prediction Method Based on Data Debiasing
By expanding the differences through embedding layers, bias removal, and feature selection layers, and combining factorization machines to optimize model parameters, the problem of duration bias in user preference prediction is solved, achieving more accurate user preference prediction and recommendation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHONGQING UNIV OF POSTS & TELECOMM
- Filing Date
- 2024-07-30
- Publication Date
- 2026-05-26
Smart Images
Figure CN118916761B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of Internet user preference prediction technology, specifically relating to a user preference prediction method based on data debiasing. Background Technology
[0002] Recommender systems are decision support systems that predict a user's interest in specific items and help the user select content of interest based on information such as historical behavior and preferences. Click-through rate (CTR) prediction aims to forecast the probability of a user clicking on an ad or item, and it is crucial for many online applications such as online advertising and recommendation systems. However, existing datasets are typically observational, analyzing user interests and preferences based on historical interactions. Such datasets often contain various biases, such as user selection bias, system exposure bias, and content bias. Models trained on biased data cannot capture the user's true interests; therefore, mitigating the influence of biased data is key to improving cache hit rates.
[0003] For user preference prediction, capturing users' true interests from biased datasets is a crucial issue that has been extensively studied. However, a method for handling duration bias in video datasets remains lacking. One effective method is to draw a complete causal graph and then use the backdoor adjustment principle to remove the influence of confounding factors. However, this method still has limitations. First, the confounding factors identified through causal graphs are primarily observable, while there is no effective way to remove unobservable confounding factors. Summary of the Invention
[0004] To address the shortcomings of existing technologies, this invention proposes a user preference prediction method based on data debiasing. The method includes: acquiring video data and user data; inputting the video data and user data into a trained user preference prediction model to obtain user preference prediction results.
[0005] The training process for a user preference prediction model includes:
[0006] S1: Obtain video data and user data for training; process the video data and user data using an embedding layer to obtain user embeddings and video embeddings;
[0007] S2: Perform bias removal processing on user embeddings and video embeddings to obtain bias-removed user embeddings and bias-removed video embeddings;
[0008] S3: A feature filtering layer is used to process user embedding, video embedding, debiased user embedding, and debiased video embedding to obtain the importance weights of user-specific features and video-specific features.
[0009] S4: The importance weights of user-specific features and video-specific features are processed using a factorization machine to obtain the user interest representation vector;
[0010] S5: Input the user interest representation vector into the output layer for processing to obtain the user preference prediction result;
[0011] S6: Calculate the model cross-entropy loss and adjust the model parameters based on the model cross-entropy loss to obtain a trained user preference prediction model.
[0012] Preferably, the process of removing bias from user embeddings and video embeddings includes: training the embedding layer and calculating the pairwise ranking loss; adjusting the embedding layer parameters according to the pairwise ranking loss to obtain the trained embedding layer; and removing bias from the user embeddings and video embeddings output by the trained embedding layer according to the user embeddings and video embeddings obtained during training to obtain the debiased user embeddings and debiased video embeddings.
[0013] Furthermore, the formulas for debiased user embedding and debiased video embedding are obtained as follows:
[0014]
[0015] in, This indicates the debiased user embedding of video i, Q i This represents the video embedding of video i, where α1 is the first control parameter. This represents the average video embedding obtained from the embedding layer over multiple iterations. P represents the debiased user embedding of user u. u This represents the user embedding of user u, and α2 is the second control parameter. This represents the average user embedding obtained from the embedding layer over multiple iterations.
[0016] Preferably, the pairwise sorting loss is:
[0017]
[0018] Among them, L BPR This represents the pairwise sorting loss, where σ represents the sigmoid activation. This represents the matching score between user u and positive sample video i. λ represents the matching score between user u and negative sample video j. Θ Let Θ represent the learning rate, Θ represent the embedding layer parameters, and D represent the learning rate. s Let (u,i,j) represent the set of triples, where (u,i,j) represents the triple of user u, positive sample video i, and negative sample video j.
[0019] Preferably, the process of obtaining the importance weights of user-specific features and video-specific features includes:
[0020] A gated network based on MLP is used to process user embedding and video embedding respectively to obtain user gate weights and video gate weights;
[0021] By stitching together all debiased user embeddings, we obtain the stitched debiased user embeddings; by stitching together all debiased video embeddings, we obtain the stitched debiased video embeddings.
[0022] The importance weights of user-specific features are calculated based on user gating weights and user embeddings after stitching and debiasing; the importance weights of video-specific features are calculated based on video gating weights and video embeddings after stitching and debiasing.
[0023] Furthermore, the formulas for calculating the importance weights of user-specific features and video-specific features are expressed as follows:
[0024] h1=2σ(g1)⊙e
[0025] h2=2σ(g2)⊙f
[0026] Where h1 represents the important weight of user-specific features, g1 represents the user gating weight, e represents the concatenated debiased user embedding, σ represents the sigmoid activation function, ⊙ represents the element-wise multiplication at corresponding positions, h2 represents the important weight of video-specific features, g2 represents the video gating weight, and f represents the concatenated debiased video embedding.
[0027] Preferably, the formula for the factorization machine to process the importance weights of user-specific features and video-specific features is as follows:
[0028]
[0029] z i =h 1,i +h 2,i
[0030] in, Let θ0 represent a constant vector representing user interests. i For linear network parameters, z i Let z represent the i-th concatenation feature. j Let j represent the j-th concatenated feature, and k represent the number of feature types, <θ i ,θ j > represents the cross-model parameter, h 1,i h represents the i-th feature representation, which indicates the importance weight of a user-specific feature. 2,i The i-th feature represents the importance weight of a specific feature in the video.
[0031] Preferably, the processing of the user interest representation vector by the output layer includes: processing the user interest representation vector using the sigmoid function to obtain the user preference prediction result.
[0032] Preferably, the cross-entropy loss is:
[0033]
[0034] Where loss represents cross-entropy loss, and N represents the number of training samples. Let y represent the predicted click probability of the j-th sample. j This represents the actual tag clicked by the user in the j-th sample.
[0035] The beneficial effects of this invention are as follows: By using categorical and numerical feature embedding representations and debiasing of the original data before embedding, this invention effectively increases the accuracy of the embedded vector representation of users and video content, solving the problem of inaccurate predictions or monotonous recommended content caused by the over-reliance on initial data in recommendation models. Simultaneously, it employs stream-specific feature selection to amplify the differences between two streams, thereby generating differentiated feature inputs. This method more efficiently utilizes the information from feature combinations, avoiding the inefficiency or even ineffectiveness caused by the crossover of invalid features during the neural network learning process. Finally, the introduction of a factorization machine to fully learn the crossover of low-order and high-order features enables accurate and effective prediction of user preferences through feature crossover combinations, improving the accuracy of user preference prediction. Attached Figure Description
[0036] Figure 1 This is a structural diagram of the user preference prediction model in this invention;
[0037] Figure 2 This is a schematic diagram of the feature selection layer in the user preference prediction model of this invention. Detailed Implementation
[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0039] This invention proposes a user preference prediction method based on data debiasing, such as... Figure 1As shown, the method includes: acquiring video data and user data; inputting the video data and user data into a trained user preference prediction model to obtain user preference prediction results. The user preference prediction model includes an embedding layer, an interest capture layer, a feature selection layer, and an output layer.
[0040] The training process for a user preference prediction model includes:
[0041] S1: Obtain video data and user data for training; process the video data and user data using an embedding layer to obtain user embeddings and video embeddings.
[0042] User data includes users' age, activity level, and occupation; video data includes category, duration, and author; training tags are whether users interacted with the video (clicks), such as watching, commenting, or downloading.
[0043] An embedding layer is used to process video data and user data, specifically:
[0044] Embedding layers are commonly used neural network layers in deep learning. Their main function is to transform low-dimensional or high-dimensional features into a single dimension, facilitating subsequent vector dimension unification. They classify different types of features, encode and vectorize these features, resulting in user embeddings and video embeddings, represented as follows:
[0045] P m =[α1,α2,...,α i ,...,α M ]
[0046] Q n =[β1,β2,...,β j ,...,β N ]
[0047] Where M and N represent the total number of user features and video features; for category features, α i For a one-hot vector, α represents a discrete feature. i It is a scalar value. Similarly, when the j-th feature of the video content is a sparse feature, β j One-hot vector, when it is a non-class feature, β j It is a scalar value. P m For the embedding representation of the m-th user, Q n This is the embedded representation of the nth video content.
[0048] S2: Perform bias removal processing on the user embedding and video embedding to obtain bias-removed user embedding and bias-removed video embedding.
[0049] The embedding layer is trained, specifically:
[0050] Users are paired with video content, and samples are categorized as positive or negative based on whether the user interacted with the video content. Specifically, if a user interacts with the video (e.g., watching, commenting, downloading), the sample pair is considered positive; otherwise, it's considered negative. This distinction is made because, for gradient updates, the model assigns greater weight to positive samples than to negative samples. Therefore, the updated gradient direction tends to favor the positive gradient direction, leading to a bias in the gradient direction learned by the model.
[0051] To reduce the impact of positive gradients on the direction of the resultant gradient, pairwise ranking loss is used to measure the difference between positive and negative samples, and the vector representation parameters of users and video content are updated accordingly. The pairwise ranking loss is calculated as follows:
[0052]
[0053] Among them, L BPR Let x represent the pairwise sorting loss, and σ represent the sigmoid activation, σ(x) = 1 / (1+exp(-x)); This represents the matching score between user u and positive sample video i. λ represents the matching score between user u and negative sample video j; Θ Θ represents the learning rate, used to control how quickly the embedding layer parameters are adjusted; D represents the embedding layer parameters. s Let (u,i,j) represent the set of triples, where (u,i,j) represents the triple of user u, positive sample video i, and negative sample video j.
[0054] By adjusting the embedding layer parameters using pairwise sorting loss, a well-trained embedding layer can be obtained.
[0055] For the embedding layer parameter Θ, the gradient of the BPR loss can be written as:
[0056]
[0057] When updating the gradient of the embedding vector:
[0058]
[0059] Then, in the t-th iteration, the forward video embedding... Negative video embedding and user embedding The latent vector update is as follows:
[0060]
[0061] Where lr is the learning rate.
[0062] Video embedding and user embedding can be broken down as follows:
[0063]
[0064] in, Used to capture the impact of video duration. Used to capture the true characteristics of content It captured user activity. This indicates that the user is consistent with other mainstream users. This represents the user's true interests. Because active users tend to engage in interactions that are more observed, a product form is used.
[0065] Because active users will have more observed interactions with video content, their cumulative gradients based on this content may be larger than those of less active users. For example, if users u1 and u2 have similar preferences for video content, but u1 is more active than u2, then the update directions of u1 and u2's embedding vectors will be similar, and u1's vector norm will be much larger. Such a large embedding will increase the gradient updates for its corresponding items, exacerbating the imbalance between positive and negative gradients. To reduce the propagation of bias among highly active users, each user should be embedded into a unit vector during the training phase.
[0066]
[0067] To mitigate the impact of video length, for a long, popular video project, positive interactions with users are frequently observed in the training samples, causing the learned embeddings to shift towards positive interactions. To correct this shift, the overall gradient is adjusted by correcting the accumulated positive gradient, thus achieving debiasing of user and video embeddings. Therefore, in the debiasing calculation layer, the formula for debiasing the user and video embeddings output by the embedding layer after training, based on the user and video embeddings obtained during training, is as follows:
[0068]
[0069]
[0070] in, This indicates the debiased user embedding of video i, Q i Let α1 represent the video embedding of video i, and let α1 represent the first control parameter, which is a scalar used to control the projection size of the initial video embedding in the average video embedding direction. This represents the average video embedding obtained from the embedding layer over multiple iterations of the training process. P represents the debiased user embedding of user u. uLet α2 represent the user embedding of user u, and let α2 represent the second control parameter, which is a scalar used to control the projection size of the initial user embedding in the average user embedding direction. This represents the average user embedding obtained from the embedding layer during multiple iterations of the training process.
[0071] S3: A feature filtering layer is used to process user embedding, video embedding, debiased user embedding, and debiased video embedding to obtain the importance weights of user-specific features and video-specific features.
[0072] The feature selection layer amplifies the differences between two streams through stream-specific feature selection, thereby generating differentiated feature inputs. Feature inputs are reweighted through a stream-specific feature module, performing feature gating from different views to generate user-specific feature importance weights and video-specific feature importance weights, respectively. Specifically, such as... Figure 2 As shown:
[0073] A gated network based on MLP is used to process user embedding and video embedding separately to obtain user gate weights and video gate weights:
[0074] g1 = Gate1(x1), g2 = Gate2(x2)
[0075] Among them, Gate i This represents a gated network based on MLP, which will flow specific conditional features x i As input, and output element-level gating weights g i It allows for flexible selection of x from a set of user / video features. i x1 is user embedding, x2 is video embedding; or x i Set as a learnable parameter.
[0076] splicing all biased user embeddings Get the stitched and de-biased user embedding e; all de-biased video embeddings of the stitched video. The resulting stitched and de-biased video embedding is f.
[0077] The importance weights of user-specific features are calculated based on user gating weights and concatenated, bias-free user embeddings; the importance weights of video-specific features are calculated based on video gating weights and concatenated, bias-free video embeddings; the calculation formula is as follows:
[0078] h1=2σ(g1)⊙e
[0079] h2=2σ(g2)⊙f
[0080] Where h1 represents the user-specific feature importance weight, g1 represents the user gating weight, e represents the concatenated debiased user embedding, σ represents the sigmoid activation function, and the feature importance weight is transformed to the range [0,2] by using the sigmoid function σ and multiplying by 2, with an average value of 1; ⊙ represents the element-wise multiplication at corresponding positions, h2 represents the video-specific feature importance weight, g2 represents the video gating weight, and f represents the concatenated debiased video embedding.
[0081] S4: Use a factorization machine to process the importance weights of user-specific features and video-specific features to obtain the user interest representation vector.
[0082] The interest capture layer employs a factorization machine, which is suitable for estimating the interactions between variables in sparse scenarios. Factorization machines destroy the independence of interaction parameters by decomposing them. Typically, this means that data from one interaction also helps in estimating the parameters of related interactions. The factorization machine for second-order interactions is calculated as follows:
[0083]
[0084] z i =h 1,i +h 2,i
[0085] in, Let θ0 represent a constant vector representing user interests. i The linear network parameters are used to learn first-order features, z i Let z represent the i-th concatenation feature. j Let j represent the j-th concatenated feature, and k represent the number of feature types, <θ i ,θ j > is the cross-model parameter used to learn second-order feature interactions, h 1,i h represents the i-th feature representation, which indicates the importance weight of a user-specific feature. 2,i The i-th feature represents the importance weight of a specific feature in the video.
[0086] S5: Input the user interest representation vector into the output layer for processing to obtain the user preference prediction result.
[0087] The output layer uses the sigmoid function to process the user interest representation vector to obtain the user preference prediction result, which is the probability of the user clicking on the video content.
[0088] S6: Calculate the model cross-entropy loss and adjust the model parameters based on the model cross-entropy loss to obtain a trained user preference prediction model.
[0089] Backpropagation is performed using cross-entropy loss to optimize model parameters. When the model loss converges or the maximum number of iterations is reached, training is stopped, the model parameters are preserved, and a well-trained user preference prediction model is obtained.
[0090] The cross-entropy loss is:
[0091]
[0092] Where loss represents cross-entropy loss, and N represents the number of training samples. Let y represent the predicted click probability of the j-th sample. j This represents the actual tag clicked by the user in the j-th sample.
[0093] After the model is trained, video data and user data are acquired and input into the trained user preference prediction model to obtain user preference prediction results. Based on the user preference prediction results, video recommendations are made to users to achieve accurate recommendations.
[0094] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A user preference prediction method based on data debiasing, characterized in that, include: Acquire video data and user data; The video data and user data are input into the trained user preference prediction model to obtain the user preference prediction results; The training process for a user preference prediction model includes: S1: Obtain video and user data for training; process the video and user data using an embedding layer to obtain user embeddings and video embeddings; user data includes user age, activity level, and occupation, and video data includes category, duration, and author; training labels are whether the user interacted with the video; S2: Perform bias removal processing on user embeddings and video embeddings to obtain bias-removed user embeddings and bias-removed video embeddings. The process of bias removal processing for user embeddings and video embeddings includes: training the embedding layer and calculating the pairwise ranking loss; adjusting the embedding layer parameters according to the pairwise ranking loss to obtain the trained embedding layer; and performing bias removal processing on the user embeddings and video embeddings output by the trained embedding layer based on the user embeddings and video embeddings obtained during training to obtain bias-removed user embeddings and bias-removed video embeddings. The formulas for obtaining bias-removed user embeddings and bias-removed video embeddings are as follows: ； ； wherein, denotes a debiased user embedding for video i, denotes a video embedding for video i, is a first control parameter, denotes an average video embedding obtained by the embedding layer in a plurality of iterations, denotes a debiased user embedding for user u, denotes a user embedding for user u, is a second control parameter, denotes an average user embedding obtained by the embedding layer in a plurality of iterations; S3: A feature filtering layer is used to process user embedding, video embedding, debiased user embedding, and debiased video embedding to obtain the importance weights of user-specific features and video-specific features. S4: The importance weights of user-specific features and video-specific features are processed using a factorization machine to obtain the user interest representation vector; S5: Input the user interest representation vector into the output layer for processing to obtain the user preference prediction result; S6: Calculate the model cross-entropy loss and adjust the model parameters based on the model cross-entropy loss to obtain a trained user preference prediction model.
2. The user preference prediction method based on data debiasing according to claim 1, characterized in that, The pairwise sorting loss is: ； wherein, represents a pair-wise ranking loss, represents activation, represents a matching score of a user u and a positive sample video i, represents a matching score of a user u and a negative sample video j, represents a learning rate, represents an embedding layer parameter, represents a set of triplets, represents a triplet of a user u, a positive sample video i, and a negative sample video j.
3. The user preference prediction method based on data debiasing according to claim 1, characterized in that, The process of obtaining the importance weights of user-specific features and video-specific features includes: A gated network based on MLP is used to process user embedding and video embedding respectively to obtain user gate weights and video gate weights; By stitching together all debiased user embeddings, we obtain the stitched debiased user embeddings; by stitching together all debiased video embeddings, we obtain the stitched debiased video embeddings. The importance weights of user-specific features are calculated based on user gating weights and user embeddings after stitching and debiasing; the importance weights of video-specific features are calculated based on video gating weights and video embeddings after stitching and debiasing.
4. The user preference prediction method based on data debiasing according to claim 3, characterized in that, The formulas for calculating the importance weights of user-specific features and video-specific features are expressed as follows: ； ； in, Indicates the importance weight of specific user features. Indicates the user gating weight. This indicates that the splicing process removes biased user embedding. express Activation function This indicates that elements at corresponding positions are multiplied. Indicates the importance weight of specific video features. Indicates the video gating weight. This indicates the embedding of debiased video.
5. The user preference prediction method based on data debiasing according to claim 1, characterized in that, The formula for factorization to process the importance weights of user-specific features and video-specific features is as follows: ； ； in, Represents a vector representing user interests. Represents a constant. For linear network parameters, Indicates the first A splicing feature, Indicates the first A splicing feature, Indicates the number of feature types. For cross-model parameters, The first, representing the importance weight of user-specific features Each feature represents, The first, representing the importance weight of specific video features. Each feature is represented.
6. The user preference prediction method based on data debiasing according to claim 1, characterized in that, The output layer's processing of the user interest representation vector includes: employing... The function processes the user interest representation vector to obtain the user preference prediction result.
7. The user preference prediction method based on data debiasing according to claim 1, characterized in that, The cross-entropy loss is: ； in, Represents cross-entropy loss, Represents the number of training samples. This represents the predicted click probability of the j-th sample. This indicates the actual tag clicked by the user in the j-th sample.