Retention prediction model training method, content recommendation method, device and apparatus

Abstract

This disclosure relates to a retention prediction model training method, content recommendation method, apparatus, and device. The method includes: acquiring multiple training samples, each training sample including a sample feature vector of a first feature dimension and its corresponding medium- to long-term reference retention label; based on each sample feature vector, using a censoring estimation model, predicting the censoring weights corresponding to the sample feature vectors, and determining the sample mask value corresponding to the sample feature vectors based on the censoring weights; inputting each sample feature vector into an initial prediction model for processing to generate the predicted retention probability corresponding to the respective training sample; based on the reference retention label, predicted retention probability, censoring weights, and sample mask value corresponding to each training sample, and the number of samples in each training sample, determining the retention loss function value, and iteratively training the initial prediction model using the retention loss function value to obtain the retention prediction model. This improves the accuracy and stability of the model's medium- to long-term retention prediction.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a retention prediction model training method, content recommendation method, apparatus and device. Background Technology

[0002] With the rapid development of various content platforms, platforms are gradually using user retention rate as a core indicator to measure content quality, content recommendation effectiveness, and user stickiness.

[0003] Current retention analysis models primarily use content as the statistical object to calculate the next-day retention probability of individual content (such as videos or news), measuring its attractiveness or recommendation effectiveness. However, this model only reflects short-term (next-day) retention and cannot reflect the impact of content on users' medium- to long-term retention behavior. To predict medium- to long-term retention, the retention observation period can be extended. However, training the model using training samples with extended observation periods can lead to the problem of right censoring, meaning that users who have not yet churned during model training cannot be observed at their actual churn time. Furthermore, significant differences in user entry time and usage cycle can result in inconsistent observable durations for different users, creating unequal observation windows. Both right censoring and unequal user observation cycles introduce statistical bias into the training samples, leading to inaccurate medium- to long-term retention predictions. Summary of the Invention

[0004] To address the aforementioned technical issues, this disclosure provides a retention prediction model training method, a content recommendation method, an apparatus, and a device.

[0005] In a first aspect, embodiments of this disclosure provide a method for training a retention prediction model, the method comprising: Multiple training samples are obtained; wherein, the training samples include a sample feature vector of a first feature dimension related to the content type to be evaluated and a reference retention label corresponding to the sample feature vector; the reference retention label is used to characterize the user's retention status within a first scrolling window, the first scrolling window being greater than or equal to two days; Based on the sample feature vectors, a censoring estimation model is used to determine the censoring weights corresponding to the sample feature vectors, and the sample mask value corresponding to the sample feature vectors is determined based on the censoring weights; wherein, the sample mask value is used to characterize whether the training sample corresponds to observation data or whether the reference retention label corresponding to the training sample is a pseudo label. Each of the sample feature vectors is input into the initial prediction model for processing to generate the predicted retention probability corresponding to the training sample; wherein, the predicted retention probability represents the retention probability of the user for the content corresponding to the sample feature vector, as output by the initial prediction model; Based on the reference retention label, the predicted retention probability, the pruning weight, and the sample mask value corresponding to each training sample, as well as the number of samples in each training sample, the retention loss function value is determined, and the model parameters of the initial prediction model are updated by backpropagation using the retention loss function value. If the convergence condition for model training is not met, the process returns to the step of inputting the feature vectors of each sample into the initial prediction model for processing and generating the predicted retention probability corresponding to the training sample, until the convergence condition is met and the retention prediction model is obtained.

[0006] Secondly, this disclosure also provides a content recommendation method, which includes: In response to a content recommendation request, obtain candidate content data for multiple candidate content items and historical behavior data within a preset time period prior to the current moment; Feature extraction is performed on the historical behavior data and the candidate content data according to the second feature dimension to generate a target feature vector; Based on the target feature vector, a retention prediction model is invoked to generate the target retention probability corresponding to each candidate content; wherein, the retention prediction model is obtained by pre-training using the retention prediction model training method described in any embodiment of this disclosure; Based on the retention probability of each target, the candidate content is sorted and filtered to obtain multiple recommended contents.

[0007] Thirdly, embodiments of this disclosure also provide a retention prediction model training apparatus, the apparatus comprising: The training sample acquisition module is used to acquire multiple training samples; wherein, the training samples include a sample feature vector of a first feature dimension related to the content type to be evaluated and a reference retention label corresponding to the sample feature vector; the reference retention label is used to characterize the user's retention status within a first scrolling window, the first scrolling window being greater than or equal to two days; The censoring weight determination module is used to determine the censoring weights corresponding to the sample feature vectors based on each sample feature vector and using a censoring estimation model, and to determine the sample mask value corresponding to the sample feature vectors based on the censoring weights; wherein, the sample mask value is used to characterize whether the training sample corresponds to observation data or whether the reference retention label corresponding to the training sample is a pseudo label. The predicted retention probability generation module is used to input the feature vectors of each sample into the initial prediction model for processing, and generate the predicted retention probability corresponding to the training sample; wherein, the predicted retention probability represents the retention probability of the user for the content corresponding to the feature vector of the sample, output by the initial prediction model. The model parameter update module is used to determine the retention loss function value based on the reference retention label, the predicted retention probability, the pruning weight, and the sample mask value corresponding to each training sample, as well as the number of samples of each training sample, and to use the retention loss function value to update the model parameters of the initial prediction model through backpropagation. The retention prediction model acquisition module is used to return to the step of inputting the feature vectors of each sample into the initial prediction model for processing and generating the predicted retention probability corresponding to the corresponding training sample if the convergence condition of the model training is not met, until the convergence condition is met and the retention prediction model is obtained.

[0008] Fourthly, embodiments of this disclosure also provide a content recommendation device, the device comprising: The data acquisition module is used to respond to content recommendation requests and acquire candidate content data of multiple candidate content and historical behavior data within a preset time period before the current moment; The target feature vector generation module is used to extract features from the historical behavior data and the candidate content data according to the second feature dimension to generate a target feature vector. The target retention probability generation module is used to generate the target retention probability corresponding to each of the candidate contents by calling the retention prediction model based on the target feature vector; wherein, the retention prediction model is obtained by pre-training using the retention prediction model training method described in any embodiment of this disclosure; The recommended content acquisition module is used to sort and filter the candidate content based on the retention probability of each target to obtain multiple recommended content.

[0009] Fifthly, embodiments of this disclosure also provide an electronic device, the electronic device comprising: Processor and memory; The processor executes the retention prediction model training method or content recommendation method described in any embodiment of this disclosure by calling the program or instructions stored in the memory.

[0010] Sixthly, embodiments of this disclosure also provide a computer-readable storage medium storing a program or instructions that cause a computer to execute the retention prediction model training method or content recommendation method described in any embodiment of this disclosure.

[0011] In a seventh aspect, embodiments of this disclosure also provide a computer program product, which is used in the retention prediction model training method or content recommendation method described in any embodiment of this disclosure.

[0012] The technical solution for training the retention prediction model in this embodiment can acquire multiple training samples. Each training sample includes a sample feature vector of a first feature dimension related to the content type to be evaluated and a reference retention label corresponding to the sample feature vector, wherein the reference retention label is used to characterize the user's medium- to long-term retention status. Based on each sample feature vector, a censoring estimation model is used to determine the censoring weights corresponding to the sample feature vectors, and a sample mask value is determined based on the censoring weights to characterize the effectiveness of the training samples. Each sample feature vector is input into an initial prediction model for processing to generate a predicted retention probability corresponding to the corresponding training sample. Based on the reference retention label, the predicted retention probability, the censoring weights, the sample mask value, and the number of samples in each training sample, a retention loss function value is determined, and the retention loss function value is used to adjust the error of the model parameters of the initial prediction model. The iterative update process of backpropagation continues until convergence is achieved, resulting in a retention prediction model. This model locates the reference retention labels of training samples to the user's medium- to long-term retention status, enabling the model to effectively assess the impact of the content being evaluated on the user's medium- to long-term retention. The corresponding censoring weights are then used in the loss function calculation to correct statistical biases in long-term retention prediction caused by right censoring in retention data and incomplete sample observations, thereby improving the accuracy and stability of the medium- to long-term reference retention labels. Furthermore, the sample mask values ​​corresponding to the training samples are also used in the loss function calculation. This effectively masks noisy or pseudo-label samples when the rolling time window of retention label statistics aligns with the actual observation time, allowing the model to learn effective signals from the actual observation distribution by updating gradients only on valid training samples. This improves the model's stability and generalization ability under non-uniform time sampling, further enhancing the accuracy and stability of the model's medium- to long-term retention prediction.

[0013] The content recommendation technical solution of this disclosure embodiment can respond to a content recommendation request by acquiring candidate content data of multiple candidate content to be evaluated and historical behavior data within a preset time period before the current moment; extracting features from the historical behavior data and the candidate content data according to a second feature dimension to generate a target feature vector; based on the target feature vector, calling the aforementioned trained retention prediction model to generate a target retention probability corresponding to each candidate content; and sorting and filtering each candidate content based on each target retention probability to obtain multiple recommended content. This realizes the use of the trained retention prediction model to infer the medium- and long-term retention of candidate content, improves the accuracy and stability of the target retention probability of candidate content, thereby improving the accuracy of content recommendation and the probability of users consuming recommended content, and further improving the probability of users' medium- and long-term retention. Attached Figure Description

[0014] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0015] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart illustrating a retention prediction model training method provided in an embodiment of this disclosure; Figure 2 A schematic diagram of the model structure of an initial prediction model / retention prediction model provided in an embodiment of this disclosure; Figure 3 A flowchart illustrating a content recommendation method provided in an embodiment of this disclosure; Figure 4 This is a schematic diagram of the structure of a retention prediction model training device provided in an embodiment of the present disclosure; Figure 5 This is a schematic diagram of the structure of a content recommendation device provided in an embodiment of the present disclosure; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0017] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be described in further detail below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0018] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0019] The following provides a detailed description of the retention prediction model training method provided in the embodiments of this disclosure.

[0020] The retention prediction model training method provided in this disclosure is mainly applicable to training scenarios for models that predict user retention for various types of content, such as training scenarios for models that predict the impact of content on user retention in the medium to long term. This retention prediction model training method can be executed by a retention prediction model training device, which can be implemented in software and / or hardware and can be integrated into an electronic device with certain data processing capabilities. This electronic device can be, for example, a laptop computer, a desktop computer, a server, or a server cluster.

[0021] Figure 1 This is a flowchart illustrating a retention prediction model training method provided in an embodiment of this disclosure. See also... Figure 1 The training method for this retention prediction model specifically includes: S110. Obtain multiple training samples; the training samples include sample feature vectors of the first feature dimension related to the content type to be evaluated and reference retention labels corresponding to the sample feature vectors. The reference retention labels are used to characterize the user's retention status within the first scrolling window, which is greater than or equal to two days.

[0022] The content type to be evaluated refers to the type of content being evaluated, such as long video, short video, music, or news. The first feature dimension is a pre-defined feature dimension used for retention analysis, which can be set according to business needs. For example, if the business needs to focus on overall user activity trends, the first feature dimension can be set as the user dimension; if the business needs to focus on the content itself, the first feature dimension can be set as the content dimension; if the business needs to focus on the impact of the content on users, the first feature dimension can be set as the user dimension, the content dimension, and the user's interaction dimension with the content. The sample feature vector is a feature vector obtained by standardizing the original sample data related to the first feature dimension. The reference retention label is an identifier of the actual retention status corresponding to the sample feature vector. In this embodiment, the reference retention label represents the user's medium- to long-term retention status, enabling the model to learn the predictive ability for medium- to long-term retention metrics. Therefore, the reference retention label is used to represent the user's retention status within the first scrolling window. Here, the first scrolling window is greater than or equal to two days to correspond to the medium- to long-term duration. The specific observation time range corresponding to the reference retention label depends on the date the training samples were acquired and the window size of the first scrolling window.

[0023] Specifically, before model training, the electronic device first obtains multiple training samples. These training samples can be obtained by cleaning, filtering, and extracting features from log data of durations such as six months, three months, or one month. To enable the model to meet the retention analysis granularity required by business needs, the training samples obtained in this embodiment may include sample feature vectors of the first feature dimension to achieve refined modeling. Furthermore, to enable the model to predict the medium- to long-term retention metrics of a certain content, the training samples obtained in this embodiment may include reference retention labels corresponding to the sample feature vectors, representing the medium- to long-term retention situation, as reference ground truth values ​​for model training.

[0024] S120. Based on the feature vectors of each sample, use the censoring estimation model to determine the censoring weights corresponding to the feature vectors of the samples, and determine the sample mask values ​​corresponding to the feature vectors of the samples based on the censoring weights.

[0025] The censoring estimation model is a pre-built model that estimates the probability of censoring in training samples (or whether the user corresponding to the training sample has been observed). It can be a logistic regression model, a lightweight multilayer perceptron, or a statistical model. Censoring weights are coefficients used to correct statistical biases caused by censored samples. A larger censoring weight corresponds to a higher probability of censoring (or a lower probability of being fully observed), and thus a higher representativeness of the training sample. The sample mask value is a numerical value used to characterize the effectiveness of the training sample. It can be one of a pre-defined discrete value or a value within a certain range (e.g., [0, 1]). Effectiveness can be characterized by whether the training sample corresponds to observed data (e.g., data recorded in logs) or whether the reference retention label corresponding to the training sample is a pseudo-label. For example, the presence of observed data or the absence of a pseudo-label indicates that the training sample is effective.

[0026] Specifically, because the reference retention labels are obtained through medium- to long-term observations, which may involve censoring and unequal observation lengths, the reference retention labels may be inaccurate and unstable. Therefore, in this embodiment, the censoring weights corresponding to each sample feature vector can be calculated to correct for potential biases in the training samples. Based on this, the sample mask values ​​corresponding to the sample feature vectors can be further calculated to determine the validity of the training samples, thereby eliminating the influence of noisy samples during model training.

[0027] In practice, the electronic device can input the feature vectors of each sample into the censoring estimation model. After processing by the model, the censoring weights corresponding to each sample feature vector are obtained. Then, through the pre-constructed correspondence between censoring weights and sample mask values ​​(such as a lookup table or a specific mathematical formula), the corresponding sample mask value is determined by the censoring weights corresponding to each sample feature vector.

[0028] In some embodiments, determining the sample mask value corresponding to the sample feature vector based on the censoring weights includes: if the reciprocal of the censoring weights is less than or equal to a set threshold, then the sample mask value corresponding to the sample feature vector is determined to be 0; if the reciprocal of the censoring weights is greater than the set threshold, then the sample mask value corresponding to the sample feature vector is determined to be 1.

[0029] The threshold is a pre-set small value, such as 0 or a value close to 0.

[0030] Specifically, the censoring weight represents the correction strength for censoring bias in training samples. The smaller the censoring weight, the smaller the censoring bias in the training samples, and the greater the probability that the corresponding training sample is observed completely, thus increasing the probability that the training sample is a valid sample. Therefore, the sample mask value and the censoring weight are inversely proportional. In this embodiment, the reciprocal of the censoring weight can be used to determine the sample mask value. Furthermore, the purpose of the sample mask value in this embodiment is to efficiently distinguish between valid samples and noise samples during model training; therefore, the sample mask value has discrete values ​​of 0 and 1 to represent noise samples and valid samples respectively. Based on this, after obtaining the censoring weight of the training samples, the electronic device can compare the reciprocal of the censoring weight with a set threshold. If the reciprocal of the censoring weight is less than or equal to the set threshold, it means that the probability of the user corresponding to the training sample being fully observed is very small. The training sample is very likely to have no observation data or the reference retention label is a pseudo label. In this case, the sample mask value can be directly set to 0. Conversely, if the reciprocal of the censoring weight is greater than the set threshold, it means that the probability of the user corresponding to the training sample being fully observed is relatively large. The training sample is very likely to have observation data or the reference retention label is not a pseudo label. In this case, the sample mask value can be directly set to 1.

[0031] In some embodiments, based on each sample feature vector, the censoring weights corresponding to the sample feature vectors are determined using a censoring estimation model, including: inputting the user-related feature vectors from each sample feature vector into the censoring estimation model, outputting the user observation probability corresponding to each sample feature vector; and determining the reciprocal of the user observation probability as the censoring weight corresponding to the corresponding sample feature vector.

[0032] The user observation probability refers to the probability that the user corresponding to the sample feature vector is effectively observed in the first scrolling window. It measures the likelihood that the user's true behavior can be captured and is a key indicator for correcting censoring bias. The user observation probability ranges from (0, 1) (a probability greater than 0 means that the sample is included in the analysis and there is at least a very small chance of observation; a probability equal to 1 means that the user's target event / target behavior can be observed 100%).

[0033] Specifically, in this embodiment, the censoring estimation model can be either the model corresponding to the Inverse Probability Weighting (IPW) algorithm or the model corresponding to the Inverse Probability Censoring Weighted (IPCW) algorithm. In this way, user-related feature vectors (such as video exposure rates or conversion rates, etc., features related to user video viewing behavior) from each sample feature vector can be input into the censoring estimation model. After processing by the model, the predicted user observation probability for each training sample can be output. Then, based on the aforementioned relationship between the user observation probability and the censoring weights, the reciprocal of the user observation probability can be calculated as the censoring weight corresponding to the corresponding sample feature vector.

[0034] S130. Input the feature vectors of each sample into the initial prediction model for processing to generate the predicted retention probability corresponding to the training sample.

[0035] The initial prediction model is an untrained initial model with the ability to evaluate content according to specific prediction metrics. These prediction metrics include, at a minimum, user retention metrics. The predicted retention probability is the retention probability calculated by the initial prediction model during model training. It represents the probability that a user will retain the content corresponding to the sample feature vector, and can also be understood as the likelihood that the content corresponding to the sample feature vector will attract the user to continue staying.

[0036] Specifically, the electronic device can input the feature vectors of each sample into the initial prediction model for content evaluation processing according to the operation requirements of the initial prediction model, and output the predicted retention probability corresponding to each training sample.

[0037] S140. Based on the reference retention label, predicted retention probability, pruning weight, and sample mask value corresponding to each training sample, as well as the number of samples for each training sample, determine the retention loss function value, and use the retention loss function value to update the model parameters of the initial prediction model through backpropagation.

[0038] The retention loss function value is the value of the loss function for the retention metric during model training.

[0039] Specifically, regarding the retention metric, this embodiment of the disclosure can construct a retention loss function by introducing a sample mask variable on the basis of IPCW-weighted binary cross-entropy, according to the following formula, so as to ensure that the loss value is calculated only for valid training samples, thereby shielding the influence of noisy samples during model training and improving the stability and generalization ability of the model under non-uniform time sampling.

[0040] ; in, L This represents the retention loss function value; N Indicates the number of samples; Indicates the first i The sample mask value corresponding to each training sample; Indicates the first i Censoring weights corresponding to each training sample; Indicates the first i Each training sample corresponds to a reference retention label (e.g., 1 represents retention, 0 represents churn); Indicates the first i The predicted retention probability corresponding to each training sample.

[0041] The electronic device can substitute the reference retention label, predicted retention probability, pruning weight, and sample mask value of each training sample, along with the number of samples for each training sample, into the aforementioned retention loss function to calculate the retention loss function value for this round of model training. Then, the electronic device can use the retention loss function value to update the model parameters of the initial prediction model through error backpropagation, obtaining new model parameters and completing this round of model training.

[0042] S150. If the convergence condition for model training is not met, return to the step of inputting the feature vectors of each sample into the initial prediction model for processing and generating the predicted retention probability corresponding to the training sample, until the convergence condition is met and the retention prediction model is obtained.

[0043] The convergence condition is a pre-set condition for determining whether the model training has ended. For example, it can be the number of training rounds or the convergence threshold corresponding to the loss function value.

[0044] Specifically, after completing one round of model training, the electronic device can determine whether the current model's training state has reached the convergence condition. If the convergence condition has not been reached, it returns to execute S130 to reuse the aforementioned training samples for the next round of model training. If the convergence condition has been reached, the last updated model parameters are fused into the initial prediction model to obtain the trained retention prediction model.

[0045] The retention prediction model training method provided in this disclosure can acquire multiple training samples. Each training sample includes a sample feature vector of a first feature dimension related to the content type to be evaluated and a reference retention label corresponding to the sample feature vector. The reference retention label is used to characterize the user's medium- to long-term retention status. Based on each sample feature vector, a censoring estimation model is used to determine the censoring weights corresponding to the sample feature vectors, and the sample mask value corresponding to the sample feature vectors is determined based on the censoring weights to characterize the effectiveness of the training samples. Each sample feature vector is input into an initial prediction model for processing to generate the predicted retention probability corresponding to the corresponding training sample. Based on the reference retention label, predicted retention probability, censoring weights, and sample mask value corresponding to each training sample, as well as the number of samples in each training sample, a retention loss function value is determined. The retention loss function value is used to iteratively update the model parameters of the initial prediction model through backpropagation until a convergence threshold is reached. The model obtains a retention prediction model. It locates the reference retention labels of training samples to the user's medium- to long-term retention status, enabling the model to effectively assess the impact of the content being evaluated on the user's medium- to long-term retention. The corresponding censoring weights are then used in the loss function calculation to correct statistical biases in long-term retention prediction caused by right censoring in retention data and incomplete sample observations, thereby improving the accuracy and stability of the medium- to long-term reference retention labels. Furthermore, the sample mask values ​​corresponding to the training samples are also used in the loss function calculation. Under the condition that the rolling time window of retention label statistics is aligned with the actual observation time, noisy or pseudo-label samples are effectively masked, allowing the model to learn effective signals from the actual observation distribution by only updating gradients on valid training samples. This improves the model's stability and generalization ability under non-uniform time sampling, further enhancing the accuracy and stability of the model's medium- to long-term retention prediction.

[0046] In some embodiments, the initial prediction model can be a nonlinear modeling framework based on a multilayer perceptron (MLP), meaning the initial prediction model includes a feature sharing layer and detection heads corresponding to multiple content evaluation metrics. The feature sharing layer is used to fuse various features to obtain a comprehensive feature vector (i.e., a combined feature vector). The detection heads are used to calculate the metric values ​​of the corresponding content evaluation metrics. These content evaluation metrics include retention metrics and other metrics. These other metrics reflect the degree of user interest in the content corresponding to the sample feature vector. Therefore, these other metrics can be jointly determined by the importance of metrics reflecting user interest in the content and business requirements.

[0047] For example, other metrics include at least one of click-through rate (CTR), completion rate (CR), and content revenue metrics. CTR is a quantitative metric used to measure the frequency with which users click on content (such as videos, advertisements, etc.) corresponding to a sample feature vector. It is determined by calculating the ratio of clicks to impressions. CTR reflects the attractiveness of content to users; a higher CTR indicates a higher level of user interest. CR measures the complete viewing rate of video content. It is the ratio of the number of users who fully watched the video to the total number of users who watched it, or the ratio of the actual playback time to the total video length. CR reflects the attractiveness of video content and the degree of user interest; a higher CR indicates a higher level of user interest. Content revenue metrics are quantitative metrics used to measure the economic benefits generated by content, and may encompass one or more revenue-related metrics. Content revenue metrics reflect the commercial value of content; a higher content revenue metric indicates that the content is more attractive to users.

[0048] See Figure 2 For video content evaluation scenarios, the initial prediction model includes a feature sharing layer 210, a retention probability detection head 220, a click-through rate detection head 230, a playback completion rate detection head 240, and a content revenue detection head 250. The feature sharing layer 210 contains multiple (e.g., m) fully connected neural networks, and each detection head can adopt network structures from related technologies, such as expert units, gating units, and tower units.

[0049] Based on the above model structure, the internal processing of the initial prediction model is as follows: the sample feature vector is input into the feature sharing layer for feature fusion processing to generate a comprehensive feature vector; the comprehensive feature vector is then input into each detection head for evaluation of the corresponding indicators to generate the predicted retention probability and other predicted indicator values ​​corresponding to the training samples.

[0050] Among these, other predictive metric values ​​correspond to other metrics in each content evaluation metric. Therefore, other predictive metric values ​​can include at least one of predicted click-through rate, predicted play completion rate, and predicted content revenue.

[0051] Specifically, the initial prediction model first inputs the sample feature vector into the feature sharing layer 210. Through inter-layer nonlinear mapping of a multi-layer fully connected neural network, it performs fusion processing between features of the first feature dimension to generate a comprehensive feature vector representing the correlation between users and content. Then, the comprehensive feature vector is input into the retention probability detection head 220, click-through rate detection head 230, playback completion rate detection head 240, and content revenue detection head 250, respectively. The model outputs the predicted retention probability, predicted click-through rate, predicted playback completion rate, and predicted content revenue for the content (such as a video) corresponding to the sample feature vector. Here, the predicted click-through rate, predicted playback completion rate, and predicted content revenue correspond to the other predicted metric values.

[0052] Based on the above embodiments, the above-mentioned update process of using the retention loss function value to perform error backpropagation on the model parameters of the initial prediction model includes: determining the comprehensive loss function value based on the loss function value corresponding to the retention loss function value and other prediction index values, and using the comprehensive loss function value to perform error backpropagation update process on the model parameters of the initial prediction model.

[0053] Specifically, if the initial prediction model outputs multiple predicted indicator values, a corresponding loss function value can be calculated for each predicted indicator value. For other predicted indicator values, the corresponding loss function values ​​can be calculated according to the loss functions in relevant technologies. Then, the retention loss function value and the loss function values ​​corresponding to other predicted indicator values ​​are weighted and fused (the weights can be customized according to business needs) to calculate the comprehensive loss function value. Afterward, the comprehensive loss function value is used to update the model parameters through error backpropagation. In this way, through joint modeling of prediction tasks for multiple content evaluation indicators, positive transfer and information sharing between tasks can be achieved, avoiding the target bias caused by single-task models focusing on local indicators, ensuring the overall balance of each task dimension, and further improving model training efficiency through flexible adjustment of multi-task weight balancing parameters, realizing collaborative optimization of the user layer and content layer and improving global benefits.

[0054] In some embodiments, S110 includes: obtaining a plurality of second scrolling windows according to a first window length and a first scrolling step, and obtaining user behavior data and content-related data corresponding to the content type for each second scrolling window from the log system; for each preset time unit within each second scrolling window, performing feature extraction on the user behavior data and content-related data according to a first feature dimension, generating a sample feature vector corresponding to the preset time unit, and determining a reference retention label corresponding to the preset time unit to constitute a training sample for the preset time unit.

[0055] The first window length and the first scrolling step are pre-set time window lengths and scrolling steps in the rolling sampling rules of the training sample collection process. These can be comprehensively set based on factors such as the performance of the electronic device, the size of the original data, and business needs. For example, the first window length can be set to 14 days, 30 days, or even longer. The first scrolling step can be set to any number of days from 1 day to the first window length. The first window length and the first scrolling step can also be dynamically determined using methods such as reinforcement learning or Bayesian optimization to achieve automated time granularity adjustment. The second scrolling window, also known as the training time window, is the time window for collecting each training sample. In other words, the second scrolling window is the time window determined during the training sample collection process based on the first window length and the first scrolling step. User behavior data refers to data related to users and their interactive behaviors during content consumption. Taking video as an example, user behavior data can include user attribute information (such as user preferences, region, etc.), user interactions with videos resulting in exposure, playback, likes, comments, reposts, playback duration, dwell time, and various interaction timestamps. Content-related data refers to data related to the content consumed by the user. Taking video type as an example, content-related data can include content subcategories, themes, duration, creator information, exposure cycles, revenue-related metrics, etc. The preset time unit is a pre-defined data processing unit, such as 1 day or 2 days.

[0056] Specifically, electronic devices can divide the timeline into multiple continuous sample windows within a preset training timeframe, based on the first window length and the first scrolling step. For example, if the preset training timeframe is 180 days (meaning the timeline runs from day 1 to day 180), the first window length is 30 days (the fixed duration of each second scrolling window), and the first scrolling step is 10 days (the time offset between two adjacent second scrolling windows), then this 180-day timeline can be divided into 16 continuous sample windows (second scrolling windows): day 1 to day 30, day 11 to day 40, ..., day 151 to day 180. Then, for each second scrolling window, user behavior data and content-related data for the corresponding time period are extracted from the log system corresponding to the content type. Next, feature extraction is performed on the user behavior data and content-related data according to the preset time unit, yielding the sample feature vector corresponding to the preset time unit. Finally, reference retention tags corresponding to the preset time unit are obtained through other platforms or statistical methods. In this way, each second scrolling window can obtain the sample feature vector and reference retention label corresponding to each preset time unit, forming the training samples. This allows for the construction of a dynamic training sample set in a time series manner, enabling the model to dynamically capture user retention trends at different time periods, avoiding model distortion problems at fixed time sections, achieving dynamic estimation of retention probability, and improving the model's adaptability to seasonal and periodic behavioral changes. In addition, retention predictions can be updated in real time after model release, supporting rapid business response.

[0057] In some embodiments, taking video as the content type to be evaluated, the content-related data includes video playback data. This allows for a greater focus on video playback behavior-related data during retention analysis, such as completion rate, effective playback duration, and user dwell time distribution, thereby enabling the model's retention analysis to be focused on playback retention probability and reflecting the video's attractiveness.

[0058] Accordingly, feature extraction is performed on user behavior data and content-related data according to the first feature dimension to generate sample feature vectors corresponding to the preset time unit. This includes: extracting features from user behavior data and video playback data corresponding to the preset time unit according to the user dimension, content dimension, and interaction dimension of user interaction with content, respectively, to generate user feature vectors, content feature vectors, and interaction feature vectors, which constitute the sample feature vectors of the preset time unit.

[0059] Specifically, to address the issue of retention analysis in related technologies focusing only on content retention or user retention at a single analytical granularity, this embodiment of the disclosure can set the first feature dimension as a user dimension, a content dimension, and an interaction dimension of user interaction with content, so as to jointly predict the probability of user playback retention from both the user and content levels. Therefore, for each preset time unit, the electronic device can extract features from user behavior data and video playback data according to the user dimension, content dimension, and interaction dimension, respectively, to generate user feature vectors, content feature vectors, and interaction feature vectors, constituting sample feature vectors. The user feature vector may include user attributes, activity frequency / number of times, average dwell time, access frequency, operation habit characteristics, etc.; the content feature vector includes video topic tags, creator metrics, video duration, playback completion rate, average viewing time, and interaction rate, etc.; the interaction feature vector includes the number of times users play videos, click-through rate, exposure conversion ratio, scrolling window number, exposure cycle, interaction time period information, number of exposures within the window, etc.

[0060] In some embodiments, determining the reference retention tag corresponding to the preset time unit includes: generating a first scrolling window corresponding to the preset time unit within the time range of the second scrolling window, according to the second window length and the second scrolling step; and determining the reference retention tag corresponding to the preset time unit based on user behavior data within the first scrolling window.

[0061] The second window length and the second scrolling step are, respectively, the time window length and scrolling step in another set of pre-set rolling observation rules for the retention observation and statistics process. These can be comprehensively set based on the performance of the electronic device, the size of the original data, and business requirements. The first scrolling window is the time window within the second scrolling window used for user retention observation and statistics; it can be called the retention observation time window and is determined according to the rolling observation rules. In this embodiment, the first scrolling window is also the statistical time window when obtaining the reference retention tag. Because the second window length is less than the first window length, each first scrolling window is a subset of the second scrolling window.

[0062] Specifically, to address the issue of unequal observation periods for users, this embodiment of the disclosure can set a rolling observation rule. That is, within the time range of the second rolling window, a first rolling window corresponding to each user within a preset time unit is generated according to the second window length and the second rolling step size. For example, if the second rolling window is 10 days (assuming the window's time range is day 1 to day 10), the business requirement is a 7-day mid-term retention probability, the second window length is 7 days (the core observation period for calculating retention probability), and the second rolling step size is 1 day (the time offset between two adjacent first rolling windows), then within the second rolling window, the first rolling window for users on the first day is days 1 to 7, the first rolling window for users on the second day is days 2 to 8, and so on. The first rolling window for users on the fifth day is days 5 to 10, and the first rolling window for users on the tenth day is only day 10. Then, for each training sample, the retention-related data (such as user interaction-related data) within the time range of its corresponding first rolling window can be statistically analyzed to obtain the reference retention label. This approach overcomes the limitation of retention metrics in related technologies, which are limited to "next-day retention." By using a rolling time window mechanism for retention observation, it dynamically models the retention of content over different time spans, enabling the model to dynamically depict the process of retention changing over time. This allows for a shift from static retention analysis to continuous dynamic estimation over time, reflecting the attractiveness and decline trend of content in the medium to long term.

[0063] The following provides a detailed description of the content recommendation method provided in the embodiments of this disclosure.

[0064] The content recommendation method provided in this disclosure is mainly applicable to scenarios involving personalized recommendations / push notifications for various types of content, or to scenarios involving content recommendation and analysis based on content evaluation results. This content recommendation method can be executed by a content recommendation device, which can be implemented in software and / or hardware and integrated into an electronic device with certain data processing capabilities. This electronic device can be a client device where the front-end application resides, or a server device. For example, the electronic device can be a mobile phone, PDA, tablet computer, laptop computer, desktop computer, or server.

[0065] Figure 3 This is a flowchart of a content recommendation method provided in an embodiment of this disclosure. See also... Figure 3 The specific methods for recommending this content include: S310. In response to the content recommendation request, obtain candidate content data of multiple candidate content and historical behavior data within a preset time period before the current moment.

[0066] The content recommendation request is used to trigger the content evaluation and recommendation process. Candidate content is the content to be evaluated. Candidate content data includes relevant data such as attribute information and timestamps. The preset duration is the pre-defined model processing time, set based on the electronic device's performance and business needs. Historical behavior data is user behavior data from historical time periods prior to the current moment.

[0067] Specifically, a user launching a relevant page with content recommendation functionality, or an upstream service issuing a content recommendation instruction, can trigger a content recommendation request for a specific content type. Upon receiving this request, the electronic device can determine the type of content to be recommended and retrieve candidate content data for multiple candidate content items of that type from the database, as well as historical behavior data within a preset time period prior to the current moment.

[0068] In some embodiments, prior to S310, the method further includes: generating a content recommendation request corresponding to the target page in response to a launch request from the target page. Here, the target page is a display page associated with a retention prediction model, and its launch command can trigger the invocation of the retention prediction model. The target page needs to display at least one piece of content (i.e., recommended content) filtered based on the content recommendation results. When a user opens the target page, the electronic device can generate the aforementioned content recommendation request in response to the launch request from the target page.

[0069] If the electronic device is the client's device, then the electronic device locally generates and processes the content recommendation request. If the electronic device is the server's device, then the client performs the steps described above to generate the content recommendation request and sends the content recommendation request to the electronic device.

[0070] S320. Extract features from historical behavior data and candidate content data according to the second feature dimension to generate target feature vector.

[0071] The second feature dimension is the feature dimension used in the model inference stage for retention analysis, and it is adapted to the first feature dimension. For example, if the first feature dimension includes the user dimension (or content dimension), then the second feature dimension includes the user dimension (or content dimension); or if the first feature dimension includes the user dimension, content dimension, and interaction dimension, then the second feature dimension includes the user dimension and content dimension.

[0072] Specifically, referring to the steps related to feature extraction in the foregoing embodiments, the electronic device can extract features from historical behavior data and candidate content data to generate a feature vector (i.e., target feature vector) corresponding to the second feature dimension.

[0073] S330. Based on the target feature vector, call the retention prediction model to generate the target retention probability corresponding to each candidate content.

[0074] The retention prediction model is obtained by pre-training using the retention prediction model training method in any of the above embodiments of this disclosure.

[0075] Specifically, electronic devices can input the target feature vector into the retention prediction model, and after processing by the model, output the retention probability (i.e., the target retention probability) of each candidate content in the model inference stage.

[0076] In some embodiments, if the retention prediction model includes a feature sharing layer and detection heads corresponding to multiple content recommendation metrics (see [reference]), Figure 2 (and related embodiments), then S330 includes: based on the target feature vector, calling the retention prediction model to generate the target retention probability and other target indicator values ​​corresponding to each candidate content.

[0077] In this embodiment, the other target indicator values, corresponding to the aforementioned other indicator values, include at least one of the target click-through rate, target playback completion rate, and target content revenue.

[0078] Specifically, referring to the descriptions of the aforementioned embodiments, when the retention prediction model's structure includes multiple detection heads, the electronic device can obtain the target retention probability and other target indicator values, such as the target click-through rate, target playback completion rate, and target content revenue. This allows for a more comprehensive assessment of users' interest in content from multiple key dimensions, providing a solid data foundation for subsequent content analysis, recommendation, and other business operations.

[0079] S340. Based on the retention probability of each target, sort and filter each candidate content to obtain multiple recommended contents.

[0080] Specifically, electronic devices can sort candidate content in descending order of target retention probability. Then, based on the number of content that can be displayed on the target page or the number of content set by the business, the same number of top-ranked candidate content is extracted as recommended content. This recommended content can be displayed sequentially on the target page. If the electronic device is a client-side device, it can obtain the recommended content from the server and display it sequentially on the target page. If the electronic device is a server-side device, it can send the recommended content and its ranking to the client so that the client can display it on the target page. This allows for more accurate content recommendation based on target retention probability, thereby increasing the probability of users consuming recommended content and ultimately improving the probability of long-term user retention.

[0081] The content recommendation method provided in this disclosure can, in response to a content recommendation request, obtain candidate content data of multiple candidate contents to be recommended and historical behavior data within a preset time period before the current moment; extract features from the historical behavior data and candidate content data according to a second feature dimension to generate a target feature vector; based on the target feature vector, call the aforementioned trained retention prediction model to generate a target retention probability corresponding to each candidate content; sort and filter each candidate content based on each target retention probability to obtain multiple recommended contents; thereby realizing the use of the trained retention prediction model to infer the medium- and long-term retention of candidate contents, improving the accuracy and stability of the target retention probability of candidate contents, thereby improving the accuracy of content recommendation and the probability of users consuming recommended content, and further improving the probability of users' medium- and long-term retention.

[0082] In some embodiments, content recommendation requests are generated based on interactions with a target page. The target page includes video display pages in video application products, such as the homepage (or main page, recommendation page) of the video application product, or a vertical page corresponding to a specific filtering category (such as a movie page, short video page, etc.). Thus, candidate content includes candidate videos that match the page attributes of the target page. Here, page attributes refer to the core features of the target page, such as the all-category video attribute of the homepage, or the vertical category of a vertical page. Candidate content is candidate videos that match the target page in terms of the core features.

[0083] Based on the above, if the retention prediction model includes multiple detection heads, then the candidate content is sorted and filtered based on the retention probability of each target to obtain multiple recommended content, including: for each candidate video, the target retention probability and other target indicator values ​​are weighted and summed to determine the comprehensive indicator value corresponding to the candidate video; based on the comprehensive indicator value of each candidate video, the candidate videos are sorted and filtered to obtain multiple recommended videos.

[0084] Specifically, as described in the foregoing embodiments, the electronic device can output a target retention probability and at least one other target indicator value for each candidate video. Then, the electronic device can perform a weighted summation of these target indicator values ​​according to the weights corresponding to each content recommendation indicator to obtain a comprehensive indicator value for the candidate video. This comprehensive indicator value can comprehensively reflect the user's level of interest in the candidate videos. Therefore, the electronic device can sort and filter each candidate video according to each comprehensive indicator value to obtain multiple recommended videos, which are then displayed sequentially on the target page. This comprehensive sorting can dynamically optimize the content recommendation list, effectively solving the problem in related technologies where content recommendation algorithms only focus on short-term clicks and ignore long-term activity. This allows the system to improve user retention and content quality while maintaining exposure conversion, thereby significantly improving the content sorting quality and revenue conversion efficiency in the actual recommendation system.

[0085] It should be noted that the weights corresponding to the above content recommendation metrics can be adjusted according to business objectives or revenue priorities. In this way, by dynamically integrating scores, the platform can flexibly balance platform revenue, user experience, and the health of the content ecosystem at the platform strategy level.

[0086] It should also be noted that the above comprehensive index values ​​can also be used for scenarios such as churn warning and revenue optimization.

[0087] It should also be noted that, in addition to the scenarios mentioned above, the target retention probability and other target metric values ​​output by the retention prediction model at different times can also be used to plot long-term change curves of relevant metrics to analyze their dynamic trends. For example, for the target retention probability, a continuous retention probability curve can be obtained to describe the entire process from user registration to sustained activity, providing clear data support for user growth teams. Furthermore, the quantitative evaluation of retention trends can verify the impact of new feature launches or operational activities on user stickiness, and so on.

[0088] Figure 4 This is a schematic diagram of a retention prediction model training device provided in an embodiment of this disclosure. Figure 4 As shown, the retention prediction model training device 400 includes: The training sample acquisition module 410 is used to acquire multiple training samples; wherein, the training samples include a sample feature vector of the first feature dimension related to the content type to be evaluated and a reference retention label corresponding to the sample feature vector; the reference retention label is used to characterize the user's retention status within the first scrolling window, which is greater than or equal to two days. The censoring weight determination module 420 is used to determine the censoring weights corresponding to the sample feature vectors based on the censoring estimation model, and to determine the sample mask value corresponding to the sample feature vectors based on the censoring weights; wherein, the sample mask value is used to characterize whether the training sample corresponds to the observation data or whether the reference retention label corresponding to the training sample is a pseudo label. The predicted retention probability generation module 430 is used to input the feature vectors of each sample into the initial prediction model for processing and generate the predicted retention probability corresponding to the training sample; wherein, the predicted retention probability represents the retention probability of the user for the content corresponding to the sample feature vector output by the initial prediction model. The model parameter update module 440 is used to determine the retention loss function value based on the reference retention label, predicted retention probability, pruning weight and sample mask value corresponding to each training sample, as well as the number of samples of each training sample, and to use the retention loss function value to update the model parameters of the initial prediction model through backpropagation. The retention prediction model acquisition module 450 is used to return to the step of inputting the feature vectors of each sample into the initial prediction model for processing and generating the predicted retention probability corresponding to the training sample if the convergence condition of the model training is not met, until the convergence condition is met and the retention prediction model is obtained.

[0089] The retention prediction model training device provided in this disclosure can locate the reference retention label of the training sample to the user's medium- to long-term retention status. This allows the model to effectively evaluate the impact of the content to be evaluated on the user's medium- to long-term retention, and then obtain the corresponding censoring weights to participate in the loss function calculation. This corrects the statistical bias in long-term retention prediction caused by right censoring in the retention data and incomplete sample observations, thereby improving the accuracy and stability of the medium- to long-term reference retention label. Furthermore, the sample mask value corresponding to the training sample can be obtained to participate in the loss function calculation. Under the condition that the rolling time window of retention label statistics is aligned with the actual observation time, noise samples or pseudo-label samples can be effectively shielded. This allows the model to only perform gradient updates on valid training samples and learn effective signals from the actual observation distribution, thereby improving the model's stability and generalization ability under non-uniform time sampling, and further enhancing the model's accuracy and stability in medium- to long-term retention prediction.

[0090] In some embodiments, the censoring weight determination module 420 is specifically used for: If the reciprocal of the censoring weight is less than or equal to the set threshold, then the sample mask value corresponding to the sample feature vector is set to 0. If the reciprocal of the censoring weight is greater than the set threshold, then the sample mask value corresponding to the sample feature vector is set to 1.

[0091] In some embodiments, the censoring weight determination module 420 is further specifically used for: The user-related feature vectors from each sample feature vector are input into the censoring estimation model, and the user observation probability corresponding to each sample feature vector is output; where the user observation probability represents the probability that the user's target behavior is observed within the first scrolling window; The reciprocal of the user observation probability is used as the censoring weight corresponding to the feature vector of the sample.

[0092] In some embodiments, the initial prediction model includes a feature sharing layer and a detection head corresponding to multiple content evaluation metrics; each content evaluation metric includes retention metrics and other metrics; the other metrics reflect the degree of user interest in the content corresponding to the sample feature vector; the other metrics include at least one of click-through rate, playback completion rate and content revenue metrics. Accordingly, the initial prediction model processes the sample feature vectors in the following manner to generate the predicted retention probability and other predicted metric values ​​for the corresponding training samples: The sample feature vectors are input into the feature sharing layer for feature fusion processing to generate a comprehensive feature vector. The comprehensive feature vector is input into each detection head for evaluation of the corresponding indicators, generating the predicted retention probability and other predicted indicator values ​​corresponding to the training samples; wherein, the other predicted indicator values ​​correspond to other indicators; the other predicted indicator values ​​include at least one of the predicted click-through rate, predicted playback completion rate and predicted content revenue; Accordingly, the model parameter update module 440 is specifically used for: Based on the retention loss function value and the loss function values ​​corresponding to other prediction index values, the comprehensive loss function value is determined, and the model parameters of the initial prediction model are updated by backpropagation using the comprehensive loss function value.

[0093] In some embodiments, the training sample acquisition module 410 is specifically used for: Based on the first window length and the first scroll step, multiple second scroll windows are obtained, and user behavior data and content-related data corresponding to the content type for each second scroll window are obtained from the log system; wherein, the second scroll window is the time window for collecting each training sample; the first scroll window is the time window for observing and statistically analyzing user retention within the second scroll window; each first scroll window is a subset of the second scroll windows; For each preset time unit within each second scrolling window, feature extraction is performed on user behavior data and content-related data according to the first feature dimension to generate sample feature vectors corresponding to the preset time units, and reference retention labels corresponding to the preset time units are determined to form training samples for the preset time units.

[0094] Furthermore, the training sample acquisition module 410 is specifically used for: Within the time range of the second scrolling window, a first scrolling window corresponding to a preset time unit is generated according to the second window length and the second scrolling step; wherein, the second window length is less than the first window length; Based on user behavior data within the first scrolling window, determine the reference retention tag corresponding to the preset time unit.

[0095] In some embodiments, content-related data includes video playback data; The training sample acquisition module 410 is also specifically used for: Feature extraction is performed on user behavior data and video playback data corresponding to preset time units according to user dimension, content dimension, and user interaction dimension, respectively, to generate user feature vector, content feature vector, and interaction feature vector, which constitute the sample feature vector of the preset time unit; among them, the user feature vector includes the average dwell time of users on each video, the content feature vector includes the video playback completion rate, and the interaction feature vector includes the number of times users played the video.

[0096] The retention prediction model training device provided in this disclosure can execute the retention prediction model training method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0097] Figure 5 This is a schematic diagram of the structure of a content recommendation device provided in an embodiment of this disclosure. Figure 5 As shown, the content recommendation device 500 includes: The data acquisition module 510 is used to respond to the content recommendation request and acquire candidate content data of multiple candidate content and historical behavior data within a preset time period before the current moment; The target feature vector generation module 520 is used to extract features from historical behavior data and candidate content data according to the second feature dimension to generate a target feature vector. The target retention probability generation module 530 is used to generate the target retention probability corresponding to each candidate content by calling the retention prediction model based on the target feature vector; wherein, the retention prediction model is obtained by pre-training through the retention prediction model training method provided in any embodiment of this disclosure; The recommended content acquisition module 540 is used to sort and filter candidate content based on the retention probability of each target, and obtain multiple recommended content.

[0098] The content recommendation device provided in this disclosure can use a trained retention prediction model to infer the medium- and long-term retention of candidate content, improve the accuracy and stability of the target retention probability of candidate content, thereby improving the accuracy of content recommendation and the probability of users consuming recommended content, and further improving the probability of users' medium- and long-term retention.

[0099] In some embodiments, the retention prediction model includes a feature sharing layer and a detection head corresponding to multiple content recommendation metrics; Accordingly, the target retention probability generation module 530 is specifically used for: Based on the target feature vector, the retention prediction model is invoked to generate the target retention probability and other target indicator values ​​for each candidate content; among them, the other target indicator values ​​include at least one of the target click-through rate, target play completion rate and target content revenue.

[0100] In some embodiments, the content recommendation request is generated based on interactive operations on a target page; the target page includes a video display page in a video application product; the candidate content includes candidate videos that are adapted to the page attributes of the target page; Accordingly, the recommended content acquisition module 540 is specifically used for: For each candidate video, the target retention probability and other target indicator values ​​are weighted and summed to determine the comprehensive indicator value corresponding to the candidate video; Based on the comprehensive index values ​​of each candidate video, the candidate videos are sorted and filtered to obtain multiple recommended videos; among them, each recommended video is used to display in sequence on the target page.

[0101] The content recommendation device provided in this disclosure can execute the content recommendation method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.

[0102] It is worth noting that in the embodiments of the above devices, the modules and sub-modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional module / sub-module are only for easy differentiation and are not used to limit the scope of protection of this disclosure.

[0103] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Figure 6 As shown, the electronic device 600 includes one or more processors 601 and memory 602.

[0104] The processor 601 may be a central processing unit (CPU) or other form of processing unit with interface layout capabilities and / or instruction execution capabilities, and may control other components in the electronic device 600 to perform desired functions.

[0105] The memory 602 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 601 may execute the program instructions to implement the retention prediction model training method, content recommendation method, and / or other desired functions described above in any embodiment of this disclosure. The computer-readable storage medium may also store various content such as user behavior data, target evaluation models and their model parameters, target interface elements and their corresponding candidate element layout information.

[0106] In one example, the electronic device 600 may further include an input device 603 and an output device 604, these components being interconnected via a bus system and / or other forms of connection mechanisms (not shown). The input device 603 may include, for example, a keyboard, a mouse, etc. The output device 604 may output various information to the outside, including element layout information, the target interface to be rendered, etc. The output device 604 may include, for example, a monitor, speakers, a printer, and a communication network and its connected remote output devices, etc.

[0107] Of course, for the sake of simplicity, Figure 6 Only some of the components of the electronic device 600 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 600 may include any other suitable components depending on the specific application.

[0108] In addition to the methods and devices described above, embodiments of this disclosure may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the retention prediction model training method or content recommendation method provided in any embodiment of this disclosure.

[0109] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0110] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the retention prediction model training method or content recommendation method provided in any embodiment of this disclosure.

[0111] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0112] It should be noted that the terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the scope of this disclosure. As shown in this specification and claims, unless the context clearly indicates otherwise, words such as "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. The term "and / or" includes any one and all combinations of one or more of the associated listed items. The terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, or apparatus. Without further limitations, an element defined by the phrase "comprising an..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element.

[0113] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for retention prediction model training, the method comprising: include: Multiple training samples are obtained; wherein, the training samples include a sample feature vector of a first feature dimension related to the content type to be evaluated and a reference retention label corresponding to the sample feature vector; the reference retention label is used to characterize the user's retention status within a first scrolling window of scrolling observation, the first scrolling window being greater than or equal to two days; Based on the sample feature vectors, a censoring estimation model is used to determine the censoring weights corresponding to the sample feature vectors, and the sample mask value corresponding to the sample feature vectors is determined based on the censoring weights; wherein, the sample mask value is used to characterize whether the training sample corresponds to observation data or whether the reference retention label corresponding to the training sample is a pseudo label. Each of the sample feature vectors is input into the initial prediction model for processing to generate the predicted retention probability corresponding to the training sample; wherein, the predicted retention probability represents the retention probability of the user for the content corresponding to the sample feature vector, as output by the initial prediction model; Based on the reference retention label, the predicted retention probability, the pruning weight, and the sample mask value corresponding to each training sample, as well as the number of samples in each training sample, the retention loss function value is determined, and the model parameters of the initial prediction model are updated by backpropagation using the retention loss function value. If the convergence condition for model training is not met, the process returns to the step of inputting the feature vectors of each sample into the initial prediction model for processing and generating the predicted retention probability corresponding to the training sample, until the convergence condition is met and the retention prediction model is obtained.

2. The method according to claim 1, characterized in that, The step of determining the sample mask value corresponding to the sample feature vector based on the pruning weights includes: If the reciprocal of the censoring weight is less than or equal to a set threshold, then the sample mask value corresponding to the sample feature vector is determined to be 0; If the reciprocal of the censoring weight is greater than the set threshold, then the sample mask value corresponding to the sample feature vector is determined to be 1.

3. The method according to claim 1, characterized in that, The step of determining the censoring weights corresponding to the sample feature vectors using a censoring estimation model based on each sample feature vector includes: The user-related feature vectors from each of the sample feature vectors are input into the censoring estimation model, and the user observation probability corresponding to each of the sample feature vectors is output; wherein, the user observation probability represents the probability that the user's target behavior is observed within the first scrolling window; The reciprocal of the user observation probability is determined as the censoring weight corresponding to the feature vector of the corresponding sample.

4. The method according to claim 1, characterized in that, The initial prediction model includes a feature sharing layer and a detection head corresponding to multiple content evaluation metrics; each content evaluation metric includes retention metrics and other metrics; the other metrics reflect the user's interest in the content corresponding to the sample feature vector; the other metrics include at least one of click-through rate, playback completion rate, and content revenue metrics; The initial prediction model processes the sample feature vectors in the following manner to generate the predicted retention probability and other predicted index values ​​corresponding to the training samples: The sample feature vector is input into the feature sharing layer for feature fusion processing to generate a comprehensive feature vector. The comprehensive feature vector is input into each of the detection heads for evaluation of corresponding indicators, generating the predicted retention probability and other predicted indicator values ​​corresponding to the training samples; wherein, the other predicted indicator values ​​correspond to other indicators; the other predicted indicator values ​​include at least one of predicted click-through rate, predicted playback completion rate, and predicted content revenue; The step of updating the model parameters of the initial prediction model using the retention loss function value through error backpropagation includes: Based on the retention loss function value and the loss function values ​​corresponding to the other prediction index values, a comprehensive loss function value is determined, and the model parameters of the initial prediction model are updated using the comprehensive loss function value through error backpropagation.

5. The method according to claim 1, characterized in that, The acquisition of multiple training samples includes: According to the first window length and the first scroll step, multiple second scroll windows are obtained, and user behavior data corresponding to each second scroll window and content-related data corresponding to the content type are obtained from the log system; wherein, the second scroll window is the time window for collecting each training sample; the first scroll window is the time window for observing and statistically analyzing user retention within the second scroll window; each first scroll window is a subset of the second scroll window; For each preset time unit within each second scrolling window, feature extraction is performed on the user behavior data and the content-related data according to the first feature dimension to generate the sample feature vector corresponding to the preset time unit, and the reference retention tag corresponding to the preset time unit is determined to constitute the training sample of the preset time unit.

6. The method according to claim 5, characterized in that, The step of determining the reference retention tag corresponding to the preset time unit includes: Within the time range of the second scrolling window, the first scrolling window corresponding to the preset time unit is generated according to the second window length and the second scrolling step; wherein, the second window length is less than the first window length; Based on the user behavior data within the first scrolling window, the reference retention tag corresponding to the preset time unit is determined.

7. The method according to claim 5, characterized in that, The content-related data includes video playback data; The step of extracting features from the user behavior data and the content-related data according to the first feature dimension to generate the sample feature vector corresponding to the preset time unit includes: Feature extraction is performed on the user behavior data and video playback data corresponding to the preset time unit according to the user dimension, content dimension, and user interaction dimension, respectively, to generate user feature vector, content feature vector, and interaction feature vector, which constitute the sample feature vector of the preset time unit; wherein, the user feature vector includes the average dwell time of the user on each video, the content feature vector includes the video playback completion rate, and the interaction feature vector includes the number of times the user plays the video.

8. A content recommendation method, characterized in that, include: In response to a content recommendation request, obtain candidate content data for multiple candidate content items and historical behavior data within a preset time period prior to the current moment; Feature extraction is performed on the historical behavior data and the candidate content data according to the second feature dimension to generate a target feature vector; Based on the target feature vector, a retention prediction model is invoked to generate the target retention probability corresponding to each of the candidate contents; wherein, the retention prediction model is obtained by pre-training using the retention prediction model training method as described in any one of claims 1 to 7; Based on the retention probability of each target, the candidate content is sorted and filtered to obtain multiple recommended contents.

9. The method according to claim 8, characterized in that, The retention prediction model includes a feature sharing layer and detection heads corresponding to multiple content recommendation metrics; The step of calling the retention prediction model based on the target feature vector to generate the target retention probability corresponding to each candidate content includes: Based on the target feature vector, the retention prediction model is invoked to generate the target retention probability and other target indicator values ​​corresponding to each candidate content; wherein, the other target indicator values ​​include at least one of target click-through rate, target playback completion rate and target content revenue.

10. The method according to claim 9, characterized in that, The content recommendation request is generated based on interactive operations on the target page; the target page includes video display pages in video-based application products; the candidate content includes candidate videos that are adapted to the page attributes of the target page; The candidate content is sorted and filtered based on the retention probability of each target to obtain multiple recommended contents, including: For each candidate video, the target retention probability and the other target indicator values ​​are weighted and summed to determine the comprehensive indicator value corresponding to the candidate video; Based on the comprehensive index values ​​of each candidate video, the candidate videos are sorted and filtered to obtain multiple recommended videos; wherein, each recommended video is used to be displayed in sequence on the target page.

11. A retention prediction model training device, characterized in that, include: The training sample acquisition module is used to acquire multiple training samples; wherein, the training samples include a sample feature vector of a first feature dimension related to the content type to be evaluated and a reference retention label corresponding to the sample feature vector; the reference retention label is used to characterize the user's retention status within a first scrolling window, the first scrolling window being greater than or equal to two days; The censoring weight determination module is used to determine the censoring weights corresponding to the sample feature vectors based on each sample feature vector and using a censoring estimation model, and to determine the sample mask value corresponding to the sample feature vectors based on the censoring weights; wherein, the sample mask value is used to characterize whether the training sample corresponds to observation data or whether the reference retention label corresponding to the training sample is a pseudo label. The predicted retention probability generation module is used to input the feature vectors of each sample into the initial prediction model for processing, and generate the predicted retention probability corresponding to the training sample; wherein, the predicted retention probability represents the retention probability of the user for the content corresponding to the feature vector of the sample, output by the initial prediction model. The model parameter update module is used to determine the retention loss function value based on the reference retention label, the predicted retention probability, the pruning weight, and the sample mask value corresponding to each training sample, as well as the number of samples of each training sample, and to use the retention loss function value to update the model parameters of the initial prediction model through backpropagation. The retention prediction model acquisition module is used to return to the step of inputting the feature vectors of each sample into the initial prediction model for processing and generating the predicted retention probability corresponding to the corresponding training sample if the convergence condition of the model training is not met, until the convergence condition is met and the retention prediction model is obtained.

12. A content recommendation device, characterized in that, include: The data acquisition module is used to respond to content recommendation requests and acquire candidate content data of multiple candidate content and historical behavior data within a preset time period before the current moment; The target feature vector generation module is used to extract features from the historical behavior data and the candidate content data according to the second feature dimension to generate a target feature vector. The target retention probability generation module is used to generate the target retention probability corresponding to each of the candidate contents by calling the retention prediction model based on the target feature vector; wherein, the retention prediction model is obtained by pre-training the retention prediction model training method as described in any one of claims 1 to 7; The recommended content acquisition module is used to sort and filter the candidate content based on the retention probability of each target to obtain multiple recommended content.

13. An electronic device, characterized in that, The electronic device includes: Processor and memory; The processor executes the retention prediction model training method as described in any one of claims 1 to 7, or the content recommendation method as described in any one of claims 8 to 10, by calling the program or instructions stored in the memory.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program or instructions that cause a computer to perform the retention prediction model training method as described in any one of claims 1 to 7, or the content recommendation method as described in any one of claims 8 to 10.

15. A computer program product, characterized in that, The computer program product is used to implement the retention prediction model training method according to any one of claims 1 to 7, or the content recommendation method according to any one of claims 8 to 10.

Images