Content retention value evaluation method, device, electronic device and storage medium

Abstract

The present application relates to the field of data processing technology, and provides a method, device, electronic device, and storage medium for evaluating the retention value of content. The method responds to a retention value evaluation request for target log content, obtains content features of the target log content and consumption behavior features of different users in a historical preset time period; based on the content features and consumption behavior features, obtains the retention score of the target log content in each historical sub-time period; based on the retention score in each historical sub-time period and the configured weight of the corresponding historical sub-time period, determines the target retention value comprehensive value of the target log content. The method determines the target retention value comprehensive value of the corresponding content by analyzing the user's consumption behavior and the content features of the consumption in real time, so as to quickly evaluate the attractiveness and retention value of the content, and improve the response speed and accuracy of the recommendation system.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, electronic device and storage medium for evaluating the retention value of content. Background Art

[0002] In the recommendation system, user retention is a crucial indicator that directly reflects the long-term effectiveness of the system. Retention-related signals are usually feedback from users after using the platform for a period of time, such as whether the user will visit the platform again, how long they stay, etc. Although these signals are important indicators for measuring the effectiveness of the recommendation system, they are indirect and comprehensive feedback signals, so it is difficult to capture the user's true reaction to the content, making it difficult to immediately guide the recommendation system to distribute content immediately. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a method, device, electronic device and storage medium for evaluating the retention value of content. The method can determine the target comprehensive retention value of different content by real-time analysis of the user's consumption behavior and the characteristics of the consumed content, so as to quickly evaluate the attractiveness and retention value of the content, thereby improving the user retention rate.

[0004] In a first aspect, a method for evaluating the retention value of content is provided, and the method may include:

[0005] In response to a retention value evaluation request for target log content, obtain content features of the target log content and consumption behavior features of different users within a historical preset time period; wherein the retention value evaluation request includes the target log content and consumption behaviors of different users on the target log content within a historical preset time period; the historical preset time period includes a plurality of historical sub-time periods of equal length;

[0006] Based on the content features and the consumption behavior features, obtaining the retention score of the target log content in each historical sub-time period;

[0007] Based on the retention points in each historical sub-time period and the configured weight of the corresponding historical sub-time period, a target retention value comprehensive value of the target log content is determined.

[0008] In a possible implementation, obtaining the retention score of the target log content in each historical sub-time period based on the content feature and the consumption behavior feature includes:

[0009] If the target log content is consumed before the historical preset time period, the refresh times of different users in each historical sub-time period are obtained;

[0010] The number of refreshes in each historical sub-time period is determined as the retention score in the corresponding historical sub-time period.

[0011] In a possible implementation, determining the target retention value comprehensive value of the target log content based on the retention points in each historical sub-time period and the configured weight of the corresponding historical sub-time period includes:

[0012] Determine the retention value assessment value of the target log content on that day based on the retention score in each historical sub-time period, the configured weight of the corresponding historical sub-time period and the total number of refreshes on that day;

[0013] The preset exponential smoothing moving average algorithm is used to process the retention value assessment value of the target log content on the current day and the target retention value comprehensive value of the target log content on the previous day to obtain the target retention value comprehensive value of the target log content on the current day; if the current day is the initial day, the target retention value comprehensive value of the target log content on the previous day is the configured initial value.

[0014] In a possible implementation, the calculation formula of the preset exponential smoothing moving average algorithm is expressed as:

[0015]

[0016] Among them, the represents the initial evaluation value on the mth day, represents the comprehensive value of the target retention value on the mth day, It represents the comprehensive value of the target retention value on the m-1th day, and K represents the number of adjacent days.

[0017] In a possible implementation, obtaining the retention score of the target log content in each historical sub-time period based on the content feature and the consumption behavior feature includes:

[0018] Input the content features and the consumption behavior features into a trained retention estimation model to obtain the retention score of the target log content in each historical sub-time period;

[0019] The retention prediction model includes an input terminal and multiple output terminals, and the multiple output terminals include an output terminal for outputting the retention score of each historical sub-time period, an output terminal for outputting the first comprehensive value of retention value, and an output terminal for outputting whether the content feature has been consumed in the historical preset time period.

[0020] In a possible implementation, determining the target retention value comprehensive value of the target log content based on the retention points in each historical sub-time period and the configured weight of the corresponding historical sub-time period includes:

[0021] Determine the second comprehensive retention value by multiplying the retention points in each historical sub-time period by the weight of the corresponding historical sub-time period;

[0022] Determine an average value of the first retention value comprehensive value and the calculated second retention value comprehensive value as the target retention value comprehensive value of the target log content at the current moment, and the current moment is the request moment of the retention value evaluation request.

[0023] In a possible implementation, the method further includes:

[0024] After determining the target retention value comprehensive values ​​corresponding to different types of target log content, a preset normalization algorithm is used to process each target retention value comprehensive value to obtain the retention value normalization values ​​corresponding to different types of target log content;

[0025] Based on the normalized values ​​of retention values ​​corresponding to different types of target log content, the control strategies in the recall stage, rough ranking stage, fine ranking stage and re-ranking stage of the recommendation system are adjusted to obtain new control strategies for the corresponding stages.

[0026] In a second aspect, a device for evaluating the retention value of content is provided, and the device may include:

[0027] an acquisition unit, configured to acquire, in response to a retention value evaluation request for target log content, content features of the target log content and consumption behavior features of different users within a historical preset time period; wherein the retention value evaluation request includes the target log content and consumption behaviors of different users on the target log content within a historical preset time period; and the historical preset time period includes a plurality of historical sub-time periods of equal length;

[0028] And, based on the content features and the consumption behavior features, obtaining the retention score of the target log content in each historical sub-time period;

[0029] The determination unit is used to determine the target retention value comprehensive value of the target log content based on the retention points in each historical sub-time period and the configured weight of the corresponding historical sub-time period.

[0030] In a third aspect, an electronic device is provided, the electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0031] Memory, used to store computer programs;

[0032] The processor is used to implement any method step described in the first aspect when executing the program stored in the memory.

[0033] In a fourth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any method step described in the first aspect is implemented.

[0034] The present application provides a method for obtaining content features of the target log content and consumption behavior features of different users in a historical preset time period in response to a retention value evaluation request for target log content; the retention value evaluation request includes the target log content and the consumption behavior of different users on the target log content in a historical preset time period; the historical preset time period includes multiple historical sub-time periods of equal length; based on the content features and consumption behavior features, the retention score of the target log content in each historical sub-time period is obtained; based on the retention score in each historical sub-time period and the weight of the corresponding historical sub-time period configured, the target retention value comprehensive value of the target log content is determined. This method can determine the target retention value comprehensive value of different content by analyzing the user's consumption behavior and the content features of the consumption in real time, so as to quickly evaluate the attractiveness and retention value of the content, thereby improving the user retention rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 A flowchart of a method for evaluating the retention value of content provided in an embodiment of the present application;

[0037] Figure 2 A schematic diagram of the structure of a device for evaluating the content retention value provided in an embodiment of the present application;

[0038] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application. Unless otherwise defined, the technical terms or scientific terms used in this application should be the common meanings understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in this application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Including" or "including" and other similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connected", "coupled" or "connected" and other similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0040] Existing technologies use a variety of different methods to evaluate the retention value of content and improve user retention, including:

[0041] 1. Methods that explore the correlation between retention goals and other goals that are easy to work on directly: In this type of method, there may be deviations in the correlation between retention goals and short-term indicators (i.e., correlation deviation), resulting in inconsistent optimization directions; and short-term indicators may not fully represent the user's long-term retention intention (i.e., the limitations of short-term indicators), resulting in inaccurate recommendation strategies.

[0042] 2. Reinforcement learning models the recommendation system as a Markov decision process (MDP):

[0043] This type of method requires a large amount of interaction data to train the model (i.e. data is scarce), which is especially difficult for new or low-frequency users. In addition, it is difficult to strike a balance between exploring unknown strategies and using known optimal strategies, which may lead to unstable recommendation quality. Excessive exploration may lead to a decline in recommendation quality, while excessive use may cause the model to fall into a local optimal solution. User behavior and preferences change over time, and the model needs to be updated frequently to adapt to environmental changes (i.e. non-stationary environment). In order to avoid excessive damage to the online user experience, some methods use offline simulation, but offline simulation lacks real interaction, usually has a large variance and is easy to deviate from the online optimization direction. Reinforcement learning usually consumes a lot of computing resources.

[0044] 3. Methods of applying causal inference to retention modeling:

[0045] Accurate estimation of causal effects in this method usually requires rigorous experimental design, but it is difficult to fully control experimental conditions in actual recommendation systems (i.e., experimental design difficulty); high-quality datasets are required to support causal inference analysis, which may be difficult to obtain in certain specific scenarios (i.e., high data requirements).

[0046] In order to solve the above problems, this application provides a method for evaluating the retention value of content, which can quickly capture the user's real reaction to the content by using instant retention signals (or "the comprehensive value of the retention value of the content to be recommended"). By analyzing the user's clicks, dwell time, return visit interval and other behaviors in real time, the attractiveness and retention value of the content can be quickly evaluated, so that the recommendation system can adjust its strategy in time and give priority to recommending high-quality content that has been proven to retain users. The instant feedback mechanism reduces delays, improves the response speed and accuracy of the recommendation system, and ensures that the recommended content is always synchronized with user needs, that is, it guides the optimization of the content distribution strategy of the recommendation system and improves user retention rate.

[0047] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0048] Example 1

[0049] Figure 1 A flow chart of a method for evaluating the content retention value provided in an embodiment of the present application. Figure 1 As shown, the method may include:

[0050] Step S110: In response to a retention value assessment request for target log content, content features of the target log content and consumption behavior features of different users within a historical preset time period are obtained.

[0051] The retention value evaluation request may include the target log content and the consumption behaviors of different users on the target log content within the historical preset time period; the consumption behaviors may include behaviors such as clicking CTR, liking, collecting, sharing, refreshing, etc. The historical preset time period may include multiple historical sub-time periods of equal length. For example, if the historical preset time period is 60 minutes, the historical sub-time periods may be 0-10 minutes, 10-20 minutes, 20-30 minutes, 30-40 minutes, 40-50 minutes, and 50-60 minutes, each spanning 10 minutes.

[0052] Step S120: Based on the content characteristics and consumption behavior characteristics, the retention score of the target log content in each historical sub-time period is obtained.

[0053] Since different users refresh the same content differently after consuming it, we need to collect as many samples as possible for statistics so that the statistical values ​​are more representative.

[0054] In the specific implementation, the retention of target log content in each historical sub-time period is The acquisition can include the following methods:

[0055] Method 1: If the target log content is consumed before the historical preset time period, the refresh times of different users in each historical sub-time period are obtained; then, the refresh times in each historical sub-time period are determined as the retention score in the corresponding historical sub-time period.

[0056] Specifically, statistics are collected on whether different users have refresh behaviors in each historical sub-time period, and a revisit sequence for the target log content is obtained, wherein each element of the revisit sequence represents the refresh situation in the corresponding historical sub-time period, with 1 indicating refresh behaviors and 0 indicating no refresh behaviors. Afterwards, the cumulative number of refresh behaviors in each historical sub-time period is calculated.

[0057] Method 2: Input the content features and consumption behavior features into the trained retention estimation model to obtain the retention score of the target log content in each historical sub-time period.

[0058] In some embodiments, the model structure of the retention prediction model is a typical deep learning multi-layer fully connected network, which calculates the estimated score of each target by encoding and mapping the input features. The retention prediction model includes an input end and multiple output ends; wherein the input end is used to input features closely related to user retention, such as user historical behavior features and content features; the multiple output ends include an output end for outputting the retention score of each historical sub-time period, an output end for outputting the first retention value comprehensive value, and an output end for outputting whether the content feature has been consumed in the historical preset time period.

[0059] Furthermore, the training process of the retention prediction model includes: obtaining features closely related to user retention, such as user population attribute labels, user historical behavior sequences, content features, etc. as training samples and the corresponding true labels y of the samples; using the training samples to train the retention prediction model to be trained to obtain the estimated score p of the model output, and then using the preset loss function to adjust the model parameters to obtain a trained retention prediction model.

[0060] Among them, the preset loss function can be expressed as: When training the built model with training samples, GPU is used for training, and samples from T-2 hours to T-1 hours are trained every hour. After the training is completed, the model breakpoints are saved; in addition, the trained retention estimation model is deployed to the recommendation system and updated hourly to ensure that all candidate content of each user in each request is scored to obtain a more accurate comprehensive value of estimated retention value.

[0061] Step S130: Determine the target retention value comprehensive value of the target log content based on the retention points in each historical sub-time period and the configured weight of the corresponding historical sub-time period.

[0062] (1) In case of adopting method 1:

[0063] In the specific implementation, the retention value assessment value of the target log content of the day is determined based on the retention points of different users in each historical sub-time period and the configured weights of the corresponding historical sub-time period; the day is the daily day in which the historical preset time period is located.

[0064] The preset exponential smoothing moving average algorithm is used to process the retention value assessment value of the target log content of the day and the target retention value comprehensive value of the target log content of the previous day to obtain the target retention value comprehensive value of the target log content of the day; among which, if the day is the initial day (i.e. the initial natural day), the target retention value comprehensive value of the target log content of the previous day is the configured initial value, such as 0 or other custom values.

[0065] In some embodiments, the calculation formula of the preset exponential smoothing moving average algorithm can be expressed as:

[0066]

[0067] in, represents the initial evaluation value on the mth day, represents the comprehensive value of the target retention value on the mth day, It represents the comprehensive value of the target retention value on the m-1th day, K represents the number of adjacent days, and is generally 2.

[0068] (2) In case of adopting method 2:

[0069] The second retention value comprehensive value is determined as the cumulative sum of the retention points in each historical sub-time period multiplied by the weight of the corresponding historical sub-time period;

[0070] Determine the average of the first retention value comprehensive value and the calculated second retention value comprehensive value as the target retention value comprehensive value of the target log content at the current moment, wherein the current moment is the request moment of the retention value evaluation request.

[0071] That is to say, the comprehensive value of the target retention value of the target log content obtained by method 1 is at the day level; the comprehensive value of the target retention value of the target log content obtained by method 1 is at the moment level.

[0072] It should be noted that the target log content involved in this application is other log content except for the content with extremely low distribution volume, and the user consumption behavior involved in this application is other consumption behavior except for the behavior of super-high active users.

[0073] Among them, content with extremely low distribution volume: content with extremely low distribution volume is usually of poor quality, and the low statistics are insignificant. Behavior of ultra-high-activity users: behaviors are intensive, and the behavior patterns of ultra-high-activity users are different from those of low-activity users, which will affect the accuracy of statistics.

[0074] Example 2

[0075] In an example of adopting method 1, the content retention value evaluation method provided by this application may include:

[0076] Step 1: In response to a retention value assessment request for target log content, obtain content features of the target log content and consumption behavior features of different users within 60 minutes after they click on the target log content on the mth day in history.

[0077] The retention value evaluation request may include the target log content and the consumption behavior of different users within the next 60 minutes after they click on the target log content on the mth day in history.

[0078] The target log content is at least one target log content except invalid or abnormal logs, wherein invalid or abnormal logs may include: extremely high active user behavior and content with extremely low distribution volume; the former has intensive return visit behavior, which is not conducive to retention attribution; the latter has extremely low distribution volume and has no statistical significance.

[0079] Step 2: Divide the next 60 minutes into T=6 sub-time periods, each of which has a span of 10 minutes (i.e., 0-10, 10-20, 20-30, 30-40, 40-50, 50-60). Count whether any user refreshes in each sub-time period. t represents the tth time period in the future, which is represented by the indicator function:

[0080]

[0081] Thus, a revisit sequence is obtained, which indicates whether there is a refresh in each of the T=6 sub-time periods, i.e., {f1, f2, ..., fT};

[0082] Step 3: Based on the return visit sequence of each user for the target log content, count the cumulative number of refreshes in each sub-time period after the corresponding user consumes the content, that is, the retention score of the target log content corresponding to each user in each historical sub-time period. .

[0083]

[0084] Wherein, n represents the nth user, and t represents the tth sub-time period.

[0085] Step 4: The calculation formula for the retention value evaluation value of the target log content on the mth day can be expressed as:

[0086]

[0087] Where N is the total number of users, T is the total number of historical sub-time periods, is the weight of the t-th historical sub-period, which is used to highlight the weight of recent return visits:

[0088]

[0089] Step 5: Use the preset exponential smoothing moving average algorithm to process the retention value assessment value of the target log content on the mth day and the target retention value comprehensive value of the target log content on the previous day to obtain the target retention value comprehensive value of the target log content on the day.

[0090] Example 3

[0091] In an example of adopting method 2, the content retention value evaluation method provided by this application may include:

[0092] Step 1: In response to a retention value assessment request for target log content, obtain content features of the target log content and consumption behavior features of different users within a historical 60 minutes.

[0093] The retention value assessment request may include the target log content and the consumption behavior of different users on the target log content within a historical preset time period.

[0094] The target log content is at least one target log content except invalid or abnormal logs, wherein invalid or abnormal logs may include: extremely high active user behavior and content with extremely low distribution volume; the former has intensive return visit behavior, which is not conducive to retention attribution; the latter has extremely low distribution volume and has no statistical significance.

[0095] Step 2: Input the content features and consumption behavior features into the trained retention prediction model to obtain the retention status and retention score of the target log content in the range of 0-10, 10-20, 20-30, 30-40, 40-50, and 50-60. Specifically, the retention prediction model can output the following content:

[0096] Whether there is a click this time: output 1 for yes, output 0 for no;

[0097] Whether to retain within the next 10 minutes, output 1 for yes and retention points; output 0 for no;

[0098] Whether to retain in the next 10 minutes to 20 minutes, output 1 for yes and retention points; output 0 for no;

[0099] Whether to retain in the next 20 minutes to 30 minutes, output 1 for yes and retention points; output 0 for no;

[0100] Whether to retain in the next 30 minutes to 40 minutes, output 1 for yes and retention points; output 0 for no;

[0101] Whether to retain in the next 40 minutes to 50 minutes, output 1 for yes and retention points; output 0 for no;

[0102] Whether to retain in the next 50 minutes to 60 minutes, output 1 for yes and retention points; output 0 for no;

[0103] The retention value of the input content features is evaluated as a first retention value comprehensive value;

[0104] Step 3: The accumulated sum of the retention points in each historical sub-time period multiplied by the weight of the corresponding historical sub-time period is determined as the second retention value comprehensive value;

[0105] Step 4: Determine the average of the first retention value comprehensive value and the calculated second retention value comprehensive value as the target retention value comprehensive value of the target log content at the current moment, where the current moment is the request moment of the retention value evaluation request.

[0106] In some embodiments, according to the obtained target retention value comprehensive value of different contents, the various stages of the recommendation system can be adjusted through personalized control strategies to improve the user retention rate. The various stages may include a recall stage, a rough sorting stage, a fine sorting stage, and a re-sorting stage.

[0107] Specifically, after determining the target retention value comprehensive values ​​corresponding to different types of target log content, a preset normalization algorithm may be used to process each target retention value comprehensive value to obtain the retention value normalization values ​​corresponding to different types of target log content;

[0108] Based on the normalized values ​​of retention values ​​corresponding to different types of target log content, the control strategies in the recall stage, rough ranking stage, fine ranking stage and re-ranking stage of the recommendation system are adjusted to obtain new control strategies for the corresponding stages.

[0109] In some embodiments, the preset normalization algorithm can be expressed as:

[0110]

[0111] Among them, r is the calculated comprehensive value of the target retention value of the content, The mean of all retained scores, The standard deviation of all retained scores, k, , are hyperparameters, representing the slope before normalization, the slope after normalization, and the bias after normalization, which can be set to 3, 0.5, and 0.75 respectively. sigmoid() represents the Sigmoid function, and s is the normalized value of the retention value of the content finally used in each link of the recommendation system.

[0112] The application process includes:

[0113] (1) In the recall phase, the adjustment of the normalized value of retention value corresponding to different types of target log content may include:

[0114] In the construction of the high-quality content pool: according to the normalized retention value s, select content with high retention potential to form a high-quality content pool. Here, select content with s above the 60th percentile as potential high-retention content; when recalling content, give priority to selecting content from the high-quality content pool to ensure that the content recommended to users is verified high-value content, thereby improving user satisfaction. In practice, this part of the content is usually detailed, novel and unique, and can bring certain medium- and long-term value to users rather than short-term pleasure, which is more in line with the tone of the community.

[0115] In filtering the low-quality content pool: Based on the normalized retention value s, identify those contents with poor performance and low user retention rate. Here, select the contents with s below the 20th percentile as the content to be filtered and suppressed; through the filtering mechanism, eliminate low-quality content, purify the recommendation environment, and improve the overall recommendation quality.

[0116] (2) In the rough sorting stage, the adjustment of the normalized value of retention value corresponding to different types of target log content may include:

[0117] After the rough ranking, in order to ensure that there are enough high-retention content to enter the subsequent fine ranking stage, and at the same time to ensure that the amount of low-retention content is as small as possible, the quantity will be limited according to s after the rough ranking. The amount of content with a value less than the average of all content's s that enters the fine ranking shall not exceed 30%. In this way, the proportion of content above the average that enters the fine ranking will exceed 70%;

[0118] After the rough ranking, the content’s s is used for weighted fusion with various multi-objective scores (such as reading, likes, duration, etc.) output by the rough ranking model.

[0119] (3) In the refined ranking stage, the adjustment of the normalized value of retention value corresponding to different types of target log content may include:

[0120] In the fine ranking stage, the fine ranking model and the retention estimation model are called at the same time, so there is no additional time-consuming overhead; the s obtained by the retention estimation model and the multi-objective (such as reading, likes, etc.) scores of the original fine ranking model are multiply fused.

[0121] (4) In the reordering stage, the adjustment of the normalized value of retention value corresponding to different types of target log content may include:

[0122] In the final sorting stage, further fine-tuning is performed based on the normalized retention value s. Content with a retention value higher than the average level is promoted, while content with a retention value lower than the average level is demoted, ensuring that content with high retention value receives more exposure.

[0123] The beneficial effects of this application include:

[0124] 1. Strong real-time performance:

[0125] Real-time retention signals can quickly capture users’ real reactions to content. By analyzing users’ clicks, dwell time, return visit intervals and other behaviors in real time, the attractiveness and retention value of content can be quickly evaluated.

[0126] This mechanism enables the recommendation system to adjust its strategy in a timely manner and prioritize high-quality content that has been proven to retain users. The instant feedback mechanism reduces latency, improves the response speed and accuracy of the recommendation system, and ensures that the recommended content always keeps pace with user needs.

[0127] 2. Rich data:

[0128] Compared with traditional daily retention signals, our model uses denser minute-level data (signal aggregation granularity) and updates the model hourly (model update granularity). Through high-frequency data collection and processing, we can capture users' immediate behaviors more finely and obtain richer feedback signals. This data density not only improves the training effect of the model, but also enables the recommendation system to more accurately reflect users' real-time needs.

[0129] 3. Multi-stage optimization:

[0130] Integrate retention value applications in multiple stages such as recall, rough ranking, and fine ranking to ensure that recommended content can be optimized at every stage. From prioritizing the recall of content with high retention value, to comprehensive evaluation of multiple indicators (such as click-through rate, diversity, and retention value) in rough ranking, to personalized value evaluation and fine-tuning of retention models in fine ranking, each stage strives to present the content in the best possible way. This multi-level optimization strategy significantly improves the overall quality of recommended content.

[0131] 4. Significant effect:

[0132] Through a series of carefully designed experiments, we successfully applied the retention value modeling solution to the recall, rough ranking, and fine ranking of new and old users. We conducted more than ten experiments and achieved significant retention benefits in the online environment. These optimizations not only improved the user experience, but also increased user stickiness, reduced user churn, and enhanced users' willingness to continue using the platform. By continuously optimizing the recommendation strategy, it helps the platform establish long-term user loyalty, thereby improving the overall user retention rate and activity, and enhancing the platform's market competitiveness.

[0133] In some embodiments, after obtaining the target retention value comprehensive value of different contents, the recommendation system can be adjusted at various stages for users with a high probability of churn. Specifically:

[0134] Acquire a mapping relationship between the identifier of each accessed content and the login time of each accessing user for each accessed content within a time period before the current time;

[0135] According to the mapping relationship, the access retention amount of each accessing user for each accessed content is determined;

[0136] According to the access retention volume, each access user is clustered to obtain the clustering result;

[0137] According to the clustering results, the accessed content corresponding to the cluster center is recommended to each accessing user.

[0138] In some embodiments, determining the access retention amount of each accessing user for each accessed content according to the mapping relationship includes:

[0139] Determine the number of times each access party accesses each accessed object according to the mapping relationship;

[0140] Based on the number of visits and the login time of each visiting user to each visited object, the access retention of each visiting user to each visited object is determined. Specifically: based on each visiting user, the first login time of each visiting user to each visited content is searched from the preset database; based on the login time of each visiting user to each visited content and the first login time, the login time difference is obtained; based on the login time difference and the number of visits, the access retention of each visiting user to each visited content is determined.

[0141] Corresponding to the above method, the embodiment of the present application also provides a device for evaluating the retention value of content, such as Figure 2 As shown, the device comprises:

[0142] The acquisition unit 210 is used to obtain content features of the target log content and consumption behavior features of different users in a historical preset time period in response to a retention value evaluation request for target log content; wherein the retention value evaluation request includes the target log content and consumption behaviors of different users on the target log content in a historical preset time period; the historical preset time period includes a plurality of historical sub-time periods of equal length;

[0143] And, based on the content features and the consumption behavior features, obtaining the retention score of the target log content in each historical sub-time period;

[0144] The determination unit 220 is used to determine the target retention value comprehensive value of the target log content based on the retention points in each historical sub-time period and the configured weight of the corresponding historical sub-time period.

[0145] The functions of each functional unit of the content retention value evaluation device provided in the above-mentioned embodiment of the present application can be realized through the above-mentioned method steps. Therefore, the specific working process and beneficial effects of each unit in the content retention value evaluation device provided in the embodiment of the present application are not repeated here.

[0146] The present application also provides an electronic device, such as Figure 3 As shown, it includes a processor 310 , a communication interface 320 , a memory 330 and a communication bus 340 , wherein the processor 310 , the communication interface 320 , and the memory 330 communicate with each other via the communication bus 340 .

[0147] Memory 330, for storing computer programs;

[0148] The processor 310 is used to execute the program stored in the memory 330 to implement the following steps:

[0149] In response to a retention value evaluation request for target log content, obtain content features of the target log content and consumption behavior features of different users within a historical preset time period; wherein the retention value evaluation request includes the target log content and consumption behaviors of different users on the target log content within a historical preset time period; the historical preset time period includes a plurality of historical sub-time periods of equal length;

[0150] Based on the content features and the consumption behavior features, obtaining the retention score of the target log content in each historical sub-time period;

[0151] Based on the retention points in each historical sub-time period and the configured weight of the corresponding historical sub-time period, a target retention value comprehensive value of the target log content is determined.

[0152] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0153] The communication interface is used for communication between the above electronic device and other devices.

[0154] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0155] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0156] The implementation methods and beneficial effects of the components of the electronic device in the above embodiments to solve the problems can be found in Figure 1 The various steps in the illustrated embodiment are implemented, therefore, the specific working process and beneficial effects of the electronic device provided by the embodiment of the present application are not repeated here.

[0157] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which instructions are stored. When the computer-readable storage medium is executed on a computer, the computer executes the method for evaluating the content retention value described in any of the above embodiments.

[0158] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the method for evaluating the content retention value described in any of the above embodiments.

[0159] Those skilled in the art will appreciate that the embodiments in the present application can be provided as methods, systems, or computer program products. Therefore, the embodiments in the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments in the present application can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0160] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0161] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0162] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0163] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present application.

[0164] Obviously, those skilled in the art can make various changes and modifications to the embodiments in the present application without departing from the spirit and scope of the embodiments in the present application. Thus, if these modifications and variations of the embodiments in the present application fall within the scope of the claims and their equivalents in the embodiments of the present application, the embodiments of the present application are also intended to include these modifications and variations.

Claims

1. A method for evaluating the retention value of content, characterized in that: The method comprises: In response to a retention value evaluation request for target log content, obtain content features of the target log content and consumption behavior features of different users within a historical preset time period; wherein the retention value evaluation request includes the target log content and consumption behaviors of different users on the target log content within a historical preset time period; the historical preset time period includes a plurality of historical sub-time periods of equal length; Based on the content features and the consumption behavior features, the retention score of the target log content in each historical sub-time period is obtained; the retention score is determined by the number of refreshes of different users in each historical sub-time period obtained after the target log content was consumed before the historical preset time period, or is obtained by processing the content features and the consumption behavior features using a trained retention estimation model; Determine a target retention value comprehensive value of the target log content based on the retention points in each historical sub-time period and the configured weight of the corresponding historical sub-time period; Wherein, if the retention score is determined by the number of refreshes by different users in each historical sub-time period obtained after the target log content is consumed before the historical preset time period, then based on the retention score in each historical sub-time period and the configured weight of the corresponding historical sub-time period, the target retention value comprehensive value of the target log content is determined, including: Determine the retention value assessment value of the target log content on that day based on the retention score in each historical sub-time period, the configured weight of the corresponding historical sub-time period and the total number of refreshes on that day; Using a preset exponential smoothing moving average algorithm, the retention value evaluation value of the target log content on the current day and the target retention value comprehensive value of the target log content on the previous day are processed to obtain the target retention value comprehensive value of the target log content on the current day; if the current day is the initial day, the target retention value comprehensive value of the target log content on the previous day is the configured initial value; The calculation formula of the preset exponential smoothing moving average algorithm is expressed as: Among them, the represents the initial evaluation value on the mth day, represents the comprehensive value of the target retention value on the mth day, represents the comprehensive value of the target retention value on the m-1th day, K represents the number of adjacent days; the calculation formula for the retention value evaluation value of the target log content on the mth day is expressed as: , where N is the total number of users, T is the total number of historical sub-time periods, is the weight of the t-th historical sub-period, expressed as: , The retention score of the target log content corresponding to each user in each historical sub-time period: If the retention score is obtained by processing the content features and the consumption behavior features using a trained retention estimation model, then based on the retention scores in each historical sub-time period and the configured weights of the corresponding historical sub-time period, a target retention value comprehensive value of the target log content is determined, including: The retention estimation model includes an input terminal and multiple output terminals, wherein the multiple output terminals include an output terminal for outputting the retention score of each historical sub-time period, an output terminal for outputting the first retention value comprehensive value, and an output terminal for outputting whether the content feature has been consumed in the historical preset time period; Determine the second comprehensive retention value by multiplying the retention points in each historical sub-time period by the weight of the corresponding historical sub-time period; An average value of the first retention value comprehensive value and the second retention value comprehensive value is determined as a target retention value comprehensive value of the target log content at a current moment, where the current moment is a request moment of the retention value evaluation request.

2. The method according to claim 1, characterized in that The method further comprises: After determining the target retention value comprehensive values ​​corresponding to different types of target log content, a preset normalization algorithm is used to process each target retention value comprehensive value to obtain the retention value normalization values ​​corresponding to different types of target log content; Based on the normalized values ​​of retention values ​​corresponding to different types of target log content, the control strategies in the recall stage, rough ranking stage, fine ranking stage and re-ranking stage of the recommendation system are adjusted to obtain new control strategies for the corresponding stages.

3. A device for evaluating the retention value of content, characterized in that: The device comprises: an acquisition unit, configured to acquire, in response to a retention value evaluation request for target log content, content features of the target log content and consumption behavior features of different users within a historical preset time period; wherein the retention value evaluation request includes the target log content and consumption behaviors of different users on the target log content within a historical preset time period; and the historical preset time period includes a plurality of historical sub-time periods of equal length; And, based on the content features and the consumption behavior features, obtaining the retention score of the target log content in each historical sub-time period; the retention score is determined by the number of refreshes of different users in each historical sub-time period obtained after the target log content was consumed before the historical preset time period, or obtained by processing the content features and the consumption behavior features using a trained retention estimation model; a determination unit, configured to determine a target retention value comprehensive value of the target log content based on the retention points in each historical sub-time period and the configured weight of the corresponding historical sub-time period; Wherein, if the retention score is determined by the number of refreshes by different users in each historical sub-time period obtained after the target log content is consumed before the historical preset time period, the determination unit is specifically used to: Determine the retention value assessment value of the target log content on that day based on the retention score in each historical sub-time period, the configured weight of the corresponding historical sub-time period and the total number of refreshes on that day; Using a preset exponential smoothing moving average algorithm, the retention value evaluation value of the target log content on the current day and the target retention value comprehensive value of the target log content on the previous day are processed to obtain the target retention value comprehensive value of the target log content on the current day; if the current day is the initial day, the target retention value comprehensive value of the target log content on the previous day is the configured initial value; The calculation formula of the preset exponential smoothing moving average algorithm is expressed as: Among them, the represents the initial evaluation value on the mth day, represents the comprehensive value of the target retention value on the mth day, represents the comprehensive value of the target retention value on the m-1th day, K represents the number of adjacent days; the calculation formula for the retention value evaluation value of the target log content on the mth day is expressed as: , where N is the total number of users, T is the total number of historical sub-time periods, is the weight of the t-th historical sub-period, expressed as: , The retention score of the target log content corresponding to each user in each historical sub-time period: If the retention score is obtained by processing the content features and the consumption behavior features using a trained retention prediction model, the determination unit is specifically configured to: The retention estimation model includes an input terminal and multiple output terminals, wherein the multiple output terminals include an output terminal for outputting the retention score of each historical sub-time period, an output terminal for outputting the first retention value comprehensive value, and an output terminal for outputting whether the content feature has been consumed in the historical preset time period; Determine the second comprehensive retention value by multiplying the retention points in each historical sub-time period by the weight of the corresponding historical sub-time period; An average value of the first retention value comprehensive value and the second retention value comprehensive value is determined as a target retention value comprehensive value of the target log content at a current moment, where the current moment is a request moment of the retention value evaluation request.

4. An electronic device, characterized in that: The electronic device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; A processor, used to implement the method described in any one of claims 1-2 when executing a program stored in a memory.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 2 is implemented.

Images