Content recommendation adjustment

Abstract

According to embodiments of the present disclosure, a solution for content recommendation adjustment is provided A method comprises: determining a predicted duration difference between a first retention duration of the target user in an application and a second retention duration of the target user in an application before and after applying a content recommendation treatment to the target user; determining a predicted cost difference between a first cost introduced by the target user and a second cost introduced by the target user before and after applying the content recommendation treatment to the target user; determining a recommendation change coefficient for the target user based on the predicted duration difference and the predicted cost difference; adjusting content recommendation parameter(s) for the target user based on the recommendation change coefficient; and providing one recommended content item(s) to the target user based on the adjusted content recommendation parameter(s).

Description

CONTENT RECOMMENDATION ADJUSTMENTField

[0001] The disclosed example embodiments relate generally to the field of computer technology, and, more particularly, to a method, apparatus, device and computer readable storage medium for content recommendation adjustment.Background

[0002] Internet provides access to a variety of resources. For example, various applications, products, and audio and video content can be accessed through the Internet. In addition, the accessible content also includes specific recommended content items (such as advertisements) related to various objects / resources. The resource providers can provide the content delivery provider with the recommended content items for delivery to users.Summary

[0003] In a first aspect of the present disclosure, there is provided a method for content recommendation. The method comprises: determining, based on feature information of a target user, a predicted duration difference between a first retention duration of the target user in an application before applying a content recommendation treatment to the target user and a second retention duration of the target user in the application after applying the content recommendation treatment to the target user; determining, based on the feature information, a predicted cost difference between a first cost introduced by the target user before applying the first content recommendation treatment to the target user and a second cost introduced by the target user after applying the content recommendation treatment to the target user; determining a recommendation change coefficient for the target user based on the predicted duration difference and the predicted cost difference; adjusting at least one content recommendation parameter for the target user based at least in part on the recommendation change coefficient; and providing at least one recommended content item to the target user based at least in part on the at least one adjusted content recommendation parameter.

[0004] Tn a second aspect of the present disclosure, there is provided an apparatus for content recommendation. The apparatus comprises: a duration difference determining module configured to determine, based on feature information of a target user, a predicted duration difference between a first retention duration of the target user in an application before applying a content recommendation treatment to the target user and a second retention duration of the target user in the application after applying the content recommendation treatment to the target user; a cost difference determining module configured to determine, based on the featureinformation, a predicted cost difference between a first cost introduced by the target user before applying the first content recommendation treatment to the target user and a second cost introduced by the target user after applying the content recommendation treatment to the target user; a coefficient determining module configured to determine a recommendation change coefficient for the target user based on the predicted duration difference and the predicted cost difference; a recommendation adjustment module configured to adjust at least one content recommendation parameter for the target user based at least in part on the recommendation change coefficient; and a content providing module configured to provide at least one recommended content item to the target user based at least in part on the at least one adjusted content recommendation parameter.

[0005] In a third aspect of the present disclosure, there is provided an electronic device. The device comprises: at least one processing unit; and at least one memory', the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. The instructions, when executed by at least one processing unit, cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, there is provided a computer-readable storage medium. The medium stores a computer program which, when executed by a processor, implements the method of the first aspect.

[0007] In a fifth aspect of the present disclosure, there is provided a computer program product. The computer program product stores computer readable instructions which, when executed by' a processor, implements the method of the first aspect.

[0008] It would be appreciated that the content described in the section is neither intended to identify' key' or essential features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily envisaged through the following description.Brief Description of the Drawings

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent in combination with the accompanying drawings and with reference to the following detailed description. In the drawings, the same or similar reference symbols refer to the same or similar elements, where:

[0010] FIG. 1 illustrates a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0011] FIG. 2 illustrates a flowchart of a process for content recommendation according to some embodiments of the present disclosure;

[0012] FIG. 3 illustrates a schematic diagram of model-based prediction according to some embodiments of the present disclosure;

[0013] FIG. 4 illustrates a schematic diagram of a timeline for treatment data collection according to some embodiments of the present disclosure;

[0014] FIG. 5 illustrates a schematic diagram of an overall architecture for content recommendation according to some embodiments of the present disclosure;

[0015] FIG. 6 illustrates a block diagram of an apparatus for content recommendation adjustment according to some embodiments of the present disclosure; and

[0016] FIG. 7 illustrates an electronic device in which one or more embodiments of the present disclosure can be implemented.Detailed Description

[0017] The embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the drawings, it would be appreciated that the present disclosure may be implemented in various forms and should not be interpreted as limited to the embodiments described herein. On the contrary, these embodiments are provided for a more thorough and complete understanding of the present disclosure. It would be appreciated that the drawings and embodiments of the present disclosure are only for the purpose of illustration and are not intended to limit the scope of protection of the present disclosure.

[0018] In the description of the embodiments of the present disclosure, the term "including" and similar terms would be appreciated as open inclusion, that is, "including but not limited to". The term "based on" would be appreciated as "at least partially based on". The term "one embodiment" or "the embodiment" would be appreciated as "at least one embodiment". The term "some embodiments" would be appreciated as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0019] It will be appreciated that the data involved in this technical solution (including but not limited to the data itself, data acquisition or use) shall comply with the requirements of corresponding laws, regulations and relevant provisions.

[0020] It will be appreciated that before using the technical solution disclosed in each embodiment of the present disclosure, users should be informed of the type, the scope of use, the use scenario, etc. of the personal information involved in the present disclosure in an appropriate manner in accordance with relevant laws and regulations, and the user’s authorization should be obtained.

[0021] For example, in response to receiving an active request from a user, a prompt message is sent to the user to explicitly prompt the user that the operation requested operation by the user will need to obtain and use the user's personal information, so that users may select whether to provide personal information to the software or the hardware such as an electronicdevice, an application, a server or a storage medium that perform the operation of the technical solution of the present disclosure according to the prompt information.

[0022] As an optional but non-restrictive implementation, in response to receiving the user's active request, the method of sending prompt information to the user may be, for example, a pop-up window in which prompt information may be presented in text. In addition, pop-up windows may also contain selection controls for users to choose “agree” or “disagree” to provide personal information to electronic devices.

[0023] It will be appreciated that the above notification and acquisition of user authorization process are only schematic and do not limit the implementations of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0024] As used herein, the term "model" can learn an association between respective inputs and outputs from training data, so that a corresponding output can be generated for a given input after training is completed. The generation of the model can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides a corresponding output by using multiple layers of processing units. A neural network model is an example of a deep learning-based model. As used herein, "model" may also be referred to as "machine learning model", "learning model", "machine learning network", or "learning network", and these terms are used interchangeably herein.

[0025] ‘ ‘Neural network” is a type of machine learning network based on deep learning. A neural network is capable of processing inputs and providing corresponding outputs, which typically includes an input layer, an output layer, and one or more hidden layers between the input layer and output layer. Neural networks used in deep learning applications typically comprise a number of hidden layers, thereby increasing the depth of the network. The layers of neural networks arc sequentially connected so that the output of the previous layer is provided as input to the latter layer, where the input layer receives the input of the neural network and the output of the output layer serves as the final output of the neural network. Each layer of a neural network comprises one or more nodes (also known as processing nodes or neurons), each of which processes input from the previous layer.

[0026] Usually, machine learning may roughly comprise three stages, namely training stage, test stage, and application stage (also known as inference stage). During the training stage, a given model can be trained using a large scale of training data, iteratively updating parameter values until the model can obtain consistent inference from the training data that meets the expected objective. Through the training, the model may be considered to learn the association between the inputs and outputs (also known as input-to-output mapping) from the training data. The parameter values of the trained model are determined. In the test stage, test inputs are applied to the trained model to test whether the model can provide correct outputs, therebydetermining the model performance. In the application stage, the model can be used to process actual inputs and determine corresponding outputs based on the parameter values obtained from training .

[0027] FIG. 1 illustrates a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. One or more content providers can use a content management system 120 to manage content on a content delivery system 1 10. One or more terminal devices 130-1, 130-2, 130-3, etc. (for case of discussion, collectively or individually referred to as terminal devices 130) are associated with the content delivery system 110, and can access various types of content provided on content delivery system 110 for example, based on corresponding users 132-1, 132-2, 132-3, etc. (for ease of discussion, collectively or individually referred to as users 132). As an example, the content delivery system 110 can be an application, a website, a webpage, or other accessible platform. Terminal device 130 can be installed with applications for accessing the content delivery system 110, or can access the content delivery system 110 in a suitable manner.

[0028] The content management system 120 can be configured to deliver one or more specific recommended content items related to one or more resources to the users according to corresponding policies (such as providing or presenting on the terminal devices 130). The recommended content items to be delivered for example, may include one or more recommended content items 142-1, 142-2, ...,142-M (for ease of discussion, collectively or individually referred to as recommended content items 142) in a content database 140.

[0029] The resources to be recommended may, for example, include various recommended objects, examples of which may include applications, physical products, virtual products, audio and video content, and so on. Recommended content items may include content presented to recommend corresponding resources. Examples of recommended content items may include advertisements (or ads for short). The user group may include one or more user members, such as users 132. User members can be any potential consumers of resources, such as users, groups, organizations, entities, and so on.

[0030] In some embodiments, the content management system 120 may distribute corresponding recommended content items on the content delivery system 110 based on requests from resource providers 150-1, 150-2, 150-3, etc. (for ease of discussion, collectively or individually referred to as resource providers 150). Tn some embodiments, the content management system 120 may deliver recommended content items 142 to corresponding users 132 on the content delivery system 110 at least based on requests from various resource providers 150. In advertising delivery scenarios, a resource provider 150 is sometimes referred to as an advertiser. In some embodiments, the resource providers may also pay the content providers for the presentation of recommended content items and subsequent conversion behaviors performed on the recommended content items.

[0031] In some embodiments, the content management system 120 may select recommended content items for presentation to a specific terminal device 130 in a content delivery opportunity (e.g., at a specific time and location) of content delivery system 110 based on bidding results. For example, the content management system 120 may receive bids from resource providers 150. In some embodiments, the content management system 120 may allocate content delivery' opportunities to the highest bidder, which means that the corresponding recommended content item can be successfully delivered in contending delivery. Bid may refer to the cost of contending for a certain recommended content item in a certain content delivery opportunity . A recommended content item that is successfully delivered at a certain cost is called a send, and the cost is called the rank bid for this delivery or send.

[0032] In the environment 100, the terminal device 130 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDA), audio / video Player, digital cameras / video cameras, positioning devices, television receivers, radio broadcast receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, terminal device 130 can also support any' type of user-specific interface (such as "wearable" circuits, etc.). In the environment 100, the content delivery' system 110 and / or the content management system 120 can be various ty'pes of computing systems / servers that can provide computing power, including but not limited to mainframes, edge computing nodes, data centers, cloud computing environments, etc.

[0033] It should be appreciated that the structure and function of each element in the environment 100 are described for illustrative purposes only, without implying any limitation on the scope of the present disclosure.

[0034] The content delivery' system usually grapples with the tough problem of gaining more user retention while minimizing the loss of cost metrics with recommended traffic re-allocation. The retention gain of a user may be measured by retention duration of the user in the application of the content delivery system, which may' be referred to as a stay duration metric, or an active time or days of the user in the application. The cost metrics may be measured as cost introduced by users and paid by the resource providers to the content delivery' system.

[0035] The content delivery’ system may rely on various content recommendation parameters to decide how the content items are recommended to the respective users. By adjusting one or more of the content recommendation parameters, it is possible to adjust the amount of recommended content items provided to the users. Unappreciated adjustments on the content recommendation parameters may result in a large decrease in cost with a less favorable exchange rate of retention gain and cost.

[0036] Usually, if more recommended content items are provided to a user, the cost or the cost metric may be raised as the resource providers may pay more for the large amount of recommended content items. However, an increase in the recommended content items may reduce the retention duration of some users in the application, which may cause a less favorable exchange rate of the retention gain and the cost.

[0037] Currently, there are no effective traffic strategies for gaining or maintaining user experience without much overall cost fluctuation. In some potential solutions, the number of active days of a user among the last 14 days are collected as an estimator signal, where an active day of a user refers to a day when the user launches the application or stay in an active stay in the application for a certain duration. Then users are divided into an active user group (e.g., an active user bucket) with the number of active days higher than a threshold number, and a less-active user group (e.g., a less-active user bucket) with the number of active days lower than the threshold number. The content delivery system may provide more recommended content items to users in the active user groups, and provide less recommended content items to users in the less-active user groups. However, as users in the active and less-active user groups divided by the number of active days arc varied in different regions and / or scenarios, it is difficult to adjust the content recommendation parameters globally. In addition, there is always traffic shift between the user groups divided based on such estimator signal, making the distribution volatile over time. Further, the number of active days fails to indicate the compromise degree of the cost and the retention duration. By providing a majority of recommended content items to the active user group, it is not sure whether the divided user groups can achieve a tradeoff between the retention duration and the cost by the users.

[0038] In embodiments of the present disclosure, an improved solution for content recommendation adjustment is proposed. In the solution, a recommendation change coefficient is introduced which is determined based on a predicted duration difference and a predicted cost difference each calculated before and after applying a content recommendation treatment to the target user. The recommendation change coefficient may be used to measure the exchange rate of the retention duration and the cost after applying the content recommendation treatment. Then at least one content recommendation parameter for the target user is adjusted based at least in part on the recommendation change coefficient, and at least one recommended content item is provided to the target user based at least in part on the at least one adjusted content recommendation parameter. In this solution, as the recommendation change coefficient can measure the exchange rate of the retention duration and the cost after applying the content recommendation treatment, a content delivery system can easily determine a more appropriate recommendation strategy, e.g., to determine whether or not the content recommendation treatment is to be applied, so as to adjust one or more content recommendation parameters. In this way, the users can still obtain reasonable recommended content items, while the contentdelivery system can retain the user stay duration in the application, while alleviating the cost reduction.

[0039] Some example embodiments of the present disclosure will be discussed in detail with reference to the accompanying drawings.

[0040] FIG. 2 illustrates a flowchart of a process 200 for content recommendation according to some embodiments of the present disclosure. The process 200 may be implemented at the content delivery system 110 or any other suitable device or system that is operative to determine a recommendation strategy for users 132 associated with the content delivery system 110. For easy of discussion, the process 200 is described from the perspective of the content delivery system 110.

[0041] In the embodiments of the present disclosure, a recommendation change coefficient is introduced to measure the exchange rate of the retention duration and the cost after applying a content recommendation treatment.

[0042] At block 210, the content delivery system 110 determines, based on feature information of a target user, a predicted duration difference between a first retention duration of the target user in an application before applying a content recommendation treatment to the target user and a second retention duration of the target user in the application after applying the content recommendation treatment to the target user. At block 220, the content delivery system 110 determines, based on the feature information, a predicted cost difference between a first cost introduced by the target user before applying the first content recommendation treatment to the target user and a second cost introduced by the target user after applying the content recommendation treatment to the target user.

[0043] The target user is any user of intertest to which content items are to be recommended. A content recommendation treatment refers to a treatment that is applied in content recommendation for the target user. Examples of the content recommendation treatment may include an increase in an amount of content items to be provided to a user, or a decrease in an amount of content items to be provided to a user. In some cases, the content recommendation treatments may be divided based on different percentage increases and / or different percentage decrease in the amount of content items to be provided to the user, e.g., a 10% increase, a 20% increase, a 10% decrease, a 20% decrease, etc. In the embodiments of the present disclosure, for each different content recommendation treatment, the predicted duration difference and the predicted cost difference arc measured to determine a recommendation change coefficient for this content recommendation treatment.

[0044] A retention duration of a user in an application may be measured as a stay duration or a playtime duration of the user in the application (e.g., using the application), a duration or a number of days when the user is active in the application, or any other suitable metrics. A cost introduced by a user is the cost of recommending content items to the user. The cost may bemeasured using various cost metrics, such as the cost paid by the resource providers requesting provision of the content items, and / or any other suitable metrics.

[0045] In some embodiments, the feature information of the target user may include user demographical information related to the target user. Examples of the user demographical information may include some attribute information that may characterize the user. In some embodiments, the feature information of the target user may alternatively or additionally include provision information related to historical content recommendation items provided to the target user, which may include aggregated user-level metrics for content recommendations such as the number of recommended content items provided to the user, and / or the cost amount introduced by the recommended content items provided to the user.

[0046] In some embodiments, the aggregated user-level metrics for content recommendations may further include interaction information of the target user with respect to the historical content recommendation items. The interaction information between the user and the recommended content items may include, for example, the number of recommended content items liked by the user during a historical time period, the number of recommended content items disliked by the user during a historical time period, and / or information of other interaction behaviors performed by the user on the recommended content items.

[0047] In some embodiments, the feature information of the target user may include information related to retention durations of the target user in the application. The information related to retention durations of the target user in the application may include aggregated user retention metrics, such as the stay duration or the playtime duration of the target user in a historical time period, the number of active time periods (e.g., active days) of the target user in the historical time period, or the count of content items viewed or played by the target users in the historical time period.

[0048] It would be appreciated that only some example feature information that would impact on the retention duration and / or the cost in the content recommendation are provided above. There would be various other feature information that may be involved in the content recommendation, which is not limited in the scope of the present disclosure.

[0049] With the content recommendation treatment applied, the retention duration of the target user in the application and / or the cost introduced by the user may be changed. The predicted duration difference and the predicted cost difference are measured in order to determine a recommendation change coefficient. Then at block 230, the content delivery system 110 determines a recommendation change coefficient for the target user based on the predicted duration difference and the predicted cost difference.

[0050] In some examples, the recommendation change coefficient may be calculated as a ratio of the predicted duration difference to the predicted cost difference or vice versa. Anexample calculation of the recommendation change coefficient SensScore may be represented as follows:where Δ(Duration) represents the predicted duration difference, and A(Cost) represents the predicted cost difference.

[0051] The calculation of Δ(Duration) may be represented as follows: Δ(Duration) = E[Duration (treatment) \X] — E[Duration (controZ)|X] (2) where E[Duration (treatment) \X] represents the second retention duration with a content recommendation treatment applied, E[Duration (control) |X] represents the first retention duration without the content recommendation treatment applied, and X represents feature information of a user.

[0052] Similarly, the calculation of A(Cost) may be represented as follows: Δ(Cost) = E[Cost (treatment)\X] — E[Cost (controZ)|X] (3) where E[Cost (treatment) |X] represents the second cost introduced by a user with a content recommendation treatment applied, E[Cost (control) |X] represents the first cost introduced by a user without the content recommendation treatment applied, and X represents feature information of a user.

[0053] A higher recommendation change coefficient may indicate that there is a relatively large shift in the retention duration and / or a relatively small shift in the cost after applying the content recommendation treatment, which is usually not a desirable result in the scenario of content recommendation. In the scenario of content recommendation, if a content recommendation treatment of providing more content items to the user is applied, it is desired to keep the retention duration of the user constant or as close as possible while increasing the cost introduced by the user. A smaller recommendation change coefficient may indicate that there is a relatively small shift in the retention duration and / or a relatively large shift in the cost after applying the content recommendation treatment, which is more desirable. The recommendation change coefficient may thus indicate the sensitivity level of the user to the content recommendation treatment to be applied.

[0054] As it is impossible to observe that a specific user is applied with and without a content recommendation treatment at the same time, it is assumed that there are a treatment group and a control group. A user with the content recommendation treatment applied may be in a treatment group, and a user without the content recommendation treatment applied may be in a control group. Thus, it may need to predict Δ(Duration) and A(Cost), or more specifically, to predict E [Dur al ion (treatment) \X] , E[Duration (controZ)|X] , E[Cost (treatment)\X] , E[Duration (controZ)|X] by assuming that a target user is classified as a user in the treatment group and classified as a user in the control group.

[0055] In some embodiments, machine learning models may be trained to determine the predicted duration difference and the predicted cost difference for the target user. The feature information may be collected for a past duration from the target user, before the target user is exposed to the content recommendation treatment. The machine learning models may be applied to predict the retention duration difference and the cost difference for the user in different treatment conditions (with and without a specific content recommendation treatment applied).

[0056] FIG. 3 illustrates a schematic diagram 300 of model-based prediction according to some embodiments of the present disclosure. As shown, for a target user, user 132, feature information of the user 132 may be provided to a trained machine learning model 310 to determine the predicted duration difference Δ(Duration) 312. The feature information of the user 132 may also be provided to a trained machine learning model 320 to determine the predicted cost difference Δ(Cost) 322. Then the predicted duration difference Δ{Duration) 312 and the predicted cost difference Δ(Cost') 322 are used to determine the recommendation change coefficient SensScore 330. Such a model architecture may be constructed based on an uplift model.

[0057] In some embodiments, the outputs of the trained machine learning model 310 and the trained machine learning model 330 may directly be the predicted duration difference Δ(Duration') 312 and the predicted cost difference Δ(Cost) 322.

[0058] In some embodiments, the machine learning model 310 may include a first duration prediction model configured to determine the first retention duration of the target user in the application without the content recommendation treatment applied, and a second duration prediction model configured to determine the second retention duration of the target user in the application with the content recommendation treatment applied. The input of both duration prediction models may be the feature information of the target user. Then the predicted duration difference Δ(Duration) 312 may be calculated based on a difference from the second retention duration to the first retention duration.

[0059] In some embodiments, the machine learning model 320 may include a first cost prediction model configured to determine a first cost introduced by the target user without the content recommendation treatment applied, and a second cost prediction model configured to determine a second cost introduced by the target user with the content recommendation treatment applied. The input of both cost prediction models may be the feature information of the target user. Then the predicted cost difference Δ(Cost) 322 may be calculated based on a difference from the second cost to the first cost.

[0060] At block 240, the content delivery system 110 adjusts at least one content recommendation parameter for the target user based at least in part on the recommendation change coefficient. In some embodiments, the at least one content recommendation parametermay be determined further based on the content recommendation treatment (which indicates the percentage increase or percentage decrease). The content recommendation parameter(s) is those that are applied by the content delivery system 110 to determine the recommendation strategy applied to the target user. The recommendation strategy is applied to determine the specific content items and / or the amount of content items to be provided by the content delivery system 1 10.

[0061] In some embodiments, in response to the recommendation change coefficient being a first value, the at least one content recommendation parameter may be determined to be at least one first parameter value for to cause a first amount of recommended content items to be provided to the target user; and in response to the recommendation change coefficient being a second value higher than the first value, the at least one content recommendation parameter may be determined to be at least one second parameter value respectively to cause a second amount of recommended content items to be provided to the target user. Then the second amount may be determined to be lower than the first amount. That is, for a user with a higher recommendation change coefficient, the at least one content recommendation parameter may be adjusted such that fewer content items arc recommended to the user.

[0062] In some embodiments, the at least one content recommendation parameter may include a time gap between content recommendation items provided to the target user. For example, by increasing the time gap, a smaller amount of content items may be recommended to the target user, and by increasing the time gap, a larger amount of content items may be recommended to the target user.

[0063] In some embodiments, the at least one content recommendation parameter an estimated cost per mille (ECPM) threshold for the metric. By increasing the ECPM threshold, a smaller amount of content items may satisfy the ECPM requirement, and a smaller amount of content items may be recommended to the target user. On the other hand, by decreasing the ECPM threshold, a larger amount of content items may satisfy the ECPM requirement, and a smaller amount of content items may be recommended to the target user.

[0064] Although some example content recommendation parameters are provided above, it would be appreciated that there would be various other parameters that are involved in the recommendation strategy and have impacts on the amount of recommended content items to be provided to the user. Tn some embodiments, only one or more than one content recommendation parameter may be adjusted based on the recommendation change coefficient.

[0065] At block 250, the content delivery system 110 provides at least one recommended content item to the target user based at least in part on the at least one adjusted content recommendation parameter. With the at least one adjusted content recommendation parameter, the recommendation strategy may be determined, to guide the content recommendation for the target user in the application.

[0066] With the machine learning models, e.g., the machine learning models 310 and 320 applied, training data may be collected for training the machine learning models 310 and 320. The training data may be collected from sample users in a treatment group with the content recommendation treatment applied, and sample users in a control group without the content recommendation treatment applied. With such settings, counterfactual data can be collected from the treatment group and the control group to be used as the training data for the machine learning models 310 and 320. In some embodiments, the sample users in the control group and the treatment group may share the same or similar feature information.

[0067] For users in the control group, the content recommendation may be applied according to a benchmark recommendation strategy, while for users in the treatment group, the content recommendation may be treated with shift traffic conditions, to increase or decrease the amount of recommended content items. By shifting at least one content recommendation parameter (e.g., the time gap, or ECPM threshold), a delta amount of content items may be added on top of the benchmark recommendation strategy. The content recommendation treatment aims to aggressively shift the amount of recommended content items sent to the users in order to observe the treatment effect on the retention duration and the cost.

[0068] FIG. 4 illustrates a schematic diagram of a timeline 400 for treatment data collection according to some embodiments of the present disclosure. Before the content recommendation treatment is applied to the treatment group at TO, sample feature information of respective sample users in the treatment group and the control group are collected. In some cases, considering that the average treatment effect (ATE) for the retention duration (and / or the cost) usually takes some time (e.g., about two weeks, or some other time depending on the actual application) to stabilize, the treatment outcomes for the retention duration and the cost are collected after a first period of time when the contention recommendation treatment is applied to the sample users in the treatment group.

[0069] As shown in the example of FIG. 4, the outcome measurement may be started at the 15thday (T0+15D) from the time TO when the content recommendation treatment is applied and ended at the 28thdays (T0+28D). The daily retention durations and costs of the sample users in both the treatment group and the control group may be collected during the measurement period. Then the daily average retention durations and costs 440 may be calculated based on the collected measurement data during the measurement period. It would be appreciated that FIG. 4 merely provides a specific example for the treatment data collection and there will be various variables that can be controlled or changed for the treatment data collection, including the measurement period, the start time for the measurement collection, and whether the average values are to be calculated. In some embodiments, the feature information of the sample users may be collected before the measurement period is started, e.g., from TO to T0+13 days in the example of FIG. 4.

[0070] In some embodiments, the machine learning model 310 may be trained with a first training dataset comprising a plurality of first training samples. A first training sample for the machine learning model 310 may include sample feature information of a first sample user in the control group and a second sample user in the treatment group. The first training sample may further include a labeled duration difference between a first sample retention duration of the first sample user in the application without the content recommendation treatment applied to the first sample user and a second sample retention duration of the second sample user in the application with the content recommendation treatment applied to the second sample user. The first sample retention duration may be collected during the measurement period for the first sample user in the control group, while the second sample retention duration may be collected during the measurement period for the second sample user in the treatment group.

[0071] In some embodiments, the machine learning model 320 may be trained with a second training dataset comprising a plurality of second training samples. A second training sample for the machine learning model 320 may include the sample feature information of the first sample user and the second sample user. The second training sample may further include a labeled cost difference between a first sample cost introduced by the first sample user without the content recommendation treatment applied to the first sample user and a second sample cost introduced by the second sample user with the content recommendation treatment applied to the second sample user. The first sample cost may be collected during the measurement period for the first sample user in the control group, while the second sample cost may be collected during the measurement period for the second sample user in the treatment group.

[0072] In some embodiments, the model structures of the machine learning models 310 and 320 may be designed according to any suitable architecture, e.g., by using a regression model as the base model which could determine user retention duration and cost characteristics. It would be appropriated that various other model structures may be applied. In some embodiments, a two-stage meta-learner (also referred to as X-learner) may be applied to train the machine learning models 310 and 320 as such leaner can have better bias-variance tradeoff than the one-stage meta-leaner (e.g., S-learner or T-leamer).

[0073] According to the two-stage meta-learner, a first training stage is to train a first duration prediction model configured to determine a retention duration of a user in an application without the content recommendation treatment applied and a second duration prediction model configured to determine a retention duration of a user in an application with the content recommendation treatment applied. The first training samples may be applied to train the two duration perdition models.

[0074] The training targets of the two duration prediction models are based on the first retention duration E[Duration (control) |X] and the second retention duration E[Duration (;reatment)\X] because the direct outputs of the two duration prediction modelsare the two kinds of retention durations, not the retention duration difference. In this case, the predicted duration difference ΔDuration may be determined by calculating the difference between the outputs of the two duration prediction models.

[0075] Similarly, the machine learning model 320 may include a first cost prediction model configured to determine a cost introduced by a user in an application without the content recommendation treatment applied, and a second cost prediction model configured to determine a cost introduced by a user with the content recommendation treatment applied. The training target of the two duration prediction models are the first retention duration E[Cost (control) |X] and the second retention duration E [Cos treatment) |X] because the direct outputs of the two cost prediction models arc the two kinds of costs, not the cost difference. In this case, the predicted cost difference ΔCost may be determined by calculating the difference between the outputs of the two cost prediction models.

[0076] At a second stage, the machine learning model 310 that outputs the predicted duration difference ΔDuration may be further trained with a training target of the first machine learning model is configured to minimize or reduce an error between the labeled duration difference and an output duration difference between a predicted retention duration output by the first duration prediction model and a predicted retention duration output by a second duration prediction model. Similarly, a training target of the machine learning model 320 that outputs the predicted cost difference ΔCost is configured to minimize or reduce an error between the labeled cost difference and an output duration difference between a predicted cost provided by the first cost prediction model and a predicted cost provided by a second cost prediction model, the predicted cost difference

[0077] Then the machine learning model 310 and machine learning model 320 trained in the second stage may be applied later for model inference, to determine the predicted duration difference and the cost difference for a target user.

[0078] In some embodiments, the one-stage meta-learner may be applied to directly train the machine learning model 310 and machine learning model 320. In this model training mechanism, the machine learning model 310 with the two duration prediction models, and the machine learning model 320 with the two cost prediction models may be applied for model inference. In some other cases, the machine learning models 310 and 320 with the direct outputs of duration difference and cost difference may be trained with the one-stage learner and applied for later model inference.

[0079] FIG. 5 illustrates a schematic diagram of an overall architecture 500 for content recommendation according to some embodiments of the present disclosure. As shown in FIG. 5, in the model training and update stage 510, treatment data are collected at 511 and are stored in a database 512. The collected treatment data may be aggregated at 513 to formulate as training samples for the machine learning models 310 and 320, with feature information and labels. Atmodel retraining 514, the machine learning models 310 and 320 are trained using the collected training samples.

[0080] At a full batch feature aggregation and inference stage 520, the machine learning models 310 and 320 may be applied at 523 for prediction, to obtain recommendation change coefficients of respective users to which content recommendation is applicable. The input to the trained machine learning models 310, 320 are obtained through user feature generation 522 from a database 521 which stores feature information of the users. Then at 524, for a specific target user, it may be allocated to a user group (also referred to as a user backet) based on the recommendation change coefficient. The user groups may be ranked, e.g., with a smaller bucket number to represent users with smaller recommendation change coefficients. Then at least one content recommendation parameter may be determined for the user group based on the recommendation change coefficients of users in the user group. A user group may share the same recommendation strategy.

[0081] As shown in FIG. 5, after allocation of the users in the user buckets, the allocation may be recorded in a database 531. At the online serving stage 530, one or more content recommendation parameters 533 may be adjusted based on recommendation change coefficients of users in each user bucket, to derive a content recommendation strategy 534. Then the content provisioning adjustment 535 may be determined for the users in the user bucket to determine the recommended content items to be provided to the users.

[0082] Through the embodiments of the present disclosure, it is possible to achieve a tradeoff between the retention duration and the cost by applying different content recommendation treatments .

[0083] FIG. 6 illustrates a block diagram of an apparatus 600 for content recommendation adjustment according to some embodiments of the present disclosure. The apparatus 600 may be implemented as or included in the content delivery system 110. Various modulcs / componcnts in the apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0084] As shown, the apparatus 600 comprises a duration difference determining module 610 configured to determine, based on feature information of a target user, a predicted duration difference between a first retention duration of the target user in an application before applying a content recommendation treatment to the target user and a second retention duration of the target user in the application after applying the content recommendation treatment to the target user; and a cost difference determining module 620 configured to determine, based on the feature information, a predicted cost difference between a first cost introduced by the target user before applying the first content recommendation treatment to the target user and a second cost introduced by the target user after applying the content recommendation treatment to the target user.

[0085] The apparatus 600 further comprises a coefficient determining module configured 630 to determine a recommendation change coefficient for the target user based on the predicted duration difference and the predicted cost difference, and a recommendation adjustment module 640 configured to adjust at least one content recommendation parameter for the target user based at least in part on the recommendation change coefficient.

[0086] The apparatus 600 further comprises a content providing module 650 configured to provide at least one recommended content item to the target user based at least in part on the at least one adjusted content recommendation parameter.

[0087] In some embodiments, the recommendation adjustment module 640 is further configured to: in response to the recommendation change coefficient being a first value, determine at least one first parameter value for the at least one content recommendation parameter to cause a first amount of recommended content items to be provided to the target user; and in response to the recommendation change coefficient being a second value higher than the first value, determine at least one second parameter value for the at least one content recommendation parameter to cause a second amount of recommended content items to be provided to the target user, the second amount being lower than the first amount.

[0088] In some embodiments, the recommendation adjustment module 640 is further configured to: allocate the target user to a user group based on the recommendation change coefficient; and determine at least one content recommendation parameter for the user group based on recommendation change coefficients of users in the user group.

[0089] In some embodiments, the duration difference determining module 610 configured to: determine the predicted duration difference using a trained first machine learning model based on the feature information. In some embodiments, the duration difference determining module 610 is configured to: determine the predicted cost difference using a trained second machine learning model based on the feature information.

[0090] In some embodiments, the first machine learning model is trained with a first training dataset comprising a plurality of first training samples, a first training sample comprising: sample feature information of a first sample user and a second sample user, and a labeled duration difference between a first sample retention duration of the first sample user in the application without the content recommendation treatment applied to the first sample user and a second sample retention duration of the second sample user in the application with the content recommendation treatment applied to the second sample user. In some embodiments, the first sample retention duration and the second sample retention duration are collected for the first sample user and the second sample user after a first period of time when the contention recommendation treatment is applied to the second sample user.

[0091] In some embodiments, the second machine learning model is trained with a second training dataset comprising a plurality of second training samples, a second training samplecomprising: the sample feature information of the first sample user and the second sample user, and a labeled cost difference between a first sample cost introduced by the first sample user without the content recommendation treatment applied to the first sample user and a second sample cost introduced by the second sample user with the content recommendation treatment applied to the second sample user. In some embodiments, the first sample cost and the second sample cost are collected for the first sample user and the second sample user after a second period of time when the contention recommendation treatment is applied to the second sample user.

[0092] In some embodiments, the first machine learning model comprises a first duration prediction model configured to determine a retention duration of a user in an application without the content recommendation treatment applied, and a second duration prediction model configured to determine a retention duration of a user in an application with the content recommendation treatment applied; and wherein a training target of the first machine learning model is configured to minimize or reduce an error between the labeled duration difference and an output duration difference between a predicted retention duration output by the first duration prediction model and a predicted retention duration output by a second duration prediction model

[0093] In some embodiments, the second machine learning model comprises a first cost prediction model configured to determine a cost introduced by a user without the content recommendation treatment applied, and a second cost prediction model configured to determine a cost introduced by a user with the content recommendation treatment applied; and wherein a training target of the second machine learning model is configured to minimize or reduce an error between the labeled cost difference and an output duration difference between a predicted cost provided by the first cost prediction model and a predicted cost provided by a second cost prediction model.

[0094] In some embodiments, the feature information of the target user comprises at least one of the following: user demographical information of the target user, provision information related to historical content recommendation items provided to the target user, interaction information of the target user with respect to the historical content recommendation items, or information related to retention durations of the target user in the application.

[0095] Tn some embodiments, the recommendation adjustment module 640 is further configured to: adjust the at least one content recommendation parameter for the target user based on the recommendation change coefficient and the content recommendation treatment.

[0096] In some embodiments, the at least one content recommendation parameter comprises at least one of the following: a time gap between content recommendation items provided to the target user, or an estimated cost per mille (ECPM) threshold.

[0097] FIG. 7 illustrates a block diagram of an electronic device 700 in which one or more embodiments of the present disclosure can be implemented. It would be appreciated that the electronic device 700 shown in FIG. 7 is only an example and should not constitute any restriction on the function and scope of the embodiments described herein. The electronic device 700 shown in FIG. 7 may be used to implement the content delivery system 110 of FIG. 1. The electronic device 700 shown in FIG. 7 may be used to implement the apparatus 600 of FIG. 6.

[0098] As shown in FIG. 7, the electronic device 700 is in the form of a general computing device. The components of the electronic device 700 may include, but are not limited to, one or more processors or processing units 710, a memory 720, a storage devices 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. The processing unit 710 may be an actual or virtual processors and can execute various processes according to the programs stored in the memory 720. In a multiprocessor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 700.

[0099] The electronic device 700 typically includes a variety of computer storage media. Such media may be any available media that is accessible to the electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 720 may be volatile memory (for example, a register, cache, a random access memory (RAM)), a non-volatile memory (for example, a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory), or any combination thereof. The storage device 730 may be any removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which can be used to store information and / or data (such as training data for training) and can be accessed within electronic device 700.

[0100] The electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage medium. Although not shown in FIG. 7, a disk driver for reading from or writing to a removable, non-volatile disk (such as a "floppy disk"), and an optical disk driver for reading from or writing to a removable, non-volatile optical disk can be provided. In these cases, each driver may be connected to the bus (not shown) by one or more data medium interfaces.

[0101] In addition, functions of components in the electronic device 700 may be implemented by a single computing cluster or multiple computing machines, which can communicate through a communication connection. Therefore, the electronic device 700 may be operated in a networking environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0102] The input device 750 may be one or more input devices, such as a mouse, a keyboard, a trackball, etc. The output device 760 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 700 may also communicate with one or more external devices (not shown) through the communication unit 740 as required. The external device, such as a storage device, a display device, etc., communicate with one or more devices that enable users to interact with the electronic device 700, or communicate with any device (for example, a network card, a modem, etc.) that makes the electronic device 700 communicate with one or more other computing devices. Such communication may be executed via an input / output (I / O) interface (not shown).

[0103] According to example implementation of the present disclosure, a computer-readable storage medium is provided, on which a computer-executable instruction or computer program is stored, where the computer-executable instructions or the computer program is executed by the processor to implement the method described above.

[0104] Various aspects of the present disclosure are described herein with reference to the flow chart and / or the block diagram of the method, the device, the equipment and the computer program product implemented in accordance with the present disclosure. It would be appreciated that each block of the flowchart and / or the block diagram and the combination of each block in the flowchart and / or the block diagram may be implemented by computer-readable program instructions.

[0105] These computer-readable program instructions may be provided to the processing units of general-purpose computers, special computers or other programmable data processing devices to produce a machine that generates a device to implement the functions / acts specified in one or more blocks in the flow chart and / or the block diagram when these instructions are executed through the processing units of the computer or other programmable data processing devices. These computer-readable program instructions may also be stored in a computer-readable storage medium. These instructions enable a computer, a programmable data processing device and / or other devices to work in a specific way. Therefore, the computer-readable medium containing the instructions includes a product, which includes instructions to implement various aspects of the functions / acts specified in one or more blocks in the flowchart and / or the block diagram.

[0106] The computer- readable program instructions may be loaded onto a computer, other programmable data processing apparatus, or other devices, so that a scries of operational steps can be performed on a computer, other programmable data processing apparatus, or other devices, to generate a computer-implemented process, such that the instructions which execute on a computer, other programmable data processing apparatus, or other devices implement the functions / acts specified in one or more blocks in the flowchart and / or the block diagram.

[0107] The flowchart and the block diagram in the drawings show the possible architecture, functions and operations of the system, the method and the computer program product implemented in accordance with the present disclosure. In this regard, each block in the flowchart or the block diagram may represent a part of a module, a program segment or instructions, which contains one or more executable instructions for implementing the specified logic function. Tn some alternative implementations, the functions marked in the block may also occur in a different order from those marked in the drawings. For example, two consecutive blocks may actually be executed in parallel, and sometimes can also be executed in a reverse order, depending on the function involved. It should also be noted that each block in the block diagram and / or the flowchart, and combinations of blocks in the block diagram and / or the flowchart, may be implemented by a dedicated hardware-based system that performs the specified functions or acts, or by the combination of dedicated hardware and computer instructions.

[0108] Each implementation of the present disclosure has been described above. The above description is example, not exhaustive, and is not limited to the disclosed implementations. Without departing from the scope and spirit of the described implementations, many modifications and changes are obvious to ordinary skill in the art. The selection of terms used in this article aims to best explain the principles, practical application or improvement of technology in the market of each implementation, or to enable other ordinary skill in the art to understand the various embodiments disclosed herein.

Claims

I / we claim:

1. A method for content recommendation, comprising: determining, based on feature information of a target user, a predicted duration difference between a first retention duration of the target user in an application before applying a content recommendation treatment to the target user and a second retention duration of the target user in the application after applying the content recommendation treatment to the target user: determining, based on the feature information, a predicted cost difference between a first cost introduced by the target user before applying the first content recommendation treatment to the target user and a second cost introduced by the target user after applying the content recommendation treatment to the target user; determining a recommendation change coefficient for the target user based on the predicted duration difference and the predicted cost difference; adjusting at least one content recommendation parameter for the target user based at least in part on the recommendation change coefficient; and providing at least one recommended content item to the target user based at least in part on the at least one adjusted content recommendation parameter.

2. The method of claim 1, wherein adjusting at least one content recommendation parameter for the target user comprises: in response to the recommendation change coefficient being a first value, determining at least one first parameter value for the at least one content recommendation parameter to cause a first amount of recommended content items to be provided to the target user; and in response to the recommendation change coefficient being a second value higher than the first value, determining at least one second parameter value for the at least one content recommendation parameter to cause a second amount of recommended content items to be provided to the target user, the second amount being lower than the first amount.

3. The method of claim 1, wherein adjusting at least one content recommendation parameter for the target user comprises: allocating the target user to a user group based on the recommendation change coefficient; and determining at least one content recommendation parameter for the user group based on recommendation change coefficients of users in the user group.

4. The method of claim 1, wherein determining the predicted duration difference comprises: determining the predicted duration difference using a trained first machine learning model based on the feature information; andwherein determining the predicted cost difference comprises: determining the predicted cost difference using a trained second machine learning model based on the feature information.

5. The method of claim 4, wherein the first machine learning model is trained with a first training dataset comprising a plurality of first training samples, a first training sample comprising: sample feature information of a first sample user and a second sample user, and a labeled duration difference between a first sample retention duration of the first sample user in the application without the content recommendation treatment applied to the first sample user and a second sample retention duration of the second sample user in the application with the content recommendation treatment applied to the second sample user, wherein the first sample retention duration and the second sample retention duration are collected for the first sample user and the second sample user after a first period of time when tire contention recommendation treatment is applied to the second sample user.

6. The method of claim 5, wherein the second machine learning model is trained with a second training dataset comprising a plurality of second training samples, a second training sample comprising: the sample feature information of the first sample user and the second sample user, and a labeled cost difference between a first sample cost introduced by the first sample user without the content recommendation treatment applied to the first sample user and a second sample cost introduced by the second sample user with tire content recommendation treatment applied to the second sample user, wherein the first sample cost and the second sample cost are collected for the first sample user and the second sample user after a second period of time when the contention recommendation treatment is applied to the second sample user.

7. The method of claim 4, wherein the first machine learning model comprises a first duration prediction model configured to determine a retention duration of a user in an application without the content recommendation treatment applied, and a second duration prediction model configured to determine a retention duration of a user in an application with the content recommendation treatment applied; and wherein a training target of the first machine loaming model is configured to minimize or reduce an error between the labeled duration difference and an output duration difference between a predicted retention duration output by the first duration prediction model and a predicted retention duration output by a second duration prediction model.

8. The method of claim 5, wherein the second machine learning model comprises a first cost prediction model configured to determine a cost introduced by a user without the content recommendation treatment applied, and a second cost prediction model configured to determine a cost introduced by a user with the content recommendation treatment applied; and wherein a training target of the second machine learning model is configured to minimize or reduce an error between the labeled cost difference and an output duration difference between a predicted cost provided by the first cost prediction model and a predicted cost provided by a second cost prediction model.

9. The method of claim 1, wherein the feature information of the target user comprises at least one of the following: user demographical information of the target user, provision information related to historical content recommendation items provided to the target user, interaction information of the target user with respect to the historical content recommendation items, or information related to retention durations of the target user in the application.

10. The method of claim 1, wherein adjusting at least one content recommendation parameter for the target user based at least in part on the recommendation change coefficient comprises: adjusting the at least one content recommendation parameter for the target user based on the recommendation change coefficient and the content recommendation treatment.

11. The method of claim 1, wherein the at least one content recommendation parameter comprises at least one of the following: a time gap between content recommendation items provided to the target user, or an estimated cost per mille (ECPM) threshold.

12. An apparatus for content recommendation, comprising: a duration difference determining module configured to determine, based on feature information of a target user, a predicted duration difference between a first retention duration of the target user in an application before applying a content recommendation treatment to the target user and a second retention duration of the target user in the application after applying the content recommendation treatment to the target user; a cost difference determining module configured to determine, based on the feature information, a predicted cost difference between a first cost introduced by the target user before applying the first content recommendation treatment to the target user and a second cost introduced by the target user after applying the content recommendation treatment to the target user;a coefficient determining module configured to determine a recommendation change coefficient for the target user based on the predicted duration difference and the predicted cost difference; a recommendation adjustment module configured to adjust at least one content recommendation parameter for the target user based at least in part on the recommendation change coefficient; and a content providing module configured to provide at least one recommended content item to the target user based at least in part on the at least one adjusted content recommendation parameter.

13. An electronic device, comprising: at least one processing unit; and at least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the device to perform the method of any of claims 1 to 11.

14. A computer readable storage medium having a computer program stored thereon which, when executed by an electronic device, causes the electronic device to perform the method of any of claims 1 to 11.

15. A computer program product storing computer readable instructions which, when executed by a processor, implements the method of any of claims 1 to 11.

Images