Systems for efficient use of data for personalization

Abstract

An off-policy evaluation of a new "target" policy is performed using historical data collected based on a previous "logging" policy to estimate the performance of the target policy. An estimator can be used where a quality-based estimator or a quality-agnostic estimator is used to weight the difference between the observed rewards in the historical data and the estimated rewards generated by the target policy. The quality-agnostic estimator can be used to evaluate the importance weights based on a threshold. In such an example, the quality-agnostic estimator clips the importance weights at the threshold when they exceed the threshold, thereby providing a fixed upper bound that is independent of the quality of the reward predictor. In other examples, a quality-based estimator is used where the upper bound incorporates the quality of the reward predictor in order to modify the importance weights used by the estimator.

Description

Background Art

[0001] In applications using contextual bandit protocols, logging policies are used to take actions based on a given context to obtain rewards. In some instances, logging policies are evaluated based on an average reward metric. However, without expensive A / B testing and large datasets, generating updated or new policies ("target policies") that achieve similar or improved average reward metrics is difficult.

[0002] It is with respect to these and other general considerations that the aspects disclosed herein are made.In addition, although relatively specific problems may be discussed, it should be understood that the examples should not be limited to solving specific problems identified in the background or elsewhere in this disclosure. Summary of the Invention

[0003] This disclosure describes systems and methods for evaluating policies and generating policies with improved performance. In some examples, off-policy evaluation is performed using historical data collected based on a previous algorithm (e.g., a "logging policy") to estimate the performance of an updated algorithm (e.g., a "target policy"). An estimator can be used in which importance weights are used to weight the difference between observed rewards in the historical data and estimated rewards generated by the target policy (e.g., which can be estimated by a reward predictor). In an example, the method involves reducing the importance weights to improve the bound on the mean squared error (MSE). In some examples, a quality-agnostic estimator is used to evaluate the importance weights based on a threshold. In such examples, when the importance weight exceeds the threshold, the threshold is used as the importance weight, thereby providing an upper limit that is independent of the quality of the reward predictor. In other examples, a quality-based estimator is used in which the upper limit incorporates the quality of the reward predictor to modify the importance weights used by the estimator.

[0004] It should be understood that in another example, the off-policy estimate can be used to generate a new target policy. In the example, an estimator that improves the bound on the MSE can be selected and used to generate the target policy accordingly. The aspects described herein for evaluating policies and for generating optimized policies are applicable to different applications and contexts. Given the same amount of historical data from a previous logging policy, this approach can result in more accurate evaluation and optimization. Therefore, less data may be required to perform an adequate evaluation and optimization of the policy, thereby reducing the difficulties associated with collecting data and potential problems caused by outdated or stale data.

[0005] This summary is provided to introduce in simplified form a series of concepts that are further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Additional aspects, features, and / or advantages of the examples will be set forth in part in the description that follows and in part will become apparent from the description, or may be learned by practice of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Non-limiting and non-exhaustive examples are described with reference to the following figures.

[0007] Figure 1 Illustrated is an overview of an example system for improved personalization techniques in accordance with various aspects described herein.

[0008] Figure 2A Illustrated is an overview of an example method for evaluating a target policy based on collected historical data generated according to a logging policy.

[0009] Figure 2B Illustrated is an overview of an example method for determining hyperparameters for off-policy evaluation.

[0010] Figure 2C Illustrated is an overview of an example method for selecting an estimator class for off-policy evaluation.

[0011] Figure 3 is a block diagram illustrating example physical components of a computing device with which aspects of the present disclosure may be practiced.

[0012] Figure 4A and Figure 4B is a simplified block diagram of a mobile computing device with which aspects of the present disclosure may be practiced.

[0013] Figure 5 is a simplified block diagram of a distributed computing system in which aspects of the present disclosure may be practiced.

[0014] Figure 6 Illustrated is a tablet computing device for performing one or more aspects of the present disclosure. DETAILED DESCRIPTION

[0015] Various aspects of the present disclosure are described more fully below with reference to the accompanying drawings, which form a part of this disclosure and illustrate specific exemplary aspects. However, the different aspects of the present disclosure can be implemented in many different forms and should not be construed as limited to the aspects set forth herein; rather, these aspects are provided so that this disclosure will be thorough and complete and will fully convey the scope of these aspects to those skilled in the art. Aspects can be practiced as methods, systems, or devices. Therefore, the following detailed description should not be taken in a limiting sense.

[0016] In the example of the contextual bandit protocol, the strategy evaluates the context, determines an action based at least in part on the context, and accumulates rewards. As used herein, a strategy (or "decision maker") can be a pre-existing "logging" strategy or a new "target" strategy. In the example, a strategy is generated whose goal is to increase or maximize the probability of accumulating rewards for a given context (e.g., based on the determined action). As an example, determining an action can include selecting a content item from a set of content items for presentation to the user. Thus, if the user engages with the content item, a reward is obtained. However, if the user does not engage with the content item, no reward is obtained. Thus, if the user engages with the content item, the reward can be represented as "1," and if the user does not engage, the reward can be represented as "0." Therefore, in such an example, a strategy that increases or maximizes the likelihood of accumulating rewards involves presenting content that may result in user engagement.

[0017] In the example, the context space from which the context is observed can be uncountably large. In other examples, it can be assumed that the action space from which an action is determined is finite. As used herein, context can relate to user activities on a user device, such as clicking a link, opening an email, sending an email, opening an application, interaction within an application, the location or time of using an application, and other user activities performed on the user device. Therefore, example actions include, but are not limited to, presenting content items (e.g., links to websites, text content, graphic content, video content, audio content, target content, etc.), applications on the user device or actions within an application, or contacts and other actions (e.g., ranking a group of content items). If the user interaction is generated by a determined action (e.g., a user clicks a link, calls a contact, etc.), a reward is triggered. As another example, if the user interacts with the presented content, an initial reward can be triggered, and if the user initiates and / or completes a purchase associated with the presented content, a subsequent reward can be triggered. It will be understood that although examples are generally described herein with a single action, similar techniques can be used to determine a group of actions (e.g., presenting multiple content items, multiple applications or actions within an application, multiple contacts, etc.). In such an example, if any one of the actions in the set of actions results in a user interaction, then the associated reward is triggered.

[0018] Off-policy evaluation can be used to evaluate the target policy. In an example, the goal of the off-policy evaluation is to use historical data collected by past policies (e.g., logging policies) to estimate the performance of the new target policy. As an example, historical data can include, but is not limited to, a set of triples, each of which includes a context, an associated action, and a resulting reward. It will be understood that historical information can include additional, less, or alternative information about user interactions associated with the logging policy. Returning to the above example, an off-policy evaluation of the target policy can be performed based on historical data about user interactions with content identified according to the logging policy. Therefore, the off-policy evaluation of the target policy includes: based on the historical interactions caused by the logging policy, determining the predicted likelihood that the target policy will identify the content that the user will participate in. Therefore, if the target policy appears to have a higher likelihood of user participation (and therefore a higher predicted reward) than the logging policy, the target policy can be implemented to replace the logging policy.

[0019] High-quality off-policy estimates can avoid expensive A / B testing and can also be used to generate improved or optimized policies. As used herein, an "optimized" policy refers to a policy that exhibits one or more improved characteristics compared to a previous policy. For example, a target policy can exhibit a reduced mean squared error or an increased average reward compared to a logging policy, among other examples. It will be understood that while the examples herein are described with respect to a user interacting with selected content, any of a variety of contexts, actions, and rewards can be used in accordance with the present disclosure.

[0020] A challenge in off-policy evaluation is distribution mismatch, where the action determined by the target policy for a given context may be different from the historical action determined by the logging policy that was in use when the historical data was collected. "Doubly robust estimation" is an example estimator that can be used to address such a challenge. In an example, doubly robust estimation uses a combination of inverse propensity scoring and direct modeling, where inverse propensity scoring can be used to correct for the distribution mismatch by reweighting the data, while direct modeling can be used to reduce the impact of larger importance weights. As an example, direct modeling includes generating and using a regression model to predict rewards. In other examples, a reward predictor can be trained and used to generate predicted rewards for a given context.

[0021] Doubly robust estimation can produce results that are less biased or unbiased and have smaller variance than the results achieved using inverse propensity scores. In addition, doubly robust estimation can be asymptotically optimal under weaker assumptions than direct modeling. However, since doubly robust estimation may use the same importance weights as inverse propensity scores, its variance is still high unless the reward predictor is very accurate. Therefore, in some examples, according to the aspects described herein, doubly robust estimation can be further improved by clipping or removing larger importance weights, such as by weighting the reward predictions used by the doubly robust estimation model using a quality-agnostic estimator or a quality-based estimator. Although weight clipping or shrinking may induce smaller biases, it can also significantly reduce variance, which can result in a lower mean squared error (MSE) than achieving doubly robust estimation without such techniques. The present disclosure presents systems and methods for improving off-policy evaluation by weight clipping.

[0022] The aspects described herein allow for better evaluation and optimization of policies. For example, evaluating a target policy may require less historical data associated with a logging policy, thereby reducing the computational overhead associated with acquiring and processing historical data. Furthermore, because less historical data is effectively used for off-policy evaluation, the complexity of obtaining relevant and current historical data is reduced, minimizing the impact of potentially stale data on evaluation. Furthermore, as noted above, expensive A / B testing can be avoided, among other benefits.

[0023] The off-policy evaluation techniques described herein can also be performed with associated hyperparameters. In an example, the method presented herein involves shrinking importance weights to optimize a bound on the MSE of an estimator. Two classes of estimators are described herein. The first class of estimators is independent of the quality of the reward predictor and is referred to herein as "quality agnostic" estimators or doubly robust estimators with "pessimism shrinkage". Quality agnostic estimators use a threshold to evaluate importance weights. In an example, when an importance weight exceeds a threshold, the threshold is used in place of the importance weight, thereby clipping the importance weights above the threshold. However, if the importance weight does not exceed the threshold, the importance weight is not changed. As an example, a quality agnostic estimator This can be modeled by the following equation:

[0024]

[0025] In the equation above, p indicates that the estimator is quality agnostic (e.g., it does not take into account the quality of the reward predictor), λ represents the value of a hyperparameter (e.g., the threshold at which weights are clipped), x represents the context of the contextualbandit policy, a represents the action associated with the context x (e.g., it could have been generated by a logging policy), w(x,a) represents the importance weight, and Denotes the new importance weight according to the quality agnostic estimator. It will be appreciated that the above equation is provided as an example equation with which to implement doubly robust estimation with pessimism shrinkage.

[0026] In other aspects, a second class of estimators is used, which are referred to herein as quality-based estimators or doubly robust estimators with "optimistic shrinkage." Unlike quality-agnostic estimators, quality-based estimators use an upper bound based on the quality of the reward predictor. For example, the importance weights generated according to the quality-based estimator can be modified or generated in a manner that combines the original importance weights. In some examples, the methods herein can bound the bias and variance based on the weighted squared loss. As an example, the quality-based estimator This can be modeled by the following equation:

[0027]

[0028] In the equation above, o indicates that the estimator is quality-based (e.g., it depends on the quality of the reward predictor), λ represents the value of the hyperparameter, x represents the context of the contextualbandit policy, a represents the action associated with the context x (e.g., it may have been generated by a logging policy), w(x,a) represents the importance weight, and Denotes the new importance weight according to the quality-based estimator. It will be appreciated that the above equation is provided as an example equation with which to implement doubly robust estimation with optimistic shrinkage.

[0029] As discussed above, each category of estimators can include hyperparameters to reduce or, in some examples, eliminate bias and variance from the estimator. In some examples, a model selection process is used to adjust hyperparameters (e.g., λ in the example equations above) and / or determine the category of estimator to use (e.g., quality-agnostic estimators, quality-based estimators, etc.). As an example, the model selection process includes selecting values ​​for the hyperparameters and determining which estimator to use so that the resulting off-policy evaluation model produces an MSE that is below a target threshold or, in some examples, within a predetermined range. Therefore, the techniques described herein achieve similar or improved results compared to previous solutions. In addition, this off-policy evaluation technique can exhibit improved finite sample performance and can therefore achieve results comparable to other techniques using relatively less historical data.

[0030] Similar techniques can be used when a policy is used to determine a list of actions rather than a single action. As an example, the overall reward of a list (which may vary depending on the action chosen and its position) can be estimated by decomposing the list into the individual contributions of each action. The contributions of all possible actions can be estimated by applying a weight matrix to the vector representation of the action list selected by the logging policy. The resulting contribution estimates can then be combined to evaluate the reward of the action list selected by the target policy. An example equation for use in such a scenario is provided below:

[0031]

[0032] In the above equation, x is the context and S is the action list (s1, ..., s l ), represents the reward predictor, represents the action s in slot j from the action list S j The contribution of The sum of each action in the list relative to the The actions that have been captured make an overall contribution.It will be understood that the above equations are provided as examples, and in other examples, different techniques may be used to apply the aspects described herein to evaluate action lists.

[0033] In addition to evaluating the target strategy, the aspects described herein can be used to generate improved or optimized strategies. In an example, an improved strategy is generated based on the analysis of historical data collected according to a logging strategy. As an example, an adaptive decision algorithm is used to determine an action for a given context. The adaptive decision algorithm is adjusted according to the context and associated actions from the historical data. In other examples, the decision algorithm can make randomized decisions at least in part when determining an action, thereby increasing the possibility of exploring different possible actions. In another example, an exploration budget is used, wherein the performance of the adaptive decision algorithm is evaluated (for example, according to one or more estimator categories and associated technologies described herein) compared to a default strategy (for example, a logging strategy, according to a fixed decision algorithm based on historical data, etc.). In some examples, if the performance of the adaptive decision algorithm exceeds the exploration budget, the default strategy can be used instead to determine the action until the performance returns to within the exploration budget, thereby limiting the potential impact of performance degradation.

[0034] Figure 1 An overview of an example system 100 for improved personalization techniques according to aspects described herein is illustrated. As illustrated, system 100 includes a server device 102 and a user device 104. In an example, server device 102 and user device 104 communicate using a network (such as a local area network, a wireless network, or the Internet, or any combination thereof). In an example, user device 104 is any of a variety of computing devices, including but not limited to a mobile computing device, a laptop computing device, a tablet computing device, or a desktop computing device. In other examples, server device 102 is a computing device, including but not limited to a desktop computing device or a distributed computing device. It will be understood that although system 100 is illustrated as including one server device 102 and one user device 104, in other examples, any number of devices may be used.

[0035] The user device 104 is illustrated as including a client application 106 and a context data store 108. The client application 106 can be a web browser or a messaging application, among other examples. In the example, the client application 106 communicates with the server 102 to access content for display to the user of the user device 104. As the user engages with the client application 106, the user interactions can be stored in the context data store 108. Thus, the context data store 108 can store any of a variety of context information, including, but not limited to, links visited, emails opened, emails sent, applications opened, interactions within applications, the location or time of application use, and other activities on the user device 104.

[0036] Server device 102 is illustrated as including action generation engine 110, policy evaluation engine 112 and historical data repository 114. In an example, action generation engine 110 uses a policy (e.g., a logging policy) to generate an action according to a given context. For example, user device 104 can provide context information from context data repository 108, which is used by action generation engine 110 to determine an action. An indication of an action can be provided to user device 104. User device 104 can then generate a display (e.g., presenting a content item, displaying a ranking list of content items, suggesting content or applications, etc.) according to the action. Therefore, depending on the result of the selected action, action generation engine 110 can receive an indication of an associated reward. In another example, instead of or in addition to the context received from user device 104, action generation engine 110 can use a context determined at server device 102 (e.g., associated with a user account, a specific cookie, a session, etc.). In an example, action generation engine 110 logs historical data and stores such information in historical data repository 114. As described above, historical data may be stored in the form of triples, including the context, the determined action, and the associated reward. It will be appreciated that in other examples, additional, less, or alternative information may be stored as historical data in the historical data repository 114.

[0037] The server device 102 also includes a policy evaluation engine 112. The policy evaluation engine 112 implements the aspects described herein to perform off-policy evaluation of a new policy (e.g., a target policy) based on historical data generated by an existing logging policy. In an example, the policy evaluation engine 112 accesses historical data from a historical data repository 114 (e.g., which may have been generated by the action generation engine 110). In an example, the policy evaluation engine 112 performs a model selection process to determine whether to apply an off-policy evaluation model with a quality-agnostic estimator or a quality-based estimator when analyzing the target policy in light of the historical data. In addition, the policy evaluation engine 112 can determine the values ​​of optional hyperparameters to further adjust the model used to evaluate the target policy. Ultimately, the target policy is evaluated based on the determined off-policy evaluation model to compare the performance of the target policy with the logging policy currently used by the action generation engine 110. In an example, the comparison includes evaluating an average reward metric, wherein the average reward induced by each policy (e.g., for a set of contexts) is compared to determine which policy induced the highest average reward. If the target policy exhibits a higher average reward metric, the action generation engine 110 may use the target policy instead of the logging policy when generating subsequent actions based on the given context. It will be understood that the target policy may be compared to the logging policy using any of a variety of other metrics, including but not limited to average variance or total reward value. Figure 2A-2C Additional example aspects of the policy evaluation engine 112 are discussed in greater detail.

[0038] While example implementations are described above with respect to a server computing device and / or a user device, it will be appreciated that the aspects described herein may be implemented using any of a variety of other devices. Similarly, while certain operations are described with respect to server device 102 or user device 104, it will be appreciated that the aspects described herein may be split among and performed by any of a variety of computing device configurations. For example, aspects of action generation engine 110 may be performed by user device 104, or, in another example, at least a sub-portion of context data repository 108 may reside on server device 102.

[0039] Figure 2A An overview of an example method 200 for evaluating a target policy based on collected historical data generated according to a logging policy is illustrated. The method 200 may be performed by one or more computing devices, including but not limited to a personal computer, a laptop computer, a tablet computer, a mobile computing device, or a distributed computing device. As an example, aspects of the method 200 may be performed by Figure 1 As another example, aspects of method 200 may be performed by server device 102 and / or user device 104 in Figure 1 The method 200 begins at operation 202, where historical data associated with a logging policy may be accessed. For example, according to aspects disclosed herein, the data may include information related to a contextual bandit protocol, including context, actions, rewards, and / or performance of the protocol. It will be understood that the data may be accessed from any of a variety of sources, including, but not limited to, local data repositories (e.g., Figure 1 ) or a remote data repository (e.g., the historical data repository 114), or any combination thereof.

[0040] At operation 204, a reward predictor is generated. As described above, the reward predictor can generate an expected reward. For example, the reward predictor uses historical data associated with the logging policy, as accessed at operation 202, to generate the expected reward. Thus, in the example, given a context and an associated action determined by the logging policy based on the context, the reward predictor generates a predicted reward. In some examples, the reward predictor can be modeled as a regression function (e.g., as may be the case when using a direct modeling approach).

[0041] It will be appreciated that in other examples, multiple reward predictors may be generated at operation 204. For example, each reward predictor may use a different kind of regression function. As an example, a first reward predictor may be generated using linear regression, while a second reward predictor may be generated based on a deep neural network. Other examples include using different weighting techniques when generating multiple reward predictors. Assuming a data weighting function z(x, a), where x is the context and a is the action, example weighting techniques include, but are not limited to, a uniform weighting function (e.g., z(x, a) = 1), a weighting function based on importance weights (e.g., assuming the weights are defined by a function w(), z(x, a) = w(x, a)), or a weighting function based on the square of the importance weights (e.g., z(x, a) = w 2 (x, a)). Another example weighting function that can be used for policy optimization is:

[0042]

[0043] In the above example equation, mu() is a function that describes the probability of choosing action a given context x when applying a given logging policy. It will be understood that the above equation is provided as an example, and any of a variety of other functions may be used in accordance with the aspects described herein.

[0044] At operation 206, the generated reward predictor(s) are used to perform a model selection process. As described herein, a set of one or more estimators, such as quality agnostic estimators and / or quality based estimators, may be used. In some examples, a plurality of quality agnostic estimators are evaluated at operation 206, wherein each quality agnostic estimator uses different hyperparameters. Similarly, a plurality of quality based estimators may be evaluated at operation 206, wherein each quality based estimator uses different hyperparameters. In fact, as discussed above, an estimator may include one or more hyperparameters that are used to tailor or shrink the importance weights of the reward predictor according to the aspects described herein. The model selection process also includes evaluating each estimator to determine which estimator produces an off-policy evaluation model with the smallest error. As an example, the set of estimators may be compared based on the least squared error associated with each estimator. It will be understood that other techniques may be used to compare each model and ultimately select a model with which to evaluate the target policy. As another example, models may be evaluated based on which models exhibit errors below a certain threshold or within a certain range. Figure 2B and Figure 2C Additional example aspects of model selection are described in more detail. In an example, model selection can identify a strategy for minimizing MSE, bias, and variance. Model selection can also identify which estimator class to use.

[0045] Flow proceeds to operation 208, where the target policy is evaluated according to the model selected at operation 206. In an example, the evaluation includes generating an average reward metric for the target policy using the selected model for a set of contexts from the historical data accessed at operation 202. It will be appreciated that any of a variety of other metrics may be used to evaluate the target policy, including, but not limited to, average variance or total reward value.

[0046] At determination 210, a determination is made as to whether the target policy is expected to perform better than the logging policy used to generate the historical data. In an example, the determination includes comparing the average reward metric for the target policy (e.g., generated at operation 208) to the average reward elicited by the logging policy. If the target policy is better than the logging policy (e.g., it exhibits a higher average reward metric than the logging policy), then the process branches "yes" to operation 212, where the target policy is used instead of the logging policy. For example, Figure 1 The action generation engine 110 in FIG. 1 may receive an indication to use the target policy instead of the logging policy. The process terminates at operation 212. However, if it is determined that the target policy is not better than the logging policy, the process branches "No" to operation 214, where the logging policy is continued to be used instead of the target policy. The process terminates at operation 214.

[0047] Figure 2B An overview of an example method 220 for determining hyperparameters for off-policy evaluation is shown. The method 220 may be performed by one or more computing devices, including but not limited to a personal computer, a laptop computer, a tablet computer, a mobile computing device, or a distributed computing device. As an example, aspects of the method 220 may be performed by Figure 1 As another example, aspects of method 220 may be performed by server device 102 and / or user device 104 in Figure 1 The policy evaluation engine 112 in the method 220 is executed. Aspects of the method 220 can be performed in Figure 2A Operation 206 in is performed as part of the model selection process.

[0048] Method 220 begins at operation 222, where hyperparameters are selected. In one example, the hyperparameters are selected based on one or more previous off-policy evaluations. In another example, the hyperparameters may be selected based on hyperparameters used to evaluate a logging policy for collecting historical data. In other examples, the hyperparameters may be selected iteratively, where the hyperparameters are iteratively increased or decreased based on reaching an error below a threshold or within a certain range. It will be appreciated that a variety of other techniques may be used to select hyperparameters.

[0049] At operation 224, the historical data is evaluated according to the model based on the selected hyperparameters. In an example, the evaluation includes using hyperparameters for one or more estimator classes, such as quality-based estimators and / or quality-agnostic estimators. The historical data can be accessed from a historical data repository, such as Figure 1 The historical data repository 114 in . The evaluation may include the MSE generated for the model according to the hyperparameters. It will be appreciated that in other examples, different metrics may be used to evaluate the model.

[0050] The process proceeds to determination 226, where it is determined whether the MSE is below a certain threshold. The threshold can be a threshold pre-configured by the user, or can be determined programmatically (e.g., based on evaluating the data according to a doubly robust estimator without applying a quality-based estimator or a quality-agnostic estimator). It will be appreciated that in other examples, a range of values ​​can be used to determine whether the MSE is acceptable. If it is determined that the MSE is not below the threshold, the process branches to "no" and returns to operation 222. As noted above, the hyperparameter selection process can be iterative, thereby determining the updated hyperparameters at operation 222. Therefore, the process loops between operations 222, 224, and 226 until the MSE is below the threshold. In some examples, determination 226 also includes a counter so that after a certain number of iterations, the process branches to "yes" to operation 228 instead. Ultimately, the process arrives at operation 228, which will be discussed below.

[0051] However, if it is determined that the MSE is below the threshold, then flow instead branches "yes" to operation 228, where the hyperparameters are used for off-policy evaluation according to aspects described herein. For example, performing off-policy evaluation based on the determined hyperparameters may include: performing Figure 2A The steps of method 200 in are as described above. The process terminates at operation 228.

[0052] Figure 2C An overview of an example method 240 for selecting an estimator class for off-policy evaluation is shown. The method 240 may be performed by one or more computing devices, including but not limited to a personal computer, a laptop computer, a tablet computer, a mobile computing device, or a distributed computing device. As an example, aspects of the method 240 may be performed by Figure 1 As another example, aspects of method 240 may be performed by server device 102 and / or user device 104 in Figure 1 Aspects of the method 240 may be performed by the policy evaluation engine 112 in Figure 2A Operation 206 in is performed as part of the model selection process.

[0053] Method 240 begins at operation 242, where historical data is evaluated according to a quality-agnostic estimator. In an example, the quality-agnostic estimator tailors weights assigned to reward predictions according to aspects described herein. Operation 242 may include generating an MSE associated with the evaluation of the historical data using the quality-agnostic estimator.

[0054] Flow proceeds to operation 244, where the historical data is evaluated based on a quality-based estimator. As discussed above, the quality-based estimator weights the reward predictions based on the quality of the reward predictor. Similar to operation 242, operation 244 may also include generating an MSE for evaluating the historical data using the quality-based estimator. It will be appreciated that in other examples, different metrics may be used, such that operations 242 and 244 generate different metrics for comparison.

[0055] Moving to determination 246, it is determined whether the quality-based estimator produces better results than the quality-agnostic estimator. Method 240 is an example where the MSE is used to evaluate two estimators. Thus, the determination includes evaluating the MSE for each estimator to determine which estimator exhibits a lower MSE. If it is determined that the quality-based estimator produces a lower MSE, then flow branches to operation 248 where the quality-based estimator is used to perform off-policy evaluation of the target policy (e.g., as described above with respect to Figure 2AHowever, if it is determined that the quality-agnostic estimator exhibits a lower MSE than the quality-based estimator, then the process branches to operation 250, where the quality-agnostic estimator is used to perform off-policy evaluation of the target policy. The process terminates at operation 248 or 250.

[0056] Figure 3-Figure 6 and the associated descriptions provide a discussion of various operating environments in which aspects of the present disclosure may be practiced. Figure 3-Figure 6 The devices and systems illustrated and discussed are for purposes of example and explanation and are not limiting of the wide variety of computing device configurations that can be utilized to practice aspects of the disclosure described herein.

[0057] Figure 3 is a block diagram illustrating the physical components (e.g., hardware) of a computing device 300 with which aspects of the present disclosure may be practiced. The computing device components described below may be suitable computing devices for implementing the aspects of the present disclosure described above. In a basic configuration, the computing device 300 may include at least one processing unit 302 and system memory 304. Depending on the configuration and type of the computing device, the system memory 304 may include, but is not limited to, volatile memory (e.g., random access memory), non-volatile memory (e.g., read-only memory), flash memory, or any combination of such memories.

[0058] System memory 304 may include an operating system 305 and one or more program modules 306 adapted to run software applications 320, such as one or more components supported by the system described herein. As an example, system memory 304 may store a client application 324 and a policy generator 326. For example, client application 324 may display content determined based on actions in a logging policy. A user may interact with such content, thereby triggering a reward. Such interactions may form part of historical user interactions. Policy generator 326 may implement aspects of method 200 to optimize a logging model and / or generate a new target model in accordance with aspects described herein. For example, operating system 305 may be adapted to control the operation of computing device 300.

[0059] Furthermore, embodiments of the present disclosure may be practiced in conjunction with graphics libraries, other operating systems, or any other application programs and are not limited to any particular application or system. Figure 3 308. The computing device 300 may have additional features or functionality. For example, the computing device 300 may also include additional data storage devices (removable and / or non-removable) such as, for example, magnetic disks, optical disks, or tapes. Such additional storage devices may be used to store data. Figure 3309 and a non-removable storage device 310.

[0060] As described above, a number of program modules and data files can be stored in system memory 304. When executed on processing unit 302, program modules 306 (e.g., applications 320) can perform processes including, but not limited to, aspects as described herein. Other program modules that can be used in accordance with aspects of the present disclosure may include email and contact applications, word processing applications, spreadsheet applications, database applications, slide presentation applications, drawing or computer-assisted applications, and the like.

[0061] Furthermore, embodiments of the present disclosure may be practiced in circuits comprising discrete electronic components, packaged or integrated electronic chips containing logic gates, circuits utilizing a microprocessor, or on a single chip containing electronic components or a microprocessor. For example, embodiments of the present disclosure may be practiced via a system on a chip (SOC) wherein Figure 3 Each or many of the components illustrated in the figure can be integrated onto a single integrated circuit. Such a SOC device may include one or more processing units, a graphics unit, a communication unit, a system virtualization unit, and various application functionalities, all of which are integrated (or "burned") onto the chip substrate as a single integrated circuit. When operated via the SOC, the functionality described herein regarding the ability to switch client protocols can be operated via dedicated logic integrated with other components of the computing device 300 on a single integrated circuit (chip). Embodiments of the present disclosure may also be practiced using other technologies capable of performing logical operations such as, for example, AND, OR, and NOT, including but not limited to mechanical, optical, fluidic, and quantum technologies. In addition, embodiments of the present disclosure may be practiced within a general-purpose computer or in any other circuit or system.

[0062] The computing device 300 may also have one or more input devices 312, such as a keyboard, a mouse, a pen, an audio or voice input device, a touch or slide input device, and the like. Output device(s) 314, such as a display, speakers, a printer, and the like, may also be included. The aforementioned devices are examples, and other devices may be used. The computing device 300 may include one or more communication connections 316 that allow for communication with other computing devices 350. Examples of suitable communication connections 316 include, but are not limited to, radio frequency (RF) transmitters, receivers, and / or transceiver circuits; universal serial bus (USB), parallel, and / or serial ports.

[0063] As used herein, the term "computer-readable media" may include computer storage media. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, or program modules. System memory 304, removable storage device 309, and non-removable storage device 310 are all examples of computer storage media (e.g., memory storage). Computer storage media may include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic tape, magnetic cassette, disk storage or other magnetic storage device, or any other article that can be used to store information and is accessible by computing device 300. Any such computer storage media may be part of computing device 300. Computer storage media does not include carrier waves or other propagated or modulated data signals.

[0064] Communication media may be embodied by computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" may describe a signal that has one or more characteristics that are set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0065] Figure 4A and Figure 4B A mobile computing device 400 is illustrated, such as a mobile phone, a smart phone, a wearable computer (such as a smart watch), a tablet computer, a laptop computer, etc., which can be used to practice embodiments of the present disclosure. In some aspects, the client can be a mobile computing device. Figure 4A , illustrates one aspect of a mobile computing device 400 for implementing these aspects. In a basic configuration, the mobile computing device 400 is a handheld computer having both input and output elements. The mobile computing device 400 typically includes a display 405 and one or more input buttons 410 that allow a user to enter information into the mobile computing device 400. The display 405 of the mobile computing device 400 can also be used as an input device (e.g., a touch screen display).

[0066] If included, optional side input element 415 allows further user input. Side input element 415 can be a rotary switch, a button, or any other type of manual input element. In alternative aspects, mobile computing device 400 can incorporate more or fewer input elements. For example, in some embodiments, display 405 can be other than a touch screen.

[0067] In yet another alternative embodiment, the mobile computing device 400 is a portable telephone system, such as a cellular phone. The mobile computing device 400 may also include an optional keypad 435. The optional keypad 435 may be a physical keypad or a "soft" keypad generated on a touch screen display.

[0068] In various embodiments, the output elements include a display 405 for displaying a graphical user interface (GUI), a visual indicator 420 (e.g., a light emitting diode), and / or an audio transducer 425 (e.g., a speaker). In some aspects, the mobile computing device 400 incorporates a vibration transducer for providing tactile feedback to the user. In yet another aspect, the mobile computing device 400 incorporates input and / or output ports, such as an audio input (e.g., a microphone jack), an audio output (e.g., a headphone jack), and a video output (e.g., an HDMI port), for sending signals to or receiving signals from external devices.

[0069] Figure 4B 4 is a block diagram illustrating the architecture of one aspect of a mobile computing device. That is, a mobile computing device 400 can incorporate a system (e.g., architecture) 402 that implements some aspects. In one embodiment, the system 402 is implemented as a "smartphone" capable of running one or more applications (e.g., a browser, email, calendar, contact manager, messaging client, games, and media client / player). In some aspects, the system 402 is integrated into a computing device, such as an integrated personal digital assistant (PDA) and a wireless phone.

[0070] One or more application programs 466 can be loaded into memory 462 and run on or in association with operating system 464. Examples of application programs include a phone dialer, an email program, a personal information management (PIM) program, a word processing program, a spreadsheet program, an Internet browser program, a messaging program, and the like. System 402 also includes a non-volatile storage area 468 within memory 462. Non-volatile storage area 468 can be used to store persistent information that should not be lost when system 402 loses power. Application programs 466 can use and store information in non-volatile storage area 468, such as email or other messages used by email applications. A synchronization application (not shown) also resides on system 402 and is programmed to interact with a corresponding synchronization application residing on a host computer to synchronize information stored in non-volatile storage area 468 with corresponding information stored on the host computer. As will be appreciated, other applications can be loaded into memory 462 and run on mobile computing device 400 as described herein (e.g., a search engine, an extractor module, a relevance ranking module, an answer scoring module, and the like).

[0071] System 402 has a power source 470, which can be implemented as one or more batteries. Power source 470 can also include an external power source, such as an AC adapter or a powered docking station to replenish or recharge the batteries.

[0072] System 402 may also include a radio interface layer 472 that performs the function of sending and receiving radio frequency communications. Radio interface layer 472 facilitates wireless connectivity between system 402 and the "outside world" via a communications carrier or service provider. Transmissions to and from radio interface layer 472 are performed under the control of operating system 464. In other words, communications received by radio interface layer 472 can be propagated to application programs 466 via operating system 464, and vice versa.

[0073] The visual indicator 420 can be used to provide a visual notification, and / or the audio interface 474 can be used to generate an auditory notification via the audio transducer 425. In the illustrated embodiment, the visual indicator 420 is a light emitting diode (LED) and the audio transducer 425 is a speaker. These devices can be directly coupled to the power supply 470 so that when activated, they remain on for the duration specified by the notification mechanism - even if the processor 460 and other components may be turned off to conserve battery power. The LED can be programmed to remain lit indefinitely until the user takes action to indicate the device's power status. The audio interface 474 is used to provide audible signals to the user and receive audible signals from the user. For example, in addition to being coupled to the audio transducer 425, the audio interface 474 can also be coupled to a microphone to receive audible input, such as to facilitate a telephone conversation. According to embodiments of the present disclosure, the microphone can also be used as an audio sensor to facilitate the control of notifications, as will be described below. The system 402 can also include a video interface 476, which enables the onboard camera 430 to operate to record still images, video streams, etc.

[0074] The mobile computing device 400 implementing the system 402 may have additional features or functionalities. For example, the mobile computing device 400 may also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage devices may be used to store data. Figure 4B , illustrated by non-volatile storage area 468 .

[0075] The data / information generated or captured by the mobile computing device 400 and stored via the system 402 may be stored locally on the mobile computing device 400, as described above, or the data may be stored on any number of storage media that can be accessed by the device via the radio interface layer 472 or via a wired connection between the mobile computing device 400 and a separate computing device associated with the mobile computing device 400 (e.g., a server computer in a distributed computing network such as the Internet). It should be appreciated that such data / information may be accessed via the mobile computing device 400 via the radio interface layer 472 or via a distributed computing network. Similarly, such data / information may be readily transferred between computing devices for storage and use according to well-known data / information transmission and storage means, including electronic mail and collaborative data / information sharing systems.

[0076] Figure 55. The diagram illustrates one aspect of the architecture of a system for processing data received from a remote source at a computing system, such as a personal computer 504, a tablet computing device 506, or a mobile computing device 508, as described above. The content displayed at the server device 502 can be stored in different communication channels or other storage types. For example, a directory service 522, a web portal 524, a mailbox service 526, an instant messaging repository 528, or a social networking site 530 can be used to store various documents.

[0077] 502. The client application 520 can be employed by a client communicating with the server device 502, and / or the policy generator 521 can be employed by the server device 502. The server device 502 can provide data to and from client computing devices, such as a personal computer 504, a tablet computing device 506, and / or a mobile computing device 508 (e.g., a smartphone), via a network 515. For example, the computer system described above can be embodied in the personal computer 504, the tablet computing device 506, and / or the mobile computing device 508 (e.g., a smartphone). In addition to receiving graphics data that can be used for pre-processing at a graphics originating system or post-processing at a receiving computing system, any of these embodiments of the computing device can also obtain content from the repository 516.

[0078] Figure 6 An exemplary tablet computing device 600 is illustrated that can perform one or more aspects disclosed herein. In addition, the aspects and functionality described herein can be operated on a distributed system (e.g., a cloud-based computing system) where application functionality, memory, data storage and retrieval, and various processing functions can be remotely operated from each other via a distributed computing network, such as the Internet or an intranet. Various types of user interfaces and information can be displayed via an onboard computing device display or via a remote display unit associated with one or more computing devices. For example, various types of user interfaces and information can be displayed and interacted with on a wall onto which various types of user interfaces and information are projected. Interaction with a variety of computing systems in which embodiments of the present invention can be practiced includes keystroke input, touch screen input, voice or other audio input, gesture input where the associated computing device is equipped with detection (e.g., camera) functionality for capturing and interpreting user gestures to control the functionality of the computing device, and the like.

[0079] As will be understood from the foregoing disclosure, one aspect of the present technology relates to a system comprising at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the system to perform a set of operations. The set of operations includes: generating a reward predictor for historical data associated with a logging policy; determining an off-policy evaluation model, wherein the off-policy evaluation model includes an estimator selected from the group consisting of a quality-agnostic estimator and a quality-based estimator; evaluating a target policy using the off-policy evaluation model to determine whether an expected reward metric of the target policy is higher than a reward metric of the logging policy; and when it is determined that the expected reward metric is higher than the reward metric of the logging policy, generating an indication to use the target policy instead of the logging policy. In an example, determining the off-policy evaluation model includes: generating a first mean squared error (MSE) metric for the quality-agnostic estimator; generating a second MSE metric for the quality-based estimator; when the first MSE is less than the second MSE, selecting the quality-agnostic estimator as the estimator; and when the second MSE is less than the first MSE, selecting the quality-based estimator as the estimator. In another example, the off-policy evaluation model includes a combination of direct modeling of a reward predictor and an inverse propensity score, and a weight of the reward predictor in the off-policy evaluation model is determined based on the estimator. In another example, determining the off-policy evaluation model includes determining hyperparameters of the estimator. In yet another example, the set of operations further includes: receiving a second indication of a context from a user device; determining an action based on the received context according to the target policy; and providing a third indication of the determined action in response to the first indication. In yet another example, the quality-agnostic estimator includes a threshold at which the importance weight is clipped if the weight exceeds the threshold. In an example, the set of operations further includes: accessing historical data from a historical data repository, wherein the historical data includes at least one context, an action associated with the context, and a reward for the action.

[0080] In another aspect, the technology relates to a method for selecting a new policy based on a previous policy. The method includes: accessing historical data associated with the previous policy, the historical data including at least one context, an action determined based on the context, and a reward for the action; evaluating the new policy using an off-policy evaluation model to determine whether to use the new policy instead of the previous policy, wherein the off-policy evaluation model includes a direct model, an inverse propensity score, and a combination of estimators selected from the group consisting of a quality-agnostic estimator and a quality-based estimator; and based on determining that the new policy should be used instead of the previous policy: generating an action for the context according to the new policy; and providing an indication of the action to a user device. In an example, when an average reward metric for the new policy is higher than an average reward metric for the previous policy, determining to use the new policy instead of the old policy, and determining the average reward metric for the new policy using the off-policy evaluation model. In an example, when the average reward metric for the new policy is higher than the average reward metric for the previous policy, determining to use the new policy instead of the old policy, and determining the average reward metric for the new policy using the off-policy evaluation model. In another example, an estimator for an off-policy evaluation model is selected by generating a first mean squared error (MSE) metric for a quality-agnostic estimator; generating a second MSE metric for a quality-based estimator; selecting the quality-agnostic estimator as the estimator when the first MSE is less than the second MSE; and selecting the quality-based estimator as the estimator when the second MSE is less than the first MSE. In yet another example, the method further comprises determining hyperparameters for the estimator for the selected estimator, wherein the hyperparameters are determined by iteratively refining the hyperparameters to reduce the MSE of the selected estimator. In yet another example, the quality-agnostic estimator includes a threshold at which the importance weights are clipped if the weights exceed the threshold. In yet another example, the direct model is a regression model on historical data, and wherein the inverse propensity score generates weights for predicting rewards.

[0081] In another aspect, the technology relates to another method for off-policy evaluation of a target policy. The method includes: generating a reward predictor for historical data associated with a logging policy; determining an off-policy evaluation model, wherein the off-policy evaluation model includes an estimator selected from the group consisting of a quality-agnostic estimator and a quality-based estimator; evaluating the target policy using the off-policy evaluation model to determine whether an expected reward metric of the target policy is higher than a reward metric of the logging policy; and when it is determined that the expected reward metric is higher than the reward metric of the logging policy, generating an indication to use the target policy instead of the logging policy. In an example, determining the off-policy evaluation model includes: generating a first mean squared error (MSE) metric for the quality-agnostic estimator; generating a second MSE metric for the quality-based estimator; when the first MSE is less than the second MSE, selecting the quality-agnostic estimator as the estimator; and when the second MSE is less than the first MSE, selecting the quality-based estimator as the estimator. In another example, the off-policy evaluation model includes a combination of direct modeling of the reward predictor and an inverse propensity score, and the weight of the reward predictor in the off-policy evaluation model is determined based on the estimator. In another example, determining the off-policy evaluation model includes generating hyperparameters for the estimator. In yet another example, the method further includes: receiving a second indication of a context from a user device; determining an action based on the received context according to the target policy; and providing a third indication of the determined action in response to the first indication. In yet another example, the quality-agnostic estimator includes a threshold at which the importance weight is clipped if the weight exceeds the threshold. In an example, the method further includes: accessing historical data from a historical data repository, wherein the historical data includes at least one context, an action associated with the context, and a reward for the action.

[0082] For example, aspects of the present disclosure are described above with reference to block diagrams and / or operational descriptions of methods, systems, and computer program products according to aspects of the present disclosure. The functions / actions noted in the blocks may not occur in the order shown in any flowchart. For example, depending on the functionality / actions involved, two blocks shown in succession may actually be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order.

[0083] The description and explanation of one or more aspects provided in this application are not intended to limit or restrict the scope of the disclosure claimed in any way. The aspects, examples and details provided in this application are considered to be sufficient to convey ownership and enable others to make and use the disclosed best mode claimed. The disclosure claimed should not be interpreted as being limited to any aspect, example or details provided in this application. No matter whether it is shown and described in combination or separately, various features (structure and method) are intended to be selectively included or omitted to produce an embodiment with a specific feature group. Description and explanation of the application have been provided, and those skilled in the art can envision the changes, modifications and alternative aspects that fall within the spirit of the broader aspects of the general inventive concept specifically implemented in this application without departing from the broader scope of the disclosure claimed in this application.

Claims

1. A system comprising: at least one processor; as well as a memory storing instructions that, when executed by the at least one processor, cause the system to perform a set of operations comprising: generating a reward predictor for historical data associated with a logging policy, wherein the reward predictor is operable to generate an expected reward based on the historical data; determining an off-policy evaluation model, wherein the off-policy evaluation model comprises an estimator selected from a group of estimators consisting of a quality-agnostic estimator and a quality-based estimator, wherein the quality-agnostic estimator comprises a threshold, and the quality-agnostic estimator uses the threshold to evaluate the importance weight; evaluating a target policy using the off-policy evaluation model and based on the historical data associated with the logging policy to determine whether an expected reward metric of the target policy is higher than a reward metric of the logging policy; and When it is determined that the expected reward metric is higher than the reward metric of the logging policy, an indication is generated to use the target policy instead of the logging policy, wherein: A first reward is generated based on the user completing a first action, and a second reward is generated based on the user completing a second action; and The reward predictor is trained to generate the expected reward for a given context.

2. The system of claim 1 , wherein determining the off-policy evaluation model comprises: generating a first mean square error (MSE) metric for the quality agnostic estimator; generating a second MSE metric for the quality-based estimator; When the first mean square error (MSE) is less than the second MSE, selecting the quality-agnostic estimator as the estimator; as well as When the second MSE is less than the first mean square error MSE, the quality-based estimator is selected as the estimator.

3. The system of claim 1 , wherein the off-policy evaluation model comprises a combination of direct modeling of reward predictors and inverse propensity scoring; and The weight of the reward predictor in the off-policy evaluation model is determined according to the estimator.

4. The system of claim 1, wherein determining the off-policy evaluation model comprises determining hyperparameters for the estimator.

5. The system of claim 1 , wherein the set of operations further comprises: receiving a second indication of a context from a user device; determining, according to the target policy, an action based on the received context; as well as In response to the second indication, a third indication of the determined action is provided. The system of claim 1 , wherein if the importance weight exceeds the threshold, the importance weight is clipped at the threshold.

7. The system of claim 1 , wherein the set of operations further comprises: The historical data is accessed from a historical data repository, wherein the historical data includes a context, an action associated with the context, and a reward for the action.

8. A method for off-policy evaluation of a target policy, the method comprising: generating a reward predictor for historical data associated with a logging policy, wherein the reward predictor is operable to generate an expected reward based on the historical data; determining an off-policy evaluation model, wherein the off-policy evaluation model comprises an estimator selected from a group of estimators consisting of a quality-agnostic estimator and a quality-based estimator, wherein the quality-agnostic estimator comprises a threshold, and the quality-agnostic estimator uses the threshold to evaluate the importance weight; evaluating the target policy using the off-policy evaluation model and based on the historical data associated with the logging policy to determine whether an expected reward metric of the target policy is higher than a reward metric of the logging policy; as well as When it is determined that the expected reward metric is higher than the reward metric of the logging policy, an indication is generated to use the target policy instead of the logging policy, wherein: A first reward is generated based on the user completing a first action, and a second reward is generated based on the user completing a second action; and The reward predictor is trained to generate the expected reward for a given context.

9. The method of claim 8, wherein determining the off-policy evaluation model comprises: generating a first mean square error (MSE) metric for the quality agnostic estimator; generating a second MSE metric for the quality-based estimator; When the first mean square error (MSE) is less than the second MSE, selecting the quality-agnostic estimator as the estimator; as well as When the second MSE is less than the first mean square error MSE, the quality-based estimator is selected as the estimator.

10. The method of claim 8, wherein the off-policy evaluation model comprises a combination of direct modeling of reward predictors and inverse propensity scoring; and The weight of the reward predictor in the off-policy evaluation model is determined according to the estimator.

11. The method of claim 8, wherein determining the off-policy evaluation model comprises generating hyperparameters for the estimator.

12. The method according to claim 8, further comprising: receiving a second indication of a context from a user device; determining, according to the target policy, an action based on the received context; as well as In response to the second indication, a third indication of the determined action is provided. The method of claim 8 , wherein if the importance weight exceeds the threshold, clipping the importance weight at the threshold.

14. The method according to claim 8, further comprising: The historical data is accessed from a historical data repository, wherein the historical data includes a context, an action associated with the context, and a reward for the action.

Images