Evaluation and adaptive sampling of proxy configurations

Abstract

This document relates to evaluation of automatic agents. One example includes a system having a processor and a storage medium. The storage medium may store instructions that, when executed by the processor, cause the system to perform two or more data collection iterations, which may include: allocating experimental units to a plurality of agents having different agent configurations according to a sampling policy; populating the event log with events representing a reaction of the environment to actions taken by the respective agents in response to the respective experimental units; and adjusting a sampling policy for subsequent data collection iterations based at least on events in the event log. The event log may provide a basis for subsequent evaluation of the plurality of agents relative to the one or more evaluation metrics.

Description

Background Art

[0001] Often, techniques such as A / B testing are used to evaluate different alternative configurations of various applications. For example, A / B testing can be used to compare two different algorithms for a web search engine, or to compare two different user interface configurations for a social networking service. However, as discussed in more detail below, A / B testing can be resource intensive in scenarios where multiple configurations are evaluated. Summary of the invention

[0002] This summary is provided to introduce a selection of concepts in a simplified form that are further described in the following detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0003] The present specification generally relates to techniques for evaluating computer-based agents. An example includes a method or technique that can be performed on a computing device. The method or technique may include performing two or more data collection iterations. Each data collection iteration may include assigning an experimental unit to multiple agents with different agent configurations according to a sampling strategy. Each data collection iteration may also include populating an event log with events representing the reaction of the environment to actions taken by each agent in response to each experimental unit. Each data collection iteration may also include adjusting the sampling strategy for subsequent data collection iterations based at least on events in the event log. The method or technique may also include predicting the performance of multiple agents relative to one or more evaluation metrics based at least on events in the event log. The method or technique may also include identifying a selected agent configuration based at least on the predicted performance of multiple agents relative to one or more evaluation metrics.

[0004] Another example includes a system having a hardware processing unit and a storage resource storing computer-readable instructions. When executed by the hardware processing unit, the computer-readable instructions may cause the hardware processing unit to perform two or more data collection iterations. Each data collection iteration may include assigning an experimental unit to multiple agents with different agent configurations according to a sampling strategy. Each data collection iteration may also include populating an event log with events representing the environment's reaction to the actions taken by each agent in response to each experimental unit. Each data collection iteration may also include adjusting the sampling strategy for subsequent data collection iterations based at least on the events in the event log. The event log may provide a basis for subsequent evaluation of multiple agents relative to one or more evaluation metrics.

[0005] Another example includes a hardware computer-readable storage medium storing computer-readable instructions. When executed by a hardware processing unit, the computer-readable instructions may cause the hardware processing unit to perform an action. The action may include obtaining an event log of events representing the reaction of the environment to actions taken by multiple agents in response to each experimental unit. The action may also include predicting the performance of each agent relative to one or more evaluation metrics based on at least corresponding events in the event log that reflect corresponding actions taken by other agents. The action may also include identifying a selected agent configuration based on at least the predicted performance of each agent relative to one or more evaluation metrics. The action may also include deploying the selected agent with the selected agent configuration.

[0006] The examples listed above are intended to provide a quick reference to aid the reader and are not intended to limit the scope of the concepts described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The detailed description is described with reference to the accompanying drawings. In the drawings, the left-most digit(s) of a reference number identifies the drawing in which the reference number first appears. The use of similar reference numbers in different instances in the description and the drawings may indicate similar or identical items.

[0008] Figure 1 An example proxy block framework consistent with some implementations of the present concepts is shown.

[0009] Figure 2 An example data collection workflow consistent with some implementations of the present concepts is shown.

[0010] Figure 3 An example data analysis workflow consistent with some implementations of the present concepts is shown.

[0011] Figure 4 An example data structure for storing performance predictions consistent with some implementations of the disclosed technology is shown.

[0012] Figure 5 An example adaptive sampling workflow consistent with some implementations of the present concepts is shown.

[0013] Fig. 6A , 6B and 6C show example graphical user interfaces consistent with some implementations of the present concepts, conveying the performance of alternative proxy configurations for evaluating metrics during adaptive sampling.

[0014] Fig. 7A An example agent that may be configured to perform reinforcement learning is shown, consistent with some embodiments of the present concepts.

[0015] Figure 7BAn example agent that may be configured to perform supervised learning is shown consistent with some embodiments of the present concepts.

[0016] Figure 8 An example system consistent with some implementations of the disclosed technology is shown.

[0017] Fig. 9 is a flow chart of an example method for evaluating an agent using an adaptive sampling strategy according to some embodiments of the present concepts.

[0018] Fig. 10A and 10B Example user experiences and user interfaces for content distribution scenarios consistent with some implementations of the present concepts are shown.

[0019] Fig.11A and 11B An example user experience and user interface for a voice or video call scenario consistent with some implementations of the present concepts is shown. DETAILED DESCRIPTION Overview

[0020] Traditionally, users who wish to select an agent to perform a computational task will directly compare agents using a testing method such as A / B testing. In an A / B test, an experiment is conducted with two different agents, and the user selects the agent with the best performance. Typically, computational resources are evenly divided between the two agents, for example, agent A may execute 100 samples on a processor, while agent B may execute another 100 samples on a processor. Samples collected by executing agent A are only used to evaluate agent A, while samples collected by executing agent B are only used to evaluate agent B.

[0021] A / B testing works quite well for specific use cases where only two alternative agents are considered. However, there are cases where users want to evaluate many different alternative agents in an efficient manner. A simple approach is to run a tournament of A / B testing, but this can take a lot of time and computing resources. For example, in a tournament with four agents, two rounds, and 100 samples per round per agent, a total of 600 samples are collected - all four agents execute on the processor in the first round, each collecting 100 samples, and then the two winning agents execute on the processor again in the second round, each collecting 100 samples.

[0022] Now, suppose a user wants to evaluate several different types of agents - supervised learning agents, unsupervised learning agents, reinforcement learning agents, and / or heuristic-based agents. Each type of agent can have a different underlying algorithm, for example, supervised learning can be implemented using neural networks, support vector machines, etc., or reinforcement learning can be implemented using policy iteration methods, context grouping, etc. In addition, each algorithm can have various model structures, hyperparameters, etc. The resulting search space of potential agent configurations is very large, so it is impractical to conduct a complete A / B test of all possible agent configurations.

[0023] The disclosed implementation uses samples collected by executing one agent to infer the performance of another agent, which can be used to evaluate different agent configurations. In other words, samples collected by agent A can be reused to infer the performance of agent B. As a result, less computing resources can be used to gain insight into the performance of different agents than is typically involved in A / B testing, where only samples collected by a given agent are used to evaluate the performance of that agent.

[0024] The disclosed implementations may also employ adaptive sampling methods that use agent behavior to change the way sampling proceeds over time. For example, in some cases, the probabilities that agents were assigned to various actions during previously collected samples may be used to adjust the probabilities that agents are assigned to handle subsequent samples. In other cases, agents may be removed from future sampling based on their performance with respect to one or more evaluation metrics. Overview of Supervised Learning

[0025] Supervised learning typically involves training an agent using labeled training data. In supervised learning, the agent updates its own internal model parameters based on a loss function defined on the labels. Supervised learning can be implemented using model structures such as support vector machines, neural networks, decision trees, etc. Supervised learning models can be used for tasks such as classification and / or regression. In some cases, a supervised learning model can output a probability distribution over a set of actions, for example, assigning a 90% probability of taking a first action and a 10% probability of taking a second action given a set of inputs (e.g., context).

[0026] Some machine learning building blocks (such as neural networks) use layers of nodes that perform specific operations. In a neural network, nodes are connected to each other via one or more edges. A neural network can include an input layer, an output layer, and one or more intermediate layers. Each node processes their respective inputs according to a predefined function and provides outputs to subsequent layers (or in some cases, to previous layers). The input of a given node can be multiplied by the corresponding weight value of the edge between the input and the node. In addition, separate bias values ​​are also used to generate outputs. Various training processes can be applied to learn edge weights and / or bias values. Reinforcement Learning Overview

[0027] Reinforcement learning typically involves an agent taking various actions in an environment according to a policy, and adjusting the policy based on how the environment reacts to those actions. Reinforcement learning does not necessarily rely on labeled training data like supervised learning does. Instead, in reinforcement learning, the agent evaluates the environment's reactions using a reward function and aims to determine a policy that tends to maximize or increase the agent's cumulative reward.

[0028] In some cases, the reward function can be defined by the user based on the environment's reaction, for example, a desired outcome is 1 point, a neutral outcome is 0 points, and a negative outcome is -1 point. The agent proceeds in a series of steps, and in each step, the agent has one or more possible actions that it can take. For each action taken by the agent, the agent observes the environment's reaction, calculates the corresponding reward according to the reward function, and can update its own policy based on the calculated reward.

[0029] Reinforcement learning can strike a balance between "exploration" and "exploitation". Typically, exploitation involves taking actions that are expected to maximize immediate reward given the current policy, while exploration involves taking actions that do not necessarily maximize expected immediate reward but search for unexplored or underexplored actions. In some cases, the agent may choose an action during the exploration phase that produces a cumulative reward greater than the best action under its current policy, and the agent may update its policy to reflect this new information.

[0030] In some reinforcement learning scenarios, an agent can use the context that describes the environment that the agent is interacting with to choose an action to take. For example, a contextual bandit receives contextual features that describe the current state of the environment and uses these features to choose the next action to take. A contextual bandit agent can maintain a history of the rewards obtained for different actions taken in different contexts and continue to modify the policy as new information is discovered.

[0031] One type of contextual bandit is a linear model, such as Vowpal Wabbit. Such a model can output a probability density function of available actions at each step, and randomly select an action from the probability density function. The model can learn feature weights applied to one or more input features (e.g., describing the context) to determine the probability density function. When the reward obtained at a given step does not match the expected reward, the agent can update the weights used to determine the probability density function. definition

[0032] For the purposes of this article, an agent is an automated entity that can determine the probability distribution of one or more actions that can be taken in an environment, and / or select a specific action to take. An agent can determine the probability distribution and / or select an action based on a policy. For example, a policy can map an environmental context to the probabilities of actions that an agent can take. Some agents can employ machine learning, for example, an agent can be updated based on the environment's response to the action chosen by the agent, either through a reward function (reinforcement learning) or a loss function (supervised learning). The term "internal parameters" is used in this article to refer to learnable values, such as weights, that can be learned by training a machine learning model (e.g., a linear model or a neural network). An experimental unit is a data item on which an agent can take action, for example, an experimental unit can specify a context and / or a set of actions for the agent to choose based on the context.

[0033] Machine learning models may also have hyperparameters that control how the agent acts and / or learns. For example, a machine learning model may have a learning rate, a loss or reward function, an exploration strategy, and the like. A machine learning model may also have feature definitions, for example, mapping information about the environment to specific features that the model uses to represent that information. A feature definition may include the type of information the model receives and how that information is represented. For example, two different feature definitions may both indicate that the model receives contextual features that describe the user's age, but one feature definition may identify specific ages (e.g., 24, 36, 68, etc.) while another feature definition may only identify corresponding age ranges (e.g., 21-30, 31-40, and 61-70).

[0034] An agent configuration is a specification of at least one characteristic of the agent, such as rules, model structure, loss or reward function, feature definition, or hyperparameters. A policy is a function that determines what action the agent takes given a context. Policies can be static or learned from the agent configuration. Note that an agent's policy can be defined heuristically or using a static probability distribution. For example, an agent can use a uniformly randomly sampled policy from a set of available actions, without having to update the policy in response to environmental reactions. Rule-based agents can have static rules that directly map context values ​​to specific actions or action probabilities.

[0035] A specific proxy configuration may be sampled by processing an experimental unit with an agent configured according to the proxy configuration. A specific proxy configuration may be evaluated by predicting the performance of the proxy configuration in terms of one or more evaluation metrics using data sampled by the proxy configuration or other proxy configurations. A specific proxy configuration may be deployed by placing the proxy configuration into a service for a specific application, e.g., executing the specific proxy configuration to select actions for a specific application in a production environment. Example Learning Blocks

[0036] Figure 1 An example is shown where an agent 102 receives context information 104, action information 106, and reaction information 108. The context information represents the state of an environment 110. The action information represents one or more available actions 112. The agent may select a selected action 114 based on the context information. The reaction information may represent how the state of the environment changes in response to the action selected by the agent. For a reinforcement learning model, the reaction information 108 may be used in a reward function to determine a reward for the agent 102 based on how the environment changes in response to the selected action. In some cases, reactions may be labeled by manual or automated techniques to train a supervised learning agent.

[0037] In some cases, the actions available to an agent may be independent of context—for example, all actions may be available to the agent in all contexts. In other cases, the actions available to an agent may be constrained by context, such that an action available to the agent in one context is not available in another context. Thus, in some implementations, context information 104 may specify the actions available to the agent in the context in which the current agent is running. Example Data Collection Workflow

[0038] Figure 2 An example data collection workflow 200 is shown in which an experimental unit 202 is received and sampled by a sampler 204. The sampler assigns each experimental unit to a plurality of agents 102(1)...102(N) according to a sampling strategy, where N is the size of the sampling pool of agents. Each agent outputs a corresponding event group 206(1)...206(N) for populating an event log 208. As discussed in more detail below, each event can include information such as the context in which a given agent takes an action, the action taken, the environment's reaction to the action, and / or the probability assigned by the agent to the action taken. Each agent can have a different agent configuration, so even given the same experimental unit, the agents can assign different probabilities to different actions.

[0039] As previously described, the experimental unit 202 may include any data that the agent may select an action. In some cases, the experimental unit includes both context information and action information, and in other cases only includes context information. As sampling is performed in multiple sampling iterations, the sampler 204 can adjust the probability of assigning each agent to handle each experimental unit. As discussed in more detail below, this can allow the sampler to sample data in a manner that allows multiple different agent configurations to be evaluated in an efficient manner using the resulting event log. In some cases, the event log can be used to evaluate other agents that were not sampled when filling the event log. Example Data Analysis Workflow

[0040] Figure 3 An example data analysis workflow 300 is shown in which event logs 208 are processed to predict the performance of different agent configurations 102(1) and 102(M), as described in more detail below. As previously described, event logs can be obtained by sampling agents 102(1)...102(N) to process experimental units according to their respective agent configurations. In some cases, M is greater than N, for example, the performance of M=100 agent configurations can be predicted based on sampling of N=25 agent configurations. In this case, the M alternative agent configurations can include the 75 agent configurations that were not sampled when populating the event logs.

[0041] The term "log agent" is used herein to refer to any agent that generates a particular event in the event log, for example, when a sampler is assigned to a given experimental unit. The term "log agent configuration" is used herein to refer to the configuration of a log agent. As previously described, each event in the event log can identify a context associated with the experimental unit, an action taken by the agent, and the reaction of the environment to the action taken by the agent. An event value 302 can be determined for each event, wherein the event value reflects the value of the event relative to one or more evaluation metrics. In some cases, a function is used to map reactions and optional context and actions to event values ​​to determine event values, as described below.

[0042] A log-based action probability 304 may be determined for each event in the event log 208, where the log-based action probability represents the likelihood of each action taken that the log agent computes for each event in the log. Thus, assume that a particular event in the event log indicates that for a given context associated with the event, the probability density function determined by the agent is {action A == 0.7, action B == 0.3}. If the log agent takes action A for the particular event, the log-based action probability for the event is 0.7, and if the agent takes action B for the particular event, the log-based action probability for the event is 0.3.

[0043] Agents 102(1)...102(M) may be configured according to various alternative agent configurations 306. The agents configured in each alternative event configuration may be used to replay events in the event log 208, such that each alternative event configuration may be used to process events in the event log offline. For each event in the event log, a predicted action probability 308 may be predicted. Here, the predicted action probability represents the probability that the agent will take the action taken in the event log if the corresponding alternative agent configuration is used instead of the log agent configuration. Thus, for example, assume that alternative agent configuration 1 is a probability density function {action A == 0.8, action B == 0.2} calculated for a particular event in the event log. If the event log indicates that the agent took action A for the event (e.g., when configured via the log agent configuration), then for alternative agent configuration 1, the predicted action probability for the event is 0.8. If the event log indicates that the agent took action B for the event, then for alternative agent configuration 1, the predicted action probability for the event is 0.2.

[0044] Evaluation metric predictor 310 can predict aggregate values ​​of one or more evaluation metrics for each alternative agent configuration to populate performance prediction 312. Here, each performance prediction conveys the predicted performance of a particular alternative agent configuration for a particular evaluation metric. By comparing the predicted performance of different agent configurations for different evaluation metrics, a selected agent configuration can be identified. Example Performance Prediction Data Structure

[0045] Figure 4 A performance prediction table 400 is shown, which is one example of a data structure that can be used to store performance predictions 312. Each row of the performance prediction table 400 represents a different alternative agent configuration, and each column of the table represents a different evaluation metric. As previously described, the evaluation metric can be based on a function that maps environmental reactions and optionally selected actions and / or contexts to the value of a given evaluation metric.

[0046] Examples of evaluation metrics and corresponding functions for specific applications are described in detail below, but for now, consider the following brief example. Suppose the function defines the following value for metric 1: For events where the agent has reaction 1 when selecting action 1 in the first context, the value of metric 1 is 1; For an event where the agent has reaction 1 when selecting action 1 in the second context, the value of metric 1 is 2; For an event where the agent has a reaction of 1 when selecting action 2 in the first context, the value of metric 1 is 10; For the event where the agent has reaction 1 when selecting action 2 in the second context, the value of metric 1 is 8; For events with reaction 2, the value of metric 1 is 0, regardless of which action the agent or context chooses.

[0047] Using this function, each event in the event log can be extracted, and the value of metric 1 can be determined for the event based on the action actually taken by the agent, the context in which the agent takes the action, and the reaction of the environment. Then, the value of metric 1 can be adjusted for each alternative agent configuration. As shown below, the value of metric 1 is multiplied by the probability that a specific alternative agent configuration is given the same action given the same context of the event, the number is divided by the probability that the agent gives the selected action in the log agent configuration, and the value is added to the column of metric 1 of the row for the specific agent configuration. These calculations can be performed for each event in the log. The resulting value conveys the expected value of metric 1 in the first column of the performance prediction table 400 for each alternative agent configuration. These steps can be performed for different evaluation metrics (e.g., calculated using different functions) to fill the rest of the table. Specific algorithm

[0048] A more detailed definition of the variables and formulas that can be used to populate the performance prediction table 400 is provided below. The term "log agent" is used below to refer to an agent configured according to the log agent configuration. In other words, "log agent" refers to the configuration state of the agent when collecting events in the event log 208. For each event in the event log, define the following: x (vector): This is called the context of the decision. It contains the environment (context) features, the possible actions that the agent can take for each event, and the action features of the available actions. The context features describe the environment in which the agent chooses a particular action; a (index): the action actually taken by the logging agent (out of the possible options specified in x); p_log (scalar between 0 and 1): probability that the logging agent takes action a, as indicated in the event log; y (vector): a vector of observation features describing the environment’s response to the action a chosen by the logging agent; r (vector): This vector defines the multidimensional value of this event, e.g., r is a way of expressing a function that maps events to the value of one or more evaluation metrics. Each entry in the vector represents the value of a particular evaluation metric for selecting action a in context x given observed feature y. This vector can be specified by the user when evaluating alternative proxy configurations using events in the log.

[0049] For each event in event log 208, the expected value of that event relative to a particular evaluation metric for a given alternative proxy configuration may be calculated as follows:

[0050] As mentioned above, r iThe value of the vector r given the action taken by the logging agent, the context in which the action was taken, and the reaction time of the environment. So, for example, if the r of a given event i Given a K-dimensional vector of {1,4,...27}, this means that the event has a value of 1 for evaluation metric 1, a value of 4 for evaluation metric 2, and a value of 27 for evaluation metric K.

[0051] r i Each value in can be adjusted by multiplying the value by the probability that a given alternative agent configuration would take an action in the log, divided by the probability that the log agent would take that action. Thus, if the alternative agent configuration is more likely to take an action than the log agent given the context of that event, then the value essentially increases r for the alternative agent configuration. i The weight of the vector value and is lower if the alternative agent configuration is less likely than the logging agent to have taken action given the context of this event.

[0052] Note that some embodiments may also define constraints on which alternative proxy configurations should be considered. For example, a constraint may specify that only alternative proxy configurations with a value of at least 1000 for a particular evaluation metric should be considered. Any proxy configurations with lower values ​​may be filtered out before selecting a new proxy configuration from the remaining available configurations. generalization

[0053] As described above, each column can represent the predicted performance of a given evaluation metric calculated over each event in the event log 208. However, in some cases, the evaluation metric can be calculated over an episode of multiple events. For example, an episode can be specified as a constant number (e.g., every 10 events), a time range (e.g., all events that occurred on a given day), or any other grouping of interest to the user. Episode values ​​calculated over the entire episode of events can replace individual event values ​​to determine performance predictions.

[0054] In addition, the previous description can be used to calculate the average expected value of each evaluation metric. However, other implementations may consider other statistical metrics, such as medians, percentile values ​​(e.g., 10th, 50th, 90th), standard deviations, etc. These statistical metrics can be calculated for each individual event or episode of multiple events in the log. In addition, confidence intervals can be calculated for each statistical measure.

[0055] The following statement: in:

[0056] Can be used to calculate a given statistical measure for any event scenario definition. Adaptive Sampling Workflow

[0057] Figure 5 An adaptive sampling workflow 500 is shown, integrating the processes described above with respect to the data collection workflow 200 and the data analysis workflow 300. Experimental units 202 are input to the data collection workflow 200 to populate the event log 208, as previously described. The data analysis workflow 300 is executed on the event log to generate performance predictions 312.

[0058] Next, sampling adaptation 502 is performed based on the performance prediction 312. In general, sampling adaptation can adjust the sampling probability of each agent 102(1)...102(N) in the data collection workflow 200. After the termination condition is reached, agent selection / configuration 504 is performed to select a final agent configuration 506 from a set of M agent configurations, which can include the sampled N agent configurations and other agent configurations that have not been sampled.

[0059] In some cases, sampling adaptation can provide different sampling modes. In the first sampling mode, the user can manually define the sampling probability of each agent configuration. For example, the user can decide to evaluate six agent configurations A, B, C, E, F, and G. The user can specify agent configurations A, B, and C to be sampled with equal sampling probability (e.g., 33%) for the first data collection iteration, where the probability of agent configurations E, F, and G is zero. The user can decide to eliminate agent C from subsequent data collection iterations and then evaluate the remaining two agent configurations, focusing on one agent, for example, the probability of agent A is 60% and the probability of agent B is 40%. The performance of all six agent configurations can be predicted based only on the data collected by agent configurations A, B, and C, and then any one of the six agent configurations can be selected for deployment.

[0060] In a second sampling mode, the sampling probability of each agent configuration can be determined according to an importance weighting scheme. In the importance weighting scheme, the probability of each agent assigning an action in the event log is divided by the probability of the action taken by the logging agent to determine the importance weight of each event. The resulting values ​​for each agent are then summed and divided by the number of events in the event log to obtain the average importance weight of each agent. Subsequently, z=abs(ln(the average of the importance weights)) is used as a sampling metric that can be calculated for each agent configuration. The probability distribution over the agent configurations can then be determined based on this sampling metric, as described in more detail below.

[0061] One way to employ the sampling metric z involves removing individual agent configurations from sampling before subsequent data collection iterations. For example, a user may define the number X of agent configurations to be sampled per data collection iteration, e.g., X = {N, 10, 5} for three data collection iterations. In each data collection iteration, z may be calculated for each agent configuration using the sampled data obtained so far, and then the X agent configurations with the highest z values ​​may be assigned a uniform sampling probability of 1 / X in the next data collection iteration.

[0062] Assuming N=25, 25 agent configurations are sampled with equal probability in the first sampling iteration. Then, the 10 agents with the highest z-values ​​among the N agents sampled in the first data collection iteration are sampled with equal probability in the second data collection iteration, while 15 agents are excluded from sampling (e.g., assigned a sampling probability of 0). Then, the 5 agents with the highest z-values ​​among the 10 agents sampled in the second data collection iteration are sampled with equal probability in the third data collection iteration, while the other 5 agents are excluded from sampling. In other cases, the sampling probability for each data collection iteration can be determined by applying a function such as softmax to the z-values ​​determined in the previous data collection iteration.

[0063] In general, when the average importance weight of a given agent is close to 1, this indicates that the data that has been collected can accurately predict the performance of the agent. Conversely, when the average importance weight of a given agent is far from 1, this indicates that the data that has been collected is not very accurate for predicting the performance of the agent. In the uniform sampling probability example given above, agent configurations with average importance weights close to 1 tend to be removed from subsequent data collection iterations, allowing other agent configurations whose existing samples are relatively less predictive to obtain more samples. Using a softmax function instead of uniform sampling has the additional benefit that higher sampling probabilities tend to be assigned to those agent configurations whose sampled data provide relatively low confidence in predicting performance, while lower sampling probabilities are assigned to those agent configurations whose sampled data provide relatively high confidence in predicting performance. Therefore, each subsequent data collection iteration tends to emphasize collecting new samples for certain agents, thereby increasing the overall confidence that the performance of the M agents can be predicted.

[0064] In the third sampling mode, sampling is performed by determining sampling probabilities based on the performance of individual agents relative to one or more evaluation metrics. For example, suppose a user specifies an evaluation metric that they wish to maximize on average. One approach is to determine an upper bound (e.g., an upper bound of a 95% confidence interval) for each agent configuration for that evaluation metric. This upper bound can then be used as a sampling metric in a manner similar to that described above with respect to z for the second sampling mode.

[0065] In each data collection iteration, the X agent configurations with the highest values ​​of a given evaluation metric upper limit can be assigned a uniform probability of 1 / X in the next data collection iteration. In this example, in the first sampling iteration, all N=25 agent configurations are sampled with equal probability. Then, in the second data collection iteration, 10 agents with the highest upper limit of the selected evaluation metric among the 25 agents sampled in the first data collection iteration are sampled with equal probability, where 15 agents are excluded from sampling. Then, in the third data collection iteration, 5 agents with the highest upper limit among the 10 agents sampled in the second data collection iteration are sampled with equal probability, where another 5 agents are excluded from sampling. In other cases, the sampling probability can be determined for each data collection iteration by applying a function such as softmax to the upper limit value determined in the previous data collection iteration.

[0066] For some cases, the user may be interested in minimizing rather than maximizing a given evaluation metric. If so, the opposite value may be maximized instead. For example, if the user wishes to minimize a given evaluation metric, the upper limit of the opposite value (e.g., negative value) of the metric may be used as the sampling metric. For cases where the user wishes to consider multiple evaluation metrics, the user may provide corresponding weights or coefficients for each evaluation metric of interest to define the sampling metric. For example, if the user selects a weight of 2 for evaluation metric 1 and a value of 5 for evaluation metric 3, the sampling metric may be calculated as (2*evaluation metric 1) + (5*evaluation metric 3) and the probabilities derived using uniform and / or softmax as previously described.

[0067] As sampling proceeds, confidence intervals will tend to get smaller. By assigning higher sampling probabilities to those agents with higher upper confidence intervals, each subsequent data collection iteration tends to emphasize collecting new samples for those agents, thereby reducing confidence intervals for the evaluation metrics and tending to prioritize sampling agent configurations with potentially optimal performance for one or more evaluation metrics. As the confidence intervals for two different agents get smaller, at some point one can be said to statistically "dominate" the other. In other words, it is statistically unlikely that agent A will perform better than agent B on one or more evaluation metrics, given the confidence intervals for those metrics. In this case, agent A can automatically be removed from subsequent data collection iterations.

[0068] Note that the first and second sampling modes can be performed without defining a function for the evaluation metric. The first sampling mode can be implemented without knowledge of the resulting events from the previous sampling iteration in the event log 208. The second sampling mode can adjust sampling based on the event log without determining the performance prediction of the agent. The third sampling mode generally involves using performance predictions to inform the sampling strategy of future data collection iterations. Example sampling adaptation for the third sampling mode

[0069] Shown below are various graphical representations of how the sampling probabilities of different surrogate configurations vary over the data collection iterations according to their different predictive performance on different evaluation metrics.

[0070] Fig. 6A , 6B 6C shows an example output graph 600 , where the y-axis represents evaluation metric 1 and the x-axis represents evaluation metric 2 . Fig. 6A represents the state of the curve after the first data collection iteration, Figure 6B represents the state of the curve after the second data collection iteration, and Figure 6C represents the state of the curve after the third data collection iteration. For the purposes of the following examples, it is assumed that agent configurations are automatically removed from sampling when they are statistically dominated by other agent configurations. Alternative approaches are described in more detail below.

[0071] Each entry on curve 600 represents an aggregate value of the evaluation metric predicted for the corresponding agent configuration. As shown in legend 602, various alternative agent configurations are represented by circular black dots 604 and rectangles 606. Each circular black dot conveys the predicted aggregate value of evaluation metrics 1 and 2 for the agent in a different alternative agent configuration. Each rectangle shows a 95% confidence interval relative to each metric. Therefore, curve 600 graphically represents the performance of predicting different alternative agent configurations for two different evaluation metrics, as well as the relative confidence of each metric given the data currently sampled in the event log.

[0072] For the purposes of the following examples, assume that users generally want to maximize the value of evaluation metric 1 while minimizing the value of evaluation metric 2. However, observe that this involves certain trade-offs, as the value of evaluation metric 2 tends to increase as evaluation metric 1 increases. In other words, those alternative agent configurations that have high values ​​of evaluation metric 1 also tend to result in relatively high values ​​of evaluation metric 2.

[0073] Thus, in general, any first point above and to the left of a second point on curve 600 may be referred to as "dominant" the second point. In other words, the second point has both a lower value for evaluation metric 1 (which the user wishes to maximize) and a higher value for evaluation metric 2 (which the user wishes to minimize).

[0074] exist Fig. 6A In the example above, most rectangles 606 have at least one point that is above or to the left of at least one point in another rectangle. Note, however, that every point in rectangle 606(1) is below and to the right of every point in rectangle 606(2). Subject to the statistical constraints of the confidence interval, it is clear that the corresponding agent configuration of rectangle 606(1) is strictly lower than the agent configuration of rectangle 606(2). In other words, rectangle 606(1) does not contain any point on the Pareto frontier of possible agent configurations.

[0075] One way to implement sampling adaptation 502 in adaptive sampling workflow 500 is to filter out any agent configuration that does not have a point on the Pareto frontier, eg, assigning that agent configuration a zero sampling probability for the next round of sampling. Figure 6B An example after filtering is shown, e.g., rectangle 606(1) and its corresponding agent configuration have not yet been sampled. The remaining agent configurations continue to be sampled, so the size of each rectangle decreases as the corresponding confidence intervals of the two metrics decrease. As previously described, in each subsequent iteration, the sampling probability of the remaining agents can be uniform or proportional to the upper limit of the one or more metrics being considered.

[0076] exist Figure 6B , note that rectangle 606(3) is now dominated by rectangle 606(4). Therefore, the agent configuration of rectangle 606(3) can be filtered from further sampling in the next data collection iteration. Note that the mean and confidence interval can change during the data collection iteration.

[0077] Figure 6C The eight remaining rectangles are shown. Since no rectangle completely dominates another, Figure 6C No filtering is performed in the data collection iteration represented by . Further sampling and evaluation can be performed as described above. The final configuration can be selected from the 8 remaining configurations automatically or based on user input, for example, for Figure 6C The GUI shown in .

[0078] The above description assumes that the proxy configurations are automatically filtered based on statistical dominance in each data collection iteration, and can be performed in a fully automated manner without user input. However, in further implementations, user input can be used to guide how to sample in subsequent data collection iterations. For example, as described previously, a user can specify how many configurations to evaluate in each data collection iteration, in which case only the top proxy configurations from the previous data collection iteration are sampled in each subsequent iteration.

[0079] In other cases, the user can specifically select one or more agent configurations via a GUI to specify configurations to be included (or excluded) for further data collection iterations. This can also involve scenarios in which the user can view multiple different GUIs that illustrate the performance of the agent for different evaluation metrics. For example, the user can provide input, request a first drawing of evaluation metrics 1 and 2 and select 10 agent configurations from the first drawing. Then, the user can provide input, request a second drawing of evaluation metrics 3 and 4 and select 5 additional agent configurations from the second drawing to select a total of 15 agent configurations to be sampled in the next data collection iteration. When selecting individual configurations, they can be graphically distinguished from other configurations, allowing the user to determine which configurations have been selected by previous drawings. This allows the user to view and select different "slices" of agent configurations for further sampling based on the performance of the metrics of interest to the user.

[0080] The user may also be provided with an option to configure the sampling probability. For example, for the second sampling mode and the third sampling mode, the user may select uniform sampling or softmax-based sampling. For example, a user with relatively less technical knowledge may prefer a uniform sampling method, while a user with relatively more technical knowledge may prefer a softmax-based sampling method. In some cases, the user may also be provided with the ability to adjust the softmax-based sampling probability. Example Proxy Type

[0081] The disclosed implementations can be used to evaluate many different types of agents. Fig. 7A and 7B Examples of proxy components are shown for two specific types of proxies, as discussed in more detail below.

[0082] Fig. 7AComponents of a reinforcement learning agent 700 are shown, including a feature generator 710 and a reinforcement learning model 720. The feature generator uses feature definitions 712 to generate context features 714 from context information 104, action features 716 from action information 106, and reaction features 718 from reaction information 108. The context features represent the context of the environment in which the agent operates, the action features represent potential actions that the agent can take, and the reaction features represent how the environment reacts to the action selected by the agent. Therefore, the reaction information may be obtained later than the context information and the action information. The reinforcement learning model 720 uses internal parameters 722 to determine the selected action 114 from the context features 714 and the action features 716. The reward function 724 calculates a reward based on the reaction features. Hyperparameters 726 can be used to adjust the internal parameters of the reinforcement learning model based on the value of the reward function.

[0083] Figure 7B Components of a supervised learning agent 750 are shown, including a feature generator 760 and a supervised learning model 770. The feature generator generates context features 764 from context information 104 using feature definitions 762, and generates action features 766 from action information 106. The context features represent the context of the environment in which the agent is operating, and the action features represent potential actions that the agent can take. The supervised learning model 770 uses internal parameters 772 to determine the selected action 114 from the context features 764. During training, a loss function 774 can be applied to labels 776, where each label indicates a correct action. Hyperparameters 726 can be used to adjust the internal parameters of the supervised learning model based on the value of the loss function.

[0084] As mentioned previously, the disclosed implementations may also be used to evaluate other types of agents, such as unsupervised learning agents, rule-based agents, and the like. Technical Effects

[0085] The disclosed implementation can use an adaptive sampling method to predict the performance of different proxy configurations for different evaluation metrics. This allows a user to evaluate different proxy configurations based on the metrics that the user is interested in, without having to fully test every potential configuration. Instead, analysis can be performed on the adaptively sampled data to obtain sufficient test data for each potential proxy configuration. Furthermore, data obtained using one proxy configuration can be used to infer the performance of other proxy configurations, allowing for the reuse of test data.

[0086] Note that the data collection and data analysis aspects disclosed herein can be used collaboratively or independently. Considering the data collection aspect first, invoking a conventional A / B test would involve taking an equal number of samples for each agent configuration being evaluated. Now, consider a scenario where a third party has an automated evaluation system that uses conventional A / B test data to evaluate different agents. Each time the sampler assigns a given experimental unit to a given agent configuration, that agent configuration consumes computing resources, such as processor cycles, memory, storage, and / or network bandwidth.

[0087] By using the above-described adaptive sampling techniques, data samples with relatively few samples can be used with third-party evaluation systems to achieve comparable results. Therefore, fewer computing resources (e.g., processor cycles and bytes of storage) are used to obtain comparable evaluations of different agents. More specifically, by using the above-described second or third sampling modes to cultivate data samples, the resulting data samples can be used to make higher confidence predictions of the performance of various agent configurations compared to simple sampling methods in conventional A / B testing. Therefore, the disclosed adaptive sampling mechanism can use fewer processing or storage resources required in conventional A / B testing because conventional A / B test data provides less average prediction information for each sample. In other words, when a data structure such as performance prediction table 400 is populated using the disclosed adaptive sampling techniques, fewer computing resources are required to test alternative agent configurations than using traditional A / B testing.

[0088] Taking the data analysis aspect next, recall that the disclosed implementations can use data samples recorded by different agents to estimate the performance of an agent relative to an evaluation metric. Thus, individual samples taken by one agent can be "reused" to evaluate multiple other agents, even agents that were not necessarily used during the sampling period. Thus, rather than expending processor cycles and / or storage resources to fully sample each agent, the disclosed data analysis techniques can conserve these resources by making predictive inferences about one agent based on samples collected by another agent. In other words, when a data structure such as performance prediction table 400 is used to infer the performance of a given agent using events collected by different agents, fewer computational resources are required to test alternative agent configurations than when using traditional A / B testing.

[0089] In addition, recall that the data analysis techniques described herein can be implemented by replaying events in the event log with different agent configurations. Therefore, different agent configurations can be evaluated offline, and in some cases, there is no need to sample those different agent configurations. In fact, even new agent configurations that may not yet exist when the event log is created can still be analyzed using the disclosed techniques. In other words, data structures such as performance prediction table 400 allow offline evaluation of new agent configurations, even without collecting any samples through the new agent configuration. This is particularly useful for scenarios where users want to evaluate prototypes of agent configurations that may not be fully used in production environments. For example, users can quickly prototype different potential agent configurations so that they are sufficient to determine action probabilities. These rapid prototypes may have unresolved security, privacy, or performance issues that prevent them from being used in production code, but they can still be evaluated using events processed by other agents that can be used in production. Therefore, developers can focus their efforts on promising prototypes without having to fully develop less promising configurations, thereby saving the development work of testing / modifying those less promising prototypes for security, privacy, and performance reasons.

[0090] Additionally, note that new evaluation metrics can be defined after the event log is created. For example, assume that an event log is created using any of the three sampling modes described above, and then the user determines a new evaluation metric of interest. The events in the log can be replayed to generate performance predictions for the new evaluation metric, e.g., a new column in performance prediction table 400. The fact that the event log is not sampled using an evaluation metric does not prevent the agent from being evaluated according to the new evaluation metric. If the confidence intervals for the new evaluation metric are too large, some additional sampling in the third sampling mode using the new evaluation metric can be used to reduce those confidence intervals so that an appropriate agent configuration can be selected. In contrast, conventional A / B testing requires a completely new test of each agent configuration according to the new evaluation metric, which in turn requires the consumption of further processing and storage resources, which can be saved by using the disclosed data sampling and / or analysis techniques to populate and / or evaluate data structures (e.g., performance prediction table 400). Example System

[0091] This implementation can be executed in various scenarios on various devices. Figure 8 An example system 800 is shown in which the present implementations may be employed, as discussed in greater detail below.

[0092] like Figure 8As shown, system 800 includes client device 810, client device 820, server 830, and server 840 connected by one or more networks 850. Note that client devices can be embodied as mobile devices such as smart phones or tablets, as well as fixed devices such as desktops, server devices, etc. Similarly, servers can be implemented using various types of computing devices. In some cases, Figure 8 Any devices shown in , particularly servers, may be implemented in a data center, a server farm, or the like.

[0093] Figure 8 Certain components of the apparatus shown in FIG. 8 may be referred to herein by bracketed reference numerals. For purposes of the following description, brackets (1) indicate that a given component is present on client device 810, (2) indicates that a given component is present on client device 820, (3) indicates that a given component is present on server 830, and (4) indicates that a given component is present on server 840. Unless a specific instance of a given component is identified, this document will generally refer to the component without brackets.

[0094] Typically, devices 810, 820, 830, and / or 840 may have corresponding processing resources 801 and storage resources 802, which will be discussed in more detail below. Devices may also have various modules that use processing and storage resources to perform the techniques discussed herein. Storage resources may include persistent storage resources, such as magnetic or solid-state drives, and volatile memory, such as one or more random access memory devices. In some cases, modules are provided as executable instructions stored on a persistent storage device, loaded into a random access memory device, and read from the random access memory by a processing resource for execution.

[0095] The server 840 may include an agent 102, a data collection module 842, a data analysis module 844, a sampling adaptation module 846, and an agent deployment module 848. The data collection module may generate an event log by sampling different agent configurations, for example using the data collection workflow 200 described above. The data analysis module may process the event log using one or more alternative agent configurations to determine an evaluation metric prediction for each alternative agent configuration, for example using the data analysis workflow 300 described above. As described above with respect to any of the three sampling modes, the sampling adaptation module may adjust the sampling strategy used by the data collection module for the next data collection iteration. The agent deployment module may manually select one of the alternative agent configurations based on user input, or automatically select one of the alternative agent configurations based on the evaluation metric prediction, and deploy the agent using the selected agent configuration. One way in which the agent configuration module automatically selects an agent configuration is to randomly sample from the Pareto frontier of the agent configuration based on the predicted performance of one or more evaluation metrics.

[0096] In other cases, the agent deployment module 848 may output a graphical user interface, such as the one shown above in Fig. 6A , 6B 6C, the graphical user interface transmits information about each alternative proxy configuration to the client device 810. The client device 810 may include a configuration interface module 811 that displays a GUI to a user and receives input from the GUI to select a particular configuration. The client device may send a communication identifying the selected proxy configuration to the server 840, and the proxy deployment module 848 on the server 840 may deploy the proxy 102 according to the selected configuration.

[0097] The server 830 may have a server application 831 that may make API calls to the agent 102 on the server 840. For example, a user on a client device 820 may be using a client application 821 that interacts with a server application. The server application may send context information and / or action information to the agent 102 on the server 840 via an API call, reflecting the context on the client device 820 and potential actions that the server application may take. The agent may select a particular action, the server application may perform the selected action, and then the server application may inform the agent how the client device should react to the selected action. When the agent is a reinforcement learning agent, the agent may calculate its own reward and potentially update its policy based on the reaction. When the agent is a supervised learning agent, the agent may update its own internal parameters given a label provided by a human or automated entity, where the label may be based on the reaction. Example Method

[0098] Fig. 9 An example method 900 consistent with some implementations of the present concepts is shown. The method 900 can be implemented on many different types of devices by a client device such as a laptop, tablet, or smartphone, or through a combination of one or more servers, client devices, and the like.

[0099] Method 900 begins at block 902 where experimental units are assigned to agents having different agent configurations according to a sampling strategy.

[0100] Method 900 begins at block 904, where an event log is populated with events. The events in the event log represent the reaction of the environment to the various actions taken by the various agents in response to the various experimental units. The events may also represent the context in which these actions were taken and / or the probabilities assigned to those actions by the agents.

[0101] Method 900 continues at block 906, where the sampling strategy is adjusted for the next data collection iteration. As previously described, the sampling strategy can be adjusted based on events in the event log. Blocks 902, 904, and 906 can correspond to data collection iterations. Each subsequent data collection iteration can use a sampling strategy adjusted based on the data sampled in the previous data collection iteration.

[0102] The method 900 continues at block 908, where the performance of the alternative agent configuration is predicted for one or more evaluation metrics. The value of each event of a given evaluation metric can be determined for each event in the event log based on a function that maps environmental reactions (and potentially selected actions and / or contexts) to the value of the evaluation metric.

[0103] Method 900 continues at block 908 where a selected proxy configuration is identified based at least on the predicted performance.For example, the selected proxy configuration may be selected automatically or in response to user input from a GUI or other user interface identifying the selected proxy configuration.

[0104] Method 900 continues at block 912 where the agent is deployed according to the selected agent configuration.

[0105] Blocks 902 and 904 may be performed by the data collection module 842. Block 906 may be performed by the sampling adaptation module 846. Block 908 may be performed by the data analysis module 844. Blocks 910 and 912 may be performed by the agent deployment module 848. Use cases for electronic content distribution

[0106] The disclosed embodiments are generally applicable to a variety of practical problems that can be solved using automated proxies. A specific use case is given below, where a given entity wishes to select a proxy configuration for distributing electronic content.

[0107] For the purpose of this example, the server application 831 on the server 830 can be an application that presents an electronic content item to a user of the client device 820 by outputting an identifier of the electronic content item to the client application 821. The agent 102 can be an agent that receives an API call from the server application, wherein the API call identifies to the agent a plurality of different potential electronic content items and a context that reflects the environment in which the electronic content will be presented to the user. Each potential electronic content item is a potential action for the agent. The agent can select a specific content item that the application is to output to the user.

[0108] Suppose that a user who oversees a video game platform wants to encourage different video game players to interact more with each other. However, also assume that the user does not necessarily care which video games the players actually play, but only that they interact with the video game platform by commenting, liking, or otherwise interacting with other video game players. Now, consider a particular video game player who enjoys playing driving video games and has never played any sports video games.

[0109] Since the video game player has played driving games for many hours, the agent may be inclined to continue to prioritize driving video games for the video game player. Fig. 10A An electronic content GUI 1000 is shown in which various electronic content 1002(1)-1002(12) are displayed to the user. Here, based on a previous agent configuration, the agent selects driving game 1002(1) as the highest ranked content item. This can be a previous rule-based agent configuration that uses a heuristic weighting scheme with static weights to select games based on how long the user has previously played similar games, a previous supervised learning agent that uses labeled training data with positive labels to train the user on how long he or she has played driving video games, or a previous reinforcement learning agent that calculates its own reward based on how long the user has played its recommended game. Because the agent recommends the driving video game, the application outputs the driving game in the largest portion of the display area. Note that sports video game 1002(4) is also displayed, but takes up less screen area.

[0110] Now, assume that players of driving video games tend to rarely interact with the video game platform, e.g., they tend not to communicate with each other or "like" specific games or game scenes. Furthermore, assume that players of sports game games are more likely to interact with the video game platform. For example, this may be because sports games have structured breaks such as timeouts, halftime, etc., which allow users time to interact more with the platform. Therefore, when evaluating various agent configurations to obtain evaluation metrics related to engagement, those agent configurations that tend to recommend more sports game games will tend to increase engagement, while those agent configurations that recommend driving video games will have the opposite effect.

[0111] Fig. 10B The electronic content GUI 1000 is shown in an alternative configuration using an agent configuration selected according to the disclosed implementation. Now, the sports video game 1002(4) occupies the largest area of ​​the screen. This may be due to the new agent configuration having a different heuristic weighting scheme, a different loss function, or a different reward function, thereby encouraging the selection of sports video games and discouraging the playing of driving video games.

[0112] Note that in this example, the user specifying the new agent configuration does not need to specifically modify, evaluate, or understand the internal functionality of the underlying agent. For example, the user does not need to specify heuristic rules, supervised learning algorithms or loss functions, reinforcement learning models, or reward functions to encourage the agent to choose a sports game over a driving game. Instead, the user cares about engagement, not the type of game the user is playing or the underlying technical details of the agent.

[0113] By replaying the event logs of various agent configurations, the disclosed technology can discover agent configurations that have good performance to meet the metrics that the user is interested in without the user having to determine how the agent itself is configured. Thus, in this example, the user is able to select an agent configuration that encourages participation without even having to recognize that the sports game tends to encourage participation, let alone manually define an agent configuration that encourages the agent to select the sports game.

[0114] As another example, suppose the agent was previously configured with a reinforcement or supervised learning agent that used a very conservative learning rate hyperparameter. As a result, even if the player recently started playing a new video game, the agent may tend to continue recommending the same video games to the player that they have played in the past. In other words, the conservative learning rate hyperparameter causes the agent to react rather slowly to the player's changing preferences.

[0115] Now, suppose that a player who plays a video game initially interacts with the platform more frequently than a player who continues to play a previously played game. Because the agent slowly adapts, the previous agent configuration may inadvertently discourage player engagement, even if the previous agent configuration tends to make the player play a lot of video games. This is because the slow learning rate will hinder the agent from reacting when the player starts playing a new video game, thus making the user return to the video game they used to play frequently.

[0116] Now, suppose a user selects a new agent configuration with a very high predicted engagement value. The new agent configuration may have a much faster learning rate than the old agent configuration. Because of the faster learning rate, the agent can react quickly when a player starts playing a new game, for example by recommending a new game over an old game that the user has played before, even if the user has only played the new game once or twice. However, the user who selects the new agent configuration does not need to know this - the user only knows that the new agent configuration will tend to increase engagement.

[0117] As another example, suppose that the agent has been configured with a feature definition that only considers features related to video games that the player has played in the past. As a result, the agent may not consider other features of the player when recommending video games, such as other interests the player may have. On the other hand, video game players with common interests may interact with each other more frequently while playing video games, even if these interactions are not necessarily related to gameplay. For example, a group of players of an online basketball game may discover that they also have a common interest in politics and discuss politics while playing the basketball game.

[0118] Now, suppose that a user selects a new agent configuration with a very high predicted engagement value. The new agent configuration might have a feature definition that takes into account other interests of the video game player, such as whether the player is a member of an online social media group on a particular topic (e.g., politics), etc. Because this feature definition enables the agent to take into account user context that conveys the external interests of the video game player, the new agent configuration increases engagement compared to the previous configuration that did not take into account the external user interests. Likewise, the user who selects the new configuration does not necessarily need to care about which features the agent uses in the new configuration, only that the new configuration tends to increase engagement relative to the previous or other alternative agent configurations. Further contextual features for content distribution

[0119] A context vector that can be used for content distribution (e.g., video games or streaming media distribution) is a user vector that characterizes one or more features of a particular user. There are many different ways to describe or define a user as a set of features or signals. The features of a user may include fixed user features, such as a user identifier (e.g., a user game identifier), age, gender, location, sexual orientation, race, language, etc. The features of a user may also include dynamic user features, such as purchase propensity, type affinity, publisher affinity, ability affinity, social affinity, purchase history, interest history, wish list history, preferences, social media contacts or groups, and features of the user's social media context or group. There may be a lot of features or signals in a user vector. A feature generator may generate a user vector containing one or more user features for each user. Other context features may represent time of day, day of the week, month of the year, season, holiday, etc. In some implementations, user information may be maintained in a privacy-preserving manner.

[0120] Each available content item is a potential action for the agent. In other words, depending on the agent's current strategy, the agent may choose to recommend any specific content item given the current context of the environment. Therefore, in this case, the action feature can be represented as a content vector for each of multiple content (e.g., games, movies, music, etc.). The content information can be provided manually or obtained from a content database. There are many ways to describe or characterize content. The content vector can include multiple features of a specific content (e.g., a specific game), such as text about the content, metadata about the content, pricing information, toxicity, content rating, age group suitability, type, publisher, social, number of users, etc. The feature generator can generate metrics for various features related to the content, such as inclusive metrics, safety metrics, toxicity metrics, etc.

[0121] Each time the agent selects an action, such as outputting a content item to the user, the environment reacts. For example, the user can click on the content item, ignore the content item, etc. For example, the user reaction can include viewing the content, selecting the content, clicking on the content or any other item on the display screen, purchasing, downloading, spending money, spending points, commenting, sharing, hovering the pointer over the content, playing, socializing, failing to select any personalized content (e.g., within a predefined time period), minimizing, idle, exiting the platform (e.g., a game store), etc. Any of these user actions can be represented by a corresponding reaction feature.

[0122] Note that some embodiments may also consider various contextual features that characterize the client device used to consume electronic content. For example, the agent may be provided with contextual features that identify the processing unit of the client device, whether the client device has a particular type of hardware acceleration capability (e.g., a graphics processing unit), the amount of memory or storage, the display resolution, the operating system version, etc. In this case, the agent may learn that certain games or other executable files will not run or will run poorly on devices that lack certain technical features. For example, reviewing Fig. 10A , the agent may learn that the driving game 1002(1) does not run well on a device with less than a specified amount of RAM, and may learn to select a sports game 1002(4) instead.

[0123] The environmental reactions measured by the agent can be the result of the user intentionally ending the game within a short period of time, or explicit signals from the client device, such as measuring memory or CPU utilization. Thus, for example, the agent may learn that device type A (e.g., lacking a GPU but having a large amount of RAM) exhibits high CPU utilization when executing driving game 1002(1), device type B (e.g., having a GPU but lacking sufficient RAM) exhibits high memory utilization when executing driving game 1002(1), and device type C exhibits moderate memory and CPU utilization when executing driving game. Thus, the agent can learn to recommend sports video games 1002(4) to devices of types A and B, while recommending driving video games to devices of type C. Use case for video calling application

[0124] For purposes of this example, server application 831 may be an application that provides video calling functionality to a user of client application 821 on client device 820. Proxy 102 may be a proxy that receives an application programming interface (API) call from the application, wherein the API call identifies to the proxy a plurality of different technical configurations and a context reflecting the technical environment in which the video call will be made. Each technical configuration is a potential action for the proxy. The proxy may return the highest ranked configuration to the application.

[0125] One example of a potential technical configuration for a video call application is the playout buffer size. The playout buffer is an area of ​​memory that stores VOIP packets, and playback is delayed by the duration of the playout buffer. In general, the use of a playout buffer can improve sound quality by reducing the effects of network jitter. However, because filling the buffer is delayed while the sound is played, if the playout buffer is too large, the conversation will appear relatively interactive. However, a larger playout buffer means a longer delay from when the packet is received to when the audio / video data is played for the receiving user, which can result in noticeable delays in the conversation.

[0126] In general, any agent that encourages high sound quality without regard to interactivity will tend to prioritize large playback buffers. Fig.11A A video call GUI 1100 is shown with a high sound quality rating but a low interactivity rating, which reflects how a human user (e.g., client device 820) might perceive call quality using this configuration. This could be due to the rule-based agent specifying a large play buffer in all cases, due to the supervised learning agent being trained using labeled training data with labels that only characterize sound quality, or due to the reinforcement learning agent having configured a reward function that calculates a reward based solely on whether the play buffer becomes empty, e.g., playback needs to be paused while waiting for a new packet.

[0127] Now suppose a new agent configuration is chosen. The new agent configuration could be a rule-based agent that configures the play buffer size as a static mathematical function of a variable, such as average packet delay. The new agent configuration could be a supervised learning agent trained with training data with labels that characterize interactivity and sound quality. The new agent configuration could be a reinforcement learning agent with a different reward function that takes into account both whether the play buffer becomes empty and the duration of the call. Any of these agents might tend to choose a medium-sized play buffer that provides reasonable call quality and interactivity. Fig. 11B A video call GUI 1100 is shown with relatively high sound quality and interactivity ratings.

[0128] A user who is selecting an agent configuration for this scenario might consider evaluation metrics such as call quality or interactivity, as these are aspects of a call that are important to the end user. Such a user might not have a great deal of technical expertise and might have difficulty specifying an agent configuration to achieve this goal. However, by specifying their evaluation metrics in an intuitive way, users can implicitly select an agent configuration that successfully achieves a balance between interactivity and call quality.

[0129] With respect to feature definitions, one feature that the agent might consider is network jitter, e.g., the variation in packet reception times. Jitter can be measured over any time interval, e.g., the variation in packet arrival times can be calculated over a few packets or over a longer duration, such as an entire call. Consider a previous agent configuration that used a network jitter feature definition that was calculated over a large number of packets. If the network jitter changes suddenly, it may take a long time for the agent to identify the change and make corresponding changes to the size of the play buffer. A new agent configuration that uses jitter measurements calculated over a shorter period of time may result in better sound quality and interactivity. Again, the user does not need to explicitly configure the agent to use a specific jitter feature definition. Instead, the various feature definitions can be evaluated using the event log to determine how they will affect sound quality and interactivity, and the user can select a configuration that balances sound quality and interactivity based on their preferences. This implicitly allows the user to specify a feature definition for jitter without having to manually define such a feature.

[0130] Contextual features, action features, and reaction features for voice call applications can be different from features used for content personalization. For example, contextual features might represent the location and identity of the parties in a given call, whether some parties have muted their microphones or turned off their video, network jitter and latency, whether the user is using a high-fidelity audio device, whether a given user is sending multicast packets, etc. Action features can describe the size of the playback buffer and any other parameters that the agent might be able to take action on, such as VOIP packet size, codec parameters, etc. Reaction features might represent buffer overflow or underrun, quiet periods during a call, call duration, etc. In some cases, reaction features or labels for supervised learning can be obtained using automatic characterization of voice quality or interactivity, for example, Rix et al., "Perceptual Evaluation of Speech Quality (PESQ)-A New Method for Speech Quality Assessment of Telephone Networks and Codecs", 2001IEEE International Conference on Acoustics, Speech, and Signal Processing. Device Implementation

[0131] As mentioned above about Figure 8 As shown, system 800 includes several devices, including client device 810, client device 820, server 830, and server 840. It should also be noted that not all device implementations may be described, and other device implementations should be apparent to those skilled in the art from the above and following descriptions.

[0132] As used herein, the terms "device," "computer," "computing device," "client device," and / or "server device" may mean any type of device having a certain amount of hardware processing capability and / or hardware storage / memory capability. The processing capability may be provided by one or more hardware processors (e.g., hardware processing units / cores) that can execute computer-readable instructions to provide functionality. Computer-readable instructions and / or data may be stored on a storage device, such as a storage device / memory and / or a data storage device. As used herein, the term "system" may refer to a single device, multiple devices, etc.

[0133] The storage resource may be internal or external to the corresponding device associated therewith. The storage resource may include any one or more of volatile or non-volatile memory, a hard drive, a flash storage device, and / or an optical storage device (e.g., a CD, a DVD, etc.), etc. As used herein, the term "computer-readable medium" may include signals. In contrast, the term "computer-readable storage medium" excludes signals. Computer-readable storage media include "computer-readable storage devices". Examples of computer-readable storage devices include volatile storage media, such as RAM, and non-volatile storage media, such as hard drives, optical disks, and flash memory, etc.

[0134] In some cases, the device is configured with a general-purpose hardware processor and storage resources. In other cases, the device may include a system-on-chip (SOC) type design. In a SOC design implementation, the functions provided by the device can be integrated on a single SOC or multiple coupled SOCs. One or more associated processors may be configured to coordinate with shared resources (such as memory, storage devices, etc.) and / or one or more dedicated resources (such as hardware blocks configured to perform certain specific functions). Therefore, the terms "processor", "hardware processor" or "hardware processing unit" as used herein may also refer to a central processing unit (CPU), a graphics processing unit (GPU), a controller, a microcontroller, a processor core, or other types of processing devices suitable for implementation in both conventional computing architectures and SOC designs.

[0135] Alternatively or additionally, the functions described herein may be at least partially performed by one or more hardware logic components. For example, but not limited to, illustrative types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0136] In some configurations, any modules / code discussed herein may be implemented in software, hardware, and / or firmware. In any case, the modules / code may be provided during the manufacture of the device or by an intermediary who prepares the device for sale to an end user. In other cases, the end user may later install these modules / codes, such as by downloading an executable code and installing the executable code on a corresponding device.

[0137] Also note that a device can generally have input and / or output capabilities. For example, a computing device can have various input mechanisms, such as a keyboard, a mouse, a touchpad, voice recognition, gesture recognition (e.g., using a depth camera, such as a stereo or time-of-flight camera system, an infrared camera system, an RGB camera system, or using an accelerometer / gyroscope, facial recognition, etc.). A device can also have various output mechanisms, such as a printer, a monitor, etc.

[0138] It is also noted that the devices described herein can function independently or in a collaborative manner to implement the described techniques. For example, the methods and functions described herein can be executed on a single computing device and / or distributed across multiple computing devices communicating via a network 850. The network 850 may include one or more local area networks (LANs), wide area networks (WANs), the Internet, etc.

[0139] Various examples are described above. Additional examples are described below. One example includes a method comprising performing two or more data collection iterations, the method comprising: assigning experimental units to multiple agents having different agent configurations according to a sampling strategy; populating an event log with events representing reactions of the environment to actions taken by the respective agents in response to the respective experimental units; and adjusting the sampling strategy for subsequent data collection iterations based at least on the events in the event log; predicting performance of multiple agents relative to one or more evaluation metrics based at least on the events in the event log; and identifying a selected agent configuration based at least on the predicted performance of the multiple agents relative to the one or more evaluation metrics.

[0140] Another example may include any of the above and / or following examples, wherein the method further comprises deploying the selected agent with the selected agent configuration.

[0141] Another example can include any of the above and / or following examples, wherein the selected agent configuration is automatically selected or selected based on user input identifying the selected agent configuration from a graphical representation of predicted performance of a plurality of agents.

[0142] Another example may include any of the above and / or following examples, wherein the method further includes determining importance weights for each agent based at least on the respective probabilities assigned to actions by each agent relative to the probabilities of actions taken by other agents stored in the event log, and adjusting the sampling strategy based at least on the importance weights.

[0143] Another example may include any of the above and / or following examples, wherein the method further comprises calculating corresponding sampling probabilities for respective agents based at least on the importance weights.

[0144] Another example may include any of the above and / or following examples, wherein adjusting the sampling strategy includes removing at least one agent from a subsequent data collection iteration based at least on the importance weight.

[0145] Another example may include any of the above and / or following examples, wherein a sampling strategy is adjusted at each data collection iteration based at least on a predicted performance of the plurality of agents with respect to one or more evaluation metrics.

[0146] Another example may include any of the above and / or following examples, wherein adjusting the sampling strategy includes determining corresponding confidence intervals for one or more evaluation metrics for each of a plurality of agents, and calculating a sampling probability for each agent based at least on an upper limit of the confidence interval.

[0147] Another example may include any of the above and / or following examples, wherein adjusting the sampling strategy includes removing at least one agent from further sampling based at least on the predicted performance.

[0148] Another example may include any of the above and / or following examples, wherein the method further includes populating a data structure with aggregate values ​​and corresponding confidence intervals for one or more evaluation metrics, outputting a graphical representation of the data structure, and identifying one or more agent configurations to be sampled in a subsequent data collection iteration based at least on user input to the graphical representation of the data structure.

[0149] Another example may include any of the above and / or following examples, wherein the method further comprises receiving user input specifying two or more evaluation metrics, and generating the graphical representation based at least on the two or more evaluation metrics specified by the user input.

[0150] Another example may include any of the above and / or following examples, wherein the method further comprises using events in the event log to predict performance of at least one other agent relative to one or more evaluation metrics, wherein the at least one other agent is not sampled when populating the event log.

[0151] Another example includes a system comprising a processor and a storage resource storing instructions that, when executed by the processor, cause the system to perform two or more data collection iterations, the two or more data collection iterations comprising assigning experimental units to multiple agents having different agent configurations according to a sampling strategy, populating an event log with events representing reactions of the environment to actions taken by the respective agents in response to the respective experimental units, and adjusting the sampling strategy for subsequent data collection iterations based at least on the events in the event log, wherein the event log provides a basis for subsequent evaluation of the multiple agents relative to one or more evaluation metrics.

[0152] Another example may include any of the above and / or following examples, wherein the respective agents include machine learning agents with different hyperparameters or different feature definitions.

[0153] Another example may include any of the above and / or following examples, wherein the agents include at least two different reinforcement learning agents with different reward functions, at least two different supervised learning agents with different loss functions, and at least two different rule-based agents with different rules.

[0154] Another example may include any of the above and / or following examples, wherein the sampling strategy is based at least on corresponding importance weights of the respective agents.

[0155] Another example may include any of the above and / or following examples, wherein the sampling strategy is adjusted based at least on predicted performance of the plurality of agents with respect to one or more evaluation metrics.

[0156] Another example may include any of the above and / or following examples, wherein adjusting the sampling strategy includes assigning corresponding probabilities to the respective agents, and randomly assigning the experimental units to the respective agents based on the corresponding probabilities.

[0157] Another example includes a computer-readable storage medium storing instructions that, when executed by a computing device, cause the computing device to perform actions including: obtaining an event log of events representing reactions of the environment to actions taken by multiple agents in response to respective experimental units; predicting performance of respective agents relative to one or more evaluation metrics based at least on respective events in the event log reflecting respective actions taken by other agents; identifying a selected agent configuration based at least on the predicted performance of the respective agents relative to the one or more evaluation metrics; and deploying the selected agent having the selected agent configuration.

[0158] Another example may include any of the above and / or following examples, wherein events of the event log are pre-sampled using an adaptive sampling strategy that adjusts sampling probabilities of various agents based on the collected events.

[0159] Another example may include any of the above and / or following examples, wherein the selected agent, when deployed in the selected agent configuration, receives an application programming interface call from the application and selects a technical configuration for the application in response to the application programming interface call.

[0160] Another example may include any of the above and / or following examples, wherein the application is a voice or video call application, and the technology configuration indicates a buffer size of a playout buffer of the application.

[0161] Another example may include any of the above and / or following examples, wherein the selected agent is a reinforcement learning agent and the selected agent configuration includes a selected reward function that takes into account whether the play buffer becomes empty and the corresponding duration of the voice or video call. in conclusion

[0162] Although the subject matter has been described in language directed to structural features and / or methodological acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Instead, the specific features and acts described above are disclosed as example forms of implementing the claims, and other features and acts recognized by those skilled in the art are intended to fall within the scope of the claims.

Claims

1. A method comprising: Perform two or more data collection iterations, including: assigning experimental units to a plurality of agents having different agent configurations, the experimental units being assigned according to a sampling strategy; populating an event log with events representing the environment's reactions to actions taken by the respective agents in response to the respective experimental units; and adjusting the sampling strategy for a subsequent data collection iteration based at least on the event in the event log; predicting performance of the plurality of agents relative to one or more evaluation metrics based at least on the events in the event log; and A selected agent configuration is identified based at least on the predicted performance of the plurality of agents relative to the one or more evaluation metrics.

2. The method according to claim 1, further comprising: The selected agent is deployed with the selected agent configuration.

3. The method of claim 2, wherein when the selected agent is deployed in the selected agent configuration, the selected agent receives an application programming interface call from an application and selects a technology configuration for the application in response to the application programming interface call.

4. The method of claim 3, wherein the application is a voice or video call application and the technology configuration indicates a buffer size for a playout buffer of the application.

5. The method of claim 1, wherein the selected agent configuration is automatically selected or is selected based on user input identifying the selected agent configuration from a graphical representation of the predicted performance of the plurality of agents.

6. The method according to claim 1, further comprising: determining an importance weight for each agent based at least on a respective probability assigned to the action by each agent relative to probabilities of the action being taken by other agents, the probabilities of the action being taken by other agents being stored in the event log; as well as A corresponding sampling probability for each agent is calculated based at least on the importance weight.

7. The method according to claim 6, wherein adjusting the sampling strategy comprises: At least one agent is removed from a subsequent data collection iteration based at least on the importance weight.

8. The method of claim 1 , wherein the sampling strategy is adjusted at each data collection iteration based at least on the predicted performance of the plurality of agents relative to the one or more evaluation metrics, and adjusting the sampling strategy comprises: determining respective confidence intervals for the one or more evaluation metrics for each agent of the plurality of agents; as well as The sampling probability of each agent is calculated based on at least the upper limit of the confidence interval.

9. The method of claim 8, wherein adjusting the sampling strategy comprises: At least one agent is removed from further sampling based at least on the predicted performance.

10. The method according to claim 8, further comprising: populating a data structure with predicted aggregate values ​​and corresponding confidence intervals for the one or more evaluation metrics; outputting a graphical representation of the data structure; as well as One or more agent configurations to be sampled in a subsequent data collection iteration are identified based at least on a user input to the graphical representation of the data structure.

11. The method according to claim 1, further comprising: Using the events in the event log, predicting the performance of at least one other agent relative to the one or more evaluation metrics, wherein the at least one other agent is not sampled when the event log is populated.

12. A system comprising: processor; as well as A storage resource storing instructions, wherein when the instructions are executed by the processor, the system: Perform two or more data collection iterations, including: assigning experimental units to a plurality of agents having different agent configurations, the experimental units being assigned according to a sampling strategy; populating an event log with events representing the environment's reactions to actions taken by the respective agents in response to the respective experimental units; and adjusting the sampling strategy for a subsequent data collection iteration based at least on the event in the event log, Wherein the event log provides a basis for subsequent evaluation of the plurality of agents relative to one or more evaluation metrics.

13. The system of claim 12, wherein the sampling strategy is based at least on respective importance weights of the respective agents.

14. The system of claim 12, wherein the sampling strategy is adjusted based at least on the predicted performance of the plurality of agents relative to the one or more evaluation metrics.

15. The system of claim 12, wherein adjusting the sampling strategy comprises assigning corresponding probabilities to respective agents, and randomly assigning the experimental units to the respective agents based on the corresponding probabilities.

Images