gated linear contextual game machine

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for selecting an action in response to each context in a sequence of contexts. One of these methods includes maintaining data specifying a respective gated linear network corresponding to each of a plurality of actions; for each context in a sequence of contexts: for each action, processing the context using the gated linear network corresponding to the action to generate a predicted probability; for each action, generating an action score for the action based on at least the predicted probability; and selecting an action to perform in response to the context based on the action scores.

Description

Technical Field

[0001] This specification relates to selecting an action in response to contextual input. Background Technology

[0002] In a contextual bandits scenario, an agent iteratively selects an action to perform from a set of possible actions. At each iteration, the agent receives contextual input associated with that iteration and then selects an action for that iteration based on that contextual input. Summary of the Invention

[0003] This specification describes a system implemented as a computer program on one or more computers at one or more locations, which selects an action to be performed in response to received context input.

[0004] Specific embodiments of the subject matter described in this specification may be implemented in order to achieve one or more of the following advantages.

[0005] The described system uses a gated linear network in a context-based game machine setting to select actions in response to contextual input. This action selection scheme will be referred to as a gated linear context game machine. Using a gated linear network to select actions results in more accurate action selections in terms of received rewards, while reducing the amount of computational resources required to generate action selections. This can be attributed to several features of the described scheme. As an example, the described scheme allows the system to estimate prediction uncertainty efficiently with zero algorithmic overhead by leveraging the data-related gating properties of GLNs, thereby allowing for the computation of more accurate pseudo-counts without increasing computational overhead and resulting in a more efficient exploration of the space of possible actions. As another example, the system can compute an action score for an action and update the weights of the gated linear network for that action in a single forward propagation, eliminating the computationally intensive backpropagation required to update the model weights of a conventional system that uses traditional deep neural networks to generate action scores. Because the gated linear network can be updated entirely online, the system does not need to store historical data other than a small amount of signature data used to compute pseudo-counts, thus significantly reducing the system's memory footprint compared to other techniques that use neural networks to select actions.

[0006] Details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the following description, drawings, and claims. Attached Figure Description

[0007] Figure 1A An example context game machine system is shown.

[0008] Figure 1B An example of a gated linear network (GLN) is shown.

[0009] Figure 2 This is a flowchart of an example process for selecting an action in response to contextual input.

[0010] Figure 3 This is a flowchart of another example process for selecting an action in response to contextual input.

[0011] The same reference numerals and labels in each of the accompanying figures indicate the same elements. Detailed Implementation

[0012] This specification generally describes a system that repeatedly selects an action to be performed in response to received context input.

[0013] Each action is selected from a predefined set of actions, and the system selects actions in an attempt to maximize the reward received in response to the selected actions.

[0014] Typically, reward is a numerical measure of the quality of the selected action. In some implementations, the reward for each action is zero or one, while in others, each reward is a value derived from a continuous range between, for example, a lower reward value and an upper reward value.

[0015] In some cases, the action is the recommendation of content items (such as videos, advertisements, images, search results, or other content fragments), and the contextual input represents a feature vector characterizing the current recommendation setting—data describing the environment in which the content item will be recommended, such as the current time, attributes of the user's device from which the recommendation will be displayed, attributes of previous content items already recommended to the user and the user's responses to those previous content items, and attributes of the setting in which the content item will be placed. In these cases, the reward value measures the quality of the recommendation. For example, the value could be one if the user interacts with the recommendation, and zero if the user does not interact with the recommendation. As another example, the reward value could be a value that measures the extent to which the user will engage with future content items recommended to the user after the current recommendation was made.

[0016] In some other cases, the action is a user interface element that can be presented to the user in the user interface, i.e., within a given presentation setting, and the context input represents a feature vector characterizing the given presentation setting, the user, or both. In these cases, the reward value measures the quality of the presented user interface element. For example, the value could be one if the user interacts with the element, and zero if the user does not interact. As another example, the reward value could be dwell time or other metrics that measure the degree of user interaction with the user interface element.

[0017] The above example describes a scenario where the reward is based on user feedback after an action has been selected. However, in some other cases, the reward is generated automatically, for example, as the output of a rating function that rates the selected action. The rating function could be, for example, a function that rates the sensitivity or accuracy of the selected action. As another example, the rating function could be a neural network that receives data characterizing the environment after an action has been selected and generates a reward for that action. Using these kinds of rewards allows the described technique to be used in many situations without a human user, such as for input classification tasks, industrial control tasks, or other tasks described below with reference to Figure 1.

[0018] To select an action, the system maintains data for one or more gated linear networks that correspond to each of a plurality of actions. When the reward is zero or one, a single gated linear network corresponds to each of the plurality of actions and is configured to predict the probability of receiving a reward if the corresponding action is performed in response to the input context, i.e., the probability that the reward will be one instead of zero.

[0019] The reward can be, for example, a continuous value derived from a range. In one approach, a gated linear network tree corresponds to each of a plurality of actions, and the range is divided into intervals. The gated linear network trees corresponding to a given action are collectively configured to predict a corresponding probability for each interval, representing the likelihood that the received reward will fall into the corresponding interval if the corresponding action is performed in response to the input context.

[0020] In order to select an action for a given context input, for each action, the system uses one or more gated linear networks corresponding to that action to process the context input to generate (i) a predicted probability of receiving a reward of 1 or (ii) a corresponding probability for each interval.

[0021] The system then uses the output of the gated linear network to select the action to be performed by generating an action score for the action based on the output of the action-gated linear network and, in some cases, a pseudo-count. The pseudo-count is also typically determined based on one or more gated linear networks corresponding to the action, as described below.

[0022] The system then selects the action to be performed based on the action score in response to the context, for example by selecting the action with the highest action score.

[0023] Figure 1 illustrates an example context game machine system 100. The context game machine system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations, in which the systems, components, and techniques described below are implemented.

[0024] System 100 includes an action selection system 110 and a training engine 150.

[0025] In response to the received context input 120, system 100 repeatedly selects an action 106 in environment 104, for example, performed by system 100 or by another system. For example, as described above, the action could be a content item recommendation to be made to the user in the environment, i.e., in settings for content item recommendation, such as on a webpage or in a software application.

[0026] The execution of each selected action 106 generally causes the environment 104 to transition to a new state and enables the system 100 to receive a reward 124 from the environment 104.

[0027] Typically, the reward 124 is a numerical value representing the quality of the selected action 106. In some implementations, the reward 124 for each action 106 is zero or one, indicating whether the action was successful, while in other implementations, the reward 124 is a value derived from a continuous range between a lower reward value and an upper reward value, that is, the quality of action 106 is represented as a value from a continuous range rather than a binary value.

[0028] Specifically, the action selection system 110 uses a set of gated linear networks (GLNs) 130 to select actions in an attempt to maximize the reward received in response to the selected actions. This set of gated linear networks includes one or more GLNs 130 corresponding to each of a plurality of actions that can be performed in response to context input 120. For example, the system 110 can select actions to maximize cumulative reward, i.e., the sum of rewards received at a certain level or the sum of time-discounted rewards received at that level.

[0029] Each GLN 130 is configured to predict the corresponding probability given an input context 120.

[0030] When the reward 124 is zero or one, a single GLN 130 corresponds to each of the multiple actions and is configured to predict the probability of receiving a reward, that is, the probability that the reward will be one instead of zero if the corresponding action is performed in response to the input context 120.

[0031] When the reward 124 is a continuous value obtained from a range between the minimum and maximum values, in one approach, the tree of GLN 130 corresponds to each of multiple actions, and the range is divided into intervals. The trees of the GLN corresponding to a given action are collectively configured to predict the corresponding probability for each interval, which represents the likelihood that the received reward will fall into the corresponding interval if the corresponding action is performed in response to the input context 120. See below for reference. Figure 3The generation of these probabilities is described in more detail. In another approach, when the reward 124 is a continuous value, regardless of whether it is within a certain range, the GLN can output parameters such as the mean and optional variance for each action, which are defined to represent a Gaussian distribution of the probability.

[0032] In some implementations, the context may correspond to an observation of the environment 104 at a time step; for example, it may include a feature vector representing the environment at the time step. The action selection system 110 may select an action to be performed at that time step in response to the observation.

[0033] In some implementations, the environment can be a real-world environment or a simulation of a real-world environment. The context can include, for example, a set of features encoded as feature vectors, representing observations of the environment, such as images or other sensor data. The action can correspond to an action to be performed in the environment to carry out a task.

[0034] In one example, the environment may include a manufacturing plant, or a service facility such as a data center, server farm, or mains power or water distribution system, or a power generation facility such as a solar or wind farm. Observations may include data from one or more sensors that monitor the operation of electronic and / or mechanical items of equipment in the plant or facility, such as current, voltage, power, temperature, or weather observations such as wind speed or solar irradiance or other characteristics. These actions may control operating conditions or impose operating conditions on equipment items, such as adjusting equipment item settings or turning equipment items on or off or adjusting the alignment of wind turbines or solar collectors. The returns may depend on the operation of the plant or facility, such as water or electricity use or generation, measurement of environmental impacts, or temperature control or equipment items.

[0035] In another example, the environment could be a packet data communication network environment. Observations could include routing metrics, such as any of the metrics derived from observations of the routing table, such as route path length, bandwidth, load, hop count, path cost, delay, maximum transmission unit (MTU), and reliability. Actions could include data packet routing actions, such as one or more steps defining the route of a data packet, or resource allocation actions, such as allocating a channel to one or more data packets. The reward (or equivalent cost) could depend on one or more of the routing metrics, such as maximizing throughput or minimizing latency.

[0036] In another example, the environment can be a computing environment. Observations can include observations of available computing resources, such as computing and / or memory capacity or internet-accessible resources. Actions can include assigning computing tasks to specific computing resources. Rewards can depend on one or more of, for example, the utilization of computing resources, power, bandwidth, and computing speed.

[0037] In another example, the environment could be a circuit routing environment, such as interconnects used to route integrated circuits like ASICs. Observations could include observations of component locations and interconnects; actions could include component placement actions, such as defining component locations or orientations, and / or interconnect routing actions, such as interconnect selection and / or placement actions. The results could depend on one or more collective routing metrics for the circuit, such as those derived from interconnect length, resistance, capacitance, impedance, loss, propagation delay, or electromagnetic emissions.

[0038] In some implementations, the context-based game machine system 100 can be used to perform classification tasks. The context can then include a set of features representing the items used for classification, such as feature vectors; actions selected from a set of possible actions can correspond to the classification of the items; and the reward can be a reward for correct classification, such as a binary reward depending on whether the classification is correct, or a continuous value reward corresponding to a rating of the classification. The system can be trained to perform any classification task. For example, features can represent observations of the state of items in equipment, plants, or facilities as described above, and the classification can represent performance measurements of equipment, plants, or facilities, such as measurements of operational efficiency, or classifications of potential faults.

[0039] Typically, a GLN consists of multiple layers, each containing one or more neurons. Each neuron within a GLN generates a prediction, or probability, of the target for the entire GLN as its output. That is, each neuron within a GLN predicts the same target, i.e., makes predictions for the same number of targets, and the final output of the GLN is the prediction generated by the neurons in the last layer.

[0040] More specifically, each neuron in each layer of the GLN receives a context 120 and a set of predictions, i.e., a set of probabilities, and outputs a mixture, specifically a geometric mixture, of the received predictions using the neuron's weight set and according to the context 120. This mixture can be thought of as a weighted combination of predictions. This mixture can be called geometric because it can be thought of as a parameterized or weighted form of the geometric mean (which is reduced to the geometric mean with equal weights).

[0041] This is in contrast to traditional neural networks, where the input to the neural network is only provided to the first layer, and neurons in each layer other than the output layer generate alternative representations for the next layer.

[0042] Specifically, within GLN, each neuron in layers following the first layer is configured to receive (i) an input context 120 and (ii) predictions from neurons in previous layers, and (iii) apply a gating function to the input context to select a weight vector, i.e., an appropriate subset of the neuron’s weights, and (iv) generate a new probability as an output, which is a geometric mixture of the predictions from neurons in previous layers, based on the selected weight vector.

[0043] Each neuron in the first layer is configured to receive (i) an input context 120 and (ii) a set of basic predictions, and (iii) apply a gating function to the input context to select a weight vector, and (iv) generate a geometric mixture of the basic predictions as output based on the selected weight vector.

[0044] The operation of GLN will be referenced below. Figure 1B To describe in more detail.

[0045] In order to select an action for a given context input 120, for each action in the action set, the system 110 uses one or more gated linear networks 130 corresponding to that action to process the context input 120 to generate (i) a predicted probability of receiving a reward or (ii) a corresponding probability for each interval.

[0046] System 100 then uses the output of the gated linear network to select the action to be performed by generating an action score 122 for the action based on the output of the gated linear network for the action, and in some cases, generating a pseudo-count. The pseudo-count is also typically determined based on one or more gated linear networks corresponding to the action, as described below.

[0047] System 100 then responds to context 120 based on action score 122 to select action 106 to be performed, for example by selecting the action with the highest action score 122.

[0048] The action selection will be referenced below. Figure 2 and 3 To describe in more detail.

[0049] To improve the quality of the selected action, the training engine 150 repeatedly updates the GLN 130 so that the action selection system 110 generates a more accurate policy output, that is, this produces a higher reward 124 received by the system 100 in response to the selected action.

[0050] In particular, due to the architecture of GLN, the training engine 150 can train GLN online in response to each received reward 124, without having to perform any backpropagation required for training traditional deep neural networks.

[0051] More specifically, when reward 124 is received, training engine 150 can use per-neuron loss to update the neuron in the GLN corresponding to the selected action (i.e., the action that produces reward 124 being received). In other words, engine 150 can locally update each neuron in a given GLN based on neuron-specific loss. The update for any given neuron depends only on reward 142 and information computed as part of the action score that generates the corresponding context, i.e., information computed during a single forward propagation GLN. In particular, the update can depend on the reward, the probability predicted by the neuron, and the probability received through the neuron as input. Therefore, training engine 150 can update neurons without requiring any additional forward propagation GLN or any backward propagation GLN.

[0052] This contrasts with traditional deep neural networks, which, in addition to one or more forward propagations, require computationally expensive backpropagation to update the weights of any given neuron within the deep neural network.

[0053] The following will refer to Figure 2 and Figure 3 A more detailed description of the updated GLN 130.

[0054] By repeatedly updating GLN 130, training engine 150 can continue to improve the quality of the actions selected by system 110.

[0055] Figure 1B An example of one of the GLN 130 is shown.

[0056] like Figure 1B As shown, GLN 130 receives edge information z, i.e., context 120, and generates probabilities as output. The probability representation will depend on the implementation. For example, when the reward is binary, the probability representation is the likelihood that the reward will be received if the action corresponding to the GLN is performed. When the reward is continuous, in some implementations, the probability representation is the likelihood that the reward will fall into one of two intervals if the action corresponding to the GLN is performed. This will be referred to below. Figure 3 A more detailed description would follow. However, the probability typically defined by the output of GLN 130 can define any exponential family probability distribution (which is closed under multiplication), such as a binomial distribution representing multi-bit binary values, or a Gaussian distribution characterized by mean and variance to represent continuous-valued data.

[0057] exist Figure 1B In the example, GLN has an input layer 0 and three additional layers 1, 2 and 3. The input layer 0 has K0 neurons, the layer 1 has K1 neurons, the layer 2 has K2 neurons, and the output layer 3 has a single neuron.

[0058] The output generated by a single neuron in the output layer is the probability p. 31 It is the predicted probability of GLN.

[0059] In an alternative implementation, instead of an output layer with a single neuron that generates a prediction probability that will be used as the final output of the GLN, the output can be provided by the output of neurons in an aggregated layer (e.g., layer 2 in this example).

[0060] The neurons in layer 0 serve as the “base model” for the remainder of the GLN and receive basic probabilities as input based on side information (i.e., context 120) or data derived from the side information (rescaled if necessary).

[0061] Specifically, because each neuron in the GLN computes a geometric mixture of probabilities given as inputs to the neuron, the inputs to neurons in layer 0 must be a set of probabilities, i.e., a set of values ​​ranging from zero to one (inclusive). Neurons in layer 0 can directly receive context inputs if the context inputs satisfy this criterion. Alternatively, system 100 or another system can use a function that maps context 120 to a set of probabilities, such as a fixed scaling function, a linear projection, or some learned nonlinear transformation.

[0062] Neurons in each of layers 1, 2, and 3 receive context 120 and probabilities generated by neurons in the previous layer of the GLN as input, and compute new probabilities by calculating a geometric mixture of the probabilities generated by the previous layer. As a specific example, neuron 1,1 in layer 1 receives probabilities generated by neurons in layer 0 as input and generates a new probability p. 11 As output.

[0063] In particular, each neuron in GLN has a set of weights that are represented as multiple weight vectors.

[0064] To generate the output probability, a given neuron applies a gating function to the context input 120 to select one of multiple weight vectors, and then uses the selected weight vector to perform a geometric mixture of the received probabilities to generate a new probability. Specifically, the process performed by neuron ij in the GLN to generate probability p... ij The operation that is output can be represented as:

[0065]

[0066] Where σ is the sigmoid function, and logit is the logit function, which is the inverse of the sigmoid function (meaning the entire neuron is linear). It is achieved by using the gating function g for neurons ij ij(z) is the weight vector chosen for the context z, and p i-1 It is the probability received as input from layer i-1.

[0067] In some implementations, to improve the stability of GLN training, the system prunes the probability values ​​so that the probabilities fall within a specific range, for example, for a constant ε between zero and one, within the range of ε to 1-ε (inclusive). In these cases, if any probability is below ε, the system sets the probability to ε. If any probability is above 1-ε, the system sets the value to 1-ε.

[0068] The gating function of each neuron is fixed before training, while the weight vector is learned during training. See below for reference. Figure 2 and 3 A more detailed description of updating the weight vector.

[0069] Neurons in GLN can use any appropriate gating function that maps context inputs to one of the weight vectors maintained by the neuron.

[0070] As a specific example, the gating function for each neuron in GLN can be a half-space gating function.

[0071] The half-space gating function is described in more detail in Veness et al.’s Gated Linear Networks arXiv:1910.01526.

[0072] exist Figure 1B In the example, each layer except the output layer (layer 3) includes bias neurons, namely neurons 00, 10, and 20, which generate outputs with fixed probabilities (i.e., probabilities set before training the GLN). However, this is optional, and in some cases, some or all layers of the GLN will not include bias neurons.

[0073] Figure 2 This is a flowchart of an example process 200 for selecting an action to be performed in response to context input. For convenience, process 200 will be described as being executed by a system of one or more computers located at one or more locations. For example, a properly programmed context game machine system (such as context game machine system 100 of Figure 1) can execute process 200.

[0074] Specifically, when the reward is a binary value and each GLN corresponds to a different action from a set of actions that can be performed in response to context input, the system can execute process 200.

[0075] That is, the system maintains data specifying the corresponding gated linear network for each of the multiple actions (step 202). The gated linear network corresponding to any given action is configured to predict the probability of receiving a reward if the corresponding given action is performed in response to the input context, i.e., the reward received will be equal to 1 rather than zero.

[0076] Then, the system executes steps 202 and 208 (and optionally 210 and 212) in response to each received context input. That is, the system can execute steps 202-208 or steps 202-212 in response to each context input in the context input sequence to select the action to be performed in response to each context input.

[0077] For each action, the system uses a gated linear network corresponding to that action to process the context input in order to generate a predicted probability for that action (step 204).

[0078] For each action, the system generates an action score based at least on the predicted probability for that action (step 206). That is, in some implementations, the predicted probability is used as the action score. In some other implementations, the system generates the action score based on the predicted probability for the action and one or more other factors.

[0079] As a specific example, the system can compute a pseudo-count for an action and then generate an action score for that action based on the predicted probability and the pseudo-count. Typically, a pseudo-count is a generalized measure of the number of times an action has been selected within a historical time window (e.g., a window of time steps in a sequence ending at the current time step). Calculating action scores based on both the pseudo-count and the predicted probability encourages exploration of the possible space of context-action pairs and leads to higher-quality action selection strategies.

[0080] As a specific example, when using pseudo-counting, the action score of action a at time step t in the context input sequence can satisfy:

[0081]

[0082] in, It is determined by GLN for action a in a given context x t The predicted probability generated under the given conditions, where C is a normal value for scaling the exploration bonus, and It is a pseudo-count.

[0083] The system can compute the pseudo-count for an action in any of a variety of ways. As a particular example, the system can compute the pseudo-count based on the overlap between (i) the signature of the context of the gating function of the neurons in the action-gated linear network and (ii) the signature of an earlier context in the sequence for which the action was selected to be performed in response to the earlier context.

[0084] As used herein, a context signature refers to the output of the gating function of a neuron in the GLN, generated by processing the context. For example, a signature for a given context for a given action could be a vector including the output of the gating function, i.e., a vector identifying the weight vector selected using the gating function, for each neuron in the GLN corresponding to a given action. In an implementation, the same set of gating functions is used for each action-specific GLN, and the signature can be reused when evaluating each GLN.

[0085] Therefore, the overlap between a signature corresponding to one time step and another signature corresponding to another time step identifies which gating functions generate the same output, that is, identifies the same weight vector for both the first time step and the other time step.

[0086] As a specific example, the system can determine, for a given action and for each neuron in the GLN for that action, how many earlier time steps in the window satisfy the following conditions: (1) the given action is selected at an earlier time step, and (2) the output of the gating function of the neuron for the context at an earlier time step is the same as the output of the gating function of the neuron for the current time step.

[0087] The system can then determine the pseudo-count by aggregating the corresponding counts of neurons, for example, by setting the pseudo-count to be equal to the average, minimum, maximum, or median of the corresponding count.

[0088] The system selects the action to be executed based on the action score of the actions in the action set in response to the context (step 208). For example, the system can select the action with the highest action score, or it can map the action scores to a probability distribution and then sample the actions according to the probability distribution.

[0089] Optionally, the system can then update the weights of the GLN neurons for the selected action by performing steps 210-212. Specifically, in some cases, the system updates the GLN weights after each context in the sequence (fully online). In other cases, the system may update the GLN weights only after every N contexts in the sequence, or it may freeze the weights after a threshold number of weight updates have been performed.

[0090] The system receives a reward as a result of performing the selected action in response to the context (step 210). As described above, when the system uses process 200 to select an action, the reward is typically a binary reward equal to one or zero.

[0091] The system updates the GLN for the selected action based on the received reward (step 212). More specifically, the system locally updates each neuron in the gated linear network based on neuron-specific losses, that is, updates the weights of each neuron in the gated linear network. Therefore, the system does not need to perform computationally expensive backpropagation to compute the loss.

[0092] Given the input probabilities to a neuron, the loss of a given neuron typically measures the error between the reward and the probability predicted by that given neuron. The system updates the weights of the neuron by computing the gradient of the loss and updating the chosen weight vector.

[0093] Specifically, the gradient of the loss for a given neuron j in layer i can satisfy:

[0094] -n(p ij -r)logit(p i-1 )),

[0095] Where n is the learning rate used to update GLN, logit is the logit function, i.e., the inverse of the sigmoid function, and p ij p is the probability predicted by neuron j in layer i, r is the received reward, and p i-1 It is the probability received as input from layer i-1. Therefore, as can be seen from this.

[0096] The system can then update the selected weight vector by adding or subtracting gradients. In this implementation, to improve the stability of the update, the system prunes the weight values ​​so that they fall within a specific range, for example, the range from -b to b(inclusive) for a constant b between 10 and 100. In these cases, if any value in the new weight vector is below the specific range, the system sets that value to the lowest value within that range. If any value in the new weight vector is above the specific range, the system sets that value to the highest value within that range.

[0097] Figure 3 This is a flowchart of another example process 300 for selecting an action to be performed in response to context input. For convenience, process 300 will be described as being performed by a system of one or more computers located at one or more locations. For example, a properly programmed context game machine system (such as context game machine system 100 of Figure 1) can perform process 300.

[0098] Specifically, when the returns are continuous values ​​from a fixed range and there is a corresponding GLN tree for each action, the system can execute process 300.

[0099] That is, the system maintains data specifying the corresponding tree of the gated linear network corresponding to each of the multiple actions (step 302). The gated linear network in the tree corresponding to any given action is configured to predict the corresponding probability for each of the multiple intervals of the range of reward values, given the input context.

[0100] In other words, the system discretizes a fixed range into multiple non-overlapping intervals, and the gated linear network trees in the tree corresponding to any given action jointly predict the corresponding probability of each interval.

[0101] The probability corresponding to each interval represents the likelihood that a reward falling into the interval will be received if the corresponding action is performed in response to the input context.

[0102] More specifically, for each action, when the number of intervals equals 2^D, the system can maintain data for a binary tree representing depth D, which includes GLN at each non-leaf node. Each non-leaf node at any level d < D in the tree has two child nodes at level d+1, and each leaf node corresponds to an interval. Therefore, at each level d in the tree, the binary tree uniformly divides the bounded reward range into 2^d intervals, each decreasing in size as d increases.

[0103] Therefore, the probability predicted by each GLN is the probability that, assuming the reward falls within the interval represented by one of the child nodes of the GLN in the tree, the reward will fall within the interval represented by one of the child nodes but not the other. For example, the right branch from each GLN can be associated with the label 1, while the left branch can be associated with the label 0. A prediction from a given GLN represents the probability that the reward falls within the interval represented by the child nodes of the GLN connected to the tree via the right branch.

[0104] Therefore, each leaf node, that is, each interval in the 2^D interval, can only be reached by a single unique path starting from the root node of the tree.

[0105] Therefore, once each GLN in the tree has generated a probability, the system can calculate the total probability for a given interval as the product of the probabilities that each GLN is assigned to the next node along the unique path for that interval. That is, continuing the example above where the right branch from each GLN is associated with a label of 1 and the left branch with a label of 0, the probability of a given GLN being assigned to a child node along the right branch is equal to the probability predicted by the given GLN, while the probability of a given GLN being assigned to a child node along the left branch is equal to 1 minus the probability predicted by the given GLN.

[0106] Specifically, if the path through the tree to a given interval is represented as a binary vector b of length D, where each value in the vector identifies the next node on the path through the tree (an earlier node on the given path), and the value is 1 if the next node is assigned label 1 and 0 if the next node is assigned label 0, then the probability for the given interval satisfies:

[0107]

[0108] Among them, b i It is the i-th value in b, GLN0(x) t ) is the GLN root node of the tree in a given context x. t The output under the condition, and GLN i-1 (x t ) is from b i-1 The output of the identified GLN.

[0109] The system also maintains a representative value for each interval, such as the midpoint between the lowest and highest values ​​in the interval.

[0110] Then, the system executes steps 302-308 (and optionally 310 and 312) in response to each received context input. That is, the system can execute steps 302-308 or steps 302-312 in response to each of a series of context inputs to select the action to be performed in response to each context input.

[0111] For each action, the system uses a gated linear network tree corresponding to that action to process the context input in order to generate the corresponding predicted probability for each interval (step 304).

[0112] For each action, the system generates an action score for that action based at least on the predicted probability for that action (step 306).

[0113] Specifically, for each action, the system can generate the expected reward for that action by calculating a weighted sum of representative values ​​for intervals, where each interval is weighted by the probability predicted for the corresponding interval by the GLN tree for that action.

[0114] In some implementations, the expected reward is used as the action score. In other implementations, the system generates an action score based on the predicted probability of the action and one or more other factors.

[0115] As a specific example and as described above, the system can compute pseudo-counts for actions and then generate action scores for those actions based on predicted probabilities and pseudo-counts. As mentioned above, computing action scores based on both pseudo-counts and predicted probabilities encourages exploration of the possible space of context-action pairs and leads to higher-quality action selection strategies.

[0116] As a specific example, when using pseudo-counting, the action score of action a at time step t in the context input sequence can satisfy:

[0117]

[0118] in, It is based on the output of the GLN tree used for action a for that action in a given context x. t The predicted probability generated under the given conditions, where C is a normal value for scaling the exploration bonus, and It is a pseudo-count.

[0119] The system can calculate pseudo-counts for actions in any of a variety of ways.

[0120] As a specific example, the system can compute a pseudo-count based on the overlap between (i) the signature of the context of the gating function of a neuron in a gated linear network across a GLN tree for an action and (ii) the signature of the context of the gating function of a neuron in a gated linear network across a GLN tree for any earlier context in the sequence, where the action for the earlier context is selected as the action performed in response to the earlier context.

[0121] As mentioned above, the overlap between a signature corresponding to one time step and another signature corresponding to another time step identifies which gating functions generate the same output for both the first and the other time step.

[0122] As a specific example, the system can determine, for a given action and for each neuron in the GLN in the GLN tree for that action, how many earlier time steps in the window satisfy the following conditions: (1) the given action is selected at an earlier time step, and (2) the output of the gating function of the neuron for the context at the earlier time step is the same as the output of the gating function of the neuron for the current time step.

[0123] The system can then determine the pseudo-count by aggregating the corresponding counts of neurons, for example, by setting the pseudo-count to be equal to the average, minimum, maximum, or median of the corresponding count.

[0124] That is, unlike when a pseudo-count is calculated when only a single GLN is maintained for each action (as described above with reference to step 206), when a GLN tree is maintained for each action, the count of all neurons in all GLNs across the tree is aggregated.

[0125] The system selects the action to be executed based on the action score of the actions in the action set in response to the context (step 308). For example, the system can select the action with the highest action score, or it can map the action scores to a probability distribution and then sample the actions according to the probability distribution.

[0126] Optionally, the system can then update the weights of the neurons in the GLN tree for the selected action by performing steps 310-312. Specifically, in some cases, the system updates the weights of the GLN in the tree after each context in the sequence (fully online). In other cases, the system may update the weights of the GLN only after every N contexts in the sequence, or it may freeze the weights after a threshold number of weight updates have been performed.

[0127] The system receives a report as a result of performing the selected action in response to the context (step 310).

[0128] The system updates the GLN tree for the selected action based on the received feedback (step 312).

[0129] More specifically, the system identifies the interval to which the received reward belongs and updates each neuron in any GLN on the path from the tree for the selected action to the identified interval.

[0130] As described above, the system locally updates the weights of each neuron in each gated linear network along the path based on a neuron-specific loss. Therefore, the system does not need to perform computationally expensive gradient descent to compute the loss.

[0131] The loss of a given neuron is typically measured as the error between the target of the GLN to which the neuron belongs (based on the received reward) and the probability predicted by the given neuron given the input probabilities to that neuron.

[0132] The system updates the weights of neurons by calculating the gradient of the loss and updating the selected weight vector. Specifically, if the next GLN on the path is associated with label 1, the system sets the target of the given GLN on the path to 1, and if the next GLN on the path is associated with label 0, the target is set to 0.

[0133] Specifically, the gradient of the loss for a given neuron j in layer i of a given GLN can satisfy:

[0134] -n(p ij -target)logit(p i-1 )),

[0135] Where n is the learning rate used to update GLN, logit is the logit function, i.e., the inverse of the sigmoid function, and p ij It is the probability predicted by neuron j in layer i, where target is the target of the GLN to which the neuron belongs and is based on the received reward r, and p i-1 It is the probability received as input from layer i-1.

[0136] The system can then update the selected weight vector by adding or subtracting gradients. In this implementation, to improve the stability of the update, the system prunes the weight values ​​so that they fall within a specific range, for example, the range from -b to b(inclusive) for a constant b between 10 and 100. In these cases, if any value in the new weight vector is below the specific range, the system sets that value to the lowest value within that range. If any value in the new weight vector is above the specific range, the system sets that value to the highest value within that range.

[0137] This specification uses the term "configured" in connection with system and computer program components. A system of one or more computers configured to perform a specific operation or action means that the system has software, firmware, hardware, or a combination thereof installed thereon, which, in operation, causes the system to perform the operation or action. A computer program configured to perform a specific operation or action means that one or more programs include instructions that, when executed by a data processing device, cause the device to perform an operation or action.

[0138] Embodiments of the subject matter and functional operation described in this specification may be implemented in digital electronic circuits, in tangibly implemented computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of these. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these. Alternatively or additionally, the program instructions may be encoded on artificially generated propagated signals (e.g., machine-generated electrical, optical, or electromagnetic signals, which are generated as encoded information to be transmitted to a suitable receiver device for execution by the data processing apparatus).

[0139] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, or multiprocessors or computers. The apparatus may also be or further include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may optionally include code that creates an execution environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0140] A computer program (also referred to or described as a program, software, software application, application, module, software module, script, or code) can be written in any programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to that program, or in multiple collaborating files (e.g., a file storing portions of one or more modules, subroutines, or code). A computer program can be deployed to execute on one or more computers located at a site or distributed across multiple sites interconnected via a data communication network.

[0141] In this specification, the term "database" is used broadly to refer to any collection of data: data that does not need to be structured in any particular way, or does not need to be structured at all, and can be stored on storage devices in one or more locations. Therefore, for example, an index database may include multiple collections of data, each of which can be organized and accessed differently.

[0142] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines may be installed and run on the same one or more computers.

[0143] The processes and logic flows described in this specification can be executed by one or more programmable computers that execute one or more computer programs, thereby achieving functionality by manipulating input data and generating outputs. The processes and logic flows can also be executed by dedicated logic circuitry (e.g., FPGA or ASIC), or by a combination of dedicated logic circuitry and one or more programmed computers.

[0144] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The components of a computer are the central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be complemented by or integrated into special-purpose logic circuitry. Generally, a computer may include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or may be operatively coupled to one or more mass storage devices to receive data from or transfer data to or from them, or both. However, a computer need not have such devices. Additionally, a computer may be embedded in another device (e.g., to name just a few, mobile phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, GPS receivers, or portable storage devices (e.g., Universal Serial Bus (USB) flash drives)).

[0145] Computer-readable media used for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD ROMs and DVD-ROMs.

[0146] To support user interaction, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user and a keyboard and pointing device through which the user can provide input to the computer. The display device is, for example, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and the pointing device is, for example, a mouse or trackball. Other types of devices can also be used to support user interaction; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. Furthermore, the computer can interact with the user by sending and receiving files from a device used by the user; for example, by sending a webpage to a web browser in response to a request received from a web browser on the user's device. Additionally, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and subsequently receiving response messages from the user.

[0147] The data processing apparatus for implementing machine learning models may also include, for example, dedicated hardware accelerator units for processing the ordinary and computationally intensive parts (i.e., inference, workload) generated during machine learning training or production.

[0148] Machine learning models can be implemented and deployed using machine learning frameworks such as TensorFlow, Microsoft Cognitive Toolkit, Apache Singa, or Apache MXNet.

[0149] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., as a data server), middleware components (e.g., an application server), or front-end components (e.g., a client computer with a graphical user interface, web browser, or application through which a user can interact with the implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected via any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0150] A computing system may include clients and servers. Generally, clients and servers are geographically separated and typically interact via a communication network. The client-server relationship is established by computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server sends data (e.g., HTML pages) to a user device acting as a client, for purposes such as displaying data to a user interacting with the device and receiving user input from that user. Data generated at the user device (e.g., the result of user interaction) can be received from that device at the server.

[0151] Although this specification contains numerous details of specific implementations, these details should not be construed as limiting the scope of any invention or the scope of possible claims, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Furthermore, although features may be described as functioning in certain combinations as described above and initially claimed in this manner, in some cases, one or more features from a claimed combination may be excluded from the combination, and the claimed combination may involve sub-combinations or variations thereof.

[0152] Similarly, although operations are depicted in a specific order in the accompanying drawings and described in the claims, this should not be construed as requiring the operations to be performed in the specific order shown or in a sequential order, or requiring the execution of all illustrated operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, but rather it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0153] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions described in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method for selecting an action from a set of actions to be performed in response to each context in a sequence of contexts, wherein each action in the set of actions is a recommendation of a content item, and each context in the sequence of contexts represents a feature vector characterizing a current recommendation setting, the current recommendation setting including data describing the environment in which the content item will be recommended, the method comprising: Maintain data for a corresponding gated linear network for each action in the set of actions, wherein each gated linear network is configured to predict the probability of receiving a reward if the corresponding action is performed in response to an input context, and wherein each gated linear network comprises multiple layers, each layer comprising one or more neurons, wherein each neuron in each layer after the first layer is configured to receive (i) the input context and (ii) predictions from neurons in the previous layer, and (iii) apply a gating function to the input context to select a weight vector, and (iv) generate a geometric mixture of predictions from neurons in the previous layer as output based on the selected weight vector; For each context in the context sequence: For each action, a gated linear network corresponding to the action is used to process the context to generate predicted probabilities; For each action, at least based on the predicted probability, an action score is generated for the action, including: Calculate a pseudo-count for the action, wherein calculating the pseudo-count includes: determining (i) the overlap between the signature of the context of the gating function of a neuron in a gated linear network for the action and (ii) the signature of any earlier context in the context sequence for which the action is selected to be performed in response to the earlier context, wherein the signature of the context represents the output of the gating function of a neuron in a gated linear network corresponding to the action generated by processing the context; An action score is generated based on the predicted probability for the action and the pseudo-count for the action; and The action to be performed is selected based on the action score in response to the context.

2. A method for selecting an action from a set of actions to be performed in response to each context in a sequence of contexts, wherein each action in the set of actions is a user interface element presented to a user in a presentation setting, and each context in the sequence of contexts represents a feature vector characterizing the presentation setting, the user, or both, the method comprising: Maintain data for a corresponding gated linear network for each action in the set of actions, wherein each gated linear network is configured to predict the probability of receiving a reward if the corresponding action is performed in response to an input context, and wherein each gated linear network comprises multiple layers, each layer comprising one or more neurons, wherein each neuron in each layer after the first layer is configured to receive (i) the input context and (ii) predictions from neurons in the previous layer, and (iii) apply a gating function to the input context to select a weight vector, and (iv) generate a geometric mixture of predictions from neurons in the previous layer as output based on the selected weight vector; For each context in the context sequence: For each action, a gated linear network corresponding to the action is used to process the context to generate predicted probabilities; For each action, at least based on the predicted probability, an action score is generated for the action, including: Calculate a pseudo-count for the action, wherein calculating the pseudo-count includes: determining (i) the overlap between the signature of the context of the gating function of a neuron in a gated linear network for the action and (ii) the signature of any earlier context in the context sequence for which the action is selected to be performed in response to the earlier context, wherein the signature of the context represents the output of the gating function of a neuron in a gated linear network corresponding to the action generated by processing the context; An action score is generated based on the predicted probability for the action and the pseudo-count for the action; and The action to be performed is selected based on the action score in response to the context.

3. The method according to claim 1 or 2, wherein selecting the action to be performed based on the action score in response to the context includes selecting the action with the highest action score.

4. The method according to claim 1 or 2, further comprising: For each context in the context sequence: Receive rewards; as well as The gated linear network for the selected action is updated based on the reward.

5. The method according to claim 4, wherein, Updating the gated linear network for the selected action includes: Each neuron in the gated linear network is locally updated based on neuron-specific losses.

6. The method according to claim 1 or 2, wherein, The last layer of each gated linear network comprises only a single neuron, and the predicted probability of the gated linear network is the output of that single neuron.

7. The method according to claim 1 or 2, wherein, Neurons in the first layer of the plurality of layers in each gated linear network receive the input context and the basic prediction set.

8. A method for selecting an action from a set of actions to be performed in response to each context in a sequence of contexts, wherein Each action in the action set is a recommendation of a content item, and each context in the context sequence represents a feature vector characterizing the current recommendation setting, which includes data describing the environment in which the content item will be recommended. The method includes: Maintain data for a corresponding gated linear network tree for each action in the action set, wherein each gated linear network tree is collectively configured to predict a corresponding probability for each of a plurality of intervals of a range of reward values, wherein the corresponding probability for each interval represents the likelihood of receiving a reward falling within the interval if the corresponding action is performed in response to an input context, and wherein each gated linear network of each gated linear network tree comprises a plurality of layers, each layer comprising one or more neurons, wherein each neuron in each layer after the first layer is configured to receive (i) the input context and (ii) predictions from neurons in the previous layer, and (iii) apply a gating function to the input context to select a weight vector, and (iv) generate a geometric mixture of predictions from neurons in the previous layer as output based on the selected weight vector; For each context in the context sequence: For each action, the context is processed using a gated linear network tree corresponding to the action to generate a corresponding probability for each of a plurality of intervals of the reward value range; For each action, at least based on the corresponding probability, an action score is generated for the action, including: Calculate a pseudo-count for the action, wherein calculating the pseudo-count includes: determining (i) the signature of the context of the gating function of a neuron in a gated linear network in a gated linear network tree for the action and (ii) the overlap between the signature of the context of the gating function of a neuron in a gated linear network tree for any earlier context in the sequence of contexts for which the action is selected to be performed in response to the earlier context, wherein the signature of the context refers to the output of the gating function of a neuron in a gated linear network tree for the action generated by processing the context; and The action to be performed is selected based on the action score in response to the context.

9. A method for selecting an action from a set of actions to be performed in response to each context in a sequence of contexts, wherein each action in the set of actions is a user interface element presented to a user in a presentation setting, and each context in the sequence of contexts represents a feature vector characterizing the presentation setting, the user, or both, the method comprising: Maintain data for a corresponding gated linear network tree for each action in the action set, wherein each gated linear network tree is collectively configured to predict a corresponding probability for each of a plurality of intervals of a range of reward values, wherein the corresponding probability for each interval represents the likelihood of receiving a reward falling within the interval if the corresponding action is performed in response to an input context, and wherein each gated linear network of each gated linear network tree comprises a plurality of layers, each layer comprising one or more neurons, wherein each neuron in each layer after the first layer is configured to receive (i) the input context and (ii) predictions from neurons in the previous layer, and (iii) apply a gating function to the input context to select a weight vector, and (iv) generate a geometric mixture of predictions from neurons in the previous layer as output based on the selected weight vector; For each context in the context sequence: For each action, the context is processed using a gated linear network tree corresponding to the action to generate a corresponding probability for each of a plurality of intervals of the reward value range; For each action, at least based on the corresponding probability, an action score is generated for the action, including: Calculate a pseudo-count for the action, wherein calculating the pseudo-count includes: determining (i) the signature of the context of the gating function of a neuron in a gated linear network in a gated linear network tree for the action and (ii) the overlap between the signature of the context of the gating function of a neuron in a gated linear network tree for any earlier context in the sequence of contexts for which the action is selected to be performed in response to the earlier context, wherein the signature of the context refers to the output of the gating function of a neuron in a gated linear network tree for the action generated by processing the context; and The action to be performed is selected based on the action score in response to the context.

10. A system for selecting actions from a set of actions to be performed, comprising one or more computers and one or more storage devices storing instructions, the instructions causing the one or more computers to perform operations according to any one of claims 1 to 9 when executed by the one or more computers.

11. A computer storage medium storing one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform the operation of the method according to any one of claims 1 to 9.

Images