Sub-additive action planning using multiple action selection policies

Abstract

Systems and methods for sub-additive action planning using multiple action selection policies. For example, sub-additive action planning can be performed using statistics of different tree searches that are guided by different action selection policies. As another example, multiple different action selection policies can be learned using intrinsic rewards that encourage diversity.

Description

[0001] SUB-ADDITIVE ACTION PLANNING USING MULTIPLE ACTION SELECTION

[0002] POLICIES

[0003] CROSS-REFERENCE TO RELATED APPLICATION

[0004] This application claims priority to U.S. Provisional Application No. 63 / 533,123, filed on August 16, 2024. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application.

[0005] BACKGROUND

[0006] This specification relates to selecting actions to be performed by a reinforcement learning agent.

[0007] Reinforcement learning agents interact with an environment by receiving an observation that characterizes the current state of the environment, and in response, performing an action. Once the action is performed, the agent receives a reward that is dependent on the effect of the performance of the action on the environment.

[0008] Some reinforcement learning systems use neural networks to select the action to be performed by the agent in response to receiving any given observation.

[0009] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks are deep neural networks that include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters.

[0010] SUMMARY

[0011] This specification generally describes a system implemented as computer programs on one or more computers in one or more locations that maintains a plurality of diverse action selection policies for selecting actions to be performed by an agent interacting with an environment.

[0012] That is, the system receives a current observation characterizing a current state of an environment being interacted with by an agent and selects an action to be performed by the agent using the plurality of maintained diverse action selection policies. The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.

[0013] In principle, a single action selection policy, i.e., action selection neural network, can learn to always select actions optimally for an agent to perform any reinforcement learning task using enough training data related to performing the task. In other words, the actions can be optimally selected to maximize the likelihood that a desired result, such as performance of a learned task, will be achieved.

[0014] For example, an action selection policy for an autonomous vehicle agent performing the task of navigating to a destination may learn to drive in a manner that is safe through many repeated attempts, e.g., through many simulations of a real-world environment that simulates the navigation of an autonomous vehicle.

[0015] But in practice it is difficult to learn a single action selection policy that always optimally selects actions for an agent to perform a task. Due to computer memory limitations, computer latency limitations, and challenges in defining appropriate rewards for a task, it is impractical for a single action selection policy to sufficiently explore the environment state space, i.e., the set of all possible states that an agent can encounter within an environment, during training to then always select the optimal action for an agent.

[0016] For example, a single action selection policy for the autonomous vehicle agent may face a real-world environment state, e.g., momentum of vehicle, friction of tires, weather conditions, and traffic arrangement, not represented in the training data due to the vast size of the environment state space. As a result, when the policy encounters an unfamiliar environment state that is not represented by the training data, the policy is unlikely to optimally select an action for the agent to perform the task, and worse, may select an action that results in poor task performance.

[0017] This specification, on the other hand, describes training a plurality of action selection policies to have diversity in terms of behavior, i.e., how often an action selection policy visits state-action pairs when selecting actions according to the action selection policy, then maintaining, and using the plurality of diverse action selection policies to select optimal actions for an agent to perform using a “sub-additive planning” method.

[0018] That is, for each of the plurality of action selection policies, the system performs a respective tree search through future states of the environment using the action selection policy to generate respective statistics for each of a plurality of actions, then the system selects an action selection policy according to the statistics, and then the system selects an action according to the statistics. The set of diverse action selection policies proves to be advantageous over a single action selection policy because the collective behavior of the set of the action selection policies can be used to select the optimal action for a greater range of environment states than the behavior of a single action selection policy could be used for. That is, different explorations of the environment state space through different behaviors starting from an unfamiliar state by performing different tree searches using the multiple diverse action selection polices can uncover the optimal action associated with a particular action selection policy during performance of a task that results in better task performance in contrast to using any single action selection policy, which is less likely to uncover the optimal action.

[0019] For example, consider an autonomous vehicle agent performing a task of safely navigating and selecting actions from two maintained action selection policies. The first action selection policy learned safe driving by following vehicles closely to prevent other vehicles from cutting in, reducing the likelihood of abrupt stops to avoid collision with merging vehicles causing accidents, and the second action-selection policy learned safe driving by following vehicles with greater distance to allow space for maneuvering, reducing the likelihood of an abrupt stop of the followed vehicle causing accidents. The policies are diverse in that they result in different agent behavior, yet both still accomplish the task of navigating safely. Suppose that the first and second action selection policies navigate autonomous vehicles equally safely. Now consider the autonomous vehicle agent selecting a follow time interval to a vehicle that recently merged ahead during rainy conditions, the rainy conditions not being present during training of the first and second action selection policies. Exploring environment state space during a tree search starting from the current rainy environment state generates statistics that indicate actions from the second action selection policy will perform better than those from the first action selection policy because during rainy conditions stopping distances, i.e., the distance a vehicle travels after applying brakes, increases and therefore a greater follow distance is less likely to result in a collision than a close following distance. While neither the first nor the second action selection policies included the information that stopping distances increase during rainy conditions, the second action selection policy uncovers this lesson, and therefore a tree search according to the second action selection policy uncovers the optimal action.

[0020] The example illustrates why the diversity in terms of behavior of the action selection policies is an asset, referred to as a ‘diversity bonus’, because if only one action selection policy, i.e., if only the first action selection policy, were available, then the optimal action would not be uncovered. The variety of methods to select actions according to the generated statistics are referred to as “sub-additive planning” methods.

[0021] The term “sub-additive planning” is in contrast to performing planning using “max- over-polices” using an oracle, i.e., an entity that has perfect knowledge of the consequences of selecting actions and action selection policies, to evaluate each action selection policy to select the action selection policy and corresponding action that will lead to successful completion of a task. In particular, performing planning using “max-over-policies” is a performance upper bound relative to any other method that evaluates the set of action selection policies to select an action. Thus, “sub -additive” planning refers to performing planning in a manner that is more computationally efficient than performing planning using max-over-policies, i.e., that does not require searching through the state tree to a state in which the task is successfully completed.

[0022] Sub-additive planning methods in practice prove to be advantageous in terms of computational cost and generalizability of task performance for evaluating action selection policies and actions compared to methods that attempt to approximate a “max-over-policies” method to a high degree.

[0023] For example, achieving oracle like evaluation of action selection policies may require tree searches with depths that approach large values to reach terminal states, i.e., states that correspond to success or failure of task performance to evaluate actions, and performing such an evaluation could be prohibitively costly to do for all action selection policies and actions, i.e., may consume an excessive amount of computational resources and result in an unacceptable cost when selecting actions.

[0024] In contrast, sub-additive planning methods are more cost effective by relying on statistics that do not require tree searches reaching a terminal state to evaluate action selection policies.

[0025] As another example, achieving oracle like evaluation of action selection policies may require selecting actions using criteria that are complex and specific, e.g., cumulative discounted task specific rewards using a reward definition that is insufficiently broad for general instances of the task, and, as a result, may work well for selecting action selection policies and corresponding actions for environment states during training but may generalize poorly to unfamiliar environment states. Conversely, sub-additive planning criteria for first selecting an action selection policy using statistics are not task-specific, resulting in better generalization to unfamiliar states. Diversity may refer to the variation or differences among a plurality of action selection policies. This variation may manifest in different ways. For example, diversity may be reflected in differences in action selection probabilities, state-action visitation frequencies, or exploration strategies. Diversity may be measured by quantifying the dissimilarity between the policies' action selection probabilities for a given state. Alternatively, diversity may be measured by comparing the frequency with which different policies visit specific state-action pairs during training or execution. In one exemplary embodiment, diversity may be measured using a metric such as the Jensen-Shannon divergence between the probability distributions over actions generated by different policies for a given state.

[0026] Sub-additive planning may be a method for selecting actions that leverages information or statistics gathered from multiple distinct action selection policies. This approach may contrast with methods that rely on a single policy or require exhaustive evaluation of all policies. Sub-additive planning may involve performing independent searches or evaluations using each of the multiple policies and then combining the results to select an action. This combination may be based on various criteria, such as the maximum value, visit count, or a combination of factors. In one example, sub-additive planning may involve selecting the policy with the highest maximum action value (e.g., Q-value) and then selecting the action with the highest value according to that policy.

[0027] An intrinsic reward may be a reward signal that is not directly tied to the achievement of the primary task goal, i.e., not directly tied to successfully completing the task, but instead encourages exploration, diversity, or other desirable behaviors in the agent or the action selection policies. Intrinsic rewards may be designed to promote the development of policies that exhibit distinct behaviors, explore different parts of the state-action space, or learn representations that capture relevant features of the environment. In one example, the intrinsic reward may be calculated based on the dissimilarity between a given policy's action selection probabilities and those of one or more other policies in the set, encouraging policies to specialize in different aspects of the task or environment.

[0028] An extrinsic reward may be a reward signal that is directly related to the achievement of the primary task goal or objective. This reward may be provided by the environment and may reflect the agent's progress towards completing the task. Extrinsic rewards may be defined in various ways, depending on the specific task. They may be sparse, providing a reward only upon successful task completion, or dense, providing rewards for intermediate progress towards the goal. In one exemple, the extrinsic reward for an autonomous navigation task may be a positive value for reaching the destination and a negative value for collisions or deviations from the desired path.

[0029] A latent representation may refer to a compressed or encoded representation of the input observation generated by the action selection neural network. This representation captures essential features of the environment's state and is used by the network to generate action probabilities, value estimates, and other outputs. For example, a latent representation may encode spatial relationships between objects in a scene or the temporal dynamics of a sequence of events. In one exemple, the latent representation may be a vector of real-valued numbers produced by one or more hidden layers of the neural network.

[0030] A dynamics neural network may refer to a type of neural network that is trained to model the dynamics or behavior of a system over time. This network learns to predict the future state of a system given its current state and any actions taken. A dynamics neural network may be used to predict the next state of the environment given the current state and the action taken by the agent or, more specifically, a latent representation of the next state of the environment given the current state and the action. This allows the system to simulate the effects of actions without directly interacting with the real world, enabling efficient planning and decision-making. In one exemple, the dynamics neural network may be a self-attention neural network (e.g. a transformer neural network), recurrent neural network (RNN), a statespace model, a convolutional neural network (CNN) or any combination thereof that takes the current state and action as input and outputs the predicted next state.

[0031] An adjusted action score may refer to a modified score or value assigned to an action that incorporates additional factors beyond the initial or estimated value. This adjustment may be used to account for uncertainty, exploration, or other considerations that influence the desirability of an action. An adjusted action score may be used to guide the selection of actions during the tree search. For example, an exploration bonus may be added to the estimated action value to encourage the exploration of less-visited or uncertain actions. This helps the agent discover potentially better actions and improve its overall performance. In one example, the adjusted action score may be calculated by adding an exploration bonus, which may be proportional to the uncertainty or the inverse of the visit count for a given action, to the estimated action value (e.g., Q-value).The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below.

[0032] According to a first aspect there is provided a method performed by one or more computers. The method includes receiving a current observation characterizing a current state of an environment being interacted with by an agent to perform a task. Then, for each of multiple action selection policies for selecting actions to be performed by the agent, performing a respective tree search through future states of the environment starting from the current state of the environment and using the action selection policy to generate a respective set of statistics for each of multiple actions that can be performed by the agent in response to the current observation. Then lastly, selecting an action to be performed by the agent using the respective sets of statistics for the multiple actions generated for the multiple action selection policies.

[0033] In some implementations, each action selection policy has a corresponding latent variable that represents the action selection policy, and each action selection policy is defined by action selection outputs generated by an action selection neural network by processing an action selection input that includes an input observation and the corresponding latent variable for the action selection policy.

[0034] In some implementations, the corresponding latent variable for each action selection policy is a one-hot vector that identifies the action selection policy.

[0035] In some cases, the respective tree search through future states of the environment starting from the current state of the environment and using the action selection policy to generate a respective set of statistics for each of multiple actions that can be performed by the agent in response to the current observation includes performing the respective tree search using action selection outputs generated by the action selection neural network while the action selection neural network is conditioned on the corresponding latent variable for the action selection policy.

[0036] In some implementations, selecting an action to be performed by the agent using the respective sets of statistics for the multiple actions generated for the multiple action selection policies includes selecting one of the multiple action selection policies using the respective sets of statistics for the multiple actions generated for the multiple action selection policies, and selecting the action to be performed by the agent using the respective set of statistics for the multiple actions generated for the selected action selection policy.

[0037] In some implementations, selecting the action to be performed by the agent using the respective set of statistics for the multiple actions generated for the selected action selection policy includes determining a respective likelihood for each action in the multiple actions from the respective set of statistics for the multiple actions generated for the selected action selection policy. Then, selecting the action using the respective likelihoods. In some implementations, the respective sets of statistics for the multiple actions generated for the multiple action selection policies includes for each action selection policy, a respective visit count for each of the multiple actions that identifies a number of times the action was selected to be performed in response to the current observation during the tree search performed using the action selection policy.

[0038] In some implementations, selecting one of the multiple action selection policies using the respective sets of statistics for the multiple actions generated for the multiple action selection policies includes determining, for each action selection policy, a respective maximum visit count of the respective visit counts for the multiple actions for the action selection policy, and selecting an action selection policy that has a highest respective maximum visit count.

[0039] In some cases, the respective sets of statistics for the multiple actions generated for the multiple action selection policies include for each action selection policy, a respective value estimate for each of the multiple actions that identifies an estimate of a value to performing the task of performing the action in response to the current observation as a result of the tree search performed using the action selection policy.

[0040] In some implementations, selecting one of the multiple action selection policies using the respective sets of statistics for the multiple actions generated for the multiple action selection policies includes determining, for each action selection policy, a respective maximum value of the respective value estimates for the multiple actions for the action selection policy, and selecting an action selection policy that has a highest respective maximum value estimate.

[0041] In some implementations, selecting one of the multiple action selection policies using the respective sets of statistics for the multiple actions generated for the multiple action selection policies includes determining, for each action selection policy, a respective maximum value of the respective value estimates for the multiple actions for the action selection policy, determining an argmax action selection policy that has a highest respective maximum value estimate, identifying each action selection policy that has a respective maximum value that is within a threshold of the highest respective maximum value estimate, and selecting, from the identified action selection policies, the action selection policy that has a lowest respective maximum value.

[0042] In some cases, the respective sets of statistics for the multiple actions generated for the multiple action selection policies includes for each action selection policy, (i) a respective value estimate for each of the multiple actions that identifies an estimate of a value to performing the task of performing the action in response to the current observation as a result of the tree search performed using the action selection policy and (ii) a respective exploration term for each of the multiple actions that identifies an estimate of a value to exploring the environment of performing the action in response to the current observation as a result of the tree search performed using the action selection policy.

[0043] In some cases, selecting one of the multiple action selection policies using the respective sets of statistics for the multiple actions generated for the multiple action selection policies includes determining, for each action selection policy, a respective maximum difference of the respective value estimates for the multiple actions for the action selection policy. The respective difference for each action is a difference between the respective value estimate for the action and the respective exploration term for the action. Then selecting an action selection policy that has a highest respective maximum difference.

[0044] In some cases, the action selection neural network includes a policy output and the action selection neural network is configured to 1) process the action selection input that includes an input observation and the corresponding latent variable for the action selection policy to generate a latent representation, and 2) process the latent representation using a policy head to generate the policy output.

[0045] In some cases, the action selection neural network is further configured to process the latent representation using an intrinsic value head to generate an intrinsic value estimate that estimates a value of the current observation to the action selection policy being diverse relative to the other action selection policies in the multiple action selection policies.

[0046] In some cases, the action selection neural network is further configured to process the latent representation using an extrinsic value head to generate an extrinsic value estimate that estimates a value of the current observation to successfully perform the task using the action selection policy.

[0047] In some implementations, performing a respective tree search through future states of the environment starting from the current state of the environment and using the action selection policy to generate a respective set of statistics for each of multiple actions that can be performed by the agent in response to the current observation includes performing the tree search by traversing a tree with nodes representing states in a state space of states of the environment.

[0048] In some implementations, performing a respective tree search through future states of the environment starting from the current state of the environment and using the action selection policy to generate a respective set of statistics for each of multiple actions that can be performed by the agent in response to the current observation includes 1) performing the tree search by traversing a tree with nodes representing states in a latent space of latent representations generated by the action selection neural network, and 2) using a dynamics neural network that is configured to receive an input latent representation of the input observation and an action and to generate a predicted latent representation of a state that the environment transitions into as a result of the action being performed in response to the input observation.

[0049] In some implementations, performing the tree search includes evaluating leaf nodes of the tree using a policy output generated from a latent representation of a state corresponding to the leaf nodes.

[0050] In some implementations, performing the tree search includes evaluating leaf nodes of the tree using a value generated from a value estimate and an intrinsic value estimate generated from a latent representation of a state corresponding to the leaf nodes.

[0051] In some implementations the one or more computers cause the agent to perform the selected action.

[0052] In some implementations the one or more computers provide, to the agent, information identifying the selected action.

[0053] According to a second aspect there is provided a method performed by one or more computers. The method includes training multiple action selection policies for controlling an agent interacting with an environment to perform a task. The multiple action selection policies include a first action selection policy and multiple second action selection policies. Each second action selection policy is trained to optimize a respective second objective that depends on a respective reward that is a combination of an extrinsic reward for the task and an intrinsic reward that measures a diversity of the multiple action selection policies. The first action selection policy is trained to optimize a respective first objective that depends on the extrinsic reward but not on the intrinsic reward. The one or more computer then select the first action selection policy as a final policy for controlling the agent.

[0054] In some implementations of the second aspect, the method includes controlling the agent to perform the task using the first action selection policy.

[0055] In some implementations of the second aspect, each action selection policy has a corresponding latent variable that represents the action selection policy, each action selection policy is defined by action selection outputs generated by an action selection neural network by processing an action selection input that includes an input observation and the corresponding latent variable for the action selection policy. Also, training each of the multiple action selection policies includes training the action selection neural network on outputs generated by the action selection neural network while the action selection neural network is conditioned on inputs that include the corresponding latent variable for the action selection policy.

[0056] In some implementations of the second aspect, training multiple action selection policies includes, for each of the action selection policies, obtaining a training observation characterizing a training state of the environment, obtaining outputs of a tree search through future states of the environment that is performed (i) starting from the training state of the environment and (ii) using the action selection policy, generating, using the action selection policy, an action selection output, generating, from the outputs of the tree search, a target action selection output for the action selection policy, and training the action selection policy using a loss function that is based on the action selection output and the target action selection output.

[0057] In some implementations of the first or second aspects, the agent is a mechanical agent and the environment is a real-world environment.

[0058] In some implementations of the first or second aspects, the agent is a robot.

[0059] In some implementations of the first or second aspects, the environment is a real- world environment of a service facility including multiple items of electronic equipment, and the agent is an electronic agent configured to control operation of the service facility.

[0060] In some implementations of the first or second aspects, the environment is a real- world manufacturing environment for manufacturing a product, and the agent is an electronic agent configured to control a manufacturing unit or a machine that operates to manufacture the product.

[0061] In some implementations of the first or second aspects, the environment is a simulation of a real-world environment. The method of the first or second aspect is performed during training of the action selection policies, and the method further includes, after the training, controlling a real-world agent in the real-world environment using one or more of the action selection policies.

[0062] In some implementations of the first or second aspects, the agent is a user of a digital assistant, the method includes obtaining information defining the task from the digital assistant, and using the digital assistant to provide information about how to perform the task to the user.

[0063] In some implementations of the previously described implementation, the method includes receiving, at the digital assistant, a request from the user for assistance, and determining, in response to the request, that the user should perform the task. Then outputting, from the digital assistant to the user, an indication of the task to be performed. The current observation is a visual or audio observation or both of the user performing the task captured by the digital assistant.

[0064] In some implementations of the first or second aspects, the environment is a data item compression environment, the task is to compress an input data item, each state of the environment corresponds to a respective state of the compression of the input data item, and the actions correspond to encoding decisions during the compression of the input data item.

[0065] In some implementations of the first or second aspects, the environment is a chip design environment, the task is to generate a chip design for a computer chip that satisfies an input specification, the state of the environment corresponds to a respective state of the chip design, and the actions correspond to chip design decisions during the generation of the chip design.

[0066] In some implementations of the first or second aspects, the environment is an algorithm optimization environment, the task is to generate a computer program that optimizes a target algorithm for execution on a target processor, each state of the environment corresponds to a respective state of the generation of the computer program, and the actions each apply a respective modification to the computer program.

[0067] In some implementations of the first or second aspects, the environment is an algorithm optimization environment, the task is to generate a sequence of modifications to a tensor that represent an algorithm that optimizes a target algorithm for execution on a target processor, each state of the environment corresponds to a respective state of the tensor, and the actions each apply a respective modification to the tensor.

[0068] In some implementations of the first or second aspects the environment is a computer system that comprises multiple computing devices, the task is to allocate one or more computational workloads across the multiple computing devices, each state of the environment corresponds to a respective state of the computer system given a current allocation of the one or more computational workloads, and the actions each apply a respective modification to the current allocation.

[0069] According to a third aspect there is provided the methods of the first or second aspect performed by one or more computers and one or more storage devices communicatively coupled to the one or more computers. The one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the respective method. According to a fourth aspect there is provided the methods of the first or second aspect performed by one or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the operations of the respective method.

[0070] Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

[0071] DESCRIPTION OF DRAWINGS

[0072] FIG. 1 shows an example reinforcement learning system.

[0073] FIG. 2 shows an example action selection neural network.

[0074] FIG. 3 is a flow diagram of an example process to select an action through subadditive planning.

[0075] FIG. 4 is a flow diagram of an example process to perform a tree search by an action selection policy to generate a respective set of statistics for each of a plurality of actions.

[0076] FIG. 5 is a flow diagram of an example process to train an action selection policy.

[0077] FIG. 6 shows an example of the performance of the describe techniques.

[0078] Like reference numbers and designations in the various drawings indicate like elements.

[0079] DETAILED DESCRIPTION

[0080] FIG. 1 shows an example reinforcement learning system 100. The reinforcement learning system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

[0081] The reinforcement learning system 100 is a system that maintains a plurality of action selection policies 106A-C for selecting actions to be performed by an agent performing a task and uses the action selection policies 106A-C to control the agent.

[0082] When controlling the agent, the system 100 controls the agent to accomplish a task by selecting actions to be performed by the agent at each of multiple time steps during the performance of an episode of the task.

[0083] An “episode” of a task is a sequence of interactions during which the agent attempts to perform an instance of the task starting from some starting state of the environment. In other words, each task episode begins with the environment being in an initial state, e.g., a fixed initial state or a randomly selected initial state, and ends when the agent has successfully completed the task or when some termination criterion is satisfied, e.g., the environment enters a state that represents a completed task (i.e. the environment meets a task completion criterion) or the agent performs a threshold number of actions without successfully completing the task.

[0084] At each time step during any given task episode, the system 100 receives an observation characterizing the current state 102 of the environment at the time step and, in response, selects an action 114 to be performed by the agent at the time step. After the agent performs the action, the environment transitions into a new state.

[0085] More specifically, to control the agent or to assist the agent at a given time step, the system 100 receives a current observation characterizing a current state 102 of the environment at the time step. That is, at any given time step, the reinforcement learning system 100 receives a current observation characterizing a current state 102 of the environment being interacted with by an agent.

[0086] The observation characterizes the state 102 of the environment in a manner that is appropriate for the context of use for the reinforcement learning system 100.

[0087] For example, when the agent is a control system for a mechanical agent interacting with the real-world environment, the observations may include sensor data captured by the sensors of the agent, e.g., images captured by sensors of the mechanical agent as it interacts with the real-world environment other sensor data captured by the sensors of the agent, or both.

[0088] As another example, when the environment is a patient diagnosis environment, the observations may be data from an electronic medical record of a current patient.

[0089] As another example, when the environment is a protein folding environment, the observations may be images of the current configuration of a protein chain, a vector characterizing the composition of the protein chain, or both.

[0090] Additional examples of actions, agents, observations, and environments are described below.

[0091] After receiving the current observation 102, for each action selection policy 106A-C, the system 100 performs a respective tree search through future states of the environment starting from the current state of the environment and using the action selection policy to generate a respective set of statistics, e.g., 112A, 112B, or 112C, for each of a plurality of actions that can be performed by the agent in response to the current observation. In some implementations, the action selection neural network 108 is configured to process an action selection input that includes an input observation 102 and a latent variable 104. Each of the action selection policies 106A-C has a corresponding latent variable 104 (which may alternatively be considered as a corresponding realization (“value”) of a single latent variable which may take any of multiple realizations), e.g., a one-hot encoding of the action selection policy or a different encoding that uniquely identifies the action selection policy. The latent variable thus represents the action selection policy. The policy is defined by action selection outputs 110 generated by an action selection neural network 108 by processing the action selection input that includes an input observation 102 and the corresponding latent variable 104 for the action selection policy. The action selection neural network may be configured to, in order to generate an output for a given action selection output, receive the latent variable 104 for the action selection policy, and to process a corresponding action selection input that includes the input observation 102 and the corresponding latent variable 104, to generate the respective action selection output(s) 110.

[0092] In some other implementations, each action selection policy can be defined by action selection outputs generated by an action selection neural network that is specific to the action selection policy, i.e., so that each action selection policy has a different corresponding action selection neural network. The input observation 102 is processed by a selected one or more, or by all, of the action selection neural networks corresponding to the action selection policies, to generate corresponding action selection output(s).

[0093] An action selection policy is “defined” by the outputs 110 of an action selection neural network when controlling an agent using the action selection policy includes using one or more outputs of the action selection neural network to select an action to be performed by the agent when the environment is in a given state. Thus, in this example, for a given action selection policy, controlling the agent using the action selection policy includes using one or more outputs of the action selection neural network (that are each generated from a respective input that includes, e.g., when there is a single action selection neural network that is shared across action selection policies, the corresponding latent variable) for the policy to select the action to be performed by the agent when the environment is in a given state. The reinforcement learning system may generate the latent variables 104 corresponding to selected ones of the action selection policies, or to all the action selection policies. For example, it may generate the latent variable for each of the action selection policies in turn, and transmit the latent variables to the action selection neural network 108. The action selection neural network 108 may use the latent variables to form successive corresponding action selection inputs to the action selection neural network, including the corresponding latent variable 104 and the observation of the state 102.

[0094] Generally, a given action selection neural network is configured to process an input that includes an observation (and, when there is a single action selection neural network that is shared across action selection policies, the latent variable for the action selection policy) to generate an action selection output 110.

[0095] The action selection outputs 110 can include a policy output that defines a score distribution over a set of actions, e.g., that includes a respective score, e.g., a respective probability or a respective action value (a “Q value”), for each action in the set or that includes parameters of a likelihood distribution over the set of actions. The Q value for a possible action represents an expected return to be received from the environment being in the state if the agent performs the possible action in response to the observation of the state.

[0096] The action selection outputs can also include a score for the observed state, e.g. a respective state value V, for the state 102. The “state value V” is the expected return output, i.e. an estimate of a return to be received from the environment being in the state.

[0097] The return, R, is typically a numeric reward or a combination, e.g., a time-discounted sum, of numeric rewards received as a result of the agent interacting with the environment, e.g., R = in which y is the discount factor, T is the episode length, t is the time step, and r is the immediate reward for the time step. The predicted expected return may be designated by a scalar value.

[0098] Generally, the immediate rewards can be extrinsic, intrinsic, or a combination of both.

[0099] An extrinsic reward is one that is received from the environment or from another system and that reflects the progress of the agent towards accomplishing the specified result of the task at the current state.

[0100] As a particular example, the extrinsic reward can be a sparse binary reward that is zero unless the task is successfully completed and one if the task is successfully completed as a result of the action performed.

[0101] As another particular example, the extrinsic reward can be a dense reward that measures a progress of the agent towards completing the task as of individual observations received during the episode of attempting to perform the task, i.e., so that non-zero rewards can be and frequently are received before the task is successfully completed.

[0102] An intrinsic reward is one that measures the difference between the action selection policy used to select the action and the other action selection policies. For example, the intrinsic reward can reflect the difference in action selection behavior of the action selection policy and the behavior of the most similar other action selection policy at the current state.

[0103] As a particular example, the intrinsic reward can be computed by the system 100 as the minimum negative inner product between the action selection policy’s policy output representing probability over each action IT and the policy output TT of all other action selection policies.

[0104] Further details of another example of computing intrinsic reward are described in more detail below with reference to FIG. 5.

[0105] As a particular example of combining extrinsic and intrinsic rewards, the combination of both extrinsic and intrinsic reward can be a weighted linear combination of extrinsic and intrinsic rewards, e.g., (1 — )rintrinsic+ rextrinsicin which A ranges from 0 to 1, for which extrinsic and intrinsic rewards are defined in any appropriate manner.

[0106] As another particular example, the combination of both extrinsic and intrinsic reward can be a non-linear combination of extrinsic and intrinsic rewards, e.g., a neural network processing of the two reward types, a user defined function for the two reward types, etc., for which extrinsic and intrinsic rewards are defined in any appropriate manner.

[0107] The action selection neural network 108 that generates the action selection outputs 110 can have any appropriate architecture in any appropriate configuration that processes latent variables 104 and a state 102 to generate action selection outputs 110, including fully connected layers, convolution layers, recurrent layers, attention-based layers, and so on, as is appropriate.

[0108] One example of the architecture of the action selection neural network 108 is described below with reference to FIG. 2.

[0109] While only three different action selection policies 106A-C are shown in FIG. 1, in practice the system 100 can include any number of action selection policies.

[0110] Generally, the system 100 performs the respective tree search for a given action selection policy using any appropriate tree search technique that performs a respective tree search (i) through future states of the environment starting from the current state of the environment and (ii) using the action selection policy to generate (iii) a respective set of statistics for each of a plurality of actions that can be performed by the agent in response to the current observation.

[0111] Depending on the implementation, the tree search can be through a tree of environment states in the environment state space, i.e., so that nodes in the tree represent states of the environment, or through a tree with nodes corresponding to latent representations of environment states.

[0112] In some implementations, the system 100 performs the tree search using a simulated version of the environment.

[0113] Generally, the simulated version of the environment is a virtualized environment, e.g., a computer simulation of the environment, that simulates how actions performed by the agent would affect the state of the environment.

[0114] For example, when the environment is a real-world environment and the agent is an autonomous or semi-autonomous vehicle, the simulated version of the environment is a motion simulation environment that simulates navigation through the real-world environment. That is, the motion simulation environment simulates the effects of various control inputs (the “action”) on the navigation of the vehicle through the real-world environment. More generally, when the environment is a real-world environment and the agent is a mechanical agent the simulated version of the environment is a dynamics model that models how actions performed by the agent change the state of the environment.

[0115] As another example, when the environment is a patient diagnosis environment, the simulated version of the environment is a patient health simulation that simulates effects of medical treatments on patients. For example, the patient health simulation may be a computer program that receives patient information and a treatment to be applied to the patient and outputs the effect of the treatment on the patient's health.

[0116] As another example, when the environment is a protein folding environment, the simulated version of the environment is a simulated protein folding environment that simulates effects of folding actions on protein chains. That is, the simulated protein folding environment may be a computer program that maintains a virtual representation of a protein chain and models how performing various folding actions will influence the protein chain.

[0117] As another example, when the environment is the virtual environment described below, the simulated version of the environment is a simulation in which a user, i.e., a user that controls the agent or a user that interacts with the agent, is replaced by another computerized agent.

[0118] In some cases, the task may be a task that requires interacting with other agents, i.e., that requires other agents to also perform actions. In these cases, the one or more other agents can be controlled by the system, e.g., by one of the action selection policies, or by the simulation of the environment. Generally, the system’s 100 tree search for an action selection policy collects statistics for each of a plurality of actions that can be performed by the agent in response to the current observation through multiple look ahead searches, i.e., multiple searches through a state tree having nodes representing states of the environment starting from the root node that represents the current state.

[0119] In particular, when performing a tree search using any given action selection policy, the system 100 maintains data representing a state tree of the environment. The state tree includes nodes that represent states of the environment and directed edges that connect nodes in the tree. An outgoing edge from a first node to a second node in the tree represents an action performed in response to an observation characterizing the first state resulting in the environment transitioning into the second state.

[0120] While the data is logically described as a tree, it can be represented by any of a variety of convenient physical data structures, e.g., as multiple triples, a hash table, or as an adjacency list.

[0121] The system 100 also maintains edge data for each edge in the state tree that includes statistics for the action represented by the edge.

[0122] The statistics for a given edge can include one or more of a respective visit count, i.e., the number of look ahead searches that take the action from the environment state, a respective total action value for the action, i.e., the sum of returns received after the tree search takes the action from the environment state over all look ahead searches that take the action, a respective mean action value, i.e., the total action value for the action divided by the visit count for the action, a prior probability for the action, i.e., how likely it is to select the action from the environment state before any look ahead searches occur, and an exploration term that identifies an estimate of a value to exploring the environment of selecting the action in response to the observation.

[0123] The set of statistics can also include statistics for nodes of the tree, e.g. a respective total state value, i.e., the sum of returns received from the environment state over all look ahead searches, a respective mean state value, i.e., the total state value divided by the total number of tree search iterations, or both.

[0124] Generally, the look ahead searches are guided by edge data, i.e., statistics for the plurality of actions that can be performed by the agent in response to the environment state. That is, at each in-tree node, i.e., a node encountered starting from the root node until reaching the leaf state, the system selects the edge to be traversed using edge data for the outgoing edges from the in-tree node representing the in-tree state. For example, a look ahead search can select an action as the edge to be traversed from the in-tree node associated with the highest adjusted action score that takes as inputs the edge data associated with the in-tree node.

[0125] For example, the adjusted action score for an action can be the sum of the Q value and a bonus term U proportional to the prior probability for the edge but inversely proportional to visits to encourage exploration, e.g., Q+U. For example, the bonus term U may be directly proportional to the product of the prior probability and a ratio that has the square root of the sum of all visit counts for all outgoing edges from the root node as the numerator and a constant, e.g., one, plus the visit count for the edge representing the action as the denominator, e.g., U = C*P*NO 5 / (1+N) where C is an exploration rate term, N is the visit count, and P is the prior probability for the edge.

[0126] At each iteration of a given tree search, the system 100 traverses the state tree until the state tree reaches a leaf state, i.e., a state that is represented by a leaf node in the state tree. Once it reaches the leaf state, the system 100 expands the leaf node using the action selection policy for the given tree search.

[0127] Performing the tree search will be described in more detail below.

[0128] In some implementations, the system 100 distributes the tree search, i.e., by running multiple different searches in parallel on multiple different machines, i.e., computing devices, or in multiple threads on one or more such machines, to collect statistics.

[0129] After generating the sets of statistics, i.e., 112A-C, for the plurality of actions that can be performed by the agent in response to the current observation generated for the plurality of action selection policies, the system 100 can use the respective sets of statistics to select an action selection policy and an action 114 to be performed by the agent.

[0130] In one described aspect, the system performs “sub-additive planning” to select the action and then uses the action to (i) control the agent interacting with the environment to perform a task in the environment or (ii) assist the agent interacting with the environment to perform a task in the environment.

[0131] “Sub-additive planning” refers to using sets of statistics 112A-C generated for multiple corresponding policies 106A-C to select one of the action selection policies and then using the respective set of statistics for the available actions associated with the selected action selection policy to then select the action 114 to be performed by the agent.

[0132] Further details of using the respective sets of statistics 112A-C for multiple action selection policies to select the action to be performed by the agent through sub-additive planning are described below with reference to FIG. 3. Further details of performing a tree search to generate a set of statistics are described in more detail below with reference to FIG. 4.

[0133] When the system 100 is controlling the agent, the system 100 controls the agent to perform the selected action 114.

[0134] Prior to using the system 100 to select actions for an agent to cause the agent to interact with an environment to perform tasks, the reinforcement learning system 100 or another training system trains the action selection policies 106A-C.

[0135] The system 100 or another training system can (e.g., concurrently) train each action selection policy 106A-C. That is, for each action selection policy, the system 100 interacts with an environment using an agent controlled using the respective action selection policy for an episode to generate experiences. Then, after generating experiences, for each action selection policy, the system computes the gradient of a respective loss function with respect to the action selection neural network 108 parameters. Lastly, the system uses the gradients of the loss to update the parameters of the neural network 108 parameters through an optimization algorithm, e.g., gradient descent.

[0136] Generally, the concurrent training of the action selection policies occurs asynchronously and in parallel. That is, multiple instances of the system 100 can each operate independently to generate experiences, compute gradients of losses, and compute updates of parameters. At any point during asynchronous updates of the instances’ action selection neural networks the learned parameters can be aggregated and scattered across all instances to synchronize the action selection neural network parameters across instances, e.g., average all action selection neural network parameters across instances.

[0137] During one episode of performing a task, while training an action selection policy, the system 100 uses the action selection policy being trained to select actions to be performed by the agent in response to observations Si through ST, i.e., starting from when the environment is in an initial state Si and until the environment reaches a terminal state ST. The terminal state of an episode may be a state in which the specified result has been achieved, or a state that the environment is in after a specified number of actions have been performed after the environment was in the initial state without the specified result.

[0138] For each of the observations encountered during the instance of performing the task, the system 100 generates an experience history that includes the observation state, i.e., a training observation characterizing a training state of the environment, and target action selection outputs generated from outputs of a tree search. For example, the system generates an experience history that includes the observation state s, target policy output TT, target extrinsic state value z, target intrinsic state value za. The experience histories may be stored in a training database.

[0139] The system can then (asynchronously from generating the experience histories), use the experience histories to update the values of the parameters of the action selection neural network 108 that maintains the action selection policy being trained. In particular, system 100 then trains the action selection neural network 108 by adjusting the values of the network parameters so that the action selection outputs more closely matches the target action selection outputs.

[0140] Further details of training the action selection policies are described below with reference to FIG. 5.

[0141] FIG. 2 shows an example action selection neural network 200.

[0142] The action selection neural network 200 can be used to represent multiple action selection policies while using a single set of parameters. In particular, when conditioned on a latent variable 104 corresponding to a particular action selection policy, the action selection network 200 generates action selection outputs 110 associated with the particular policy and the input 102. The latent variable 104 therefore enables the neural network 200 to manage multiple action selection policies within a unified neural network architecture.

[0143] In order to generate the action selection outputs 110 associated with a particular action selection policy with a corresponding particular latent variable 104 value, the main body 202 of the neural network 200 processes the latent variable 104 and the state 102 to generate a latent representation 204 to be processed by various heads, i.e., a policy head 206, an extrinsic state value head 208A, an extrinsic state-action values head 208B, an intrinsic state value head 210A, and an intrinsic state action-values head 210B. The outputs of these heads are the action selection outputs 110 which include a policy output (i.e., a score or probability associated with each action), an predicted extrinsic state value, predicted extrinsic state-action values, an predicted intrinsic state value, and predicted intrinsic state-action values.

[0144] The main body 202 can include fully connected layers, convolution layers, recurrent layers, attention-based layers, and so on, depending on if the input state 102 includes images, videos, text, numerical vectors, or any combination of these data types.

[0145] For example, if a state 102 is represented at least in part by images, then the main body 202 may include convolutional layers, e.g., Conv2D; if a state 102 is represented at least in part by text (or other sequence like data), then the main body 202 may include recurrent layers, e.g., RNN, LSTM, and GRU; and if a state 102 is represented more generically through numerical representations of environment state 102 properties, then the main body 202 may include fully connected layers, or attention-based layers, e.g., scaled dotproduct attention and multi -head attention.

[0146] In some implementations, as described below, during a tree search a dynamics neural network can process the latent representation 204 along with a selected action to generate a predicted latent representation of a state that the environment transitions into as a result of the action being performed.

[0147] FIG. 3 is a flow diagram of an example process 300 for selecting an action through sub-additive planning. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a reinforcement learning system, e.g., the reinforcement learning system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 300.

[0148] The system receives a current observation characterizing a current state of the environment being interacted with by an agent to perform a task (step 302).

[0149] The system, for each of a plurality of action selection policies for selecting actions to be performed by the agent, performs a respective tree search through future states of the environment starting from the current state of the environment and using the action selection policy to generate a respective set of statistics for each of a plurality of actions that can be performed by the agent in response to the current observation (step 304).

[0150] That is, a tree search, e.g., the tree search as described with reference to FIG. 4, for each action selection policy of the system generates statistics for each action available for a response by an agent to the current observation state.

[0151] For example, the statistics can include, for each action selection policy, a respective visit count for each of the plurality of actions that identifies a number of times the action was selected to be performed in response to the current observation during the tree search performed using the action selection policy.

[0152] As another example, the statistics can include, for each action selection policy, a respective value estimate for each of the plurality of actions that identifies an estimate of a value to performing the task of performing the action in response to the current observation as a result of the tree search performed using the action selection policy.

[0153] As another example, the statistics can include, for each action selection policy, (i) a respective value estimate for each of the plurality of actions that identifies an estimate of a value to performing the task of performing the action in response to the current observation as a result of the tree search performed using the action selection policy and / or (ii) a respective exploration term for each of the plurality of actions that identifies an estimate of a value to exploring the environment of performing the action in response to the current observation as a result of the tree search performed using the action selection policy.

[0154] The statistics generated may include any one or more of the above statistics.

[0155] The system selects an action to be performed by the agent using the respective sets of statistics for the plurality of actions generated for the plurality of action selection policies (step 306).

[0156] For example, the system can perform “sub-additive planning.” That is, the system selects an action selection policy according to statistics from respective tree searches and then further selects the action according to the statistics of the selected action selection policy.

[0157] For example, sub-additive planning can involve determining, for each action selection policy, a respective maximum visit count of the respective visit counts for the plurality of actions for the action selection policy and selecting an action selection policy that has a highest respective maximum visit count. That is, sub-additive planning can select the action selection policy according to t «- arg max max Ni(s, a) where i is the index of the selected

[0158] 7 “ action selection policy, j is a variable representing action selection policy indices, a is a variable representing actions, s is a variable representing states, N is the visit count.

[0159] As another example, sub-additive planning can involve determining, for each action selection policy, a respective maximum value, e.g., extrinsic value, intrinsic value, or a combination of both, of the respective value estimates for the plurality of actions for the action selection policy and selecting an action selection policy that has a highest respective maximum value estimate. That is, sub-additive planning can select the action selection policy according to, i <- arg max V (s) where V7(s) = maxaQi (s, a) for which V is the state j value and Q is the action value.

[0160] As another example, sub-additive planning can involve determining, for each action selection policy, a respective maximum value of the respective value estimates for the plurality of actions for the action selection policy. Then, determining an argmax action selection policy that has a highest respective maximum value estimate. Next, identifying each action selection policy that has a respective maximum value that is within a threshold of the highest respective maximum value estimate. And then, selecting, from the identified action selection policies, the action selection policy that has a lowest respective maximum value. That is, sub-additive planning can select the action selection policy according to t <- arg min V (s) where Gap(s) is a function that defines the threshold j E { : (s)>arg max / cKfc(s)-Gap(s) } and k is another variable representing action selection policy indices.

[0161] As another example, sub-additive planning can involve determining, for each action selection policy, a respective maximum difference of the respective value estimates for the plurality of actions for the action selection policy, wherein the respective difference for each action is a difference between the respective value estimate for the action and the respective exploration term for the action. And then, selecting an action selection policy that has a highest respective maximum difference. That is, sub-additive planning can select the action selection policy according to t «- arg max max Qi(s, a) — l (s, a) where t / is the

[0162] Jaexploration term for the action.

[0163] As another example, the sub-additive planning can involve selecting the action selection policy with the maximum score of a custom function, e.g., lower-confidence bound function, that includes visitation counts and predicted action-values.

[0164] Note that the sub-additive planning can involve any one or more of these examples.

[0165] Sub-additive planning reduces the cost and complexity of action selection by efficiently combining the outputs of multiple tree searches from the diverse set of action selection policies. This approach does not require the tree search to reach a terminal or successful state, allowing for action selection without simulating the entire episode.

[0166] FIG. 4 is a flow diagram of an example process 400 for performing a tree search using an action selection policy to generate a respective set of statistics for each of a plurality of actions that can be performed by the agent in response to the current observation environment state. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a reinforcement learning system, e.g., the reinforcement learning system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 400.

[0167] Specifically, the system can perform the process 400 to perform a tree search using action selection outputs generated by an action selection neural network while the action selection neural network is conditioned on the corresponding latent variable for the action selection policy.

[0168] Performing the tree search involves traversing a tree with nodes representing environment states and directed edges representing actions that can be performed by an agent in response to the state represented as the origin node of the edge. The tree search process can therefore be described in terms of state-action pairs, i.e., the state in the pair is the presently visited state and the action in the pair is an action that can be performed by the agent in response to the visited state leading to a new future state.

[0169] As described earlier, the system can maintain data representing a state tree of the environment. The state tree includes nodes that represent states of the environment and directed edges that connect nodes in the tree. An outgoing edge from a first node to a second node in the tree represents an action that was performed in response to an observation characterizing the first state and resulted in the environment transitioning into the second state. In this way the system can maintain statistics of all state-action pairs occurring during a tree search through maintained edge data for each edge in the state tree.

[0170] Depending on the implementation, the tree search can be through a tree of environment states in the state space of the environment, i.e., so that nodes in the tree represent states of the environment, or through a tree with nodes corresponding to latent representations of environment states.

[0171] In some implementations, when tree nodes correspond to latent representations of environment states, performing the tree search involves traversing a tree with nodes representing states in a latent space of latent representations generated by the action selection neural network and using a dynamics neural network that is configured to receive an input latent representation of the input observation and an action to generate a predicted latent representation of a state that the environment transitions into as a result of the action being performed in response to the input observation.

[0172] The system receives data identifying a root node for the search, i.e., a node representing the state characterized by the current observation (step 402).

[0173] That is, the system receives the state of the environment along with the latent variable corresponding to the action selection policy used to perform the tree search and generates action selection outputs. For example, the latent representation of the state can be processed to generate a policy output, a predicted extrinsic state value, a predicted intrinsic state value, etc. using a policy head, an extrinsic value head, an intrinsic value head, etc., e.g., as described with reference to FIG. 2.

[0174] The system traverses the state tree until the state tree reaches a leaf state, i.e., a state that is represented by a leaf node in the state tree (step 404).

[0175] That is, at each in-tree node, i.e., a node encountered starting from the root node until reaching the leaf state, the system selects the edge to be traversed using the edge data for the outgoing edges from the in-tree node representing the in-tree state. The system may select the edge based on the action score or may determine an adjusted action score for selecting an edge.

[0176] For example, the system can select an action for the presently visited state associated with a maximum adjusted action score, e.g., the score determined as the sum of Q value and bonus term U as described earlier, to traverse to a future state. That is, the action selection outputs (e.g., policy output, extrinsic state-action values, and / or intrinsic state-action values) e.g. along with statistics of state-action pairs of the presently visited state (e.g., visit counts of state-action pairs) and exploration terms (e.g., exploration rate) are used to select the action to transition to a future state, i.e., new tree node.

[0177] The system continues traversing the state tree in this manner until a leaf node in the state tree is reached. Generally, a leaf node is a node in the state tree that has no child nodes, i.e., is not connected to any other nodes by an outgoing edge (i.e. an edge directed away from the root node).

[0178] The system then expands the leaf node (step 406).

[0179] To expand the leaf node, the system may add a respective new edge to the state tree for each action that is a valid action to be performed by the agent in response to a leaf observation characterizing the state represented by the leaf node. The system also initializes the edge data for each new edge by setting the visit count and action scores for the new edge to zero.

[0180] The system evaluates the leaf node using the action selection neural network in accordance with the current values of the parameters to generate a respective prior probability for each new edge (step 408). To determine the prior probability for each new edge, the system may process the leaf observation using the action selection neural network and uses the action probabilities from the distribution defined by the network output as the prior probabilities for the corresponding edges. The system may also generate a predicted return (extrinsic, intrinsic, or both) for the leaf observation from the results of the processing of the leaf observation by the neural network.

[0181] The system then updates the edge data for the edges traversed during the search based on the predicted return for the leaf node (step 410).

[0182] In particular, for each edge that was traversed during the search, the system increments the visit count for the edge by a predetermined constant value, e.g., by one. The system also updates the action score for the edge using the predicted expected return for the leaf node by setting the action score equal to the new average of the predicted expected returns of all searches that involved traversing the edge. After completing step 410, the system repeats the described exploration from a root node to a leaf node through steps 402-410, i.e., the look ahead search process, multiple (e.g. many) times, i.e., performs multiple look ahead searches, without resetting the statistics associated with each state-action pair, but instead augmenting them with each run. Once the criterion / criteria for ending the tree search’s look ahead searches, e.g., max number look ahead searches, time limit, convergence of statistics, etc., have been met, the system generates a final respective set of statistics for each of a plurality of actions that can be performed by the agent in response to the current observation.

[0183] As described above, in some cases the system uses a simulated version of the environment to perform the look ahead search, e.g., to identify which state taking an action leads to when in a leaf state, to identify which states taking an action leads to when the outgoing edge for the action is not connected to any node in the tree, or to verify that the edge data for an in-tree node accurately reflects the transitions that will occur when a given action is selected.

[0184] In some implementations, the system distributes the searching of the state tree, i.e., by running multiple different searches in parallel on multiple different machines, i.e., computing devices, or in multiple threads on one or more such machines.

[0185] For example, the system may implement an architecture that includes a master machine that executes the main search. The entire state tree may be stored on the master, which only executes the in-tree phase of each simulation. The leaf positions are communicated to one or more workers, which execute the expansion and evaluation phase of the simulation.

[0186] In some cases, the system does not update the edge data until a predetermined number of look ahead searches have been performed since a most-recent update of the edge data, e.g., to improve the stability of the search process in cases where multiple different searches are being performed in parallel.

[0187] In some implementations, the system trains one action selection policy, i.e., a first action selection policy, using an objective or loss functions that only include extrinsic rewards, and concurrently trains the remainder action selection policies, i.e., second action selection policies, using an objective or loss functions that include respective combinations of extrinsic and intrinsic rewards to create a set of high quality and diverse action selection policies. As a consequence, the first action selection policy achieves the highest extrinsic return among all action selection policies during training. However, in practice, using the final set of all action selection policies, including those that do not individually maximize extrinsic rewards during training, leads to better overall task performance when controlling an agent than using only the first action selection policy to control the agent.

[0188] Generally, when using respective objective or loss functions that includes combinations of extrinsic and intrinsic rewards to train multiple action selectin policies, the final trained set of action selection policies results in multiple distinct, policies. When these policies work in concert to select an action, e.g., the action selection process as described with reference to FIG. 4, they outperform any single policy of the set. When encountering an unfamiliar environment state, having a set of diverse policies to select the best action increases the likelihood of performing a task optimally when compared to selecting the best action provided by a single policy.

[0189] FIG. 5 is a flow diagram of an example process 500 for training an action selection policy. For convenience, the process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a reinforcement learning system, e.g., the reinforcement learning system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 500.

[0190] For a particular action selection policy being trained, the system uses the generated experience history for each of the observations encountered during an instance of performing the task using the action selection policy. While the particular action selection policy being trained determines the selected actions during the instance of performing the task, the other action selection policies can contribute to training, e.g., being used to compute target intrinsic state values as described below.

[0191] The system obtains a training observation characterizing a training state of the environment (step 502). For example, the training observation can be one of various environment states from an experience history, e.g. one stored in a training database of the reinforcement learning system 100.

[0192] The system obtains outputs of a tree search through future states of the environment that is performed (i) starting from the training state of the environment and (ii) using the action selection policy (step 504).

[0193] A tree search, e.g., the tree search as described with reference to FIG. 4, can be performed starting from the training state of the environment and using the action selection policy. That is, the action selection policy that is being trained performs the tree search using any appropriate tree search method to generate outputs, e.g., visit counts for each action available in response to the training state of the environment, extrinsic rewards from the root to the leaf node, intrinsic rewards from the training state root node to a leaf node, etc. The system generates, using the action selection policy, action selection outputs (step

[0194] 506).

[0195] For example, the action selection neural network can process a latent representation of the training state using action selection output heads, e.g., policy head, extrinsic value head, intrinsic value head, etc., as described with reference to FIG. 2, to generate action policy outputs, e.g., policy output, predicted extrinsic state value, predicted intrinsic state value, etc.

[0196] The system generates, from the outputs of the tree search, target action selection outputs for the action selection policy (508).

[0197] For example, if the tree search outputs of step 504 include visit counts for each action available to an agent in response to the training state of the environment, then a target action selection output can include the vector it representing a score distribution over the set of

[0198] N(a) actions, e.g., a respective probability for each action in the set, e.g., n(a) where

[0199] Wb)

[0200] 7t(a) is the probability for action a, IV(a) represents the visit count for action a and N b) denotes the total number of visit counts over all actions.

[0201] As another example, if the output of a tree search’s look ahead search includes extrinsic rewards from the training state root node to a terminal state leaf node, then a target action selection output can include the extrinsic state value z associated with the accumulation of extrinsic rewards.

[0202] As another example, if the output of a tree search’s look ahead search includes intrinsic rewards from the training state root node to a leaf node, then a target action selection output can include the intrinsic state value zd associated with the accumulation of intrinsic rewards. For this particular example, because the intrinsic reward reflects the diversity of behavior of an action selection policy relative to other action selection policies, all action selection policies being considered for the computation of intrinsic rewards perform a tree search and the outputs of these tree searches are included to determine the target intrinsic reward.

[0203] As a particular example of determining the target intrinsic reward from tree searches of the action selection policy being trained and the action selection policies being compared to, the intrinsic reward can be defined (i|jl— v|F) where rds, a) is the intrinsic reward for the currently being trained action selection policy after selecting action a from state s, 4>lis the resultant state after taking action a from state 5, ip is the average resultant state over taking all actions weighted according to probabilities associated with a according to the policy output it, l0is a hyperparameter, the superscript ‘i’ refers to the index representing the action selection policy being trained, and the superscript ‘j ’ refers to the index representing the action selection policy with the most similar ip value to the currently being trained action selection policy. The actions from state s, resultant states , and the policy output it are all determined from outputs of tree searches for each action selection policy. For the purposes of computing the intrinsic reward, “states” can be represented as observations or latent representations of observations generated by the action selection neural network. More generally, the state that is represented by and the states that are used compute the various t|rs can be represented in the computation by any appropriate set of numerical values, e.g., observations or latent representations of observations.

[0204] The system trains the action selection policy using a loss function that is based on the action selection outputs and the target action selection outputs (step 510).

[0205] In order to train an action selection policy the parameters of the action selection neural network are adjusted through any of variety of optimization techniques, e.g., gradient descent methods using backpropagation, to minimize any of a variety of loss functions that depend on predicted action selection outputs and target action selection outputs.

[0206] For example, the parameters 0 of an action selection neural network that processes a training state 5 to generate action selection outputs that include a probability distribution p over a set of actions, a predicted extrinsic state value v for starting at state s, and a predicted intrinsic state value va for starting at state .s; i.e, f^s) = ( / ?, v, v< / ), can be adjusted to generate outputs more similar to target action selection outputs from an experience history that include a target probability distribution n over a set of actions and a target extrinsic state value z for starting at state 5, and a target intrinsic state value zd for starting at state .s.

[0207] For this example, the training data is a set that includes a corresponding training state 5 and corresponding target action selection outputs (TT, zd, z) to create a set (.s, TT, Z, zd). The parameters 0 can then be adjusted to minimize a loss function L such as, for example, L = (zd - Vd)2+ (z - v)2- 7TTlog p.

[0208] In addition, the loss function can also be part of a greater objective function that includes regularization terms for the learnable parameters, such as the L-l (LASSO regularization) and L-2 (ridge regression regularization) variants of the L-p norm.

[0209] The loss functions, e.g., L = (zd - v<y)2+ (z - v)2- 7rTlog / ?, that include combinations of intrinsic rewards and extrinsic rewards are responsible for training an action selection policy to have unique behavior among the set of action selection policies while also performing a task near optimally during training. Minimizing loss on intrinsic rewards encourage unique behavior of the action selection policy while minimizing loss on extrinsic rewards encourages the action selection policy to still perform the task well.

[0210] As another example, a reward r’ = (1 — )rintrinsic+ (Arextrmsic^canbe usec[ t0train a neural network to produce an action selection output state value v’ that approximates target state value z’ =f=Qytr't+1using a loss function that includes the term (z ’ - v ’)2.

[0211] Generally, training includes minimizing the objective function over many experience histories.

[0212] The above description describes that, after training, sub-additive planning is used to select an action.

[0213] In some other implementations, however, the “first” action selection policy, i.e., the one trained using only extrinsic rewards, is used to select actions after training. Using solely the first action selection policy is appropriate because training using only extrinsic rewards means the action selection policy is directly optimized for task performance and, therefore, has no dependence on the other action selection policies and will outperform any other single action selection policy. In addition, the sole use of the first action selection policy from the set of trained action selection policies offers latency and memory advantages for action selection while still achieving acceptable task performance.

[0214] FIG. 6 shows an example of the performance of the describe techniques.

[0215] FIG. 6 shows an example 600 of the performance of the described techniques (subadditive action planning using multiple action selection policies) relative to a conventional technique (action planning using a single action selection policy trained solely).

[0216] In particular, the example 600 shows a summary of chess matchups (e.g., overall winrate and relative Elo score, i.e., a metric for player skill level) between a first agent controlled using a diverse set of action selection policies relative to a second agent controlled by the conventional single action selection policy. In particular, the columns refer to the first agent being controlled by “player 0”, i.e., a first action selection policy trained solely using extrinsic rewards of multiple action selection policies that are diverse, “sub-additive planning”, i.e., a diverse set of action selection policies that uses sub-additive planning to select actions, or “max-over-latents”, i.e., a diverse set of action selection policies that uses an oracle, i.e., an entity that has perfect knowledge of the consequences of selecting actions and action selection policies, to evaluate each action selection policy to select the action selection policy and corresponding action that will lead to successful completion of a task.

[0217] As can be seen from example 600, the “sub-additive planning” column shows how a diverse set of action selection policies that selects actions using sub-additive planning outperforms the conventional single policy, as indicated by the greater than 50 winrate and positive Elo score. Additionally, using only the first action selection policy of the set, ’’player 0”, still outperforms the conventional policy while requiring similar computational resources to execute as the conventional policy. Furthermore, using “sub-additive planning” performs almost as well as “max-over-latent” while requiring far fewer computational resources to execute.

[0218] As can be seen from example 600, using a diverse set of action selection policies trained on combinations of extrinsic and intrinsic rewards along with sub-additive planning to select actions for an agent provides a performance advantage over using a single conventional action skeleton policy.

[0219] Examples of environments, actions, and agents that the system can control follows.

[0220] When the system is assisting the agent, the system provides, to the agent, information about how to perform the task that is generated using the selected action. For example, the system can use a generative neural network to map the selected action to a natural language output, a speech output, an image output, or a video output, that identifies the selected action and, optionally, describes how to perform the select action and provide the output for presentation to the user.

[0221] For example, the agent that is assisted is a human. For example assisting the agent, i.e. the human, using the answer can comprise communicating with a human user of a digital assistant (also referred to as a virtual assistant) such as a smart speaker or display, mobile, or other device, that implements the method.

[0222] In more detail, in some implementations the agent comprises a human user of a digital assistant such as a smart speaker, smart display, or other device. Then, information defining the task can be obtained from the digital assistant, and the digital assistant can be used to provide information (instructions) to the user based on the latent vector. For example, this may comprise receiving, at the digital assistant, a request from the user for assistance and determining, in response to the request, one or more tasks for the user to perform, e.g., steps or sub-tasks of an overall task. Then for one or more tasks of the series of tasks, e.g., for each task, e.g., until a final task of the series the digital assistant can be used to output to the user information indicating how to perform the task. This may be done using natural language, e.g., on a display and / or using a speech synthesis subsystem of the digital assistant. Visual, e.g., video, and / or audio observations of the user performing the task may be captured, e.g., using the digital assistant.

[0223] As an illustrative example a user may be interacting with a digital assistant and ask for help performing an overall task consisting of multiple steps, e.g., cooking a pasta dish. While the user performs the task, the digital assistant receives audio and / or video inputs representative of the user's progress on the task, e.g., images or video or sound clips of the user cooking. The digital assistant uses a system as described above, in particular by providing it with the captured audio and / or video to determine how the user should complete each step.

[0224] In a further aspect there is provided a digital assistant device including a system as described above. The digital assistant can also include a user interface to enable a user to request assistance and to output information. In implementations this is a natural language user interface and may comprise a keyboard, voice input-output subsystem, and / or a display. The digital assistant can further include an assistance subsystem configured to determine, in response to the request, a series of tasks for the user to perform. In implementations this may comprise a generative (large) language model, in particular for dialog, e.g., a conversation agent such as LaMDA or Gemini. The digital assistant can have an observation capture subsystem to capture visual and / or audio observations of the user performing a task; and an interface for the above-described language model neural network (which may be implemented locally or remotely). The digital assistant can also have an assistance control subsystem configured to assist the user. The assistance control subsystem can be configured to perform the steps described above, for one or more tasks, e.g., of a series of tasks, e.g., until a final task of the series. More particularly the assistance control subsystem can capture, using the observation capture subsystem, visual or audio observations of the user performing the task, determine how to perform the task, and provide information about how to perform the task.

[0225] Consider, for instance, a user requesting help with assembling furniture. The digital assistant, using its camera, captures visual observations of the user's progress and the current state of the assembly. This visual input is processed as the current observation of the environment. The system maintains a plurality of action selection policies, each trained on diverse assembly strategies and common furniture types. For each policy, the system performs a respective tree search, simulating potential outcomes of different actions the user could take. These simulations, guided by a dynamics neural network trained on assembly instructions, generate statistics for each possible action, such as the estimated probability of successful assembly and the expected time to completion. The digital assistant then analyzes these statistics and selects the action with the highest overall score, balancing factors like probability of success and efficiency. The assistant then guides the user, suggesting this next best step through natural language instructions, images, and / or augmented reality overlays.

[0226] Some examples of the types of agents the system can control now follow.

[0227] In some implementations, the environment is a real-world environment, the agent is a mechanical agent interacting with the real-world environment, e.g., a robot or an autonomous or semi-autonomous land, air, or sea vehicle operating in or navigating through the environment, and the actions are actions taken by the mechanical agent in the real-world environment to perform the task. For example, the agent may be a robot interacting with the environment to accomplish a specific task, e.g., to locate an object of interest in the environment or to move an object of interest to a specified location in the environment or to navigate to a specified destination in the environment.

[0228] In these implementations, the observations may include, e.g., one or more of: images, object position data, and sensor data to capture observations as the agent interacts with the environment, for example sensor data from an image, distance, or position sensor or from an actuator. For example in the case of a robot, the observations may include data characterizing the current state of the robot, e.g., one or more of: joint positionjoint velocityjoint force, torque or acceleration, e.g., gravity-compensated torque feedback, and global or relative pose of an item held by the robot. In the case of a robot or other mechanical agent or vehicle the observations may similarly include one or more of the position, linear or angular velocity, force, torque or acceleration, and global or relative pose of one or more parts of the agent. The observations may be defined in 1, 2 or 3 dimensions, and may be absolute and / or relative observations. The observations may also include, for example, sensed electronic signals such as motor current or a temperature signal; and / or image or video data for example from a camera or a LIDAR sensor, e.g., data from sensors of the agent or data from sensors that are located separately from the agent in the environment.

[0229] In these implementations, the actions may be control signals to control the robot or other mechanical agent, e.g., torques for the joints of the robot or higher-level control commands, or the autonomous or semi-autonomous land, air, sea vehicle, e.g., torques to the control surface or other control elements, e.g., steering control elements of the vehicle, or higher-level control commands. The control signals can include for example, position, velocity, or force / torque / accel eration data for one or more joints of a robot or parts of another mechanical agent. The control signals may also or instead include electronic control data such as motor control data, or more generally data for controlling one or more electronic devices within the environment the control of which has an effect on the observed state of the environment. For example in the case of an autonomous or semi-autonomous land or air or sea vehicle the control signals may define actions to control navigation, e.g., steering, and movement, e.g., braking and / or acceleration of the vehicle.

[0230] In some implementations the environment is a simulation of the above-described real- world environment, and the agent is implemented as one or more computers interacting with the simulated environment. For example the simulated environment may be a simulation of a robot or vehicle and the reinforcement learning system may be trained on the simulation and then, once trained, used in the real-world.

[0231] In some implementations the environment is a real-world manufacturing environment for manufacturing a product, such as a chemical, biological, or mechanical product, or a food product. As used herein a “manufacturing” a product also includes refining a starting material to create a product, or treating a starting material, e.g., to remove pollutants, to generate a cleaned or recycled product. The manufacturing plant may comprise a plurality of manufacturing units such as vessels for chemical or biological substances, or machines, e.g., robots, for processing solid or other materials. The manufacturing units are configured such that an intermediate version or component of the product is moveable between the manufacturing units during manufacture of the product, e.g., via pipes or mechanical conveyance. As used herein manufacture of a product also includes manufacture of a food product by a kitchen robot.

[0232] The agent may comprise an electronic agent configured to control a manufacturing unit, or a machine such as a robot, that operates to manufacture the product. That is, the agent may comprise a control system configured to control the manufacture of the chemical, biological, or mechanical product. For example the control system may be configured to control one or more of the manufacturing units or machines or to control movement of an intermediate version or component of the product between the manufacturing units or machines.

[0233] As one example, a task performed by the agent may comprise a task to manufacture the product or an intermediate version or component thereof. As another example, a task performed by the agent may comprise a task to control, e.g., minimize, use of a resource such as a task to control electrical power consumption, or water consumption, or the consumption of any material or consumable used in the manufacturing process. The actions may comprise control actions to control the use of a machine or a manufacturing unit for processing a solid or liquid material to manufacture the product, or an intermediate or component thereof, or to control movement of an intermediate version or component of the product within the manufacturing environment, e.g., between the manufacturing units or machines. In general the actions may be any actions that have an effect on the observed state of the environment, e.g., actions configured to adjust any of the sensed parameters described below. These may include actions to adjust the physical or chemical conditions of a manufacturing unit, or actions to control the movement of mechanical parts of a machine or joints of a robot. The actions may include actions imposing operating conditions on a manufacturing unit or machine, or actions that result in changes to settings to adjust, control, or switch on or off the operation of a manufacturing unit or machine.

[0234] The rewards or return may relate to a metric of performance of the task. For example in the case of a task that is to manufacture a product the metric may comprise a metric of a quantity of the product that is manufactured, a quality of the product, a speed of production of the product, or to a physical cost of performing the manufacturing task, e.g., a metric of a quantity of energy, materials, or other resources, used to perform the task. In the case of a task that is to control use a resource the matric may comprise any metric of usage of the resource.

[0235] In general observations of a state of the environment may comprise any electronic signals representing the functioning of electronic and / or mechanical items of equipment. For example a representation of the state of the environment may be derived from observations made by sensors sensing a state of the manufacturing environment, e.g., sensors sensing a state or configuration of the manufacturing units or machines, or sensors sensing movement of material between the manufacturing units or machines. As some examples such sensors may be configured to sense mechanical movement or force, pressure, temperature; electrical conditions such as current, voltage, frequency, impedance; quantity, level, flow / movement rate or flow / movement path of one or more materials; physical or chemical conditions, e.g., a physical state, shape or configuration or a chemical state such as pH; configurations of the units or machines such as the mechanical configuration of a unit or machine, or valve configurations; image or video sensors to capture image or video observations of the manufacturing units or of the machines or movement; or any other appropriate type of sensor. In the case of a machine such as a robot the observations from the sensors may include observations of position, linear or angular velocity, force, torque or acceleration, or pose of one or more parts of the machine, e.g., data characterizing the current state of the machine or robot or of an item held or processed by the machine or robot. The observations may also include, for example, sensed electronic signals such as motor current or a temperature signal, or image or video data for example from a camera or a LIDAR sensor. Sensors such as these may be part of or located separately from the agent in the environment.

[0236] In some implementations the environment is the real-world environment of a service facility comprising a plurality of items of electronic equipment, such as a server farm or data center, for example a telecommunications data center, or a computer data center for storing or processing data, or any service facility. The service facility may also include ancillary control equipment that controls an operating environment of the items of equipment, for example environmental control equipment such as temperature control, e.g., cooling equipment, or air flow control or air conditioning equipment. The task may comprise a task to control, e.g., minimize, use of a resource, such as a task to control electrical power consumption, or water consumption. The agent may comprise an electronic agent configured to control operation of the items of equipment, or to control operation of the ancillary, e.g., environmental, control equipment.

[0237] In general the actions may be any actions that have an effect on the observed state of the environment, e.g., actions configured to adjust any of the sensed parameters described below. These may include actions to control, or to impose operating conditions on, the items of equipment or the ancillary control equipment, e.g., actions that result in changes to settings to adjust, control, or switch on or off the operation of an item of equipment or an item of ancillary control equipment.

[0238] In general observations of a state of the environment may comprise any electronic signals representing the functioning of the facility or of equipment in the facility. For example a representation of the state of the environment may be derived from observations made by any sensors sensing a state of a physical environment of the facility or observations made by any sensors sensing a state of one or more of items of equipment or one or more items of ancillary control equipment. These include sensors configured to sense electrical conditions such as current, voltage, power or energy; a temperature of the facility; fluid flow, temperature or pressure within the facility or within a cooling system of the facility; or a physical facility configuration such as whether or not a vent is open.

[0239] The rewards or return may relate to a metric of performance of the task. For example in the case of a task to control, e.g., minimize, use of a resource, such as a task to control use of electrical power or water, the metric may comprise any metric of use of the resource. In some implementations the environment is the real-world environment of a power generation facility, e.g., a renewable power generation facility such as a solar farm or wind farm. The task may comprise a control task to control power generated by the facility, e.g., to control the delivery of electrical power to a power distribution grid, e.g., to meet demand or to reduce the risk of a mismatch between elements of the grid, or to maximize power generated by the facility. The agent may comprise an electronic agent configured to control the generation of electrical power by the facility or the coupling of generated electrical power into the grid. The actions may comprise actions to control an electrical or mechanical configuration of an electrical power generator such as the electrical or mechanical configuration of one or more renewable power generating elements, e.g., to control a configuration of a wind turbine or of a solar panel or panels or mirror, or the electrical or mechanical configuration of a rotating electrical power generation machine. Mechanical control actions may, for example, comprise actions that control the conversion of an energy input to an electrical energy output, e.g., an efficiency of the conversion or a degree of coupling of the energy input to the electrical energy output. Electrical control actions may, for example, comprise actions that control one or more of a voltage, current, frequency or phase of electrical power generated.

[0240] The rewards or return may relate to a metric of performance of the task. For example in the case of a task to control the delivery of electrical power to the power distribution grid the metric may relate to a measure of power transferred, or to a measure of an electrical mismatch between the power generation facility and the grid such as a voltage, current, frequency or phase mismatch, or to a measure of electrical power or energy loss in the power generation facility. In the case of a task to maximize the delivery of electrical power to the power distribution grid the metric may relate to a measure of electrical power or energy transferred to the grid, or to a measure of electrical power or energy loss in the power generation facility.

[0241] In general observations of a state of the environment may comprise any electronic signals representing the electrical or mechanical functioning of power generation equipment in the power generation facility. For example a representation of the state of the environment may be derived from observations made by any sensors sensing a physical or electrical state of equipment in the power generation facility that is generating electrical power, or the physical environment of such equipment, or a condition of ancillary equipment supporting power generation equipment. Such sensors may include sensors configured to sense electrical conditions of the equipment such as current, voltage, power or energy; temperature or cooling of the physical environment; fluid flow; or a physical configuration of the equipment; and observations of an electrical condition of the grid, e.g., from local or remote sensors. Observations of a state of the environment may also comprise one or more predictions regarding future conditions of operation of the power generation equipment such as predictions of future wind levels or solar irradiance or predictions of a future electrical condition of the grid.

[0242] As another example, the environment may be a chemical synthesis or protein folding environment such that each state is a respective state of a protein chain or of one or more intermediates or precursor chemicals and the agent is a computer system for determining how to fold the protein chain or synthesize the chemical. In this example, the actions are possible folding actions for folding the protein chain or actions for assembling precursor chemicals / intermediates and the result to be achieved may include, e.g., folding the protein so that the protein is stable and so that it achieves a particular biological function or providing a valid synthetic route for the chemical. As another example, the agent may be a mechanical agent that performs or controls the protein folding actions or chemical synthesis steps selected by the system automatically without human interaction. The observations may comprise direct or indirect observations of a state of the protein or chemical / intermediates / precursors and / or may be derived from simulation.

[0243] In a similar way the environment may be a drug design environment such that each state is a respective state of a potential pharmaceutically active compound and the agent is a computer system for determining elements of the pharmaceutically active compound and / or a synthetic pathway for the pharmaceutically active compound. The drug / synthesis may be designed based on a reward derived from a target for the drug, for example in simulation. As another example, the agent may be a mechanical agent that performs or controls synthesis of the drug.

[0244] In some further applications, the environment is a real-world environment and the agent manages distribution of tasks or workloads, e.g., processes, jobs, virtual machines, and so on, across computing resources, e.g., on a mobile device and / or in a data center. In these implementations, the actions may include assigning tasks to particular computing resources.

[0245] For example, consider a large cloud data center hosting thousands of virtual machines (VMs) running various applications for different clients. The goal may be to efficiently allocate these VMs across physical servers to maximize resource utilization, minimize energy consumption, and ensure each application's performance requirements are met. The Al agent, observing server load, memory usage, network bandwidth, and application performance, may take actions such as placing new VMs on specific servers, migrating existing VMs, or adjusting resource allocation for individual VMs. The agent's task is to learn an optimal allocation strategy, guided by rewards that reflect high server utilization, low energy consumption, and meeting performance targets. Intrinsic rewards may encourage the agent to explore diverse allocation strategies, potentially discovering novel solutions for workload distribution. To make allocation decisions, the agent employs sub-additive planning: it utilizes multiple, diverse action selection policies, each trained to handle workload distribution differently. When a decision is needed, the agent performs independent tree searches using each policy, simulating potential outcomes of different VM allocation choices. These simulations leverage a dynamics neural network trained on historical data center performance, allowing the agent to anticipate the impact of its actions on resource utilization, energy consumption, and application performance. The agent then selects the policy that promises the highest expected reward, considering both extrinsic and intrinsic rewards, and executes the action recommended by that policy. This approach allows the agent to leverage the strengths of diverse strategies, adapting its decision-making to the dynamic complexities of the data center environment. This example demonstrates how the described techniques can be applied to complex resource management problems in dynamic computing environments, leveraging multiple action selection policies and sub-additive planning to achieve significant improvements in efficiency, cost-effectiveness, and performance.

[0246] As further example, the actions may include presenting advertisements, the observations may include advertisement impressions or a click-through count or rate, and the reward may characterize previous selections of items or content taken by one or more users.

[0247] In some cases, the observations may include textual or spoken instructions provided to the agent by a third-party (e.g., an operator of the agent). For example, the agent may be an autonomous vehicle, and a user of the autonomous vehicle may provide textual or spoken instructions to the agent (e.g., to navigate to a particular location).

[0248] As another example the environment may be an electrical, mechanical or electromechanical design environment, e.g., an environment in which the design of an electrical, mechanical or electro-mechanical entity is simulated. The simulated environment may be a simulation of a real-world environment in which the entity is intended to work. The task may be to design the entity. The observations may comprise observations that characterize the entity, i.e., observations of a mechanical shape or of an electrical, mechanical, or electromechanical configuration of the entity, or observations of parameters or properties of the entity. The actions may comprise actions that modify the entity, e.g., that modify one or more of the observations. The rewards or return may comprise one or more metric of performance of the design of the entity. For example rewards or return may relate to one or more physical characteristics of the entity such as weight or strength or to one or more electrical characteristics of the entity such as a measure of efficiency at performing a particular function for which the entity is designed. The design process may include outputting the design for manufacture, e.g., in the form of computer executable instructions for manufacturing the entity. The process may include making the entity according to the design. Thus a design an entity may be optimized, e.g., by reinforcement learning, and then the optimized design output for manufacturing the entity, e.g., as computer executable instructions; an entity with the optimized design may then be manufactured.

[0249] As a particular example, the entity may be a computer chip and the task may be to generate a chip design for the computer chip. For example, the task may be to generate an optimized logic circuit or to generate an optimized chip floorplan for the computer chip. In these cases, each action modifies the current design of the chip, e.g., by adding a component to the design or by modifying an existing component in the design.

[0250] As another example, the environment may be an algorithm optimization environment. In this case, the task can be to generate a computer program that optimizes a target algorithm for execution on a target processor, each state of the environment can correspond to a respective state of the generation of the computer program, and the actions can each apply a respective modification to the computer program. An example of such an environment is described in DOI: 10.1038 / s41586-023 -06004-9.

[0251] In another case of the environment being an algorithm optimization environment, the task can be to generate a sequence of modifications to a tensor that represent an algorithm that optimizes a target algorithm for execution on a target processor, each state of the environment can correspond to a respective state of the tensor, and the actions can each apply a respective modification to the tensor. An example of such an environment is described in DOI: 10.1038 / s41586-022-05172-4.

[0252] As another example, the environment may be a compression environment, e.g., a software system running on one or more computers for compressing a data item, e.g., an image, a video, an audio sample, and so on. In this case, each action can modify the current compression of the data item. For example, each action can correspond to a different parameter of the compression algorithm. As a particular example, the actions can correspond to quantization parameter values, block partitioning values, reference frame selections, or some combination. For example, a video compression system may aim to achieve the highest possible visual quality at a target bitrate. The Al agent, observing the characteristics of the video frames and the current compression settings, may take actions to adjust these parameters. The agent's task is to learn an optimal compression strategy, guided by rewards that reflect both the achieved compression ratio and the perceived visual quality of the compressed video. Intrinsic rewards may encourage the agent to explore diverse compression strategies, potentially discovering novel combinations of parameter settings that achieve a better balance between compression efficiency and visual fidelity. To make compression decisions, the agent employs sub-additive planning: it utilizes multiple, diverse action selection policies, each trained to prioritize different aspects of the compression process. When a decision is needed, the agent performs independent tree searches using each policy, simulating potential outcomes of different compression parameter choices. These simulations leverage a dynamics neural network trained on video content, allowing the agent to anticipate the impact of its actions on both the compression ratio and the resulting visual quality. The agent then selects the policy that promises the highest expected reward, considering both extrinsic and intrinsic rewards, and executes the action recommended by that policy. This approach allows the agent to leverage the strengths of diverse strategies, adapting its decision-making to the specific characteristics of the video content being compressed. This example demonstrates how the described techniques can be applied to optimize complex compression algorithms, leveraging multiple action selection policies and sub-additive planning to achieve superior compression performance while maintaining high visual quality. As previously described the environment may be a simulated environment. Generally in the case of a simulated environment the observations may include simulated versions of one or more of the previously described observations or types of observations and the actions may include simulated versions of one or more of the previously described actions or types of actions. For example the simulated environment may be a motion simulation environment, e.g., a driving simulation or a flight simulation, and the agent may be a simulated vehicle navigating through the motion simulation. In these implementations, the actions may be control inputs to control the simulated user or simulated vehicle. Generally the agent may be implemented as one or more computers interacting with the simulated environment.

[0253] The simulated environment may be a simulation of a particular real-world environment and agent. For example, the system may be used to select actions in the simulated environment during training or evaluation of the system and, after training, or evaluation, or both, are complete, may be deployed for controlling a real-world agent in the particular real -world environment that was the subject of the simulation. This can avoid unnecessary wear and tear on and damage to the real-world environment or real-world agent and can allow the control neural network to be trained and evaluated on situations that occur rarely or are difficult or unsafe to re-create in the real -world environment. For example the system may be partly trained using a simulation of a mechanical agent in a simulation of a particular real-world environment, and afterwards deployed to control the real mechanical agent in the particular real-world environment. Thus in such cases the observations of the simulated environment relate to the real-world environment, and the selected actions in the simulated environment relate to actions to be performed by the mechanical agent in the real- world environment.

[0254] Optionally, in any of the above implementations, the observation at any given time step may include data from a previous time step that may be beneficial in characterizing the environment, e.g., the action performed at the previous time step, the reward received at the previous time step, or both.

[0255] This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0256] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0257] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0258] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0259] In this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0260] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers. Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0261] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0262] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0263] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.

[0264] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework or a Jax framework.

[0265] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0266] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0267] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination. Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0268] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

CLAIMS1. A method performed by one or more computers, the method comprising: receiving a current observation characterizing a current state of an environment being interacted with by an agent to perform a task; for each of a plurality of action selection policies for selecting actions to be performed by the agent, performing a respective tree search through future states of the environment starting from the current state of the environment and using the action selection policy to generate a respective set of statistics for each of a plurality of actions that can be performed by the agent in response to the current observation; and selecting an action to be performed by the agent using the respective sets of statistics for the plurality of actions generated for the plurality of action selection policies.

2. The method of claim 1, wherein: each action selection policy has a corresponding latent variable that represents the action selection policy, and each action selection policy is defined by action selection outputs generated by an action selection neural network by processing an action selection input that includes an input observation and the corresponding latent variable for the action selection policy.

3. The method of claim 2, wherein the corresponding latent variable for each action selection policy is a one-hot vector that identifies the action selection policy.

4. The method of claim 2 or claim 3, wherein performing a respective tree search through future states of the environment starting from the current state of the environment and using the action selection policy to generate a respective set of statistics for each of a plurality of actions that can be performed by the agent in response to the current observation comprises: performing the respective tree search using action selection outputs generated by the action selection neural network while the action selection neural network is conditioned on the corresponding latent variable for the action selection policy.

5. The method of any preceding claim, wherein selecting an action to be performed by the agent using the respective sets of statistics for the plurality of actions generated for theplurality of action selection policies comprises: selecting one of the plurality of action selection policies using the respective sets of statistics for the plurality of actions generated for the plurality of action selection policies; and selecting the action to be performed by the agent using the respective set of statistics for the plurality of actions generated for the selected action selection policy.

6. The method of claim 5, wherein selecting the action to be performed by the agent using the respective set of statistics for the plurality of actions generated for the selected action selection policy comprises: determining a respective likelihood for each action in the plurality of actions from the respective set of statistics for the plurality of actions generated for the selected action selection policy; and selecting the action using the respective likelihoods.

7. The method of any preceding claim, wherein the respective sets of statistics for the plurality of actions generated for the plurality of action selection policies comprise: for each action selection policy, a respective visit count for each of the plurality of actions that identifies a number of times the action was selected to be performed in response to the current observation during the tree search performed using the action selection policy.

8. The method of claim 7 when dependent on claim 5, wherein selecting one of the plurality of action selection policies using the respective sets of statistics for the plurality of actions generated for the plurality of action selection policies comprises: determining, for each action selection policy, a respective maximum visit count of the respective visit counts for the plurality of actions for the action selection policy; and selecting an action selection policy that has a highest respective maximum visit count.

9. The method of any one of claims 1-8, wherein the respective sets of statistics for the plurality of actions generated for the plurality of action selection policies comprise: for each action selection policy, a respective value estimate for each of the plurality of actions that identifies an estimate of a value to performing the task of performing the action in response to the current observation as a result of the tree search performed using the actionselection policy.

10. The method of claim 9 when dependent on claim 5, wherein selecting one of the plurality of action selection policies using the respective sets of statistics for the plurality of actions generated for the plurality of action selection policies comprises: determining, for each action selection policy, a respective maximum value of the respective value estimates for the plurality of actions for the action selection policy; and selecting an action selection policy that has a highest respective maximum value estimate.

11. The method of claim 9 when dependent on claim 5, wherein selecting one of the plurality of action selection policies using the respective sets of statistics for the plurality of actions generated for the plurality of action selection policies comprises: determining, for each action selection policy, a respective maximum value of the respective value estimates for the plurality of actions for the action selection policy; determining an argmax action selection policy that has a highest respective maximum value estimate; identifying each action selection policy that has a respective maximum value that is within a threshold of the highest respective maximum value estimate; and selecting, from the identified action selection policies, the action selection policy that has a lowest respective maximum value.

12. The method of any one of claims 1-11, wherein the respective sets of statistics for the plurality of actions generated for the plurality of action selection policies comprise: for each action selection policy, (i) a respective value estimate for each of the plurality of actions that identifies an estimate of a value to performing the task of performing the action in response to the current observation as a result of the tree search performed using the action selection policy and (ii) a respective exploration term for each of the plurality of actions that identifies an estimate of a value to exploring the environment of performing the action in response to the current observation as a result of the tree search performed using the action selection policy.

13. The method of claim 12 when dependent on claim 5, wherein selecting one of the plurality of action selection policies using the respective sets of statistics for the plurality ofactions generated for the plurality of action selection policies comprises: determining, for each action selection policy, a respective maximum difference of the respective value estimates for the plurality of actions for the action selection policy, wherein the respective difference for each action is a difference between the respective value estimate for the action and the respective exploration term for the action; and selecting an action selection policy that has a highest respective maximum difference.

14. The method of any preceding claim, when dependent on claim 2, wherein the action selection neural network comprises a policy output and the action selection neural network is configured to: process the action selection input that includes an input observation and the corresponding latent variable for the action selection policy to generate a latent representation; and process the latent representation using a policy head to generate the policy output.

15. The method of claim 14, wherein the action selection neural network is further configured to: process the latent representation using an intrinsic value head to generate an intrinsic value estimate that estimates a value of the current observation to the action selection policy being diverse relative to the other action selection policies in the plurality of action selection policies.

16. The method of claim 14 or claim 15, wherein the action selection neural network is further configured to: process the latent representation using an extrinsic value head to generate an extrinsic value estimate that estimates a value of the current observation to successfully performing the task using the action selection policy.

17. The method of any one of claims 14-16, wherein performing a respective tree search through future states of the environment starting from the current state of the environment and using the action selection policy to generate a respective set of statistics for each of a plurality of actions that can be performed by the agent in response to the current observation comprises performing the tree search by traversing a tree with nodes representing states in astate space of states of the environment.

18. The method of any one of claims 14-16, wherein performing a respective tree search through future states of the environment starting from the current state of the environment and using the action selection policy to generate a respective set of statistics for each of a plurality of actions that can be performed by the agent in response to the current observation comprises performing the tree search by traversing a tree with nodes representing states in a latent space of latent representations generated by the action selection neural network and using a dynamics neural network that is configured to receive an input latent representation of the input observation and an action and to generate a predicted latent representation of a state that the environment transitions into as a result of the action being performed in response to the input observation.

19. The method of any one of claims 17 or 18, wherein performing the tree search comprises: evaluating leaf nodes of the tree using a policy output generated from a latent representation of a state corresponding to the leaf nodes.

20. The method of claim 19, when dependent on claim 16, wherein performing the tree search comprises: evaluating leaf nodes of the tree using a value generated from a value estimate and an intrinsic value estimate generated from a latent representation of a state corresponding to the leaf nodes.

21. The method of any preceding claim, further comprising: causing the agent to perform the selected action.

22. The method of any preceding claim, further comprising: providing, to the agent, information identifying the selected action.

23. A method performed by one or more computers, the method comprising: training a plurality of action selection policies for controlling an agent interacting with an environment to perform a task, wherein: the plurality of action selection policies comprise a first action selection policy and aplurality of second action selection policies, each second action selection policy is trained to optimize a respective second objective that depends on a respective reward that is a combination of an extrinsic reward for the task and an intrinsic reward that measures a diversity of the plurality of action selection policies, and the first action selection policy is trained to optimize a respective first objective that depends on the extrinsic reward but not on the intrinsic reward; and selecting the first action selection policy as a final policy for controlling the agent.

24. The method of claim 23, further comprising: controlling the agent to perform the task using the first action selection policy.

25. The method of claim 23 or claim 24, wherein: each action selection policy has a corresponding latent variable that represents the action selection policy, each action selection policy is defined by action selection outputs generated by an action selection neural network by processing an action selection input that includes an input observation and the corresponding latent variable for the action selection policy, and training each of the plurality of action selection policies comprises training the action selection neural network on outputs generated by the action selection neural network while the action selection neural network is conditioned on inputs that include the corresponding latent variable for the action selection policy.

26. The method of any one of claims 23-25, wherein training a plurality of action selection policies comprises, for each of the action selection policies: obtaining a training observation characterizing a training state of the environment; obtaining outputs of a tree search through future states of the environment that is performed (i) starting from the training state of the environment and (ii) using the action selection policy; generating, using the action selection policy, an action selection output; generating, from the outputs of the tree search, a target action selection output for the action selection policy; and training the action selection policy using a loss function that is based on the action selection output and the target action selection output.

27. The method of any preceding claim, wherein the agent is a mechanical agent and the environment is a real-world environment.

28. The method of claim 27, wherein the agent is a robot.

29. The method of any preceding claim, wherein the environment is a real-world environment of a service facility comprising a plurality of items of electronic equipment and the agent is an electronic agent configured to control operation of the service facility.

30. The method of any preceding claim, wherein the environment is a real-world manufacturing environment for manufacturing a product and the agent comprises an electronic agent configured to control a manufacturing unit or a machine that operates to manufacture the product.

31. The method of any preceding claim, wherein the environment is a simulation of a real-world environment, wherein the method is performed during training of the action selection policies, and wherein the method further comprises: after the training, controlling a real-world agent in the real-world environment using one or more of the action selection policies.

32. The method of claim 31, wherein the agent comprises a user of a digital assistant, the method comprising: obtaining information defining the task from the digital assistant; and using the digital assistant to provide information about how to perform the task to the user.

33. The method of claim 32, further comprising receiving, at the digital assistant, a request from the user for assistance; determining, in response to the request, that the user should perform the task; and outputting, from the digital assistant to the user, an indication of the task to be performed; wherein the current observation is a visual or audio observation or both of the user performing the task captured by the digital assistant.

34. The method of any preceding claim, wherein: the environment is a data item compression environment; the task is to compress an input data item, each state of the environment corresponds to a respective state of the compression of the input data item; and the actions correspond to encoding decisions during the compression of the input data item.

35. The method of any preceding claim, wherein: the environment is a chip design environment; the task is to generate a chip design for a computer chip that satisfies an input specification, the state of the environment corresponds to a respective state of the chip design; and the actions correspond to chip design decisions during the generation of the chip design.

36. The method of any preceding claim, wherein: the environment is an algorithm optimization environment; the task is to generate a computer program that optimizes a target algorithm for execution on a target processor, each state of the environment corresponds to a respective state of the generation of the computer program; and the actions each apply a respective modification to the computer program.

37. The method of any preceding claim, wherein: the environment is an algorithm optimization environment; the task is to generate a sequence of modifications to a tensor that represent an algorithm that optimizes a target algorithm for execution on a target processor, each state of the environment corresponds to a respective state of the tensor; and the actions each apply a respective modification to the tensor.

38. The method of any preceding claim, wherein: the environment is a computer system that comprises a plurality of computing devices; the task is to allocate one or more computational workloads across the plurality of computing devices, each state of the environment corresponds to a respective state of the computer system given a current allocation of the one or more computational workloads; and the actions each apply a respective modification to the current allocation.

39. A system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the respective method of any one of claims 1-37.

40. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the operations of the respective method of any one of claims 1-37.

Images