Planning for Agent Control Using Restart Extended Lookahead Search

Abstract

It's about choosing your actions effectively. Methods, systems, and apparatus, including computer programs encoded on computer storage media, for selecting an action from a set of actions to be performed by an agent interacting with an environment to cause the agent to perform a task. One of the methods includes receiving current observations characterizing a current environmental state of the environment, and selecting an action to be performed by the agent in response to the current observations by performing multiple iterations of an external lookahead search, where performing the multiple iterations of the external lookahead search includes, in each external lookahead search iteration, determining a proper subset of possible future states of the environment, determining that one or more internal lookahead search initiation criteria are satisfied, and in response thereto, performing an internal lookahead search of the proper subset of the possible future states of the environment.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit under 35 U.S.C. §119(e) of U.S. Patent Application No. 63 / 320,905, filed March 17, 2022, the contents of which are incorporated herein by reference in their entirety. [Background technology]

[0002] This specification relates to selecting an action to be performed by an agent.

[0003] For example, the agent may be an agent controlled by a reinforcement learning system. A reinforcement learning agent interacts with its environment by receiving observations that characterize the current state of the environment and performing actions in response. As actions are performed, the agent receives feedback, called a reward, which is typically a number that depends on the effect of the action's execution on the environment and can be used to update a model that determines which actions the agent will perform.

[0004] Some reinforcement learning systems use neural networks to select actions to be performed by an agent in response to receiving any given observation. Summary of the Invention

[0005] This specification describes a system for controlling an agent that interacts with an environment by processing, at each of a number of time steps, data characterizing the current state of the environment at the time step, or "observations," to select an action to be performed by the agent. At each time step, the state of the environment at the time step depends on the state of the environment at previous time steps and on the actions performed by the agent at the previous time steps.

[0006] In general, the system receives current observations and performs multiple iterations of an external lookahead search. Within each external lookahead search iteration, the system first determines an appropriate subset of possible future states of the environment to be explored, and then performs an internal lookahead search of the appropriate subset of possible future states of the environment. The system then selects an action to be performed in response to the current observations based on the results of the multiple iterations of the external lookahead search.

[0007] Particular embodiments of the subject matter described herein can be implemented to achieve one or more of the following advantages: Actions to be performed by an agent interacting with an environment to perform a complex task having a very large state space can be effectively selected. In other words, actions can be effectively selected to maximize the likelihood of achieving a desired result, such as performance of a learned task. In particular, actions can be effectively selected when the environment has a state tree that is too large to explore, even when techniques such as alpha-beta algorithms or Monte Carlo search guided by neural network output are used. By incorporating machine learning to determine which specific subsets of different states of a large state tree should be investigated, the described techniques reduce the amount of computational resources, time, or both consumed by the exploration process, since exhaustive searching through the state tree is no longer required, while still maintaining effective performance. Instead, only a relatively small number of states of the environment need be evaluated during each exploration iteration. This can enable a system to control an agent to achieve comparable performance with fewer computational and memory requirements and reduced runtime latency than may be required by previous agent control systems.

[0008] The details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an exemplary agent control system. [Figure 2] FIG. 1 is a flow diagram of an example process for selecting an action to be performed by an agent interacting with an environment. [Figure 3] FIG. 3 is a flow diagram of a sub-step of one of the steps of the process of FIG. 2. [Figure 4] FIG. 2 is a diagram of an example state tree from an example process for determining a proper subset of possible future states. [Figure 5] FIG. 1 is a flow diagram of an example process for solving a search problem by searching a search space containing multiple candidate solutions.

[0010] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION OF THE INVENTION

[0011] This specification describes a system for controlling an agent that interacts with an environment by processing, at each of a number of time steps, data characterizing the current state of the environment at the time step, i.e., "observations," to select an action to be performed by the agent.

[0012] At each time step, the state of the environment at the time step may depend on the state of the environment at previous time steps and the response of the environment to actions taken by the agent at previous time steps.

[0013] In some implementations, the environment is a real-world environment and the agent is a mechanical agent that interacts with the real-world environment, such as a robot or an autonomous or semi-autonomous land, air, or sea vehicle that navigates through the environment.

[0014] In these implementations, the observations may include, for example, image data, object position data, or sensor data captured as the agent interacts with the environment, or a combination thereof, such as sensor data from image, distance, or position sensors, or from actuators.

[0015] For example, in the case of a robot, the observables may include data characterizing the current state of the robot, such as one or more of joint positions, joint velocities, joint forces, torques or accelerations, e.g., gravity-compensated torque feedback, and global or relative poses of items held by the robot.

[0016] In the case of a robot or other mechanical agent or vehicle, the observables may similarly include one or more of the position, linear or angular velocity, force, torque or acceleration, and global or relative pose of one or more parts of the agent or vehicle. The observables may be defined in one, two, or three dimensions and may be absolute or relative observables.

[0017] The observations may also be, for example, sensed electronic signals, such as motor current or temperature signals, or image or video data, for example from a camera or ranging sensor, such as a LIDAR (Light Detection and Ranging) sensor, for example, data from a sensor on the agent or from a sensor located remotely from the agent in the environment.

[0018] In these implementations, the actions may be control inputs for controlling a robot, such as torques or high-level control commands for the joints of the robot, or may be control inputs for controlling an autonomous or semi-autonomous land, air, or sea vehicle, such as torques or high-level control commands for the control surfaces or other control elements of the vehicle.

[0019] An action may include, for example, a position, velocity, force, torque, or acceleration control signal for one or more joints of a robot or part of another mechanical agent. Action signals may additionally or alternatively include electronic control data, e.g., motor control data, or more generally, data for controlling one or more electronic devices in an environment, the control of which affects the observed state of the environment. For example, in the case of an autonomous or semi-autonomous land, air, or sea vehicle, actions may include actions for controlling navigation, e.g., steering, and movement, e.g., braking or acceleration of the vehicle.

[0020] In some other applications, agents can control actions in a real-world environment. The environment may include items of equipment. For example, agents can control actions in a data center, a power or water distribution system, a manufacturing plant or service facility, or an infrastructure facility, e.g., a road, an airport, a train station, etc. The observations may relate to the operation of the plant or facility. For example, the observations may include observations of power or water usage by equipment, or observations of power generation or distribution control, or observations of resource or waste generation usage. The actions may include actions that control or impose operating conditions on items of equipment of the plant or facility, or actions that result in a change in settings in the operation of the plant or facility, e.g., adjusting, turning on, or turning off components of the plant or facility.

[0021] In the case of electronic agents, the observables may include data from one or more sensors, e.g., current, voltage, power, temperature, or other sensors, monitoring a portion of a plant or service facility. The observables may also include electronic signals representing the function of electronic or mechanical items of equipment. For example, the real-world environment may be a manufacturing plant or service facility, and the observables may relate to, e.g., the operation of the plant or facility or resource usage, e.g., power consumption, and the agent may control actions or operations in the plant or facility, e.g., to reduce resource usage. In some other implementations, the real-world environment may be a renewable energy plant, and the observables may relate to the operation of the plant, e.g., to maximize current or future planned power generation, and the agent may control actions or operations in the plant to achieve this.

[0022] For example, the environment may be a real-world environment of a service facility including multiple items of electronic equipment, such as a server farm or data center, e.g., a telecommunications data center, or a computer data center for storing or processing data, or any service facility. The service facility may also include auxiliary controls that control the operating environment of the items of equipment, e.g., environmental control equipment such as temperature controls, e.g., cooling equipment, airflow controls, or air conditioning equipment. The tasks may include tasks that control, e.g., minimize, resource usage, such as tasks to control power consumption or water consumption. The agents may include electronic agents configured to control the operation of the items of equipment or to control the operation of auxiliary, e.g., environmental control, equipment.

[0023] In general, the actions may be any actions that affect the observed state of the environment, for example, actions configured to adjust any of the sensed parameters described below. These may include actions to control or impose operating conditions on an item of equipment or an auxiliary control, for example, actions that result in a change of settings to adjust, control, or switch on or off the operation of an item of equipment or an auxiliary control.

[0024] In general, observations of an environmental state can include any electronic signal that represents the function of a facility or equipment within a facility. For example, a representation of an environmental state can be derived from observations made by any sensors that sense the state of the facility's physical environment, or by any sensors that sense the state of one or more items of equipment or auxiliary controls. These include sensors configured to sense electrical conditions such as current, voltage, power, or energy, facility temperature, fluid flow, temperature, or pressure within the facility or in the facility's cooling system, or physical facility configuration, such as whether a vent is open.

[0025] As another example, the environment may be a real-world environment of a power generation facility, e.g., a renewable power generation facility such as a solar farm or wind farm. The task may include a control task to control power generated by the facility, e.g., to control the supply of power to a distribution grid to meet demand, reduce the risk of mismatches between elements of the grid, or for some other reason, or to maximize the power generated by the facility. The agent may include an electronic agent configured to control the generation of power by the facility or the coupling of the generated power to the grid. The action may include an action to control the electrical or mechanical configuration of a generator, e.g., an action to control the electrical or mechanical configuration of one or more renewable power generation elements, e.g., a wind turbine or one or more solar panels or mirrors, or an action to control the electrical or mechanical configuration of a rotating generator. The mechanical control action may include, e.g., an action to control the conversion of an energy input to an electrical energy output, e.g., the efficiency of the conversion or the degree of coupling of the energy input to the electrical energy output. The electrical control action may include, e.g., an action to control one or more of the voltage, current, frequency, or phase of the generated power.

[0026] In some other implementations, the environment is a real-world environment and the agent is a computer system that generates output for presentation to a user.

[0027] For example, the environment may be a patient diagnostic environment where each state is a respective patient state of a patient, i.e., reflected by health data characterizing the patient's health, and the agent may be a computer system programmed to suggest treatments for the patient. In this example, the actions in the set of actions are possible medical treatments for the patient, and the objectives may include one or more of maintaining the patient's current health, improving the patient's current health, minimizing the patient's medical costs, etc.

[0028] Quite different from real-world environments of the type described above are simulated environments, some examples of which are described below.

[0029] In some implementations, the environment is a simulated environment and the agent is implemented as one or more computer programs that interact with the simulated environment. For example, the simulated environment may be a virtual environment in which a user competes against a computerized agent to achieve a goal, where the agent is a computerized agent. In this example, the actions in the set of actions are possible actions that can be performed by the computerized agent, and the goal may be, for example, to win the race against the user.

[0030] As another example, the simulated environment may be a motion simulation environment, such as a driving simulation or a flight simulation, and the agent may be a simulated vehicle that navigates through the motion simulation. In these implementations, the actions may be control inputs for controlling a simulated user or a simulated vehicle.

[0031] As another example, the simulated environment may be a chemical synthesis or protein folding environment, where each state is a respective state of a protein chain or one or more intermediate or precursor chemicals, and the agent is a computer system for determining how to cause the folding of the protein chain or how to synthesize a chemical. In this example, the actions are simulated possible folding actions that cause the folding of the protein chain or simulated actions for assembling precursor chemicals or intermediates, and the achieved results may include, for example, folding a protein so that it is stable and achieves a specific biological function, or providing a valid synthetic route for a chemical. As another example, the agent may be a software program that performs or controls protein folding actions or chemical synthesis steps automatically selected by the system without human interaction. The observables may include direct or indirect observations of the state of the protein or chemical intermediates or precursors. Some observables may be derived from a simulation.

[0032] In the case of a simulated environment, the observations include simulated versions of one or more of the aforementioned observations or types of observations, and the actions include simulated versions of one or more of the aforementioned actions or types of actions.

[0033] Optionally, in any of the above implementations, the observations at any given time step may include data from previous time steps that characterize the environment, e.g., actions performed in previous time steps, rewards received in previous time steps, etc. Rewards may be issued to the agent by the environment and are typically specified by scalar numeric values, e.g., characterizing the agent's progress toward completing assigned tasks. Thus, the system can be used to select actions to be performed by the agent, e.g., to maximize a reward function that measures a time-adjusted sum of total rewards.

[0034] 1 illustrates an exemplary agent control system 100. Agent control system 100 is an example of a system in which the systems, components, and techniques described below are implemented as computer programs on one or more computers at one or more locations.

[0035] The agent control system 100 selects actions to be performed by an agent 102 interacting with an environment 104 to perform a task, achieve a goal, or both. That is, the agent control system 100 receives observations, each observation being data characterizing a respective state of the environment 104, and in response to each received observation, selects an action from a set of actions to be performed by the agent 102 in response to the observation.

[0036] Once the agent control system 100 selects an action to be performed by the agent 102, the agent control system 100 causes the agent 102 to perform the selected action, for example, by commanding the agent to perform the action or by passing a control signal to a control system for the agent. Generally, the agent 102 performing the selected action causes the environment 104 to transition to a different state.

[0037] The observations characterize the state of the environment in a manner appropriate to the context of use of the agent control system 100 .

[0038] For example, if agent control system 100 is a control system for a mechanical agent that interacts with a real-world environment, the observations may be or include images captured by sensors of the mechanical agent as it interacts with the real-world environment, and optionally other sensor data captured by sensors of the agent.

[0039] As another example, if the environment 104 is a patient diagnostic environment, the observations may be or include data from the current patient's electronic medical record.

[0040] As another example, if the environment 104 is a protein folding environment, the observations may be or include images of the current configuration of the protein chains, vectors characterizing the composition of the protein chains, or both.

[0041] In particular, the agent control system 100 selects actions by executing a planning process using the restart engine 110 and the search engine 130 each time an action needs to be selected. Each planning process involves performing multiple iterations of external look-ahead search, starting from the current state of the environment, to traverse a portion of the space of possible future states of the environment. The current state of the environment is the state of the environment 104 characterized by the current observations received by the agent control system 100.

[0042] To assist in each planning process, some implementations of the agent control system 100 maintain data representing possible states of the environment 104. Some implementations of the agent control system 100 also have access to an environment simulator that provides a simulated version of the environment 104 that simulates the effects of actions performed by the agent 102 on the environment 104, and can use such simulator to determine to which state the environment 104 will transition as a result of a given action being performed in a given state.

[0043] In other words, each planning process that involves traversing multiple future states of the environment reflects a different approach to performing the same specific task or achieving the same specific goal, assuming the agent performs a particular action. Thus, the agent control system 100 can also be viewed as an optimization system that solves the problem of finding a final solution to an agent control technical problem as a result of exploring a search space containing multiple candidate solutions, where each candidate solution is represented by a sequence of multiple environmental states. In general, the longer the sequence of environmental states, the more complete the candidate solution may be with respect to solving the technical problem.

[0044] The restart engine 110 is a machine learning subsystem of the control system 100 that includes the machine learning model 120. In each iteration of the external lookahead search, the restart engine 110 uses the machine learning model 120 to determine an appropriate subset of possible future states of the environment 104 starting from a current state characterized by the current observations.

[0045] The machine learning model 120 is configured to receive input including data specifying a current subset of possible future states of the environment that have already been visited in previous outer lookahead search iterations, and to process the input to generate output including data specifying one or more unvisited possible future states of the environment, each of which may be a successor state to any of the ending states of the previous outer lookahead search iteration at which the previous outer lookahead search iteration ended.

[0046] The machine learning model 120 may have any suitable architecture that enables the model 120 to map a current subset of already-visited possible future states to one or more unvisited possible future states of the environment. For example, the machine learning model 120 may be a neural network, a support vector machine (SVM) model, a decision forest model, including a gradient boosting decision forest model, or any other type of trainable machine learning model. The machine learning model 120 may be trained on data generated as a result of prior interactions of the agent 102 or another agent with the environment 104 or another instance of an environment when performing a task similar to the assigned task.

[0047] The agent control system 100 then updates the appropriate subset of possible future states of the environment 104 determined during the previous external lookahead search iteration to include one or more unvisited possible future states determined by the restart engine 110 in the current iteration of the external lookahead search. Collectively, the appropriate subset of possible future states represents a smaller, sometimes much smaller, and therefore much more focused, portion of the entire space of possible future states of the environment that needs to be traversed in the current external lookahead search iteration.

[0048] Then, within the current iteration of the external lookahead search, the search engine 130 performs an internal lookahead search to search a proper subset of possible future states of the environment 104. The search engine 130 may generally use any search-based optimization algorithm to search a proper subset of the possible future states to determine which action from the set of possible actions should be performed by the agent, i.e., as a result of applying an optimization algorithm to the proper subset of the possible future states. The search-based optimization algorithm may be a heuristic search algorithm, such as a depth- or breadth-first search algorithm, a greedy search algorithm, or an A(n) search algorithm, to name just a few. * It can be a search algorithm.

[0049] In some implementations, the search engine 130 can perform an internal look-ahead search by performing a tree search guided by the output of the trained action selection model 140, and the search-based optimization algorithm can include, for example, a look-ahead tree search using an alpha-beta method, or a Monte Carlo tree search.

[0050] In some implementations, the action selection model 140 is a neural network model configured to receive network inputs including observations and process the network inputs according to parameters of the action selection model to generate network outputs. The observations may be observations characterizing possible future states of the environment 104 in a proper subset of possible future states determined by the restart engine 110, e.g., simulated observations generated by a simulator. The network outputs include action selection outputs that define an action selection policy for selecting actions to be performed by the agent in response to the observations.

[0051] Exemplary configurations of such action selection model 140, as well as how an internal look-ahead search can be performed with the guidance of the action selection output, are described in U.S. Patent Nos. 11,449,750 (B2) and 10,867,242 (B2), which are incorporated herein by reference. In these examples, performing an internal look-ahead search with the guidance of the action selection output to select an action for an agent means that instead of directly using the action selection output generated by action selection model 140 to select an action, the system performs a guided look-ahead search guided by action selection model 140, and then selects an action based on the results of the guided look-ahead search.

[0052] In this way, in each iteration of the external look-ahead search, the agent control system 100 uses the restart engine 110 to determine a proper subset of the possible future states of the environment 104 from all of the possible future states of the environment 104, and traverses only the states in the proper subset using the search engine 130. By doing so, the agent control system 100 reduces the number of possible future states of the environment that need to be traversed by using the search engine 130 in the internal look-ahead search during each iteration of the external look-ahead search, while still allowing precise control of the agent 102, i.e., the selection of high-quality actions in response to any given observation.

[0053] The number of states in the relevant subset is generally much less than the total number of states in the space of all possible states of the environment 104. For example, if the space of possible states is about 2 100 Even if the number of possible states is as small as 2, the system can still accurately control the agent. 20 Only states are included in the appropriate subset, e.g., 2 10 This action is added to the appropriate subset during each outer lookahead search iteration.

[0054] This can enable the agent control system 100 to control the agent 102 with reduced latency while consuming fewer computational resources than existing approaches that always attempt to explore the entire space of possible states of the environment 104 when selecting actions in response to observations.

[0055] 2 is a flow diagram of an exemplary process 200 for selecting an action to be performed by an agent interacting with an environment. For convenience, process 200 is described as being implemented by one or more computer systems located at one or more locations. For example, an agent control system suitably programmed in accordance with this specification, such as agent control system 100 of FIG. 1, can perform process 200.

[0056] The system receives current observations characterizing the current environmental state of the environment (step 202). In some implementations, the observations may include visual data, e.g., images or video frames, while in other implementations, the observations may be multimodal observations that further incorporate information about text, e.g., natural language instructions, rewards, or other sensory inputs, including touch, smell, sound, or temperature.

[0057] The system selects an action to be performed by the agent in response to the current observations (step 204). To facilitate this selection, the system performs multiple iterations 212 of external lookahead search to generate estimates of possible future states of the environment, starting from the current environmental state, i.e., states that are successors of the current environmental state in the environment. Each external lookahead search iteration 212 includes steps 206-210, and thus performing multiple iterations 212 of external lookahead search includes repeating steps 206-210 multiple times.

[0058] The system determines an appropriate subset of possible future states of the environment to be explored (step 206). In each external lookahead search iteration, the system can use a machine learning model to identify unvisited possible future states of the environment to add to the current subset to determine an appropriate subset of possible future states of the environment. Step 206 is described in more detail with reference to FIG. 3, which shows substeps 302-306 of step 206.

[0059] The system maintains data specifying a current subset of possible future states of the environment that have already been visited in previous outer lookahead search iterations (step 302).

[0060] 4 is a diagram of an exemplary state tree from an exemplary process for determining a proper subset of possible future states. The system maintains data representing a state tree 400 of an environment. The state tree 400 includes nodes that represent states of the environment and directed edges that connect the nodes in the tree. An outgoing edge from a first node to a second node in the tree represents an action that, when performed in response to an observation that characterizes the first state, will cause the environment to transition to the second state. While the data is logically described as a tree in the example of FIG. 4, the data could be represented by any of a variety of convenient physical data structures, for example, as multiple triples or an adjacency list.

[0061] In state tree 400, nodes 402, 412, 414, 422, 424, 426, and 428 constitute a current subset of possible future states of the environment already visited in previous outer lookahead search iterations. Among these nodes, the system can identify a current node in the state tree, e.g., root node 402, that represents the current environmental state characterized by current observations. The system can also identify nodes, e.g., leaf nodes 422, 424, 426, and 428, that represent the ending states of previous outer lookahead search iterations where the previous outer lookahead search iteration ended. In a state tree, e.g., state tree 400 generated during the most recent outer lookahead search iteration, a leaf node is a node that has no child nodes, i.e., is not connected to any other nodes by an outgoing edge. Thus, a leaf node is sometimes referred to as an "unexpanded" node in the state tree representing the environment.

[0062] The system determines one or more unvisited possible future states starting from the end state represented by the identified leaf node (step 304), i.e., the system expands the leaf node, particularly by using a machine learning model, optionally together with a simulator of the environment.

[0063] When used by the system, the machine learning model is configured to receive input including data specifying a current subset of possible future states of the environment already visited in previous outer lookahead search iterations, and to process the input to generate output including data specifying one or more unvisited possible future states of the environment, each of which may be a successor state to the ending state of one of the previous outer lookahead search iterations at which the previous outer lookahead search iteration ended.

[0064] In some implementations, the machine learning model can generate these unvisited possible future states over multiple time steps, i.e., generate these unvisited possible future states sequentially, e.g., one after the other in an autoregressive manner.

[0065] In some implementations, these unvisited possible future states are generated deterministically, e.g., by the output of a machine learning model. In other implementations, these unvisited possible future states are generated probabilistically, e.g., the output of a machine learning model parameterizes a distribution, e.g., a score distribution, among the space of possible future states of the environment from which one or more unvisited possible future states may be sampled.

[0066] The one or more unvisited possible future states thus generated include future states in the space of the environment to which the environment can transition starting from the ending state of any of the previous outer lookahead search iterations as a result of the agent performing one or more valid actions selected from a set of actions that can be performed by the agent.

[0067] 4, the system uses a machine learning model to process inputs including data describing nodes 402, 412, 414, 422, 424, 426, and 428, and processes the inputs to generate outputs including data describing nodes 432, 434, 442, 444, 446, and 448, each of which represents an unvisited possible future state of the environment. In this example, node 432 represents the immediate future state to which the environment transitions in response to the agent performing a valid action in the set of actions when the environment is in the end state represented by node 426, and node 442 (or node 444) represents the next immediate future state to which the environment transitions in response to the agent performing a valid action in the set of actions when the environment is in the state represented by node 432.

[0068] In some cases, instead of, or in addition to, unvisited possible future states of the environment, the output of the machine learning model may include data specifying one or more already visited states of the environment. For example, as shown in Figure 4, the output of the machine learning model may instead or additionally include data describing nodes 402, 412, or 414.

[0069] The system adds one or more unvisited possible future states to the current subset of possible future states of the environment that were already visited in the previous outer lookahead search iteration (step 306). That is, the system updates the current subset of possible future states of the environment specified in the maintained data to also include one or more unvisited possible future states determined using the machine learning model. In the example of FIG. 4, the unvisited possible future states represented by nodes 432, 434, 442, 444, 446, and 448 are added to the current subset of possible future states of the environment to be explored. If only the already-visited states of the environment were determined using the machine learning model, there is no need to add additional environment states to the current subset of possible future states of the environment specified in the maintained data.

[0070] When the state space is large, determining a suitable subset of possible future states of the environment by using a machine learning model in this manner can require long processing times and can be computationally intensive and consume a significant amount of computational resources. Thus, whenever one or more internal lookahead search initiation criteria are met, the system determines that an internal lookahead search should begin and correspondingly, in step 206, stops the ongoing determination of a suitable subset of possible future states.

[0071] The system determines that one or more internal lookahead search initiation criteria are met, step 208. As described above, such internal lookahead search initiation criteria specify when process 200 should proceed to steps 206-210.

[0072] For example, the entry criteria may include a maximum number criteria that specifies that process 200 should proceed to steps 206-210 whenever a predetermined number of different states of the environment have been generated by using the machine learning model.

[0073] As another example, the starting criteria can include a sufficiency criterion that specifies that process 200 should proceed to steps 206-210 whenever a proper subset of possible future states of the environment to be investigated is sufficient for generating an assessment of possible future states of the environment starting from the current environment state. To determine sufficiency in this example, the system can evaluate the proper subset of possible future states of the environment by computing a stopping function to generate a binary classification result specifying whether the proper subset of possible future states of the environment is sufficient. For example, the stopping function can be implemented as a trained classifier model or can be realized deterministically as a parametric equation.

[0074] As another example, the entry criteria may include a timeout criteria that specifies that process 200 should proceed to steps 206-210 whenever the time spent determining unvisited possible future states by using the machine learning model exceeds a predetermined length of time, e.g., 1 second, 5 seconds, 10 seconds, etc.

[0075] In response, the system performs an internal look-ahead search of an appropriate subset of possible future states of the environment (step 210). In some implementations, the internal look-ahead search is performed using a heuristic search algorithm, such as a depth- or breadth-first search algorithm, a greedy search algorithm, or an A * Implement a search algorithm. In some other implementations, the internal look-ahead search is a Monte Carlo tree search guided by the action selection model according to the values ​​of the network parameters, such that the Monte Carlo tree search depends on the action selection output from the action selection model.

[0076] Specifically, starting from a current state characterized by current observations, the system performs an internal lookahead search to traverse only a proper subset of the environment's possible future states, i.e., not the entire space of the environment's possible future states, until one or more internal lookahead search termination criteria are met.

[0077] For example, the system may start at a root node of the state tree representing the current state of the environment, traverse the state tree through nodes representing appropriate subsets of possible future states of the environment, until it reaches a leaf node in the state tree.

[0078] After performing multiple iterations of the external lookahead search, the system can proceed to select an action to be performed by the agent in response to the current observation using statistics generated during the multiple iterations of the external lookahead search for the root node of the state tree that represents the current observation.

[0079] When different variants of a heuristic search algorithm or Monte Carlo tree search are used to perform an internal look-ahead search of an appropriate subset of possible future states of the environment during each iteration of the external look-ahead search, different statistics may be compiled. For example, the statistics may include, for each outgoing edge connected to the root node representing a corresponding action performed by the agent in response to the current observation, a respective action score for the action represented by the edge. Each action score specifies the likelihood of the agent completing the task if the action is performed. In this example, the system may select the action with the highest action score as the action to be performed by the agent in response to the current observation.

[0080] As another example, the statistics may include, for each outgoing edge connected to the root node representing a corresponding action performed by the agent in response to the current observation, a respective visit count for the action represented by the edge. Each visit count represents the number of times the action was considered by the agent as an action that could be performed in response to the current observation. In this example, the system may select the action with the highest visit count as the action to be performed by the agent in response to the current observation.

[0081] In either example, the system may maintain, along with data representing the state tree, edge data for each edge in the state tree that includes (i) an action score for the action represented by the edge, (ii) a visit count for the action represented by the edge, or both. The system may use an exploration engine to update the maintained data representing the state tree and the edge data for the edges in the state tree from the agent's interactions with the simulated version of the environment.

[0082] The system can repeat process 200 for each observation received during an episode of an agent's interaction with the environment. An episode refers to a series of time steps during which the agent interacts with the environment. An episode can end, for example, when the agent has interacted with the environment for a predetermined number of time steps or when the agent completes a task.

[0083] 5 is a flow diagram of an exemplary process 500 for solving a problem by exploring a search space containing multiple candidate solutions. For convenience, process 500 is described as being performed by one or more computer systems located at one or more locations. For example, an agent control system suitably programmed in accordance with this specification, such as agent control system 100 of FIG. 1, can perform process 500.

[0084] Each candidate solution can be considered a different approach to solving the problem. In various implementations, the search space of multiple candidate solutions can be, for example, provided or otherwise defined by a user, or randomly generated by the system. Process 500 finds a solution by iteratively identifying a proper subset of each of the multiple candidate solutions and then analyzing the candidate solutions within each proper subset until a satisfactory solution is reached. For example, the problem can be a problem in the fields of engineering, biology, or robotics defined in the form of a combinatorial optimization problem, e.g., an assignment problem involving assigning a group of workers to perform a set of tasks; a packing problem involving finding how to pack a set of items of a given size into a container of fixed dimensions; a scheduling program, e.g., a factory job scheduling problem involving assigning people and resources to tasks at specific times; or a network flow problem involving transporting goods or materials through a network such as a railroad system.

[0085] In summary, the system uses a machine learning subsystem to repeatedly perform steps 502 and 504 to identify a respective good subset of multiple candidate solutions across multiple outer search iterations. For each good subset identified during each outer search iteration, the system then performs steps 508 and 510 using a search engine to search the candidate solutions in the good subset and identify a respective candidate final solution.

[0086] The system receives, at the machine learning subsystem, a machine learning subsystem input that includes data specifying a current subset of a plurality of candidate solutions already explored in a previous outer search iteration (step 502).

[0087] The system processes the machine learning subsystem input using a machine learning subsystem configured to generate a machine learning subsystem output including data specifying one or more new candidate solutions to be searched for in the current outer search iteration (step 504). In some implementations, the machine learning subsystem is configured to generate at most a predetermined, e.g., user-specified, number of new candidate solutions in each current outer search iteration.

[0088] The system maintains data specifying a current subset of the plurality of candidate solutions already searched in a previous outer search iteration. After step 504, the system updates the current subset of the plurality of candidate solutions by adding one or more new candidate solutions to the current subset.

[0089] The system searches the candidate solutions to identify each candidate final solution by using a search engine (step 508). In particular, the system does this by using the search engine to search an updated current subset of candidate solutions, including the one or more new candidate solutions determined in step 504, in each current outer search iteration.

[0090] In some implementations, searching for candidate solutions includes first determining that one or more look-ahead search initiation criteria are met (optional step 506) and then proceeding to perform step 508. For example, determining that one or more look-ahead search initiation criteria are met may include determining that an updated current subset of candidate solutions is sufficient with respect to generating evaluation results that represent all of the multiple candidate solutions in the search space. In this example, the system may determine whether the updated current subset of candidate solutions is sufficient based on evaluating each candidate final solution identified from the updated current subset of candidate solutions by calculating a stopping function to generate a binary classification result that specifies whether each candidate final solution is a valid solution to the problem. In other words, the updated current subset of candidate solutions is considered sufficient by the system if a valid solution can be obtained from the updated current subset.

[0091] As another example, determining that one or more lookahead search initiation criteria are met may include determining that the time spent processing the machine learning system input to generate the machine learning subsystem output exceeds a predetermined length of time.

[0092] In each current outer search iteration, the system identifies respective candidate final solutions to the problem (step 508). To identify candidate final solutions, the system performs a look-ahead search of possible continuation solutions starting from at least one or more new candidate solutions specified by the machine learning subsystem output determined in step 504 until one or more look-ahead search termination criteria are met.

[0093] For example, the search problem can be represented by a search tree, and the look-ahead search can be a look-ahead tree search, e.g., a Monte Carlo tree search. In this example, the root node of the search tree for the look-ahead tree search represents an initial partial solution to the problem, and each child node on the path from the root node represents a candidate continuation of the initial partial solution. The look-ahead tree search then involves traversing paths connecting particular child nodes of the search tree, each of which represents one of the new candidate solutions specified by data included in the machine learning subsystem output, until a leaf node of the search tree is reached.

[0094] The system then selects, from the plurality of candidate solutions, a candidate solution selected as a result of performing the look-ahead search as a respective candidate final solution. In some implementations, the system generates evaluation results for the one or more new candidate solutions by, for example, evaluating the one or more new candidate solutions determined in step 504 using statistics compiled from the look-ahead search regarding solving the problem according to an objective function associated with the problem, and then using the evaluation results to select a selected candidate solution. For example, the evaluation results may include a likelihood that the new candidate solution is a mathematically optimal solution to the problem.

[0095] After performing the multiple outer search iterations, the system proceeds to identify a final solution to the problem from each of the candidate final solutions generated in the multiple outer search iterations, for example, by using a search engine to identify the candidate final solution that is most likely to be the optimal solution to the problem.

[0096] The subject matter and actions and operations described herein can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware, or a combination of one or more of them, including the structures disclosed herein and their structural equivalents. The subject matter and actions and operations described herein can be implemented as or in one or more computer programs, e.g., one or more modules of computer program instructions encoded on a computer program carrier, for execution by or to control the operation of a data processing apparatus. The carrier can be a tangible, non-transitory computer storage medium. Alternatively, or additionally, the carrier can be an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to a suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be, or be part of, a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. The computer storage medium is not a propagating signal.

[0097] The term "data processing apparatus" encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. A data processing apparatus may include special-purpose logic circuitry, such as an FPGA (field programmable gate array), an ASIC (application-specific integrated circuit), or a GPU (graphics processing unit). In addition to hardware, an apparatus may also include code that creates an execution environment for a computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0098] A data processing device for implementing machine learning models may also include, for example, a dedicated hardware accelerator unit for handling the typical, computationally intensive part of machine learning training or production, i.e., inference, workloads.

[0099] The machine learning model can be implemented and deployed using a machine learning framework, for example, the TensorFlow framework or the JAX framework.

[0100] Computer programs can be written in any type of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including stand-alone programs, for example, as apps, or as modules, components, engines, subroutines, objects, or other units suitable for execution in a computing environment, which environment can include one or more computers interconnected by a data communications network at one or more locations.

[0101] A computer program may, but need not, correspond to a file in a file system. A computer program may be stored in one or more scripts stored in a portion of a file that holds other programs or data, for example, in a markup language document, in a single file dedicated to the program in question, or in multiple cooperating files, for example, files that store one or more modules, sub-programs, or portions of code.

[0102] The processes and logic flows described herein may be performed by one or more computers executing one or more computer programs to perform operations by operating on input data to generate output. The processes and logic flows may also be implemented by special purpose logic circuitry, such as an FPGA, an ASIC, or a GPU, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0103] A computer suitable for executing a computer program may be based on a general-purpose or special-purpose microprocessor, or both, or any other kind of central processing unit. Generally, the central processing unit receives instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory may be supplemented by, or incorporated in, special-purpose logic circuitry.

[0104] Generally, a computer also includes or is operably coupled to one or more mass storage devices and is configured to receive data from or transfer data to the mass storage devices. A mass storage device can be, for example, a magnetic, optical-magnetic, or optical disk, or a solid-state drive. However, a computer need not have such a device. Furthermore, a computer can be incorporated into another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name just a few.

[0105] To provide for interaction with a user, the subject matter described herein can be implemented on one or more computers having or configured to communicate with a display device, e.g., an LCD (liquid crystal display) monitor or a virtual reality (VR) or augmented reality (AR) display, for displaying information to a user, and an input device, e.g., a keyboard and pointing device, e.g., a mouse, trackball, or touchpad, through which a user can provide input to the computer. Other types of devices can be used to provide interaction with a user as well; for example, feedback and responses provided to a user can be any form of sensory feedback, e.g., visual, auditory, speech, or tactile, and input from a user can be received in any form, including acoustic, speech, or tactile input, including touch movements or gestures, or kinetic movements or gestures, or directional movements or gestures. Additionally, a computer can interact with a user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser, or by interacting with an app running on the user device, for example, a smartphone or electronic tablet. A computer can also interact with a user by sending text messages or other types of messages to a personal device, for example, a smartphone running a messaging application, and receiving reply messages from the user in return.

[0106] The term "configured" is used herein in connection with systems, devices, and computer program components. When one or more computer systems are configured to perform a particular operation or action, it means that the system has installed thereon software, firmware, hardware, or a combination thereof, which, when in operation, causes the system to perform the operation or action. When one or more computer programs are configured to perform a particular operation or action, it means that the one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action. When special-purpose logic circuitry is configured to perform a particular operation or action, it means that the circuitry has electronic logic that performs the operation or action.

[0107] The subject matter described herein can be implemented in a computing system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server, or includes a front-end component, such as a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described herein, or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0108] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some implementations, a server sends data, e.g., HTML pages, to a user device for the purpose of displaying the data to and receiving user input from a user interacting with the device acting as a client. Data generated at the user device, e.g., a result of a user interaction, may be received from the device at the server.

[0109] While the disclosed inventive concepts include those defined in the appended claims, it should be understood that the inventive concepts may also be defined according to the following embodiments.

[0110] In addition to the embodiments in the appended claims and the embodiments described above, the following numbered embodiments are also innovative.

[0111] Embodiment 1 is a method for selecting an action to be performed by an agent interacting with an environment from a set of actions to cause the agent to perform a task, the method comprising: receiving current observations characterizing a current environmental state of the environment; selecting an action to be performed by the agent in response to the current observations by performing multiple iterations of an external look-ahead search to generate an assessment of possible future states of the environment starting from the current environmental state, wherein performing multiple iterations of the external look-ahead search includes, in each external look-ahead search iteration: determining a suitable subset of possible future states of the environment to be investigated; determining that one or more internal lookahead search initiation criteria are met; In response thereto, performing an internal lookahead search of a proper subset of possible future states of the environment until one or more internal lookahead search termination criteria are met.

[0112] Embodiment 2 is the method according to claim 1, wherein determining the appropriate subset of possible future states comprises: maintaining data specifying a current subset of possible future states of the environment already visited in previous outer lookahead search iterations; determining one or more unvisited possible future states starting from one or more end states of any of the previous outer lookahead search iterations at which the previous outer lookahead search iteration ended; adding one or more unvisited possible future states to a current subset of possible future states of the environment that were already visited in a previous outer lookahead search iteration.

[0113] Example 3 is the method of example 2, wherein determining one or more unvisited possible future states includes selecting at most a predetermined number of different states of the environment.

[0114] Embodiment 4 is the method according to any one of embodiments 2 to 3, wherein the one or more unvisited possible future states include a future state to which the environment transitions from an end state of any of the previous outer lookahead search iterations in response to the agent performing a valid action in the set of actions when the environment is in the end state.

[0115]

[0023] Embodiment 5 is the method according to any one of embodiments 1 to 4, wherein determining that one or more internal lookahead search initiation criteria are met comprises: determining that a suitable subset of possible future states of the environment to be investigated is sufficient for generating an assessment of possible future states of the environment starting from the current environmental state.

[0116]

[0023] Embodiment 6 is the method of embodiment 5, wherein determining that a proper subset of possible future states of the environment to be investigated is sufficient comprises: and evaluating the appropriate subset of possible future states of the environment by computing a stopping function to generate a binary classification result specifying whether the appropriate subset of possible future states of the environment is sufficient.

[0117] Embodiment 7 is the method according to any one of embodiments 1 to 4, wherein determining that one or more internal lookahead search initiation criteria are met comprises: Determining that the time spent determining one or more unvisited possible future states exceeds a predetermined amount of time.

[0118]

[0023] Embodiment 8 is the method of any one of embodiments 1 to 7, wherein performing an internal look-ahead search of a proper subset of possible future states of the environment until one or more termination criteria are met comprises: It involves traversing the state tree starting from the root node of the state tree, which represents the current environmental state, until a leaf node in the state tree is reached.

[0119]

[0023] Embodiment 9 is the method of embodiment 8, wherein selecting an action to be performed by the agent comprises: In response to the current observation, selecting an action to be performed by the agent using statistics generated during multiple iterations of the external lookahead search for the root node of the state tree representing the current observation.

[0120]

[0023] Embodiment 10 is the method of embodiment 9, wherein the statistics include, for each outgoing edge connected to the root node representing a corresponding action performed by the agent in response to the current observation, a respective action score for the action represented by the edge, the action score specifying the likelihood of the agent completing the task if the action is performed; Selecting an action to be performed by the agent includes selecting the action with the highest action score.

[0121]

[0023] Embodiment 11 is the method of embodiment 9, wherein the statistics include, for each outgoing edge connected to the root node representing a corresponding action considered by the agent as an action that could be performed in response to the current observation, a respective visit count for the action represented by the edge, the visit count representing the number of times the action was considered by the agent as an action that could be performed in response to the current observation; Selecting an action to be performed by the agent includes selecting the action with the highest visit count.

[0122] A twelfth embodiment is the method according to any one of the first to eleventh embodiments, in which performing an internal lookahead search includes performing a Monte Carlo tree search.

[0123] Example 13 is the method of example 12, wherein performing a Monte Carlo tree search includes performing a Monte Carlo tree search guided by outputs of a neural network, the neural network being configured to receive input observations characterizing a state of the environment, process the input observations according to network parameters, and generate network outputs specifying actions to be taken by the agent in response to the input observations.

[0124] Embodiment 14 is the method according to any one of embodiments 1 to 11, wherein performing the internal lookahead search includes performing the lookahead tree search using alpha-beta algorithm.

[0125] Embodiment 15 is a method according to any one of embodiments 1 to 14, wherein the agent is a mechanical agent, the environment is a real-world environment, and the current observations include data from one or more sensors configured to sense the real-world environment.

[0126] Embodiment 16 is a method according to claim 15, wherein the agent comprises a robot or a vehicle.

[0127] Example 17 is the method of example 15 or 16, wherein the agent includes a data processing device configured to process current observations and generate control signals that cause the agent to perform selected actions in the real-world environment.

[0128] Embodiment 18 is a method according to any one of embodiments 1 to 14, wherein the agent is a computer program, the environment is a real-world environment, the observations include data from one or more sensors configured to sense the real-world environment, and the agent performs a task by providing instructions specifying actions in the real-world environment.

[0129] Embodiment 19 is a system comprising one or more computers and one or more storage devices that store instructions, which, when executed by the one or more computers, are operable to cause the one or more computers to perform a method described in any one of embodiments 1 to 18.

[0130] Embodiment 20 is a computer storage medium encoded with a computer program, the program comprising instructions operable, when executed by a data processing device, to cause the data processing device to perform a method according to any one of embodiments 1 to 18.

[0131]

[0023] Embodiment 21 is a computationally implemented system for solving a search problem by searching a search space containing multiple candidate solutions, the system comprising: a machine learning subsystem and a search engine; The machine learning subsystem is configured to identify a proper subset of the plurality of candidate solutions, and the machine learning subsystem, in each current outer search iteration of the plurality of outer search iterations, receiving a machine learning subsystem input including data specifying a current subset of a plurality of candidate solutions already explored in a previous outer search iteration; processing the machine learning subsystem input to generate a machine learning subsystem output including data specifying one or more new candidate solutions to be searched for in the current outer search iteration; using a search engine to explore the candidate solutions to identify each candidate final solution, the explored candidate solutions including one or more new candidate solutions; The search engine, in each current outer search iteration, The system is configured to identify each candidate final solution to the search problem by performing a look-ahead search of possible subsequent solutions starting from at least one or more new candidate solutions specified by the machine learning subsystem output until one or more look-ahead search termination criteria are met, and selecting, from the plurality of candidate solutions, a candidate solution selected as a result of performing the look-ahead search as a respective candidate final solution.

[0132]

[0033] Embodiment 22 is the system of embodiment 21, wherein the machine learning subsystem, at each current outer search iteration: The method is further configured to maintain data specifying a current subset of the plurality of candidate solutions already searched in a previous outer search iteration, including updating the current subset of the plurality of candidate solutions by adding one or more new candidate solutions.

[0133] A twenty-third embodiment is the system according to the twenty-first or twenty-second embodiment, wherein the selection of the candidate solution selected as a result of the look-ahead search is performed by: generating evaluation results of the one or more new candidate solutions by evaluating the one or more new candidate solutions with respect to solving the search problem; and using the evaluation results to select a selected candidate solution.

[0134] Embodiment 24 is the system described in embodiment 23, wherein the evaluation results include a likelihood that the new candidate solution is a mathematically optimal solution to the search problem.

[0135] Embodiment 25 is a system described in any of embodiments 21 to 24, wherein the search engine is further configured to identify a final solution to the search problem from each candidate final solution generated in multiple outer search iterations.

[0136] A twenty-sixth embodiment is the system according to any one of the twenty-first to twenty-fifth embodiments, in which the search problem includes a factory job scheduling problem.

[0137] Embodiment 27 is a system according to any one of embodiments 21 to 26, wherein the machine learning subsystem is configured to generate at most a predetermined number of new candidate solutions in each current external search iteration.

[0138] Embodiment 28 is a system described in any of embodiments 21 to 27, wherein searching candidate solutions to identify each candidate final solution using a search engine includes determining that one or more look-ahead search initiation criteria are met.

[0139]

[0072] Embodiment 29 is the system of embodiment 28, wherein determining that one or more lookahead search initiation criteria are met comprises: determining that the updated current subset of candidate solutions is sufficient to generate evaluation results for all of the candidate solutions in the search space; or determining that the time spent processing the machine learning system inputs to generate the machine learning subsystem outputs exceeds a predetermined length of time.

[0140] Embodiment 30 is the system of embodiment 29, wherein determining that the updated current subset of candidate solutions is sufficient comprises: In this example, the system includes evaluating each candidate final solution identified from the updated current subset of candidate solutions by computing a stopping function to generate a binary classification result specifying whether each candidate final solution is a valid solution to the search problem.

[0141] A thirty-first embodiment is the system according to any one of the twenty-first to thirty-first embodiments, in which the look-ahead search includes a Monte Carlo tree search.

[0142] Embodiment 32 is one or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to implement the machine learning subsystem and search engine described in any of embodiments 21 to 31.

[0143] Embodiment 33 is a method including operations configured to be performed by the machine learning subsystem and search engine described in any of embodiments 21 to 31.

[0144] While this specification contains many details of specific implementations, these should not be construed as limitations on the scope of what is claimed, as defined by the claims themselves, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described herein in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as working in a particular combination and initially claimed as such, one or more features from a claimed combination can, in some cases, be deleted from the combination, and the claims may be directed to the subcombination or variations of the subcombination.

[0145] Similarly, although operations are depicted in the figures or recited in the claims in a particular order, this in itself should not be understood as requiring that such operations be performed in the particular order, or sequential order, shown, or that all of the illustrated operations be performed, to achieve desired results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products.

[0146] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some instances, multitasking and parallel processing may be advantageous.

Claims

1. 1. A method for selecting an action to be performed by an agent interacting with an environment from a set of actions to cause said agent to perform a task, said method comprising: receiving, by an agent control system, current observations characterizing a current environmental state of the environment; selecting, by the agent control system, an action to be taken by the agent in response to the current observations by performing multiple iterations of an external look-ahead search to generate an estimate of possible future states of the environment starting from the current environmental state; Including, The performing multiple iterations of the outer lookahead search comprises, in each outer lookahead search iteration: performing, by the agent control system, an ongoing determination of an appropriate subset of the possible future states of the environment to be explored; determining, by the agent control system, that one or more internal lookahead search initiation criteria are satisfied in response to suspending the ongoing determination of the appropriate subset; thereafter, performing, by the agent control system, an internal look-ahead search of only the proper subset of the possible future states of the environment without considering remaining possible future states of the environment not included in the proper subset until one or more internal look-ahead search termination criteria are met; A method comprising:

2. The method of claim 1 , wherein the agent is a mechanical agent, the environment is a real-world environment, and the current observations include data from one or more sensors configured to sense the real-world environment.

3. The method of claim 2 , wherein the agent comprises a robot or a vehicle.

4. The method of claim 2 , wherein the agent comprises a data processing device configured to process the current observations and generate control signals that cause the agent to perform the selected action in a real-world environment.

5. 10. The method of claim 1, wherein the agent is a computer program, the environment is a real-world environment, the observations include data from one or more sensors configured to sense the real-world environment, and the agent performs the task by providing instructions that specify actions in the real-world environment.

6. Determining the relevant subset of the possible future states comprises: maintaining, by said agent control system, data specifying a current subset of possible future states of said environment that have already been visited in previous external lookahead search iterations; determining by the agent control system one or more unvisited possible future states starting from one or more end states of any of the previous external lookahead search iterations at which the previous external lookahead search iterations ended; adding, by the agent control system, the one or more unvisited possible future states to the current subset of possible future states of the environment already visited in the previous outer lookahead search iteration.

7. The method of claim 6 , wherein determining the one or more unvisited possible future states comprises selecting, by the agent control system, at most a predetermined number of different states of the environment.

8. 7. The method of claim 6, wherein the one or more unvisited possible future states comprises a future state to which the environment transitions from the end state of any of the previous outer lookahead search iterations in response to the agent executing a valid action in the set of actions when the environment is in an end state.

9. Determining that the one or more internal lookahead search initiation criteria are met includes:

2. The method of claim 1, comprising determining, by the agent control system, that the appropriate subset of the possible future states of the environment to be explored is sufficient for generating the assessment of the possible future states of the environment starting from the current environmental state.

10. determining that the relevant subset of the possible future states of the environment to be explored is sufficient; 10. The method of claim 9, comprising evaluating, by the agent control system, the proper subset of the possible future states of the environment by computing a stopping function to generate a binary classification result specifying whether the proper subset of the possible future states of the environment is sufficient.

11. Determining that the one or more internal lookahead search initiation criteria are met includes:

2. The method of claim 1, including determining by the agent control system when the time spent determining the one or more unvisited possible future states exceeds a predetermined amount of time.

12. performing the internal look-ahead search of the relevant subset of the possible future states of the environment until the one or more termination criteria are met; 2. The method of claim 1, comprising traversing the state tree by the agent control system starting from a root node of the state tree representing the current environmental state until a leaf node in the state tree is reached.

13. Selecting the action to be performed by the agent comprises:

13. The method of claim 12, comprising selecting, by the agent control system, the action to be performed by the agent in response to the current observation using statistics generated during multiple iterations of the external lookahead search for the root node of the state tree that represents the current observation.

14. the statistics include, for each outgoing edge connected to the root node representing a corresponding action performed by the agent in response to the current observation, a respective action score for the action represented by the edge, the action score specifying the likelihood of the agent completing the task if the action is performed; The method of claim 13 , wherein selecting the action to be performed by the agent comprises selecting, by the agent control system, the action having the highest action score.

15. the statistics include, for each outgoing edge connected to the root node representing a corresponding action considered by the agent as an action that could be performed in response to the current observation, a respective visit count for the action represented by the edge, the visit count representing the number of times the action was considered by the agent as an action that could be performed in response to the current observation; The method of claim 13 , wherein selecting the action to be performed by the agent includes selecting, by the agent control system, the action having the highest number of visits.

16. The method of claim 1 , wherein performing the internal look-ahead search comprises performing a Monte Carlo tree search by the agent control system.

17. 17. The method of claim 16, wherein performing the Monte Carlo tree search comprises performing, by the agent control system, the Monte Carlo tree search guided by outputs of a neural network, the neural network configured to receive input observations characterizing states of the environment, process the input observations according to network parameters, and generate network outputs specifying actions to be taken by the agent in response to the input observations.

18. The method of claim 1 , wherein performing the internal look-ahead search comprises performing, by the agent control system, a look-ahead tree search using an alpha-beta algorithm.

19. 19. A system comprising one or more computers and one or more storage devices storing instructions, the instructions being operable when executed by the one or more computers to cause the one or more computers to perform the method of any one of claims 1 to 18.

20. One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 18.

Images