Active offline policy selection using policy representations

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for obtaining one or more final policies for controlling agents in an environment. In one aspect, one of the methods includes obtaining a candidate policy set including a plurality of candidate policies for controlling agents in an environment; obtaining an offline data set storing a plurality of historical trajectories, where each historical trajectory includes a plurality of historical observations each characterizing a respective historical state of the environment; and generating a behavioral representation for each candidate policy.

Description

Cross-reference to related applications

[0001] This application claims priority to U.S. Provisional Application No. 63 / 467,286, filed May 17, 2023. The disclosure of the earlier application is considered part of the disclosure of this application and is incorporated herein by reference. Background Technology

[0002] This specification relates to the use of neural networks to control intelligent agents.

[0003] A neural network is a machine learning model that uses one or more layers of non-linear units to predict the output from a given input. Some neural networks are deep neural networks that include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output from the received input based on the current values ​​of its corresponding set of parameters. Summary of the Invention

[0004] This specification describes a system for obtaining one or more final action selection policies (or simply "final policies") for an intelligent agent in a control environment by a computer program implemented on one or more computers at one or more locations.

[0005] According to one aspect, a computer-implemented method is provided, comprising: obtaining a candidate policy set including a plurality of candidate policies for controlling an agent in an environment; obtaining an offline dataset storing a plurality of historical trajectories, wherein each historical trajectory includes a plurality of historical observations representing a corresponding historical state of the environment; and generating a behavioral representation for each candidate policy, wherein for each candidate policy, the method comprises: for each historical observation in a subset of the plurality of historical observations, generating a successor feature value for the historical observation under the candidate policy by using a successor feature neural network corresponding to the candidate policy, wherein the successor feature neural network is configured to receive and process the successor feature network input to generate the successor feature value for the historical observation, the successor feature network input including the historical observation and a corresponding action to be performed by the agent in response to the historical observation selected by using the candidate policy; generating a feature vector for the candidate policy based on the successor feature values ​​already generated for one or more historical observations in a subset of the plurality of historical observations; and determining the behavioral representation of the candidate policy from the feature vector of the candidate policy.

[0006] Generating a candidate strategy's feature vector may include calculating the average of subsequent feature values ​​generated for one or more historical observations in a subset of multiple historical observations.

[0007] The method may further include: determining an estimated performance metric for each candidate policy based on the behavioral representation of each candidate policy.

[0008] Determining the estimated performance metric for each candidate policy may include: using a learned model to process the behavioral representation of the candidate policy to generate the estimated reward of that candidate policy as the estimated performance metric.

[0009] Multiple candidate strategies may include at least one candidate strategy that has been evaluated, and wherein determining the estimated performance metric for each candidate strategy may include: for each remaining candidate strategy that has not yet been evaluated: determining the estimated performance metric for the remaining candidate strategies based on: (a) (i) the difference between the behavioral representation of the remaining candidate strategies and (ii) the behavioral representation of the candidate strategies that have been evaluated and (b) the known performance metric of the candidate strategies that have been evaluated.

[0010] Multiple candidate strategies can include a larger number of un-evaluated candidate strategies than the candidate strategies that have already been evaluated.

[0011] The method may further include obtaining a final policy for controlling the agent in the environment based on the estimated performance metric of each candidate policy.

[0012] Obtaining a final policy for controlling an agent in an environment may include: selecting a subset of multiple candidate policies based on estimated performance metrics of multiple candidate policies; controlling the agent in the environment using each candidate policy in the subset to determine the actual performance metric of the candidate policy, including selecting a sequence of actions to be performed by the agent based on the candidate policy; and selecting the candidate policy as the final policy based on the actual performance metric.

[0013] Obtaining a final policy for an agent to control an environment may include: receiving a current observation representing the current state of the environment; generating a final policy by applying a general policy improvement algorithm to multiple candidate policies; and selecting a current action to be performed by the agent in response to the current observation, based on the final policy.

[0014] The method may further include using an encoder neural network and training different successor feature neural networks corresponding to different candidate strategies using an offline dataset.

[0015] Training a successor feature neural network corresponding to a candidate policy may include: obtaining a training history trajectory from an offline dataset that includes a sequence of historical observations; processing each historical observation in the sequence of historical observations starting from a given historical observation using an encoder neural network to generate a corresponding encoded representation of the historical observation; processing a given historical observation in the sequence of historical observations using a successor feature neural network to generate a successor feature value for the given historical observation; evaluating a loss function that measures the difference between (i) the successor feature value of the given historical observation and (ii) the time-discounted sum of the corresponding encoded representations of the sequence of historical observations starting from the given historical observation; and determining one or more updates to the current values ​​of the parameters of the successor feature neural network based on the gradient of the calculated loss function with respect to the parameters of the successor feature neural network.

[0016] The encoder neural network may include one of the following: a convolutional neural network with randomly initialized or learned parameter values; an image encoder in an image-text neural network pre-trained on an image captioning task; a visual Transformer neural network pre-trained on an image classification task; or a neural network pre-trained on a video tracking task.

[0017] According to another aspect, a computer-implemented method is provided, comprising: obtaining data for a specified pre-trained model; maintaining an offline dataset storing multiple historical trajectories, wherein each historical trajectory includes multiple historical observations representing corresponding historical states of an environment; and training a successor feature neural network corresponding to a candidate policy using the pre-trained model and the offline dataset, wherein the training may include: obtaining training historical trajectories including a sequence of historical observations from the offline dataset; for each historical observation in the sequence of historical observations starting from a given historical observation, processing the model input including the historical observations using the pre-trained model to generate a corresponding encoded representation of the historical observation; and using... The successor feature neural network processes the input to the successor feature network to generate successor feature values ​​for a given historical observation, the input including (i) the given historical observation in the historical observation sequence, and (ii) an action selected using a candidate policy in response to the given historical observation; evaluates a loss function that measures the difference between (i) the successor feature value of the given historical observation and (ii) the time-discounted sum of the corresponding encoded representations of the historical observation sequence starting from the given historical observation; and determines one or more updates to the current values ​​of the parameters of the successor feature neural network based on the gradient of the loss function with respect to the parameters of the successor feature neural network.

[0018] The method may further include using a trained successor feature neural network to generate behavioral representations of candidate policies.

[0019] The method may further include using the behavioral representation of the candidate policy to determine whether to deploy the candidate policy to control an agent in the environment, or whether to use the candidate policy to generate a final policy for controlling the agent.

[0020] Pre-trained models can include pre-trained visual models.

[0021] The intelligent agent can be a mechanical intelligent agent, and the environment can be a real-world environment.

[0022] Intelligent agents can be robots.

[0023] The environment can be a real-world environment that includes a service facility with multiple electronic devices, and the intelligent agent can be an electronic intelligent agent configured to control the operation of the service facility.

[0024] The environment can be a real-world manufacturing environment used to manufacture products, and the intelligent agent can include an electronic intelligent agent configured to control the manufacturing unit or operate a machine to manufacture the product.

[0025] The environment may be a simulation of a real-world environment, and the method may further include: after generating a final policy, providing data specifying the final policy for use in controlling a real-world agent in a real-world environment.

[0026] According to another aspect, one or more computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the operations described above.

[0027] According to another aspect, a system is provided, comprising: one or more computers; and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform corresponding operations in respect of the above-described method.

[0028] The subject matter described in this specification can be implemented in specific embodiments to achieve one or more of the following advantages.

[0029] High-performance strategies can be effectively selected, which can be used to choose actions to be performed by agents interacting with the environment, thereby performing tasks with high performance (e.g., higher long-term, time-discounted rewards received by the agent while performing the task). In particular, a relatively small number of high-performance strategies can be identified even when a very large number of candidate strategies makes it impractical to try each candidate strategy individually to determine which strategies to use to control the agent—such as when the system cannot access the agent, or when online interaction with the environment may consume too much time and / or computational resources or may cause damage or depletion to the agent.

[0030] By generating behavioral representations of different candidate policies using different successor feature neural networks, which can be generated using, for example, a common policy-independent base model, the system described in this specification can automatically and accurately evaluate the performance of a set of candidate policies using primarily recorded data with minimal online environment interaction. Specifically, given historical observations of the environment, the behavioral representations generated by the successor feature neural networks can characterize how each candidate policy alters the environment by modeling the anticipated future occurrence of the environment's state, provided the candidate policies are followed.

[0031] In particular, it is no longer necessary to repeatedly evaluate the performance of all candidate policies through actual deployment on the agent; instead, the system can evaluate only a relatively small number of candidate policies that have already been selected based on its behavioral representation. In practice, the described system is highly useful in many technical use cases, such as industrial or robotic tasks, especially where there is a large amount of recorded data (e.g., demonstration data or past experience data) but a limited online interaction budget—e.g., due to the lengthy time or high hardware costs required to repeatedly deploy different candidate policies on the agent. From another perspective, this improvement in evaluation efficiency allows for the rapid and efficient exploration of a more diverse pool of candidate policies to obtain a better final policy.

[0032] Details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of this subject matter will become apparent from the description, the drawings, and the claims. Attached Figure Description

[0033] Figure 1 An example intelligent agent control system and an example policy evaluation system are shown.

[0034] Figure 2 This is an example illustration of the operations performed by the strategy evaluation system.

[0035] Figure 3 This is an example illustration of the architecture of a subsequent feature neural network.

[0036] Figure 4 This is a flowchart of an example process for obtaining the final policy of an agent used to control an environment.

[0037] Figure 5 yes Figure 4 A flowchart of a sub-step within a process.

[0038] Figure 6 yes Figure 4 A flowchart of a sub-step within a process.

[0039] Figure 7 This is a flowchart of an example process for training a subsequent feature neural network.

[0040] Figure 8 This is an example illustration of training a successor feature neural network.

[0041] The same reference numerals and names in the various figures indicate the same elements. Detailed Implementation

[0042] Figure 1 An example intelligent agent control system 100 and an example policy evaluation system 120 are illustrated. The intelligent agent control system 100 and the policy evaluation system 120 are examples of systems implemented as computer programs on one or more computers at one or more locations, in which the systems, components and techniques described below are implemented.

[0043] The agent control system 100 uses one or more final action selection policies 136 (or simply "final policies") to select an action 102 to be performed by the agent 104 interacting with the environment 106 at each of a plurality of consecutive time steps. At each time step, the system 100 receives data characterizing the current state of the environment 106 and selects an action 102 to be performed by the agent 104 in response to the received data, based on one or more final policies 136.

[0044] Each final policy 136 defines which action (according to the final policy) should be performed by the agent at each of the multiple time steps. Data characterizing the state of the environment 106—such as an image of the environment 106—will be referred to as observation 108 in this specification.

[0045] Once the agent control system 100 selects an action to be performed by the agent 104, the agent control system 100 can cause the agent 104 to perform the selected action. In other words, the agent control system 100 can use one or more final policies 136 to control the agent 104 to automatically perform a specific task without any user intervention.

[0046] For example, the system 100 can instruct the agent 104, and the agent 104 can perform the selected action. As another example, the system 100 can directly generate control signals for one or more controllable elements of the agent 104. As yet another example, the system 100 can transmit data specifying the selected action to the control system of the agent 104, which controls the agent 104 to perform the action.

[0047] Typically, agent 104 performs a selected action, causing environment 106 to transition to different states. By repeatedly causing agent 102 to act in environment 106, agent control system 100 can control agent 104 to complete a specific task.

[0048] The intelligent agent control system 100 described in this paper has wide applications and is not limited to a particular implementation. However, for illustrative purposes, a small number of example implementations are described below.

[0049] In some implementations, environment 106 is a real-world environment, and agent 104 is a mechanical agent that interacts with the real-world environment, such as a robot or an autonomous or semi-autonomous land, air, or sea vehicle that operates or navigates through the environment. The actions are performed by the mechanical agent in the real-world environment to execute tasks. For example, a robot's mechanical agent can interact with the environment to complete specific tasks, such as locating or manipulating an object of interest in the environment, moving the object of interest to a designated location in the environment, or navigating to a designated destination in the environment.

[0050] In these implementations, observation 108 may include one or more of the following: images, object position data, and sensor data, to capture observations as the agent interacts with its environment, such as sensor data from images, distance or position sensors, or actuators. For example, in the case of a robot, observation 108 may include data characterizing the robot's current state, such as one or more of the following: joint positions, joint velocities, joint forces, torques or accelerations (e.g., gravity-compensated torque feedback), and the global or relative pose of an object held by the robot. In the case of a robot or other mechanical agent or vehicle, observation 108 may similarly include one or more of the following: the position, linear or angular velocity, force, torque or acceleration of one or more parts of the agent, and the global or relative pose. Observation 100 may be defined in one, two, or three dimensions and may be an absolute and / or relative observation. Observation 100 may also include, for example, sensed electronic signals, such as motor current or temperature signals; and / or, for example, image or video data from a camera or LIDAR sensor, such as data from a sensor of an agent or data from a sensor located separately from an agent in the environment.

[0051] In these implementations, action 102 can be a control signal for controlling a robot or other mechanical agent, such as torque or higher-level control commands for the robot's joints; or it can be a control signal for controlling an autonomous or semi-autonomous land, air, or sea vehicle, such as torque or higher-level control commands for controlling surfaces or other control elements (e.g., steering control elements of the vehicle). The control signal can include, for example, position, velocity, or force / torque / acceleration data for one or more joints of a robot or parts of another mechanical agent. The control signal can also, or alternatively, include electronic control data, such as motor control data, or more generally, data for controlling one or more electronic devices within an environment, the control of which affects the observed state of the environment. For example, in the case of an autonomous or semi-autonomous land, air, or sea vehicle, the control signal can define actions for controlling navigation (e.g., steering) and movement (e.g., braking and / or acceleration of the vehicle).

[0052] In some implementations, environment 106 is a simulation of the aforementioned real-world environment, and agent 104 is implemented as one or more computers interacting with the simulated environment. For example, the simulated environment could be a simulation of a robot or vehicle, and agent control system 100 could be trained on this simulation and then used in the real world once trained.

[0053] In some implementations, environment 106 is a real-world manufacturing environment used to manufacture products such as chemical, biological, or mechanical products or food. As used herein, "manufacturing" a product also includes refining starting materials to create the product, or processing starting materials, for example, to remove contaminants, to produce a cleaned or recycled product. A manufacturing plant may include multiple manufacturing units, such as containers for chemical or biological substances, or machines for handling solids or other materials, such as robots. Manufacturing units are configured such that intermediate versions or components of the product can move between manufacturing units during product manufacturing, for example, via piping or mechanical transport. As used herein, product manufacturing also includes the manufacture of food by kitchen robots.

[0054] Intelligent agent 104 may include an electronic intelligent agent configured to control a manufacturing unit or machine, such as a robot, to manufacture a product. That is, the intelligent agent may include a control system configured to control the manufacture of a chemical, biological, or mechanical product. For example, the control system may be configured to control one or more manufacturing units or machines, or to control the movement of intermediate versions or components of a product between manufacturing units or machines.

[0055] As an example, the task performed by agent 104 may include manufacturing a product or an intermediate version or component thereof. As another example, the task performed by the agent may include controlling (e.g., minimizing) the use of resources, such as controlling electricity or water consumption or the consumption of any materials or consumables used in the manufacturing process.

[0056] Action 102 may include control actions to control the use of a machine or manufacturing unit for handling solid or liquid materials to manufacture a product or its intermediate products or components, or to control the movement of intermediate versions of a product or components within a manufacturing environment, such as between manufacturing units or machines. Generally, an action can be any action that affects the state of the observed environment, such as an action configured to adjust any of the sensed parameters described below. These actions may include actions for adjusting the physical or chemical conditions of a manufacturing unit, or actions for controlling the movement of mechanical parts of a machine or joints of a robot. Actions may include actions that impose operating conditions on a manufacturing unit or machine, or actions that result in changes to settings to adjust, control, or turn the operation of the manufacturing unit or machine on or off.

[0057] Rewards or rewards can be related to metrics of task performance. For example, in the case of a task involving the manufacture of a product, metrics could include the quantity of the product manufactured, the quality of that product, the rate at which the product was manufactured, or metrics of the physical costs of performing the manufacturing task, such as the amount of energy, materials, or other resources used to perform the task. In the case of a task involving the control of resource usage, metrics could include any measure of resource usage.

[0058] Generally, observations 108 of the state of environment 106 may include any electronic signals representing the operation of electronic and / or mechanical equipment. For example, the representation of the state of the environment may be derived from observations made by sensors that sense the state of the manufacturing environment (e.g., sensors that sense the state or configuration of a manufacturing cell or machine, or sensors that sense the movement of materials between manufacturing cells or machines). As some examples, such sensors may be configured to sense mechanical movement or forces, pressure, temperature; electrical conditions such as current, voltage, frequency, impedance; the quantity, level, flow / movement rate, or flow / movement path of one or more materials; physical or chemical conditions such as physical state, shape, or configuration, or chemical conditions such as pH; the configuration of a unit or machine, or valve configuration, such as the mechanical configuration of the unit or machine; image or video sensors for capturing image or video observations of the manufacturing cell or machine or its movement; or any other suitable type of sensor. In the case of a machine such as a robot, observations from sensors may include observations of the position, linear or angular velocity, force, torque, or acceleration, or attitude of one or more parts of the machine, such as data characterizing the current state of the machine or robot or an item held or handled by the machine or robot. Observations may also include, for example, sensed electronic signals, such as motor current or temperature signals, or image or video data, such as from camera or lidar sensors. Sensors like these may be part of an agent in the environment or located separately from the agent.

[0059] In some implementations, environment 106 is a real-world environment comprising a service facility including multiple electronic devices, such as server clusters or data centers, like telecommunications data centers, or computer data centers for storing or processing data, or any service facility. The service facility may also include auxiliary control equipment for controlling the operating environment of the equipment, such as environmental control equipment, such as temperature control (e.g., cooling) equipment, or airflow control or air conditioning equipment, such as heaters, coolers, humidifiers, or other hardware that modifies the properties of air in the real-world environment. Tasks may include tasks for controlling (e.g., minimizing) the use of resources, such as controlling power consumption or water consumption. Agent 104 may include an electronic agent configured to control the operation of the equipment or to control the operation of auxiliary (e.g., environmental) control equipment.

[0060] Generally, action 102 can be any action that affects the state of the observed environment, such as an action configured to adjust any of the sensed parameters described below. These actions can include actions that control or assist control equipment or impose operating conditions on the equipment or assist control equipment, such as actions that cause a change in settings to adjust, control, or turn the equipment or assist control equipment on or off.

[0061] Generally, the observation 108 of the state of environment 106 may include any electronic signals indicating the operation of the facility or equipment within the facility. For example, the representation of the state of the environment may be derived from observations made by any sensor sensing the state of the physical environment of the facility or by any sensor sensing the state of one or more pieces of equipment or one or more auxiliary control devices within the equipment. These include sensors configured to sense: electrical conditions such as current, voltage, power, or energy; the temperature of the facility; fluid flow rate, temperature, or pressure within the facility or its cooling system; or physical facility configurations such as whether vents are open.

[0062] Rewards or rewards can be related to metrics of task performance. For example, in the case of tasks that control (e.g., minimize) the use of resources, such as controlling the use of electricity or water, the metric can include any measure of resource usage.

[0063] In some implementations, environment 106 is the real-world environment of a power generation facility (e.g., a renewable power generation facility, such as a solar power plant or a wind power plant). Tasks may include control tasks related to controlling the electricity generated by the facility (e.g., controlling the delivery of electricity to a distribution network, such as to meet demand or reduce the risk of mismatch between network components, or to maximize the electricity generated by the facility). Intelligent agents may include electronic agents configured to control the generation of electricity by the facility or the coupling of the generated electricity to the grid. Actions may include actions controlling the electrical or mechanical configuration of a generator (such as the electrical or mechanical configuration of one or more renewable power generation elements) (e.g., controlling the configuration of a wind turbine or solar panel or mirror, or the electrical or mechanical configuration of a rotating power generation machine). Mechanical control actions may, for example, include actions controlling the transformation of energy input to electrical output (e.g., transformation efficiency or the degree of coupling between energy input and electrical output). Electrical control actions may, for example, include actions controlling one or more of the voltage, current, frequency, or phase of the generated electricity.

[0064] Rewards or rewards can be related to metrics of task performance. For example, in the case of a task controlling the delivery of electricity to the distribution network, the metric could be related to a measure of the electricity delivered, or a measure of electrical mismatches (such as voltage, current, frequency, or phase mismatches) between the generating facility and the grid, or a measure of power or energy losses in the generating facility. In the case of a task maximizing the delivery of electricity to the distribution network, the metric could be related to a measure of the electricity or energy delivered to the grid, or a measure of power or energy losses in the generating facility.

[0065] Generally, observations 108 of the state of environment 106 may include any electronic signals representing the electrical or mechanical operation of the power generation equipment in the power generation facility. For example, the representation of the state of the environment may be derived from observations by any sensor that senses the physical or electrical state of the equipment in the power generation facility generating electricity, or the physical environment of such equipment, or the state of auxiliary equipment supporting the power generation equipment. Thus, such observations may include observations of wind speed or solar irradiance, or of local time, date, or season. Such sensors may include sensors configured to sense: the electrical state of the equipment, such as current, voltage, power, or energy; the temperature or cooling of the physical environment; fluid flow; or the physical configuration of the equipment; and observations of the electrical state of the power grid, such as from local or remote sensors. Observations of the state of the environment may also include one or more predictions of future operating conditions of the power generation equipment, such as predictions of future wind speeds or solar irradiance, or predictions of future electrical states of the power grid.

[0066] As another example, the environment can be a chemical synthesis or protein folding environment, such that each state is a corresponding state of a protein chain or one or more intermediate or precursor chemicals, and the agent is a computer system for determining how to fold the protein chain or synthesize the chemical. In this example, actions are possible folding actions for folding the protein chain or actions for assembling the precursor chemical / intermediate, and the desired outcome can include, for example, folding the protein to make it stable and enabling it to perform a specific biological function, or providing an efficient synthetic route for the chemical. As another example, the agent can be a mechanical agent that indirectly performs or controls protein folding actions or chemical synthesis steps, for example, by controlling synthetic steps automatically selected by the system without human intervention. Observation can include direct or indirect observation of the state of the protein or chemical / intermediate / precursor, and / or can be derived from a simulation. Thus, the system can be used to automate the synthesis of proteins with specific functions, such as proteins with binding site shapes, for example, binding with sufficient affinity to obtain ligands that can be used as drugs for their biological effects. For example, it can be an agonist or antagonist of a receptor or enzyme; or it can be an antibody configured to bind to an antibody target (such as a viral capsid protein or a protein expressed on a cancer cell), for example, to act as an agonist of a specific receptor or to prevent the binding of another ligand and thus prevent activation of the associated biological pathway.

[0067] In a similar manner, environment 106 can be a drug design environment, such that each state is a potential pharmaceutically active compound, i.e., the corresponding state of a drug, and the agent is a computer system for determining the elements of the pharmaceutically active compound and / or the synthetic pathway for the pharmaceutically active compound. For example, in a simulation, a drug / synthesis can be designed based on a reward derived from the target of the pharmaceutically active compound. Agent 104 can be or can include a mechanical agent that performs or controls the synthesis of the pharmaceutically active compound; and therefore, the process as described herein can include the preparation of such pharmaceutically active compounds.

[0068] For example, environment 106 can be a computer-simulated drug design environment, such as a molecular docking environment, and agent 104 can be a computer system for determining the elemental or chemical structure of the drug. The drug can be a small molecule or a biological drug. Observation 108 can be an observation of a simulated combination of the drug and its target. Actions can be actions to modify the relative position, orientation, or conformation of the drug and its target (or these can be performed automatically) and / or actions to modify the chemical composition of the drug and / or to select candidate drugs from a drug candidate library. One or more rewards can be defined based on one or more of the following: a measure of the interaction between the drug and its target, such as a measure of coordination or binding between the drug and its target; the estimated potency of the drug; the estimated selectivity of the drug; the estimated toxicity of the drug; the estimated pharmacokinetic properties of the drug; the estimated bioavailability of the drug; the estimated ease of synthesis of the drug; and one or more fundamental chemical properties of the drug. The interaction between a drug and its target can be measured by, for example, protein-ligand bonds, van der Waals interactions, electrostatic interactions, and / or contact surface area or energy; this measurement may include, for example, docking fraction. After identifying the elemental or chemical structure of the drug in a simulation, the method may further include the preparation of the drug. The drug may be prepared partially or entirely by an automated chemical synthesis system.

[0069] In some applications, agent 104 can be a software agent, i.e., a computer program, configured to perform tasks. For example, environment 106 can be a circuit or integrated circuit design or routing environment, and the agent can be configured to perform the task of designing or routing interconnects for routing circuits or integrated circuits (e.g., ASICs). The reward can then depend on one or more routing metrics, such as interconnect length, resistance, capacitance, impedance, loss, speed, or propagation delay; and / or physical line parameters, such as width, thickness, or geometry, and design rules. The reward may also, or alternatively, include one or more rewards related to global properties of the routed circuit system, such as component density, operating speed, power consumption, material usage, cooling requirements, electromagnetic radiation levels, etc. Observations can be, for example, observations of component locations and interconnects; actions can include, for example, component placement actions and / or interconnect routing actions for defining component locations or orientations, such as interconnect selection and / or placement actions. Tasks can be, for example, optimizing circuit operation to reduce electrical losses, local or external interference, or heat generation; or increasing operating speed; or minimizing or optimizing the use of available circuit area. The method may include fabricating circuits or integrated circuits according to a design, or fabricating circuits or integrated circuits using interconnects laid out as determined by the method.

[0070] In some applications, agent 104 is a software agent, and the environment is a real-world computing environment. In one example, the agent manages the distribution of tasks across computing resources, such as on a mobile device and / or in a data center. In these applications, observations may include observations of computing resources such as computing and / or memory capacity or internet-accessible resources; and actions may include assigning tasks to specific computing resources. Rewards can be configured to maximize or minimize one or more of the following: utilization of computing resources, power, bandwidth, and computing speed.

[0071] In another example, the software agent manages the processing of a queue of consecutively arriving jobs, for example, through one or more real-world servers. Observations may include observations of the departure times of consecutive jobs, or the time intervals between the departures of consecutive jobs, or the time taken by the server to process each job (e.g., the start and end of a time frame), or the arrival times of consecutive jobs, or the time intervals between the arrivals of consecutive jobs, or observations of data characterizing the type of job. These actions 102 may include actions such as assigning a specific job to a specific computing resource; rewards may be configured to minimize total queuing or processing time or queuing or processing time for one or more individual jobs; or generally to optimize any metric based on observations.

[0072] As another example, environment 106 may include a real-world computer system or network, observation 108 may include any observation characterizing the operation of the computer system or network, actions performed by the software agent may include actions for controlling the operation, such as actions for limiting or correcting abnormal or undesirable operations, such as those caused by the presence of viruses or other security vulnerabilities, and rewards may include any metric characterizing the desired operation of the computer system or network.

[0073] In some applications, environment 106 is a real-world computing environment, and the software agent manages the distribution of tasks / jobs across computing resources (e.g., on mobile devices and / or in data centers). In these implementations, observation 108 may include observations related to the operation of computing resources in processing tasks / jobs, actions may include assigning tasks / jobs to specific computing resources, and rewards may be related to one or more metrics of using computing resources to process tasks / jobs, such as metrics of computing resource, bandwidth, or power usage, or metrics of processing time or numerical precision, or one or more metrics related to the expected load balancing among computing resources.

[0074] In some applications, the environment is a data packet communication network environment, and the agent is part of a router used to route data packets through the communication network. Actions can include data packet routing actions, and observations can include, for example, observations of a routing table that includes routing metrics such as path length, bandwidth, load, hop count, path cost, latency, maximum transmission unit (MTU), and reliability. Rewards can be defined relative to one or more of the routing metrics, i.e., configured to maximize one or more of the routing metrics.

[0075] In some other applications, the environment is an internet or mobile communication environment, and the agent is a software agent that manages personalized recommendations for users. Observations may include previous actions taken by the user, such as characteristics characterizing those actions; these actions may include recommending items such as content items to the user. The reward may be configured to maximize one or more of the following: the estimated likelihood that the user will respond favorably to the recommended (content) item, the suitability or unsuitability of one or more recommended items, the cost of the recommended item, and optionally the number of recommendations received by the user over a time span.

[0076] As a further example, action 102 may include presenting an advertisement, observation 108 may include ad flashes or click counts or click-through rates, and rewards may represent previous selections of an item or content by one or more users.

[0077] In some cases, observation 108 may include text or verbal instructions provided to the agent by a third party (e.g., the agent's operator). For example, agent 104 may be an autonomous vehicle, and the user of the autonomous vehicle may provide text or verbal instructions to the agent (e.g., to navigate to a specific location).

[0078] As another example, environment 106 may be an electromechanical or mechatronic design environment, such as an environment in which the design of an electromechanical or mechatronic entity is simulated. The simulated environment may be a simulation of a real-world environment in which the entity is intended to operate. The task may be designing the entity. Observation 108 may include observations characterizing the entity, i.e., observations of the entity's mechanical shape or electromechanical or mechatronic configuration, or observations of the entity's parameters or properties. Action 102 may include actions that modify the entity, such as actions that modify one or more of the observations. Rewards or rewards may include one or more metrics of the performance of the entity's design. For example, rewards or rewards may be related to one or more physical characteristics of the entity (such as weight or force); or to one or more electrical characteristics of the entity (such as a measure of efficiency in performing a specific function designed for the entity). The design process may include outputting a design for manufacturing, for example in the form of computer-executable instructions for manufacturing the entity. The process may include manufacturing the entity according to the design. Thus, the design of the entity can be optimized, for example through reinforcement learning, and then the optimized design is output for manufacturing the entity, for example as computer-executable instructions; the entity with the optimized design can then be manufactured.

[0079] As previously mentioned, environment 106 can be a simulated environment. Generally, in the case of a simulated environment, observations can include simulated versions of one or more of the previously described observations or observation types, and actions can include simulated versions of one or more of the previously described actions or action types. For example, the simulated environment can be a motion simulation environment, such as a driving simulation or flight simulation, and the agent can be a simulated vehicle navigating through the motion simulation. In these implementations, actions can be control inputs controlling the simulated user or the simulated vehicle. Typically, the agent can be implemented as one or more computers interacting with the simulated environment.

[0080] The simulated environment can be a simulation of a specific real-world environment and an intelligent agent. For example, the system can be used to select actions in the simulated environment during training or evaluation, and after training or evaluation, or both, can be deployed to control a real-world agent in the specific real-world environment that is the object of the simulation. This avoids unnecessary depletion and damage to the real-world environment or the real-world agent, and allows the control neural network to be trained and evaluated in situations where it is rare, difficult, or unsafe to recreate in the real-world environment. For example, the system can be partially trained using a simulation of a mechanical agent in a simulation of a specific real-world environment, and then deployed to control a real mechanical agent in that specific real-world environment. Therefore, in such cases, the observation of the simulated environment is relevant to the real-world environment, and the selected actions in the simulated environment are relevant to the actions to be performed by the mechanical agent in the real-world environment.

[0081] In some implementations, observation 108 is an observation of the real-world environment in which a human is performing a task, such as image observation from an image sensor and / or language observation from a speech recognition system; and action is, for example, a linguistic action that uses natural language or image control (instruction) to perform an action in the real-world environment to perform a task. A linguistic action can be an action that outputs a natural language sentence, for example by defining a sequence of linguistic lemmas to be emitted at consecutive time steps, such as words or word fragments.

[0082] Therefore, agent 104 may include a user interface device, such as a digital device (“digital assistant”), like a smart speaker or smart display, or other device, that controls (instructs) a human user to perform tasks, such as controlling (instructing) a human user to perform tasks. Generally, such a digital device may be a mobile device with a natural language interface for receiving natural language requests from a human user and providing natural language responses. It may also include vision-based input, such as a camera and / or a display screen. The digital device may include a language model or language generation neural network system that is locally stored or remotely accessed, or both. The user interface device may include, for example, a mobile device, a keyboard (and optionally a display screen), or a voice-based input mechanism, such as audio data representing speech waveforms that represent speech input from a user in natural or computer language, and converting the audio data into lexical units representing speech in natural or computer language, i.e., lexical units representing the transcription of spoken input. The user interface may also include text- or speech-based output, such as a display screen and / or a text-to-speech subsystem.

[0083] Therefore, in this implementation, the agent's actions contribute to task performance. A monitoring system (e.g., a camera system) can be provided to monitor the actions actually performed by the user at each time step (if any), in case (e.g., due to human error) differ from the actions instructed by the agent control system 100. The monitoring system can be used to determine whether the task has been completed. Training data can be collected by recording the actions actually performed by the user based on instructions. For example, reward values ​​for actions can be generated by comparing the user's actions with a corpus showing data on human experts performing tasks, for example, using techniques known from imitation learning or in some other way—e.g., using a trained reward model. This type of system can learn how to guide humans in performing tasks, thereby, for example, avoiding actions that are difficult to perform.

[0084] Optionally, in any of the above implementations, the observation 108 at any given time step may include data from previous time steps that may be helpful in characterizing the environment, such as actions performed at previous time steps, rewards received at previous time steps, or both.

[0085] To generate one or more final strategies 136 for use by the agent control system 100, the strategy evaluation system 120 may first obtain or pre-generate a candidate strategy set 130, which includes multiple candidate strategies 134A-N for controlling the agent 104 to perform a specific task. Then, the strategy evaluation system 120 performs a quantitative evaluation of each of the multiple candidate strategies 134A-N included in the candidate strategy set 130 to generate a quantitative evaluation result for each candidate strategy, and uses the quantitative evaluation result to determine one or more final strategies 136 for use by the agent control system 100 to select action 102 for the agent 104 to perform when performing a specific task.

[0086] Typically, the agent control system 100 is located locally on agent 104. For example, the agent control system 100 may be on-board on agent 104, such as on one or more computers, local workstations, or local servers with relatively small processing and memory resources.

[0087] In some implementations, the policy evaluation system 120 is located locally on agent 104. For example, like agent control system 100, policy evaluation system 120 may also be on agent 102. Furthermore, in some of these implementations, policy evaluation system 120 may be part of agent control system 100, which causes agent 104 to perform action 144. In these implementations, agent control system 100 and policy evaluation system 120 may be managed by the same organization.

[0088] In other implementations, the policy evaluation system 120 is located remotely from the agent 104. For example, unlike the agent control system 100, the policy evaluation system 120 may be hosted in a data center, which may be a distributed computing system with hundreds or thousands of computers in one or more locations. In these implementations, the agent control system 100 and the policy evaluation system 120 may be managed by different organizations.

[0089] The candidate policy set 130 includes multiple candidate policies 134A-N, each defining which action (according to the candidate policy) the agent should perform at each time step across multiple time steps. The multiple candidate policies 134A-N reflect different methods for performing the same specific task. In other words, the multiple candidate policies 134A-N represent different solutions to the same technical problem of agent control.

[0090] An example of a candidate policy is a non-trainable action selection policy, such as an action selection policy that does not use neural network layers to select actions and / or is non-differentiable. For example, a candidate policy could be one that, when used, continuously controls the agent to perform the same single action. As another example, a candidate policy could be one that selects actions with uniform randomness. As yet another example, a candidate policy could be a manually programmed policy that selects actions based on some hard-coded logic.

[0091] Another example of a candidate policy is a trainable action selection policy. In this example, the candidate policy can be implemented as a trained candidate policy neural network, which has been configured, for example, to process the candidate policy network input, including a current observation representing the current state of the environment, at each of multiple time steps based on learned values ​​of the network parameters, using conventional reinforcement learning techniques, to generate a candidate policy network output specifying the action to be performed by the agent in response to the current observation.

[0092] As a more general example, the candidate policy can be any black-box action selection policy, such that the policy evaluation system 120 does not need to access the internal workings of the action selection logic of the candidate policy. In the example where the candidate policy is implemented as a trained candidate policy neural network, the policy evaluation system 120 does not need to access the parameters or architecture of the trained candidate policy neural network.

[0093] The candidate policy set 130 can be obtained in any of a variety of ways. For example, the policy evaluation system 120 can receive data defining candidate policies as uploaded content from a remote user of the system via a data communication network, for example, using an application programming interface (API) available to the system. As a particular example, candidate policies can be provided by a third-party policy developer—that is, a developer who is neither affiliated with the manufacturer of agent 104 nor with the organization managing the policy evaluation system 120. As another example, the policy evaluation system 120 can receive input from a user specifying which data, already maintained by system 120 or another system accessible to system, should be used as data defining candidate policies. As yet another example, the policy evaluation system 120 or another system can use conventional reinforcement learning techniques to generate candidate policies implemented as a trained candidate policy neural network.

[0094] To evaluate each of the multiple candidate policies 134A-N, the policy evaluation system 120 includes or accesses multiple successor feature neural networks 150A-N and a performance prediction model 160. The policy evaluation system 120 also includes or accesses an offline dataset 140, which includes multiple historical observations of the environment 106.

[0095] Multiple successor feature neural networks 150A-N correspond to multiple candidate policies 134A-N, respectively. That is, each successor feature neural network corresponds to a different candidate policy from the multiple candidate policies 134A-N that can be used to control the agent 104 to perform a specific task. For example, in... Figure 1 In this context, the successor feature neural network 150A corresponds to the candidate policy 134A, the successor feature neural network 150B corresponds to the candidate policy 134B, and so on.

[0096] Each candidate policy neural network can have any suitable architecture, such as feedforward or recurrent architectures, such as a multilayer perceptron (MLP) including tanh activation layers (or another layer with a chosen activation function), or a convolutional neural network, which allows the candidate neural network to map observations to the candidate policy network output for selecting actions. In this example, training of each candidate policy neural network can be performed locally on the policy evaluation system 120, or alternatively on a remote, cloud-based training system. An example of how to obtain the successor feature neural networks 150A-N will be referenced below. Figures 7 to 8 To describe in more detail.

[0097] For example, the output of a candidate policy network can be a probability distribution over a set of possible actions. As another example, the output can include a Q-value, which is an estimate of the long-term, time-discounted reward that would be received if the agent performs a particular action in response to an observation. As yet another example, the output can identify a specific action, for instance, by defining the mean and variance of the torque of each of several movable components, such as a joint, to be applied to a robot.

[0098] Offline dataset 140 stores empirical data obtained as a result of interactions between agent 104 or another agent and environment 106 or another instance of the environment. Additionally or alternatively, offline dataset 140 stores empirical data derived from environmental interaction information obtained from any of a variety of other relevant sources, such as driving logs in the case of an autonomous or semi-autonomous vehicle. Therefore, offline dataset 140 can be obtained before evaluating candidate policies 134A-N using policy evaluation system 120.

[0099] For example, the agent can be controlled by this action control system 100 or another control system, including control by a human-operated or another machine learning-based control system, control by a hard-coded processing strategy that selects actions to be performed by the agent according to pre-programmed logic, or simply control by a random strategy that selects actions with uniform randomness, etc.

[0100] The empirical data includes multiple historical trajectories 144A-N. Each historical trajectory then includes multiple historical observations. Each historical observation represents a historical state of environment 106 or another instance of the environment with which agent 104 or other agents interact.

[0101] In some implementations, the policy evaluation system 120 may similarly obtain the offline dataset 140, for example, as content uploaded from a remote user of the system or from another system. In other implementations, the policy evaluation system 120 may obtain the offline dataset 140 by controlling the agent 104 to interact with the environment 104 and then including the observations received by the system during the interaction in the offline dataset 140.

[0102] Figure 2 This is an example illustration 200 of the operation performed by the policy evaluation system 120 to generate quantitative evaluation results for each of the multiple candidate policies 134A-N included in the candidate policy set 130. For each candidate policy, the quantitative evaluation result includes an estimated performance metric for that candidate policy, i.e., an estimate of the performance of that candidate policy when it is used to select actions to be performed by the agent to interact with the environment to perform a specific task.

[0103] The policy evaluation system 120 generates a behavioral representation for each candidate policy by using a successor feature neural network corresponding to each candidate policy in the candidate policy set 130 and by accessing an offline dataset 140. As used herein, the behavioral representation is, for example, a vector or other data structure of numerical values—e.g., floating-point values ​​or other values—with a predetermined dimension in the latent space.

[0104] More specifically, for each candidate policy in the candidate policy set 130, the policy evaluation system 120 can select a subset of historical observations by sampling from all historical observations stored in the offline dataset 140, for example, with uniform randomness. Then, for each candidate policy in the candidate policy set 130, and for each selected historical observation in the subset, the policy evaluation system 120 processes the subsequent feature network input using a subsequent feature neural network corresponding to that candidate policy to generate a subsequent feature value for that historical observation. The subsequent feature network input includes (i) the historical observation (in this particular example, the historical observation includes visual state features) and (ii) data identifying the corresponding action to be performed by the agent 104 in response to the historical observation, which can be selected using the candidate policy. As discussed below, in some cases, the subsequent feature values ​​can have a dimension greater than one, i.e., the feature values ​​can be an N-dimensional vector, where It is the dimension of the successor eigenvalues.

[0105] therefore, Figure 2 It is shown that for the first candidate strategy The strategy evaluation system 120 selects a subset of historical observations s from the offline dataset 140, and for each historical observation s included in the subset, uses the strategy with the first candidate strategy. The corresponding first successor feature neural network Process the input of the successor feature network to generate successor feature values ​​for historical observations s The subsequent feature network input includes (i) historical observations, and (ii) identifications that can be obtained by using a first candidate strategy. The data selected is the corresponding action to be performed by agent 104 in response to the historical observation, and in some implementations, (iii) for example, non-visual state features of proprioceptive information, or data derived from non-visual state features, or both. Subsequent feature values ​​can indicate the expected future occurrence of the state of the environment for the corresponding policy, given the historical observation.

[0106] The strategy evaluation system 120 then combines the subsequent feature values ​​generated for the historical observations s included in the subset to generate a first candidate strategy. eigenvectors The strategy evaluation system 120 can implement this in any suitable manner. For example, feature vectors. It could be the average of subsequent feature values ​​already generated for historical observations s included in the subset. As another example, the feature vector... It can be the sum of successor feature values ​​already generated for the historical observations s included in the subset.

[0107] exist Figure 2 In the middle, by using the second candidate strategy respectively The corresponding second successor feature neural network and third candidate strategies The corresponding third successor feature neural network Furthermore, by accessing the offline dataset 140, the policy evaluation system 120 can similarly generate a second candidate policy. eigenvectors and third candidate strategy eigenvectors .

[0108] The successor feature network can typically have any suitable neural network architecture that enables each successor feature network in the successor feature network to perform the functions described therein, namely: receiving inputs to the successor feature network, which include (i) historical observations, (ii) data identifying corresponding actions selected by using candidate strategies, and in some implementations, (iii) non-visual state features such as proprioceptive information, or data derived from non-visual state features, or both; and processing the inputs to the successor feature network to generate successor feature values ​​for the historical observations.

[0109] However, in general, each successor feature neural network can be a neural network that includes multiple neural network components that cooperate to generate successor feature values. For example, each successor feature neural network can include any suitable type (e.g., fully connected layers, convolutional layers, attention layers, recurrent layers, etc.) and any suitable number of layers (e.g., 5, 10, or 25 layers) connected in any suitable configuration (e.g., as a linear sequence of layers).

[0110] In some implementations, the successor feature neural networks 150A-N, including those in or accessed by the policy evaluation system 120, have the same architecture. For example, the successor feature neural networks 150A-N can have the same architecture but can have different parameter values. In other implementations, the architectures of the successor feature neural networks 150A-N differ from one another. For example, Figure 1 The successor feature neural network 150A-N can include different numbers of neural network layers, different types of neural network layers, or both.

[0111] The specific example architecture of the subsequent feature neural network will now be described.

[0112] Figure 3 This is a diagram 300 illustrating an example architecture of a subsequent feature neural network. The following will refer to... Figure 7 Further discussion is needed to obtain the desired result as shown in Figure 9. Figure 3 The example architecture shown or any other architecture is a subsequent feature of the neural network.

[0113] exist Figure 3 In the example, the successor feature neural network includes multiple embedded subnetworks 310, 320, 330 and a multilayer perceptron subnetwork 340. A subnetwork of a neural network refers to a group of one or more neural network layers in the neural network.

[0114] The plurality of embedding subnetworks includes a first embedding subnetwork 310, which is configured to process data recognizing a selected action to generate action embedding data. The plurality of embedding subnetworks also includes a second embedding subnetwork 320, which is configured to process historical observations to generate historical observation embeddings. Optionally, the plurality of embedding subnetworks further includes a third embedding subnetwork 330, which is configured to process non-visual state features, or data derived from non-visual state features, or both, to generate non-visual state feature embeddings. For example, non-visual state features may include proprioceptive information.

[0115] As an example, the first embedding subnetwork 310 and the third embedding subnetwork 330 can each be a corresponding fully connected subnetwork, i.e., a corresponding fully connected subnetwork including one or more fully connected neural network layers. In some implementations, the first embedding subnetwork 310 can be the same as the third embedding subnetwork 330, i.e., the subsequent feature neural network uses the shared embedding subnetwork to generate action embeddings and non-visual state feature embeddings.

[0116] The second embedded subnetwork 320 can be, for example, a convolutional subnetwork, i.e., a convolutional subnetwork that includes one or more convolutional neural network layers.

[0117] The multilayer perceptron subnetwork 340 includes one or more fully connected neural network layers. The multilayer perceptron subnetwork 340 operates on combinations (e.g., cascades) of (i) action embeddings, (ii) historical observation embeddings, and optional (iii) non-visual state feature embeddings to generate subsequent feature values.

[0118] Subsequent feature values ​​can be generated deterministically, for example, through the output of the multilayer perceptron subnetwork 340; or randomly, for example, where the output of the multilayer perceptron subnetwork 340 parameterizes one or more distributions from which it samples subsequent feature values. As a specific example of the latter, the multilayer perceptron subnetwork 340 generates... The output is a vector of n dimensions, where It is the number of intervals (or gaps) in the distribution, and It is the dimension of the successor eigenvalues.

[0119] For each of the multiple candidate policies 134A-N included in the candidate policy set 130, after the feature vector of the candidate policy has been generated in this manner, the policy evaluation system 120 then determines the behavioral representation of the candidate policy from the feature vector.

[0120] In some implementations, the feature vector generated by combining (e.g., averaging or summing) subsequent feature values ​​generated by the subsequent feature neural network is directly used by the policy evaluation system 120 as a behavioral representation of the candidate policy. In other implementations, the feature vector is further processed by the policy evaluation system 120 to provide a behavioral representation. For example, one or more linear or nonlinear neural network layers can be used to process the feature vector to generate a behavioral representation.

[0121] After the behavioral representation has been generated, the policy evaluation system 120 determines the estimated performance metric for each of the multiple candidate policies 134A-N included in the candidate policy set 130 based on the behavioral representation of the candidate policies.

[0122] Specifically, the strategy evaluation system 120 can achieve this by processing the behavioral representation of each candidate strategy using a reward estimation model 160 to generate an estimated reward for that candidate strategy, for example, by generating the estimated reward for the candidate strategy from the state of the environment as characterized by historical observations. The estimated reward is then used as an estimated performance metric for the candidate strategy.

[0123] The return estimation model 160 has any suitable architecture that allows the model 160 to map inputs, including behavioral representations, to the estimated return. For example, the return estimation model 160 can be a linear model, i.e., a model that multiplies the inputs to the model by a parameter matrix and then optionally adds the bias values ​​to the product to generate the estimated return. As another example, the return estimation model 160 can be a neural network model that processes the inputs to the model through multiple neural network layers to generate the estimated return.

[0124] In any example, in some implementations, such a reward estimation model 160 may be trained on training data comprising multiple training tuples based on an optimized loss function, each specifying (i) a behavioral representation of an action selection policy that can be used, for example, to select an action to be performed by an agent in response to an observation representing the state of the environment, and (ii) a reward received by the agent after the action selected by the action selection policy has been performed in response to the observation, the loss function evaluating for each training tuple the difference between (i) the estimated reward generated by the model based on the processing behavioral representation and (ii) the reward specified by that training tuple.

[0125] After generating estimated performance metrics for each candidate policy, the policy evaluation system 120, based on the multiple candidate policies 134A-N included in the candidate policy set 130 and their estimated performance metrics, obtains one or more final policies 136 for controlling the agent 104 to perform a specific task in the environment 106. (Refer to the following...) Figures 5 to 6 As discussed further, there are many ways in which the strategy evaluation system 120 can achieve this.

[0126] In some implementations, the policy evaluation system 120 provides data specifying a final policy 136 to the agent control system 100—and the agent control system 100 deploys the final policy 136 and uses it at each of multiple time steps throughout the trajectory to select actions 102 to be performed by the agent 104, thereby controlling the agent 104's interaction with the environment 106. For example, such a final policy 136 generated by the policy evaluation system 120 would be used to select actions at different time steps throughout the trajectory.

[0127] A trajectory represents a sequence of interactions between agent 104 and environment 106 over a time-step sequence. For example, a trajectory may represent an episode of a specified task over a time-step sequence during which the agent attempts to perform the specified task. For example, a task episode may last for a predetermined number of time steps, or until a reward indicating that the task has been successfully completed is received.

[0128] Furthermore, in some implementations, the policy evaluation system 120 can provide data, such as data from another agent control system, specifying a final policy 136 already generated by the policy evaluation system 120 for use in controlling the agent to perform a specific task. This allows the other system to store the final policy for other purposes, such as modifying it for use when selecting actions to be performed by the agent to perform other tasks. In some implementations, the agent control system 100 can also output estimated performance metrics to a user or system that has provided candidate policies 134A-N.

[0129] Figure 4 This is a flowchart of an example process 400 for controlling an intelligent agent in an environment. For convenience, process 400 will be described as being executed by a system of one or more computers located in one or more locations. For example, a properly programmed policy evaluation system, such as... Figure 1 The strategy evaluation system 120 can execute process 400.

[0130] The system obtains a set of candidate policies, including multiple candidate policies (step 402). Each final policy defines which action should be performed by the agent at each of the multiple time steps (according to the candidate policy). The multiple candidate policies may include untrainable action selection policies, such as random action selection policies or action selection policies defined by hard-coded processing logic; trainable action selection policies, such as candidate policies of a trained candidate policy neural network implemented to map observations to data specifying actions to be performed by the agent in response to those observations; or both untrainable and trainable action selection policies.

[0131] The system obtains an offline dataset generated as a result of an interaction between an agent or another agent and the environment or another instance of the environment (step 404). Each historical trajectory includes multiple historical observations. Each historical observation represents a corresponding historical state of the environment. For example, each historical observation may correspond to a specific time during a task segment and may include visual state features of the environment and / or non-visual state features of the environment.

[0132] The system then performs steps 406 to 412 to generate an estimated performance metric for each candidate policy. In other words, by repeating steps 406 to 412 multiple times, the system can generate corresponding estimated performance metrics for multiple candidate policies included in the candidate policy set.

[0133] For each historical observation in a subset of multiple historical observations, the system generates a successor feature value for that historical observation under the candidate policy by using a successor feature neural network corresponding to the candidate policy (step 406).

[0134] The system can obtain a subset of historical observations by sampling from all historical observations stored in an offline dataset, for example, with uniform randomness. By sampling from the offline dataset, the system typically selects different historical observations for different candidate strategies over multiple iterations in steps 406 to 412.

[0135] Furthermore, the system maintains multiple successor feature neural networks, each corresponding to a specific candidate strategy in the candidate strategy, and uses different successor feature neural networks for different candidate strategies in multiple iterations from steps 406 to 412.

[0136] For each historical observation in a subset of multiple historical observations, the successor feature neural network is configured to: receive a successor feature network input comprising (i) the historical observation, (ii) data identifying a corresponding action to be performed by the agent in response to the historical observation, which can be selected by using a candidate strategy, and in some implementations, (iii) non-visual state features such as proprioceptive information, or data derived from non-visual state features, or both; and process the successor feature network input to generate a successor feature value for the historical observation.

[0137] The system generates feature vectors for candidate strategies based on the combination of subsequent feature values ​​generated for subsets of multiple historical observations—e.g., averaging or summing—(step 408).

[0138] The system determines the behavioral representation of the candidate policy from the feature vectors of the candidate policy (step 410). In some implementations, the feature vectors are directly used by the system as the behavioral representation of the candidate policy, while in others, the feature vectors are further processed by the system to provide a behavioral representation. For example, one or more linear or nonlinear neural network layers can be used to process the feature vectors to generate a behavioral representation.

[0139] The system determines the estimated performance metric of the candidate policy based on its behavioral representation (step 412). This can be achieved by processing the behavioral representation with a reward estimation model to generate the estimated reward of the candidate policy, and then using the estimated reward as the estimated performance metric of the candidate policy.

[0140] In some cases, multiple candidate strategies include at least one candidate strategy that has already been evaluated, i.e., a candidate strategy with a known estimated performance metric. For example, the set of candidate strategies may include a larger number of unevaluated candidate strategies than the candidate strategies that have already been evaluated; for example, the candidate strategies may include 100, 200 or more unevaluated candidate strategies and no more than 5 evaluated candidate strategies.

[0141] In these cases, for each remaining candidate policy in the candidate policy set that has not yet been evaluated, the system can determine the estimated performance metric of the remaining candidate policies based on: (a) (i) the difference between the estimated behavioral representation of the remaining candidate policies and (ii) the behavioral representation of the evaluated candidate policies, and (b) the known performance metric of the evaluated candidate policies. For example, the larger the difference between behavioral representations (e.g., in terms of distance in the latent space), the larger the difference between the estimated performance metric for the remaining candidate policies and the known performance metric of the evaluated candidate policies. Doing so allows the system to quickly and more accurately determine the estimated performance metrics of a very large number, potentially infinitely large number, of candidate policies from a relatively small number of evaluated candidate policies.

[0142] After repeating steps 406 through 412 multiple times to generate estimated performance metrics for multiple candidate policies included in the candidate policy set, the system continues to obtain a final policy for controlling the agent in the environment based on the multiple candidate policies and the estimated performance metrics of each of the multiple candidate policies (step 414). There are many ways in which the system can obtain such a final policy based on the estimated performance metrics.

[0143] The following is for reference. Figure 5 Describe in more detail an example of how the system can obtain the final strategy.

[0144] Figure 5 It is a flowchart of sub-steps 502 to 506 of step 414 of process 400 according to an implementation method.

[0145] The system selects a subset of candidate policies based on their estimated performance metrics (step 502). For example, the system may select one or more candidate policies with the highest estimated performance metric from among all the estimated performance metrics of the candidate policies included in the candidate policy set. As another example, the system may select one or more candidate policies with estimated performance metrics that satisfy, for example, a performance metric greater than a given threshold.

[0146] The system determines the actual performance measure of each candidate policy in a subset by controlling the agent in the environment using candidate policies (step 504). That is, for each candidate policy in the subset, the system controls the agent's interaction with the environment by selecting an action to be performed by the agent according to the candidate policy at each of a plurality of time steps, for example, at each of a fixed number of time steps. The system monitors the rewards received by the agent at the plurality of time steps and then determines the actual performance measure of the candidate policy based on those rewards, for example, as the cumulative rewards received by the agent at the plurality of time steps.

[0147] The system selects a candidate policy as the final policy for controlling the agent to perform a specific task in the environment based on actual performance metrics (step 506). For example, the system can select the candidate policy with the highest actual performance metric among all candidate policies in a subset.

[0148] In this example, after a candidate policy has been selected as the final policy, the system or another agent controlling the system can then deploy the final policy and then use the final policy to control the agent interacting with the environment by selecting the action to be performed by the agent at each of the multiple time steps throughout the trajectory.

[0149] Specifically, in this example, the number of candidate policies in the proper subset is typically much smaller than the total number of candidate policies in the candidate policy set. For example, even when the candidate policy set includes 100, 500, or more candidate policies, the system can still accurately control the agent when the subset includes only 10 candidate policies, i.e., when using 10 candidate policies to generate the final policy.

[0150] This allows the system to evaluate the performance of different candidate policies with reduced latency, while consuming fewer computational resources than conventional offline policy selection methods. When candidate policies are needed to actually control a real-world agent in a real-world environment for performance evaluation, this avoids unnecessary overhead and damage to the real-world environment or the real-world agent, or both, because only a small number of candidate policies will need to be evaluated—that is, used to select the action to be performed by the agent.

[0151] See below for reference Figure 6 Here is another example of how the system can obtain the final strategy, described in more detail.

[0152] Figure 6 It is a flowchart of sub-steps 602 to 606 of step 414 of process 400 according to an implementation method.

[0153] The system receives the current observation representing the current state of the environment (step 602).

[0154] The system applies a policy improvement algorithm, such as a general policy improvement algorithm, to the multiple candidate policies based on estimated performance metrics to generate a final policy (step 604). By applying the (general) policy improvement algorithm, the system can generate a composite policy as the final policy from two or more candidate policies in the candidate policy set. The composite policy can be an action selection policy that switches between subsets of candidate policies with a given switching probability. When used, the composite policy defines at each of multiple time steps which (and at most one) candidate policy will be used to control the agent according to the given switching probability, and then selects the action to be performed by the agent according to the determined candidate policy.

[0155] For example, to generate composite policies, the system can apply any of the general policy improvement techniques described in "Successor features for transfer in reinforcement learning" by Barreto, A et al. at the 2017 Conference on Advances in Neural Information Processing Systems and "Generalised Policy Improvement with Geometric Policy Composition" by Thakoor, S et al. at the 2022 International Conference on Machine Learning, PMLR, to identify which candidate policies should be included in a subset and how to switch from one candidate policy to another in the subset.

[0156] Similar to the examples discussed above, after a composite policy has been generated as the final policy, the system or another agent controlling the system can then deploy the final policy and then use the final policy to control the agent interacting with the environment by using the final policy at each of the multiple time steps throughout the trajectory to select an action to be performed by the agent, for example, in response to the current observation received at that time step (step 606).

[0157] Because obtaining multiple successor feature neural networks corresponding to a single candidate policy when there are a large number of candidate policies can be computationally intensive and consume a considerable amount of computational resources, we will now discuss techniques for training successor feature neural networks for each candidate policy in a computationally efficient manner.

[0158] Figure 7This is a flowchart of an example process 700 for training a successor feature neural network corresponding to a candidate policy. For convenience, process 700 will be described as being executed by a system of one or more computers located in one or more locations. For example, a properly programmed system—e.g. Figure 1 The intelligent agent control system 100 or another training system can execute process 700.

[0159] Before the iteration of process 700, the system obtains an offline dataset and data from the specified encoder neural network. The offline dataset stores multiple historical trajectories obtained as a result of interaction between the agent or another agent and the environment or another instance of the environment.

[0160] The encoder neural network can be or be included in a neural network with any suitable architecture. However, in general, the system uses the same encoder neural network in multiple iterations of process 700 to train different successor feature neural networks corresponding to different candidate policies respectively (although different encoder neural networks can be used in some cases). That is, the encoder neural network is agnostic to any particular candidate policy and can be broadly applied to any candidate policy from a variety of candidate policies.

[0161] For example, an encoder neural network can be a convolutional neural network that includes one or more convolutional layers. The parameters of the encoder neural network can have randomly initialized values, or alternatively, learned parameter values.

[0162] As another example, the encoder neural network could be an image encoder neural network, which is included as part of a multimodal neural network (e.g., an image-text neural network) that has already been pre-trained on a multimodal task (e.g., a task that requires processing both text and image data to generate the task's output, such as an image caption task).

[0163] As another example, the encoder neural network can be a visual Transformer (ViT) neural network that has been pre-trained on computer vision tasks such as image classification, image embedding generation, object detection, image segmentation, etc.

[0164] As another example, the encoder neural network can be a video processing neural network that has been pre-trained on video tracking or action recognition tasks.

[0165] As another example, the encoder neural network can be a pre-trained model of a pre-trained base model, or can be included in that pre-trained model. For example, a pre-trained model can be a model that has already been trained using a self-supervised training method. A pre-trained “base” model refers to a model that has been pre-trained, for example, on (extensive) data using self-supervision at a large scale and can be adapted to a wide range of downstream tasks. For example, a pre-trained base model can typically be pre-trained to perform one or more tasks and then later adapted or extended to perform one or more tasks that are not part of the pre-training. For example, a pre-trained base can be any of the pre-trained visual base models mentioned in Rishi Bommasani et al.'s "the opportunities and risks of foundation models" arXiv preprint arXiv:2108.07258, 2021.

[0166] The system obtains historical trajectories, for example, from an offline dataset through sampling (step 702). The historical trajectory comprises multiple historical observations. Each historical observation represents a corresponding historical state of the environment. The multiple historical observations included in the historical trajectory can include a time series of historical observations. Multiple historical observations are referred to as a time series because the historical observations are arranged according to the order in which the corresponding observations were generated.

[0167] For each historical observation in the time series of historical observations starting from a given historical observation, the system uses an encoder neural network to process the encoder network input including that historical observation to generate a corresponding encoded representation of that historical observation (step 704).

[0168] The system uses a successor feature neural network to process the input of the successor feature network to generate successor feature values ​​for a given historical observation. The input of the successor feature network includes (i) the given historical observation, (ii) data that identifies the appropriate action to be performed by the agent in response to the given historical observation, which can be selected by using a candidate policy, and in some implementations, (iii) non-visual state features (step 706).

[0169] The system evaluates a loss function that measures the difference between (i) the subsequent feature values ​​of a given historical observation and (ii) the time-discounted sum of the corresponding encoded representations of the historical observation sequence starting from that given historical observation (step 708). For example, the loss function could be a squared loss function, a mean squared error loss function, etc.

[0170] Figure 8This is an example illustration 800 of training a successor feature neural network on a historical trajectory. The historical trajectory includes a time series of N historical observations, such as the historical observation at time t, the historical observation at time t+1, the historical observation at time t+2, and so on, up to the historical observation at time t+N.

[0171] For example, suppose in Figure 8 In the example where the given historical observation is the historical observation at time t, the system can use an encoder neural network. The system processes each historical observation in a time series starting from a given historical observation to generate a corresponding encoded representation of the historical observation. The system calculates the time-discounted sum of the corresponding encoded representations of the historical observation sequence starting from the historical observation at time t. (in (This is the discount factor). The system evaluates the loss function, which measures the difference between: (i) the subsequent feature values ​​of a given historical observation generated by the subsequent feature neural network and (ii) the sum discounted over time. .

[0172] The system determines one or more updates to the current values ​​of the parameters of the successor feature neural network (step 710). This can be achieved by computing the corresponding gradient of the loss function with respect to the parameters of the successor feature neural network via backpropagation through the appropriate parameters of the successor feature neural network. The system can then determine the update by applying an update rule—for example, the Adam update rule, the RMSProp update rule, or the stochastic gradient descent (SGD) update rule—to the corresponding gradient.

[0173] The system can repeatedly perform process 700 iterations on different historical trajectories sampled from offline datasets to update the parameters of the subsequent feature neural network.

[0174] For each candidate policy, the system can continue to iterate through process 700 until the termination criteria for training the subsequent feature neural network corresponding to the candidate policy are met, such as until the parameters have converged, until a threshold amount of clock time has elapsed, or until a threshold amount of iterations of process 700 has been performed.

[0175] By iteratively executing process 700 for different candidate policies, the system can obtain multiple trained successor feature neural networks, each corresponding to a different candidate policy among multiple candidate policies. However, because the system can perform iterations of process 700 using the same encoder neural network and by accessing the same offline dataset, it can avoid the costs associated with training different encoder neural networks for different candidate policies and obtaining different offline datasets. Moreover, in some implementations, the system parallelizes the training of successor feature neural networks to reduce the total training time.

[0176] After the successor feature neural networks are trained, they can be used in any suitable manner. As an example only, the system can use the trained successor feature neural networks corresponding to the candidate policies to generate behavioral representations of the candidate policies. In this example, the system can further use the behavioral representations of the candidate policies to determine whether to deploy the candidate policies for controlling the agents in the environment, or whether to use the candidate policies to generate the final policy for controlling the agents.

[0177] This specification uses the term "configured" in conjunction with system and computer program components. For configuring one or more computer systems to perform a specific operation or action, it means that software, firmware, hardware, or a combination thereof are installed on the system that causes the system to perform that operation or action during operation. For one or more computer programs configured to perform a specific operation or action, it means that the one or more programs include instructions that, when executed by a data processing device, cause that device to perform that operation or action.

[0178] Embodiments of the subject matter and functional operation described in this specification may be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in combinations thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing device or for controlling the operation of the data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination thereof. Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals (e.g., machine-generated electrical, optical, or electromagnetic signals) that are generated to encode information for transmission to a suitable receiver device for execution by the data processing device.

[0179] The term "data processing device" refers to data processing hardware and includes all kinds of devices, apparatuses, and machines for processing data, such as programmable processors, computers, or multiple processors or computers. The device may also be or further include special-purpose logic circuit systems, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0180] A computer program (which may also be referred to or described as a program, software, software application, app, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions). A computer program can be deployed to execute on a single computer or on multiple computers located at a site or distributed across multiple sites and interconnected via a data communication network.

[0181] In this specification, the term "database" is used broadly to refer to any collection of data: data that does not need to be structured in any particular way, or does not need to be structured at all, and can be stored on storage devices in one or more locations. Thus, for example, an indexed database may include multiple collections of data, each of which can be organized and accessed differently.

[0182] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in others, multiple engines may be installed and run on the same one or more computers.

[0183] The processes and logic flows described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and generating outputs. The processes and logic flows can also be executed by a dedicated logic circuit system, such as an FPGA or ASIC, or by a combination of a dedicated logic circuit system and one or more programmed computers.

[0184] A computer suitable for executing computer programs can be based on a general-purpose microprocessor or a special-purpose microprocessor or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory or both. The basic components of a computer are the central processing unit for making or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into a special-purpose logic circuit system. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from or transfer data to one or more mass storage devices or both. However, a computer does not necessarily need to have such devices. Furthermore, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0185] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD ROMs and DVD-ROMs.

[0186] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser in response to a request received from a web browser on the user's device. Additionally, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and receiving responsive messages from the user in response.

[0187] Data processing devices used to implement machine learning models may also include, for example, dedicated hardware accelerator units for handling the general and computationally intensive portions of machine learning training or production (i.e., inference, workloads).

[0188] Machine learning models can be implemented and deployed using machine learning frameworks such as TensorFlow or JAX.

[0189] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a client computer having a graphical user interface, web browser, or app that a user can interact with through an implementation of the subject matter described in this specification), or any combination of one or more such back-end components, middleware components, or front-end components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0190] A computing system may include clients and servers. Clients and servers are typically geographically separated and interact via a communication network. The client-server relationship is established by computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., HTML pages) to a user device, for example, for the purpose of displaying data to a user interacting with the device acting as a client and receiving user input from that user. Data generated at the user device, such as the result of user interaction, may be received at the server from the device.

[0191] While this specification contains numerous details of specific implementations, these details should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features that may be characteristic of particular embodiments of a particular invention. Certain features described in the context of individual embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as operating in certain combinations and even initially claimed in this way, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may involve sub-combinations or variations thereof.

[0192] Similarly, although operations are depicted in the accompanying drawings and described in a specific order in the claims, this should not be construed as requiring such operations to be performed in the specific order shown or in sequential order, or requiring all shown operations to be performed to achieve the desired result. In some contexts, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0193] Specific embodiments of this subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the figures do not necessarily require a specific order or sequence to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous.

Claims

1. A computer-implemented method comprising: obtaining a candidate policy set comprising a plurality of candidate policies for controlling an agent in an environment; obtaining an offline dataset storing a plurality of historical trajectories, wherein each historical trajectory comprises a plurality of historical observations each characterizing a respective historical state of the environment; and generating a behavioral representation for each candidate policy, comprising, for each candidate policy: for each historical observation in a subset of the plurality of historical observations, generating a successor feature value for the historical observation under the candidate policy by using a successor feature neural network corresponding to the candidate policy, wherein the successor feature neural network is configured to: receive a successor feature network input comprising the historical observation and a respective action to be performed by the agent in response to the historical observation selected by using the candidate policy, and process the successor feature network input to generate the successor feature value for the historical observation; generating a feature vector for the candidate policy based on the successor feature values that have been generated for one or more historical observations in the subset of the plurality of historical observations; and determining the behavioral representation for the candidate policy from the feature vector for the candidate policy.

2. The method of claim 1, wherein, Generating the feature vector for the candidate policy comprises computing a mean of the successor feature values that have been generated for one or more historical observations in the subset of the plurality of historical observations.

3. The method of any one of claims 1-2, further comprising: determining an estimated performance measure for each candidate policy based on the behavioral representation for each candidate policy.

4. The method of claim 3, wherein, Determining the estimated performance measure for each candidate policy comprises: processing the behavioral representation for the candidate policy using a learned model to generate an estimated return for the candidate policy as the estimated performance measure.

5. The method of any one of claims 3-4, wherein, The plurality of candidate policies comprises at least one candidate policy that has been evaluated, and wherein determining the estimated performance measure for each candidate policy comprises, for each remaining candidate policy that has not yet been evaluated: determining the estimated performance measure for the remaining candidate policy based on (a) a difference between (i) a behavioral representation for the remaining candidate policy and (ii) a behavioral representation for the candidate policy that has been evaluated and (b) a known performance measure for the candidate policy that has been evaluated.

6. The method of claim 5, wherein, The plurality of candidate policies comprises a greater number of candidate policies that have not yet been evaluated than candidate policies that have been evaluated.

7. The method of any one of claims 3-6, further comprising: obtaining a final policy to control the agent in the environment based on the estimated performance measure for each candidate policy.

8. The method of claim 7, wherein, Obtaining the final policy to control the agent in the environment comprises: selecting a subset of the plurality of candidate policies based on the estimated performance measures for the plurality of candidate policies; determining an actual performance measure for the candidate policy by controlling the agent in the environment using each candidate policy in the subset, including selecting a sequence of actions to be performed by the agent in accordance with the candidate policy; and selecting a candidate policy as the final policy based on the actual performance measures.

9. The method of claim 7, wherein, obtaining the final policy to control the agent in the environment includes: receiving a current observation characterizing a current state of the environment; generating the final policy based on applying a general-purpose policy improvement algorithm to the plurality of candidate policies; and selecting a current action to be performed by the agent in response to the current observation in accordance with the final policy.

10. The method of any one of claims 1-9, further comprising training different successor feature neural networks corresponding to different candidate policies by using an encoder neural network and by using the offline dataset.

11. The method of claim 10, wherein, training a successor feature neural network corresponding to a candidate policy includes: obtaining a training history trajectory including a sequence of historical observations from the offline dataset; processing each historical observation in the sequence of historical observations starting from a given historical observation using the encoder neural network to generate a respective encoded representation of the historical observation; processing the given historical observation in the sequence of historical observations using the successor feature neural network to generate a successor feature value for the given historical observation; evaluating a loss function that measures a difference between (i) the successor feature value for the given historical observation and (ii) a temporally discounted sum of the respective encoded representations of the sequence of historical observations starting from the given historical observation; and determining one or more updates to current values of parameters of the successor feature neural network based on computing gradients of the loss function with respect to the parameters.

12. The method of any one of claims 10-11, wherein, the encoder neural network includes one of: a convolutional neural network with randomly initialized or learned parameter values; an image encoder in an image-text neural network pre-trained on an image illustration task; a visual Transformer neural network pre-trained on an image classification task; or a neural network pre-trained on a video tracking task.

13. A computer-implemented method comprising: obtaining data specifying a pre-trained model; maintaining an offline dataset storing a plurality of history trajectories, where each history trajectory includes a plurality of historical observations each characterizing a respective historical state of an environment; and training a successor feature neural network corresponding to a candidate policy using the pre-trained model and the offline dataset, where the training includes: obtaining a training history trajectory including a sequence of historical observations from the offline dataset; for each historical observation in the sequence of historical observations starting from a given historical observation, processing a model input including the historical observation using the pre-trained model to generate a respective encoded representation of the historical observation; processing a successor feature network input using the successor feature neural network to generate a successor feature value for the given historical observation, the successor feature network input including (i) the given historical observation in the sequence of historical observations, and (ii) an action selected using the candidate policy in response to the given historical observation; evaluating a loss function that measures a difference between (i) the successor feature value for the given historical observation and (ii) a temporally-discounted sum of the respective encoded representations of the sequence of historical observations starting from the given historical observation; and determining one or more updates to current values of parameters of the successor feature neural network based on computing gradients of the loss function with respect to the parameters of the successor feature neural network.

14. The method of claim 13, further comprising generating a behavioral representation of the candidate policy using the trained successor feature neural network.

15. The method of claim 14, further comprising using the behavioral representation of the candidate policy to determine whether to deploy the candidate policy for controlling an agent in the environment, or whether to use the candidate policy to generate a final policy for controlling the agent.

16. The method of any one of claims 13 to 15, wherein, the pre-trained model comprises a pre-trained visual model.

17. The method of any one of claims 1 to 16, wherein, the agent is a mechanical agent and the environment is a real-world environment.

18. The method of claim 17, wherein, the agent is a robot.

19. The method of any one of claims 1 to 16, wherein, the environment is a real-world environment of a service facility that includes a plurality of electronic equipment, and the agent is an electronic agent configured to control operation of the service facility.

20. The method of any one of claims 1 to 16, wherein, the environment is a real-world manufacturing environment for manufacturing a product, and the agent includes an electronic agent configured to control a manufacturing cell or a machine operating to manufacture the product.

21. The method of any one of claims 1 to 16, wherein, the environment is a simulation of a real-world environment, and wherein the method further comprises: after generating the final policy, providing data specifying the final policy for use in controlling a real-world agent in the real-world environment.

22. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the operations of the respective method of any one of claims 1-21.

23. A computer storage medium encoding instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the respective method of any one of claims 1-21.

Images