Large-scale hierarchical reinforcement learning

Abstract

The present invention describes a system and method for controlling an agent that interacts with an environment to perform a task. The method includes, for each of a plurality of first time steps from a plurality of time steps, receiving an observation characterizing the state of the environment at the first time step; determining a target representation for the first time step characterizing a target state of the environment that the agent should reach; using a low-level controller neural network to process the observation and the target representation to generate a low-level policy output defining an action to be performed by the agent in response to the observation, the low-level controller neural network comprising a representation neural network configured to process the observation to generate an internal state representation of the observation, and a low-level policy head configured to process the observation state representation and the target representation to generate the low-level policy output; and controlling the agent using the low-level policy output.

Description

[Technical Field]

[0001] Cross-reference of related applications This application claims the benefits of U.S. Provisional Application No. 63 / 349,968, filed on 7 June 2023. The disclosures of that prior application are deemed to be part of the disclosures of this application and are incorporated by reference in the disclosures of this application.

[0002] This specification relates to processing data using machine learning models. [Background technology]

[0003] Machine learning models receive input and, based on the received input, generate an output, such as a predicted output. Some machine learning models are parametric models, which generate an output based on the received input and the values ​​of the model's parameters.

[0004] Some machine learning models are deep models that utilize multiple layers of models to generate an output for a given input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that apply nonlinear transformations to the received inputs, each to generate an output. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Espeholt et al., arXiv:1802.01561 [Overview of the project] [Problems that the invention aims to solve]

[0006] This specification generally describes a system implemented as a computer program on one or more computers located in one or more locations, which controls an agent that interacts with an environment and performs tasks in that environment.

[0007] At each time step, an agent receives input observations and executes one of the actions from a set of actions. For example, the set of actions may include a fixed number of actions or may be a continuous action space.

[0008] Generally, a system controls an agent using a hierarchical controller. The hierarchical controller includes a low-level controller neural network and a high-level controller neural network.

[0009] The hierarchical controller generates options that can be used to control the agent. Generally, an option is a generalization of an action. For example, an option may include an action (as specified above), a high-level action that affects the selection of actions by the hierarchical controller, a target output (a "target option") that characterizes a target state of the environment that the agent should reach, and any other generalizable behavior.

[0010] This specification also describes target conditioning techniques for training the low-level controller neural network and the high-level controller neural network so that the hierarchical controller can be used to effectively control an agent using a target representation generated from a target output produced by the high-level controller neural network.

[0011] Particular embodiments of the subject matter described in this specification may be implemented to realize one or more of the following advantages.

[0012] Unlike existing hierarchical agents, the hierarchical systems described in this specification can exhibit performance comparable to or exceeding that of other flat (non-hierarchical) techniques for difficult real-world tasks, such as visually complex partially observable 3D environments.

[0013] In addition, the described system can train hierarchical controllers to learn target-conditioned behavior from any experience generated through the training processes of low-level controllers, high-level controllers, or both. This represents a significant advance in the field of goal-conditioned reinforcement learning, which has previously relied on expert-generated data in the environment to train agents on target representations.

[0014] This specification also describes techniques for training low-level controllers on multiple tasks simultaneously, which enhances the ability of hierarchical controllers to generalize across multiple tasks. This provides a concrete example of the latent abilities of abstraction, transfer, and skill reuse that characterize hierarchical reinforcement learning in challenging environments.

[0015] Furthermore, this specification describes techniques that enable hierarchical controllers to be used "on a large scale," that is, to be trained and used effectively for large-scale industrial tasks, and in some cases, to be generalized to new industrial tasks that were not encountered during training for low-level controllers, high-level controllers, or both.

[0016] To effectively control agents using hierarchical controllers, this specification describes various techniques that can be used together or separately to improve agent performance in such complex environments. [Means for solving the problem]

[0017] In one exemplary method described herein, a method for controlling an agent that interacts with an environment to perform a task comprises: receiving observations characterizing the state of the environment at each of a plurality of first time steps from a plurality of time steps; determining a target representation for the first time step characterizing the target state of the environment to which the agent should reach; and processing the observations and target representation using a low-level controller neural network to generate a low-level policy output that defines the action to be performed by the agent in response to the observations. The low-level controller neural network comprises a representation neural network configured to process the observations to generate an internal state representation of the observations, and a low-level policy head configured to process the observation state representation and the target representation to generate a low-level policy output. The method comprises controlling the agent using the low-level policy output.

[0018] The step of determining a target representation for a first time step that characterizes a target state of the environment to which the agent should reach may include a step of determining whether a criterion for generating a new target representation is met in the first time step. If the criterion is not met, the method may use a target representation from a preceding first time step as the target representation for the first time step. In one exemplary implementation, if the criterion is met, the method may include the steps of generating high-level observations for the first time step, processing a high-level input containing the high-level observations using a high-level controller neural network to generate a high-level policy output containing a target output characterizing the target state, and processing the target output using a target encoder neural network to generate a target representation. The high-level policy output may further include an indication of whether (i) the agent is controlled using a low-level controller, or (ii) the agent is controlled using a basic action identified by the high-level policy output. In one exemplary implementation, the steps of determining a target representation for a first time step, processing the observations and target representation using a low-level controller neural network, and controlling the agent using a low-level policy output may be performed only in response to a decision, based on an indication, to control the agent using a low-level controller. The method may further comprise the steps of: generating high-level observations for a second time step in each of one or more second time steps from a plurality of time steps; processing the high-level observations for the second time step using a high-level controller neural network to generate a high-level policy output for the second time step; deciding to control the agent using a basic action identified by a high-level policy output, based on an indication of (i) controlling the agent using a low-level controller, or (ii) controlling the agent using a basic action identified by a high-level policy output; and, in response, controlling the agent using a basic action identified by a high-level policy output.

[0019] High-level observations for the first time step may include observations received in the first time step. High-level observations for the first time step may include data identifying the number of time steps since the most recent time step in which the criterion was met. High-level observations for the first time step may include data characterizing the rewards received from the environment since the most recent time step in which the criterion was met. High-level observations for the first time step may include data characterizing observations received in time steps since the most recent time step in which the criterion was met. A criterion may be met if any of the criteria in the set of criteria are met. High-level observations for a time step may include data identifying which criteria were met in the first time step.

[0020] A target output characterizing a target state may be a target vector characterizing the target state. A target output characterizing a target state may be a text string describing the target state. A high-level policy output may have hyperparameters defining some aspect of training. A step of controlling an agent using a low-level policy output may include applying hyperparameters to the low-level policy output to generate a tuned policy output, and using the tuned policy output to select an action. A high-level policy may tune the low-level policy output using a temperature parameter as a hyperparameter.

[0021] The step of determining whether the criteria for generating a new target expression are met in the first time step may include the step of determining that the criteria for generating a new target expression are met when the first time step is the first time step in the task episode. The step of determining whether the criteria for generating a new target expression are met in the first time step may include the step of determining that the maximum number of time steps have elapsed since the most recent time step in which the criteria were met.

[0022] A low-level controller neural network may further comprise a value head configured to process observational representations and target representations to generate a value estimate of the value of an environment in a state characterized by an observation that the target state characterized by the target representation has been reached. The step of determining whether a criterion for generating a new target representation is met in a first time step may comprise the step of determining that the criterion for generating a new target representation is met if the value estimate generated by the value head by processing observational representations of observations in a preceding time step and the target representation for the preceding time step is less than an unreachability threshold.

[0023] The step of determining whether the criteria for generating a new target representation are met in the first time step may include determining that the criteria for generating a new target representation are met when the value estimate and the target representation for the preceding time step, generated by the value head by processing the observational representation of the observations in the preceding time step, are greater than an achieved threshold indicating that the goal characterized by the target representation for the preceding time step has been achieved. The achieved threshold may be included in the high-level policy output.

[0024] The step of determining whether the criteria for generating a new target representation are met in a first time step may include: using a classifier neural network to process a classifier input that characterizes (i) observations in a preceding time step and (ii) a target state for the preceding time step to generate a classifier output indicating how many time steps remain until the target state is achieved; and determining that the criteria for generating a new target representation are met when the classifier output indicates that zero time steps remain until the target state is achieved.

[0025] In another exemplary method described herein, a method for training a low-level controller neural network comprises the steps of: acquiring one or more trajectories of observation-action pairs and a respective goal for each trajectory; for each trajectory, generating a respective reward for each pair in the trajectory based on whether the goal was achieved in a state characterized by the observation in the pair; and training a low-level controller neural network for one or more trajectories and one or more rewards to optimize a reinforcement learning objective, wherein the reinforcement learning objective comprises (i) a behavior imitation loss and (ii) a proximity regularization term that penalizes a low-level policy output generated by a low-level controller neural network for deviation from a behavior imitation policy output generated by a behavior imitation head, which is configured to process a given observation representation and a given goal representation for a given observation generated by a representation neural network to generate a behavior imitation policy output for an observation.

[0026] The behavior imitation policy output and the low-level policy output can each define a probability distribution across multiple actions. The proximity regularization term may be based on the divergence between the low-level policy output generated by the low-level policy head by (i) processing the observation representation of the observation generated by the representation neural network and the target representation of the target for the trajectory, and (ii) the behavior imitation policy output generated by the behavior imitation policy head by processing the observation representation of the observation generated by the representation neural network and the target representation of the target for the trajectory, for each observation in each trajectory. For example, this divergence could be a KL divergence. The reinforcement learning objective may further include, for each observation-action pair in each trajectory, a term that is the product of (i) a term based on the reward for the pair and (ii) the ratio of the probability assigned to the action in the pair by the low-level policy output generated by the low-level policy head by processing (a) the observation representation of the observation and the target representation of the target for the trajectory generated by the representation neural network, and (b) the probability assigned to the action in the pair by the behavior imitation policy output generated by the behavior imitation policy head by processing the observation representation of the observation and the target representation of the target for the trajectory generated by the representation neural network. For example, the term based on the reward for the pair may be the V-Trace policy gradient term.

[0027] The method may further include a step of training a behavior imitation policy head for one or more trajectories to minimize the behavior imitation loss.

[0028] For one or more of the trajectories, the target may be an image of the environment in the target state. The training step may further comprise processing the image using an image target encoder neural network to generate a target representation to be processed by a low-level policy head and a behavior-imitating policy head. For one or more of the trajectories, the target may be text describing the target state of the environment. The training step may further comprise processing the text using a text target encoder neural network to generate a target representation to be processed by a low-level policy head and a behavior-imitating policy head.

[0029] One or more trajectories may be sampled from offline data generated while the agent was controlled by one or more different behavioral policies. Low-level controllers may be trained on offline data only.

[0030] Alternatively, the low-level controller can be trained on both offline data and online data generated by controlling agents using both high-level policy outputs generated by the high-level controller and low-level policy outputs generated by the low-level controller.

[0031] The method may further comprise the step of training the high-level controller neural network through reinforcement learning to maximize the expected reward for a task resulting from controlling an agent based on the high-level policy output generated by the high-level controller neural network and the low-level policy output generated by the low-level controller neural network. In some exemplary implementations, no gradients are passed to the low-level controller neural network as a result of training the high-level controller neural network.

[0032] One or more trajectories may include one or more online trajectories generated while controlling the agent during the training of a high-level or low-level controller. In an alternative implementation, one or more trajectories may not include any online data generated while controlling the agent during the training of a high-level controller.

[0033] The reinforcement learning objective may include one or more auxiliary loss terms. Each auxiliary loss term may correspond to a separate auxiliary task. Each auxiliary task may require generating predictions that characterize a target distribution conditioned on the output produced by the representational neural network.

[0034] The method may further comprise the step of generating a goal for each of one or more trajectories of observation-action pairs. The generation step may comprise the step of selecting a goal for one or more of the trajectories that describes the state of the environment characterized by the last observation in the trajectory.

[0035] The method may further comprise the step of generating a goal for each of one or more trajectories of observation-action pairs. The generation step may comprise the step of selecting a goal for one or more of the trajectories that describes an environmental state not characterized by any of the observations in the trajectory.

[0036] The agent may be a mechanical agent, and the environment may be a real-world environment. For example, the agent may be a robot. The environment may be a real-world environment of a service facility with multiple electronic devices, and the agent may be an electronic agent configured to control the operation of the service facility. The environment may be a real-world manufacturing environment for manufacturing a product, and the agent may include an electronic agent configured to control a manufacturing unit or machine that operates to manufacture the product.

[0037] Details of one or more embodiments of the subject matter described herein are shown in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawing]

[0038] [Figure 1] This is a diagram illustrating an exemplary behavioral choice system. [Figure 2A] This is an illustrative process flowchart for generating options for a hierarchical controller. [Figure 2B] This diagram shows in more detail how high-level and low-level controllers interact. [Figure 3] This diagram provides a more detailed illustration of an exemplary high-level controller and its components. [Figure 4] This diagram provides a more detailed illustration of an exemplary low-level controller and its components. [Figure 5] This figure illustrates an exemplary offline-online training system for a hierarchical controller. [Figure 6] This is a block diagram illustrating an exemplary process for hindsight selection of target representations in an offline-online training system. [Figure 7] This figure shows the benchmark performance of the described technique compared to conventional techniques. [Figure 8] This diagram illustrates the importance of having a proximity regularization penalty. [Modes for carrying out the invention]

[0039] Similar reference numbers and designations in various drawings refer to the same elements.

[0040] Figure 1 shows an exemplary behavioral selection system 100. The behavioral selection system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations where the systems, components, and techniques described below are implemented.

[0041] The action selection system 100 controls the agent 104 as it interacts with the environment 106 and performs the task by selecting an action 108 to be performed by the agent 104 at each of several time steps during the execution of the task episode.

[0042] As a general example, a task may include one or more of the following: moving to a specified location in environment 106, identifying a specific object in environment 106, manipulating a specific object in a specified manner, controlling multiple devices to meet certain criteria, or distributing resources to multiple devices. More generally, a task is defined by the reward 130 received, i.e., the episodic return is maximized when the task is successfully completed. Rewards and returns are described in more detail below. Examples of agents, tasks, and environments are also given below.

[0043] A task “episode” is a series of interactions in which agent 104 attempts to perform a single task, starting from some initial state of environment 106. In other words, each task episode begins with environment 106 in an initial state, such as a fixed or randomly selected initial state, and ends when agent 104 successfully completes the task, or when some termination criterion is met, such as environment 106 entering a state designated as a termination state, or when agent 104 performs a threshold number of actions 108 without successfully completing the task.

[0044] At each time step in any given task episode, the system 100 receives observations 110 that characterize the current state of the environment 106 at that time step, and in response, selects an action 108 that should be performed by agent 104 at that time step. After agent 104 performs action 108, the environment 106 transitions to a new state, and the system 100 receives a reward 130 from the environment 106.

[0045] Generally, the reward 130 is a scalar value that characterizes agent 104's progress toward completing the task.

[0046] As a specific example, the reward 130 could be a sparse binary reward, meaning it is 0 unless the action successfully completes the task; that is, it is not 0, for example, equal to 1, only if the action successfully completes the task.

[0047] As another specific example, reward 130 could be a dense reward that measures the agent's progress toward completing the task at each observation point received during an episode in which the agent attempts to perform the task; that is, a non-zero reward may be received, and frequently, before the agent successfully completes the task.

[0048] While executing any given task episode, the system 100 selects an action 108 in an attempt to maximize the return received during the task episode.

[0049] In other words, at each time step between episodes, the system 100 selects an action 108 that attempts to maximize the return received for the remainder of the task episode starting from that time step.

[0050] In general, the return received at any given time step is a combination of the rewards received at time steps following that time step in the episode.

[0051] For example, at a certain time step t, the return is Σ i γ i-t-1 r i The following conditions can be met: i is over all time steps after t in the episode, or over any fixed number of time steps after t in the episode, γ is a discount factor greater than 0 and less than or equal to 1, and r i is the reward at time step i.

[0052] To control agent 108, at each time step in the episode, the action selection subsystem 102 of system 100 processes the observations 110 using the hierarchical controller 130 and selects an action 108 to be performed by agent 104 at that time step.

[0053] The hierarchical controller 130 includes a high-level controller (HLC) 126 and a low-level controller (LLC) 122.

[0054] LLC122 is a neural network configured to receive an input that includes an observation 110 and a target output that characterizes the target state of environment 106 that agent 104 should reach when interacting with environment 106, the input being processed into a target representation ("target") 124. LLC processes the input to generate a low-level output that defines the action 108 that agent 104 should perform in response to the observation 110, which is conditioned by the target representation 124.

[0055] HLC126 is a neural network configured to process high-level inputs and generate high-level policy outputs, including target option outputs.

[0056] In some implementations, the high-level policy output includes only the target output.

[0057] In some other implementations, the high-level policy output includes additional data.

[0058] In a specific example, high-level policy output can also identify basic behaviors.

[0059] Basic actions may be used to advance the system among multiple objectives 124 or to generate training data for low-level controllers. In this case, the high-level policy output includes an indication of whether (i) the agent is controlled using the low-level controller or (ii) the agent is controlled using the basic actions identified by the high-level policy output.

[0060] As another specific example, high-level policy outputs can also identify high-level actions that change how the policy outputs are processed to select actions. Specifically, this may be done to aid in agent research during training.

[0061] The components of HLC neural network 126 are shown in Figure 3, and the components of LLC neural network 122 are shown in Figure 4.

[0062] To select an action 108 at a given time step between episodes using the controller 130, the system determines whether criteria for generating a new option are met at that time step. The determination of whether the criteria are met is described in more detail below in Figure 2A.

[0063] In some cases where the criteria are not met, the system may use a target expression 124 from a preceding time step in the episode as the target 124 for the time step.

[0064] The system then uses the LLC neural network 122 to select an action 108 that is conditioned on the target representation 124.

[0065] If the criteria are met, the system uses the HLC neural network 126 to generate a high-level policy output.

[0066] In an implementation where the high-level policy output includes only the target output, the system uses the target output to generate a new target representation 124 and uses the new target representation as the target representation for the time step. The system then uses the LLC neural network 122 to select an action 108 that is conditioned on the target representation 124.

[0067] In an implementation where the high-level policy output indicates whether (i) the agent is controlled using LLC122 or (ii) the agent is controlled using the basic action identified by the high-level policy output, the hierarchical controller 130 can use either the policy output of HLC126 or LLC122 to select the action 108 that should be performed by agent 104 at a given time step based on the indication.

[0068] If the controller 130 decides to control the agent using LLC 122, the controller 130 generates a new target representation 124 using the target output and uses the new target representation as the target representation for the time step. The system then uses LLC neural network 122 to select an action 108 that is conditioned on the target representation 124.

[0069] If the controller 130 decides to control the agent using the HLC 126, the controller 130 selects a basic action from the high-level output as action 108 for the time step.

[0070] In some cases, the HLC126 can output a sequence of basic actions to advance the system before selecting a new target output.

[0071] In a particular example, at any given step in an episode, the action selection subsystem 102 may use the HLC 126 to process the observation result 110 and generate a target output to be sent to the LLC 122 to be processed into a target expression 124. The time step in which the HLC 126 generates the target output used to generate the target expression 124, and the time step in which the LLC 122 acts using the target expression 124, are referred to herein as the “first time step”.

[0072] In this example, LLC122 can then process the observations 110 to generate a policy output that is conditioned on the target expression 124. System 100 can then use the policy output to select an action 108.

[0073] Furthermore, at another time step within the episode, the low-level controller 110 can take in the observation results 110 received after the first action 108 and send them to the high-level controller 126 in the form of a high-level input.

[0074] In some examples, this high-level input may include high-level observations summarizing agent 104's behavior 108 in environment 106 since the last time HLC 126 provided the target output used to generate the target representation 124. Criteria for generating a new target representation are discussed in Figure 2A.

[0075] The high-level controller 126 can process this high-level input and generate a high-level policy output that characterizes whether to control agent 104 using LLC 122, which has a target output that is processed into a target representation 124 in successive time steps.

[0076] At some time step, the action selection subsystem 102 can use the HLC 126 to process high-level inputs and generate basic action options that use high-level policy outputs to control agent 104 in environment 106. That is, a “basic action” is one of the actions from a set or space of actions that can be performed by the agent, and is called “basic” because it is generated as part of or defined by a high-level policy output. The time step at which the HLC generates basic action options is referred to herein as the “second time step”.

[0077] As another example, at some time step, the action selection subsystem 102 can use the high-level controller 126 to generate high-level action options that modify how actions are selected from the policy output. An example of a high-level action option is discussed in more detail in Figure 3.

[0078] Therefore, the agent runs under options provided by the hierarchical controller, namely, options that specify whether to directly select an action, change how an action is selected, or provide a target output that the lower-level controller 122 processes into a target representation 124 in order to control agent 104.

[0079] Possible criteria for generating options using a hierarchical controller are discussed in more detail in Figure 2A.

[0080] Controllers 122 and 126 may use either the same or different methods to process the policy output into action 108. As discussed below in the context of possible processing methods, “policy output” can refer to either the low-level controller policy output or the portion of the high-level output that, when used, can identify the underlying action.

[0081] In one example, the policy output may include a numerical probability value for each action in a fixed set. System 102 can select action 108, for example, by sampling actions according to their probability values ​​for an action index, or by selecting the action with the highest probability value.

[0082] In another example, the policy output may include the respective Q-values ​​for each action in a fixed set. System 102 can process the Q-values ​​(for example, using a soft-max function) to generate the respective probability values ​​for each action that can be used to select action 108 (as described earlier), or it can select the action with the highest Q-value.

[0083] The Q-value for an action is an estimate of the return, resulting from agent 104 performing action 108 in response to the current observation 110, and then selecting future actions 108 to be performed by agent 104 according to the current values ​​of the parameters of controllers 122 and 126.

[0084] As another example, when the action space is continuous, the policy output may include parameters of a probability distribution over the continuous action space, and system 102 can select action 108 by sampling from the probability distribution or by selecting the mean action. A continuous action space is an action space containing infinitely many actions, i.e., each action is represented as a vector having one or more dimensions, for each dimension the action vector can take any value within the range of the dimension, and the only constraint is the precision of the numerical format used by system 100.

[0085] As yet another example, when the action space is continuous, the policy output may include a regressed action, i.e., a regression vector representing an action from the continuous space, and system 102 can select the regressed action as action 108.

[0086] Before using controllers 122 and 126 to control the agents, a training system 190 within system 100 or another training system can train controllers 122 and 126.

[0087] Specifically, the training system 190 can train the HLC 126 to select target options that instruct the LLC 122 to interact with the environment 106 in order to effectively perform a task, and can train the LLC 122 to effectively select actions under a given target representation.

[0088] Training the hierarchical controller 130 using the training system 190 will be discussed in more detail below with reference to Figure 2A.

[0089] In some implementations, the environment is a real-world environment, the agent is a mechanical agent that interacts with the real-world environment, such as a robot that operates or moves through the environment, or an autonomous or semi-autonomous land, air, or sea vehicle, and the action is an action taken by the mechanical agent in the real-world environment to perform a task. For example, the agent could be a robot that interacts with the environment to perform a specific task, such as finding an object of interest in the environment, moving an object of interest to a specified location in the environment, or moving to a specified destination in the environment.

[0090] In these implementations, observations may include, for example, images, object position data, and one or more sensor data for capturing observations as the agent interacts with the environment, such as sensor data from image sensors, distance sensors, or position sensors, or from actuators. For example, in the case of a robot, observations may include data characterizing the robot's current state, such as one or more of joint position, joint velocity, joint force, torque, or acceleration, such as gravity-compensated torque feedback, and the global or relative orientation of an object held by the robot. In the case of a robot or other mechanical agent or vehicle, observations may similarly include one or more of position, linear or angular velocity, force, torque, or acceleration, and the global or relative orientation of one or more parts of the agent. Observations may be defined in one, two, or three dimensions, and may be absolute and / or relative observations. The observations may also include, for example, detected electronic signals such as motor current or temperature signals, and / or image or video data from, for example, a camera or LiDAR sensor, for example, data from the agent's sensors or data from sensors located separately from the agent in the environment.

[0091] In these implementations, the actions may be control signals for controlling a robot or other mechanical agent, such as torque or high-level control commands for a robot's joints, or control signals for controlling an autonomous or semi-autonomous land, air, or sea vehicle, such as torque or high-level control commands for a control surface or other control element, such as steering control elements of the vehicle. The control signals may include, for example, position, velocity, or force / torque / acceleration data for one or more joints of a robot or part of another mechanical agent. The control signals may further or instead include electronic control data, such as motor control data, or more generally, data for controlling one or more electronic devices in the environment whose control has an effect on the observed state of the environment. For example, in the case of an autonomous or semi-autonomous land, air, or sea vehicle, the control signals may define actions to control navigation, such as steering, and actions to control movement, such as braking and / or acceleration of the vehicle.

[0092] In some implementations, the environment is a simulation of a real-world environment as described above, and the agent is implemented as one or more computers interacting with the simulated environment. For example, the simulated environment may be a simulation of a robot or a vehicle, and the reinforcement learning system may be trained on the simulation and, once trained, used in the real world.

[0093] In some implementations, the environment is a real-world manufacturing environment for producing products such as chemical, biological, or mechanical products, or food. In this specification, “production” of a product includes refining raw materials to produce a product, or processing raw materials to produce a clean or recycled product, for example, by removing contaminants. A manufacturing plant may comprise multiple manufacturing units, such as containers for chemical or biological substances, or machines for processing solid or other materials, such as robots. The manufacturing units are configured such that intermediate versions or components of a product can be moved between manufacturing units during product production, for example, via piping or mechanical transport. In this specification, product production also includes food production by kitchen robots.

[0094] The agent may include an electronic agent configured to control a manufacturing unit, or a machine such as a robot that operates to manufacture a product. In other words, the agent may include a control system configured to control the manufacture of a chemical, biological, or mechanical product. For example, the control system may be configured to control one or more of the manufacturing units or machines, or to control the movement of intermediate versions or components of a product between the manufacturing units or machines.

[0095] For example, tasks performed by an agent may include tasks for manufacturing a product or an intermediate version or component thereof. Another example is tasks for controlling, for example, the use of resources, such as controlling the consumption of electricity, or water, or any materials or consumables used in the manufacturing process, or minimizing them.

[0096] Actions may include control actions for controlling the use of a machine or manufacturing unit for processing solid or liquid materials in order to manufacture a product or an intermediate version or component thereof, or to control the movement of an intermediate version or component of a product within a manufacturing environment, for example, between manufacturing units or machines. Generally, actions may be any action that affects the observed state of the environment, for example, an action configured to adjust any of the detected parameters described below. These may include actions to adjust the physical or mechanical conditions of a manufacturing unit, or actions to control the movement of mechanical parts of a machine or joints of a robot. Actions may include actions that impose operating conditions on a manufacturing unit or machine, or actions that bring about changes in settings to adjust, control, or turn on or off the operation of a manufacturing unit or machine.

[0097] The reward or return may relate to a measure of the performance of the task. For example, in the case of a task to manufacture a product, the measure may be a measure of the quantity of the product manufactured, the quality of the product, the speed of the product's manufacture, or the physical cost of performing the manufacturing task, such as the amount of energy, materials, or other resources used to perform the task. In the case of a task to control resource use, the measure may be any measure of resource usage.

[0098] In another example, the agent may be a human or animal agent, and controlling the agent may involve outputting commands or signals configured to cause the agent to perform an action. For example, the commands or signals may be output to the agent using an output device such as a display device, speaker, or haptic device. In this case, the task may be a task in a real-world environment, which may be any real-world environment, including the examples described above. In some implementations, the user may be the user of a digital assistant, such as a smart speaker, smart display, or other device, and the digital assistant may be used to command the user to perform an action. For example, controlling the agent may involve commanding the digital assistant to issue commands to the human user via the digital assistant for actions that the user should perform at each of several time steps. The commands may be generated in natural language form, transmitted, for example, as sound and / or text on a screen, based on actions selected by a reinforcement learning system. Actions may be selected to contribute to the performance of the task. An observation capture subsystem, such as a monitoring system, such as a video camera or audio capture system, may be provided to capture visual and / or auditory observations of the user performing the task. This can be used to monitor any actions the user actually takes at each time step, and to provide observations that characterize the state of the environment.

[0099] In some of the implementations described above, the environment may include people or animals. For example, the agent may be an autonomous vehicle in an environment where people, such as pedestrians or drivers / passengers of other vehicles and / or animals, are located, or the autonomous vehicle itself may be carrying people. As another example, the environment may include at least one room in, for example, a dwelling, in which one or more people are located. People or animals may be elements of the environment involved in the task.

[0100] In a further example, the environment may include a user of a person interacting with an agent in the form of a user device, such as a computer, mobile device, or digital assistant. The user device provides a user interface between the user and a computer system, which may be the same computer system or a different computer system that implements the controller. The user interface may also allow the user to input data into the computer system and / or receive data from the computer system, and the agent may be controlled by the controller to perform information transfer tasks related to the user, such as providing the user with information about a topic and / or allowing the user to specify the components of a task that the computer system should perform. For example, an information transfer task might be teaching the user skills such as how to speak a certain language or how to navigate around a certain geographical location. In another example, a task might be allowing the user to define a shape to a computer system so that the computer system can control an additional manufacturing (3D printing) system to produce an object with a certain three-dimensional shape. Actions may include providing information to the user and / or configuring the interface to receive input from the user, for example, in a certain format, at a certain rate, etc. For example, an action may involve setting a problem for the user to perform regarding a skill, such as asking the user to choose from several options regarding the correct usage of a language, or asking the user to read a passage of language aloud, and / or receiving input from the user, such as registering one of the options, or using a microphone to record a spoken passage of language.

[0101] In general, observations of the state of the environment may include any electronic signals that represent the function of electronic and / or mechanical equipment. For example, a representation of the state of the environment may be derived from observations made by sensors that detect the state of the manufacturing environment, such as sensors that detect the state or configuration of a manufacturing unit or machine, or sensors that detect the movement of raw materials between manufacturing units or machines. In some examples, such sensors may be configured to detect mechanical movement or force, electrical conditions such as pressure, temperature, current, voltage, frequency, impedance, the quality, level, flow rate / movement speed or flow path / movement path of one or more raw materials, physical or chemical conditions such as physical state, shape, or configuration, or chemical state such as pH, the configuration of a unit or machine such as the mechanical configuration of a unit or machine, or the configuration of a valve, and may be image or video sensors for capturing image or video observations of a manufacturing unit or machine or movement, or any other suitable type of sensor. In the case of machines such as robots, observations from sensors may include data characterizing the current state of the machine or robot, or an article held or processed by the machine or robot, such as the position, linear or angular velocity, force, torque or acceleration, or posture of one or more parts of the machine. The observations may also include detected signals, such as motor current or temperature signals, or images or video data from, for example, a camera or LiDAR sensor. These sensors may be part of the agent in the environment or placed separately.

[0102] In some implementations, the environment is a real-world environment of a service facility comprising multiple electronic devices, such as a server farm or data center, e.g., a telecommunications data center, or a computer data center for storing or processing data, or any service facility. The service facility may also include auxiliary control devices that control the operating environment of its equipment, e.g., environmental control devices such as temperature control, e.g., cooling equipment, or airflow control or HVAC equipment. Tasks may include tasks that control resource use, e.g., minimizing it, such as tasks for controlling power consumption or water consumption. Agents may include electronic agents configured to control the operation of the equipment or to control the operation of auxiliary control devices, e.g., environmental control devices.

[0103] In general, an action can be any action that affects the observed state of the environment, for example, an action configured to adjust any of the detected parameters described below. These may include actions to control the device or auxiliary control device, or to impose operating conditions on them, for example, actions that result in a change of settings to adjust, control, or turn on or off the operation of a device or auxiliary control device.

[0104] In general, observations of the state of the environment may include any electronic signals that represent the function of a facility or equipment within a facility. For example, a representation of the state of the environment may be derived from observations made by any sensors that detect the state of the physical environment of the facility, or from observations made by any sensors that detect the state of one or more pieces of equipment or one or more auxiliary control devices. These include sensors configured to detect electrical conditions such as current, voltage, power, or energy; physical facility configurations such as the temperature, flow rate, temperature or pressure within the facility or in the facility's cooling system, or whether a vent is open.

[0105] The reward or return may relate to a measure of task performance. For example, in the case of a task to control resource use, such as minimizing it, such as a task to control the use of electricity or water, the measure may be any measure of resource use.

[0106] In some implementations, the environment is the real-world environment of a power generation facility, such as a renewable power generation facility like a solar or wind power plant. Tasks may include control tasks to control the power generated by the facility, such as controlling the delivery of power to the distribution system, such as meeting demand, or reducing the risk of mismatches between system elements, or maximizing the power generated by the facility. Agents may include electronic agents configured to control the power generation by the facility or the coupling of the generated power to the system. Actions may include actions to control the electrical or mechanical configuration of a generator, such as the electrical or mechanical configuration of one or more renewable power generation elements, such as the configuration of a wind turbine or one or more solar panels or mirrors, or the electrical or mechanical configuration of a rotating generator. Mechanical control actions may include actions to control the conversion of energy input to electrical energy output, such as the efficiency or degree of coupling of the conversion of energy input to electrical energy output. Electrical control actions may include actions to control one or more of the voltage, current, frequency, or phase of the generated power.

[0107] The reward or return may relate to a measure of the task's performance. For example, in a task to control the delivery of power to a distribution system, the measure may relate to a measure of the power being delivered, or a measure of electrical mismatches between the power generation facility and the system, such as voltage, current, frequency, or phase mismatches, or a measure of power or energy loss at the power generation facility. In a task to maximize the delivery of power to a distribution system, the measure may relate to a measure of the power or energy being delivered to the system, or a measure of power or energy loss at the power generation facility.

[0108] In general, observations of environmental conditions may include any electronic signals that represent the electrical or mechanical function of a generator in a power generation facility. For example, a representation of environmental conditions may be derived from observations made by any sensors that detect the physical or electrical state of equipment within a power generation facility that is generating power, or from the physical environment of such equipment, or from the state of auxiliary equipment that supports the generator. Such sensors may include sensors configured to detect the electrical state of equipment, such as current, voltage, power, or energy; the temperature or cooling of the physical environment, flow rate, or the physical configuration of equipment; and observations of the electrical state of the system, for example, from local or remote sensors. Observations of environmental conditions may also include one or more predictions regarding the operating conditions of the generator in the future, such as predictions of future wind levels or solar irradiation, or predictions of the future electrical state of the system.

[0109] In another example, the environment may be a chemosynthesis or protein folding environment where each state is the respective state of a protein chain or one or more intermediates or precursor chemicals, and the agent is a computer system for determining how to fold the protein chain or synthesize the chemicals. In this example, the actions are possible folding actions for folding a protein chain or actions for assembling precursor chemicals / intermediates, and the results to be achieved may include, for example, folding the protein so that it stably achieves a particular biological function, or providing an effective synthesis pathway for a chemical. In another example, the agent may be a mechanical agent that automatically performs or controls protein folding actions or chemosynthesis steps selected by the system without human interaction. Observations may consist of direct or indirect observations of the state of the protein or chemicals / intermediates / precursors, and / or may be derived from simulations.

[0110] Similarly, the environment may be a drug discovery environment where each state is a possible state of a compound that may be drug-active, and the agent is a computer system for determining the elements of a drug-active compound and / or the synthesis route of a drug-active compound. The drug / synthesis may be designed based on a reward derived from the drug's target, for example, in a simulation. In another example, the agent may be a mechanical agent that performs or controls the synthesis of a drug.

[0111] In some further applications, the environment is a real-world environment, and the agent manages the distribution of tasks across computing resources, for example, on a mobile device and / or in a data center. In these implementations, actions may include assigning tasks to specific computing resources.

[0112] As further examples, the actions may include presenting advertisements, the observations may include ad impressions or click-through counts or rates, and the rewards may characterize previous selections of items or content made by one or more users.

[0113] In some cases, the observations may include text or spoken commands given to the agent by a third party (e.g., the agent's operator). For example, the agent may be an autonomous vehicle, and the user of the autonomous vehicle may provide the agent with text or spoken commands (e.g., to move to a specific location).

[0114] As another example, the environment could be an electrical, mechanical, or electromechanical design environment, for example, an environment in which the design of an electrical, mechanical, or electromechanical entity is simulated. The simulated environment could be a simulation of a real-world environment in which the entity is intended to operate. The task could be to design the entity. Observations may include observations that characterize the entity, i.e., observations of its mechanical shape, or its electrical, mechanical, or electromechanical configuration, or observations of its parameters or properties. Actions may include actions that modify the entity, for example, modifying one or more of the observations. Rewards may include one or more measures of the performance of the entity's design. For example, rewards may relate to one or more physical properties of the entity, such as weight or strength, or to one or more electrical properties of the entity, such as a measure of the efficiency of the entity in performing a particular function for which it is designed. The design process may include outputting a design for manufacture, for example, in the form of computer-executable instructions for manufacturing the entity. The process may include making the entity according to that design. Therefore, the entity design can be optimized, for example, by reinforcement learning, and then optimized as an optimized design output for manufacturing the entity, such as a computer executable instruction. The entity can then be manufactured using the optimized design.

[0115] As previously described, the environment may be a simulated environment. Generally, in a simulated environment, observations may include one or more simulated versions of the previously described observations, or the types of observations and actions may include one or more simulated versions of the previously described actions or types of actions. For example, the simulated environment may be a motion simulation environment, such as a driving simulation or a flight simulation, and the agent may be a simulated vehicle that moves through the motion simulation. In these implementations, actions may be control inputs for controlling a simulated user or a simulated vehicle. Generally, the agent may be implemented as one or more computers that interact with the simulated environment.

[0116] The simulated environment may be a simulation of a specific real-world environment and agent. For example, a system may be used to select actions in a simulated environment during training or evaluation of the system, and after training, evaluation, or both are complete, it may be deployed to control a real-world agent in the specific real-world environment that was the subject of the simulation. This can avoid unnecessary wear and tear and damage to the real-world environment or the real-world agent, and allows the control neural network to be trained and evaluated for rare situations or situations that are difficult or unsafe to reproduce in the real world. For example, a system may be partially trained using a simulation of a mechanical agent in a simulation of a specific real-world environment, and then deployed to control a real mechanical agent in that specific real-world environment. In such a case, the observations of the simulated environment relate to the real-world environment, and the selected actions in the simulated environment relate to actions that should be performed by the mechanical agent in the real-world environment.

[0117] Optionally, in any of the above implementations, observations at any given time step may include data from previous time steps that may be useful in characterizing the environment, such as actions performed in previous time steps, rewards received in previous time steps, or both.

[0118] Figure 2A is a flowchart of an exemplary process 200 of a goal representation training subsystem for a hierarchical controller, illustrating how new options are generated by the high-level controller. For convenience, process 200 is described as being performed by a system of one or more computers located at one or more locations. For example, an action selection system appropriately programmed according to this specification, e.g., the action selection system 100 in Figure 1, can perform process 200.

[0119] The system may execute process 200 at several time steps during a series of time steps within an episode, for example, at each time step (step 202) where certain criteria for generating a new option are met.

[0120] The criteria for generating new options can also be called the criteria for generating new target expressions.

[0121] A set of criteria may include one or more of various criteria. Some specific criteria that may be included in a set of one or more criteria are described here.

[0122] In one example, these criteria might be based on the fact that previous options have been performed.

[0123] For example, if the previous option was a basic action, the system may determine that the criterion is met in the next time step (even if none of the other criteria described below are met).

[0124] As another example, a system may determine that a criterion is met when the objective expressed by the current objective representation is fulfilled (or achieved). Exemplary techniques for determining when the current objective is fulfilled are described in more detail below.

[0125] As another example, these criteria for generating new options may be accompanied by the same termination criteria that indicate the end of a task episode. Specifically, they may correspond to the end of an episode in which the task is completed, i.e., the objective specified by the previous objective expression is met, or a certain threshold number of steps have been taken while attempting to perform the task.

[0126] As another example, these criteria can define a new option every N environmental time steps, where N is an integer greater than or equal to 1.

[0127] As another example, these criteria may be based on measures corresponding to goal achievement, such as determining whether it is possible to achieve a specified goal representation in a discrete number of future time steps from the state of the environment in the current time step. That is, the criterion may be met when the system determines that it is not possible to achieve the goal represented by the current goal representation within that number of future time steps.

[0128] In some examples, the feasibility of achieving a goal can be determined by comparing the value estimate provided by a low-level controller to a threshold. If the value estimate is higher than the achieved threshold, this indicates that the goal characterized by the goal representation has been achieved. If the value estimate is lower than the unattainability threshold, this indicates that the goal representation is not achievable within a selected number of time steps. These threshold values ​​are discussed in more detail below in Figures 4 and 6.

[0129] In some other examples, auxiliary computations, such as logic-based systems or learnable methods, like networks or linear mappings trained to classify whether a goal is achievable in a set number of future steps, may be used to determine the unachievability of the goal at the current time step. These auxiliary computations are discussed in more detail in Figure 4. If it is determined that the goal is unachievable, the execution of the agent under the chosen goal representation may be terminated early.

[0130] After determining that the criteria for generating a new option are met, the system generates the new option using a high-level controller (step 204).

[0131] The high-level controller output may include one or more types of options, including single-step or multi-step options, as well as criteria for selecting from them.

[0132] In some examples, the output may contain only goal options. Goal options are multi-step options that characterize the target state of the environment that the agent should reach. Goal options can be represented as text, images, or any other type of data that can express the goal that the agent should achieve.

[0133] Specifically, the goal options may include short text describing the target state, such as "the ball on the table."

[0134] In another specific example, the target options might include images that characterize the target state, such as an image showing a ball on a table.

[0135] In some other examples, the output may include other option types.

[0136] In some examples, the output may include basic actions, such as a one-step option to directly control agents in the environment using high-level policy output and select actions for those agents.

[0137] In another example, the output may include high-level actions, such as one-step options, which can influence how the hierarchical controller chooses an action by setting criteria for option termination, such as a maximum option duration threshold, or by applying noise or a function to the policy output used to select the agent's action. In some examples, low-level policy outputs are influenced by these high-level actions.

[0138] In certain examples that allow for more than one type of option, the high-level controller output may also include an indication for selecting the type of option it will output in that time step (step 206).

[0139] Specifically, the high-level policy output may include an indication of whether (i) the agent is controlled using the low-level controller, or (ii) the agent is controlled using the basic behavior identified by the high-level policy output.

[0140] In this case, the high-level controller can use a learned model or a logic-based rule system to decide which option to choose.

[0141] In some examples, the indication of which output to select can be determined using a learned multi-class classification.

[0142] Specifically, a high-level controller may include a high-level option decision network trained to output an option probability distribution used to select from multiple option classes. In this case, either the option with the highest probability may be selected, or the options may be sampled from the distribution.

[0143] For example, a high-level controller can generate goal options, basic options, and high-level actions, which can then determine: (i) output goal options that a low-level controller can process into a goal representation to generate a policy output that controls the agent's actions regarding this goal; (ii) output a one-step basic action that instructs the agent to directly select an action using the high-level policy; and (iii) output a high-level action that can change how the action is determined from the low-level policy output.

[0144] Once the options are generated, the system uses the new options to control the agent's interaction with the environment (step 208).

[0145] For example, the system can select a goal option, use a low-level policy to process the goal option into a goal expression, and then select actions that are conditioned on the goal expression.

[0146] In this case, the high-level controller may decide to reuse multi-step goal options from previous steps in consecutive time steps until the above criteria are met. This decision effectively relinquishes control of the agent to the low-level controller for a certain number of time steps until it is determined that the goal has been achieved or is unattainable, in which case a new goal expression or another option may be generated as detailed above. As another example, the system may select a basic action and then select action 108 using the high-level policy directly.

[0147] As yet another example, the system can select a high-level action and use that high-level action to change the threshold for option termination.

[0148] As yet another example, the system could select a high-level action and use that high-level action to change how action 108 is selected by the low-level policy output.

[0149] In some cases, this change may only affect the choice of one action.

[0150] In other cases, this high-level action imposes a permanent or irreversible change on how the low-level policy output chooses its actions.

[0151] Specifically, a high-level behavior can define parameters that influence a study over consecutive time steps until another high-level behavior sets new parameters.

[0152] For example, high-level behavior can dictate changes to the temperature parameter τ used in the temperature-dependent softmax equation. This type of high-level behavior is discussed in more detail in Figure 3.

[0153] The system continues executing process 200 until a certain number of iterations have been performed to train the hierarchical controller, or until the termination criteria for training the hierarchical controller are met, for example, until the task is successfully executed, until the environment reaches a specified termination state, or until the maximum number of time steps have elapsed between episodes.

[0154] Figure 2B shows in more detail how HLC126 and LLC122 interact.

[0155] The HLC126 includes an observation encoder 301, an RNN core 302, and a policy network 303. These components are described in more detail with reference to Figure 3.

[0156] LLC122 includes a target encoder 408, an observation encoder 405A, one or more target models 406, an RNN core 405B, and a policy and value network 409. These components are described in more detail with reference to Figure 4.

[0157] In this example, HLC126 outputs either a target option to LLC122, or a basic action that directly interacts with the environment 106, i.e., an action that should be used directly as an action to be performed by agent 104 in the environment 106.

[0158] When HLC126 outputs target options to LLC122, the target options are processed into a target expression 124, which is used to condition the policy output of LLC122. LLC122 controls agent 104 in environment 106 under this target expression 124 until the criteria dealt with in Figure 2A are met.

[0159] In this particular case, when the criteria are met, LLC122 can generate a summarized empirical output that serves as input to HLC126. This summarized output is used to guide HLC126 to select a future target option for LLC122, which is discussed in more detail in Figure 4. That is, LLC122 generates an input to HLC126 that summarizes the agent's interaction with the environment since the time prior to when HLC126 generated the target output.

[0160] In some cases, this summarized output may include the reason for the termination of target expression 124. This is discussed in more detail in Figure 6.

[0161] When HLC126 outputs a basic action, HLC126 controls agent 104 in environment 106 to perform this action 108.

[0162] In this particular case, the criterion in Figure 2A may be met immediately after action 108 is performed. As dealt with in Figure 3, HLC126 can process the input of observation 110 to generate a new optional output.

[0163] The replay buffer 510 shown here is used by the training system 190 to perform online-offline training protocols for HLC126 and LLC122, which are discussed in more detail in Figure 5. As part of this training, a target model 406 ("target predictor") may be used to generate auxiliary predictions 407 that can improve the training.

[0164] Figure 3 shows an exemplary HLC126 and its components in more detail.

[0165] At any given time step in which HLC126 is used, HLC126 receives a high-level policy input 310.

[0166] For example, input 310 could be an observation 110 that comes directly from the environment. For instance, input 310 could be an observation received at a given time step.

[0167] As another example, input 310 could be a high-level observation provided by LLC122.

[0168] In a particular example, LLC122 may provide HLC126 with a series of observations 110 collected while attempting to achieve the target representation 124 over a certain number of time steps.

[0169] In another example, an LLC can reduce or summarize a series of observations into high-level observations.

[0170] In a particular example, high-level observations may be constructed using the element-wise average across this series of observations.

[0171] In other examples, these high-level observations may be created by encoding a series of observations through a recurrent neural network (RNN) or another suitable type of machine learning model.

[0172] Therefore, the high-level input summarizes the agent's interaction with the environment since the time prior to when the criteria described with respect to Figure 2B were met.

[0173] As another example, high-level input may contain additional contextual information.

[0174] For example, high-level inputs may include data that identifies the number of time steps since the most recent target option was generated, i.e., the current time duration of the executed option, data characterizing the rewards received from the environment during this time, and data that identifies which criteria for generating a new option were met in this time step.

[0175] In some cases, this criterion for generating new options could be comparing the value estimate from LLC122 for target expression 124 to a threshold.

[0176] Specifically, if the unattainability threshold is not met, the goal may be terminated early, and in this case, a new goal is required.

[0177] Similarly, if the feasibility threshold is exceeded, the goal is considered achieved, and a new goal 124 is required.

[0178] Other possible reasons for early termination are explained below with respect to Figure 6.

[0179] In the example in Figure 3, the high-level policy input 310 is first processed using a neural network that processes inputs to the observation result representation.

[0180] In some cases, this neural network is an encoder. An encoder is a model that generates an encoded representation of an input, which means that the encoder transforms the input information into a latent space of outputs with a defined, fixed shape but different dimensions.

[0181] Specifically, the observation encoder neural network 301 can be used to encode the input 310. In this case, the observation encoder 301 takes in the input 310 and outputs an observation latent space representation.

[0182] This encoded output is then passed to the recurrent neural network (RNN) core 302.

[0183] The RNN core 302 could be a long-short-term memory (LSTM) network, a stacked LSTM, a gated recurrent unit (GRU), or any other variation of a recurrent neural network.

[0184] In other examples, the RNN302 can be replaced by a Transformer model.

[0185] The output of RNN302 is then passed to the high-level policy network 303.

[0186] The high-level policy network 303 processes the output of RNN302 to generate high-level policy output.

[0187] In some examples where there are more than one option, the high-level policy network 303 may include one or more option encoders 303A trained to embed latent options and a high-level option decision network 303B trained to provide an indication of which option to use, in order to generate a high-level policy output 304.

[0188] Specifically, there can be one option encoder for each type of option available for selection by the HLC126.

[0189] In some examples, the available options for the HLC output 304 may be limited by regularizing or restricting the latent output of the optional encoder 303A.

[0190] The option encoder 303A functions similarly to the observation encoder 301. The option encoder 303A processes the observation representation transferred by the RNN 302 to generate an option latent space representation of the type of option it encodes.

[0191] For example, if there is one option encoder for each option type, there could be a target option encoder and a base behavior encoder.

[0192] The various components of HLC126 can be learned together using reinforcement learning to maximize an expected reward 130, which measures the performance of the neural network in controlling agent 104 on a given task. In this case, gradients are passed between subnetwork components during the learning process.

[0193] As a specific example, HLC122 may be trained according to the online Muesli protocol, which incorporates a measure from the training of the high-level policy network 303A as an additional loss in the reinforcement learning process.

[0194] More specifically, the Muesli protocol offers flexibility in the output of both a continuous option (multiple steps) and a discrete option (one step) that can be used to control the low-level controller 122.

[0195] This high-level policy output 304 may include any number of different types of options as well as indicators for making a decision among them.

[0196] In some examples, policy output 304 may contain only the target output option subcomponent.

[0197] In other examples, policy output 304 may include target output option subcomponents, basic action subcomponent options, and a binary indication of which option should be chosen at that time step, provided by the high-level option decision network 303B.

[0198] In this particular case, the binary indicator controls the LLC neural network 122 to determine whether HLC126 directly performs a basic action in environment 106, or to select an action 108 that is conditioned on a target output returned in output 304 which is processed into a target representation 124.

[0199] In a further example, policy output 304 may include additional high-level behavioral options. In this case, the indication of which option to choose is a multi-class indication of which option to choose at that time step, provided by the high-level option decision network 303B.

[0200] In particular, high-level behaviors may encompass behaviors taken by HLC126 to further control the selection of behaviors by LLC122.

[0201] For example, high-level behavior may include parameters that adjust the LLC122 policy output z.

[0202] In particular, high-level behaviors are

[0203]

number

[0204] It is possible to specify a change to the temperature parameter τ used in a temperature-dependent softmax formula, such as defined by , where z(i) is the score for action i in the low-level policy output, and the sum is the sum over action j in the set of actions that can be performed by the agent. Thus, adding the temperature parameter τ to the argument of the exponential in a standard softmax formula produces different probability distributions over the set of potential actions.

[0205] Therefore, in this example, when there are multiple possible options, the high-level controller 130, or more generally the behavior selection system 100, can analyze the high-level policy output and, based on the indications, decide which option to select at any given time step.

[0206] Although not shown in this example, HLC126 could instead be implemented as a Large-Scale Language Model (LLM) neural network, such as a text-based LLM neural network, or a Visual Language Model (VLM) neural network that accepts both images and text as input. The LLM may be either pre-trained or fine-tuned, and can take a high-level policy input 310 and output a high-level policy output 304 as a natural language instruction. That is, in this example, the high-level policy output could specify a goal as natural language text, or specify a basic action as one or more text tokens when a basic action should be performed.

[0207] In particular, HLC 126-LLM can process an exemplary text high-level policy input 310 to produce an output 304. As another example, HLC 126-VLM can process an exemplary image high-level policy input 310 or an exemplary high-level policy input containing both image and text to produce an output 304. As a further example, a multimodal Transformer-based HLC126 may be implemented to process and produce other data modalities of input 310 or output 304.

[0208] Figure 4 shows an exemplary low-level controller 122 and its components in more detail. This figure separates policy execution 122A from auxiliary task 122B into a separate branch to provide a complete overview of the LLC 122's functionality both (i) during policy execution 122a for selecting action 108 and (ii) during training with both policy execution 122A and auxiliary task 122B.

[0209] During training, both policy execution branch 122A and auxiliary task branch 122B run together, passing information as shown by the dotted line in Figure 4 to update the parameters of the policy and value network 409, including the policy head and value head that use the auxiliary output 407.

[0210] The policy head includes one or more policy neural networks trained to learn the probability of the action to be taken for each observation 110 in that time step.

[0211] The value head includes one or more value neural networks trained to learn the value of observation 110 at that time step. The value of observation 110 at that time step represents the value of the environment in a state characterized by the observation reaching the target state characterized by the target representation. That is, the value is a score that represents the value of the environment in a state characterized by the observation reaching the target state characterized by the target representation.

[0212] After training, as shown by the solid line in Figure 4, no parameter updates are required, so only the policy execution branch 122A functions without the auxiliary task branch 122B, producing a low-level policy output.

[0213] As described above, in some implementations, the criteria for generating a new goal representation may include one or more criteria that depend on goal attainability, goal unattainability, or both. In this case, goal attainability or unattainability may be determined using thresholds corresponding to the output values.

[0214] LLC122 has a low-level policy input 410 which may include options such as target option 424 or high-level action, and the current time step O t and previous action A t-1 It receives and encodes a low-level policy input 410, which may include observation-behavior pairs containing data from the source.

[0215] This input 410 may originate directly from the environment 106 during or after online training, or it may originate from a sample of observations 110 recorded in the replay buffer during offline training.

[0216] In the case of sampling from the replay buffer during offline training, LLC122 can use hindsight selection of the target to select and encode its intrinsic target representation 124. This process is discussed in more detail in Figure 6.

[0217] For target option 424 as input 410 during training or for options proposed by HLC after training, the target encoder 408 in auxiliary task branch 122B receives the target 124 and embeds the target representation G t Encode it as 124 into the latent target space.

[0218] By embedding the target into latent space, the target encoder 207 can be agnostic about the target type. This setup allows the use of any target modality, considering that the encoder can be trained from target to the shared embedding space of targets.

[0219] Goal expression G t124 can be represented by text, images, or any other type of data that can be embedded into the shared target space.

[0220] In some examples, the target encoder can be an autoencoder-like network that outputs the parameters of a multivariate normal distribution with diagonal covariance as the shared embedding space of the target. The system can then sample the target representation 124 from this multivariate normal part.

[0221] [[ID=​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​Policy and Value Network 409 is a low-level targeted policy μ LLC , Low-level behavior policy π LLC , and value estimator V LLC Includes.

[0228] Low-level targeting policy μ LLC This is a policy that is trained to learn a target conditional probability distribution across multiple actions using the rewards 130 received for each of the 108 actions. This policy is trained through offline reinforcement learning to learn the behavior policy π LLC It is trained using behavioral mimicry from offline data to serve as an expert policy that can improve the effectiveness of training. In other words, the system trains low-level target policies through behavioral mimicry of trajectories from offline datasets, for example, the same dataset used to train the behavioral policies or different offline datasets.

[0229] Behavioral policy π LLC This is a policy configured to process a given observation result 110 and target expression 124 to generate a policy output that can select an action for the agent.

[0230] In some examples, the behavior policy output may be influenced by high-level options that change how behavior 108 is selected from the behavior policy output.

[0231] In particular, π LLC While training, the system will use a trained target policy π that mimics the behavior of the trained target policy π. LLC Using π LLC Regularize the training.

[0232] Specifically, π LLC To train the model, a proximity regularization term can be incorporated into the loss function of the behavior policy.

[0233] In some cases, this proximity regularization term is π LLC and μ LLCThis can be based on the discrepancy with [the other party].

[0234] After training, this becomes a behavior policy π that can select action 108 according to a target conditional probability distribution across multiple actions. LLC It can produce.

[0235] Value Estimator V LLC This is based on learning the expected cumulative reward for observation 110 over consecutive time steps, and the value V for that observation 110. LLC The system generates a value, and the reward at a given time step represents whether the current goal was achieved at that time step. For example, the system can train a value estimator using a regression loss on the ground truth value for that time step.

[0236] Behavioral policy π LLC It can be trained using any appropriate actor-critic method.

[0237] In particular, the behavior policy π LLC This may be trained offline using the IMPALA (Importance Weighted Actor-Learner Architecture) protocol (Espeholt et al., arXiv:1802.01561), which can be scaled using distributed computing to consist of one or more decoupled learners.

[0238] In addition, IMPALA is π LLC To train the model and correct policy lag inconsistencies in offline configurations with multiple learners, incorporate the use of the V-trace term.

[0239] Specifically, the behavior policy π LLC This can be optimized by following the V-Trace policy gradient.

[0240] The V-Trace term consists of (i) a favorable term (favorable estimate) based on the reward 130 for the observation-behavior pair compared to the average of possible behaviors performed under that observation 110, and (ii) (a) the observational representation b of observation 110. t By processing the low-level policy head π LLC The probability a of being assigned to one of the 108 actions in a pair. t (b) Observational representation of the target for the trajectory t And by processing the embedded target representation g124, the behavior imitation policy head μ LLC The probability assigned to the 108 actions in the pair by the behavior imitation policy output generated by

[0241]

number

[0242] It is the product of the ratios of . This term is

[0243]

number

[0244] It can be expressed as, Adv t This is a favorable estimate at time t, calculated using the return and value estimates of V-Trace for time step t. This is described in more detail in Espeholt et al., arXiv:1802.01561.

[0245] In particular, the total loss is calculated using the V-Trace term and the KL divergence π. LLC and target policy μ LLC It may include a regularization term weighted by a hyperparameter α that penalizes discrepancies with the formula.

[0246]

number

[0247] In the modified V-Trace algorithm, π LLC The gradient can be calculated with respect only to the parameter.

[0248] In other words, the system can use the above loss to train the low-level controller components offline.

[0249] The auxiliary task branch 122B may also include a target predictor 406 to improve the performance of LLC 122 with respect to the target representation 124.

[0250] In auxiliary task branch 122B, the output C of the observation result encoder 405A t This may be transmitted to the target evaluator 406, which is an intermediate observation result representation C. t and embedded target representation G t Use 124 to evaluate the target objective 124.

[0251] These target predictors 406 may be learned mappings, such as neural networks or other functions and measures, which can be used to assess the feasibility and availability of previously trained targets.

[0252] In some examples, the goal predictor 406 may include a goal achievement evaluator 406A that predicts whether the goal 124 is achievable within one or more predetermined integer steps.

[0253] In another example, the predictor 406 may include a target similarity score calculator 406B that uses a high-dimensional vector distance criterion to reject targets 124 that are too close to previously trained targets.

[0254] In some cases, a high-dimensional vector distance criterion can be a cosine similarity measure.

[0255] The auxiliary output 407 of the target evaluator 406 can serve as an auxiliary task for training the policy execution branch 122A. This means that when training the policy and value network 109, the output 407 can also be used as an additional loss.

[0256] Once training is complete, these auxiliary tasks are no longer necessary. In this case, goal achievement is determined by the value network, and value estimation is used to determine when to generate new goal options.

[0257] Specifically, the criterion for generating a new target representation is met when the value estimate generated by the value head 409 by processing the observed representation and target representation 124 for the preceding time step exceeds an achieved threshold indicating that the target 124 for the preceding time step has been achieved.

[0258] Similarly, the criterion for generating a new target representation 124 is satisfied when the value estimate generated by the value head 409 by processing the observed representation and target representation 124 for the preceding time step falls below an unreachability threshold indicating that the target 124 for the preceding time step will not be achieved in a certain discrete number of time steps.

[0259] Figure 5 shows an exemplary training system 190, specifically an offline-online training system 500, which can be used to train a hierarchical controller 130 to perform new tasks on data collected throughout the training process using unorganized data, i.e., policies with varying capabilities.

[0260] In such a system, HLC126 is trained online via an online protocol 501 (represented here by a dashed arrow), and LLC122 is trained offline via an offline protocol (represented here by a solid circular box). Online training means that HLC(126) is trained using an online dataset, and its network gradient is updated after receiving immediate feedback, for example, in the form of the next-day result 110 and reward 130 from time t+1 immediately after it acted on the environment 106, or in the form of a reduced observation 124 received from LLC122 at time t. Offline training means that LLC(122) is trained using an offline dataset, and its network gradient is updated, for example, by sampling past observations recorded in a replay buffer 510.

[0261] Here, the replay buffer 510 can be any data structure capable of storing experience objects. For example, the replay buffer 510 can store a set of past experience trajectory objects 515.

[0262] Each trajectory object 515 can be a series of observation-action pairs collected as a result of the interaction of agent 104 with environment 106 while being controlled using different policies, including policies with varying capabilities.

[0263] For example, different policies may be fixed control policies, learned policies, or controlled by experts, such as human users.

[0264] This offline-online training system 500 can be trained on unorganized data using hindsight selection of objectives, which is discussed in more detail in Figure 6.

[0265] In this specific example, HLC126 can output either the target option 424 or the basic action 511.

[0266] If the output of HLC126 is the basic action 511, the basic action 511 bypasses LLC122 and directly interacts with the environment 106 to advance it, resulting in the basic observation 512.

[0267] Specifically, the basic observation 512 may include the next observation 110 and reward 130 from time t + 1 immediately after it acts on the environment 106 at time t.

[0268] As an exemplary interaction, HLC126 can output the basic action 511 only at the beginning and end of an episode.

[0269] As another exemplary interaction, HLC126 may output the target option 424 that bypasses the environment 106 and is sent directly to LLC122. In this case, the target option 424 is processed into the target representation 124 used for training.

[0270] In yet another example, the target hindsight selection enables the training of the hierarchical controller 130 for any data related to multiple targets in the replay buffer 510, particularly data not generated by an expert, which is dealt with in more detail in Figure 6.

[0271] At the end of training, LLC122 can provide the reduced observation 514 to HLC126. In this case, LLC122 is the cause of what HLC126 observes from the environment.

[0272] The reduced observation 514 is a component of the information that HLC126 needs to determine which option to execute next, i.e., which option to indicate in the policy output 304.

[0273] These abridged observations 514 may include any one of the following: environmental observations 110, reasons for early termination, a compressed history of observations, or any other information sufficient to summarize LLC 122's interaction with the environment 106 during offline training 502.

[0274] As an example of interaction, in the course of a single episode, HLC126 may receive an initial observation 110 from LLC122 at the beginning of the episode and output a basic behavior 511. Upon receiving the basic online observation 512, HLC126 may then output a target option 424 to LLC122, which initiates LLC122 sampling of past experience trajectories 515.

[0275] In some cases, this sampling can utilize the process for hindsight selection of targets, as discussed in Figure 6.

[0276] In the second-to-last step of the episode, LLC122 may provide HLC126 with abridged observations 514, which include a history of observations, at which point HLC126 initiates the online training protocol 501.

[0277] As another example, as the episode progresses, HLC126 may receive reasons for early termination of the initially selected goal option 424 for the episode as part of the abridged observations 514. Possible reasons for early termination are discussed in Figure 6.

[0278] In some implementations, the LLC network 122 may also be frozen during training of the HLC, such as when a high-level controller is trained separately from the LLC. Keeping a neural network "frozen" during training means not passing gradients for updates, i.e., keeping parameter values ​​fixed while changing parameter values ​​in another neural network.

[0279] This exemplary offline-online training system 500 may be deployed "on a large scale," meaning the system may be distributed across multiple agents and learners as defined by the IMPALA protocol.

[0280] In particular, the system 500 may be deployed such that the replay buffer 510 from which LLC 122 samples in the offline protocol 502 is filled by one or more agents interacting in one or more environments.

[0281] Furthermore, system 500 may be used to train a hierarchical controller 130 so that agent 104 can be controlled to generalize to new industrial tasks not encountered during training LLC 122, HLC 126, or both. This is because the training system 500 facilitates the training of autonomous agent 105, which can learn how to generate and use its own goal representation 124 from scratch using its own unorganized experience.

[0282] Figure 6 is a block diagram of an exemplary process 600 for offline hindsight selection of a target using LLC122 under the online-offline training protocol 500 shown in Figure 5. For convenience, process 600 is described as being performed by a system of one or more computers located in one or more locations. For example, a well-programmed training system, e.g., training system 190 in Figure 1, can perform process 600.

[0283] In this case, LLC122 trains offline on a plurality of tasks sampled from replay buffer 510. This multi-task training has the advantage of providing HLC126 with reduced observations corresponding to a plurality of target representations 124, thereby inducing online learning of HLC126 as to which option to select for LLC122. In particular, these reduced observations may include reasons for early termination of the target.

[0284] Specifically, LLC122 can sample a past experience trajectory 515 from the observations recorded in replay buffer 510, and generate a plurality of tasks 610 from this sampled past experience trajectory 515. These tasks 610 form components of segment partial trajectories, and each partial trajectory is a subset of a given observation 515 from start time t start to end time t end Typically, the target is fixed for the task, i.e., the target is achieved at t end

[0285] By sampling a plurality of partial trajectories, LLC122 can be trained on a plurality of tasks 610 at once.

[0286] In particular, this pruning of the sampled past experience trajectory 515 can be optimized in some way to assist in the training of LLC122.

[0287] For example, tasks 610 can be improved to increase the frequency of segments that are non-zero reward components, or to produce specific instances of tasks desired by the user.

[0288] In some examples, non-goals, i.e., tasks without a target state of the target to be achieved, may also be included to assist in agent exploration.

[0289] In yet another example, tasks can be sampled randomly from the past experience trajectory 515.​

[0290] LLC122 then performs training across multiple tasks 610, which is created by completing a loop across all tasks 610 generated from sampled past experiences 515.

[0291] In some cases, this loop, spanning multiple tasks, can constitute a single episode of LLC122 training.

[0292] In other cases, this loop, spanning multiple tasks, can be a component of an integer number of steps in the environment, which may be one or more.

[0293] Within the loop, each task 610 acts as an input to the target encoder 408 and the value head 409.

[0294] The target encoder 408 is fixed to the target representation G of its sampled partial trajectory task 610. t The output is 124. This output is then evaluated using the target evaluator 406, specifically the target achievement evaluator 406A and the similarity score calculator 406B.

[0295] If the goal achievement evaluator 406A determines that goal 124 is too far off temporarily, the reason for early termination may be provided to the HLC 126 at the discretion of the judge, as part of the reduced observations 514.

[0296] In some implementations, the goal achievement evaluator can be a classifier that directly learns the termination function.

[0297] If the distance similarity score calculator 406B determines that objective 124 of task 610 is too close to other objectives 124 in the episode, the reason for early termination may be provided to the HLC 126 at the discretion of the user, as part of the reduced observations 514.

[0298] In some implementations, the similarity score calculator 406B can be a measure of distance, such as cosine similarity, which is evaluated to exclude all targets from training up to a user-defined similarity level.

[0299] For example, according to this scale, objective 124 with a similarity greater than 60% may be rejected from training.

[0300] For each task 610, value estimation V from the value network 409 is performed. LLC It uses a value-based logic rule to reject target 124 whose value falls below a threshold, and a pre-set unreachability threshold V unreachability threshold It can be compared to this.

[0301] If the value estimate is too low, HLC126 may receive a reason for early termination that indicates that the target 124 was unattainable.

[0302] This threshold can be provided by HLC126 or set to a fixed value.

[0303] Other implementations may include reasons for early termination not explicitly stated here, which are intended to simplify training for LLC122.

[0304] If Task 610 is not rejected for any of the reasons for early termination mentioned above, or for any reason not mentioned above, then LLC122 will be trained.

[0305] Training can be carried out through any appropriate reinforcement learning method.

[0306] In particular, training may include the regularized offline V-Trace algorithm 620 described above.

[0307] Figure 7 shows the performance of the offline-online hierarchical system (H2O2) in Figures 5 and 6, using average episode returns per different task taken from the DeepMind Hard Eight task suite, compared to the latest flat agent baseline.

[0308] Figure 7 shows six plots corresponding to a subset of six tasks from the DeepMind Hard Eight task suite, namely the baseball task, drawbridge task, navigation cubes task, push blocks task, wall sensor task, and wall sensor stack task. These tasks involve task components in a visually complex and partially observable 3D environment, which has been challenging for hierarchical agents to date.

[0309] The results for the throw-across and remember sensor tasks have been excluded because neither agent showed any evolution in these tasks during the tests.

[0310] In most tasks, agents trained using the described hierarchical training system performed better than or at least as well as other agents. In particular, H2O2 achieved better results in the baseball task, the wall sensor task, and the navigation cubes task.

[0311] While H2O2 has not achieved better results in the drawbridge or push blocks tasks, it still holds its own against the standards of modern flat agents, representing an effort to address previous challenges in hierarchical reinforcement learning, such as demonstrating performance comparable to other flat (non-hierarchical) techniques in visually complex and partially observable 3D environments.

[0312] In practice, the reason why hierarchical agents might perform worse than flat agents for these two tasks could be that the LLC design substantially alters the quasi-Markov decision process (SMDP) that the HLC experiences. If the SMDP is easier to solve than the task's original Markov decision process (MDP), the hierarchical agent may only be on par with the flat agent.

[0313] Furthermore, Figure 7 is important because H2O2's performance stems from learning target representation behavior offline from arbitrary experiences generated by the agent, rather than relying on datasets organized by the expert agent.

[0314] In Figure 8, the power of including the KL regularization term in the modified V-Trace algorithm is demonstrated in the case of an offline-online implementation of H2O2 for a representative task. In this representative task, removing the penalty could cause the learned policy to deviate significantly from the behavioral policy, resulting in considerably poor performance.

[0315] This specification uses the term “configured” in relation to systems and computer program components. One or more computer systems being configured to perform a particular operation or action means that the system has installed software, firmware, hardware, or a combination thereof that causes the system to perform that operation or action during operation. One or more computer programs being configured to perform a particular operation or action means that one or more programs contain instructions that, when executed by a data processing device, cause the device to perform that operation or action.

[0316] The subject matter and functional operating embodiments described herein may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed herein and their structural equivalents, or in one or more combinations thereof. Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible non-temporary storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer-readable storage medium may be a machine-readable storage device, a machine-readable storage board, a random or serial access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions may be encoded into artificially generated propagating signals, such as machine-generated electrical signals, optical signals, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0317] The term "data processing device" refers to data processing hardware and encompasses all types of devices, machines, and equipment for processing data, including, for example, programmable processors, computers, or multiple processors or computers. A device may be a dedicated logic circuit, such as an FPGA (Field-Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit), or may further include these. In addition to hardware, a device may optionally include code that constitutes a component of an execution environment for computer programs, such as processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0318] Computer programs, also called or sometimes described as programs, software, software applications, apps, modules, software modules, scripts, or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A program may, but does not have to, correspond to a file in a file system. A program may be stored in a part of a file that holds other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple collaborative files, such as a file that stores one or more modules, subprograms, or parts of code. A computer program may be deployed to run on one computer, or on multiple computers located in one place or distributed across multiple locations and interconnected by a data communication network.

[0319] In this specification, the term “engine” is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers are dedicated to a particular engine, while in other cases, multiple engines may be installed and running on the same one or more computers.

[0320] The processes and logic flows described herein may be implemented by one or more programmable computers that perform their functions by executing one or more computer programs to manipulate input data and generate outputs. The processes and logic flows may also be implemented by dedicated logic circuits, such as FPGAs or ASICs, or by a combination of dedicated logic circuits and one or more programmed computers.

[0321] A computer suitable for running computer programs may be based on a general-purpose microprocessor, a dedicated microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory or random-access memory, or both. The basic elements of a computer are a central processing unit for executing or running instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory may be complemented by or incorporated into dedicated logic circuits. Generally, a computer is one or more mass storage devices for storing data, including, for example, magnetic disks, magneto-optical disks, or optical disks, or operably coupled to them to receive data from them or to transfer data to them, or both. However, a computer does not need to have such devices. Moreover, a computer may be incorporated into another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, such as a Universal Serial Bus (USB) flash drive.

[0322] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0323] To enable user interaction, embodiments of the subject matter described herein may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices may also be used to enable user interaction. For example, the feedback given to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and the input from the user may be received in any form, including acoustic input, speech input, or tactile input. In addition, the computer may interact with the user by sending and receiving documents to and from a device used by the user, for example, by sending a web page to a web browser in response to a request received from the user's device's web browser. The computer may also interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving response messages from the user in return.

[0324] Data processing equipment for implementing machine learning models may also include, for example, dedicated hardware accelerator units for handling the general, computationally intensive parts of machine learning training or product, i.e., inference workloads.

[0325] Machine learning models can be implemented and deployed using machine learning frameworks, such as the TensorFlow framework.

[0326] Embodiments of the subject matter described herein may be implemented in a computing system that includes, for example, a data server as a backend component, or a middleware component, such as an application server, or a client computer having a graphical user interface, a web browser, or an application through which a user can interact with an implementation of the subject matter described herein, or one or more such combinations of backend components, middleware components, or frontend components. The components of the system may be interconnected by digital data communications of any form or medium, such as communication networks. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0327] A computing system may include clients and servers. Clients and servers are generally geographically separated and typically interact through a communication network. The client-server relationship arises when computer programs run on each computer and have a client-server relationship with each other. In some embodiments, the server sends data, such as an HTML page, to a user device for the purpose of displaying data to a user interacting with a device acting as a client and receiving user input from the user. For example, data generated on the user device as a result of user interaction may be received from the device to the server.

[0328] This specification includes many details of specific implementations, but these should not be interpreted as limitations on the scope of the invention or the scope of what can be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Some features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may be implemented separately in multiple embodiments or in any suitable partial combination. Furthermore, features may be described above as operating in several combinations, and may even be initially claimed in this manner, but one or more features from a claimed combination may be omitted from the combination in some cases, and the claimed combination may cover a partial combination or a variation of a partial combination.

[0329] Similarly, while operations are shown in the drawings and described in the claims in a specific order, this should not be understood as requiring that such operations be performed in a specific or sequential order shown, or that all shown operations be performed, in order to achieve the desired result. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0330] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions described in the claims may be performed in a different order and still achieve the desired results. As an example, the process shown in the accompanying drawings does not necessarily require the specific order or sequential order shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of Symbols]

[0331] 100 Action Selection System 102 Behavior Selection Subsystem 104 Agent 106 Environment 108 Action 110 Observation Results 122 Low-level controllers 124 Goal expression 126 High-Level Controllers 130 Rewards 190 Training Systems 301 Observation Results Encoder 302 RNN cores 303 Policy Network 405A Observation Result Encoder 405B RNN core 406 Target Model 407 Supplementary forecasts 408 Target Encoder 409 Policy and Value Network 410 Low-level policy input 411 LLC output 424 Target Options 500 Offline-Online Training Systems 501 Online Protocol 502 Offline Protocol 510 Replay Buffer 511 Basic actions 512 Basic OBS 514 Abridged OBS 515 Past Experiences and Trajectories 610 tasks 611 Value Estimation 620 Regularized Offline V-Trace Learning

Claims

1. A method performed by a system implemented on one or more computers for controlling an agent that interacts with the environment to perform a task, wherein in each of a plurality of first time steps from a plurality of time steps, The system includes the step of receiving observational results that characterize the state of the environment in the first time step, The system determines a target representation for a first time step that characterizes the target state of the environment to which the agent should reach, The step of determining the target representation for the first time step that characterizes the target state of the environment that the agent should reach is: A step of determining whether the criteria for generating a new target representation are met in the first time step, If the above criteria are not met, the step of using the target expression from the preceding first time step as the target expression for the first time step, When the above criteria are met, A step of generating high-level observational results for the first time step, A step of processing a high-level input comprising the high-level observation results using a high-level controller neural network to generate a high-level policy output comprising a target output characterizing the target state and an indication of whether (i) the agent is controlled using a low-level controller neural network or (ii) the agent is controlled using basic actions identified by the high-level policy output, The steps of: processing the target output using a target encoder neural network to generate the target representation; It has steps, The system processes the observations and target representation using the low-level controller neural network to generate a low-level policy output that defines the action to be performed by the agent in response to the observations, wherein the low-level controller neural network A representation neural network configured to process the observation results and generate an internal state representation of the observation results, A low-level policy head configured to process the internal state representation and the target representation of the observation results to generate the low-level policy output, Equipped with steps and The system controls the agent using the low-level policy output. A method that includes [a certain feature].

2. The method according to claim 1, wherein the steps of determining the target representation for the first time step, processing the observations and the target representation using the low-level controller neural network, and controlling the agent using the low-level policy output are performed only in response to a decision, based on the indication, to control the agent using the low-level controller neural network.

3. In each of the one or more second time steps from the plurality of time steps, A step of generating high-level observational results for the second time step, The steps include processing high-level observation results for the second time step using the high-level controller neural network to generate a high-level policy output for the second time step, (i) controlling the agent using the low-level controller neural network, or (ii) controlling the agent using the basic behavior identified by the high-level policy output, based on the indication, the step of deciding to control the agent using the basic behavior identified by the high-level policy output, In response, the steps include controlling the agent using the basic behavior identified by the high-level policy output, and The method according to claim 2, comprising:

4. The method according to claim 1, wherein the high-level observation result for the first time step comprises the observation result received in the first time step.

5. The high-level observation results for the first time step are, The method according to claim 1, further comprising data that identifies the number of time steps since the most recent time step in which the aforementioned criteria were met.

6. The high-level observation results for the first time step are, The method according to claim 1, comprising data characterizing the rewards received from the environment since the most recent time step in which the aforementioned criteria were met.

7. The high-level observation results for the first time step are, The method according to claim 1, comprising data characterizing observations received in time steps since the most recent time step in which the aforementioned criteria were met.

8. When any of the criteria in the set of criteria are met, the criteria are met and the high-level observation result for the time step is The method according to claim 1, comprising data that identifies which criteria were met in the first time step.

9. The method according to claim 1, wherein the target output characterizing the target state is a target vector characterizing the target state.

10. The method according to claim 1, wherein the target output characterizing the target state is a text string describing the target state.

11. The high-level policy output comprises hyperparameters defining some aspect of training, and the low-level policy output is used to control the agent. The steps include applying the hyperparameters to the low-level policy output to generate a modified policy output, The steps include selecting an action using the aforementioned adjusted policy output and The method according to claim 1, comprising:

12. The method according to claim 11, wherein the high-level policy output comprises a temperature parameter as a hyperparameter for adjusting the low-level policy output.

13. The step of determining whether the criteria for generating a new target representation are met in the first time step is: The method according to claim 1, further comprising the step of determining that the criteria for generating a new target representation are met when the first time step is the first time step in a task episode.

14. The step of determining whether the criteria for generating a new target representation are met in the first time step is: The method according to claim 1, further comprising the step of determining that the maximum number of time steps have elapsed since the most recent time step in which the aforementioned criteria were met.

15. The method according to claim 1, further comprising a value head configured to process an observational representation and the target representation to generate a value estimate of the value of the environment in a state characterized by the observation that the target state has been reached, as characterized by the target representation.

16. The step of determining whether the criteria for generating a new target representation are met in the first time step is: The method of claim 15, further comprising the step of determining that the criteria for generating a new target representation are met when the value estimate generated by the value head by processing the observational representation of the observation in a preceding time step and the target representation for the preceding time step is less than an unreachability threshold.

17. The step of determining whether the criteria for generating a new target representation are met in the first time step is: The method of claim 15, comprising the step of determining that the criterion for generating a new target representation is met when the value estimate and the target representation for the preceding time step, generated by the value head by processing the observation representation of the observation result in the preceding time step, are greater than an achieved threshold indicating that the goal characterized by the target representation for the preceding time step has been achieved.

18. The method according to claim 17, wherein the achieved threshold is included in the high-level policy output.

19. The step of determining whether the criteria for generating a new target representation are met in the first time step is: The steps include using a classifier neural network to process a classifier input that characterizes (i) observations in a preceding time step and (ii) a target state for the preceding time step, and to generate a classifier output that indicates how many time steps remain until the target state is achieved, When the classifier output indicates that zero time steps remain until the target state is achieved, the step of determining that the criteria for generating a new target representation are met. The method according to claim 1, comprising:

20. A method for training the low-level controller neural network described in Claim 1 using the system, The system includes the steps of acquiring one or more trajectories of observation result-behavior pairs and the respective goals for each trajectory, The system includes the steps of generating a reward for each pair in the trajectory based on whether the goal was achieved in the state characterized by the observation result in the pair, The steps of the system include optimizing a reinforcement learning objective by training the low-level controller neural network for one or more trajectories and one or more rewards, wherein the reinforcement learning objective comprises a proximity regularization term that penalizes the low-level policy output generated by the low-level controller neural network for deviation from the behavior imitation policy output generated by a behavior imitation head, which is (i) trained for behavior imitation loss and (ii) configured to process a given observation result representation and a given target representation for a given observation result generated by the representation neural network to generate a behavior imitation policy output for the observation result. A method that includes [a certain feature].

21. The method according to claim 20, wherein the behavior imitation policy output and the low-level policy output each define a probability distribution over a plurality of actions, and the proximity regularization term is based on the difference between (i) a low-level policy output generated by the low-level policy head by processing the observation result representation of the observation result and the target representation of the target for the trajectory generated by the representation neural network, and (ii) a behavior imitation policy output generated by the behavior imitation policy head by processing the observation result representation of the observation result and the target representation of the target for the trajectory generated by the representation neural network, for each observation result in each trajectory.

22. The method according to claim 21, wherein the aforementioned divergence is a KL divergence.

23. The method according to claim 21, wherein the reinforcement learning objective further comprises, for each observation-action pair in each trajectory, (i) a term based on the reward for the pair, and (ii) a term which is the product of (a) the probability that the action in the pair is assigned by the low-level policy output generated by the low-level policy head by processing the observation representation of the observation and the target representation of the target for the trajectory generated by the representation neural network, and (b) the probability that the action in the pair is assigned by the behavior imitation policy output generated by the behavior imitation policy head by processing the observation representation of the observation and the target representation of the target for the trajectory generated by the representation neural network.

24. The method according to claim 23, wherein the term based on the reward for the pair is a V-Trace policy gradient term.

25. The method according to claim 20, further comprising the step of training the behavior imitation policy head and the representation neural network for one or more trajectories to minimize the behavior imitation loss.

26. For one or more of the aforementioned trajectories, the target is an image of the environment in the target state, and the training step is The method according to claim 20, further comprising the step of processing the image using an image target encoder neural network to generate a target representation to be processed by the low-level policy head and the behavior mimicry policy head.

27. For one or more of the aforementioned trajectories, the objective is a text describing the target state of the environment, and the training step is The method according to claim 20, further comprising the step of processing the text using a text target encoder neural network to generate a target representation to be processed by the low-level policy head and the behavior mimicry policy head.

28. The method according to claim 20, wherein the one or more trajectories are sampled from offline data generated while the agent was controlled by one or more different behavioral policies.

29. The method according to claim 28, wherein the low-level controller neural network is trained only on the offline data.

30. The method according to claim 28, wherein the low-level controller neural network is trained on both the offline data and the online data generated by controlling the agent using the low-level policy output generated by the low-level controller neural network.

31. The method of claim 20, further comprising the step of training the high-level controller neural network through reinforcement learning to maximize the expected reward for the task, which is generated as a result of controlling the agent based on the high-level policy output generated by the high-level controller neural network and the low-level policy output generated by the low-level controller neural network.

32. The method according to claim 31, wherein, as a result of the training of the high-level controller neural network, no gradient is passed to the low-level controller neural network.

33. The method according to claim 31, wherein the one or more trajectories include one or more trajectories sampled from online data generated while controlling the agent during the training of the high-level controller neural network.

34. The method according to claim 31, wherein the one or more trajectories do not include any trajectories sampled from online data generated while controlling the agent during the training of the high-level controller neural network.

35. The method according to claim 21, wherein the reinforcement learning objective comprises one or more auxiliary loss terms, each auxiliary loss term corresponding to a respective auxiliary task, and each of the auxiliary tasks requires generating a prediction that characterizes a target distribution conditioned on the output generated by the representation neural network.

36. The system further comprises a step of generating a target for each of the one or more trajectories of the observation result-action pair, wherein the generating step is for one or more of the trajectories The method according to claim 21, further comprising the step of selecting a target that describes the state of the environment characterized by the last observation in the trajectory.

37. The system further comprises a step of generating a target for each of the one or more trajectories of the observation result-action pair, wherein the generating step is for one or more of the trajectories The method according to claim 21, further comprising the step of selecting a target that describes the state of the environment not characterized by any of the observation results in the trajectory.

38. The method according to claim 1, wherein the agent is a mechanical agent and the environment is a real-world environment.

39. The method according to claim 38, wherein the agent is a robot.

40. The method according to claim 1, wherein the environment is a real-world environment of a service facility comprising a plurality of electronic devices, and the agent is an electronic agent configured to control the operation of the service facility.

41. The method according to claim 1, wherein the environment is a real-world manufacturing environment for manufacturing a product, and the agent comprises an electronic agent configured to control a manufacturing unit or machine that operates to manufacture the product.

42. One or more computers, A system comprising one or more storage devices communicably coupled to one or more computers, wherein, when executed by one or more computers, one or more storage devices store instructions causing one or more computers to perform the operation of each of the methods described in any one of claims 1 to 41.

43. One or more non-temporary computer storage media, which, when executed by one or more computers, store instructions causing the one or more computers to perform the operation of each of the methods described in any one of claims 1 to 41.

Images