Options with meta-gradient learning action selection in multi-task reinforcement learning

Abstract

A reinforcement learning system, method, and computer program code for controlling an agent to perform multiple tasks while interacting with an environment. The system learns options, where an option comprises a sequence of primitive actions performed by the agent under the control of an option policy neural network. In implementations, while the agent interacts with the environment, the system discovers options useful for multiple different tasks by a meta-learning reward for training the option policy neural network.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 035,467, filed June 5, 2020. The disclosure of the earlier application is considered part of the disclosure of this application and is incorporated herein by reference. Background Technology

[0003] This manual relates to reinforcement learning.

[0004] In reinforcement learning systems, agents interact with the environment by performing actions selected by the reinforcement learning system in response to receiving observations that represent the current state of the environment.

[0005] Some reinforcement learning systems select actions to be performed by an agent based on the output of a neural network in response to a given observation.

[0006] A neural network is a machine learning model that uses one or more non-linear units to predict the output of a received input. Some neural networks are deep neural networks, which include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as the input to the next layer in the network, i.e., the next hidden layer or output layer. Each layer of the network generates an output from the received input based on the current values ​​of its corresponding set of parameters. Summary of the Invention

[0007] This specification generally describes a reinforcement learning system for controlling an agent to perform multiple tasks while interacting with its environment. The system learns options, which consist of sequences of primitive actions performed by the agent under the control of an option policy neural network. The primitive actions are actions performed by the agent at time steps. In the implementation, the system discovers options useful for multiple different tasks through meta-learning, more specifically, by learning rewards used to train the option policy neural network while the agent interacts with its environment.

[0008] Therefore, in one aspect, a system for controlling an agent to perform multiple tasks while interacting with an environment is described. The system is configured such that each processing at multiple time steps includes input of an observation characterizing the current state of the environment to generate an output for selecting an action to be performed by the agent, such as one from a predetermined set of possible actions. A task reward is given in response to an action being received; for some or many time steps, the task reward may be zero.

[0009] The system may include a set of manager neural networks and option policy neural networks, each of which is used to select a sequence of actions to be performed by the agent according to a corresponding option policy. The manager neural network can be configured to process data—such as observations and identifying one of the tasks currently being performed by the agent (e.g., task embeddings)—based on the parameter values ​​of the manager neural network at each time step, for example, at each time step where it needs to select an option or raw action, to generate output for selecting a manager action from a set of manager actions. The set of manager actions may include possible (raw) actions that can be performed by the agent, such as each action in a predetermined set of possible actions. The set of manager actions may also include a set of option selection actions, each of which selects one from the option policy neural networks.

[0010] Each option policy neural network can be configured to process observations at each of a series of time steps (e.g., the time step when an option is selected), based on the option policy defined by the parameter values ​​of the option policy neural network, to generate an output for selecting the action to be performed by the agent—i.e., the original action. When the selected manager action is an option selection action, the option policy neural network selected by the manager action generates its output at successive time steps until the option termination criterion is met. When the selected manager action is one of the possible (original) actions that can be performed by the agent, the output for selecting the action is the selected manager action.

[0011] The system may include an array of option reward neural networks, each for each corresponding option policy neural network. Each option reward neural network can be configured to process observations at time steps based on the parameter values ​​of the option reward neural network to generate an option reward for the corresponding option policy neural network. Option rewards can be generated at each time step in which the corresponding option is selected.

[0012] In the implementation, the system is configured to use task rewards to train an ensemble of option reward neural networks and a manager neural network. The system can also be configured to train each option policy neural network using the option rewards of the respective option policy neural networks.

[0013] Various neural networks and training can be performed using any model-based or model-free reinforcement learning method, such as stochastic or deterministic policy gradient techniques, such as, by way of example only, actor-judge (AC) methods, trust region policy optimization (TRPO) methods, deep deterministic policy gradient (DDPG) methods; or function approximation techniques such as deep Q-networks (DQN) methods. Generally, in this specification, when the output is used in a policy neural network for selecting actions or options, the output can directly define the action or option, or it can define a score for each action or option in the set of actions or options, or it can assign a corresponding probability to each action or option, for example, by providing parameters of one or more probability distributions. The task reward can characterize the agent's progress toward completing the identified task.

[0014] In the implementation, the system is configured to train each option reward neural network using task rewards in a meta-gradient training technique. Specifically, in the meta-gradient training technique, under the control of the corresponding option policy neural network, the parameter values ​​of the option reward neural network are adjusted during the agent's interaction with the environment, i.e., based on the interaction, to optimize the reward from the environment. In this context, the reward can refer to a cumulative measure of the task rewards received by the agent, such as the sum of time-discounted rewards, where future rewards are discounted by a discount factor γ when calculating the reward.

[0015] For example, after the option selection action and over a series of time steps until a termination criterion is met, training may include updating the parameter values ​​of the manager neural network using task rewards, and updating the parameter values ​​of the selected option policy neural network using the option rewards of the corresponding option policy neural network. During this phase, the parameters of the option reward neural network may be fixed. Then, after the termination criterion is met, training may include updating the parameter values ​​of the option reward neural network used for the corresponding option policy neural network using task rewards.

[0016] Task rewards can define the objective function of the manager (reinforcement learning); option rewards can define the objective function of the option policy (reinforcement learning). The gradients of both the manager and option policy objective functions can each include the corresponding policy gradient. Updating the parameter values ​​of the manager neural network can include backpropagating the gradient of the manager objective function; similarly, updating the parameter values ​​of the option policy neural network can include backpropagating the gradient of the option policy objective function.

[0017] In the implementation, updating parameter values ​​may include generating a trajectory along with corresponding observations and task rewards. The trajectory comprises a sequence of one or more actions selected by a previously chosen option policy neural network. The task rewards obtained from the trajectory can be used to update the parameter values ​​of the option reward neural network used by the corresponding option policy neural network. This may involve backpropagating the gradient of the option reward objective function, such as the policy gradient objective function, based on the task rewards from the trajectory. The gradient can be backpropagated through the corresponding option policy neural network to the corresponding option reward neural network, for example, by the option policy neural network operating at each time step during its operation to select the (original) action.

[0018] In some implementations, the same form of estimated reward can be used, for example, in policy gradient updates for updating manager neural networks, option reward neural networks, and option termination neural networks (described later).

[0019] In some implementations, the policy gradients used to update the manager neural network, the option reward neural network, and the option termination neural network (described later) also depend on the value estimate. The same value estimate can be used to update each of these neural networks. In this context, the value estimate represents the value of the environment in the current state for successfully performing the identified task. For example, it can include an estimate of the reward for the identified task arising from the environment in the current state as represented by observations.

[0020] In an actor-judge-based implementation, the manager neural network can, for example, generate value estimates for other neural networks trained with task rewards based on data from the identification task. Each option policy neural network can generate its own value estimate. In the actor-judge-based implementation, the manager neural network and each option policy neural network can include neural networks having a policy head that provides corresponding action selection data and a value head that provides value estimates. One or more of the manager neural network, option policy neural network, and option reward neural network can be updated based on the estimated n-step reward, i.e., the reward calculated over n action selection time steps.

[0021] Some implementations of the system include an ensemble of option termination neural networks, each serving a corresponding option policy neural network. Each option termination neural network can be configured to process observations at each time step based on the parameter values ​​of the option reward neural network to generate an option termination value for the corresponding option policy neural network. For each option reward neural network, the option termination value can determine whether an option termination criterion is satisfied. The option reward neural network can generate the option termination value directly or indirectly; for example, it can parameterize a probability distribution from which the option termination value can be sampled.

[0022] The system can be configured to train an option-terminating neural network using task rewards in a meta-gradient training technique. This can be a technique in which, under the control of a corresponding option policy neural network, the parameter values ​​of the option-terminating neural network are adjusted based on the interaction between the agent and the environment to optimize the reward from the environment.

[0023] Therefore, the system can be configured to train an ensemble of option termination neural networks by updating the parameter values ​​of the option reward neural network used by the corresponding option policy neural network after the termination criterion for the corresponding option policy neural network is satisfied, using task rewards. This can include generating trajectories and corresponding observations and task rewards, the trajectories comprising sequences of one or more actions selected by the corresponding option policy neural network (i.e., selected by the option selection action). The parameter values ​​of the option termination neural network of the corresponding option policy neural network can then be updated using the task rewards from the trajectories. This can involve backpropagating the gradient of the option termination objective function based on the task rewards from the trajectories through the option policy neural network and through the corresponding option termination neural network.

[0024] In the implementation, the system is configured to train the manager neural network based on an estimated reward, including the expected task reward from the environment, when selecting a manager action according to the current parameter values ​​and the transition cost. The transition cost incentivizes the manager neural network to select an option policy neural network to encourage the discovery of temporarily expanded options. In the implementation, the transition cost is configured to decrease the task reward or return used to update the parameters of the manager neural network at each time step. It can include small negative values.

[0025] The set of option policy neural networks can be implemented as a set of option policy neural network heads sharing the body of the option policy neural network. Similarly, the set of option reward neural networks can include a set of option reward neural network heads sharing the body of the option reward neural network, and the set of option termination neural networks can include a set of option termination neural network heads sharing the body of the option termination neural network.

[0026] In the implementation, only the manager neural network receives and processes data that identifies the task being performed.

[0027] Before being fed into the aforementioned neural network, observations can be preprocessed. For example, when the observation is an image, preprocessing can use a stack of convolutional neural network layers, optionally followed by one or more fully connected neural network layers and / or one or more recurrent neural network layers, which map the observation to a more compact representation. When the observation includes additional features, such as proprioceptive features, these can be fed as input to one or more fully connected layers with convolutional stack outputs.

[0028] On the other hand, a method for a computer-implemented training system is described, which controls an agent to perform multiple tasks while interacting with an environment. The system is configured such that each processing step in multiple time steps includes input of an observation characterizing the current state of the environment, to generate an output for selecting an action to be performed by the agent, and to receive a task reward in response to that action.

[0029] The method may include obtaining observations and task identification data identifying one of the tasks currently being performed by an agent; and processing the observations and task identification data according to parameter values ​​of a manager neural network to generate output for selecting a manager action from a set of manager actions. The set of manager actions may include possible actions that can be performed by the agent and a set of option selection actions, each option selection action being used to select one of a set of option policy neural networks. When the selected manager action is one of the possible actions that can be performed by the agent, the method may include using the selected manager action to generate output for selecting an action to be performed by the agent. When the selected manager action is an option selection action, the method may include selecting an option policy neural network and, for a series of time steps, such as until an option termination criterion is met, processing the observations for that time step according to an option policy defined by the parameter values ​​of the selected option policy neural network to generate output for selecting an action to be performed by the agent.

[0030] The method may further include maintaining a set of option reward neural networks, each for a corresponding option policy neural network. During the processing of observations by the selected option policy neural network over a series of time steps, i.e., based on the processing of observations by the selected option policy neural network over a series of time steps, the method may include training the selected option policy neural network using option rewards generated by processing observations according to the parameter values ​​of the corresponding option reward neural network; and training the manager neural network using task rewards. Training the selected option policy neural network may include updating the parameter values ​​of the selected option policy neural network using option rewards from the corresponding option reward neural network. After the selected option policy neural network has processed observations over a series of time steps, for example, after a termination condition is met, the method may use task rewards to train the corresponding option reward neural network.

[0031] In a general sense, this method implements a call-and-return option running model, which trains the option policy neural network by rewarding the learned options in the inner training loop, and updates the option reward neural network in the outer training loop, which updates the parameters of the option reward neural network, and performs backpropagation in the presence of an option terminating neural network, for example, through updates in the inner loop.

[0032] The method may include maintaining a set of option termination neural networks, each for a corresponding option policy neural network. Each option termination neural network provides an option termination value based on its parameter values, determining whether an option termination criterion is satisfied for the corresponding option policy neural network. The parameter values ​​of the option termination neural networks (and option reward neural networks) may be fixed during the processing of observations by the selected option policy neural network over a series of time steps. After the selected option policy neural network has processed the observations over a series of time steps, the corresponding option termination neural network can be trained using task rewards.

[0033] Training an option reward neural network can include using a (partially) trained selected option policy neural network to generate new trajectories, i.e., in response to one or more observations, to select one or more further actions to perform in the environment in order to receive one or more task rewards. The task rewards received in response to further actions can be used to train the corresponding option reward neural network. A corresponding option termination neural network can be trained in a similar manner.

[0034] In some implementations of the above systems and methods, the task is multiple different agent control tasks, that is, tasks that include controlling the same mechanical agent to enable the agent to perform different objectives in the same real-world environment or a simulated version of the real-world environment.

[0035] The agent can be, for example, a robot or an autonomous or semi-autonomous vehicle. For instance, tasks can include navigating the agent to different locations in the environment, locating different objects, picking up different objects, or moving different objects to one or more specified locations. Options can define simpler subtasks related to the main task.

[0036] More specifically, in some applications, the environment is a real-world environment and the agent is a mechanical agent that interacts with the real-world environment. For example, the agent could be a robot that interacts with the environment to perform one or more specific tasks. As another example, the agent could be an autonomous or semi-autonomous land, air, or water vehicle that navigates within the environment. In these implementations, actions can be control signals that control the physical behavior of the robot or vehicle.

[0037] Generally, observation can include one or more of, for example, images, object position data, and sensor data that captures observations as the agent interacts with the environment, such as sensor data from image, distance, or position sensors, or from actuators. In the case of robots or other mechanical agents or vehicles, observation can similarly include one or more of position, linear or angular velocity, force, torque, or acceleration, and global or relative pose of one or more parts of the agent. Observation can be defined in one, two, or three dimensions and can be absolute and / or relative. For example, in the case of a robot, observation can include data characterizing the robot's current state, such as one or more of the following: joint position, joint velocity, joint force, torque, or acceleration, and global or relative pose of a part of the robot, such as an arm and / or an object held by the robot. Observation can also include, for example, sensed electronic signals, such as motor current or temperature signals; and / or image or video data, such as data from sensors of the agent or from sensors placed separately from the agent in the environment.

[0038] In these applications, actions can be control signals or higher-level control commands that control a robot—for example, the torque of a robot's joints; or control signals or higher-level control commands that control an autonomous or semi-autonomous land, air, or sea vehicle—for example, the torque of a vehicle's control surfaces or other control elements; or, for example, motor control data. In other words, actions can include, for example, the position, velocity, or force / torque / acceleration data of one or more joints of a robot or a component of another mechanical agent. Action data can include data on these actions and / or electronic control data, such as motor control data, or more generally, data for controlling one or more electronic devices in an environment, the control of which affects the observed state of the environment. For example, in the case of autonomous or semi-autonomous land, air, or sea vehicles, actions can include actions that control navigation—for example, steering—and motion—for example, braking and / or acceleration of the vehicle.

[0039] Rewards can include one or more rewards for approaching or reaching one or more target positions, one or more target poses, or one or more other target configurations. For example, for a robot, rewards may depend on joint orientation (angle) or velocity, end effector position, center of mass position, or position and / or orientation of body part groups. Costs (i.e., negative rewards) can be defined similarly, for example, depending on the forces applied when interacting with an object, energy usage, or the position of robot body parts.

[0040] The system can be used to train vehicles or robots to perform tasks such as autonomous tasks in warehousing, logistics, or factories, such as collecting, placing, or moving stored goods or portions of goods during their manufacturing; or the tasks performed may include package delivery control tasks. Actions may include actions related to steering or other directional control actions, and observations may include observations of the position or movement of other vehicles or robots. Robots or vehicles can be simulated and trained before being used in real-world environments.

[0041] In some applications, the agent can be a static or mobile software agent, i.e., a computer program configured to operate autonomously and / or perform tasks in conjunction with other software agents or humans. For example, the environment can be an integrated circuit routing environment, and the system can be configured to learn to perform routing tasks for routing interconnects of integrated circuits such as ASICs. The reward (or cost) can then depend on one or more routing metrics, such as interconnect resistance, capacitance, impedance, loss, speed or propagation delay, physical line parameters such as width, thickness, or geometry, and design rules. Observations can be observations of component locations and interconnects; actions can include, for example, component placement actions defining component locations or orientations and / or interconnect routing actions, such as interconnect selection and / or placement actions.

[0042] In some applications, the agent may be an electronic agent, and observations may include data from one or more sensors that monitor a part of a plant or service facility, such as current, voltage, power, temperature, and other sensors and / or electronic signals representing the function of electronic and / or mechanical items of the equipment. The agent may control actions in a real-world environment including equipment items, such as actions in facilities like data centers, server farms, or grid trunk power or water distribution systems, or actions in manufacturing plants or service facilities, or actions in power generation facilities such as solar or wind farms. Observations may then be related to the operation of the plant or facility; for example, they may include observations of the equipment's electricity or water usage, or observations of power generation or distribution control, or observations of resource usage or waste generation. Actions may include actions that control or impose operating conditions on equipment items in the plant / facility, and / or actions that result in changes to settings in the operation of the plant / facility, such as adjusting or turning on / off components of the plant / facility. Tasks may include multiple different control tasks, such as resource usage, like water or electricity control; environmental impact control, electricity or other power consumption control; heating, cooling, or temperature control; and general control of items within the facility.

[0043] In some applications, the environment may be a data packet communication network environment, and the agent may include a router to route data packets over the communication network. Actions may include data packet routing actions, and observations may include, for example, observations of a routing table that includes routing metrics such as path length, bandwidth, load, hop count, path cost, latency, maximum transmission unit (MTU), and reliability. Tasks may include packet routing tasks with rewards / costs to maximize or minimize one or more routing metrics.

[0044] In some other applications, the agent is a software agent that manages the distribution of computing tasks across computing resources, such as mobile devices and / or data center resources. In these implementations, observations may include observations of computing resources or internet-accessible resources, such as computing and / or memory capacity; and actions and associated tasks may include assigning computing tasks to specific computing resources. Rewards may depend on, for example, the utilization of computing resources, power consumption, bandwidth, and computing speed.

[0045] In some other applications, the environment is an internet or mobile communication environment and the agent is a software agent that manages personalized recommendations for the user. Observations may include previous actions taken by the user (characteristics representing the user's previous actions); the task may include the action of recommending items such as content items to the user. Rewards may include an estimated probability that the user will respond favorably to the recommended (content) item, or the number of recommendations the user receives (optionally over a time span); costs may depend on the suitability of one or more recommended items and the cost of the recommended items.

[0046] In some further applications, the environment is a cybersecurity environment. For example, observations may include data characterizing the state of a computer network or distributed computing system, and actions may be tasks to protect the computer system from cybersecurity attacks by, for example, one or more other agents.

[0047] The subject matter described in this specification can be implemented in certain embodiments to achieve one or more of the following advantages.

[0048] Some implementations of these techniques are able to discover options, i.e., original action sequences, which is useful for multiple tasks, including tasks with discrete actions and continuous control tasks. In contrast, manually designing action sequences useful for more than one task can be challenging. The described techniques are general, not limited to any specific application domain, and are capable of learning options without human supervision or domain-specific knowledge.

[0049] In model-based reinforcement learning, the described techniques improve learning speed and data efficiency because they provide updates applicable to longer action sequences. In model-free reinforcement learning, they enable agents to perform intentional actions, including action sequences that allow for better exploration.

[0050] Learning options useful for multiple tasks, rather than manually designing options, helps reduce the risk of options slowing down learning, as useless options increase the action space. The described technique also mitigates the risk of the manager neural network learning to perform a task using only raw actions. The implementation of this system is able to learn the timescale and behavior of options via option termination values.

[0051] Therefore, the system's implementation enables the learning of high-quality multi-task policies in an extremely stable and data-efficient manner. This makes the described technique particularly useful for tasks performed by real-world robots or other mechanical agents, as the risk of wear and tear and mechanical failure due to repeated interaction with the environment can be significantly reduced. Compared to traditional techniques, new and complex tasks can be learned using less data collected by the system more quickly, while consuming fewer computational resources. By discovering and learning useful options, knowledge can be effectively shared across multiple tasks, allowing the system to efficiently control the agent to perform all tasks.

[0052] Throughout the specification, for example, using a neural network to process an image refers to using a neural network to process the intensity values ​​associated with the pixels of the image.

[0053] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of this subject matter will become apparent from the specification, drawings, and claims. Attached Figure Description

[0054] Figure 1 A reinforcement learning neural network system showing learning options.

[0055] Figure 2 Showing the use Figure 1 This is an example process of using a system to perform tasks.

[0056] Figure 3 Showing the training Figure 1 An example of the system's internal loop process.

[0057] Figure 4 Showing the training Figure 1 An example of the outer loop process of a system.

[0058] In the accompanying drawings, the same reference numerals denote the same elements. Detailed Implementation

[0059] This specification describes a neural network-based reinforcement learning system and method that learns and discovers reusable options, i.e., original action sequences, across different tasks. Options learned in the context of a specific task can mitigate learning for another different task. The technique described herein enables the discovery of general and practical options, facilitating learning across multiple different tasks. This is achieved by learning what reward to provide to the option policy neural network that guides the options, and also, in implementation, learning when to terminate the options.

[0060] In implementations, tasks are agent control tasks, typically involving controlling the same mechanical agent, such as a robot, to accomplish different objectives within the same real-world environment. For example, tasks might require the agent to pick up one or more objects in the environment, move them to a specific location, arrange or manipulate them, or they might require the agent to navigate within the environment. In some implementations, the system can be trained on a simulated version of a real-world environment and then used in the real-world environment, where it may or may not be further trained. Tasks can be specified by their objectives. Generally, only one task can be performed at a time.

[0061] Figure 1 A reinforcement learning neural network system 100 is shown, which can be implemented as one or more computer programs on one or more computers at one or more locations.

[0062] System 100 has an output that provides control signals to agent 102 to control the actions of the agent in environment 104. Typically, when the agent performs an action, the environment transitions from the current state to a new state, and by repeatedly causing agent 102 to perform actions in environment 104, system 100 can control agent 102 to complete a specified task. Agent actions can be discrete actions selected from a set of possible actions, or continuous actions, i.e., actions defined by continuous variables such as motor torque.

[0063] System 100 also has one or more inputs to receive data from the environment. This includes data from one or more sensors that provide environmental state s characterizing a given time t. t Information from the environment. This data is referred to as observation 106. Data from the environment also includes task rewards r. Typically, task rewards can be scalar values ​​characterizing the agent's progress toward a task objective. Task rewards can be received as the task progresses or only at the end of the task, for example, to indicate successful completion of the task (or otherwise). Rewards can be positive or negative; that is, as used herein, "reward" includes costs.

[0064] Typically, system 100 receives observations of the environmental state and processes these observations to provide control signals to the agent, enabling the agent to perform tasks. At least during learning, system 100 uses rewards to learn what control signals to provide to perform tasks.

[0065] System 100 is capable of learning to perform multiple different tasks, and in some implementations, includes receiving input 110 that identifies the task to be performed. The data identifying the task to be performed may come from an external source, such as a user, or it may be autonomously selected, for example, by a larger system including system 100.

[0066] The techniques described in this paper do not rely on any particular way of identifying tasks. In some implementations, a task may be identified by its objective g, such as a representation of the final state to be achieved by the task. In some other implementations, a task may simply be identified by a label (and the specific reward received may vary depending on the task).

[0067] System 100 includes a manager neural network 120, an option policy neural network set 130, an option reward neural network set 140, and an optional option termination neural network set 150. The functions of the neural networks are described below.

[0068] At least during training, system 100 also includes a training engine 160. The training engine operates to implement the training process, for example, as described below, to update the parameters of the neural network.

[0069] The training engine uses reinforcement learning to train the manager neural network 120 and the option policy neural network 130 based on their corresponding rewards. The training engine also trains the option reward neural network 140 and the option termination neural network 150. This uses the meta-gradient method because variations in the parameters of these neural networks indirectly affect the performance of the manager neural network 120 and the option policy neural network 130. The meta-gradient method evaluates the performance of the manager neural network and the option policy neural network on multiple tasks, and uses this evaluation to train the option reward neural network and the option termination neural network. In this way, the option reward neural network and the option termination neural network discover useful option reward and termination criteria across multiple tasks.

[0070] The manager neural network 120 is configured to receive and process observation 106 and data 110 identifying the currently performed task based on the parameter values ​​of the manager neural network, in order to generate the manager neural network output (π). MThe manager neural network output is configured to select a manager action from a set of manager actions, which includes possible discrete or continuous actions that can be performed by the agent, and an option selection action set, each of which selects one from an option policy neural network. In some implementations, the manager neural network output defines one or more parameters of the probability distribution of possible manager actions; that is, it defines a random action selection policy. The manager action can then be selected, for example, by sampling from the probability distribution or by selecting the action with the highest probability. The training engine 160 is configured to train the manager neural network to maximize task rewards.

[0071] Each option policy neural network 130 is configured to receive and process observations 106 based on the parameter values ​​of the option policy neural network to generate the option policy neural network output. The output is configured to select possible actions that can be performed by the agent. In some implementations, the option policy network output defines one or more parameters of the probability distribution over possible discrete or continuous agent actions; that is, it defines the random action selection policy. When selected by the manager neural network 120, the option policy neural network 130 selects actions that will be performed by the agent over a series of time steps until the option termination criterion is met. For example, actions can be selected by sampling from the probability distribution or by selecting the action with the highest probability. The training engine 160 is configured to train each option policy neural network to maximize the option reward generated by the corresponding option reward neural network 140.

[0072] Each option reward neural network 140 is configured to receive and process observations 106 based on parameter values ​​of the option reward neural network to generate option rewards for its corresponding option policy neural network 130. In the implementation, the option reward is a scalar value.

[0073] Each option termination neural network 150 is configured to receive and process observations 106 based on parameter values ​​of the option termination neural network to generate option termination values ​​for its corresponding option policy neural network 130. The option termination value determines whether the option termination criterion is satisfied for its corresponding option policy neural network 130. In the implementation, the option termination value is a scalar value, such as a value randomly sampled from a classification distribution such as {0, 1}, where one value indicates termination; or a scalar in a range such as [0, 1] that indicates the probability of termination.

[0074] The options correspond to the learning sub-objectives of the task. The training process described below enables the achievement of these sub-objectives, or "options" useful for learning multiple different tasks. Each option corresponds to one in the option-policy neural network. As just an example, an implementation can have 1-20 options.

[0075] like Figure 1 As shown, each of the option policy neural network 130, option reward neural network 140, and option termination neural network 150 can include different neural network heads on a shared neural network body. In some implementations, each neural network body can have essentially the same architecture. However, in this implementation, only the manager neural network 120 receives the data 110 identifying the task—the task objective; the other neural networks do not receive the task objective. This helps in discovering task-independent options that are useful for multiple tasks.

[0076] exist Figure 1 Not shown, but observations 106 may be encoded and / or different types of observation features, such as image features and proprioceptive features, may be combined before being processed by one or more of the manager neural network 120, option policy neural network 130, option reward neural network 140, or option termination neural network 150. By way of example only, for the manager neural network 120, data 110 identifying the task currently being performed may be combined with observations including images as an additional channel of the encoded version of the images. Typically, the manager neural network 120, option policy neural network 130, option reward neural network 140, and option termination neural network 150 can have any architecture; for example, they may include one or more feedforward, convolutional, recursive, batch normalized, or layer normalized neural network layers.

[0077] Figure 2 An example process is shown for performing a task using a reinforcement learning neural network system 100. Figure 2 The process can be performed by a system of one or more computers located in one or more locations during and / or after training.

[0078] In step 200, the system receives and processes the observation s of the environmental state at the current time step time t. t 106 and the data g 110 identifying the task (target), and in step 202, this is processed using a manager neural network 120 to generate, for example, a manager neural network output π defining a probability distribution. M .

[0079] The process then uses the output of the manager neural network to select a manager action, which can be either the agent's action (i.e., the original action) or an option selection action from the option policy neural network (step 204). Selecting a manager action may include selecting from the probability distribution π. M Mid-sampling. If the manager action is an option selection action, the selected option policy neural network is used to select an action for the agent (step 206).

[0080] The process then generates output to cause the agent to perform the selected action, such as outputting a control signal (step 208). As a result, the next observation of the environment is received, along with an optional task reward.

[0081] The output for selecting the action to be performed can be generated by the manager neural network 120 or the selected option policy neural network. If the manager neural network outputs an option selection action, the process continues to use the selected option policy neural network to receive and process observations and generate outputs for selecting the surrogate action until the option termination criterion is met (step 210). The process can loop back, processing the most recent observations again with the manager neural network.

[0082] In implementation, when the option policy neural network outputs an option termination value, such as a classification value, indicating that the option should be terminated, the corresponding option termination neural network output is used to indicate the option termination value. The option termination criterion is satisfied when the termination probability is greater than a threshold scalar value. When the option termination criterion is satisfied, a new manager neural network output is generated to select a new option or the original action.

[0083] An example training process for the reinforcement learning neural network system 100 to perform a task has an inner loop and an outer loop. The inner loop updates the parameters of the manager neural network 120 and the option policy neural network 130. The outer loop updates the parameters of the option reward neural network 140 and the option termination neural network 150. At the beginning of the training process, the parameters of the neural networks of system 100 are initialized, for example, randomly.

[0084] Figure 3 An example of an inner loop process is shown; this process can be executed by one or more computers located in one or more locations. The process begins by selecting a task, for example by randomly sampling from possible tasks (step 300).

[0085] The environmental state observation 106 is then obtained and processed by the manager neural network 120 along with the task identification data 110 to generate the manager neural network output. In this example, as previously described, sampling is performed to select either a proxy action or an "option," i.e., one of the option policy neural networks 130 (step 302). Again, as previously described, if the manager neural network selects a proxy action, that action is executed; if an option is selected, the proxy action is selected by the chosen option policy neural network. The manager neural network can select multiple proxy actions before selecting an option.

[0086] If an option is selected, observation 106 is processed by each of the selected option policy neural network, the option reward neural network corresponding to the selected option policy neural network, and the option termination neural network corresponding to the selected option policy neural network. In such a way... Figure 1 In the implementation shown, each neural network 130-150 includes multiple neural network heads on a shared body, and all neural networks can process observations.

[0087] The selected option policy neural network is used to select agent actions (step 304). Then, the system generates output to control the agent to perform the selected actions (step 306), the environment transitions to a new state, observations of the new state are obtained, and task rewards can be received from the environment.

[0088] Options are followed until termination, indicated, for example, by the option termination value from the corresponding option terminating the neural network, such as β. o =1. Therefore, steps 304 and 306 are repeated. In this way, the n-step trajectory of the transition is obtained by the inner loop (step 308).

[0089] The transition trajectory comprises transitions for each of n time-step sequences (where n can be 1). Each transition includes data representing the observation, the selected agent action, the task reward (if any), and the option reward from the option reward neural network corresponding to the selected option policy neural network. The transition may also include option termination values ​​from the corresponding option termination neural network. Optionally, the trajectory may include data identifying the task and data indicating the selected option.

[0090] The process then uses any reinforcement learning technique, employing an n-step trajectory to train the manager neural network 120 and the selected option policy neural network (step 310). Specifically, the manager neural network is trained using task rewards (or multiple rewards), and the selected option policy neural network is trained using option rewards from the option reward neural network corresponding to the selected option policy neural network. In the implementation, the manager neural network 120 is trained using only transitions, where the manager neural network output is used to select an option (or agent action).

[0091] Training each neural network may include backpropagating gradients of a reinforcement learning objective function, which depends on the rewards (or more) of that neural network, through the network to update its parameters. Specifically, the manager neural network may be trained to maximize the time-discounted sum of task rewards, i.e., the reward from the environment, and the chosen option policy neural network may be trained to maximize the time-discounted sum of option rewards. In some implementations, the option termination value from the corresponding option termination neural network may be used as a time-discount factor (e.g., when the option termination value is not a categorical variable).

[0092] In one example implementation, an actor-judge reinforcement learning approach is used. In this implementation, the manager neural network output and the output of each option policy neural network include outputs defining the corresponding state values, such as outputs from individual heads. The state value is an estimate of the environment's current state when the option policy neural network is used to select an action, representing the estimated reward of the task from the current state as characterized by the current observation. Then, at each time t, the parameter values ​​θ of the option policy neural network are... o The update can be determined as follows:

[0093]

[0094] Where π o (a t |s t ) is the environmental state s at a given time t. t In the case of selecting the action a at time t t The option policy neural network output, such as action a t The probability value or fraction; v o (s t ) is state s t The option strategy neural network state value output; κ o It is a hyperparameter that weights the state value update relative to the policy update (usually first order, such as 0.5); It refers to the gradient used for the parameter values ​​of the option policy neural network; α is the target value or objective of the option policy neural network output at time t for selecting actions, i.e., the option policy objective function; and α is the learning rate.

[0095] Similarly, when the option is sampled, at time t o For the parameter values ​​θ of the manager neural network M The update can be determined as follows:

[0096]

[0097] in Given a task identifier g and time t o environmental conditions In the case of selecting time t o The manager of the options neural network outputs, such as the probability value or score of option o; It is a state The manager's neural network state value output; k M It is a hyperparameter that weights the state value update relative to the policy update (usually first order, such as 0.5); It is the gradient applied relative to the parameter values ​​of the manager neural network; At time t o The target value or objective of the manager neural network output used to select options is the manager objective function; and α is the learning rate mentioned above or another learning rate.

[0098] The target value or target output of the option policy neural network and the target value or target output of the manager neural network. Each of these can be a corresponding n-step reward. When the option termination neural network indicates that the option should end, the n-step reward can be evaluated on the n-step trajectory obtained by the inner loop, i.e., at time t. o The trajectory that starts and ends after n time steps. The n-step reward of the option policy neural network's output is evaluated using the option reward from the option reward neural network used for the option policy neural network. For example, if the option policy neural network has an option termination value at time t... Used as a time discount factor:

[0099]

[0100] The n-step reward of the option policy neural network output is evaluated using task rewards from the environment. For example:

[0101]

[0102] Here, γ is the time discount factor; and -c represents the transformation cost, which is added to the reward of each step updated by the manager neural network when the option terminates. The transformation cost is a hyperparameter that encourages the manager neural network to choose temporarily expanded options, thereby aiding their discovery. As an example only, c can have a value comparable to but smaller than the reward size.

[0103] Figure 4 An example of an outer loop process is shown; similarly, this process can be executed by one or more computers located in one or more locations. This begins by executing an inner loop process one or more times (step 400). For example, the inner loop can be executed up to 10 times before the subsequent outer loop step.

[0104] Once the inner loop has been executed, optionally, a new task can be selected; this can but does not need to be a different task (or multiple tasks) from the inner loop (step 402). The task is executed as described above, i.e., using a manager neural network to select an option (or surrogate action), and then using a policy neural network to select a surrogate action using the selected option, until the option is terminated. In this case, a new task is executed for, for example, m steps, to obtain a new m-step "verification" trajectory (step 404).

[0105] The process then uses validation trajectories, specifically task rewards from the validation trajectories, to train an option reward neural network and an option termination neural network for the selected options (or more) (step 406). This may include backpropagating the gradients of the option reward objective function, which depends on the task reward, through the selected option policy neural network and into the corresponding option reward neural network to update the parameters of the option reward neural network. This may further include backpropagating the gradients of the option termination objective function, which depends on the task reward, through the selected option policy neural network and into the corresponding option termination neural network to update the parameters of the option termination neural network.

[0106] The option reward objective function and the option termination objective function can be any type of reinforcement learning objective function that depends on the task reward or reward. The same objective function can be used for each of the option reward neural networks and each of the option termination neural networks. For example, if using an actor-judge reinforcement learning method, the parameter values ​​for the option reward neural network... and the parameter values ​​for terminating the neural network. The update can be determined as follows:

[0107]

[0108] and

[0109]

[0110] in and It is the gradient applied relative to the parameter values ​​of the reward neural network and the termination neural network for the corresponding options, and α η It refers to the learning rate. For two updates, the learning rate can be, but does not need to be, the same.

[0111] In some implementations, the equations determining the updates to the parameter values ​​of the manager neural network, option policy neural network, option reward neural network, and option termination neural network may include entropy regularization terms to suppress premature convergence during training, such as gradients of the parameter values ​​relative to the entropy reward, such as those dependent on -logπ(a|s). t The gradient of the subterms of ).

[0112] The above gradient and These can be called meta-gradients because they update meta-parameters, which are the parameters of the corresponding option reward neural network and option termination neural network, which generate option reward and option termination values ​​that control the action of the selected option policy neural network.

[0113] In the implementation, the option reward neural network and the option termination neural network can be trained to maximize the time discount sum of the task rewards, i.e., the rewards from the environment, such as the rewards used to train the manager neural network in the inner loop.

[0114] Conceptually, validation trajectories are used to evaluate the option policies learned in the inner loop and update the option rewards and termination values ​​that guide the learning of these policies. Therefore, in the inner loop, the parameters of the option reward neural network and the option termination neural network are fixed, while the parameters of the manager neural network and the option policy neural network are updated. Then, in the outer loop, the meta-gradient is determined, which depends on the change in the outer loop objective relative to the parameters of the option reward neural network and the option termination neural network, and therefore on the effect of the updates to the manager neural network and the option policy neural network on the outer loop objective. The updates to the parameters of the option reward neural network and the option termination neural network can be computed by backpropagation through the inner loop updates, for example, by storing a copy of each inner loop update in the parameters of the selected option policy neural network. In practice, this can be achieved by backpropagating the parameters of the option reward neural network.

[0115] The outer loop is executed multiple times until training is complete, for example, until the maximum number of training time steps is reached. Although the outer loop can update only the parameters of the option reward neural network and the option termination neural network corresponding to the selected option, over time, the parameters of all option reward neural networks and option termination neural networks are updated.

[0116] The implementation of the aforementioned system and method enables the learning of different objectives for each option policy neural network, such as objectives parameterized by option reward neural networks and option termination neural networks. In the sense of discovering objectives useful for multiple tasks, the options are non-entangled. This can accelerate the learning of new tasks and contribute to obtaining higher rewards during training.

[0117] For example, in some implementations, the parameters of the option policy neural network, option reward neural network, and option termination neural network can be fixed after some training. The manager neural network can then be trained to perform new (unseen) tasks, facilitated by the learned options. At this stage, the transformation cost can be omitted when training the manager neural network.

[0118] Generally, once the system is trained, it can be used to control an agent to perform one or more learned tasks in an environment. Alternatively, the parameters of the trained option policy neural network, option reward neural network, and option termination neural network can be provided to a second, similar system, such that the manager neural network of the second system can be trained to perform one or more tasks using previously learned options, i.e., using the option policy neural network, option reward neural network, and option termination neural network with the provided parameters.

[0119] In systems discussed here that utilize data that may include personal information, such data may be processed in one or more ways, such as aggregation and anonymization, before being stored or used, making it impossible to identify such personal information from the stored or used data. Furthermore, the use of such information may make it impossible to identify personally identifiable information from the output of the system using such information.

[0120] This specification uses the term "configuration" in conjunction with system and computer program components. For a system of one or more computers configured to perform specific operations or actions, this means that software, firmware, hardware, or a combination thereof are installed on the system, which, in operation, causes the system to perform those operations or actions. One or more computer programs configured to perform specific operations or actions means that one or more programs include instructions that, when executed by a data processing device, cause that device to perform those operations or actions.

[0121] Embodiments of the subject matter and functional operation described in this specification can be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware, including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium, for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or combinations thereof. Alternatively or additionally, the program instructions can be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus.

[0122] The term "data processing device" refers to data processing hardware and includes all kinds of devices, apparatuses, and machines for processing data, such as programmable processors, computers, or multiple processors or computers. The device may also be or further include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0123] A computer program can also be called or described as a program, software, software application, app, module, software module, script, or code. It can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program can, but does not need to, correspond to a file in a file system. A program can be stored as a portion of a file that holds other programs or data, for example, as one or more scripts stored in a markup language document, as a single file dedicated to the program in question, or as multiple collaborative files, for example, as a file storing one or more modules, subroutines, or code portions. A computer program can be deployed to execute on a single computer or on multiple computers located in one place or distributed across multiple locations and interconnected via a data communication network.

[0124] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in others, multiple engines can be installed and run on the same or more computers.

[0125] The processes and logic flows described in this specification can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by manipulating input data and generating outputs. These processes and logic flows can also be executed by dedicated logic circuits, such as FPGAs or ASICs, or by a combination of dedicated logic circuits and one or more programmable computers.

[0126] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for executing instructions and one or more storage devices for storing instructions and data. The central processing unit and memory can be supplemented or incorporated therein by dedicated logic circuitry. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, to receive data from or transfer data to, or both. However, a computer does not need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name just a few examples.

[0127] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor storage devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and DVD-ROM disks.

[0128] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, that the user can use to provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from a web browser. Additionally, the computer can interact with the user by delivering text messages or other forms of messages to personal devices, such as smartphones running messaging applications, and receiving response messages from the user.

[0129] The data processing apparatus used to implement machine learning models can also include, for example, dedicated hardware accelerator units for processing machine learning training or production, i.e., the general and computationally intensive parts of inference and workloads.

[0130] Machine learning models can be implemented and deployed using machine learning frameworks, such as the TensorFlow framework.

[0131] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes, for example, a backend component acting as a data server, or a middleware component including, for example, an application server, or a frontend component including, for example, a client computer with a graphical user interface, a web browser, or an application through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0132] A computing system can include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship arises from computer programs running on the required computers, and they have a client-server relationship. In some embodiments, the server sends data, such as HTML pages, to a user device, for example, to display data to a user interacting with the device acting as a client and to receive user input from that user. It is possible to receive data generated on the user device, such as the results of user interactions, from the device at the server.

[0133] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features characteristic of particular embodiments of a particular invention. Some features described in this specification in the context of independent embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations, and even initially claimed in this way, one or more features from a claimed combination can be removed from that combination in some cases, and the claimed combination may be for sub-combinations or variations thereof.

[0134] Similarly, although the operations are described in a specific order in the accompanying drawings and recited in a specific order in the claims, this should not be construed as requiring these operations to be performed in the specific order or sequence shown, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0135] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions described in the claims can be performed in different orders and still achieve the desired result. As an example, the processes depicted in the figures do not necessarily require the specific or sequential order shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented system for controlling an agent to perform multiple tasks while interacting with an environment, wherein, The system is configured to, at each of a plurality of time steps, process input including observations characterizing the current state of the environment to generate output for selecting an action to be performed by an agent, and receive a task reward in response to said action, the system comprising: A set of manager neural networks and option policy neural networks, each option policy neural network being used to select the sequence of actions to be performed by the agent according to the corresponding option policy; The manager neural network is configured such that, at time steps: Based on the parameter values ​​of the manager neural network, data is processed to observe and identify one of the tasks currently being performed by the agent, in order to generate output for selecting a manager action from the set of manager actions, wherein the set of manager actions includes possible actions that can be performed by the agent and a set of option selection actions, each of which selects one of the options policy neural networks; Each option policy neural network is configured such that, at each step in a series of time steps: Based on the option policy defined by the parameter values ​​of the option policy neural network, the observations at this time step are processed to generate an output for selecting the action to be performed by the agent; Specifically, when the selected manager action is an option selection action, the option policy neural network selected by the manager action generates an output for the selection action over a series of time steps until the option termination criterion is met; and when the selected manager action is one of the possible actions that can be executed by the agent, the output for the selection action is the selected manager action; and An ensemble of option reward neural networks, each option reward neural network serving a corresponding option policy neural network, each option reward neural network being configured to, at time steps: The observations are processed based on the parameter values ​​of the option reward neural network to generate the option reward for the corresponding option policy neural network; The system is configured to train an option reward neural network set and a manager neural network using task rewards, and to train each of the option policy neural networks using the option rewards of the corresponding option policy neural networks, wherein: i) The environment is a real-world environment or a simulated version of a real-world environment; the agent is a mechanical proxy that interacts with the real-world environment or a simulated version of the real-world environment; and the observation is one or more of images, object location data, and sensor data that captures the observation while the agent interacts with the environment; the action is a control signal used to control the physical behavior of the agent; or ii) The environment is a real-world integrated circuit routing environment or a simulated version of a real-world integrated circuit routing environment; the proxy is a static or software proxy; the observations are component locations and interconnects; and the actions are component placement actions and / or interconnect routing actions; or iii) The environment is a real-world factory or service facility or a simulated version of a real-world factory or service facility; the agent is an electronic agent; observation is data from one or more sensors monitoring the factory or service facility; and actions include actions that control or impose operating conditions on equipment items in the factory or facility and / or actions that cause changes in settings during the operation of the factory / facility; or iv) The environment is a real-world packet communication network environment or a simulated version of a real-world packet communication network environment, the agent is a router to route data packets through the communication network, the observation is an observation of the routing table, and the actions include data packet routing actions; or v) The environment is a real-world mobile device and / or data center, or a simulated version of a real-world mobile device and / or data center; the agent is a software agent; the observation is an observation of computing resources or internet-accessible resources; and the action includes assigning computing tasks to specific computing resources; or vi) The environment is a real-world internet or mobile communication environment or a simulated version of a real-world internet or mobile communication environment; the agent is a software agent that manages personalized recommendations for the user, observing actions including those previously taken by the user, and those actions being actions related to the recommended content items; or vii) The environment is a real-world cybersecurity environment or a simulated version of a real-world cybersecurity environment. Observations include data characterizing the state of a computer network or distributed computing system. The agent is a security agent, and the action is a task to protect the computer system from cybersecurity attacks by one or more other agents.

2. The system according to claim 1, wherein, The system is configured to use task rewards in a meta-gradient training technique to train each option reward neural network to optimize the reward from the environment. In the meta-gradient training technique, the parameter values ​​of the option reward neural network are adjusted based on the agent's interaction with the environment under the control of the corresponding option policy neural network.

3. The system according to claim 1 or 2, wherein, The system is configured to train an ensemble of option reward neural networks and a manager neural network using task rewards, and to train each of the option policy neural networks using the option rewards of the respective option policy neural networks: After the action is selected, and for a series of time steps until the termination criterion is met: The parameter values ​​of the task reward update manager neural network are used, and the parameter values ​​of the corresponding option policy neural network selected by the option selection action are updated using the option reward of the corresponding option policy neural network; then After the termination criteria are met: The task rewards are used to update the parameter values ​​of the option reward neural network used in the corresponding option policy neural network.

4. The system according to claim 3, wherein, The parameter values ​​of the option reward neural network used to update the corresponding option policy neural network using task rewards include: Generate trajectories and corresponding observations and task rewards. The trajectories consist of sequences of one or more actions selected by a corresponding option-selection policy neural network based on the option selection action; and The task rewards from the trajectory are used to update the parameter values ​​of the option reward neural network used in the corresponding option policy neural network.

5. The system according to claim 4, wherein, Updating the parameter values ​​of the option reward neural network used for the corresponding option policy neural network using the task reward from the trajectory includes: backpropagating the gradient of the option reward objective function based on the task reward from the trajectory through the corresponding option policy neural network and through the option reward neural network used for the corresponding option policy neural network.

6. The system according to claim 1 or 2, wherein, The update manager neural network parameter values, the corresponding option policy neural network parameter values, and the option reward neural network parameter values, including one or more of these, are based on... Update the report step by step.

7. The system according to claim 1 or 2, wherein, The parameter values ​​of the task reward update manager neural network include the gradient of the backpropagation manager objective function, wherein the parameter values ​​of the update corresponding option policy neural network include the gradient of the backpropagation option policy objective function, and wherein the manager objective function and the option policy objective function each include the corresponding reinforcement learning objective function.

8. The system according to claim 7, wherein, The gradient of the manager's objective function and the gradient of the option policy objective function include the corresponding policy gradient.

9. The system of claim 1 or 2, further comprising an option termination neural network set, each option termination neural network being used for a corresponding option policy neural network, each option termination neural network being configured to, at each time step: The observations are processed based on the parameter values ​​of the option reward neural network to generate the option termination value of the corresponding option policy neural network. in, For each option, the reward neural network is used, and the option termination value determines whether the option termination criterion is met.

10. The system according to claim 9, wherein, The system is configured to use task rewards in a meta-gradient training technique to train an option-terminating neural network to optimize rewards from the environment, in which the parameter values ​​of the option-terminating neural network are adjusted based on the agent-environment interaction under the control of a corresponding option policy neural network.

11. The system according to claim 1 or 2, wherein, The system is configured to train an option termination neural network set after the termination criterion is met for the corresponding option policy neural network by: The task rewards are used to update the parameter values ​​of the option termination neural network used for the corresponding option policy neural network.

12. The system according to claim 11, wherein, Using task rewards to update the parameter values ​​of the option termination neural network used for the corresponding option policy neural network includes: Generate trajectories and corresponding observations and task rewards. The trajectories consist of sequences of one or more actions selected by a corresponding option-selection policy neural network based on the option selection action; and The task rewards from the trajectory are used to update the parameter values ​​of the option termination neural network used for the corresponding option policy neural network.

13. The system according to claim 12, wherein, Updating the parameter values ​​of the option termination neural network for the corresponding option policy neural network using task rewards from the trajectory includes: backpropagating the gradient of the option termination objective function based on the task rewards from the trajectory through the corresponding option policy neural network and through the option termination neural network for the corresponding option policy neural network.

14. The system according to claim 1 or 2, wherein, The system is configured to train a manager neural network based on an estimated reward and a transformation to the original value. The estimated reward includes the expected task reward from the environment when a manager action is selected based on the current parameter values ​​of the manager neural network.

15. The system according to claim 14, wherein, The conversion cost is configured to reduce the task reward or incentive used to update the parameter values ​​of the manager neural network.

16. The system according to claim 1 or 2, wherein, The set of option policy neural networks includes a set of option policy neural network heads on a shared option policy neural network body, and the set of option reward neural networks includes a set of option reward neural network heads on a shared option reward neural network body.

17. A computer-implemented method for training a system for controlling an agent to perform multiple tasks while interacting with an environment, wherein, The system is configured such that each process in multiple time steps includes input of observations characterizing the current state of the environment to generate output for selecting an action to be performed by an agent, and receives a task reward in response to the action, the method comprising: Obtain and identify task identification data for one of the tasks currently being executed by the agent; The observation and task identification data are processed based on the parameter values ​​of the manager neural network to generate output for selecting manager actions from the set of manager actions. The manager action set includes possible actions that can be performed by the agent and a set of option selection actions, each of which is used to select one of the option policy neural network sets. When the selected manager action is one of the possible actions that can be performed by the agent, the selected manager action is used to generate output for selecting the action that will be performed by the agent, and When the selected manager action is an option selection action, the option policy neural network is selected, and for a series of time steps until the option termination criterion is met, the observations of that time step are processed according to the option policy defined by the parameter values ​​of the selected option policy neural network to generate an output for selecting the action to be performed by the agent. Maintain an ensemble of option reward neural networks, where each option reward neural network is used in conjunction with a corresponding option policy neural network. Based on the observations processed by the selected option policy neural network at a series of time steps. The selected option policy neural network is trained using option rewards generated by processing observations based on the parameter values ​​of the corresponding option reward neural network. Use task rewards to train the manager neural network; and After the selected option policy neural network processes observations over a series of time steps, the corresponding option reward neural network is trained using task rewards, where: i) The environment is a real-world environment or a simulated version of a real-world environment; the agent is a mechanical proxy that interacts with the real-world environment or a simulated version of the real-world environment; and the observation is one or more of images, object location data, and sensor data that captures the observation while the agent interacts with the environment; the action is a control signal used to control the physical behavior of the agent; or ii) The environment is a real-world integrated circuit routing environment or a simulated version of a real-world integrated circuit routing environment; the proxy is a static or software proxy; the observations are component locations and interconnects; and the actions are component placement actions and / or interconnect routing actions; or iii) The environment is a real-world factory or service facility or a simulated version of a real-world factory or service facility; the agent is an electronic agent; observation is data from one or more sensors monitoring the factory or service facility; and actions include actions that control or impose operating conditions on equipment items in the factory or facility and / or actions that cause changes in settings during the operation of the factory / facility; or iv) The environment is a real-world packet communication network environment or a simulated version of a real-world packet communication network environment, the agent is a router to route data packets through the communication network, the observation is an observation of the routing table, and the actions include data packet routing actions; or v) The environment is a real-world mobile device and / or data center, or a simulated version of a real-world mobile device and / or data center; the agent is a software agent; the observation is an observation of computing resources or internet-accessible resources; and the action includes assigning computing tasks to specific computing resources; or vi) The environment is a real-world internet or mobile communication environment or a simulated version of a real-world internet or mobile communication environment; the agent is a software agent that manages personalized recommendations for the user, observing actions including those previously taken by the user, and those actions being actions related to the recommended content items; or vii) The environment is a real-world cybersecurity environment or a simulated version of a real-world cybersecurity environment. Observations include data characterizing the state of a computer network or distributed computing system. The agent is a security agent, and the action is a task to protect the computer system from cybersecurity attacks by one or more other agents.

18. The method according to claim 17, wherein, Training the corresponding option reward neural network includes: after training, using the selected option policy neural network to select one or more further actions to be performed in the environment in response to one or more observations to receive one or more task rewards, and using the task rewards received in response to the further actions to train the corresponding option reward neural network.

19. The method according to claim 17 or 18, further comprising: A set of option termination neural networks is maintained, with each option termination neural network serving a corresponding option policy neural network. Each option termination neural network provides an option termination value based on its parameter values. The option termination value determines whether the option termination criterion is met for the corresponding option policy neural network. During the observations processed by the selected option policy neural network at a series of time steps, the parameter values ​​of the option-terminated neural network are fixed, and After the selected option policy neural network processes observations over a series of time steps, the corresponding option termination neural network is trained using task rewards.

20. The method according to claim 19, wherein, Using task rewards to train the corresponding option termination neural network includes: using the trained selected option policy neural network to select one or more further actions to be performed in the environment in response to one or more observations to receive one or more task rewards, and using the task rewards received in response to the further actions to train the corresponding option termination neural network.

21. Computer-readable instructions or one or more computer storage media storing computer-readable instructions, which, when executed by one or more computers, cause the one or more computers to implement the system of any one of claims 1-16 or the method of any one of claims 17-20.

22. A system comprising one or more computers and one or more storage devices storing instructions, which, when executed by the one or more computers, cause the one or more computers to perform a corresponding operation of the method of any one of claims 17-20.

Images