Multi-objective reinforcement learning using target-specific action-value functions

Abstract

A method is provided for training a neural network system via reinforcement learning, the neural network system being configured to receive input observations representing the state of an environment interacting with by an agent and to select and output actions according to a policy aimed at satisfying multiple objectives. The method includes obtaining a set of one or more trajectories. Each trajectory includes the state of the environment, an action applied to the environment by the agent in response to the state according to a prior policy, and a set of rewards for that action, each reward being associated with a corresponding objective among a plurality of objectives. The method further includes determining an action-value function for each of the plurality of objectives based on the set of one or more trajectories. Each action-value function determines an action value representing an estimated reward for the corresponding objective, which will be generated by the agent performing a given action in response to a given state according to a prior policy. The method further includes determining an updated policy based on a combination of action-value functions for the plurality of objectives.

Description

Background Technology

[0001] This manual relates to reinforcement learning.

[0002] In a reinforcement learning system, an agent interacts with the environment by performing actions that are selected by the reinforcement learning system in response to receiving observations that represent the current state of the environment.

[0003] Some reinforcement learning systems respond to a given observation by selecting the action an agent will perform based on the output of a neural network.

[0004] A neural network is a machine learning model that uses one or more layers of non-linear units to predict an output in response to a received input. Some neural networks are deep neural networks that include one or more hidden layers in addition to an output layer. In the network, the output of each hidden layer is used as the input to the next layer (i.e., the next hidden layer or output layer). Each layer of the network produces an output from the received input based on the current values ​​of its corresponding parameter set. Summary of the Invention

[0005] This specification generally describes methods for training reinforcement learning systems that select actions to be performed by a reinforcement learning agent interacting with its environment. These methods can be used to train reinforcement learning systems with multiple potential conflicting objectives.

[0006] In one aspect, a method is provided for training a neural network system via reinforcement learning, the neural network system being configured to receive input observations representing the state of an environment interacting with by an agent and to select and output actions according to a policy aimed at satisfying multiple objectives. The method includes obtaining a set of one or more trajectories. Each trajectory includes a state of the environment, an action applied to the environment by the agent in response to the state according to a prior policy, and a set of rewards for the action, each reward being associated with a corresponding objective among the multiple objectives. The method further includes determining an action-value function for each of the multiple objectives based on the set of one or more trajectories. Each action-value function determines an action value representing an estimated reward for the corresponding objective, which would be generated by the agent performing a given action in response to a given state according to a prior policy. The method further includes determining an updated policy based on a combination of the action-value functions of the multiple objectives.

[0007] By determining an individual action-value function for each objective, the method described in this paper is able to effectively balance competing objectives during reinforcement learning. When combining the action values ​​of individual objectives, the action-value function overcomes the problems associated with determining optimal weights. Furthermore, the individual action-value function provides scale invariance with respect to the magnitude of the reward for each objective, thereby preventing one or more objectives from gaining learning dominance through the relative magnitude of their rewards.

[0008] The set of one or more trajectories can be obtained from storage (i.e., pre-computed) or by applying an agent to one or more states. The set of one or more trajectories may include multiple trajectories, thus allowing batch learning. Alternatively, as part of online learning, a trajectory may be provided with each update.

[0009] It should be noted that although this article discusses the term "reward," these rewards can be negative. In the case of negative rewards, these rewards can be equivalently regarded as costs. In this case, the overall objective of the reinforcement learning task will be to minimize the expected cost (rather than to maximize the expected reward or return).

[0010] In some implementations, each action-value function provides a distribution of action values ​​for a corresponding objective across a range of potential state-action pairs for the scope of the previous policy. Each action-value function may output an action value representing the expected cumulative discounted reward for the corresponding objective when a given action is selected in response to a given state. This cumulative discounted reward may be calculated over multiple subsequent actions implemented according to the previous policy. The action-value function for each objective may be considered an objective-specific action-value function.

[0011] In some implementations, determining the updated policy includes determining a target-specific policy for each of a plurality of targets. Each target-specific policy can be determined based on a corresponding action-value function for the corresponding target. The method may also include determining the updated policy by fitting the set of policy parameters of the updated policy to a combination of target-specific policies. The combination of target-specific policies can be the sum of the target-specific policies. Target-specific policies are also referred to herein as action distributions (not to be confused with action-value functions) because they provide a probability distribution of actions over states. Based on the above, the updated policy can be determined by combining the action-value functions via a combination of target-specific policies already derived from the action-value functions. This policy is then fitted to the combination of target-specific policies.

[0012] The method described in this paper combines objectives in the distribution space by combining them via a goal-specific policy. This contrasts with combining objectives in the reward space (e.g., by converting a multi-objective reward vector into a single scalar reward). By combining objectives in the distribution space, the combination is scale-invariant to the reward. The relative contribution of each objective to the updated policy can be scaled by leveraging constraints on the determined goal-specific policy.

[0013] In some implementations, fitting the updated set of policy parameters to a combination of target-specific policies includes determining a set of policy parameters that minimizes the difference between the updated policy and the combination of target-specific policies.

[0014] It is possible to constrain the minimization of the difference between the updated policy and the combination of target-specific policies, such that the difference between the updated policy and the previous policy does not exceed a trust zone threshold. In other words, it is possible to constrain the set of policy parameters used for the updated policy, such that the difference between the updated policy and the previous policy cannot exceed the trust zone threshold. The trust zone threshold can be considered as a hyperparameter that limits the overall change of the policy to improve learning stability.

[0015] The differences between the strategies discussed in this paper can be calculated using any other appropriate measure of the differences between distributions, such as the Kullback-Leibler (KL) divergence.

[0016] In some implementations, determining a target-specific strategy for each objective includes determining target-specific strategy parameters for the target-specific strategy that will increase the expected return relative to a previous strategy based on an action-value function for the corresponding objective.

[0017] In some implementations, determining the target-specific policy for each objective includes being constrained by the possibility that the target-specific policy may not differ from a previous policy by more than a corresponding difference threshold, and determining the target-specific policy parameters for the target-specific policy that maximize the expected return relative to the previous policy based on the action-value function for the corresponding objective. The difference between the target-specific policy and the previous policy can be determined based on any other appropriate measure of the difference between the Kullback-Leibler divergence or distributions.

[0018] Therefore, each target-specific policy can be determined by the constraint that it does not differ from the previous policy by more than a corresponding difference threshold. This corresponding difference threshold can be considered as representing the relative contribution of the corresponding target to the updated policy. Thus, the relative contribution of each target to the updated policy can be adjusted by adjusting the corresponding difference threshold. In other words, the relative weights between each target are encoded in the form of constraints on the influence of each target on policy updates.

[0019] In some implementations, the target-specific policy is a nonparametric policy. This reduces the computational complexity of determining the target-specific policy while still satisfying constraints regarding the corresponding difference thresholds. This is because the constraint optimization can be solved in closed-form for each state.

[0020] Each target-specific strategy q k (a|s) can be determined based on a scaled action-value function of the target according to a target-specific strategy, wherein the scaled action-value function is scaled by a value that depends on the preference for the target. The value that depends on the preference for the target can depend on a difference threshold used for the target. The value that depends on the preference for the target can be a temperature parameter η that depends on the difference threshold. k .

[0021] For example, each target-specific policy q k (a|s) can be determined by calculating the following formula:

[0022]

[0023] in:

[0024] N is a normalization constant;

[0025] k is the target;

[0026] 'a' represents the action;

[0027] s is a state;

[0028] π old (a|s) is the previous policy;

[0029] Q k (s, a) is the action-value function for the target; and

[0030] η k It is a temperature parameter.

[0031] For each target k, the temperature parameter η k This can be determined by solving the following equation:

[0032]

[0033] in:

[0034] ∈ k It is the difference threshold used for the corresponding target; and

[0035] μ(s) is the access distribution.

[0036] Each temperature parameter can be determined via gradient descent.

[0037] In another embodiment, a method is provided for training a neural network system via reinforcement learning, the neural network system being configured to receive input observations representing the state of an environment interacting with by an agent and to select and output actions according to a policy aimed at satisfying multiple objectives. The method may include obtaining a set of one or more trajectories, each trajectory including the state of the environment, an action applied to the environment by the agent in response to the state according to a previous policy, and a set of rewards for that action, each reward being associated with a corresponding objective among the multiple objectives. The method may further include determining a probability distribution (such as an action distribution or a state-action distribution) for each of the multiple objectives based on the set of one or more trajectories, each probability distribution providing an action probability distribution that would increase the expected reward relative to the policy according to the corresponding objective. The method may further include determining an updated policy based on a combination of the probability distributions for the multiple objectives.

[0038] Determining the probability distribution for each of the plurality of objectives may include, for each objective: defining a value function that represents the value of the expected reward according to the corresponding objective, which will be generated by the agent following the previous policy from a given state; and determining the probability distribution for that objective based on the value function.

[0039] Each probability distribution can be a state-action distribution defining the probability distribution of state-action pairs, and the value function for each objective can be a state value function defining the value representing the expected reward according to the corresponding objective, which will be generated by the agent following the previous policy from a given state. This may be applicable to online policy learning.

[0040] Optionally, each probability distribution can be a target-specific policy (action distribution) that defines the probability distribution of actions over states, and the value function can be an action value function representing the expected reward according to the corresponding target, which will be generated by the agent performing a given action according to the previous policy in response to a given state. This may be suitable for offline policy learning.

[0041] The methods described herein can be implemented using one or more computing devices and / or one or more computer storage media.

[0042] In one aspect, a system is provided comprising one or more computers and one or more storage devices storing instructions, wherein when the one or more computers execute the instructions, the instructions cause the one or more computers to perform the following operations: obtaining a set of one or more trajectories, each trajectory including a state of an environment, an action applied to the environment by an agent according to a prior policy in response to the state, and a set of rewards for the action, each reward being associated with a corresponding objective among a plurality of objectives; determining an action value function for each of the plurality of objectives based on the set of one or more trajectories, each action value function determining an action value representing an estimated reward for the corresponding objective, the estimated reward being generated by the agent performing a given action in response to a given state according to a prior policy; and determining an updated policy based on a combination of the action value functions for the plurality of objectives.

[0043] In one aspect, one or more computer storage media are provided, storing instructions that, when executed by one or more computers, cause the one or more computers to perform the following operations: obtaining a set of one or more trajectories, each trajectory including a state of an environment, an action applied to the environment by an agent according to a previous policy in response to the state, and a set of rewards for the action, each reward being associated with a corresponding objective among a plurality of objectives; determining an action value function for each of the plurality of objectives based on the set of one or more trajectories, each action value function determining an action value representing an estimated reward for the corresponding objective, the estimated reward being generated by the agent performing a given action in response to a given state according to a previous policy; and determining an updated policy based on a combination of action value functions for the plurality of objectives.

[0044] In order for the agent to interact with the environment, the system receives data representing the current state of the environment and selects an action to be performed by the agent in response to the received data. The data representing the state of the environment is referred to as observation in this specification.

[0045] In some applications, the environment is a real-world environment, and the agent is a mechanical intelligent agent that interacts with the real-world environment. For example, the agent could be a robot that interacts with the environment to perform a specific task. As another example, the agent could be an autonomous or semi-autonomous land, air, or water vehicle that navigates within the environment. In these implementations, actions can be control inputs used to control the physical behavior of the robot or vehicle.

[0046] Generally, observations can include one or more of, for example, images, object position data, and sensor data, to capture observations as the agent interacts with its environment, such as sensor data from images, distance or position sensors, or from actuators. In the case of robots or other mechanical agents or vehicles, observations can similarly include one or more of position, linear or angular velocity, force, torque or acceleration, and global or relative pose of one or more parts of the agent. Observations can be defined in one, two, or three dimensions and can be absolute and / or relative. For example, in the case of a robot, observations can include data characterizing the robot's current state, such as joint positions, joint velocities, joint forces, torque or acceleration, and one or more of global or relative pose of a part of the robot (such as an arm) and / or items held by the robot. Observations can also include, for example, sensed electronic signals, such as motor current or temperature signals; and / or image or video data, such as data from sensors of the agent or from sensors located separately from the agent in the environment.

[0047] In these applications, actions can be control inputs for controlling a robot, such as torques or higher-level control commands on the robot's joints; or for controlling autonomous or semi-autonomous land, air, or sea vehicles, such as torques or higher-level control commands on the vehicle's control surfaces or other control elements; or, for example, motor control data. In other words, actions can include, for example, position, velocity, or force / torque / acceleration data of one or more joints of a robot or components of another mechanical agent. Action data can include data used for these actions and / or electronic control data such as motor control data, or more generally, data used to control one or more electronic devices in an environment whose control has an impact on observed environmental states. For example, in the case of autonomous or semi-autonomous land, air, or sea vehicles, actions can include actions for controlling navigation (e.g., steering) and movement (e.g., braking and / or vehicle acceleration).

[0048] In these applications, the objectives and associated rewards / costs may include or be based on the following definitions:

[0049] i) One or more rewards for approaching or achieving one or more target positions, one or more target poses, or one or more other target configurations. One or more rewards depend on any of the aforementioned observations, such as the position or pose of the robot or vehicle. For example, in the case of a robot, the reward may depend on joint orientation (angle) or velocity, end effector position, center of mass position, or position and / or orientation of a group of body parts.

[0050] ii) One or more costs (e.g., negative rewards) can be defined similarly. Negative rewards or costs can also be associated with the force applied by the actuator or end effector, for example, depending on a threshold or maximum applied force when interacting with an object. Negative rewards can also depend on energy or power usage, excessive motion speed, for example, one or more locations of one or more robot body parts used to constrain movement.

[0051] These reward goals can be associated with different preferences—such as a high preference for safety-related goals like the work envelope or the force applied to an object.

[0052] The robot may be part of an autonomous or semi-autonomous mobile vehicle. Similar objectives may then be applied. And, or alternatively, such a vehicle may have one or more objectives related to the physical movement of the vehicle, such as those depending on the following (rewards): energy / power usage during movement, such as maximum or average energy usage; speed of movement; route taken during movement, such as penalties for taking a longer route than the shorter route between two points, measured by distance or time. Such a vehicle or robot may be used to perform tasks such as warehousing, logistics, or factory automation, such as collecting, placing, or moving stored goods or portions of goods during their manufacture; or the tasks performed may include package delivery control tasks. Therefore, one or more objectives may relate to such tasks, actions may include actions related to steering or other directional control actions, and observations may include observations of the position or movement of other vehicles or robots.

[0053] In some other applications, the same observations, actions, and objectives can be applied to the simulation of the physical system / environment described above. For example, a robot or vehicle can be trained in a simulation before being used in a real environment.

[0054] In some applications, the agent can be a static or mobile software agent, i.e., a computer program configured to operate autonomously and / or in conjunction with other software agents or persons to perform tasks. For example, the environment can be an integrated circuit routing environment, and the agent can be configured to perform routing tasks for routing interconnects of integrated circuits such as ASICs. The objective (reward / cost) can then depend on one or more routing metrics, such as interconnect resistance, capacitance, impedance, loss, speed or propagation delay, physical line parameters (such as width, thickness, or geometry), and design rules. These objectives can include one or more objectives related to the global characteristics of the routed circuit, such as component density, operating speed, power consumption, material usage, or cooling requirements. Observations can be observations of component locations and interconnects; actions can include component placement actions, such as those defining component location or orientation, and / or interconnect routing actions, such as interconnect selection and / or placement actions.

[0055] In some applications, the agent may be an electronic agent, and observations may include data (such as current, voltage, power, temperature) from one or more sensors monitoring a portion of a workshop or service facility, and other sensor and / or electronic signals representing the function of electronic and / or mechanical items of the equipment. The agent may control actions in a real-world environment including the items of the equipment—e.g., within a facility, such as a data center, server farm, or mains power supply or water distribution system, or in a manufacturing plant or service facility. Observations may then relate to the operation of the workshop or facility; for example, they may include observations of the equipment's use of electricity or water, or observations of power generation or distribution control, or observations of the use of resources or waste production. Actions may include actions that control or impose operating conditions on the items of the equipment in the workshop / facility, and / or actions that result in changes to settings in the operation of the workshop / facility, such as adjusting or turning on / off components of the workshop / facility. Objectives (to be maximized or minimized) may include one or more of the following: measurements of efficiency, such as resource use; measurements of the environmental impact of operations in the environment, such as waste production; electrical or other power consumption; heating / cooling requirements; resource use in the facility, such as water use; the temperature of the facility; and counts of the characteristics of items within the facility.

[0056] In some applications, the environment may be a data packet communication network environment, and the agent may include a router that routes data packets through the communication network. Actions may include data packet routing actions, and observations may include, for example, observations of a routing table, which includes routing metrics such as path length, bandwidth, load, hop count, path cost, latency, maximum transmission unit (MTU), and reliability. Objectives may include maximizing or minimizing one or more routing metrics.

[0057] In some other applications, an agent is a software agent that manages the allocation of tasks across computing resources (e.g., on mobile devices and / or in data centers). In these implementations, observation may include observations of computing resources such as computing and / or memory capacity, or internet-accessible resources; and actions may include assigning tasks to specific computing resources. Objectives may include those that depend on (e.g., maximizing or minimizing) one or more of the following: computing resource utilization, power consumption, bandwidth, and computing speed.

[0058] In some other applications, the environment is an internet or mobile communication environment, and the agent is a software agent that manages personalized recommendations for users. Observations may include previous actions taken by the user (characteristics representing them); actions may include recommending items such as content items to the user. Objectives may include maximizing or minimizing one or more of the following: the estimated probability that the user will respond favorably to the recommended (content) item, constraints on the suitability of one or more recommendations, the cost of the recommendations, and the number of recommendations received by the user (optionally over a time span).

[0059] The features corresponding to those previously described can also be used in the context of the aforementioned systems and computer storage media.

[0060] The subject matter described in this specification can be implemented in specific embodiments to achieve one or more of the following advantages. The subject matter described in this specification introduces a reinforcement learning method for learning a policy in which multiple potentially conflicting objectives exist. This is achieved by determining objective-specific action-value functions. By utilizing these objective-specific functions, the method described herein provides scale-invariant objective-specific functions; that is, the reward scale for a given objective does not affect the relative weights between objectives.

[0061] The scale invariance of this method offers two main advantages. First, the weights between objectives do not need to be adjusted over time due to varying reward sizes. This is particularly advantageous in reinforcement learning, where agents are likely to improve during task performance as they are trained, resulting in larger rewards over time. Second, objectives with relatively large rewards do not necessarily dominate training. Furthermore, by making the objective weights scale-invariant with respect to reward, the method is easier to implement, avoiding the need for trial and error when selecting weights for different reward sizes. This paper presents specific implementations that provide improvements in computational efficiency, for example, through the use of nonparametric objective-specific policies.

[0062] Some implementations of the described technique are capable of learning to perform tasks that take into account multiple different, potentially conflicting, objectives. Unlike some prior art, the technique described herein can adapt to rewards or penalties at different scales that can vary over time. In principle, the described technique can be applied to any reinforcement learning system that uses an action-value function (e.g., a Q-value function), although it is particularly useful for MPO (Maximum A posteriori policy optimization). The described technique allows reinforcement learning systems with multiple different objectives to learn in a faster and more stable manner, thus reducing memory and computational requirements compared to existing systems. The described technique is applicable to both discrete actions and real-world high-dimensional continuous control tasks.

[0063] In the implementation, the preference variable (∈ kThe weights are assigned to each objective to control the extent to which the objective contributes to the update of the combined action selection policy of the reinforcement learning system. This is used to adjust the "temperature" associated with the objective, which scales the action (Q) value associated with the objective. In the implementation, the "temperature" is associated with a variety of actions that help evaluate the overall action selection policy. Therefore, the weights between objectives can be scale-invariant even if the scale of the reward changes or the Q function is non-stationary. This allows the user to set preferences a priori among various objectives.

[0064] Details of one or more embodiments of the subject matter of this specification are set forth in the following drawings and description. Other features, aspects, and advantages of the subject matter will become apparent from the specification, drawings, and claims. Attached Figure Description

[0065] Figure 1 An example neural network system for reinforcement learning is shown.

[0066] Figure 2 This illustrates a method for training via multi-objective reinforcement learning, based on the arrangement.

[0067] Figure 3 This paper illustrates a method for training via multi-objective reinforcement learning, which includes a two-step policy update process, based on an arrangement.

[0068] In the various figures, the same reference numerals and names denote the same elements. Detailed Implementation

[0069] This specification describes learning techniques used in situations where one or more intelligent agents are designed to perform tasks with multiple competing objectives. This is common in the real world, where agents often must balance competing objectives. For example, an autonomous vehicle, such as a robot, might be needed to complete a task (Objective 1) while minimizing energy consumption or environmental damage (Objective 2). Other examples of such agents include factory or shop floor automation systems and computer systems. In this context, the agent could be a robot, an equipment item in a factory or shop floor, or a software agent in a computer system that, for example, controls the allocation of tasks to hardware items or the routing of data over a communication network.

[0070] This specification generally describes a reinforcement learning system implemented as a computer program on one or more computers at one or more locations, which uses a neural network to select actions to be performed by a reinforcement learning agent interacting with the environment. This specification also describes how such a system can tune the parameters of the neural network.

[0071] In order to interact with the environment, the system receives data representing the current state of the environment and, in response to the received data, determines the action to be performed by the agent from the action space—that is, the discrete action space or the continuous action space. In this specification, the data representing the state of the environment will be referred to as observations. The agent performs a selected action that causes a change in the state of the environment.

[0072] In some implementations, the environment is a simulated environment, and the agent is implemented as one or more computers that interact with the simulated environment. For example, a robot or vehicle may be trained in a simulation before being used in a real-world environment.

[0073] In other implementations, the environment is a real-world environment, and the agent is a mechanical agent that interacts with the real-world environment. For example, the agent may be a robot that interacts with the environment to perform a specific task, or an autonomous or semi-autonomous vehicle that navigates within the environment. In these cases, observations may be data captured by one or more sensors of the agent—e.g., cameras, LiDAR sensors, temperature sensors, etc.—as the agent interacts with the environment.

[0074] The specific arrangement described in this paper provides a method for training reinforcement learning systems with multiple potential conflicting objectives (multi-objective reinforcement learning).

[0075] Traditional reinforcement learning (RL) methods have done excellent work in training policies to optimize a single scalar reward function. However, many real-world tasks involve multiple potentially competing objectives. For example, controlling an energy system may require trade-offs between performance and cost; controlling an autonomous vehicle may require trade-offs between fuel cost, efficiency, and safety; and controlling a robotic arm may require trade-offs between speed, energy efficiency, and safety. Multi-objective reinforcement learning (MORL) methods aim to address such problems. One approach is scalarization: based on preferences across objectives, the multi-objective reward vector is transformed into a single scalar reward (e.g., by employing a convex combination), and then this scalar reward is optimized using standard RL.

[0076] However, for professionals, selecting an appropriate scalarization for desired preferences across objectives is challenging because objectives are often defined in different units and / or scales. For example, suppose we want an agent to complete a task while minimizing energy use and mechanical wear. Task completion might correspond to a sparse reward or the number of square feet a vacuum cleaner has cleaned, and reducing energy use and mechanical wear could be implemented by penalties on power consumption (in kWh) and actuator force (in N or Nm), respectively. Professionals would need to resort to trial and error to select a scalarization that ensures the agent prioritizes (and is therefore useful) task performance relative to energy conservation.

[0077] To overcome this problem, this application proposes a scale-invariant method for encoding preferences derived from RL as an inference perspective. The arrangement described herein learns the action-value function and action distribution for each objective of improving the current policy. Then, to obtain a single updated policy that makes these trade-offs, supervised learning can be used to fit the policy to a combination of these action distributions.

[0078] To weight relative objectives, instead of using scalarization, professionals set constraints on each objective. These constraints control the impact of each objective on the policy, for example, by constraining the KL divergence between each objective's specific distribution and the current policy. The higher the constraint value, the greater the objective's influence. Therefore, the expected preference for objectives can be encoded as the relative magnitude of these constraint values.

[0079] Essentially, scalarization combines objectives in the reward space. In contrast, the proposed method combines objectives in the distribution space in a way that is scale-invariant with respect to reward. In principle, this method can be combined with any RL method, regardless of whether it is an offline or online policy. The specific arrangement described in this paper combines it with Maximum A posteriori Policy Optimization (MPO)—an offline policy actor-evaluator RL method—and V-MPO—an online policy variant of MPO. These two approaches are referred to in this paper as Multi-Objective MPO (MO-MPO) and Multi-Objective VMPO (MO-V-MPO), respectively.

[0080] Finally, this method provides a distributional view of multi-objective reinforcement learning (MORL) that enables scale-invariant encoding of preferences. This is a theoretically fundamental approach, derived from RL using MORL as an inference perspective. Empirically, the mechanism of MO-MPO has been analyzed, and it has been shown that it finds all Pareto-optimal policies in popular MORL benchmark tasks. MO-MPO and MO-V-MPO outperform scalarized methods on multi-objective tasks spanning several challenging high-dimensional continuous control domains.

[0081] Figure 1 An example neural network system 100 for reinforcement learning is shown. The neural network system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, wherein the system, components and techniques described below are implemented.

[0082] Neural network system 100 includes an action selection policy neural network 110, which determines actions 102 to be applied to an agent 104 and output to the agent. Neural network system 100 operates over multiple time steps t. Each action a... t 102 Based on the current state s of the characterizing environment tThe observation 106 is used to determine the initial state s0106. After the initial observation representing the initial state s0106 of the environment is input, the neural network system 100 determines the action a0102 and outputs the action 102 to the agent 104. After the agent 104 has applied the action 102 to the environment 104, the updated state s1106 is input into the neural network 100. The neural network 100 thus operates on multiple time steps t in response to the input observation s0106. t 106 to select action a t 102. Each action a t 102 Policy π based on a set that depends on policy parameters θ θ To determine this. In one arrangement, the action selection policy neural network 110 is a feedforward neural network, although other types of neural networks can be utilized. For each time step, it also receives information about the previous action a. t-1 The reward set r t 108. The reward set r t 108 includes rewards for each objective.

[0083] Observations can include images of the environment and / or other sensor or input data from the environment. Such observations are typically preprocessed, for example, by one or more convolutional neural network layers and / or one or more recurrent neural network layers.

[0084] System 100 also includes a training engine 120, which is configured to update the parameters of the policy based on the reward 108 received for each target. When training the neural network system 100, the system can operate on one or more time steps t, thereby updating the policy parameters based on the reward 108 received for each action a. t 102 reward r t Before updating the policy parameter θ, based on the current state s t 108 and the current strategy π θ Select one or more corresponding actions a t 102. Batch training can be utilized, where a given policy is applied to multiple time steps before updating the given policy.

[0085] Formally, this arrangement applies a multi-objective RL problem defined by a multi-objective Markov decision process (MO-MDP). The MO-MDP consists of states... and actions Initial state distribution p(s0) and transition probability p(s) t+1 |s t ;a t The transition probability p(s) is composed of ) t+1 |s t ;a t Defined in taking action a t From state st Change to s t+1 The probability of [the objective]. In this arrangement, the neural network system 100 applies multiple objectives. Therefore, a reward function is assigned for each objective k. Provide a discount factor γ∈[0,1) to be applied to the reward. Strategy π θ (a|s) is defined as the state-conditional distribution over actions parameterized by θ. Together with the transition probabilities, this yields the state-access distribution μ(s).

[0086] In addition to the target-specific reward, an action-value function (Q-function) is provided for each target. The action-value function maps states and actions to values. The action-value function for target k is defined as the expected reward (i.e., the cumulative discounted reward) based on the action a chosen for target k in state s and then following policy π: This function can use recursive expressions. It means that, among them, It is a function of the value of π for the target k.

[0087] During training, System 100 attempts to identify the optimal policy. For any MO-MDP, there exists a set of non-dominant policies, known as the Pareto front. A policy is non-dominant if there are no other policies that improve its expected return on a target without decreasing the expected return on at least one other target. Given a preference setting, the goal of this method is to find non-dominant policies π that satisfy those preferences. θ In this method, the constraints are not set directly to specific scalar values. Instead, by changing these constraint settings, the Pareto front of the policy can be explored.

[0088] Typically, training involves a two-step approach to updating the policy. First, an action distribution (target-specific policy) is determined for each target based on the corresponding action-value function. Then, the overall policy is fitted to a combination of action distributions.

[0089] Figure 2 A method for training via multi-objective reinforcement learning based on an arrangement is shown. This method divides the reinforcement learning problem into two sub-problems and iterates until convergence:

[0090] 1. Policy Evaluation: Given policy π θ Estimating the Q-function

[0091] 2. Strategy Improvement: Given a Q-function update strategy

[0092] Algorithm 1 summarizes the two-step multi-objective policy improvement process.

[0093] In each iteration of the training method, a trajectory set 210 is obtained before determining the action-value function 220 for each target. Then, an updated policy 230 is determined based on the combination of action-value functions. The method determines whether a termination criterion 240 has been met (e.g., a fixed number of iterations have been performed, or the policy meets a given performance level). If not, the updated policy π is used. θ Perform another iteration. If so, output policy 250 (e.g., store locally or send to an external device, such as an agent to implement the policy).

[0094] Each trajectory includes a state s t Based on strategy π θ Determine and apply the action a to the environment t and the action a at one or more time steps t (the sum of up to N time steps). t The reward set r t The reward set r for each action t Each reward in the system is associated with a corresponding objective. Each reward can be received from an external source (e.g., the environment) or based on the state of the environment. t (For example, based on the corresponding reward function) to determine. Alternatively, multiple trajectories can be obtained from multiple different starting states s0 over multiple rounds, depending on the batch size (L) of the total number of rounds defined.

[0095] When determining the action-value function 220 for each target, this can be an update of the previous action-value function for that target. This determination / update is based on the already obtained trajectories (i.e., based on actions, states, and rewards from one or more trajectories). The details of this determination will be discussed in more detail below.

[0096]

[0097]

[0098] Multi-objective strategy evaluation

[0099] Neural network systems evaluate previous policies π by learning a state-action value (Q) function. old Following the Q-decomposition method, a separate Q-function is trained for each target. In principle, any Q-learning algorithm can be used, as long as it is relative to π. old The target Q-value can be calculated (based on the strategy prior to the current iteration of the update).

[0100] Typically, Q-learning aims to learn an approximation of the action-value function. To achieve this, the following update can be applied at each iteration of training to learn the action-value function for each target k via φ. k Parameterized Q function

[0101]

[0102] in, It is a target action-value function (target Q-function) based on a vector of state S, action A, and reward R. The target Q-value is an estimate of the sum of discounted rewards (e.g., determined from one or more trajectories obtained through running a policy).

[0103] Different types of objective Q-functions exist, and they are equally applicable to this method. In a particular implementation, the Retrace objective is used to learn the pass φ for each objective k. k Parameterized Q-function as follows:

[0104]

[0105] in, It is used for target k and previous policy π old The Retrace target, and It is a replay buffer containing the collected transitions (state-action pairs). This minimizes the mean squared error between the Retrace target and the learned Q-function.

[0106] In this implementation, the Retrace objective is as follows:

[0107]

[0108] in

[0109]

[0110]

[0111] Importance weight c z Defined as

[0112]

[0113] Where b(a) z |s z This represents the behavioral strategy used to collect trajectories in the environment. When j = t, the method is set...

[0114] In a particular implementation, two networks are maintained for the Q-function for each target: an online network and a target network, wherein the parameters are respectively determined by φ. k and φ′ kSimilarly, an online network and a target network can be maintained for the policy, where the parameters are represented by θ and θ′, respectively. The target network can be updated every fixed number of steps by copying parameters from the online network. In each learning iteration, the online network can be updated using any suitable update method such as gradient descent. The target policy network is referred to above as the old policy π. old .

[0115] In a particular implementation, an asynchronous actor-learner setup can be used, where the actor periodically retrieves policy parameters from the learner and acts in the environment, writing these transitions to a replay buffer. This policy is called the behavioral policy. The learner uses the transitions in the replay buffer to update its (online) Q-function and policy. This method is illustrated in more detail in Algorithm 2.

[0116] Algorithm 2 is based on a given strategy π θ Methods for obtaining the trajectory. The system obtains the current policy π as defined. θ The current policy parameters θ are then determined. The system then collects a set of trajectories over multiple time steps T. Each trajectory τ includes the state s for each time step. t Action a t and a reward set r. This reward set r includes the reward r for each objective. k The system works by applying the current state s to each time step. t And based on the current strategy π θ Determine action a t And based on the set of reward functions Determine the next state s generated by the action to be performed. t+1 The reward is used to obtain trajectory τ. Trajectory τ is obtained for each of the L rounds. Each trajectory τ is stored in the replay buffer. Then, the stored trajectories are used to update the Q-function for each target.

[0117]

[0118] Multi-objective strategy improvement

[0119] Given the previous policy π old (a|s) and the associated Q-function The next step is to improve the previous policy given the access distribution μ(s). This can be achieved by taking the expected value on the access distribution from the replay buffer. To do this, the system learns the action distribution (target-specific policy) for each Q-function and combines these to obtain the next policy π. new (a|s).

[0120] Figure 3A method for training based on an arrangement via multi-objective reinforcement learning, including a two-step policy update process, is shown. This method is similar to... Figure 2 The method is roughly the same; however, the policy update step is replaced with the following two steps:

[0121] The action distribution 330 for each target is determined based on the corresponding action value function.

[0122] The updated policy is determined based on the combination of action functions of multiple targets.335

[0123] In the first step 330, for each target k, an improved action distribution q is learned. k (a|s) makes The state s is obtained from the access distribution μ(s) (e.g., from the replay buffer). In other words, the improved action distribution q is learned. k (a|s) such that the expectation of the Q-function with respect to the action distribution is greater than or equal to the expectation of the Q-function with respect to the policy.

[0124] In the second step 335, the improved distribution q k By minimizing the difference between the distribution and the new parameter policy, they are combined and distilled into the new parameter policy π. new (with parameter θ) new This can be achieved by minimizing the KL divergence between the distribution and the new parameter strategy, i.e.,

[0125]

[0126] Among them, KL(q) k (a|s)||π θ (a|s)) is the action distribution q for target k. k (a|s) and strategy π θ The Kullback-Leibler divergence between (a|s) is used to determine the distribution q. k The supervised learning loss is estimated using maximum likelihood estimation. These two steps will be explained in more detail below.

[0127] Obtain the motion distribution for each target (Step 1)

[0128] To obtain the improved action distribution q for each objective k (a|s), optimize the reinforcement learning objective Q for each objective. k :

[0129]

[0130] st∫ s μ(s)KL(q k(a|s)||π old (a|s))ds<∈ k

[0131] Where, ∈ k Let represent the allowed expected KL divergence for objective k. These ∈ k Used to encode preferences on a target. More specifically, ∈ k Define the permissible impact of target k on policy changes.

[0132] For nonparametric action distribution q k The constrained optimization problem (a|s) can be solved in closed-form for each state s sampled from μ(s).

[0133]

[0134] Wherein, temperature η k By solving the following convex bifunction based on the corresponding ∈ k To calculate:

[0135]

[0136] To evaluate q k Based on (a|s) and the above integrals, the system can obtain L states from the replay buffer, and for each state, from the current policy π old M actions are sampled. In practice, each target maintains a temperature parameter η. k We have found that by studying η k Performing several steps of gradient descent to optimize the bifunction is effective. This method initializes the function with the solution found in previous policy iterations. Since η k It should be positive, so the projection operator can be used after each gradient step to maintain η. k >0.

[0137] As shown in Algorithm 1, the action distribution q for each target is... k (a|s) can be calculated using the following formula:

[0138]

[0139] Use optimizer based on To update η k Then determine the action distribution q k (a|s)

[0140]

[0141] Due to constraint ∈ kSince preferences on the objective are encoded, solving this optimization problem with good constraint satisfaction is important for learning strategies that satisfy desired preferences. This applies to the nonparametric action distribution q. k (a|s) can precisely satisfy these constraints. Alternatively, any policy gradient method can be used to obtain q in parametric form. k (a|s). However, solving for the parameter q k The constraint optimization of (a|s) is inaccurate, and the constraints may not be satisfied well, which hinders the use of ∈ k Preferences are encoded. Furthermore, it is assumed that the parameter q... k (a|s) requires maintaining a function approximator for each objective (e.g., a neural network), which can significantly increase the complexity of the algorithm and limit scalability.

[0142] Strategy for fitting new parameters (Step 2)

[0143] In the previous section, for each target k, the improved action distribution q has been obtained. k (a|s)(Improved target-specific strategy). Next, these distributions need to be combined to obtain the desired result based on the established constraints ∈ k To compromise on the single-parameter policy of the target, this method addresses the supervised learning problem of fitting the parameter policy to the distribution of each target action from step 1.

[0144]

[0145] st∫ s μ(s)KL(π old (a|s)||n θ (a|s))ds<β,

[0146] Here, θ represents the parameters of the policy neural network, while the KL constraint restricts the trust region of the overall change in the policy parameters β. The KL constraint in this step has a regularization effect, preventing the policy from overfitting to the sample-based action distribution, thus avoiding premature convergence and improving the stability of the learning process.

[0147] Similar to the first policy improvement step, the integral can be evaluated using L states sampled from the replay buffer and M actions per state sampled from the old policy. To optimize the above using gradient descent, Lagrange relaxation can be implemented.

[0148] As shown in Algorithm 1, strategy π θ (a|s) can be updated by calculating the following formula:

[0149]

[0150] Subject to the aforementioned regularization constraints, the policy parameters can then be used by an optimizer (e.g., via gradient descent) based on δ. π Update.

[0151] Online strategy learning

[0152] The above implementation discussed batch learning. The method described herein can also be applied to online policy learning. In this case, in order to evaluate the previous policy π old The advantage A(s, a) is estimated from the learned state-value function V(s) rather than the state-action-value function Q(s, a) as in the offline policy implementation. A separate V-function for each objective is trained by regressing to n-step regressions associated with each objective.

[0153] More specifically, given a trajectory segment τ = {(s0, a0, r0), ..., (s T a T r T )}, where r t This represents the reward vector consisting of rewards for all N objectives. Value function parameter φ k This can be achieved by optimizing the following objectives:

[0154]

[0155] Here, G (T) (s t a t ) is the T-step objective k for the value function, which uses the actual reward in the trajectory and the bootstrap from the current value function for the rest: Then, the advantage was estimated as

[0156] Given the previous policy π old (a|s) and the estimated advantage associated with the strategy for each objective. The goal is to improve previous strategies. To this end, the method first learns the variation distribution q of the improvement for each objective. k (s, a), then combine the variation distributions and distill them into a new parameter strategy π. new (a|s). Unlike the offline strategy implementation, this implementation uses a joint distribution q. k (s, a) instead of the local policy q k (s|a) is because, without a learned Q-function, only one action is available for learning per state. Each joint distribution provides the probability of a state-action pair given a corresponding objective.

[0157] To obtain the improved variation distribution qk (s, a), this method optimizes the RL objective for each objective.

[0158]

[0159] stKL(q k (s, a)||p old (s, a))<∈ k ,

[0160] Wherein, the KL-divergence is calculated over all (s, a), ∈ k Let p represent the allowed expected KL divergence. old (s, a) = μ(s)π old (a|s) is related to π old Associated state-action distribution.

[0161] In the offline strategy implementation, the online strategy implementation uses ∈ k Define preferences on objectives. More specifically, ∈ k The allowed contribution of objective k to policy changes is defined. Therefore, for a specific ∈ k The larger the value of k relative to others, the more preferred the objective k becomes. On the other hand, if ∈ k If the value is 0, then target k will not contribute to the change of policy and will be effectively ignored.

[0162] The above equation can be solved in closed-form:

[0163]

[0164] Wherein, temperature η k By solving the following convex biproblem based on the constraint ∈ k To calculate

[0165]

[0166] By using η k A gradient descent step is taken, along with the loss, to perform optimization, and this can be initialized using the solution found in previous policy iteration steps. Since η k It must be positive, so the projection operator can be used after each gradient step to maintain η. k >0.

[0167] In practice, training can be performed using samples that correspond to the proportion of the largest advantage in each batch of data (e.g., the top 50%).

[0168] The next step is to combine the state-action distributions obtained in the previous steps and distill them according to ∈ kA single-parameter strategy π that specifies preferences and benefits all objectives new (a|s). Therefore, supervised learning problems that conform to the following parameter strategy can be solved:

[0169]

[0170] st∫ s，a KL(π old (a|s)||π θ (a|s))ds<β,

[0171] Here, θ is the parameter of the function approximator (neural network), which is determined according to the previous strategy π. old The weights are initialized, and the KL constraint is applied to limit the trust region of the overall change β in the policy parameters to improve the stability of learning. As in the offline policy implementation, the KL constraint in this step has a regularization effect, which prevents the policy from overfitting to local policies and thus avoids premature convergence.

[0172] To optimize the above equation, Lagrange relaxation can be used.

[0173] Choose ∈ k

[0174] Through ∈ k Encoding preferences through scalar weights is more intuitive than scalar weights because the former is scale-invariant with respect to reward. In other words, cross-objective preference scaling is used for ∈ k The range of reasonable scalar weights is determined, but not narrowed. To identify reasonable scalar weights, RL professionals need additional familiarity with the scalarization scale of the reward for each objective. In practice, we have found that learning performance is more important for ∈ k It is robust across a wide range of scales. Regarding encoding preferences on the target, it is important that ∈ k Relative scale—specific ∈ k The larger the value of k relative to others, the more preferred the objective k becomes. On the other hand, if ∈ k If = 0, then the target k will have no effect and will be effectively ignored. Typically, specific implementations apply ∈ in the range of 0.001 to 0.1. k .

[0175] When all objectives are equally important, the general rule is to include all ∈ k Set them to the same value. Conversely, in linear scalarization, choosing appropriate weights to encode equal preferences can be difficult—setting all weights to be equal 1 / K (where K is the number of objectives) is only appropriate if the rewards of the objectives have similar scales.

[0176] Even when all ∈k When set to the same value, ∈ k The absolute value of ∈ will also affect learning. k The larger the value, the greater the impact of the objective on the policy update steps. Since the evaluator for each objective is learned in parallel with the policy, setting ∈ k Too high a value often leads to unstable learning because, early in training, when evaluators produce unreliable Q-values, their influence on the policy can lead to incorrect direction. On the other hand, if the value is set ∈ k Too low a value slows down learning because it only allows the action distribution for each target to deviate very small from the current policy, and the updated policy is obtained through supervised learning of combinations of these action distributions. However, as long as ∈ k If the target is not set too high, the learning will eventually converge to more or less the same strategy.

[0177] When there are differences in preferences across objectives, ∈ k The relative scale is important. ∈ k and ∈ l The larger the relative scale, the greater the impact of target k on policy updates compared to target l. In the extreme case, when ∈ l When the target l is close to zero, the target l will not affect the policy update and will be effectively ignored.

[0178] A common example of unequal preferences is when an agent needs to complete a task while minimizing other objectives—such as energy expenditure, applied force (e.g., a "pain" penalty), etc. In this case, the ∈ of the task objective should be higher than the ∈ of other objectives to incentivize the agent to prioritize actually performing the task. If the ∈ used for the penalty is too high, the agent will focus more on minimizing the penalty (which can often be achieved simply by not taking any action) than on performing the task, which is not particularly useful.

[0179] ∈ k The scale has a similar effect to that in the case of equal preferences. If ∈ k If the scale is too high or too low, the same problem arises as discussed for the same preferences. If all ∈ k If all factors are increased or decreased by the same (moderate) factor on the scale, and thus their relative scales remain the same, they will generally converge to more or less the same strategy. As mentioned above, ∈ in the range of 0.001 to 0.1 k Good results can be obtained.

[0180] The subject matter described in this specification introduces a reinforcement learning approach for learning policies where multiple potentially conflicting objectives exist. This is achieved by determining objective-specific action-value functions. By utilizing these objective-specific functions, the method described herein provides an objective-specific function that is independent of the scale of the reward for a given objective. This means that when the size of the reward changes, it is not necessary to adjust the weights between objectives over time. Furthermore, larger rewards do not necessarily favor training. Moreover, by keeping the weighted objective scale constant relative to the reward, the method is easier to implement, thus avoiding the need for continuous iterative trials when selecting weights for varying reward sizes. Specific implementations that provide improvements in computational efficiency (e.g., by using nonparametric objective-specific policies) are presented herein.

[0181] In some implementations, preference variables (∈ k The weighting of each objective is assigned to control how much it contributes to the update of the combined action selection policy of the reinforcement learning system. This is used to adjust the "temperature" associated with the objective, which scales the action (Q) value associated with the objective. In the implementation, "temperature" refers to the diversity of actions that contribute to the evaluation of the overall action selection policy. Therefore, the weighting between objectives can be scale-invariant even if the scale of the reward changes or the Q function is not fixed. This allows the user to set preferences a priori among various objectives.

[0182] For a system of one or more computers configured to perform a specific operation or action, this means that the system has software, firmware, hardware, or a combination thereof installed thereon, which, in operation, cause the system to perform the operation or action. For one or more computer programs configured to perform a specific operation or action, this means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform those operations or actions.

[0183] Embodiments of the subject matter and functional operation described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed in this specification and their equivalents, or in combinations thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs—that is, one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier—for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively, or otherwise, program instructions may be encoded on artificially generated propagating signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus. Computer storage media may be machine-readable storage devices, machine-readable storage substrates, random or serial access memory devices, or combinations thereof. However, computer storage media are not propagating signals.

[0184] The term "data processing apparatus" includes all kinds of devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. Apparatus may include dedicated logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, apparatus may also include code that creates an execution environment for computer programs, such as code consisting of processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0185] Computer programs (also referred to or described as programs, software, software applications, modules, software modules, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A computer program may, but does not necessarily, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program under discussion, or in multiple collaborating files (e.g., files storing one or more modules, subroutines, or multiple parts of code). A computer program can be deployed and executed on one or more computers located at a single site or distributed across multiple sites interconnected by a communication network.

[0186] As used herein, "engine" or "software engine" refers to a software-implemented input / input system that provides outputs other than inputs. An engine can be a functionally coded block, such as a library, platform, software development kit (SDK), or object. Each engine can be implemented on any suitable type of computing device, including one or more processors and computer-readable media, such as a server, mobile phone, tablet computer, laptop computer, music player, e-book reader, laptop or desktop computer, PDA, smartphone, or other fixed or portable device. Furthermore, two or more engines can be implemented on the same computing device or on different computing devices.

[0187] The processes and logic flows described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and producing output. The processes and logic flows can also be executed by dedicated logic circuits, and the apparatus can also be implemented as dedicated logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). For example, the processes and logic flows can be executed by a graphics processing unit (GPU), and the apparatus can also be implemented as a graphics processing unit (GPU).

[0188] A computer suitable for executing computer programs includes, for example, a central processing unit (CPU) that may be based on a general-purpose or special-purpose microprocessor or both, or any type. Generally, the CPU receives instructions and data from read-only memory or random access memory or both. Essential components of a computer are the CPU that implements or executes instructions and one or more storage devices that store the instructions and data. Generally, a computer will also include one or more mass storage devices for storing data or operatively coupled to receive data from or transfer data to said one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks. However, a computer does not necessarily have to have these devices. Moreover, a computer can be embedded in another device, such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, GPS receiver, or portable storage device (e.g., a Universal Serial Bus (USB) flash drive), to name just a few.

[0189] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor storage devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. Processors and memory may be supplemented or integrated therein by dedicated logic circuitry.

[0190] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile feedback. Additionally, the computer can interact with the user by sending and receiving documents from the device used by the user; for example, in response to a request received from a web browser, sending a webpage to a web browser on the user's client device.

[0191] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., as a data server), middleware components (e.g., an application server), front-end components (e.g., a client computer having a graphical user interface or web browser through which a user can interact with embodiments of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected via any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), such as the Internet.

[0192] A computing system may include clients and servers. Clients and servers are typically located far apart and generally interact through a communication network. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other.

[0193] Although this specification contains many specific implementation details, these details should not be construed as limiting the scope of any invention or claimable content, but rather as interpretations of features applicable to specific embodiments of a particular invention. Certain features described in the context of individual embodiments may also be implemented in combination within a single embodiment. Conversely, various features described in the context of a single embodiment may be implemented separately in multiple embodiments or in any suitable sub-combinations. Moreover, although features may be described above as acting in certain combinations and even initially claimed in this way, in some cases one or more features from a claimed combination may be stripped from the combination, and the claimed combination may be for sub-combinations or variations thereof.

[0194] Similarly, although operations are described in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order, or to perform all the illustrated operations in order to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Moreover, the separation of the various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0195] Specific embodiments of this subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. For example, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing may be advantageous.

Claims

1. A method for training a neural network system via reinforcement learning, the neural network system being configured to receive input observations representing the state of an environment interacted with by an agent and to select and output actions according to a policy aimed at satisfying multiple objectives, wherein: The environment is a real-world environment, and the intelligent agent is a mechanical proxy that interacts with the real-world environment. The intelligent agent includes a robot that interacts with the real-world environment to complete a specific task, or an autonomous or semi-autonomous vehicle that navigates within the real-world environment. The input observations include at least one of the following: an image of the environment, object orientation data describing the orientation of one or more objects in the environment, or sensor data used to capture observations when the intelligent agent interacts with the environment. The environment is a simulated environment, the agent is implemented as one or more computer programs that interact with the simulated environment, and the input observations include simulated versions of one or more of the following: an image of the environment, object location data describing the location of one or more objects in the environment, or sensor data used to capture observations when the agent interacts with the environment, or The environment is a workshop or facility that includes equipment items, and the intelligent agent controls actions within the workshop or facility, including actions to control or apply operating conditions to the equipment items, or actions that cause changes to settings in the operation of the workshop or facility. The method includes: A set of one or more trajectories is obtained, each trajectory including the state of the environment, an action applied by the agent to the environment in response to the state according to a previous policy, and a set of rewards for the action, each reward being associated with a corresponding target among the plurality of targets; Based on the set of one or more trajectories, a separate action value function is determined for each of the plurality of targets, each separate action value function determining an action value representing an estimated reward based on the corresponding target, the estimated reward being generated by the agent performing a given action in response to a given state according to the prior policy; Combining the individual action value functions of the multiple targets; and An updated target-specific policy is determined based on a combination of the individual action value functions of the multiple targets, wherein determining the updated target-specific policy includes: Determine a target-specific policy for each of the plurality of targets, wherein each target-specific policy is determined based on a separate action-value function corresponding to the target; and The updated policy is determined by fitting the set of policy parameters of the updated target-specific policy to the combination of the target-specific policies.

2. The method according to claim 1, wherein, Fitting the set of policy parameters of the updated policy to the combination of target-specific policies includes determining the set of policy parameters that minimizes the difference between the updated policy and the combination of target-specific policies.

3. The method according to claim 1, wherein, The set of policy parameters of the updated policy is constrained such that the difference between the updated policy and the previous policy cannot exceed the trust zone threshold.

4. The method according to claim 1, wherein, Determining a target-specific strategy for each target includes determining target-specific strategy parameters for the target-specific strategy, which increase the expected return relative to the previous strategy based on the action-value function of the corresponding target.

5. The method according to claim 4, wherein, Determining the target-specific strategy for each target includes: subjecting the target-specific strategy to not differ from the previous strategy by more than a corresponding difference threshold, determining the target-specific strategy parameters that maximize the expected return based on a separate action value function for the corresponding target, relative to the previous strategy.

6. The method according to claim 5, wherein, The corresponding difference threshold represents the relative contribution of the corresponding target to the updated strategy.

7. The method according to claim 1, wherein, The target-specific policy is a nonparametric policy.

8. The method according to claim 5, wherein, Specific strategies for each target The action value function is determined based on a scaled action value function of the target according to the target-specific strategy, wherein the scaled action value function is scaled by values ​​that depend on the preference for the target.

9. The method according to claim 8, wherein, The value, which depends on the preference for the target, depends on the difference threshold of the target.

10. The method according to claim 7, wherein, Specific strategies for each target It is determined by calculating the following formula: in: N It is a normalization constant; k It is the stated objective; a It is an action; s It is a state; This is the previously mentioned strategy; It is the separate action value function for the target; and It is a temperature parameter.

11. The method according to claim 10, wherein, For each target k The temperature parameter This is determined by solving the following equation: in: It is the difference threshold of the corresponding target; and It is the access distribution.

12. The method according to claim 11, wherein, Each temperature parameter was determined using gradient descent.

13. The method according to any one of claims 1-12, wherein, Each individual action-value function provides the distribution of action values ​​for the corresponding target among the multiple targets across the range of potential state-action pairs for the previous policy.

14. The method according to any one of claims 1-12, wherein, Each individual action value function outputs an action value representing the expected cumulative discount reward for the corresponding objective when a given action is selected in response to a given state.

15. A method for training a neural network system by reinforcement learning, the neural network system being configured to receive input observations representing the state of an environment interacting with an agent and to select and output actions according to a policy aimed at satisfying multiple objectives, wherein: The environment is a real-world environment, and the intelligent agent is a mechanical proxy that interacts with the real-world environment. The intelligent agent includes a robot that interacts with the real-world environment to complete a specific task, or an autonomous or semi-autonomous vehicle that navigates within the real-world environment. The input observations include at least one of the following: an image of the environment, object orientation data describing the orientation of one or more objects in the environment, or sensor data used to capture observations when the intelligent agent interacts with the environment. The environment is a simulated environment, the agent is implemented as one or more computer programs that interact with the simulated environment, and the input observations include simulated versions of one or more of the following: an image of the environment, object location data describing the location of one or more objects in the environment, or sensor data used to capture observations when the agent interacts with the environment, or The environment is a workshop or facility that includes equipment items, and the intelligent agent controls actions within the workshop or facility, including actions to control or apply operating conditions to the equipment items, or actions that cause changes to settings in the operation of the workshop or facility. The method includes: A set of one or more trajectories is obtained, each trajectory including the state of the environment, an action applied by the agent to the environment in response to the state according to a previous policy, and a set of rewards for the action, each reward being associated with a corresponding target among the plurality of targets; Based on the set of one or more trajectories, a separate probability distribution is determined for each of the plurality of targets, each separate probability distribution providing a distribution of the probability of actions that will increase the expected reward relative to the policy based on the corresponding target; Combining the individual probability distributions of the plurality of targets; and The updated target-specific strategy is determined based on a combination of the individual probability distributions of the multiple targets.

16. The method of claim 15, wherein: Determining the individual probability distribution for each of the plurality of targets includes, for each target: Define a value function representing the expected reward based on the corresponding objective, the expected reward being generated by the agent following the previous policy from a given state; and The individual probability distribution of the target is determined based on the value function.

17. The method of claim 16, wherein: Each individual probability distribution is a state-action distribution that defines the probability distribution of state-action pairs, and the value function for each objective is a state value function that defines the value representing the expected reward according to the corresponding objective, which will be generated by the agent following the previous policy from a given state; or Each individual probability distribution is a target-specific policy that defines the probability distribution of actions over states, and the value function is an action value function that represents the expected reward according to the corresponding target, which will be generated by the agent performing a given action in response to a given state according to the previous policy.

18. A system comprising one or more computers and one or more stores of stored instructions, said instructions, when executed by said one or more computers, causing said one or more computers to perform the method according to any one of claims 1-17.

19. A computer storage medium storing one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform the method according to any one of claims 1-17.

Images