A task-prioritized experience replay algorithm for reinforcement learning

Abstract

The Task Prioritized Experience Replay (TaPER) algorithm enables the simultaneous learning of multiple RL tasks in an off-policy manner. The algorithm can prioritize samples that were part of fixed-length episodes that led to task accomplishment. This allows the agent to quickly learn task policies by bootstrapping over early successes. Finally, TaPER can simultaneously improve performance on all tasks, a desirable property for multitask RL. Unlike traditional ER algorithms that apply to single-RL task learning settings, require rewards to be binary or abundant, or are provided as parameterized specifications of goals, TaPER does not impose such constraints and supports arbitrary reward and task specifications.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE Embodiments of the present invention relate generally to reinforcement learning methods. In particular, the present invention relates to a system and method for training an agent using a task-prioritized experience replay algorithm that enables the simultaneous learning of multiple reinforcement learning tasks in an off-policy manner. [Background technology]

[0002] The following background information may present examples of certain aspects (e.g., but not limited to, techniques, facts, or conventional wisdom) of the prior art that are expected to serve to further inform the reader of additional aspects of the prior art, but should not be construed as limiting the invention or any embodiment thereof to anything stated or implied therein or inferred therefrom.

[0003] Reinforcement Learning (RL) describes a class of problems (and solutions) in which an artificial agent optimizes the control of a system (or "environment") when the dynamics of that system are unknown and must be learned by the agent through interaction and observation. A reinforcement learning system can include four main elements: an agent, a policy, a reward, and a value function. The behavior of an agent at any point in time is defined in terms of a policy.

[0004] A policy π(a,s) is a probabilistic mapping between actions a and states s. Since not all policies are optimal, methods are developed to improve the policies. Such methods are classified as either on-policy or off-policy methods. On-policy methods attempt to evaluate or improve the same policy used to make the decision. On the other hand, off-policy methods evaluate or improve a different policy than the one used to generate the data. Off-policy reinforcement learning allows learning a policy to solve a task from experience collected from different behavior policies.

[0005] Experience replay (ER) algorithms play a key role in training gradient-based deep RL architectures. Training these architectures online often requires that the inputs are repeated and uncorrelated in time. ER algorithms efficiently address these issues by using a buffer to store experiences and sampling from the experiences independently and equally distributed. ER algorithms can be further classified into two subclasses: those that sample experiences uniformly and those that sample specific experiences with higher priority.

[0006] Traditional ER algorithms have mostly been applied in single-RL task learning settings. Few that have been applied in multi-task settings require the reward to be binary or enriched. Some of these also require a parameterized specification of the goal. Summary of the Invention [Problem to be solved by the invention]

[0007] In view of the above, there is a need for systems and methods that improve off-policy reinforcement learning. [Means for solving the problem]

[0008] An embodiment of the invention is a method for training an agent in a control loop, the method comprising the steps of: t ) for behavioral policy (π b ) sampled from (a t ); and storing a transition tuple in a main buffer, the transition tuple comprising: TIFF0007602027000001.tif6170, where TIFF0007602027000002.tif6170 is the reward vector for each task in the environment, and s t+1 is an action t), where the next environment state is a next environment state after the main buffer; updating transition priorities for each transition tuple stored in the main buffer; sampling mini-batches of transition tuples to update a task network; and optimizing task policies from the updated task network using an off-policy algorithm.

[0009] In some embodiments, the tasks in the environment are unknown to the agent.

[0010] In some embodiments, the control loop is episodic, and after each episode the agent's state is reset to an initial state.

[0011] In some embodiments, the control loop is continuous, where the agent performs actions without resetting the agent's state.

[0012] An embodiment of the present invention is a method for training an agent, the method comprising the steps of: t ) for behavioral policy (π b ) sampled from (a t ); and storing a transition tuple in a main buffer, the transition tuple comprising: TIFF0007602027000003.tif6170, where TIFF0007602027000004.tif6170 is the reward vector for each task in the environment, s t+1 is an action tupdating a transition priority for each transition tuple stored in the main buffer; sampling a mini-batch of transition tuples to update a task network; and optimizing a task policy from the updated task network using an off-policy algorithm, wherein transitions belonging to a set of transition indicators that result in the achievement of task-j during the i-th episode are given higher priority than transitions that do not result in the achievement of task-j during the i-th episode.

[0013] An embodiment of the present invention also provides a non-transitory computer readable storage medium having stored thereon an executable program, the program being configured to execute, by the agent, an observation (s t ) for behavioral policy (π b ) sampled from (a t ); and storing a transition tuple in a main buffer, the transition tuple comprising: TIFF0007602027000005.tif6170, where TIFF0007602027000006.tif6170 is the reward vector for each task in the environment, s t+1 is an action t updating a transition priority for each transition tuple stored in the main buffer; sampling a mini-batch of transition tuples to update a task network; and optimizing a task policy from the updated task network using an off-policy algorithm, wherein transitions belonging to a set of transition indicators that result in the achievement of task-j during the i-th episode are given higher priority than transitions that do not result in the achievement of task-j during the i-th episode.

[0014] These and other features, aspects, and advantages of the present invention will become better understood with reference to the following drawings, description, and claims.

[0015] Some embodiments of the present invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like reference symbols may indicate similar elements and in which: [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a control flow diagram between an environment and an agent with TaPER according to an exemplary embodiment of the present invention. [Diagram 2] FIG. 2 illustrates the internals of an off-policy enforcement agent using TaPER, according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Unless otherwise indicated, pictorial illustrations are not necessarily to scale.

[0018] A better understanding of the invention and its various embodiments can be obtained by reference to the following detailed description which describes the illustrated embodiments, with the express understanding that the illustrated embodiments are presented by way of example and are not intended to limit the invention as ultimately defined by the claims.

[0019] The terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting of the present invention. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. As used herein, the singular forms "a," "an," and "the" are intended to include the plural in addition to the singular, unless the context clearly indicates otherwise. Furthermore, it should be understood that the terms "comprises" and / or "comprising," as used herein, indicate the presence of stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0020] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by those skilled in the art to which the present invention belongs. Furthermore, terms as defined in commonly used dictionaries should be interpreted to have a meaning in accordance with these meanings in the context of the relevant art and this disclosure, and should not be interpreted in an ideal or overly formal sense unless expressly defined herein.

[0021] In the description of the present invention, it will be understood that multiple techniques and steps are disclosed. Each of these has its own advantages and can also be used with one or more, or in some cases all, of the other techniques disclosed. Thus, for the sake of clarity, the description will not unnecessarily repeat all possible combinations of the individual steps. However, the specification and claims should be read with the understanding that such combinations are fully within the scope of the present invention and the claims.

[0022] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without these specific details.

[0023] The present disclosure should be considered as an exemplification of the invention and is not intended to limit the invention to the specific embodiments shown in the following figures or description.

[0024] Devices or system modules that are in at least general communication with each other need not be in continuous communication with each other, unless otherwise specified, and devices or system modules that are in at least general communication with each other may communicate directly or indirectly through one or more intermediary devices.

[0025] A description of an embodiment including multiple components in communication with each other does not imply that all such components are required, but rather a variety of optional components are described to illustrate the wide range of possible embodiments of the present invention.

[0026] A "computer" or "computing device" may mean one or more devices and / or one or more systems that can accept structured input, process the structured input according to prescribed rules, and produce the results of the processing as output. Examples of computers or computing devices may include computers, fixed and / or portable computers, computers with a single processor, multiple processors, or multi-core processors capable of operating in parallel and / or non-parallel, supercomputers, mainframes, superminicomputers, minicomputers, workstations, microcomputers, servers, clients, interactive televisions, web appliances, communications devices with Internet access, hybrid combinations of computers and interactive televisions, portable computers, tablet personal computers (PCs), personal digital assistants (PDAs), mobile phones, application specific hardware that emulates a computer and / or software, such as digital signal processors (DSPs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific instruction set processors (ASIPs), chips, multiple chips, systems on a chip or chipsets, data acquisition devices, optical computers, quantum computers, biocomputers, and generally devices that accept data, process the data in accordance with one or more stored software programs, and generate results, and which may typically include input devices, output devices, storage devices, arithmetic units, logic units, and control units.

[0027] "Software" or "application" can mean prescribed rules for operating a computer. Examples of software or applications can include code segments, graphical and / or textual instructions, applets, precompiled code, interpreted code, compiled code, and computer programs in one or more computer readable languages.

[0028] These computer program instructions, which can direct a computer, other programmable data processing apparatus, or other device to function in a particular manner, can also be stored on a computer readable medium such that the instructions stored on the computer readable medium produce an article of manufacture that includes instructions that perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0029] Additionally, although process steps, method steps, algorithms, or the like may be described in a certain order, such processes, methods, and algorithms may be configured to work in other orders. In other words, any order or sequence of steps that may be described does not necessarily indicate that the steps must be performed in this order. Steps of processes described herein may be performed in any order that is practical. Additionally, some steps may be performed simultaneously.

[0030] It will be readily apparent that the various methods and algorithms described herein can be implemented, for example, by appropriately programmed general purpose computers and computing devices. Typically, a processor (e.g., a microprocessor) receives instructions from a memory or similar device and executes those instructions to perform the process defined by those instructions. Furthermore, programs that implement such methods and algorithms can be stored and transmitted using a variety of known media.

[0031] The term "computer-readable medium" as used herein means any medium that participates in providing data (e.g., instructions) that may be read by a computer, a processor, or a similar device. Such media may take many forms, including, but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical or magnetic disks and other persistent memory. Volatile media include dynamic random access memory (DRAM), which typically constitutes the main memory. Transmission media include coaxial cables, copper wire, and fiber optics, including the wires that comprise a system bus coupled to the processor. Transmission media may include or convey acoustic waves, light waves, and electromagnetic radiation, such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer readable media include, for example, a floppy disk, a flexible disk, a hard disk, a magnetic tape, any other magnetic medium, a CD-ROM, a DVD, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, RAM, PROM, EPROM, FLASHEEPROM, any other memory chip or cartridge, a carrier wave as described below, or any other medium from which a computer can read.

[0032] Various forms of computer readable media may be involved in carrying the set of instructions to the processor, for example, the set of instructions may (i) be provided to the processor from a RAM, (ii) be carried over a wireless transmission medium, and / or (iii) be formatted according to a number of formats, standards, or protocols, such as Bluetooth, TDMA, CDMA, 3G, etc.

[0033] Embodiments of the present invention may include apparatuses for performing the operations disclosed herein, which may be specially constructed for the desired purposes, or which may comprise general-purpose devices selectively activated or reconfigured by programs stored in the devices.

[0034] Unless otherwise stated, and as will become apparent from the following description and claims, throughout this specification, descriptions using terms such as "processing," "calculating," "computing," or "determining" should be understood to mean operations and / or processes of a computer, computer system, or similar electronic computing device that manipulates data represented as physical quantities, such as electronic quantities in a computer system's registers and / or memory, and / or transforms data into other data that is similarly represented as physical quantities in a computer system's memory, registers, or other such information storage, transmission, or display device.

[0035] Similarly, the term "processor" may refer to any device or part of a device that processes electronic data from registers and / or memory and converts this electronic data into other electronic data that can be stored in registers and / or memory or transmitted to an external device so as to cause a physical change or action in an external device.

[0036] The term "agent" or "intelligent agent" or "artificial agent" or "artificial intelligence agent" is intended to mean any artificial entity that selects an action in response to an observation. "Agent" can mean, without limitation, a robot, a simulated robot, a software agent or "bot", an adaptive agent, an internet or web bot.

[0037] The term "robot" can refer to any system controlled directly or indirectly by a computer or computer system that issues actions or commands in response to sensing or observation. The term can refer, without limitation, to a traditional physical robot having physical sensors such as cameras, touch sensors, and distance sensors, or a simulated robot that exists in a virtual simulation, or a "bot" such as a mailbot or searchbot that exists as software in a network. The term can also refer, without limitation, to any limbed robot, walking robot, industrial robot (including, but not limited to, robots used for automating assembly, painting, repair, maintenance, etc.), wheeled robot, cleaning or lawnmower robot, personal assistant robot, service robot, medical or surgical robot, flying robot, driving robot, aircraft or spacecraft robot, or any other robot that operates substantially under autonomous control, including stationary robots such as vehicular or otherwise, real or simulated, intelligent household or work equipment.

[0038] The term "observation" refers to any information received by an agent by any means about the agent's environment or the agent itself. In some embodiments, this information can be sensory information or signals received through sensory devices, such as, but not limited to, cameras, touch sensors, distance sensors, temperature sensors, wavelength sensors, acoustic or voice sensors, position sensors, pressure or force sensors, speed or acceleration or other motion sensors, location sensors (e.g., GPS), etc. In other embodiments, this information can also include, but is not limited to, compiled, abstract or situational information created from a combination of a set of sensory devices and stored information. In a non-limiting example, an agent can receive abstract information about the location or characteristics of itself or other objects as observations. In some embodiments, this information can refer to people or customers, or characteristics such as their purchasing habits, personal contact information, personal preferences, etc. In some embodiments, an observation can be information about the inside of the agent, such as, but not limited to, proprioceptive information or other information about the agent's current or past actions, information about the agent's internal state, or information already calculated or processed by the agent.

[0039] The term "behavior" refers to any means by which an agent can control, affect, or influence the agent's environment, the agent's physical or simulated self, or the agent's internal functions, ultimately controlling or influencing the agent's future actions, action selections, or behavioral preferences. In many embodiments, these actions can directly control physical or simulated servos or actuators. In some embodiments, these actions can be expressions of preferences or sets of preferences ultimately intended to influence the agent's choices. In some embodiments, information regarding the agent's action(s) can include, but is not limited to, probability distributions over the agent's action(s) and / or outgoing information intended to influence the agent's ultimate action selection.

[0040] The terms "state" or "state information" refer to any collection of information regarding the state of the environment or an agent, which may include, but is not limited to, information regarding the current and / or past observations of the agent.

[0041] The term "policy" refers to any function or mapping from any complete or partial state information to any action information. A policy can be hard coded or modified, adapted or trained using any suitable learning or teaching method, including but not limited to any reinforcement learning or control optimization method. A policy can be an explicit or implicit mapping, such as but not limited to, one that may result from optimizing a particular measure, value or function. A policy can include associated further information, features or characteristics, such as but not limited to, starting conditions (or probabilities) reflecting under what conditions the policy can be started or continued, and ending conditions (or probabilities) reflecting under what conditions the policy can be terminated.

[0042] In general, embodiments of the present invention provide a Task Prioritized Experience Replay (TaPER) algorithm that enables the simultaneous learning of multiple RL tasks in an off-policy manner. The algorithm can prioritize samples that were part of fixed-length episodes that led to the accomplishment of a task. This allows the agent to quickly learn task policies by bootstrapping over early successes. Finally, TaPER can improve performance on all tasks simultaneously, which is a desirable property for multitask RL. Unlike traditional ER algorithms that are applied in single RL task learning settings, or that require rewards to be binary or abundant, or that are provided as parameterized specifications of goals, TaPER does not impose such constraints and supports arbitrary reward and task specifications.

[0043] TaPER meets several desirable properties for multitask RL. For example, TaPER improves performance on all tasks simultaneously, which is desirable when sharing parameters between task networks. TaPER supports transfer learning between tasks by sharing data prioritized for one task with the other. Agents with TaPER learn task policies quickly by bootstrapping over early successes compared to uniform experience replay. Finally, TaPER uses a probabilistic prioritization method, which helps resist overfitting, especially early when only a small amount of experience has been collected.

[0044] As described in more detail below, embodiments of the invention generally relate to systems and computerized methods for prioritized off-policy RL that use a task-prioritized experience replay algorithm. The systems and computerized methods provide technological improvements in the efficiency and accuracy of computer systems by enabling the simultaneous learning of multiple RL tasks off-policy, prioritizing samples that were part of fixed-length episodes that led to task accomplishment, allowing agents to rapidly learn task policies by bootstrapping across early successes, and improving performance on all tasks simultaneously.

[0045] We describe herein the algorithmic implementation of TaPER in the context of a multitask RL setting with reference to the following notations: n is the number of tasks available to learn. s t is the observation of the environmental state at time t. a t is the action choice that can be performed in the environment at time t. π b (s t ) is the observation t is the action probability distribution parameter of the agent's behavior policy for π b (s t ) Actions sampled from t Execute. TIFF0007602027000007.tif6170 is the reward vector TIFF0007602027000008.tif6170, where each component TIFF0007602027000009.tif6170 is the scalar reward signal received by the agent in the i-th task. TIFF0007602027000010.tif6170 is the given observation t is a termination function for each task j that returns a binary value 0 or 1 for βj If it returns 1, the task is finished, otherwise it returns 0. TIFF0007602027000011.tif6170 shows the binary exit vector for all tasks. The exit value of a task is the agent's behavior policy π b (s t ) may not be affected. Transitions are tuples means TIFF0007602027000012.tif6170. Episode: A sequence of T transitions, where T is a fixed integer. TIFF0007602027000013.tif77 is the main buffer that stores all observed transitions. TIFF0007602027000014.tif6170 is an episode buffer that stores indices of transitions observed during the ongoing episode. The task to be accomplished is conditional, within the context of the episode. This means that task-j is considered to be accomplished when TIFF0007602027000015.tif6170 is true. TIFF0007602027000016.tif6150 shows the set of transition indicators that led to the accomplishment of task-j during the i-th episode. TIFF0007602027000017.tif6170 shows the union over all tasks, TIFF0007602027000018.tif6170 shows the union over all tasks and episodes. |X| denotes the number of samples stored in buffer X. For example, TIFF0007602027000019.tif911 shows the number of transitions stored in the main buffer. p i is the priority value of the transition with index i that is stored in the main buffer. P(i) is the probability of sampling a transition containing index i from the main buffer. TIFF0007602027000020.tif714 shows a mini-batch B of transitions sampled from D with a probability distribution P defined over the transition indices. π i (s t ) is the observation t is the action probability distribution parameter of the policy for update task i. i When trained, it solves the task optimally. TIFF0007602027000021.tif6170 is the task policy π i This is an off-policy RL algorithm that optimizes the network.

[0046] TaPER makes the following assumptions about the environment: The agent interacts with the environment at discrete time steps. Starting at t = 0, the agent is presented with n tasks to solve. The agent knows the number n, but does not know the task description. At any given time t, the agent can observe the reward signals for all tasks. That is, the agent Find the value of TIFF0007602027000022.tif6150. For any given time t and corresponding observation s t In this case, the agent determines the final value β(s t ) can be observed. That is, the agent can Find the value of TIFF0007602027000023.tif6170. Agent behavior policy π b requires that the agent be exploratory enough to reach the end state of each task at least once.

[0047] Referring to Figure 1, there is shown an overview of an agent 10 that uses TaPER to learn in a multi-task RL setting. Figure 1 shows a high-level control flow between the agent 10 and an environment 12. The environment 12 may contain n task descriptions, namely task-1 14 through task-n 16, that are unknown to the agent. At any time t and given the observed environment state s t Now, agent 10 takes action a t You can run the following environmental conditions: t+1 Observe that agent 10 receives scalar rewards from all tasks. TIFF0007602027000024.tif6170 and binary end value You can also receive TIFF0007602027000025.tif6170. Transition tuple TIFF0007602027000026.tif6170 and end vector TIFF0007602027000027.tif6170 is used to update the inside of the agent 10. This control loop 18 can continue indefinitely or until all tasks are considered solved.

[0048] The agent's control loop 18 can be broadly classified as (a) episodic or (b) continuous. In the episodic case, at the end of each episode, the agent's state is reset to an initial state (which may be different from the last observed state). An episode in this case has non-overlapping transitions. On the other hand, in the continuous case, the agent continues to execute actions without resetting. This can be treated as a special case of the episodic case, where the agent's initial state is the last observed state. An episode in a continuous loop is considered to be a moving window of T-1 overlapping transitions.

[0049] The agent can include four operational blocks, shown clockwise in FIG. 2 and summarized as follows: (1) Block 20, which samples the behavior of an agent, samples the observed t Behavioral strategies for b Actions sampled from a t (2) The transition storage block 22 is a main buffer. The new transition can be stored in TIFF0007602027000028.tif77. (3) The update priority and sample mini-batch (B) block 24 can update the transition priorities and sample mini-batch B of transitions. (4) The optimize task policy block 26 can optimize the task priorities by using the off-policy algorithm. TIFF0007602027000029.tif6170 and mini-batch B can be used to perform the optimization step of the task strategy.

[0050] Behavioral Sampling With respect to sample action block 20, the agent 10 selects an exploratory behavior strategy π that it uses to sample and execute actions at each time step. b The behavior policy has a task policy π i , It can be independent of TIFF0007602027000030.tif6170. The default choice for the behavioral strategy is a uniform random strategy. However, in complex environments, a uniform random strategy may not be sufficient to explore effectively. Other choices include using an optimistic bias for unseen regions, human-designed strategies, etc.

[0051] Transition Memory With respect to the transition storage block 22, the actions sampled from the behavior policy are used to generate new transitions. Build TIFF0007602027000031.tif6170 and save it as the main buffer The main buffer can be stored in TIFF0007602027000032.tif67. The main buffer can be implemented using an unsized queue container, or a finite-sized first-in-first-out queue if there are memory constraints. The TaPER algorithm is agnostic to the memory design choice of the main buffer. The algorithm only needs to index and order the transitions in time, where the new transition has the highest index (for now) associated with it.

[0052] Priority updates and mini-batch sampling Regarding the update priorities and sample mini-batches block 24, this block 24 has two subparts: (a) updating transition priorities and (b) sampling mini-batches.

[0053] (a) Updating transition priority Regarding the subpart of updating transition priorities, the following details how transition priorities are calculated: i is the episode buffer for the i-th episode The set of transition indicators that led to the achievement of task-j during the i-th episode is given by Eq. (1), TIFF0007602027000034.tif19160, where TIFF0007602027000035.tif6170 shows the set of transition indices from the start of the episode up to t time steps. TIFF0007602027000036.tif6170 is the union over all tasks and observed episodes, as in equation (2). TIFF0007602027000037.tif18158One focus of TaPER is, The transition belonging to TIFF0007602027000038.tif6170 is The first variant is to set a priority value for each transition, as in equation (3): Set a constant priority value for all transitions belonging to TIFF0007602027000040.tif6170. TIFF0007602027000041.tif19161

[0054] Small non-zero priority values ​​for transitions not included in TIFF0007602027000042.tif6170 TIFF0007602027000043.tif6170 does not exclude the transition from being sampled completely.

[0055] A second variant of the prioritization is also possible, where the magnitude of the time difference error δ of the transitions is used to determine: Prioritize also among the samples contained in TIFF0007602027000044.tif6170. TIFF0007602027000045.tif19160

[0056] The time-difference error δ is a reasonable proxy for the learning progress of an RL algorithm. For example, in a standard Q-learning algorithm, δ is calculated as r+Q(s t+1 , argmax a Q(s t+1 , a)) - Q(s t , a t ) Thus, by using the magnitude of δ as a priority value, transitions that are useful for speeding up learning are preferred over transitions that are not. Add TIFF0007602027000046.tif7170 and set the priority Ensure that TIFF0007602027000047.tif6170 is non-zero.

[0057] There is a trade-off between both variants. The second variant seems more appealing since it can potentially speed up learning by focusing on transitions that result in faster learning progress. However, the second variant is computationally expensive since the priorities need to be constantly updated after each optimization step, since δ changes over time. This may also affect network latency if the buffers are stored remotely in the cloud. In the first variant, the priorities are set only once, so computation is not an issue. Depending on the network and computation budget, choosing the variant is a design choice of the TaPER algorithm. The following description relies on the use of the first variant.

[0058] (b) Mini-batch sampling Using the set priorities, we can then sample mini-batches of transitions to update the task network. In particular, the set When the size of TIFF0007602027000048.tif6170 is small, simply using the priority value to sample the mini-batch results in a small set This can lead to premature convergence of the strategy over time (overfitting). To address these issues, a stochastic prioritization technique can be used that interpolates between greedy prioritization and uniform random sampling, as in Equation (5): TIFF0007602027000050.tif19160, where TIFF0007602027000051.tif6170 determines how much prioritization should be used. iSince α is nonzero, all transitions have a nonzero probability of being selected. By setting α=0, TaPER reduces to a uniform sampling ER. It is advisable to set α closer to 0 initially and increase its value to 1 over time. This forces TaPER to behave more like a uniform ER in the early stages, and reduces the probability that TaPER will be taken for small Try to avoid overfitting across the transitions referenced in TIFF0007602027000052.tif6170. Use the transition probabilities to Optimize the task policy by sampling mini-batch B of transitions from TIFF0007602027000053.tif96.

[0059] Task Policy Optimization Using a mini-batch B of sampled transitions, we apply the off-policy algorithm Update the task policy network using TIFF0007602027000054.tif6170. For the Temporal Difference (TD) Error Prioritization variant (equation (4) above), We update the priorities for the transitions in B using the TD error calculated by TIFF0007602027000055.tif6170. TaPER is a method It is agnostic to the selection of TIFF0007602027000056.tif6170.

[0060] Algorithm 1 below formally describes the episodic TaPER algorithm in the context of a multitask RL setting. In the continuous case, instead of a hard reset at the end of each episode, we use Implement TIFF0007602027000057.tif6170 as a moving window of size T. TIFF0007602027000058.tif135170

[0061] As described above, the Task Prioritized Experience Replay (TaPER) algorithm enables the simultaneous learning of multiple RL tasks in an off-policy manner. The algorithm prioritizes samples that were part of fixed-length episodes that led to the accomplishment of a task. This allows the agent to quickly learn a task policy by bootstrapping over early successes. Finally, the TaPER algorithm improves performance on all tasks simultaneously, which is a desirable property for multitask RL.

[0062] Those skilled in the art may make many changes and modifications without departing from the spirit and scope of the present invention. It is therefore to be understood that the illustrated embodiments are presented by way of example only and should not be construed as limiting the invention as defined by the following claims. For example, although the elements of the claims are presented below in specific combinations, it should be expressly understood that the invention includes other combinations of elements that are less, more, or different than those disclosed.

[0063] The words used herein to describe the invention and its various embodiments are to be understood to include not only their generally defined meaning, but also the generic structures, materials or acts for which they represent a single species by specific definitions herein.

[0064] Therefore, the definition of the words or elements in the following claims is set forth herein to include more than just the combination of elements expressly stated in the text. In this sense, it is therefore considered that the equivalent substitute of two or more elements may be used in place of any one of the elements in the following claims, or a single element may be used in place of two or more elements in the claims. Although elements may be described above as functioning in a particular combination and may be initially claimed in this manner, it is to be clearly understood that in some cases, one or more elements resulting from the claimed combination may be deleted from these combinations, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0065] The claims are therefore to be understood to include what is specifically illustrated and described above, what is conceptually equivalent, what is obviously substituted, and what incorporates the basic idea of ​​the invention. [Explanation of symbols]

[0066] 10. Agent 12 Environment 14 Task-1 16 Task-n 18 Control Loop 20 Sampling Behavior 22 Remember Transitions 24 Update priorities and sample mini-batches 26 Optimize task policies

Claims

1. 1. A computer-implemented method for training an agent in a control loop, comprising: The agent observes t ) for the behavior policy (π b ) sampled from the action (a t ) Storing a transition tuple in a main buffer, the transition tuple comprising: where is the reward vector for each task in the environment, and s t+1 is an action (a t ) the next environmental state after updating a transition priority for each transition tuple stored in the main buffer; updating the task network by sampling a mini-batch of transition tuples; optimizing a task policy from the updated task network using an off-policy algorithm; Including, The set of transition indices that lead to the accomplishment of task-j during the i-th episode is given by: Where: denotes the set of transition indices from the start of the episode up to t time steps, is the union over all tasks and observed episodes.

2. 2. The method of claim 1, further comprising the step of continuing the control loop until all of the tasks in the environment are resolved.

3. 2. The method of claim 1, wherein the control loop is episodic and after each episode, the agent's state is reset to an initial state.

4. 2. The method of claim 1, wherein the control loop is continuous, where the agent executes actions without resetting the agent's state.

5. The method of claim 1 , wherein the behavioral policy is a uniform random policy.

6. The transition belonging to 2. The method of claim 1, wherein a transition is given higher priority than a transition that is not included in 7. The method of claim 6, wherein the priority values ​​for the transitions belonging to are assigned constant values.

8. The method of claim 7, wherein transitions not included in are given a non-zero priority value.

9. The method of claim 6, wherein the priority values ​​for the transitions belonging to are assigned variable values ​​based on the magnitude of the time difference error of each transition.

10. The sampling of the mini-batches is performed using a stochastic prioritization technique that interpolates between greedy prioritization and uniform random sampling according to the following formula: where P(i) is the probability of sampling a transition containing index i from the main buffer, and p i is the priority value of the transition containing the index i from the main buffer, 2. The method of claim 1, wherein: determines how much prioritization should be used.

11. 2. The method of claim 1, wherein the step of optimizing the task policy is agnostic to the selection of the off-policy algorithm.

12. 1. A computer-implemented method for training an agent, comprising: The agent observes t ) for the behavior policy (π b ) sampled from the action (a t ) Storing a transition tuple in a main buffer, the transition tuple comprising: where is the reward vector for each task in the environment, and s t+1 is an action (a t ) the next environmental state after updating a transition priority for each transition tuple stored in the main buffer; updating the task network by sampling a mini-batch of transition tuples; optimizing a task policy from the updated task network using an off-policy algorithm; Including, A transition belonging to a set of transition indicators that results in the achievement of task-j during the i-th episode is given a higher priority than a transition that does not result in the achievement of task-j during said i-th episode; The set of transition indicators that result in the accomplishment of task-j during the i-th episode is given by: Where: denotes the set of transition indices from the start of the episode up to t time steps, is the union over all tasks and observed episodes.

13. 13. The method of claim 12, characterized in that priority values ​​for the transitions that belong to the set of transition indicators that result in the achievement of task-j during the i-th episode are given a constant value, and transitions that are not included in the set of transition indicators that result in the achievement of task-j during the i-th episode are given a non-zero priority value.

14. 13. The method of claim 12, wherein the priority values ​​for the transitions in the set of transition indicators that result in the accomplishment of task-j during the i-th episode are given variable values ​​based on the magnitude of the time difference error of each transition.

15. A non-transitory computer readable storage medium having stored thereon an executable program, the executable program comprising: The agent observes t ) for the behavior policy (π b ) sampled from the action (a t ) Storing a transition tuple in a main buffer, the transition tuple comprising: where is the reward vector for each task in the environment, and s t+1 is an action (a t ) the next environmental state after updating a transition priority for each transition tuple stored in the main buffer; updating the task network by sampling a mini-batch of transition tuples; optimizing a task policy from the updated task network using an off-policy algorithm; instructing one or more processors to execute A transition belonging to a set of transition indicators that results in the achievement of task-j during the i-th episode is given a higher priority than a transition that does not result in the achievement of task-j during said i-th episode; The set of transition indicators that result in the accomplishment of task-j during the i-th episode is given by: Where: denotes the set of transition indices from the start of the episode up to t time steps, 16. A non-transitory computer-readable storage medium, wherein: x is the union over all tasks and observed episodes.

16. 16. The non-transitory computer-readable storage medium of claim 15, wherein: (a) priority values ​​for the transitions in the set of transition indicators that result in the achievement of task-j during the i-th episode are assigned a constant value and transitions not included in the set of transition indicators that result in the achievement of task-j during the i-th episode are assigned a non-zero priority value; or (b) priority values ​​for the transitions in the set of transition indicators that result in the achievement of task-j during the i-th episode are assigned a variable value based on a magnitude of a time difference error for each transition.

Images