Training actor-critic algorithms in a laboratory setting

Abstract

Reinforcement learning methods can use actor-critic networks that (1) use additional lab-specific state information to train policies that work well in production settings without this additional lab-specific information, or (2) distill complex, resource-demanding policies into less demanding policies that can be more easily implemented in production with limited computational resources. A frozen version of a large critic network previously trained with a large actor network can be used to optimize a production actor network. Aspects of these methods can utilize actor-critic methods, in which the critic network models an action-value function as opposed to a state-value function.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE Embodiments of the present invention relate generally to reinforcement learning methods, and more particularly, to systems and methods for improving the performance of actor networks when training actor-critic models. [Background technology]

[0002] The following background information may present examples of certain aspects (e.g., but not limited to, techniques, facts, or conventional wisdom) of the prior art that are expected to serve to further inform the reader of additional aspects of the prior art, but should not be construed as limiting the invention or any embodiment thereof to anything stated or implied therein or inferred therefrom.

[0003] Reinforcement Learning (RL) refers to a class of problems (and solutions) in which an artificial agent optimizes the control of a system (or "environment") when the dynamics of the system are unknown and must be learned by the agent through interaction and observation. Typically, RL problems and solutions consider the learning and optimization to be performed within a single continuous lifetime of the agent. However, because the initial performance of learning agents is poor and often unsafe in industrial settings, industrial applications of RL benefit from separating RL into two phases: (1) a learning phase that occurs in a safe laboratory, and (2) a production phase where the lab-trained controller, or "policy," is deployed without any subsequent learning. Summary of the Invention [Problem to be solved by the invention]

[0004] In view of the above, there is a need for systems and methods that better utilize these two phases to improve reinforcement learning, especially in agents with limited computational resources. [Means for solving the problem]

[0005] An embodiment of the invention provides a method for training an agent, the method including: training a policy using an actor network and a critic network using state information available in both a laboratory setting and a production setting, where the state information available in the laboratory setting is greater than the state information available in the production setting; optimizing the critic network using an actor network; and providing a production actor network for the agent in the production setting.

[0006] In some embodiments, the critic network is trained based on an action value function.

[0007] In some embodiments, the method further includes replicating the critic network into a frozen critic network when the training is complete, and optimizing the production actor network using the frozen critic network.

[0008] In some embodiments, a first actor-critic algorithm is executed during the training of the critic network, and a second actor-critic algorithm is executed during the optimizing the production actor network using the frozen critic network, in some embodiments, the first actor-critic algorithm is the same as the second actor-critic algorithm.

[0009] An embodiment of the present invention further provides a method of training an agent, the method including the steps of training a policy using an actor network and a critic network, optimizing the critic network using an actor network, providing a production actor network for the agent in a production setting, replicating the critic network to a frozen critic network when the training is complete, and optimizing the production actor network using the frozen critic network, where the production actor network is smaller than the actor network.

[0010] An embodiment of the invention also provides a non-transitory computer-readable storage medium having stored thereon an executable program that instructs one or more processors to perform the steps of: training a policy using an actor network and a critic network using state information available in both a laboratory setting and a production setting, where the state information available in the laboratory setting is greater than the state information available in the production setting; optimizing the critic network using an actor network; and providing a production actor network for the agents in the production setting.

[0011] These and other features, aspects, and advantages of the present invention will become better understood with reference to the following drawings, description, and claims.

[0012] Some embodiments of the present invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like reference symbols may indicate similar elements and in which: [Brief description of the drawings]

[0013] [Figure 1]Schematic diagram of the actor-critic model of reinforcement learning. [Diagram 2] This is a schematic diagram of how to train a critic network using state information available in both laboratory and production settings, while training an actor network using state information available in the production setting. [Diagram 3] FIG. 1 illustrates a conventional process of optimizing a large actor network using an actor-critic approach and then distilling production-ready actors directly from the large actor network. [Figure 4] Figure 1 illustrates a process according to an exemplary embodiment of the invention in which the actor-critic algorithm is run twice: first, using a large actor network to help optimize the critic network, and then optimizing a production-ready actor network by reusing a frozen copy of the optimized critic network. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Unless otherwise indicated, pictorial illustrations are not necessarily to scale.

[0015] A better understanding of the invention and its various embodiments can be obtained by reference to the following detailed description which describes the illustrated embodiments, with the express understanding that the illustrated embodiments are presented by way of example and are not intended to limit the invention as ultimately defined by the claims.

[0016] The terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting of the present invention. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. As used herein, the singular forms "a," "an," and "the" are intended to include the plural in addition to the singular, unless the context clearly indicates otherwise. Furthermore, it should be understood that the terms "comprises" and / or "comprising," as used herein, indicate the presence of stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0017] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by those skilled in the art to which the present invention belongs. Furthermore, terms as defined in commonly used dictionaries should be interpreted to have a meaning in accordance with these meanings in the context of the relevant art and this disclosure, and should not be interpreted in an ideal or overly formal sense unless expressly defined herein.

[0018] In the description of the present invention, it will be understood that multiple techniques and steps are disclosed. Each of these has its own advantages and can also be used with one or more, or in some cases all, of the other techniques disclosed. Thus, for the sake of clarity, the description will not unnecessarily repeat all possible combinations of the individual steps. However, the specification and claims should be read with the understanding that such combinations are fully within the scope of the present invention and the claims.

[0019] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without these specific details.

[0020] The present disclosure should be considered as an exemplification of the invention and is not intended to limit the invention to the specific embodiments shown in the following figures or description.

[0021] Devices or system modules that are in at least general communication with each other need not be in continuous communication with each other, unless otherwise specified, and devices or system modules that are in at least general communication with each other may communicate directly or indirectly through one or more intermediary devices.

[0022] A description of an embodiment including multiple components in communication with each other does not imply that all such components are required, but rather a variety of optional components are described to illustrate the wide range of possible embodiments of the present invention.

[0023] A "computer" or "computing device" may mean one or more devices and / or one or more systems that can accept structured input, process the structured input according to prescribed rules, and produce the results of the processing as output. Examples of computers or computing devices may include computers, fixed and / or portable computers, computers with a single processor, multiple processors, or multi-core processors capable of operating in parallel and / or non-parallel, supercomputers, mainframes, superminicomputers, minicomputers, workstations, microcomputers, servers, clients, interactive televisions, web appliances, communications devices with Internet access, hybrid combinations of computers and interactive televisions, portable computers, tablet personal computers (PCs), personal digital assistants (PDAs), mobile phones, application specific hardware that emulates a computer and / or software, such as digital signal processors (DSPs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific instruction set processors (ASIPs), chips, multiple chips, systems on a chip or chipsets, data acquisition devices, optical computers, quantum computers, biocomputers, and generally devices that accept data, process the data in accordance with one or more stored software programs, and generate results, and which may typically include input devices, output devices, storage devices, arithmetic units, logic units, and control units.

[0024] "Software" or "application" can mean prescribed rules for operating a computer. Examples of software or applications can include code segments, graphical and / or textual instructions, applets, precompiled code, interpreted code, compiled code, and computer programs in one or more computer readable languages.

[0025] These computer program instructions, which can direct a computer, other programmable data processing apparatus, or other device to function in a particular manner, can also be stored on a computer readable medium such that the instructions stored on the computer readable medium produce an article of manufacture that includes instructions that perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0026] Additionally, although process steps, method steps, algorithms, or the like may be described in a certain order, such processes, methods, and algorithms may be configured to work in other orders. In other words, any order or sequence of steps that may be described does not necessarily indicate that the steps must be performed in this order. Steps of processes described herein may be performed in any order that is practical. Additionally, some steps may be performed simultaneously.

[0027] It will be readily apparent that the various methods and algorithms described herein can be implemented, for example, by appropriately programmed general purpose computers and computing devices. Typically, a processor (e.g., a microprocessor) receives instructions from a memory or similar device and executes those instructions to perform the process defined by those instructions. Furthermore, programs that implement such methods and algorithms can be stored and transmitted using a variety of known media.

[0028] The term "computer-readable medium" as used herein means any medium that participates in providing data (e.g., instructions) that may be read by a computer, a processor, or a similar device. Such media may take many forms, including, but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical or magnetic disks and other persistent memory. Volatile media include dynamic random access memory (DRAM), which typically constitutes the main memory. Transmission media include coaxial cables, copper wire, and fiber optics, including the wires that comprise a system bus coupled to the processor. Transmission media may include or convey acoustic waves, light waves, and electromagnetic radiation, such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer readable media include, for example, a floppy disk, a flexible disk, a hard disk, a magnetic tape, any other magnetic medium, a CD-ROM, a DVD, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, RAM, PROM, EPROM, FLASHEEPROM, any other memory chip or cartridge, a carrier wave as described below, or any other medium from which a computer can read.

[0029] Various forms of computer readable media may be involved in carrying the set of instructions to the processor, for example, the set of instructions may (i) be provided to the processor from a RAM, (ii) be carried over a wireless transmission medium, and / or (iii) be formatted according to a number of formats, standards, or protocols, such as Bluetooth, TDMA, CDMA, 3G, etc.

[0030] Embodiments of the present invention may include apparatuses for performing the operations disclosed herein, which may be specially constructed for the desired purposes, or which may comprise general-purpose devices selectively activated or reconfigured by programs stored in the devices.

[0031] Unless otherwise stated, and as will become apparent from the following description and claims, throughout this specification, descriptions using terms such as "processing," "calculating," "computing," or "determining" should be understood to mean operations and / or processes of a computer, computer system, or similar electronic computing device that manipulates data represented as physical quantities, such as electronic quantities in a computer system's registers and / or memory, and / or transforms data into other data that is similarly represented as physical quantities in a computer system's memory, registers, or other such information storage, transmission, or display device.

[0032] Similarly, the term "processor" may refer to any device or part of a device that processes electronic data from registers and / or memory and converts this electronic data into other electronic data that can be stored in registers and / or memory or transmitted to an external device so as to cause a physical change or action in an external device.

[0033] The term "agent" or "intelligent agent" or "artificial agent" or "artificial intelligence agent" is intended to mean any artificial entity that selects an action in response to an observation. "Agent" can mean, without limitation, a robot, a simulated robot, a software agent or "bot", an adaptive agent, an internet or web bot.

[0034] The term "robot" can refer to any system controlled directly or indirectly by a computer or computer system that issues actions or commands in response to sensing or observation. The term can refer, without limitation, to a traditional physical robot having physical sensors such as cameras, touch sensors, and distance sensors, or a simulated robot that exists in a virtual simulation, or a "bot" such as a mailbot or searchbot that exists as software in a network. The term can also refer, without limitation, to any limbed robot, walking robot, industrial robot (including, but not limited to, robots used for automating assembly, painting, repair, maintenance, etc.), wheeled robot, cleaning or lawnmower robot, personal assistant robot, service robot, medical or surgical robot, flying robot, driving robot, aircraft or spacecraft robot, or any other robot that operates substantially under autonomous control, including stationary robots such as vehicular or otherwise, real or simulated, intelligent household or work equipment.

[0035] The term "observation" refers to any information received by an agent by any means about the agent's environment or the agent itself. In some embodiments, this information can be sensory information or signals received through sensory devices, such as, but not limited to, cameras, touch sensors, distance sensors, temperature sensors, wavelength sensors, acoustic or voice sensors, position sensors, pressure or force sensors, speed or acceleration or other motion sensors, location sensors (e.g., GPS), etc. In other embodiments, this information can also include, but is not limited to, compiled, abstract or situational information created from a combination of a set of sensory devices and stored information. In a non-limiting example, an agent can receive abstract information about the location or characteristics of itself or other objects as observations. In some embodiments, this information can refer to people or customers, or characteristics such as their purchasing habits, personal contact information, personal preferences, etc. In some embodiments, an observation can be information about the inside of the agent, such as, but not limited to, proprioceptive information or other information about the agent's current or past actions, information about the agent's internal state, or information already calculated or processed by the agent.

[0036] The term "behavior" refers to any means by which an agent can control, affect, or influence the agent's environment, the agent's physical or simulated self, or the agent's internal functions, ultimately controlling or influencing the agent's future actions, action selections, or behavioral preferences. In many embodiments, these actions can directly control physical or simulated servos or actuators. In some embodiments, these actions can be expressions of preferences or sets of preferences ultimately intended to influence the agent's choices. In some embodiments, information regarding the agent's action(s) can include, but is not limited to, probability distributions over the agent's action(s) and / or outgoing information intended to influence the agent's ultimate action selection.

[0037] The terms "state" or "state information" refer to any collection of information regarding the state of the environment or an agent, which may include, but is not limited to, information regarding the current and / or past observations of the agent.

[0038] The term "policy" refers to any function or mapping from any complete or partial state information to any action information. A policy can be hard coded or modified, adapted or trained using any suitable learning or teaching method, including but not limited to any reinforcement learning or control optimization method. A policy can be an explicit or implicit mapping, such as but not limited to, one that may result from optimizing a particular measure, value or function. A policy can include associated further information, features or characteristics, such as but not limited to, starting conditions (or probabilities) reflecting under what conditions the policy can be started or continued, and ending conditions (or probabilities) reflecting under what conditions the policy can be terminated.

[0039] Generally, embodiments of the invention provide reinforcement learning methods that use actor-critic networks to (1) use additional lab-specific state information to train policies that work well in production settings without this additional lab-specific information, and (2) distill complex resource-demanding policies into less demanding policies that can be more easily implemented in production with limited computational resources. Aspects of these methods can utilize actor-critic methods, in which the critic network models an action value function as opposed to a state value function.

[0040] RL problems are usually framed as Markov decision processes (MDPs), where M =<S, A, P, R> where: S represents the state space of the environment. TIFF0007665735000001.tif6170 shows the complete situation in the environment up to time t. A is the set of actions that the system can take. P is a transition function that gives the transition probability of reaching a particular next state from a given state-action pair. R is the state Action from TIFF0007665735000002.tif6170 TIFF0007665735000003.tif6170 is performed, and the environment is changed from state s to a new state The reward function R(s) defines the reward received when transitioning to TIFF0007665735000004.tif6170. t , a t , s t+1 )).

[0041] The goal of an RL algorithm is to find a mapping from states to actions, i.e., a policy π(s), that maximizes the cumulative reward over time when the agent follows the policy. In practice, an agent may only be able to partially observe the current MDP state. In such cases, the agent must find a policy π(o), a mapping from observations to actions, that maximizes the cumulative reward.

[0042] In a RL setting, an agent does not have direct access to a transition function. Instead, the agent can interact with the environment (choose an action) and observe the outcomes (states and rewards). From these observations, the agent can record a dataset and use this dataset to learn a policy.

[0043] The two functions that are often estimated in RL algorithms are the state value function V π (s) and the action value function Q π (s, a). The state value function V π (s) is the expected cumulative future reward when following the policy π from state s. The action value function Q π (s; a) is the expected cumulative future reward when action a is first taken from state s and then policy π is followed from the subsequent resulting state.

[0044] Deep actor-critic RL algorithms are a family of algorithms that can be used to learn policies and typically utilize two different deep neural networks: the actor network, which represents the policy network π(o) described in the MDP definition, and is responsible for selecting the control action. Once training is complete, only this network is needed to compute actions during evaluation or testing. The critic network predicts long-term rewards for actions taken by the actor. Depending on the algorithm, the critic network directly estimates either a state value function or an action value function for the policy defined by the actor network. The actor improves by optimizing the actor's choice of action against this critic network. Figure 1 shows a simplified version of an exemplary interaction between the actor network 10 and the critic network 12. Here, the critic network 12 generates a value function 14 (e.g., an action value function) for the policy 16. This function can generate an error, such as a time difference (TD) error, that is used to improve the actor network 10 and the critic network 12.

[0045] An embodiment of the invention provides a method for improving the performance of actor networks for a class of problems that satisfy the following constraints (1) and (2): (1) training can be done via data collected in a laboratory setting where more state information and computational resources are available. For example, an autonomous vehicle may have additional / improved sensors in the lab than on the real road once in production. (2) Laboratory on-site and off-lab training is not required, allowing a policy to be trained in the lab and then deployed to production.

[0046] A problem that satisfies these two constraints can be defined as a laboratory RL problem. Exemplary methods of the invention as described herein consider laboratory RL problems and use actor-critic reinforcement learning. In particular, embodiments of the invention can utilize actor-critic methods, in which a critic network models an action value function (as opposed to a state value function).

[0047] For efficient optimization of actor networks, the critic network should accurately estimate the action-value function. However, in typical RL systems that do not distinguish between laboratory training and production runs, the critic network is trained using only the observations available in production. As a result, the critic may produce less accurate estimates of the action-value function, leading to poorer training of actors.

[0048] 2, in production, the critic 20 (also referred to as the critic network 20) ​​is not required, so an embodiment of the invention can train the critic 20 using all state information available in both the lab (lab state information 24) and production (production state information 26), while training the actor 22 using only the production state information 26 that provides the observations available in production. This difference in the information provided to each network ensures that the actor network 22 behaves as well as possible with the information available during production, without limiting the optimization process.

[0049] Given the computational and memory constraints in a production setting, it may only be possible to run actor networks of a certain size on a production device. Since only the critic network is needed during training, the size of the critic network does not depend on the production constraints. At the same time, optimization usually works better with actor networks that are larger than what is ultimately needed to represent the final policy.

[0050] According to embodiments of the present invention, laboratory training of agents can be performed with both large critic networks and large actor networks that allow for efficient optimization. From the final frozen critic network, a form of distillation can then be used to create actor networks that can run within the computational limits of production, as described below.

[0051] A typical distillation method, such as that shown in FIG. 3, aims for the output of a small (production) actor network 30 to be close to the output of a larger actor network 32 that was first trained using a large critic network 34, using, for example, an L2 loss norm.

[0052] In an embodiment of the invention, as shown in Figure 4, in training phase 1, a large critic network 40 and a large actor network 42 can be optimized through an actor-critic algorithm when training an agent. The large actor network 42 can then be distilled into a small actor network 44 suitable for use as an actor network in production by having the small actor network 44 optimize a frozen version of a pre-trained critic network 46. This process can be performed quickly as it does not require further interaction with the lab environment.

[0053] Those skilled in the art may make many changes and modifications without departing from the spirit and scope of the present invention. It is therefore to be understood that the illustrated embodiments are presented by way of example only and should not be construed as limiting the invention as defined by the following claims. For example, although the elements of the claims are presented below in specific combinations, it should be expressly understood that the invention includes other combinations of elements that are less, more, or different than those disclosed.

[0054] The words used herein to describe the invention and its various embodiments are to be understood to include not only their generally defined meaning, but also the generic structures, materials or acts for which they represent a single species by specific definitions herein.

[0055] Therefore, the definition of the words or elements in the following claims is set forth herein to include more than just the combination of elements expressly stated in the text. In this sense, it is therefore considered that the equivalent substitute of two or more elements may be used in place of any one of the elements in the following claims, or a single element may be used in place of two or more elements in the claims. Although elements may be described above as functioning in a particular combination and may be initially claimed in this manner, it is to be clearly understood that in some cases, one or more elements resulting from the claimed combination may be deleted from these combinations, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0056] The claims are therefore to be understood to include what is specifically illustrated and described above, what is conceptually equivalent, what is obviously substituted, and what incorporates the basic idea of ​​the invention. [Explanation of symbols]

[0057] 10. Actor Network 12 Critic Network 14 Value Function 16 Measures 20 Critic / Critic Network 22 Actor / Actor Network 24 Laboratory Status Information 26 Production Status Information 30 Production Actor Network 32 Large actor networks 34 Large Critic Network 40 Large Critic Network 42 Large actor networks 44 Production Actor Network 46 Frozen Big Critic Network

Claims

1. 1. A method of training an agent, comprising: training a policy using an actor network and a critic network using state information available in both a laboratory setting and a production setting, the state information available in the laboratory setting being greater than the state information available in the production setting; optimizing the critic network using an actor network; providing a production actor network for the agents in the production setting; When the training is complete, replicating the critic network into a frozen critic network; optimizing the production actor network using the frozen critic network; The method of claim 1, wherein the production actor network is smaller than the actor network.

2. 10. The method of claim 1, wherein the state information in the laboratory setting includes information from sensors that are not available in the production setting.

3. The method of claim 1 , wherein the critic network is modeled based on an action value function.

4. 2. The method of claim 1, further comprising: executing a first actor-critic algorithm during the training of the critic network; and executing a second actor-critic algorithm during the step of optimizing the production actor network using the frozen critic network.

5. 5. The method of claim 4, wherein the first actor-critic algorithm is the same as the second actor-critic algorithm.

6. 1. A method of training an agent, comprising: training a policy using the actor network and the critic network; optimizing the critic network using an actor network; providing a production actor network for said agents in a production setting; When the training is complete, replicating the critic network into a frozen critic network; optimizing the production actor network using the frozen critic network, the production actor network being smaller than the actor network; Including, 11. The method of claim 1, wherein the training of the strategy uses state information available in both a laboratory setting and in the production setting, the state information available in the laboratory setting being greater than the state information available in the production setting.

7. 7. The method of claim 6, further comprising: executing a first actor-critic algorithm during the training of the critic network; and executing a second actor-critic algorithm during the step of optimizing the production actor network using the frozen critic network.

8. 8. The method of claim 7, wherein the first actor-critic algorithm is the same as the second actor-critic algorithm.

9. The method of claim 6, wherein the critic network is modeled based on an action value function.

10. A non-transitory computer readable storage medium having stored thereon an executable program, the program comprising: training a policy using an actor network and a critic network using state information available in both a laboratory setting and a production setting, the state information available in the laboratory setting being greater than the state information available in the production setting; optimizing the critic network using an actor network; providing a production actor network for agents in the production setting; When the training is complete, replicating the critic network into a frozen critic network; optimizing the production actor network using the frozen critic network; instructing one or more processors to execute The production actor network is smaller than the actor network.

11. 11. The non-transitory computer-readable storage medium of claim 10, wherein the critic network is modeled based on an action-value function.

Images