Training actor-critic algorithms in a laboratory environment

Abstract

Reinforcement learning methods can use actor-critic networks in which (1) additional lab-only state information is used to train a policy that takes many actions without such additional lab-only information in production; and (2) a complex resource demand policy is distilled into a less demanding policy that can more easily run with limited computational resources in production. Production actor networks can be optimized using a frozen version of a large critic network previously trained with a large actor network. Aspects of these methods can utilize actor-critic methods in which the critic network models an action value function rather than a state value function.

Description

Technical Field

[0001] Embodiments of the present invention generally relate to reinforcement learning methods. More specifically, the present invention relates to systems and methods for improving the performance of actor networks when training actor-critic models. Background Technology

[0002] The following background information may present examples of specific aspects of the prior art (e.g., but not limited to schemes, facts, or common sense) that are intended to help further educate the reader about other aspects of the prior art and should not be construed as limiting the invention or any embodiments thereof to anything stated or implied therein or inferred therefrom.

[0003] Reinforcement learning (RL) describes a class of problems (and solutions) in which an artificial agent optimizes its control over a system (or “environment”) when the system dynamics are unknown and must be learned by an artificial agent through interaction and observation. Typically, RL problems and solutions consider performing learning and optimization over a single, continuous lifecycle of the agent. However, since the initial performance of the learning agent will be poor and it is often unsafe in industry environments, industry applications of RL benefit from dividing RL into two phases: (1) a learning phase that occurs in a safe laboratory, and (2) a production phase in which the controller or “policy” learned in the laboratory is deployed without any further learning.

[0004] In light of the foregoing, there is a need for systems and methods that better utilize these two phases to improve reinforcement learning, especially in agents with limited computational resources. Summary of the Invention

[0005] Embodiments of the present invention provide a method for training an agent, comprising training a strategy using an actor network and a critic network by using state information available in both laboratory and production settings, wherein more state information is available in the laboratory setting than in the production setting; optimizing the critic network using the actor network; and providing the agent with a production actor network in the production setting.

[0006] In some embodiments, the critic network is trained based on an action-value function.

[0007] In some embodiments, when training is complete, the method further includes replicating the critic network as a frozen critic network and using the frozen critic network to optimize the production actor network.

[0008] In some embodiments, the first actor-critic algorithm runs during the training of the critic network, and the second actor-critic algorithm runs during the step of optimizing the production actor network using the frozen critic network. In some embodiments, the first actor-critic algorithm and the second actor-critic algorithm are identical.

[0009] Embodiments of the present invention also provide a method for training an agent, including training a policy using an actor network and a critic network; optimizing the critic network using the actor network; providing a production actor network for the agent in a production setting; replicating the critic network as a frozen critic network when training is complete; and optimizing the production actor network using the frozen critic network, wherein the production actor network is smaller than the actor network.

[0010] Embodiments of the present invention also provide a non-transitory computer-readable storage medium having an executable program stored thereon, wherein the program instructs one or more processors to perform the following steps: training a strategy using an actor network and a critic network by using state information available in both laboratory and production settings, wherein more state information is available in laboratory settings than in production settings; optimizing the critic network using the actor network; and providing a production actor network for an agent in a production setting.

[0011] These and other features, aspects, and advantages of the invention will be better understood by referring to the following drawings, description, and claims. Attached Figure Description

[0012] Some embodiments of the present invention are described as examples and are not limited to the figures in the accompanying drawings, in which the same reference numerals may indicate similar elements.

[0013] Figure 1 The illustration shows a schematic diagram of the actor-critic model using reinforcement learning;

[0014] Figure 2 The illustration shows how to train a critic network using state information available in both laboratory and production settings, and how to train an actor network using state information available in production settings.

[0015] Figure 3 illustrates the typical process of using the actor-critic method to optimize a large actor network and then directly extracting production-ready actors from the large actor network.

[0016] Figure 4 The illustration depicts a process according to an exemplary embodiment of the invention, wherein the actor-critic algorithm is run twice: first, a large actor network is used to help optimize the critic network, and then a production-ready actor network is optimized by reusing a frozen copy of the optimized critic network.

[0017] Unless otherwise indicated, the illustrations in each figure are not necessarily drawn to scale.

[0018] The invention and its various embodiments can now be better understood by turning to the following detailed description of the illustrated embodiments. It should be clearly understood that the illustrated embodiments are merely illustrative and not intended to limit the invention as ultimately defined in the claims. Detailed Implementation

[0019] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. As used herein, the singular forms "a," "an," and "the" are intended to include both the plural and singular forms unless the context clearly indicates otherwise. It will also be understood that when the terms "comprising" and / or "including" are used in this specification, they specify the presence of the stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0020] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It will also be understood that terms such as those defined in common dictionaries shall be interpreted as having the same meaning as they have in the relevant field and in the context of this disclosure, and shall not be interpreted as having an idealized or overly formalized meaning unless so explicitly defined herein.

[0021] In describing this invention, it will be understood that numerous techniques and steps are disclosed. Each of these techniques and steps has its own advantages and can be used in combination with one or more, or in some cases, all of the other disclosed techniques. Therefore, for clarity, this description will avoid unnecessarily repeating every possible combination of the individual steps. However, this specification and claims should be read with the understanding that such combinations are fully within the scope of this invention and the claims.

[0022] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention may be practiced without these specific details.

[0023] This disclosure is considered to be illustrative of the invention and is not intended to limit the invention to the specific embodiments illustrated in the following drawings or description.

[0024] Devices or system modules that generally communicate with each other do not need to communicate continuously with each other, unless otherwise expressly stated. Furthermore, devices or system modules that generally communicate with each other can communicate directly or indirectly through one or more intermediaries.

[0025] The description of an embodiment having several components that communicate with each other does not imply that all of these components are necessary. Rather, a variety of optional components are described to illustrate a variety of possible embodiments of the invention.

[0026] "Computer" or "computing device" can refer to one or more devices and / or one or more systems that are capable of accepting structured input, processing the structured input according to prescribed rules, and producing a processing result as output. Examples of computers or computing devices may include: computers; fixed and / or portable computers; computers having a single processor, multiple processors, or multi-core processors that can operate in parallel and / or not in parallel; supercomputers; mainframes; supermicrocomputers; minicomputers; workstations; microcomputers; servers; clients; interactive televisions; web appliances; telecommunications equipment with Internet access; hybrid combinations of computers and interactive televisions; portable computers; tablet personal computers (PCs); personal digital assistants (PDAs); portable telephones; dedicated hardware for emulating computers and / or software, such as, for example, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific instruction set processors (ASIPs), a single chip, multiple chips, a system-on-a-chip, or a chipset; data acquisition devices; optical computers; quantum computers; biometric computers; and devices that can generally accept data, process data according to one or more stored software programs, generate results, and generally include input, output, storage, arithmetic, logic, and control units.

[0027] "Software" or "application" can refer to the prescribed rules used to operate a computer. Examples of software or applications may include code segments in one or more computer-readable languages; graphical and / or text instructions; applets; pre-compiled code; interpreted code; compiled code; and computer programs.

[0028] These computer program instructions may also be stored in a computer-readable medium that can instruct a computer, other programmable data processing apparatus or other device to operate in a particular manner, causing the instructions stored in the computer-readable medium to produce an article of art, including instructions that implement the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0029] Furthermore, although processing steps, method steps, algorithms, etc., may be described in a sequential order, such processes, methods, and algorithms can be configured to operate in an alternative order. In other words, any sequence or order of steps that may be described does not necessarily indicate that the steps must be performed in that order. The steps of the process described herein can be performed in any actual order. Moreover, some steps may be performed simultaneously.

[0030] It will be clear that the various methods and algorithms described herein can be implemented, for example, by a properly programmed general-purpose computer and computing device. Typically, a processor (e.g., a microprocessor) receives instructions from memory or a similar device and executes those instructions, thereby performing the processing defined by those instructions. Furthermore, a variety of known media can be used to store and transmit programs implementing these methods and algorithms.

[0031] As used herein, the term "computer-readable medium" refers to any medium that participates in providing data (e.g., instructions) that can be read by a computer, processor, or similar device. Such media can take many forms, including, but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical discs or magnetic disks and other persistent storage devices. Volatile media include dynamic random access memory (DRAM), which typically constitutes main memory. Transmission media include coaxial cables, copper wires, and optical fibers, including lines that constitute a system bus coupled to a processor. Transmission media can include or transmit sound waves, light waves, and electromagnetic radiation, such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tape, any other magnetic media, CD-ROMs, DVDs, any other optical media, punched cards, paper tape, any other physical media with a perforated pattern, RAM, PROM, EPROM, FLASHEEPROM, or any other memory chip or cartridge, a carrier wave as described below, or any other medium from which a computer can read.

[0032] Various forms of computer-readable media can involve carrying sequences of instructions to a processor. For example, sequences of instructions (i) can be transferred from RAM to the processor, (ii) can be carried on a wireless transmission medium, and / or (iii) can be formatted according to various formats, standards, or protocols such as Bluetooth, TDMA, CDMA, 3G, etc.

[0033] Embodiments of the present invention may include means for performing the operations disclosed herein. The means may be specifically constructed for a desired purpose, or the means may include a general-purpose means selectively activated or reconfigured by a program stored in the means.

[0034] Unless otherwise specified, and as may be clearly understood from the following description and claims, throughout the description, terms such as “processing,” “computing,” “operation,” “determining,” etc., are used to refer to the actions and / or processing of a computer or computing system or similar electronic computing device that manipulate and / or transform data represented as physical (e.g., electronic) quantities within the registers and / or memory of the computing system into other data similarly represented as physical quantities within the memory, registers, or other such information storage, transmission, or display devices of the computing system.

[0035] In a similar manner, the term "processor" can refer to any device or part of a device that processes electronic data from registers and / or memory to transform that electronic data into other electronic data that can be stored in registers and / or memory or transferred to external devices to cause physical changes or actuation of those external devices.

[0036] The terms “agent” or “intelligent agent” or “human agent” or “artificial intelligence agent” refer to any artificial entity that selects an action in response to observation. “Agent” can refer to, but is not limited to, robots, simulated robots, software agents, or “bots”, adaptive agents, internet or web bots.

[0037] The term "robot" can refer to any system that is directly or indirectly controlled by a computer or computing system and issues actions or commands in response to sensation or observation. The term can refer to, but is not limited to, traditional physical robots with physical sensors (such as cameras, touch sensors, distance sensors, etc.), simulated robots existing in virtual simulations, or "web robots" existing as software on a network, such as mail bots or search bots. It can refer to, but is not limited to, any limbed robot, walking robot, industrial robot (including but not limited to robots used for automated assembly, painting, repair, maintenance, etc.), wheeled robot, vacuuming or lawnmowing robot, personal assistant robot, service robot, medical or surgical robot, flying robot, driving robot, aircraft or spacecraft robot, or any other robot, vehicle, or other real or simulated robot that is substantially autonomously operable, including stationary robots such as smart home or workplace appliances.

[0038] The term "observation" or "multiple observations" refers to any information received by the agent regarding the agent's environment or itself by any means. In some embodiments, this information may be sensory information or signals received by sensing devices such as, but not limited to, cameras, touch sensors, range sensors, temperature sensors, wavelength sensors, sound or voice sensors, position sensors, pressure or force sensors, speed or acceleration or other motion sensors, location sensors (e.g., GPS), etc. In other embodiments, this information may also include, but is not limited to, compiled, abstract, or situational information compiled from a collection of sensing devices combined with stored information. In a non-limiting example, the agent may receive abstract information as an observation relating to the location or characteristics of itself or other objects. In some embodiments, this information may refer to a person or customer or their characteristics, such as purchasing habits, personal contact information, personal preferences, etc. In some embodiments, an observation may be information about internal parts of the agent, such as, but not limited to, ontology information or other information relating to the agent's current or past actions, information about the agent's internal state, or information that has been calculated or processed by the agent.

[0039] The term "action" refers to any component of an agent that controls, acts upon, or influences the agent's environment, the agent's physical or simulated self, or the agent's internal functions, ultimately controlling or influencing the agent's future actions, action choices, or action preferences. In many embodiments, an action may directly control a physical or simulated servo or actuator. In some embodiments, an action may be a preference or expression of a set of preferences ultimately intended to influence the agent's choices. In some embodiments, information about one or more of the agent's actions may include, but is not limited to, probability distributions over one or more of the agent's actions, and / or output information intended to influence the agent's final action choice.

[0040] The term “state” or “state information” refers to any collection of information relating to the state of an environment or agent, which may include, but is not limited to, information about current and / or past observations of the agent.

[0041] The term "policy" refers to any function or mapping from any complete or partial state information to any action information. A policy can be hard-coded or modified, adjusted, or trained using any suitable learning or teaching method (including, but not limited to, any reinforcement learning or control optimization method). A policy can be an explicit mapping or an implicit mapping, such as, but not limited to, a mapping resulting from optimizing a particular metric, value, or function. A policy may include associated additional information, features, or properties, such as, but not limited to, starting conditions (or probabilities) reflecting under what conditions the policy can begin or continue, and termination conditions (or probabilities) reflecting under what conditions the policy can terminate.

[0042] In a broader sense, embodiments of the present invention provide reinforcement learning methods using actor-critic networks, wherein (1) additional lab-only state information is used to train policies that take many actions in production settings where such additional lab-only information is unavailable; and (2) complex resource-demanding policies are refined into less demanding policies that can be more easily run in production with limited computational resources. Aspects of these methods can leverage actor-critic methods in which the critic network models action-value functions rather than state-value functions.

[0043] RL problems are typically constrained as Markov decision processes (MDPs). An MDP can be expressed as M =<S,A,P,R> in:

[0044] S represents the state space of the environment. Given a state s t ∈S completely describes the state of the environment before time t;

[0045] A is the set of actions that the system can take;

[0046] P is a transition function that gives the probability of transitioning from a given state-action pair to a specific next state; and

[0047] R is the reward function R(s) t a t s t+1 It defines when from state s t ∈S take action a t ∈A and the environment transitions from state s to a new state s. t+1 The reward received when ∈S.

[0048] The goal of RL algorithms is to find a policy π(s), which is a mapping from states to actions that maximizes the cumulative reward over time when the agent follows it. In practice, the agent may only have partial observations of the current MDP state. In this case, the agent must find a policy π(o), which is a mapping from the observed actions that will maximize the cumulative reward.

[0049] In the context of Reinforcement Learning (RL), the agent does not have direct access to transition functions. Instead, the agent can interact with the environment (choose actions) and observe the consequences (state and reward). Based on these observations, the agent can record a dataset and use that dataset to learn a policy.

[0050] Two functions frequently estimated in RL algorithms are the state-value function V. π (s) and action value function Q π (s, a). State-value function V π(s) is the expected cumulative future reward when following policy π from state s. Action-value function Q π (s, a) is the expected cumulative future reward when action a is first performed from state s, and then the resulting state follows policy π.

[0051] The Deep Actor-Critic RL algorithm is a family of algorithms that can be used to learn policies and typically utilizes two different deep neural networks. The actor network represents the policy network π(o) as described in the definition of an MDP and is responsible for selecting control actions. Once trained, only this network is needed to compute actions during evaluation or testing. The critic network predicts the long-term reward for the actions taken by the actor. Depending on the algorithm, the critic network will directly estimate the state-value function or action-value function of the policy defined by the actor network. The actor is improved by optimizing its action selection for this critic network. Figure 1 The illustration depicts a simplified version of an exemplary interaction between actor network 10 and critic network 12. Here, critic network 12 generates value functions 14, such as action value functions, for policy 16. This function can generate errors, such as time difference (TD) errors used to improve the time difference between actor and critic networks 10 and 12.

[0052] Embodiments of the present invention provide a method for improving the performance of actor networks for a class of problems that satisfy the following constraints: (1) Training can be accomplished using data collected in laboratory settings with more state information and available computational resources. For example, autonomous vehicles can have more / better sensors in a laboratory than once on the road in production. (2) Training in the field and outside the laboratory is unnecessary, thus allowing the policy to be trained in the laboratory and then deployed to production.

[0053] Problems satisfying these two constraints can be defined as laboratory RL problems. As described herein, exemplary methods of the present invention relate to laboratory RL problems and utilize actor-critic reinforcement learning. In particular, embodiments of the present invention may utilize an actor-critic approach, wherein a critic network models an action-value function (rather than a state-value function).

[0054] For the actor network to be effectively optimized, the critic network should accurately estimate the action-value function. However, in typical RL systems that do not distinguish between lab training and production execution, the critic network is trained using only observations that will be available at production time. Therefore, the critic may produce less accurate estimates of the action-value function, leading to poorer training of the actor.

[0055] refer to Figure 2Because critic 20, also known as critic network 20, is not required in production, embodiments of the invention can train critic 20 using all state information available in both the lab and production environments (laboratory state information 24 and production state information 26, respectively), while actor 22 is trained using only production state information 26, which provides observations available during production. This difference in information provided to each network ensures that actor network 22 will perform as best as possible using the information available during production, without limiting the optimization process.

[0056] Given the computational and memory constraints of production environments, it may be possible to run actor networks of a specific size only on production devices. Since the critic network is only needed during training, its size is not dependent on production constraints. Furthermore, optimization is generally more suitable for larger actor networks compared to the network that ultimately needs to represent the final policy.

[0057] According to embodiments of the invention, laboratory training of the agent can be performed using both a large actor network and a large critic network that allow for efficient optimization. Then, an actor network that can operate within production computational constraints can be created from the final, frozen critic network using a refined form, as discussed below.

[0058] As shown in Figure 3, a typical refinement method aims to approximate the output of the small (production) actor network 30 with the output of the large actor network 32, which was initially trained using a large critic network 34, using a loss norm such as L2. However, using this conventional approach, small errors in the policy output can lead to large errors in the accumulated future rewards.

[0059] In various aspects of the invention, such as Figure 4 As shown, in training phase one, the large critic network 40 and the large actor network 42 can be optimized using the actor-critic algorithm while training the agent. Then, by having the small actor network 44 optimize a frozen version of the pre-trained critic network 46, the large actor network 42 can be refined into a small actor network 44 suitable for use as an actor network in production. This process can be performed quickly because it does not require any further interaction with the lab environment.

[0060] Many changes and modifications can be made by those skilled in the art without departing from the spirit and scope of the invention. Therefore, it must be understood that the illustrated embodiments are set forth for illustrative purposes only and should not be construed as limiting the invention as defined in the appended claims. For example, although the elements of the claims are set forth in a certain combination below, it must be clearly understood that the invention includes fewer, more, or different combinations of the disclosed elements.

[0061] The terms used in this specification to describe the invention and its various embodiments should be understood not only in their usual sense, but also, by their specific definitions, in this specification to include the single kind of general structure, material, or action they represent.

[0062] Therefore, the definitions of words or elements in the appended claims are intended in this specification to include not only combinations of elements literally stated. Thus, in this sense, any element in the following claims may be replaced by two or more equivalent elements, or two or more elements in the claims may be replaced by a single element. While elements may be described above as functioning in certain combinations and even initially claimed in this way, it should be clearly understood that in some cases, one or more elements from the claimed combination may be removed from that combination, and the claimed combination may be indicated as a sub-combination or a variation of a sub-combination.

[0063] Therefore, the claims should be understood to include the content specifically shown and described above, the conceptually equivalent content, the content that can be obviously substituted, and the content that incorporates the basic ideas of the invention.

Claims

1. A method for training an agent, comprising: In laboratory settings: The strategy is trained using an actor-critic algorithm by employing an actor network and a critic network. The critic network uses state information available in both laboratory and production settings, while the actor network uses state information available only in production settings. The state information used in training the strategy in the laboratory setting includes information collected in the laboratory setting through at least one of the following: (1) additional sensors of the agent available in the laboratory setting but not in the production setting; and (2) more computational resources available for the agent in the laboratory setting than in the production setting, wherein the agent refers to any artificial entity that selects an action in response to observation; and Compared to critic networks, actor networks optimize motion selection; After training the policy in a laboratory setting, before deploying the policy to production: Provide production actor networks for agents in production settings; Copying the critics' network into a frozen critics' network; and Using a frozen critic network outside of a laboratory setting to optimize the production actor network. A network of critics is required only during training in laboratory settings; and The commentator network is modeled based on action-value functions rather than state-value functions.

2. The method of claim 1, wherein the actor production network is the same as the actor network.

3. The method of claim 1, wherein the status information in the laboratory setting includes information from sensors that are not available in the production setting.

4. The method of claim 1, wherein the production actor network is smaller than the actor network.

5. The method of claim 1, wherein the first actor-critic algorithm runs during the training of the critic network, and the second actor-critic algorithm runs during the step of optimizing the production actor network using the frozen critic network.

6. The method of claim 5, wherein the first actor-critic algorithm is the same as the second actor-critic algorithm.

7. The method of claim 1, wherein more state information is available in a laboratory setting than in a production setting.

8. A non-transitory computer-readable storage medium having an executable program stored thereon, wherein the program instructs one or more processors to perform the following steps: In laboratory settings: The strategy is trained using an actor-critic algorithm by employing an actor network and a critic network. The critic network uses state information available in both laboratory and production settings, while the actor network uses state information available only in production settings. The state information used in training the strategy in the laboratory setting includes information collected in the laboratory setting through at least one of the following: (1) additional sensors of the agent available in the laboratory setting but not in the production setting; and (2) more computational resources available for the agent in the laboratory setting than in the production setting, wherein the agent refers to any artificial entity that selects an action in response to observation; and Compared to critic networks, actor networks optimize motion selection; After training the policy in a laboratory setting, before deploying the policy to production: Provide production actor networks for agents in production settings; Copying the critics' network into a frozen critics' network; and Using a frozen critic network outside of a laboratory setting to optimize the production actor network. A network of critics is required only during training in laboratory settings; and The commentator network is modeled based on action-value functions rather than state-value functions.

9. The non-transitory computer-readable storage medium of claim 8, wherein the production actor network is smaller than the actor network.

Images