Behavior of the training agent

Abstract

A device for training the behavior of an agent in a physical or digital environment is described. The device includes a memory storing the position of at least one reward token in the environment. The position has been specified by a user. At least one processor executes the agent in the environment according to a behavior policy. The processor is configured to observe values of variables, including observations of the agent, the agent's actions, and any rewards generated from the reward token. The processor is configured to update the behavior policy using reinforcement learning based on the observed values.

Description

Background Art

[0001] An agent is a physical or digital entity that behaves in its environment according to a behavioral policy. A non-exhaustive list of agent examples includes: a robotic vacuum cleaner, a robotic arm on a robotic assembly line in a manufacturing plant, an avatar in a video game, a digital assistant (chatbot).

[0002] Typically, behavioral policies are defined by human operators, such as the manufacturer or supplier of the agent. Behavioral policies are often implemented using software created by expert programmers. However, implementing behavioral policies in this way can be difficult, making it difficult to deploy agents and get them to work well.

[0003] The embodiments described below are not limited to implementations that address any or all of the shortcomings of known methods of training agent behavior. Summary of the Invention

[0004] The following is a simplified summary of the present disclosure to provide the reader with a basic understanding. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Its sole purpose is to present a selection of concepts disclosed herein in a simplified form as a prelude to the more detailed description that is presented later.

[0005] A device for training the behavior of an agent in a physical or digital environment is described. The device includes a memory storing the position of at least one reward token in the environment. The position has been specified by a user. At least one processor executes the agent in the environment according to a behavior policy. The processor is configured to observe values of variables, including observations of the agent, the agent's actions, and any rewards generated from the reward token. The processor is configured to update the behavior policy using reinforcement learning based on the observed values.

[0006] Many of the attendant features will be more readily appreciated as the same become better understood by reference to the following detailed description considered in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The present description will be better understood when the following detailed description is read in light of the accompanying drawings, in which:

[0008] Figure 1 is a schematic diagram of the agent in the environment;

[0009] Figure 2 is a schematic diagram of a robotic vacuum cleaner in a living room and illustrating a reward token positioned by a user in the living room;

[0010] Figure 3 is a schematic diagram of a robotic arm in a manufacturing plant and showing a user positioned reward token;

[0011] Figure 4A This is a diagram of a chatbot and a decision tree;

[0012] Figure 4B is a schematic diagram of an avatar in a platform game with a user-located reward token;

[0013] Figure 5 is a diagram of a reinforcement learning cycle;

[0014] Figure 6 is a flow chart of a method performed by an agent for training a behavior of the agent;

[0015] Figure 7 is a flow chart of a method performed by an agent for training a skill of an aggregated skill;

[0016] Figure 8 An exemplary computing-based device is shown in which embodiments of an agent are implemented.

[0017] The same reference numerals are used throughout the drawings to denote the same components. DETAILED DESCRIPTION

[0018] The detailed description provided below in conjunction with the accompanying drawings is intended as a description of this example and is not intended to represent the only way to construct or utilize this example. This description sets forth the functions of this example and the sequence of operations for constructing and operating this example. However, the same or equivalent functions and sequences may be implemented by different examples.

[0019] As mentioned above, agents have behavioral policies that are typically defined by a human operator, such as the agent's manufacturer or supplier. For example, a behavioral policy is software that includes manually crafted rules that specify how the agent should behave in its environment. Creating software-coded manual rules for controlling an agent is a time-consuming and complex task that requires skilled programmers.

[0020] The behavioral policy for an agent can be implemented by the manufacturer and installed into the agent during manufacturing. In this case, the behavioral policy must be created for a generic environment because it is not known at the time of manufacturing in which specific environment the agent will be deployed. In the case where the agent is a robotic vacuum cleaner, the behavioral policy is created for an average household room. However, when the robotic vacuum cleaner is deployed in a specific household room, such as the living room of the reader's home, the behavioral policy is likely to be suboptimal. As a result, the robotic vacuum cleaner will bump into furniture and may damage soft furnishings or other items in the home. In order to tailor the behavioral policy to the specific household in question, the robotic vacuum cleaner must be trained when it is deployed, that is, after manufacturing. However, it is difficult for non-professionals to train the agent.

[0021] This technology uses reinforcement learning to train an agent to behave in a specific environment according to a behavior policy. A behavior policy is a definition of how an agent should behave in an environment and is defined as a function that takes the values of observation variables as arguments and computes the value of an action variable. An observation is a portion of the environment about which the agent has information at the current time. This technology uses any well-known reinforcement algorithm to update the behavior policy based on a value tuple, where the value tuple is the value of the observation variable at the current time, the value of the action variable for the action taken by the agent at the current time, and the value of any applicable reward tokens. Note that observations omit the reward token(s) because the agent cannot directly observe the reward tokens and can only learn from their effects. Reward tokens are virtual, visual elements in the environment that can be located by the user. Reward tokens are either positive or negative reward tokens. The agent's behavior is trained even if it does not "see" the reward tokens in its observations. Agent learning associates rewards with observations and actions taken, rather than reward tokens, because reward tokens are only visible / observable to the end user, not the agent.

[0022] By using reinforcement learning, an agent's behavior can be trained in an environment in an efficient and effective manner. However, using reinforcement learning to train specific behaviors for an agent typically requires machine learning expertise. Constructing a good reward function for reinforcement learning is often difficult without knowing what's happening "under the hood." In contrast, this technology enables the end user to simply place one or more reward tokens in the agent's environment to train it. The end user doesn't need to know the details of how reinforcement learning works.

[0023] Figure 1 is a schematic diagram of an agent 100 in an environment. In this example, the agent is a robotic vacuum cleaner 102 in a home, a robotic arm 104 in a manufacturing plant, or an avatar 106 in a video game. An end user 110 places 108 one or more reward tokens in the agent's 100 environment. The agent is then trained in its environment using reinforcement learning 112, resulting in a trained agent 114 that is trained to behave in a specific environment. By placing reward tokens in the environment, the user is able to influence the outcome of the training, so that the resulting trained agent 114 behaves as the user desires. This is achieved without the user 110 having to understand reinforcement learning 112 or write any software.

[0024] Alternatively or additionally, the functionality of the agent is at least partially performed by one or more hardware logic components. For example, and not limitation, illustrative types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard I / O products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and graphics processing units (GPUs).

[0025] In some examples, the functionality of the agent is performed at least in part by computing facilities in the cloud.

[0026] exist Figure 2 In the example shown, the agent is a robotic vacuum cleaner 200 in a home's living room, where various objects 206 and a chair 208 are present. A person in the room 204 is a user who has placed three reward tokens in the room. The reward tokens are virtual tokens placed at locations in the room by a user operating an augmented reality computing device (not shown). The user has placed the reward tokens so that they are approximately half the width of the robotic vacuum cleaner 200 from a piece of furniture. In this way, the robotic vacuum cleaner is trained to avoid bumping into the furniture while simultaneously cleaning as close to it as possible (because it receives a reward when it moves to a location with a token).

[0027] exist Figure 3 In the example of [ 15 ], the agent is a robotic arm 300 in a manufacturing plant. A conveyor belt 306 transports boxes 302, 304 for the robotic arm 300 to pick up using its grippers 310. The boxes 302, 304 are at various locations on the conveyor belt, and the robotic arm must use a camera 308 to detect the boxes in order to pick them up. The robotic arm must be trained to pick up the boxes 302, 304 without damaging them. To facilitate training, reward tokens 312 are placed in the environment by the user. The spacing between the reward tokens and the boxes is similar to the width of the robotic arm's grippers. In this way, the robotic arm learns to pick up the boxes without damaging them because it receives a reward when its grippers are properly spaced from the boxes.

[0028] exist Figure 4A In the example shown, the agent is a chatbot 400 on a desk 404 in an office. On desk 404 is a set of keys 402 illustrating the relative size of chatbot 400. The chatbot has multiple well-known skills and must decide which skill to activate in a given situation. A non-exhaustive list of example skills is: turning on a light, turning off an oven, making a restaurant reservation, making a calendar entry.

[0029] Figure 4ADecision tree 406 is shown in FIG, and is a schematic representation of a decision tree within the software implementing chatbot 400 in the cloud. The leaves of the decision tree represent the chatbot's skills, and the internal split nodes of the tree represent decision points. The chatbot's software searches the decision tree based on the chatbot's current situation to find the leaf node that represents which skill to perform in that situation. In this example, the agent's environment is represented by a decision tree, and a user uses a graphical user interface to place one or more tokens 408, 410 in the decision tree. In this case, the reward tokens are graphical elements. The reward tokens enable the agent to be trained to use specific skills in specific situations. In this way, the user can train the chatbot without a detailed understanding of reinforcement learning.

[0030] exist Figure 4B In the example, the agent is an avatar (in this case, a 15-inch TV character) in a 2D platform game. The user places a diamond in the environment, which in this case is a 2D platform game. The avatar's goal is to reach the Figure 4B The black doorway is shown. The user has positioned the blue diamond to facilitate the avatar learning to jump over the gap ( Figure 4B ) and moves in the direction of the black doorway ( Figure 4B the lower part of the ).

[0031] Figure 5 This is a diagram of a reinforcement learning cycle. Figure 5 An agent 500 is shown with a reinforcement learning component 502. For clarity, the agent 500 has Figure 5 The symbol π in is used to denote the behavior policy. The agent is the robotic vacuum cleaner 102, or the robotic arm 104, or the avatar 106, or the chatbot 400, or any other agent.

[0032] When the agent 500 executes in its environment 506, it performs actions 504. When the agent 500 is first deployed in the environment 506, the behavior policy is initialized to random or manually configured settings. The agent 500 calculates what action 504 to perform from the initialized behavior policy based on its current observation of the environment 506. Actions 504 occur in the environment 506 and can change the environment or the observations. The agent now has a set of three values, including information about the action 504, the observation 508, and any reward tokens that have been encountered in the environment as a result of the action 504. The agent uses the set of three values as input to the reinforcement learning component 502. The reinforcement learning component 502 uses an update algorithm to update the behavior policy based on the set of three observations. The behavior policy is updated to be more likely to generate actions that will generate rewards in the future. Figure 5The illustrated loop repeats continuously, or at least until there is little change in the behavioral policy, such as when the value of the observed variable is within a specified constant range.

[0033] By using information about the reward tokens placed by the user as part of the process of updating the behavioral policy, the agent can be efficiently and effectively trained to take those reward tokens into account. This is achieved even if the agent does not "see" the reward tokens in its observations. The agent learns to associate its actions 504 with the rewards from the reward tokens placed by the user. In this way, the user is able to train the agent's behavior without having to understand anything about reinforcement learning or write any software.

[0034] Due to the nature of reinforcement learning, once an agent has been trained on a behavior, there are often slight variations in how the agent performs the behavior each time it performs the behavior. These small variations are beneficial in many ways. These variations lead to robustness in the agent's behavior because if the environment changes over time, the agent will be better able to cope with these changes (without having to be retrained) rather than getting stuck or failing. These variations also maintain interest in human users, such as where the agent is an avatar in a video game.

[0035] Figure 6 6 is a flow chart of a method performed by an agent. The agent receives 600 user input specifying the location of one or more reward tokens in its environment. The reward tokens are virtual and specify locations in the agent's environment. The agent 602 executes according to its behavior policy and the agent observes 604 its observations, the actions it takes, and any rewards generated from the user-positioned reward token(s). The agent uses reinforcement learning and observations to update 606 the behavior policy. The agent decides whether to stop learning, and if so, it will use the behavior policy to continue controlling itself. Optionally, it removes the tokens 610 from the environment.

[0036] If the agent decides to continue training, then Figure 6 The process is repeated from operation 602.

[0037] In some cases, users want to train an agent on a complex behavior that is a collection of two or more skills. Figure 7 is an example of a method for training aggregate skills at an agent, which are complex behaviors composed of two or more skills.

[0038] The user considers a complex behavior and breaks it down into two or more skills. The user specifies one of the skills 700 and places one or more reward tokens into the environment to train the agent on the specified skill. Figure 7The method performed by the agent is shown. For a specified skill 700, the agent receives 702 one or more locations where a user places a reward token. It uses Figure 6 The process of training 704 oneself to perform skills.

[0039] The agent then checks at decision point 706 whether the user has specified more skills. If so, it moves to the next specified skill and repeats operations 700, 702, and 704. If all specified skills have been trained, then at decision point 706 the agent proceeds to operation 708. At operation 708, the agent will now train to perform the specified aggregate skill 708. It receives 710 the reward token positions placed by one or more users for the aggregate skill. At operation 704, it uses Figure 6 Once trained, the agent is now capable of performing complex behaviors. In this way, non-experts can train the agent to perform complex behaviors without having to understand machine learning or how to write computer software.

[0040] The agent uses reward token positions placed by the user, allowing the agent to operate in unconventional ways, enabling training of complex behaviors by non-experts.

[0041] By using the locations of reward tokens placed by the user, the functionality of the underlying computing device can be improved by enabling it to be trained to behave in a simple and effective manner. It is not necessary for humans to know what is happening "behind the scenes".

[0042] An alternative approach is ex post facto feedback. With ex post facto feedback, an agent is executed in its environment, and a human observes the agent's behavior. The human later provides feedback to the agent regarding its behavior. Using ex post facto feedback is not very effective. Ex post facto feedback allows the agent to cause damage in certain examples, such as when a robotic vacuum cleaner collides with furniture. Ex post facto feedback is time-consuming because it requires a human to observe the agent's long-term behavior trajectory, such as watching a robotic vacuum cleaner clean for a period of time, in order to subsequently criticize the agent for any errors.

[0043] Another alternative approach is to give preference-based feedback. Preference-based feedback involves the agent performing a task twice, and then the human giving feedback on which of the two instances was better. This approach doesn't work well. Preference-based feedback involves the human spending a lot of time observing a large number of agent actions.

[0044] Figure 8 Various components of an exemplary computing-based device 804 are shown, with the device 804 being implemented as any form of computing and / or electronic device and in some examples implementing embodiments of an agent in the device 804 .

[0045] The computing-based device 804 includes one or more processors 800, which are microprocessors, controllers, or any other suitable type of processors, for processing computer-executable instructions to control the operation of the device so as to execute an agent in an environment according to a behavioral policy 822 and train one or more behaviors of the agent using a reinforcement learning component 820. In some examples, such as where a system-on-chip architecture is used, the processor 800 includes one or more fixed function blocks (also referred to as accelerators) that implement portions of the method for executing the agent and / or training the agent in hardware (rather than software or firmware). Platform software including an operating system 816 or any other suitable platform software is provided at the computing-based device to enable application software 818 to execute on the device. A data repository 824 stores the values of observation variables, action variables, and the locations of reward tokens placed by the user.

[0046] Computer-executable instructions are provided using any computer-readable medium accessible by the computing-based device 804. Computer-readable media include, for example, computer storage media, such as memory 814 and communication media. Computer storage media, such as memory 814, include volatile and non-volatile removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, etc. Computer storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), flash memory or other storage technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium for storing information accessed by a computing device. In contrast, communication media embodies computer-readable instructions, data structures, program modules, etc. in a modulated data signal (such as a carrier wave or other transmission mechanism). As defined herein, computer storage media does not include communication media. Therefore, computer storage media should not be interpreted as propagating signals themselves. Although computer storage media (memory 814 ) is shown within computing-based device 804 , it should be appreciated that in some examples storage is distributed or remotely located and accessed via a network or other communications link (eg, using communications interface 812 ).

[0047] The device 804 based on calculation also includes an output interface 810, which is arranged to output display information to a display device 832, which can be separated or integrated with the device 804 based on calculation. The output interface 810 also sends instructions to the device control system 834, such as a mechanical arm of a mechanical arm or a motor of a robot vacuum cleaner where the agent is a mechanical arm. Display information can provide a graphical user interface. The input interface 802 is arranged to receive and process input from one or more devices, such as a user input device (e.g., a mouse 830, a keyboard 828, a game controller 826, a capture device 836, a camera, a microphone or other sensors). In some examples, the user input device detects voice input, user gestures or other user actions, and provides a natural user interface (NUI). The user input can be used to locate a reward token in the environment. In an embodiment, if the display device 832 is a touch-sensitive display device, the display device 832 is also used as a user input device.

[0048] Alternatively or in addition to other examples described herein, examples include any combination of:

[0049] Clause A. An apparatus for training the behavior of an agent in a physical or digital environment, the apparatus comprising:

[0050] a memory storing a position of at least one reward token in the environment, the position having been 15 specified by a user;

[0051] at least one processor configured to execute an agent in the environment according to a behavioral policy;

[0052] The processor is configured to observe values of the variables, including observations of the agent, the agent's actions, and any rewards generated from reward tokens; and

[0053] The processor is configured to update the behavior policy using reinforcement learning based on the observed values.

[0054] Clause B. The apparatus of clause A, wherein the processor is configured to control the agent according to the updated behavior policy.

[0055] Clause C. An apparatus according to clause A or clause B, wherein the reward generated by the reward token is generated by the position of the reward token relative to the environment.

[0056] Clause D. The apparatus of Clause A or Clause B or Clause C, wherein the apparatus comprises a user interface configured to receive user input specifying a location of the at least one reward token.

[0057] Clause E. The apparatus of clause D, wherein the environment is a digital environment graphically represented at the user interface, and wherein the at least one reward token is a graphical element placed by the user in the graphical representation of the digital environment.

[0058] Clause F. The apparatus of clause D, wherein the user interface is an augmented reality user interface.

[0059] Clause G. The apparatus of any preceding clause, wherein the environment is a physical environment, and wherein at least one reward token is a virtual element placed in the physical environment by the user using an augmented reality user interface.

[0060] Clause H. An apparatus according to any preceding clause, wherein the agent is a chatbot, and wherein the environment is a tree structure representing possible actions of the chatbot, and wherein reward tokens are placed at split nodes of the tree.

[0061] Clause I. An apparatus according to any preceding clause, wherein the stored location is specified by a user with respect to a first skill to be learned by the agent, and wherein the memory stores a second location, the second location being the location of a second reward token for a second skill to be learned by the agent.

[0062] Clause J. The apparatus of any preceding clause, wherein the behavior comprises a plurality of skills, and wherein the apparatus comprises a memory that stores, for each of the skills, a location of at least one reward token in the environment.

[0063] Clause K. An apparatus according to any preceding clause, wherein the agent is any of the following: a robotic vacuum cleaner, a manufacturing robot arm, a chatbot, an avatar in a video game.

[0064] Clause L. A method for training behavior of an agent in a physical or digital environment, the method comprising:

[0065] receiving a location of at least one reward token in the environment, the location having been specified by a user;

[0066] Execute the agent in the environment according to the behavioral policy;

[0067] The values of the observation variables, including: observations of the agent, the agent's actions, and any rewards generated by reward tokens; and

[0068] Use reinforcement learning to update the behavior policy based on observed values.

[0069] Clause M. The method of clause L, wherein the behavior comprises: a plurality of skills, and wherein the device comprises a memory comprising storing, for each of the skills, the location of at least one reward token in the environment.

[0070] Clause N. The method of clause M, comprising: performing the receiving, executing, observing, and updating for each of the skills, and using at least one reward token for the skill.

[0071] Clause O. The method of clause N, comprising: controlling the agent using the behavior policy after the behavior policy is updated for each of the skills.

[0072] Clause P. A method according to any of clauses L to O, repeated while observing the value of a variable over time.

[0073] Clause Q. The method according to any of Clauses L to P is repeated until the value of the variable is within a specified constant range.

[0074] Clause R. The method of any of clauses L to Q, wherein the environment is a physical environment and wherein at least one reward token is a virtual element placed in the physical environment by the user using an augmented reality user interface.

[0075] 10. Clause S. A method according to any of clauses L to R, wherein the agent is a chatbot and wherein the environment is a tree structure representing possible actions of the chatbot and wherein reward tokens are placed at split nodes of the tree.

[0076] Clause T. An apparatus for training an agent in a physical or digital environment to perform a behavior comprising a plurality of skills, the apparatus comprising:

[0077] a memory storing, for each skill, a location of at least one reward token in the environment, the location having been specified by a user;

[0078] at least one processor configured to execute an agent in the environment according to a behavioral policy;

[0079] The processor is configured to observe values of the variables, including observations of the agent, the agent's actions, and any rewards generated from reward tokens; and

[0080] The processor is configured to update the behavior policy using reinforcement learning based on the observed values.

[0081] The terms "computer" or "computing-based device" are used herein to refer to any device that has processing capabilities that enable it to execute instructions. Those skilled in the art will appreciate that such processing capabilities are incorporated into many different devices, and thus the terms "computer" and "computing-based device" include personal computers (PCs), servers, mobile phones (including smartphones), tablet computers, set-top boxes, media players, game consoles, personal digital assistants, wearable computers, and many other devices.

[0082] In some examples, the methods described herein are performed by software in a machine-readable form on a tangible storage medium, such as in the form of a computer program including computer program code means adapted to perform all of the operations of one or more methods described herein when the program is run on a computer, and wherein the computer program may be embodied on a computer-readable medium. The software is suitable for execution on a parallel processor or a serial processor, such that the method operations may be performed in any suitable order or simultaneously.

[0083] This recognizes that software is a valuable, separately tradable commodity. It is intended to encompass software that runs on or controls "simple" or standard hardware to perform a desired function. It is also intended to encompass software that "describes" or defines the configuration of hardware, such as HDL (Hardware Description Language) software used to design silicon chips or to configure general-purpose programmable chips to perform a desired function.

[0084] Those skilled in the art will recognize that the storage device for storing program instructions is optionally distributed across a network. For example, a remote computer can store an example of a process described as software. Local or terminal computers can access the remote computer and download part or all of the software to run the program. Alternatively, the local computer can download fragments of the software as needed, or execute some software instructions at a local terminal, and execute some software instructions at a remote computer (or computer network). Those skilled in the art will also recognize that, by utilizing conventional techniques known to those skilled in the art, all or part of the software instructions can be performed by dedicated circuits such as digital signal processors (DSPs), programmable logic arrays, etc.

[0085] It will be apparent to the skilled artisan that any range or device value given herein may be expanded or altered without losing the effect sought.

[0086] Although the subject matter has been described in language specific to structural features and / or methodological acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

[0087] It should be understood that the benefits and advantages described above may relate to one embodiment or to several embodiments. The embodiments are not limited to embodiments that solve any or all of the problems described or have any or all of the benefits and advantages described. It should also be understood that references to "an" item refer to one or more of these items.

[0088] The operations of the methods described herein can be performed in any suitable order, or simultaneously where appropriate. Additionally, individual blocks can be deleted from any method without departing from the scope of the subject matter described herein. Aspects of any of the examples described above can be combined with aspects of any other examples described to form additional examples without losing the effects sought.

[0089] The term "comprising" is used herein to mean including identified method blocks or elements, but that such blocks or elements do not comprise an exclusive list and the method or apparatus may contain additional blocks or elements.

[0090] The term "subset" is used herein to refer to a subset in place such that the subset of a set does not include all elements of the set (ie, at least one element of the set is missing from the subset).

[0091] It should be understood that the above description is given only as an example and that various modifications may be made by those skilled in the art. The above description, examples, and data provide a complete description of the structure and use of the exemplary embodiments. Although various embodiments have been described above with a certain degree of particularity or with reference to one or more individual embodiments, those skilled in the art may make various changes to the disclosed embodiments without departing from the scope of this specification.

Claims

1. An apparatus for training the behavior of an agent in a physical or digital environment, the apparatus comprising: a memory storing a location of at least one reward token in said physical environment or said digital environment, said location having been specified by an end user of said device; at least one processor configured to execute the agent in the physical environment or the digital environment according to a behavior policy, the agent being a physical or digital entity that behaves in the physical environment or the digital environment according to the behavior policy, the behavior policy being defined by a manufacturer or supplier of the agent; the processor being configured to observe values of variables, including observations of the agent, actions of the agent, and any rewards generated by the reward token, the rewards being observed by the processor and not input by the end user or the manufacturer or the supplier of the agent; as well as The processor is configured to update the behavior policy using reinforcement learning based on the observed values. 2 . The apparatus according to claim 1 , wherein the processor is configured to control the agent according to the updated behavior policy. 3 . The apparatus of claim 1 , wherein the reward generated by the reward token is generated by a position of the reward token relative to the physical environment or the digital environment.

4. The apparatus of claim 1, wherein the apparatus comprises a user interface configured to receive user input from the end user specifying the location of the at least one reward token.

5. An apparatus according to claim 4, wherein the physical environment or the digital environment is a digital environment graphically represented at the user interface, and wherein the at least one reward token is a graphical element placed by the end user in the graphical representation of the digital environment. The apparatus of claim 4 , wherein the user interface is an augmented reality user interface.

7. The apparatus of claim 1 , wherein the physical environment or the digital environment is a physical environment, and wherein the at least one reward token is a virtual element placed in the physical environment by the end-user using an augmented reality user interface.

8. An apparatus according to claim 1, wherein the agent is a chatbot, and wherein the physical environment or the digital environment is a digital environment including a tree structure representing possible actions of the chatbot, and wherein the reward tokens are placed at split nodes of the tree structure, wherein leaves of the tree structure represent skills of the chatbot.

9. An apparatus according to claim 1, wherein the stored location is specified by the end user with respect to a first skill to be learned by the agent, and wherein the memory stores a second location, the second location being the location of a second reward token for a second skill to be learned by the agent, the agent being trained to perform an aggregate skill, the aggregate skill comprising the first skill and the second skill.

10. The apparatus of claim 1, wherein the behavior comprises a plurality of skills, and wherein the apparatus comprises a memory that stores the at least one reward token at the location in the physical environment or the digital environment for each of the skills.

11. The apparatus of claim 1 , wherein the agent is any of the following: a robotic vacuum cleaner, a manufacturing robot arm, a chatbot, an avatar in a video game.

12. A method for training behavior of an agent in a physical or digital environment, the method comprising: receiving a location of at least one reward token in said physical environment or said digital environment, said location having been specified by an end user; executing the agent in the physical environment or the digital environment according to a behavior policy, the agent being a physical entity or a digital entity that behaves in the physical environment or the digital environment according to the behavior policy, the behavior policy being defined by a manufacturer or supplier of the agent; observing, by a processor, values of variables including observations of the agent, actions of the agent, and any rewards generated by the reward token, the rewards being observed by the processor and not input by the end user or the manufacturer or supplier of the agent; as well as The behavior policy is updated using reinforcement learning based on the observed values.

13. The method of claim 12, wherein the agent is an avatar in a 2D platform game, the method further comprising: A change in behavior of the avatar is observed from the updated behavior policy, wherein the avatar is not retrained following the observed change in behavior.

14. The method according to claim 12, wherein the behavior comprises a plurality of skills, the method comprising: The receiving, the performing, the observing, and the updating are performed for each of the skills, and the at least one reward token for the skill is used.

15. The method according to claim 14, comprising: After the behavior policy has been updated for each of the skills, the agent is controlled using the updated behavior policy.

16. The method of claim 12, said method being repeated as the value of said variable is observed over time.

17. The method of claim 12, said method being repeated until the value of said variable is within a specified constant range.

18. The method of claim 12, wherein the physical environment or the digital environment is a physical environment, and wherein the at least one reward token is a virtual element placed in the physical environment by the end-user using an augmented reality user interface.

19. The method of claim 12, wherein the agent is a chatbot and wherein the physical environment or the digital environment is a digital environment comprising a tree structure representing possible actions of the chatbot, and wherein the reward tokens are placed at split nodes of the tree structure.

20. An apparatus for training an agent's behavior in a physical or digital environment, the behavior comprising a plurality of skills, the apparatus comprising: a memory storing, for each skill, a location of at least one reward token in the physical environment or the digital environment, the location having been specified by an end user of the apparatus; at least one processor configured to execute the agent in the physical environment or the digital environment according to a behavior policy, the agent being a physical or digital entity that behaves in the physical environment or the digital environment according to the behavior policy, the behavior policy being defined by a manufacturer or supplier of the agent; the processor being configured to observe values of variables, including observations of the agent, actions of the agent, and any rewards generated by the reward token, the rewards being observed by the processor and not input by the end user or the manufacturer or the supplier of the agent; as well as The processor is configured to update the behavior policy using reinforcement learning based on the observed values.

Images