Controlling robots by using entropy constraints

Abstract

Methods, systems, and apparatus, including a computer program encoded on a computer storage medium, for training a policy neural network having policy parameters. One of the methods includes: obtaining trajectory data including one or more tuples; updating a current value of the policy parameter using a maximum entropy reinforcement learning technique using the trajectory data, the maximum entropy reinforcement learning technique maximizing both (i) a reward term and (ii) an entropy term, wherein a relative weight between the entropy term and the reward term in the maximization is determined by a temperature parameter; and updating the temperature parameter to adjust an expected entropy of the probability distribution to at least equal a minimum expected entropy value using a probability distribution defined by the policy output generated according to the current value of the policy parameter for the tuple in the trajectory data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Patent Application No. 62 / 773,932, filed on November 30, 2018, the entire contents of which are incorporated herein by reference. Background Art

[0003] This instruction manual is concerned with controlling a robot to perform specific tasks.

[0004] Some systems control a robot by using a policy neural network that generates an output that defines a probability distribution over actions that can be performed by the robot, i.e., the system selects an action to be performed by the robot based on the output generated by the policy neural network.

[0005] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict outputs for received inputs. Some neural networks are deep neural networks that include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output from a received input based on the current values ​​of a corresponding set of parameters. Summary of the invention

[0006] This specification describes a system implemented as one or more computer programs on one or more computers located in one or more locations for training a policy neural network. The policy neural network is used to control a robot so that the robot performs a specific task in an environment. For example, the task can be an industrial robot task involving navigating in an environment, moving an object in an environment, or both.

[0007] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0008] Applying deep reinforcement learning methods on real-world robots is exceptionally difficult due to both their sample complexity (i.e., the number of training samples required to learn an effective control policy) and, equally important, their sensitivity to hyperparameters. In particular, tuning hyperparameters directly on a real-world robotic platform (i.e., a platform involving a real-world robot interacting with a real-world environment) can be problematic. More specifically, a robot (e.g., a quadruped or other mobile robot) can be damaged or at least severely worn out after extensive trial-and-error learning, which is typically necessary to determine effective hyperparameter settings for the hyperparameters used in the reinforcement learning training process.

[0009] Specifically, in maximum entropy reinforcement learning, the relative weight between the entropy term and the reward term in the maximization is determined by a parameter, which is called the temperature parameter. Conventional maximum entropy reinforcement learning techniques require a lot of hyperparameter tuning to determine the best setting for the temperature parameter for a given task. When learning the policy on a real-world robot, this hyperparameter search can lead to damage, wear and tear, or both to the robot before the training process even begins.

[0010] However, the described techniques require minimal or no hyperparameter tuning because the temperature parameter is learned during the training of the policy neural network. In other words, the claimed techniques automatically trade off exploration versus exploitation by dynamically and automatically adjusting the temperature parameter used to determine the stochasticity of the policy. Thus, control policies can be efficiently determined without excessive wear or damage to the real-world robot because the need for trial-and-error learning is reduced or even eliminated.

[0011] Additionally, the described techniques automatically adjust the temperature parameter as the policy improves over time during training. This can further reduce the number of training iterations and the degree of robot wear required for the robot to learn a high-quality control policy. More generally, automatically adjusting the temperature parameter results in the policy being free to explore more in regions where the best action is uncertain, but maintaining greater certainty in states where there is a clear distinction between good and bad actions, thereby improving the quality of the final policy learned.

[0012] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 An example control system is shown.

[0014] Figure 2 is a flow chart of an example process for training a policy neural network.

[0015] Figure 3 is a flow chart of an example process for updating current temperature parameters.

[0016] Like reference numbers and numerals in the various drawings indicate like elements. DETAILED DESCRIPTION

[0017] Figure 1An example control system 100 is shown. The control system 100 is an example of a system implemented as a computer program on one or more computers located in one or more locations in which the systems, components, and techniques described below are implemented.

[0018] The system 100 includes a policy neural network 110 , a training engine 150 , and one or more memories storing a set of policy parameters 118 and temperature parameters 152 for the policy neural network 110 .

[0019] The system 100 controls the robot 102 to interact with the environment 104 by selecting an action 106 to be performed by the robot 102 and then causing the robot 102 to perform the selected action 106 .

[0020] The robot 102 performing the selected action 106 generally causes the environment 104 to transition to a new state. The system 100 can control the robot 102 to complete a specified task by repeatedly causing the robot 102 to perform actions in the environment 104. For example, the task can be an industrial robot task involving navigating in an environment, moving objects in an environment, or both.

[0021] Specifically, the control system 100 controls the robot 102 by using the policy neural network 110 so that the robot 102 performs a designated task in the environment 104 .

[0022] The policy neural network 110 is a neural network with parameters (referred to in this specification as “policy parameters”) that receives a state representation 120 characterizing the state of the environment 104 with which the robot 102 interacts, and generates a policy output 122 that defines a probability distribution over a set of possible actions that can be performed by the robot 102.

[0023] Each action in the set of possible actions is typically a multi-dimensional action, such as a multi-dimensional vector. For example, each dimension may correspond to a different joint or motor of the robot 102 and may be a control input to the joint or motor, such as a torque to be applied to the joint or a position or setting of the motor. As an example, if the robot is controlled by eight actuator motors positioned at different points along the frame of the robot, each action will typically be an eight-dimensional vector, where each dimension corresponds to one of the eight motors and the value at that dimension corresponds to the expected motor position of that motor.

[0024] When the action set is discrete, the policy output 122 may include a corresponding probability for each action in the set. When the action set is continuous, the policy output 122 may include, for each dimension, the parameters of the distribution over the possible values ​​of the dimension, such as the mean and covariance of a Gaussian distribution.

[0025] Typically, the state representation 120 is generated from sensor data captured by sensors of the robot 102. For example, the state representation 120 may include observations including one or more sensor readings characterizing the current state of the robot 102, such as sensors for measuring the angles of certain joints or motors of the robot 102, an IMU for measuring the orientation and angular velocity of the robot 102, and so on.

[0026] In some cases, observations may also include images captured by a camera sensor, laser data captured by a laser sensor, or both.

[0027] In some cases, the state representation 120 includes additional data, such as one or more recently received observations in addition to the current observation, data specifying the most recent action performed by the robot, or data 100 specifying the most recent reward 124 received by the system 100.

[0028] Policy neural network 110 may have any suitable neural network architecture that allows policy neural network 110 to map received state representation 120 to policy output 122. For example, when state representation 120 includes high-dimensional image or laser data, policy neural network 110 may be a convolutional neural network. As another example, when state representation 120 includes only relatively low-dimensional inputs (e.g., sensor readings used to characterize the current state of the robot), policy neural network 110 may be a multilayer perceptron.

[0029] The system 100 uses the policy output 122 to control the robot, i.e., selects the action 106 to be performed by the robot at the current time step according to the action selection policy, and then causes the robot to perform the action 106, for example by transmitting a control signal directly to the robot, or by transmitting data for identifying the action 106 to the control system of the robot 102.

[0030] The system 100 may select an action to be performed by the robot 102 based on the action selection output 122 by using any of a variety of action selection strategies, such as by sampling actions according to their probability values, or by selecting the action with the highest probability value.

[0031] The system 100 may receive a corresponding reward 124 at each time step, where the reward 124 is specified by a scalar value and represents, for example, the progress of the robot 102 toward completing a specified task.

[0032] In order to improve the control of the robot 102, the training engine 150 repeatedly updates the policy parameters 118 of the policy neural network 110 so that the policy neural network 110 generates a more accurate policy output, i.e., a policy output that causes the system 100 to receive a higher reward 124, and thus, improves the performance of the robot 102 on the intended task. In other words, the training engine 150 trains the policy neural network 110 by repeatedly updating the policy parameters 118.

[0033] Specifically, the training engine 150 updates the policy parameters 118 by using a maximum entropy reinforcement learning technique. In this technique, the training engine 150 updates the policy parameters 118 to maximize both (i) a reward term measuring the total reward in the tuple in the trajectory data used for the update and (ii) an entropy term based on the current value of the policy parameters, wherein the reward term measures the total reward in the tuple in the trajectory data used for the update, and the entropy term measures the entropy of the probability distribution defined by the policy output generated by processing the state representation in the tuple in the trajectory data. Although the term "entropy" refers throughout to measuring the 'entropy' of a probability distribution, it is understood that when the action space is continuous (and the output of the policy network is a parameter of a continuous probability distribution), the term "entropy" typically measures the differential entropy of the probability distribution.

[0034] Each trajectory in the trajectory data is a sequence of tuples, where each tuple includes a state representation, an action performed in response to the state representation, and a reward obtained in response to the robot performing the action.

[0035] The relative weight between the entropy term and the reward term in the maximization is determined by a parameter referred to as the temperature parameter 152. For example, the entropy term may be multiplied by the temperature parameter that is optimized in the overall objective. That is, the objective function may include an overall term that is the difference between the reward term and the product of the temperature parameter 152 and the entropy term.

[0036] Specifically, the objective function J(π) can satisfy:

[0037]

[0038] where E is the expectation operator, τ is a trajectory sampled from the trajectory distribution induced by the policy neural network π, i.e., a trajectory generated by selecting actions from the policy output 122 generated by the policy neural network 110, which includes time steps 0 to T, and r(s t , a t ) is the robot 102 responding to the state representation s tExecute action a t The reward received by the system 100 as a result of t , a t ) is the probability distribution assigned to action a generated by processing the state representation st using the policy neural network t The probability of , and α is the temperature parameter.

[0039] Unlike conventional maximum entropy reinforcement learning techniques that treat the temperature parameter 152 as a hyperparameter that must be set before training, the training engine 152 also repeatedly updates the temperature parameter 152 when updating the policy parameter 118, i.e., in conjunction with training the policy neural network 110. Specifically, the training engine 150 repeatedly updates the temperature parameter 152 to adjust the expected entropy of the probability distribution defined by the output of the policy neural network 110 to be at least equal to (i.e., greater than or equal to) the minimum expected entropy value.

[0040] Refer to the following Figure 2 and Figure 3 Training the policy neural network 110 and updating the temperature parameters 152 are described in more detail.

[0041] Figure 2 is a flow chart of an example process 200 for training a policy neural network. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed control system (e.g., Figure 1 The control system 100) can perform process 200.

[0042] The system can repeatedly perform process 200 on trajectory data (e.g., trajectory data sampled from a replay buffer or other memory used to store trajectory data generated as the robot interacts with the environment (while being controlled by a policy neural network or some other control strategy)) to train the policy neural network and adjust the temperature parameter in conjunction with the training.

[0043] The system obtains trajectory data (step 202). The trajectory data includes one or more tuples, each of which identifies a state representation characterizing a state of an environment, an action performed by a robot when the environment is in the state represented by the state representation, a reward received by the robot in response to the robot performing the action, and a next-state representation characterizing a next state of the environment after the robot performs the action.

[0044] Using the trajectory data, the system updates the current values ​​of the policy parameters by using a maximum entropy reinforcement learning technique that maximizes both (i) a reward term measuring the total reward in a tuple in the trajectory data and (ii) an entropy term based on the current values ​​of the policy parameters, wherein the reward term measures the total reward in a tuple in the trajectory data and the entropy term measures the entropy of a probability distribution defined by policy outputs generated by processing state representations in the tuple in the trajectory data, wherein a relative weight between the entropy term and the reward term in the maximization is determined by a current temperature parameter (step 204).

[0045] Typically, as part of such updating, the system processes each state representation in each tuple in the trajectory data using a policy neural network to generate a policy output for the state representation based on current values ​​of the policy parameters.

[0046] More specifically, the system can determine the update by using the current temperature parameter and any appropriate maximum entropy reinforcement learning technique. As a specific example, the system can use a Soft Actor-Critic (SAC) algorithm to determine an update to the current value of a policy parameter. In the SAC algorithm, the entropy term is scaled by the current temperature parameter value, i.e., multiplied by the current temperature parameter value, before calculating the error and gradient. The SAC algorithm is described in more detail in "Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learningwith a Stochastic Actor" published by Haarnoja et al. in 2018 (available at arXiv:1801.01290).

[0047] The system updates the temperature parameter using the probability distribution defined by the policy output generated according to the current value of the policy parameter for the tuple in the trajectory data (step 206). Specifically, the system updates the temperature parameter to adjust the expected entropy of the probability distribution defined by the policy output to at least equal the minimum expected entropy value.

[0048] The minimum expected entropy value is a fixed value that adjusts the expected minimum entropy of the probability distribution defined by the policy outputs generated by the policy neural network during training.

[0049] In some embodiments, the minimum expected entropy is a constant provided as an input to the system.

[0050] In some other embodiments, the system determines the minimum expected entropy value before training. For example, when the action space is continuous, the minimum expected entropy value can be determined based on the number of action dimensions of the actions in the set of possible actions. As a specific example, the minimum expected entropy value can be the negative of the number of action dimensions, i.e., such that when there are eight action dimensions, the minimum expected entropy value will be -8.

[0051] Typically, the system determines the update by using an objective function that depends, for each of the probability distributions, on a temperature parameter, an entropy of the probability distribution, and a minimum expected entropy value.

[0052] As a specific example, the objective function J(α) may satisfy

[0053]

[0054] Where α is the temperature parameter, E is the expectation operator, and a t is the state representation s in the tuple by processing the t-th tuple according to the current value of the policy parameter t And from the policy neural network π t The actions sampled by the generated probability distribution, and is the minimum expected entropy value.

[0055] As can be seen from the above formula of the objective function J(α), the optimizer of the objective function J(α) is always positive or negative infinity, since the function ignores the dependence of the probability distribution generated by the policy neural network on the temperature parameter. Therefore, instead of minimizing J(α), the system takes only one (or a few) gradient steps towards the minimization at each iteration, and then updates π t To better match the updated α, that is, the reinforcement learning objective and J(α) are jointly updated by double gradient descent.

[0056] Refer to the following Figure 3 To describe updating the current temperature parameter in more detail.

[0057] Figure 3 3 is a flow chart of an example process 300 for updating current temperature parameters. For convenience, process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a properly programmed control system (e.g., Figure 1 The control system 100) can perform process 300.

[0058] For each tuple, the system samples actions from a probability distribution generated by the policy neural network for the tuple (step 302). In other words, the system selects actions from a set of possible actions, where each action has a probability of being selected that depends on (e.g., is equal to) the probability of the action in the probability distribution.

[0059] For each tuple, the system determines the gradient of the objective function with respect to the temperature parameter, ie, the difference between (i) the negative of the logarithm of the probability assigned to the sampled action by the probability distribution and (ii) the minimum expected entropy value (step 304).

[0060] The system combines the gradients of the tuples (step 306), i.e., by summing or averaging the gradients of the tuples, and determines an update to the temperature parameter from the combined gradients (step 308). The system can determine the update by applying an update rule (e.g., an ADAM update rule or a stochastic gradient descent update rule) to the combined gradients used to determine the update.

[0061] The system then applies the update to the current temperature parameter, ie, subtracts the update from the current temperature parameter, to determine an updated temperature parameter (step 310).

[0062] The description of process 300 describes taking one gradient step for each tuple to update the temperature parameter. In some cases, the system performs multiple gradient steps (ie, two, five, or ten gradient steps) to iterate before determining the final update to the current temperature parameter.

[0063] This specification uses the term "configuration" in conjunction with system and computer program components. For a system of one or more computers, to be configured to perform a particular operation or action means that software, firmware, hardware, or a combination thereof that causes the system to perform the operation or action in operation has been installed on the system. For one or more computer programs, to be configured to perform a particular operation or action means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0064] The embodiments and functional operations of the subject matter described in this specification may be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium to be executed by a data processing device or to control the operation of the data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical signal, optical signal, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0065] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer, or a plurality of processors or computers. The apparatus may also be or further include a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may optionally include code that creates an execution environment for a computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0066] Computer programs (also referred to as or described as programs, software, software applications, apps, modules, software modules, scripts or codes) can be written in any form of programming language (including compiled languages ​​or interpreted languages ​​or declarative languages ​​or programming languages), and can be deployed in any form (including as independent programs or modules, components, subroutines, or other units suitable for computing environments). Programs can but do not have to correspond to files in a file system. Programs can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), or in a single file dedicated to the program in question, or in multiple collaborative files (e.g., files for storing one or more modules, subroutines or partial codes). Computer programs can be deployed to execute on one computer or on multiple computers, which are located at one location or distributed across multiple locations and interconnected by a data communication network.

[0067] In this specification, the term "database" is used broadly to refer to any collection of data: the data need not be structured in any particular way, or at all, and which may be stored in one or more storage devices in one or more locations. Thus, for example, an index database may include multiple collections of data, each of which may be organized and accessed differently.

[0068] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers located in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same one or more computers.

[0069] The processes and logic flows described in this specification may be performed by one or more programmable computers that execute one or more computer programs to perform functions by operating input data and generating outputs. The processes and logic flows may also be performed by a dedicated logic circuit system (e.g., FPGA or ASIC) or a combination of a dedicated logic circuit system and one or more programmed computers.

[0070] The computer that is suitable for executing computer programs can be based on a general or special microprocessor or both or any other kind of central processing unit. Generally speaking, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The elements of a computer are a central processing unit for carrying out or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by a special logic circuit system or can be incorporated into the special logic circuit system. Usually, the computer will also include one or more mass storage devices (such as disks, magneto-optical disks or optical disks) for storing data, or the computer will also be operably coupled to receive data from the mass storage device or transfer data to the mass storage device or carry out both. However, the computer is not necessarily provided with such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (such as a universal serial bus (USB) flash drive), just to name a few.

[0071] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (such as EPROM, EEPROM, and flash memory devices), magnetic disks (such as internal hard disks or removable disks), magneto-optical disks, CD-ROM disks, and DVD-ROM disks.

[0072] To provide interaction with a user, an embodiment of the subject matter described in this specification may be implemented on a computer having: a display device for displaying information to a user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor; and a keyboard and a pointing device, such as a mouse or a trackball, through which a user can provide input to the computer. Other types of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form (including acoustic input, voice input, or tactile input). Additionally, a computer may interact with a user by sending documents to and receiving documents from a device used by the user (e.g., by sending a web page to a web browser on a user's device in response to a request received from a web browser). Moreover, a computer may interact with a user by sending text messages or other forms of messages to a personal device (e.g., a smart phone running a messaging application) and receiving a response message back from the user.

[0073] The data processing apparatus for implementing the machine learning model may also include, for example, dedicated hardware accelerator units for handling the common and compute-intensive portions of the machine learning training or production (i.e., inference) workload.

[0074] Machine learning models can be implemented and deployed using a machine learning framework, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0075] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server) or a computing system that includes a middleware component (e.g., an application server) or a computing system that includes a front-end component (e.g., a user computer with a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in the present invention), or a computing system that includes any combination of one or more such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local area networks ("LANs") and wide area networks ("WANs"), such as the Internet.

[0076] The computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on corresponding computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., an HTML page) to a user device, for example, in order to display data to a user interacting with the device (the device acting as a client) and receive user input from the user. Data generated at the user device (e.g., the result of user interaction) can be received from the device at the server.

[0077] Although this specification includes many specific implementation details, these details should not be regarded as limitations on the scope of any invention or the content that may be required, but as descriptions of the features that can be directed to the specific embodiments of a particular invention. Some features described in the context of the separate embodiments in this specification can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or according to any suitable sub-combination. In addition, although the above may describe the features as working with some combinations and even initially requiring so, in some cases, one or more features from the required combination can be deleted from the combination. And the required combination can point to the variation of a sub-combination or a sub-combination.

[0078] Similarly, although operations are depicted in the drawings and described in the claims in a particular order, it should not be understood that such operations need to be performed in the particular order shown or in a sequential order, or that all illustrated operations need to be performed to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0079] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions set forth in the claims may be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or sequential order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method for controlling a robot to perform an action, the method comprising: Training a policy neural network having a plurality of policy parameters and for controlling the robot interacting with an environment, wherein the policy neural network is configured to: receive as input a state representation characterizing a state of the environment, and process the state representation according to the plurality of policy parameters to generate a policy output, wherein the policy output defines a probability distribution over a set of actions that can be performed by the robot, the training comprising: obtaining trajectory data comprising one or more tuples, each tuple identifying a state representation characterizing a state of the environment, an action performed by the robot when the environment is in the state represented by the state representation, a reward received by the robot in response to performing the action, and a next-state representation characterizing a next state of the environment after the robot performs the action; updating current values ​​of the plurality of policy parameters using the trajectory data using a maximum entropy reinforcement learning technique that maximizes both (i) a reward term measuring a total reward in the one or more tuples in the trajectory data and (ii) an entropy term based on the current values ​​of the plurality of policy parameters, the reward term measuring the total reward in the one or more tuples in the trajectory data and the entropy term measuring the entropy of a probability distribution defined by policy outputs generated by processing the state representations in the one or more tuples in the trajectory data, wherein a relative weight between the entropy term and the reward term in the maximization is determined by a temperature parameter; and updating the temperature parameter to adjust the expected entropy of the probability distribution to be at least equal to a minimum expected entropy value by using the probability distribution defined by the policy output generated according to the current values ​​of the plurality of policy parameters for the one or more tuples in the trajectory data, and The robot is controlled using the policy neural network according to the updated values ​​of the plurality of policy parameters.

2. The method of claim 1 , wherein updating the temperature parameter comprises: The temperature parameter is updated by using an objective function, which, for each of the probability distributions, depends on the temperature parameter, the entropy of the probability distribution, and the minimum expected entropy value.

3. The method of claim 2, wherein the minimum expected entropy value is based on the number of action dimensions of the actions in the action set.

4. The method according to claim 3, wherein the minimum expected entropy value is the negative of the number of action dimensions.

5. The method of claim 2, wherein updating the temperature parameter comprises: determining, for each tuple of the one or more tuples, a gradient of the temperature parameter with respect to the objective function; as well as The temperature parameter is updated by using the determined gradient.

6. The method of claim 5, wherein determining the gradient comprises, for each tuple of the one or more tuples: sampling an action from the probability distribution generated by the policy neural network for the tuple; and A difference is determined between (i) the negative of the logarithm of the probability assigned to the sampled action by the probability distribution and (ii) the minimum expected entropy value.

7. The method according to any one of claims 2 to 6, wherein the objective function satisfies: Where α is the temperature parameter, E is the expectation operator, a t is for the t-th tuple by processing the state representation s in the tuple according to the current values ​​of the plurality of policy parameters t And from the policy neural network π t the action of sampling the probability distribution generated, and is the minimum expected entropy value.

8. One or more computer-readable storage media storing instructions, which, when executed by one or more computers, cause the one or more computers to perform corresponding operations in the method according to any one of claims 1-7.

9. A system comprising one or more computers and one or more storage devices storing instructions, wherein when the instructions are executed by the one or more computers, the one or more computers are caused to perform corresponding operations in the method according to any one of claims 1 to 7.

Images