Reinforcement learning with integration of discriminator model

Abstract

This specification describes a method performed by one or more data processing devices, comprising: sampling latent variables from a possible set of latent variables; selecting an action to be performed by an agent to interact with an environment on a time-step sequence using an action selection neural network conditioned on the sampled latent variables; determining a corresponding reward to be received for each time step in the time-step sequence using an ensemble of discriminator models; and training the action selection neural network based on the reward using reinforcement learning techniques. Each discriminator model can process observations to generate a corresponding prediction output, the corresponding prediction output predicting which latent variable the action selection neural network conditioned on to bring the environment into a state represented by the observation.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Application Serial No. 63 / 194,798, filed May 28, 2021, entitled “REINFORCEMENT LEARNING USING AN ENSEMBLE OF DISCRIMINATOR MODELS”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This manual relates to processing data using machine learning models. Background Technology

[0004] Machine learning models receive input and generate outputs, such as predicted outputs, based on the received input. Some machine learning models are parametric models and generate outputs based on the received input and the values ​​of the model's parameters.

[0005] Some machine learning models are deep models that use multiple layers to generate outputs from received inputs. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, each of which applies a non-linear transformation to the received input to generate an output. Summary of the Invention

[0006] This specification generally describes a system implemented as a computer program on one or more computers at one or more locations, which trains an action selection neural network to select actions to be performed by an agent interacting with the environment.

[0007] Even in the absence of "task rewards," such as extrinsic rewards that characterize the agent's progress in completing tasks in the environment, the system described in this paper can train an action selection neural network.

[0008] For example, the system can initially use reinforcement learning techniques to train the action selection neural network based solely on "unsupervised" rewards generated by an ensemble of discriminator models. After pre-training the action selection neural network using unsupervised rewards, the system can then train the neural network based on task rewards or a combination of task rewards and unsupervised rewards.

[0009] The system can use any appropriate reinforcement learning technique, such as Q-learning or policy gradient techniques, to train the action selection neural network.

[0010] According to a first aspect, a method is provided performed by one or more data processing devices, comprising: sampling latent variables from a possible set of latent variables; selecting an action to be performed by an agent to interact with an environment on a time-step sequence using an action selection neural network conditioned on the sampled latent variables; determining a corresponding reward to be received for each time step in the time-step sequence; and training the action selection neural network based on the reward using reinforcement learning techniques.

[0011] Determining the appropriate reward received for each time step in a time-step sequence may include, for each of one or more time steps: providing each discriminator model in an ensemble of multiple discriminator models with an observation representing the state of the environment at that time step, wherein each discriminator model processes the observation to generate a corresponding predictive output that predicts which latent variable from a possible set of latent variables the action selection neural network should conditioned to bring the environment into the state represented by the observation, and determining the reward for the time step based at least in part on a measure of inconsistency between the predictive outputs generated by the ensemble of multiple discriminator models.

[0012] In some implementations, for each of one or more time steps, the prediction output generated by each discriminator model for that time step includes a corresponding score distribution over a set of possible latent variables, which defines a corresponding score for each latent variable in the set of possible latent variables.

[0013] In some implementations, the method further includes, for each of one or more time steps, determining a measure of inconsistency between the predicted outputs generated by the ensemble of multiple discriminator models, including: determining a combined score distribution over a possible set of latent variables for the time step by combining the respective score distributions generated by each discriminator model for the time step; and determining the inconsistency measure based on: (i) the combined score distribution for the time step, and (ii) the respective score distribution generated by each discriminator model for the time step.

[0014] In some implementations, determining the combined score distribution over the possible set of latent variables for a given time step involves averaging the score distribution generated by the discriminator model for that time step.

[0015] In some implementations, determining the inconsistency metric based on (i) the combined score distribution for the time step and (ii) the corresponding score distribution generated by each discriminator model for the time step includes: determining the entropy of the combined score distribution for the time step; determining the corresponding entropy of the corresponding score distribution generated by each discriminator model for the time step; and determining the inconsistency metric based on the corresponding entropy of the combined score distribution for the time step and the corresponding score distribution generated by each discriminator model for the time step.

[0016] In some implementations, determining the inconsistency metric based on the corresponding entropy of the combined score distribution for the time step and the corresponding score distribution generated by each discriminator model for the time step includes: determining the inconsistency metric as the difference between (i) the entropy of the combined score distribution for the time step and (ii) the average of the entropies of the score distributions generated by each discriminator model for the time step.

[0017] In some implementations, determining the reward for each of one or more time steps includes: determining the corresponding accuracy of the prediction output generated by each discriminator model for the time step; and determining the reward for the time step based at least in part on the accuracy of the prediction output generated by the discriminator model for the time step.

[0018] In some implementations, determining the reward for a time step based at least in part on the accuracy of the prediction output generated by the discriminator model for that time step includes determining the reward based at least in part on the average accuracy of the prediction output generated by the discriminator model for that time step.

[0019] In some implementations, each discriminator model has a corresponding set of discriminator model parameters, and the values ​​of the corresponding set of discriminator model parameters are different for each discriminator model.

[0020] In some implementations, each discriminator model is trained on batches of training examples sampled independently from the replay memory.

[0021] In some implementations, each training example in the replay memory includes: (i) training observations representing the state of the environment during previous interactions between the agent and the environment, and (ii) a target output defining the action selection neural network as a latent variable conditioned to bring the environment into the state represented by the observations.

[0022] In some implementations, before being trained, the corresponding values ​​of the discriminator model parameter set for each discriminator model are initialized to be different from the discriminator model parameter set for each other discriminator model.

[0023] In some implementations, the method further includes, for each time step in the time step sequence: determining a reward for that time step based at least in part on a task reward that measures the agent’s progress in completing a task in the environment.

[0024] In some implementations, the set of possible latent variables includes only a finite number of possible latent variables.

[0025] In some implementations, using an action selection neural network conditioned on sampled latent variables to select actions to be performed by the agent to interact with the environment over a sequence of time steps includes, for each time step: using the action selection neural network to process observations and sampled latent variables characterizing the state of the environment at that time step to generate an action selection output; and selecting an action to be performed at that time step based on the action selection output.

[0026] In some implementations, for each time step, the action selection output includes the corresponding score for each action in the set of possible actions.

[0027] In some implementations, selecting the action to be performed at each time step based on the action selection output includes selecting the action with the highest score based on the action selection output.

[0028] In some implementations, the method further includes: after training the action selection neural network, using the action selection neural network to select actions to be performed by a real-world agent interacting with the real-world environment.

[0029] According to a second aspect, a system is provided, comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of any of the corresponding methods of the foregoing aspects.

[0030] According to a third aspect, one or more non-transient computer storage media are provided that store instructions, which, when executed by one or more computers, cause the one or more computers to perform the operations of any of the corresponding methods of the foregoing aspects.

[0031] The subject matter described in this specification can be implemented in particular embodiments to achieve one or more of the following advantages.

[0032] The system described in this specification can use "unsupervised" rewards to train an action selection neural network. These "unsupervised" rewards are generated using an ensemble of discriminator models, as an alternative to or in combination with "task" rewards for training the action selection neural network. To train the action selection neural network using unsupervised rewards, the system can sample latent variables and then select actions to be performed by the agent to interact with the environment on a time-step sequence, while the action selection neural network is conditioned on the sampled latent variables. Unsupervised rewards can be different from external (e.g., task) rewards, for example, by not being received from external sources (e.g., from the environment). For example, unsupervised (or intrinsic) rewards can be rewards determined by the system based on observations of the environment after the action is performed. Unsupervised rewards can also be based on latent variables used by the system to regulate action selection.

[0033] Based on the states of the environment during the agent's interaction with the environment, the system can determine unsupervised rewards in part based on the accuracy with which the discriminator model can predict the latent variables conditioned on by the action selection neural network. Training the action selection neural network on these unsupervised rewards allows it to learn "skills," resulting in consistent and recognizable changes to the environmental state. A skill is defined as an action selection policy conditioned on latent variables by the action selection neural network.

[0034] Action selection neural networks can learn to perform tasks in an environment more quickly by leveraging diverse, consistent, and identifiable sets of skills learned using unsupervised rewards, for example, with less training data and fewer training iterations. However, determining unsupervised rewards based on the accuracy of the discriminator model in predicting latent variables conditioned on it can hinder exploration of the environment. Specifically, the discriminator model's predictive accuracy may decrease as the agent explores new parts of the environment (because they haven't yet been trained on observations from these new parts), thus reducing the reward based on the discriminator model's predictive accuracy. This hindering of the agent's exploration of the environment can limit its ability to learn how to perform tasks effectively within that environment.

[0035] To encourage exploration, the system described herein uses an ensemble of discriminator models, where each discriminator model is trained on independently sampled batches of training examples and generates unsupervised rewards based at least in part on a measure of inconsistency between predictions generated by the discriminator models. As the agent explores new parts of the environment, the predictions generated by the discriminator models tend to be inconsistent, thus tending to increase the value of the unsupervised reward and encouraging exploration. By determining the unsupervised reward based on a measure of inconsistency between discriminator models, the system can encourage the agent to explore the environment, enabling the agent to learn to perform tasks more effectively in the environment, for example, using less training data and fewer training iterations. In contrast, systems using a single discriminator model cannot leverage the inconsistency of discriminator models as a mechanism to encourage exploration, and therefore their performance may be less efficient than that of the system described in this specification.

[0036] Details of one or more embodiments of the subject matter of this specification are set forth in the following drawings and description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0037] Figure 1 This is a block diagram of an example action selection system that integrates a discriminator model.

[0038] Figure 2 The illustration shows an example technique for determining a measure of inconsistency between predicted outputs generated by an ensemble of discriminator models.

[0039] Figure 3 This is a flowchart of an example process for training an action selection neural network.

[0040] Figure 4A and Figure 4B The illustration shows the results of an example experiment implemented using an action selection system.

[0041] In the various figures, the same reference numerals and names indicate the same elements. Detailed Implementation

[0042] Figure 1 An example action selection system 100 is shown that can control an agent 104 interacting with an environment 106. The action selection system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, wherein the systems, components and techniques described below are implemented in this example.

[0043] System 100 can select an action 102 to be performed by agent 104 to interact with environment 106 over a sequence of time steps to achieve a goal. At each time step, system 100 can receive data characterizing the current state of environment 106 and select the action 102 to be performed by agent 104 in response to the received data. The data characterizing the state of environment 106 will be referred to as observation 110 in this specification and may include, for example, images or any other suitable data. In some cases, agent 104 may be, for example, a robot, and observation 110 may include, for example, joint positions, velocities, and torques or any other suitable data, as described in more detail below. At each time step, the state of environment 106 at that time step (as characterized by observation 110) depends on the state of environment 106 at the previous time step and the action 102 performed by agent 104 at the previous time step.

[0044] System 100 can use an action selection neural network 120 conditioned on latent variable 118 to select an action 102 to be performed by agent 104 to interact with environment 106 over a time-step sequence. System 100 can sample latent variable 118 from a possible set of latent variables (e.g., randomly) and condition the action selection neural network 120 on latent variable 118. System 100 can then use the action selection neural network 120 conditioned on latent variable 118 to select the action 102 to be performed by agent 104 over a time-step sequence. Typically, "conditioning" the action selection neural network 120 on latent variable 118 can mean providing latent variable 118 as input to the action selection neural network 120. In some implementations, the possible set of latent variables may include only a finite number of possible latent variables, such as 10, 100, 1000, 10000, or any other suitable number of latent variables. In some implementations, the set of possible latent variables can include an infinite number of possible latent variables; for example, the set of possible latent variables can be a continuous range [0,1].

[0045] Each latent variable can represent a "skill," such as a set of actions characterizing the behavior of agent 104 over a time-step sequence. That is, each latent variable can prompt agent 104 to interact with environment 106 by causing the state of environment 106 to change in a consistent and identifiable manner over the time-step sequence. In other words, a "skill" can refer to an action selection policy defined by conditioned the action selection neural network 120 on latent variables 118. For example, a skill can refer to the action selection policy π. θ(a|s, z), which is a distribution that maps from state s and latent variable z to action a, where θ is the set of parameters of action selection neural network 120. As described in more detail below, by training action selection neural network 120 on unsupervised reward 109, system 100 enables action selection neural network 120 to learn different identifiable skills.

[0046] As a specific example, if agent 104 is a physical robot with a robotic arm, then the first latent variable can define a first set of possible actions for the robot, which includes actions of moving forward, backward, left, and right. The second latent variable can define a second set of possible actions for the robot, which includes actions of moving the robotic arm forward, backward, left, and right. In this case, the first and second latent variables can respectively represent different identifiable skills of the robot.

[0047] After conditioned the action selection neural network 120 on latent variable 118, the system 100 can use the action selection neural network 120 to select an action 102 to be performed by the agent 104 to interact with the environment 106 over a sequence of time steps. For example, at each time step, the system 100 can use the action selection neural network 120 to process observations 110 and latent variables 118 that characterize the current state of the environment 106 at that time step to generate an action selection output 122. In some implementations, in addition to processing observations 110 that characterize the current state of the environment 106 at a time step, the system 100 may also process one or more observations, each representing the state of the environment at a corresponding previous time step.

[0048] Action selection output 122 may include a score for each action in the set of possible actions that can be performed by agent 104. In some implementations, system 100 may select the action with the highest score based on action selection output 122 as the action to be performed by agent 104 at time step. In some implementations, system 100 selects action 102 to be performed by agent 104 based on an exploration strategy. For example, system 100 may use a greedy exploration strategy. In this example, system 100 may select the action with the highest score with probability 1-∈ (based on action selection output 122) and randomly select an action with probability ∈, where ∈ is a number between 0 and 1.

[0049] After the system 100 selects an action 102 to be performed by the agent 104 at a time step, the agent 104 interacts with the environment 106 by performing the action 102, and the system 100 can receive a reward based on the interaction, such as a task reward 108, an unsupervised reward 109, or both.

[0050] For example, at each time step, system 100 can receive a task reward 108 based on the current state of environment 106 and the action 102 of agent 104 at that time step. Typically, task reward 108 can be represented as a numerical value. Task reward 108 can be based on any event in or around environment 106. For example, task reward 108 can indicate whether agent 104 has completed a task (e.g., physically manipulated an object in the environment in a desired manner) or the progress agent 104 has made in completing the task.

[0051] Additionally or alternatively, system 100 may receive an unsupervised reward 109. System 100 may use an ensemble 150 of multiple discriminator models to determine a corresponding unsupervised reward 109 for each of one or more time steps. Each discriminator model in the ensemble 150 may be configured to process observations 110 representing the current state of environment 106 and generate a corresponding predictive output that predicts which latent variable 118 the action selection neural network 120 conditioned on to cause environment 106 to enter the state represented by observation 110. In other words, each discriminator model may generate a prediction about which skill the action selection neural network 120 uses to generate action selection output 122.

[0052] The predictive output of a discriminator model can include a score distribution over a set of possible latent variables, which defines the corresponding score for each latent variable in the set. For example, at time step t, with a parameter set φ i The discriminator model i can handle observations that characterize the current state of environment 106. t To generate the possible set of latent variables z t Fractional distribution on

[0053] Typically, the input processed by the discriminator model at a time step can include: (i) the observation at that time step, and optionally, (ii) the observation for one or more previous time steps. In some cases, the discriminator model can process the observation directly. In other cases, the discriminator model can process feature representations of the observation, such as feature representations generated by processing the observation using the encoder subnetwork of an autoencoder neural network.

[0054] See below for reference. Figure 2In more detail, system 100 can use the predicted output of discriminator model 150 to determine the unsupervised reward 109 for that time step. For example, at each of one or more time steps, system 100 can determine an inconsistency measure between the predicted outputs generated by the ensemble of multiple discriminator models 150, and determine the unsupervised reward 109 based on the inconsistency measure. (An "inconsistency measure" between the predicted outputs generated by the ensemble of discriminator models refers to a measure of dissimilarity between the predicted outputs generated by the ensemble of discriminator models. Several examples of inconsistency measures will be described in more detail below). In some implementations, system 100 can additionally determine the unsupervised reward based on the accuracy of the predicted outputs generated by the discriminator models. Accuracy can reflect, for example, the effective number of skills that the action selection neural network 120 has learned at the time step. As a specific example, the unsupervised reward r at that time step can be represented as:

[0055] r = r skil +λr DIS (1)

[0056] Where r skill It is a reward term that characterizes the accuracy of the predicted output generated by the ensemble of discriminator models at time step, r DIS The reward term is a measure of inconsistency between the predicted outputs generated by the ensemble of discriminator models at time steps 150, and λ is a tunable weighting factor. The reward term r skill It can be in the following forms, for example:

[0057] f skil =logq φ (z t |o t )-logp(z t (2)

[0058] Where q φ (z t |o t ) is the average score predicted by the discriminator model ensemble against the action selection neural network 120, conditioned on latent variables 118 (e.g., correct latent variables). In other words, q φ (z t |o t ) represents (characterizes) the accuracy of the predicted output generated by the integration of the discriminator model 150.

[0059] In equation (2), p(z) t () is the set of possible latent variables z at that time step. t The prior score distribution on the [z]. In some implementations, system 100 can be based on the prior score distribution p(z). tThe latent variable 118 is sampled. For example, if the previous score distribution on the set of latent variables is uniform, then system 100 can sample the latent variables with equal probability.

[0060] In some implementations, the prior score distribution p(z) on the set of latent variables t The numbers are fixed uniformly, with a total of N skills. In this case, if the discriminator model generates a uniform prediction of the action selection neural network 120 with its latent variables as conditions, then logq φ (z t |o t ) = -logN, resulting in r skill =0. This could indicate that the estimated number of skills already learned by the action selection neural network 120 is low or negligible, corresponding to low or negligible rewards. On the other hand, if the discriminator model accurately predicts the latent variables conditioned on the action selection neural network 120, then log q φ (z t |o t ) = 0, resulting in r skill =logN. This indicates that the estimated number of skills learned is relatively high, thus leading to a higher reward.

[0061] As described above, system 100 can also determine r based on a measure of inconsistency between the predicted outputs generated by the ensemble 150 of the discriminator models, for example, by determining r in equation (1) above. DIS To generate unsupervised rewards. The following will refer to... Figure 2 Describe the process in more detail.

[0062] Typically, a high inconsistency metric between predictions generated by the discriminator model 150 can indicate that the agent 104 has encountered a “new” state of the environment 106 at that time step, such as a part of the environment 106 that it was unlikely to have previously accessed. In other words, a high inconsistency metric can indicate that the discriminator model is substantially inconsistent about which latent variable 118 the action selection neural network 120 conditioned to change the environment state to the state represented by the observation 110 at that time step. To encourage exploration of different parts of the environment 106, the system 100 can accordingly select r that leads to the above equation (1). DIS The neural network 120 selects reward actions based on the increasing value of the action, thereby responding to new states encountered in the environment 106.

[0063] In contrast, a low inconsistency metric can indicate that some or all of the discriminator models 150 are consistent with their predictions, thus indicating a relatively high probability that agent 104 previously visited a portion of the environment 106 represented by observation 110 at a time step. In this case, system 100 can generate a relatively low r in equation (1) above. DIS Value. By generating unsupervised reward 109 according to equation (1) and training action selection neural network 120 on unsupervised reward 109, system 100 can encourage action selection neural network 120 to learn more distinguishable and recognizable skills and explore new states of environment 106.

[0064] As described above, in addition to receiving the unsupervised reward 109, the system may also receive a task reward 108 representing the progress of agent 104 in achieving the goal. For example, if agent 104 completes the task at a time step, the task reward may be positive (e.g., 1). Otherwise, the reward may be zero. In some implementations, at each time step, the action selection system 100 may also determine the overall reward based on the task reward 108 and the unsupervised reward 109. The overall reward for that time step may be, for example, a linear combination of the task reward 108 and the unsupervised reward 109. In some implementations, the system 100 may generate the unsupervised reward 109 at each time step in the time step sequence or only at one or more time steps in the time step sequence. For example, the system 100 may generate the unsupervised reward 109 only at the last time step, such as at the end of a scene, where a "scene" refers to a sequence of time steps conditioned on the action selection neural network 120 with the same latent variable 118.

[0065] The action selection neural network 120 can have any suitable neural network architecture that enables it to perform the described function, such as processing latent variables 118 and observations 110 to generate an action selection output 122 that represents an action 102 to be performed by agent 103 in response to observations 110. For example, the action selection neural network 120 may include any suitable type (e.g., fully connected layers, convolutional layers, attention layers, transformer layers, etc.) and any suitable number of layers (e.g., 5, 10, or 25 layers) connected in any suitable configuration (e.g., as a linear sequence of layers).

[0066] Each discriminator model in the ensemble of discriminator models 150 can have any suitable machine learning model architecture, such as a neural network architecture, a support vector machine architecture, or a random forest architecture. In some implementations, each discriminator model shares the same architecture (e.g., the same neural network architecture), but the corresponding values ​​of the discriminator model parameters are different for each discriminator model, for example, due to training on independently sampled batches of training examples, or due to different initialized parameter values. The ensemble of multiple discriminator models 150 can include any suitable number of discriminator models, such as 2, 10, 100, 1000, or any other suitable number of discriminator models.

[0067] Training engine 112 can train action selection neural network 120 using reinforcement learning techniques based on the unsupervised reward 109 described above with reference to equation (1). Training engine 112 trains action selection neural network 120 by iteratively adjusting the parameters of action selection neural network 120. Training engine 112 can adjust the parameters of action selection neural network 120 by iteratively backpropagating the gradient of reinforcement learning objective function (e.g., Q-learning objective function, policy gradient objective function, or any other suitable reinforcement learning objective function) through action selection neural network 120. By training action selection neural network 120, training engine 112 can make action selection neural network 120 select actions that increase the cumulative measure (e.g., the cumulative total reward of long-term time discount) of the unsupervised reward 109 received by action selection neural network 120. In some implementations, training engine 112 can train action selection neural network 112 based on the total reward (e.g., a combination of unsupervised reward 109 and task reward 108) using the reinforcement learning techniques described above.

[0068] Furthermore, the training engine 112 can use any suitable training technique (e.g., supervised learning) to train the ensemble 150 of multiple discriminator models. Each discriminator model in the ensemble 150 can have a corresponding set of discriminator model parameters. In some implementations, the training engine 112 can initialize the corresponding values ​​of the respective discriminator model parameter sets to be different for each discriminator model in the ensemble 150. For example, the training engine 112 can initialize the parameters by randomly sampling each parameter from a distribution (e.g., a standard normal distribution).

[0069] In some implementations, training engine 112 may use training examples stored in replay memory 114 to train the discriminator model ensemble 150. Replay memory 114 may be implemented as, for example, a logical data storage area or a physical data storage device. Replay memory 114 may store a corresponding “empirical tuple” for each of a plurality of previous time steps (e.g., memory 114 may store a corresponding empirical tuple for each time step prior to the current time step). An empirical tuple for a time step refers to data characterizing the interaction between agent 104 and environment 106 at a previous time step. For example, an empirical tuple for a previous time step may include: (i) training observations 110 characterizing the state of environment 106 during previous interactions between agent 104 and environment 106, and (ii) a target output defining a latent variable 118 for action selection neural network 120 to conditionally cause environment 106 to enter the state characterized by observations 110.

[0070] In each training iteration, training engine 112 can sample batches of training examples from replay memory 114 and use the ensemble 150 of discriminator models to process the training input specified by the training examples. In some implementations, training engine 112 can train each discriminator model in ensemble 150 on independently sampled batches of training examples; for example, training engine 112 can independently sample the corresponding batches of training examples from replay memory 114 and provide the training input specified by each batch to the corresponding discriminator model.

[0071] Each discriminator model can process the corresponding training observations specified by the corresponding training example according to the corresponding discriminator model parameter set to generate a corresponding prediction output, which includes the corresponding score distribution on the possible set of latent variables.

[0072] For each discriminator model, the training engine 112 can determine the gradient of the objective function with respect to the corresponding discriminator model parameters. The objective function can measure: (i) the score distribution defined by the predicted output, and (ii) the error, such as cross-entropy error, between the target output of the action selection neural network 120 and its latent variables 118 conditioned upon it. As a specific example, the training engine 112 can train the discriminator model parameters φ for each discriminator model by minimizing the objective function:

[0073]

[0074] in This is the score distribution over the set of possible latent variables generated by the discriminator model. For each discriminator model, the training engine 112 can, for example, use backpropagation to determine the gradient of the objective function with respect to the discriminator model parameters. The training engine 112 can use the gradient to update the discriminator model parameters using the update rule of a gradient descent optimization algorithm (e.g., Adam or RMSprop).

[0075] After training the action selection neural network 120, the discriminator model integration 150, or both, the system 100 can use the action selection neural network 120 to select the action 102 to be performed by the agent 104 to interact with the environment 106.

[0076] The following sections will describe examples of possible agents, environments, and tasks in more detail.

[0077] In some implementations, the environment is a real-world environment, and the agent is a mechanical agent that interacts with the real-world environment. For example, the agent can be a robot that interacts with the environment to achieve a goal (or task), such as locating an object of interest in the environment, moving the object of interest to a designated location in the environment, physically manipulating the object of interest in the environment in a designated manner, or navigating to a designated destination in the environment; or the agent can be an autonomous or semi-autonomous land, air, or sea vehicle that navigates to a designated destination in the environment.

[0078] In these implementations, observations may include one or more of, for example, images, object location data, and sensor data, to capture observations, such as from images, distance or position sensors, or from actuators, as the agent interacts with the environment.

[0079] For example, in the case of a robot, observations may include data characterizing the robot’s current state, such as one or more of the following: joint positions, joint velocities, joint forces, torques or accelerations (e.g., gravity-compensated torque feedback), and the global or relative orientation of an object held by the robot.

[0080] In the case of robots or other intelligent mechanical agents or vehicles, observation can similarly include the position, linear or angular velocity, force, torque or acceleration of one or more parts of the agent, and one or more of its global or relative attitude. Observation can be defined in 1D, 2D, or 3D, and can be absolute and / or relative.

[0081] Observations may also include, for example, data obtained by one or more sensor devices that sense the real-world environment; such as sensed electronic signals, like motor current or temperature signals; and / or, for example, image or video data from a camera or LIDAR sensor, such as data from a sensor of an agent or data from a sensor located separately from the agent in the environment.

[0082] In the case of electronic agents, observation can include data from one or more sensors monitoring a part of a factory or service facility, such as current, voltage, power, temperature, and other sensors and / or electronic signals representing the operation of electronic and / or mechanical items of the equipment. Electronic agents can generate actions to control, for example, machines in an industrial facility or cooling systems in a data center.

[0083] The action can be a control signal that controls a robot or other intelligent mechanical agent, such as the torque of a robot's joints or a higher-level control command, or an autonomous or semi-autonomous land, air, or sea vehicle, such as the torque of a control surface or other control element of the vehicle or a higher-level control command.

[0084] Control signals may include, for example, position, velocity, or force / torque / acceleration data of one or more joints of a robot or a part of another mechanical agent. Control signals may also, or alternatively, include electronic control data, such as motor control data, or more generally, data for controlling one or more electronic devices within an environment, the control of which has an impact on the observed environmental state. For example, in the case of autonomous or semi-autonomous land, air, or sea vehicles, control signals may define actions to control navigation (e.g., steering) and movement (e.g., braking and / or acceleration of the vehicle).

[0085] In some implementations, the environment is a simulated environment, and the agent is implemented as one or more computers interacting with the simulated environment. In other implementations, the environment is a simulation of the aforementioned real-world environment. For example, the simulated environment could be a simulation of a robot or vehicle, and the reinforcement learning system could be trained on the simulation and then used in the real world once trained.

[0086] For example, the simulation environment can be a motion simulation environment, such as a driving simulation or a flight simulation, and the intelligent agent can be a simulated vehicle navigating in the motion simulation. In these implementations, the action can be a control input that controls the simulated user or the simulated vehicle.

[0087] In another example, the simulated environment could be a video game, and the agent could be a simulated user playing the video game.

[0088] In some implementations, the environment is a real-world manufacturing environment used to manufacture products (such as chemical, biological, or mechanical products or food). As used herein, "manufacturing" a product also includes refining raw materials to produce the product, or disposing of raw materials, such as removing contaminants to produce a clean or recyclable product. A manufacturing plant may include multiple manufacturing units (such as containers for chemical or biological substances) or machines (such as robots) for handling solids or other materials. Manufacturing units are configured such that intermediate versions or components of the product are movable between manufacturing units during product manufacturing, for example, via pipes or mechanical transport. As used herein, product manufacturing also includes manufacturing food using kitchen robots.

[0089] The intelligent agent may include an electronic intelligent agent or a machine, such as a robot, configured to control a manufacturing unit to manufacture a product. That is, the intelligent agent may include a control system configured to control the manufacture of a chemical, biological, or mechanical product. For example, the control system may be configured to control one or more manufacturing units or machines, or to control the movement of intermediate versions or components of a product between manufacturing units or machines.

[0090] As an example, tasks performed by an agent may include manufacturing a product or an intermediate version or component thereof. As another example, tasks performed by an agent may include controlling (e.g., minimizing) the use of resources, such as controlling electricity or water consumption or the consumption of any materials or consumables used in the manufacturing process.

[0091] These actions can include control actions to control the use of machines or manufacturing cells for handling solid or liquid materials to manufacture a product or its intermediates or components, or to control the movement of intermediate versions of a product or components within a manufacturing environment (e.g., between manufacturing cells or machines). Typically, the action can be any action that affects observed environmental conditions, such as actions configured to adjust any sensed parameters described below. These can include actions that adjust the physical or chemical conditions of a manufacturing cell, or actions that control the movement of mechanical parts of a machine or joints of a robot. These actions can include actions that impose operating conditions on a manufacturing cell or machine, or actions that result in changes to settings to adjust, control, or turn the operation of the manufacturing cell or machine on or off.

[0092] Rewards or rewards can be related to performance criteria for a task. For example, in the case of a task that involves manufacturing a product, the criteria could include the quantity of products manufactured, the quality of the products, the production speed of the products, or the physical costs of performing the manufacturing task, such as the amount of energy, materials, or other resources used to perform the task. In the case of a task that involves controlling the use of resources, the criteria could include any criteria for resource usage.

[0093] Typically, observations of environmental conditions can include any electronic signals representing the operation of electronic and / or mechanical components of a device. For example, representations of environmental conditions can be derived from observations made by sensors that sense the state of the manufacturing environment (e.g., sensors that sense the state or configuration of a manufacturing cell or machine, or sensors that sense the movement of materials between manufacturing cells or machines).

[0094] As some examples, such sensors can be configured to sense mechanical movement or force, pressure, temperature; electrical conditions such as current, voltage, frequency, impedance; quantity, level, flow / movement rate, or flow / movement path of one or more materials; physical or chemical conditions such as physical state, shape, or configuration, or chemical state such as pH; configuration of a unit or machine, such as the mechanical configuration of a unit or machine or valve configuration; image or video sensors for capturing image or video observations of the manufacturing unit or machine or movement; or any other suitable type of sensor. In the case of a machine such as a robot, observations from sensors can include observations of the position, linear or angular velocity, force, torque or acceleration, or attitude of one or more parts of the machine, such as data characterizing the current state of the machine or robot or the current state of an item held or handled by the machine or robot. Observations can also include, for example, sensed electronic signals such as motor current or temperature signals, or image or video data, for example, from a camera or LiDAR sensor. Sensors such as these can be part of an agent in the environment or located separately from the agent.

[0095] In some other applications, intelligent agents can control actions in real-world environments, including equipment and objects, such as in data centers or mains power or water distribution systems, or in manufacturing plants or service facilities. Observations may then be related to the operation of the plant or facility. For example, observations may include observations of equipment's electricity or water usage, or observations of power generation or distribution control, or observations of resource usage or waste generation. Intelligent agents can control actions in the environment to achieve efficiency goals, such as by reducing resource usage, and / or reducing the environmental impact of operations in the environment, such as by reducing waste. These actions may include actions of objects that control operating conditions or impose them on equipment in the plant / facility, and / or actions that cause changes in settings during the operation of the plant / facility, such as adjusting or turning on / off components of the plant / facility.

[0096] In some implementations, the environment is the real-world environment of a service facility comprising multiple items including electronic devices, such as a server farm or data center, like a telecommunications data center, or a computer data center for storing or processing data, or any service facility. The service facility may also include auxiliary control equipment that controls the operating environment of the equipment items, such as environmental control equipment, such as temperature control (e.g., cooling) equipment, or airflow control or air conditioning equipment. The task may include a task of controlling (e.g., minimizing) resource usage, such as controlling power consumption or water consumption. The agent may include an electronic agent configured to control the operation of the equipment items, or to control the operation of auxiliary (e.g., environmental) control equipment.

[0097] Typically, this action can be any action that affects the observed environmental state, such as an action configured to adjust any sensed parameters described below. These can include actions that control or apply operating conditions to equipment or auxiliary control devices, such as actions that cause settings to change to adjust, control, or turn the equipment or auxiliary control devices on or off.

[0098] Typically, observations of environmental conditions can include any electronic signals indicating the operation of a facility or equipment within a facility. For example, representations of environmental conditions can be derived from observations made by any sensor sensing the physical condition of the facility or by any sensor sensing the condition of one or more items of equipment or auxiliary control equipment. These include sensors configured to sense electrical conditions such as current, voltage, electricity, or energy; the temperature of the facility; the flow rate, temperature, or pressure of fluids within the facility or its cooling system; or physical facility configurations, such as whether vents are open.

[0099] Rewards or rewards can be related to performance criteria for a task. For example, in the case of a task that controls (e.g., minimizes) resource usage, such as controlling the use of electricity or water, the criterion can include any criterion for resource usage.

[0100] In some implementations, the environment is the real-world environment of the power generation facility, such as a renewable power generation facility like a solar farm or wind farm. The task may include control tasks related to controlling the electricity generated by the facility, such as controlling the delivery of electricity to the distribution network to meet demand or reduce the risk of mismatch between grid components, or maximizing the electricity generated by the facility. The intelligent agent may include an electronic agent configured to control the electricity generated by the facility or couple the generated electricity into the grid.

[0101] These actions can include actions that control the electrical or mechanical configuration of a generator (such as the electrical or mechanical configuration of one or more renewable power generation elements), for example, controlling the configuration of a wind turbine or one or more solar panels or reflectors, or the electrical or mechanical configuration of a rotating generator. For example, mechanical control actions can include actions that control the conversion of energy input to electrical output, such as the conversion efficiency or the degree to which energy input is coupled to electrical output. Electrical control actions can, for example, include actions that control one or more of the voltage, current, frequency, or phase of the generated electricity.

[0102] Rewards or rewards can be related to performance criteria for a task. For example, in the case of a task controlling the delivery of electricity to a distribution network, the criterion could involve a measure of the amount of electricity delivered, or a measure of electrical mismatch between the generating facility and the grid, such as voltage, current, frequency, or phase mismatch, or a measure of power or energy loss in the generating facility. In the case of a task maximizing the delivery of electricity to the distribution network, the criterion could involve a measure of the amount of power or energy delivered to the grid, or a measure of power or energy loss in the generating facility.

[0103] Typically, observations of environmental conditions can include any electronic signals representing the electrical or mechanical operation of the power generation equipment in a power generation facility. For example, representations of environmental conditions can be derived from observations made by any sensors that sense the physical or electrical condition of equipment in a power generation facility that is generating electricity, or the physical environment of such equipment, or the conditions of auxiliary equipment supporting the power generation equipment. Such sensors can include sensors configured to sense the electrical conditions of the equipment, such as current, voltage, power, or energy; temperature or cooling of the physical environment; fluid flow; or the physical configuration of the equipment; and observations of the electrical conditions of the power grid, such as from local or remote sensors. Observations of environmental conditions can also include one or more predictions about the future operating conditions of the power generation equipment, such as predictions of future wind speeds or solar irradiance, or predictions of future electrical conditions of the power grid.

[0104] As another example, the environment can be a chemical synthesis or protein folding environment, such that each state is a corresponding state of a protein chain or one or more intermediate or precursor chemicals, and the agent is a computer system for determining how to fold the protein chain or synthesize the chemical. In this example, the action is a possible folding action for folding the protein chain or an action for assembling the precursor chemical / intermediate, and the desired outcome can include, for example, folding the protein to make it stable and enabling it to perform a specific biological function or providing an efficient synthetic route for the chemical. As another example, the agent can be a mechanical agent that executes or controls protein folding actions or chemical synthesis steps automatically selected by the system without human interaction. Observations can include direct or indirect observation of the state of the protein or chemical / intermediate / precursor, and / or can be derived from a simulation.

[0105] Similarly, the environment could be a drug design environment, where each state corresponds to a potential drug state, and the agent is a computer system for determining the elements of the drug and / or the synthetic pathway. The drug / synthesis can be designed based on rewards derived from, for example, drug targets in a simulation. As another example, the agent could be a mechanical agent that performs or controls drug synthesis.

[0106] In some other applications, the environment is a real-world environment, and the agent manages the distribution of tasks across computing resources, such as on mobile devices and / or in a data center. In these implementations, actions may include assigning tasks to specific computing resources, and the objective may include minimizing the time required to complete a set of tasks using the specified computing resources.

[0107] As yet another example, the action could include presenting an advertisement, the observation could include ad impressions or click counts or rates, and the reward could represent one or more users' previous selections of an item or content. In this example, the objective to be achieved could include maximizing the selections of one or more users of an item or content.

[0108] In some cases, observation may include text or verbal instructions provided to the agent by a third party (such as the agent's operator). For example, the agent may be an autonomous vehicle, and the user of the autonomous vehicle may provide text or verbal instructions to the agent (such as navigating to a specific location).

[0109] As another example, the environment can be an electrical, mechanical, or mechatronic design environment, such as an environment that simulates the design of an electrical, mechanical, or mechatronic entity. The simulation environment can be a simulation of the real-world environment in which the entity is intended to operate. The task can be designing the entity. Observations can include observations characterizing the entity, i.e., observations of the entity's mechanical shape or electrical, mechanical, or mechatronic configuration, or observations of the entity's parameters or characteristics.

[0110] Actions can include actions that modify the entity, such as actions that modify one or more observations. Rewards or rewards can include one or more criteria for the entity's design performance. For example, rewards or rewards could relate to one or more physical characteristics of the entity (such as weight or force) or to one or more electrical characteristics of the entity (such as efficiency measures in performing the specific function the entity is designed to perform). The design process can include, for example, outputting a design for manufacturing in the form of computer-executable instructions for manufacturing the entity. The process can include manufacturing the entity according to the design. Thus, the design of the entity can be optimized, for example, through reinforcement learning, and then the optimized design output is used to manufacture the entity, for example, as computer-executable instructions; then the entity with the optimized design can be manufactured.

[0111] As previously described, the environment can be a simulated environment. Typically, in the case of a simulated environment, observations can include simulated versions of one or more of the previously described observations or observation types, and actions can include simulated versions of one or more of the previously described actions or action types. For example, the simulated environment can be a motion simulation environment, such as a driving simulation or flight simulation, and the agent can be a simulated vehicle navigating in the motion simulation. In these implementations, actions can be control inputs controlling a simulated user or a simulated vehicle. Typically, the agent can be implemented as one or more computers interacting with the simulated environment.

[0112] The simulated environment can be a simulation of a specific real-world environment and an intelligent agent. For example, the system can be used to select actions in the simulated environment during system training or evaluation, and after training or evaluation, or both, the system can be deployed to control a real-world agent in the specific real-world environment as the simulated agent. This avoids unnecessary wear and tear on the real-world environment or the real-world agent, and allows control of neural networks trained and evaluated in situations that rarely occur in the real-world environment or are difficult or unsafe to recreate. For example, a system can be partially trained using a simulation of a mechanical agent in a specific real-world environment, and then deployed to control a real mechanical agent in that specific real-world environment. Therefore, in this case, the observations in the simulated environment are relevant to the real-world environment, and the actions selected in the simulated environment are relevant to the actions to be performed by the mechanical agent in the real-world environment.

[0113] Optionally, in any of the above implementations, observations at any given time step may include data from previous time steps that may be helpful in characterizing the environment, such as actions performed at previous time steps, rewards received at previous time steps, etc.

[0114] The integration of the example discriminator model will be described in more detail below.

[0115] Figure 2 The diagram illustrates the method used to determine the reference above. Figure 1 An example technique for measuring inconsistency 240 between the predicted outputs generated by the integration of discriminator models included in the described action selection system 100.

[0116] System 100 can use a discriminator model to determine the unsupervised reward at each of one or more time steps in a time-step sequence. System 100 can use the unsupervised reward to train an action selection neural network 120 using reinforcement learning techniques. After training, system 100 can use the action selection neural network 120 to select actions to be performed by an agent interacting with the environment.

[0117] Each discriminator model can be configured to process observations 210 representing the environmental state at that time step and generate a corresponding prediction output that predicts which latent variable the action selection neural network 120 should condition to bring the environment into the state represented by observations 210. (Refer to the above...) Figure 1 The predicted output can be included as a score distribution over a set of possible latent variables, which defines the corresponding score for each latent variable in the set of possible latent variables.

[0118] To determine the unsupervised reward at each of one or more time steps in a time-step sequence, such as the reward defined above by equation (1), system 100 can determine an inconsistency measure 240 between the predicted outputs generated by the ensemble of multiple discriminator models. System 100 can determine r in equation (1) based on the inconsistency measure 240. DIS To determine unsupervised rewards.

[0119] In some implementations, to determine the inconsistency metric 240, system 100 can determine a combined score distribution 230 for that time step by combining (e.g., averaging) the respective score distributions 220 generated by each discriminator model in the ensemble for that time step. As a specific example, system 100 can determine the combined score distribution q. φ (Z|O), as follows:

[0120]

[0121] Where i represents the i-th discriminator model, and the sum is over N discriminator models. After determining the combined score distribution, for example by using equation (4), system 100 can determine the inconsistency measure based on the combined score distribution 230 for time step and the corresponding score distribution 220 generated by each discriminator model for time step.

[0122] In some implementations, system 100 can determine the entropy of each fraction distribution over a time step. For example, system 100 can determine the entropy (H) of the combined fraction distribution 230, as defined by equation (4) above, as follows:

[0123]

[0124] System 100 can also determine the corresponding entropy of the corresponding score distribution 220 generated by each discriminator model for the time step as follows:

[0125]

[0126] The system can use equations (5) and (6) above to determine the inconsistency metric 240 for a time step as the difference between the entropy of the combined score distribution for that time step and the average entropy of the score distribution generated by each discriminator model for that time step. Based on the inconsistency metric 240, the system 100 can determine the reward r for that step as follows: DIS :

[0127]

[0128] If the score distributions on the set of latent variables generated by some or all of the discriminator models in the ensemble are approximately equal, then the two terms in equation (7) are also approximately equal, resulting in a small or negligible reward r. DIS In contrast, if some or all of the discriminator models generate different score distributions on the set of latent variables, then the reward r... DIS Significantly higher. Therefore, by determining the unsupervised reward through the inconsistency measure 240 between the predicted outputs generated by the discriminator model, system 100 can encourage the exploration of new states.

[0129] In some implementations, system 100 can determine an inconsistency measure 240 between the predicted outputs generated by the ensemble of multiple discriminator models by determining the divergence measure of each pair of discriminator models in the ensemble. For example, system 100 can determine the divergence measure by determining the corresponding divergence (e.g., Kullback-Leibler divergence) between the score distributions generated by each pair of discriminator models (i.e., over the possible set of latent variables). System 100 can determine the inconsistency measure 240 based on the divergence measure, such as the mean or median of the divergence measure. System 100 can then determine the unsupervised reward based on the inconsistency measure 240, for example, in a manner similar to that described above.

[0130] In some implementations, system 100 may also determine the unsupervised reward r based on a measure of the accuracy of the predicted outputs generated by each discriminator model in the ensemble, as illustrated in equation (2). In another example, system 100 may determine the unsupervised reward as the average of the accuracy of the predicted outputs.

[0131] The following section will describe in more detail an example process for training the integration of the action selection neural network and the discriminator model.

[0132] Figure 3 It is used to train action selection neural networks (e.g.) Figure 1 A flowchart of an example process of an action selection neural network 120 (in this specification). For convenience, process 300 will be described as being executed by a system of one or more computers located in one or more locations. For example, an action selection system appropriately programmed according to this specification (e.g., Figure 1 The action selection system 100) can execute process 300.

[0133] The system samples latent variables from the set of possible latent variables (302). In some implementations, the set of possible latent variables may include only a finite number of possible latent variables. For example, the system may randomly sample latent variables from the set of possible latent variables.

[0134] The system uses an action selection neural network conditioned on sampled latent variables to select actions to be performed by the agent to interact with the environment at a time-step sequence (304). For example, the system can use the action selection neural network to process observations and sampled latent variables characterizing the environmental state at a time step to generate an action selection output. The action selection output may include, for example, a corresponding score for each action in a set of possible actions. The system can then select the action to be performed at that time step based on the action selection output; for example, the system can select the action with the highest score.

[0135] The system determines the appropriate reward (306) for each time step in the time step sequence. For each of the one or more time steps, the system provides each discriminator model in the ensemble of multiple discriminator models with an observation representing the environmental state at that time step. Then, for each of the one or more time steps, the system determines the reward for that time step based at least in part on a measure of inconsistency between the predicted outputs generated by the ensemble of the multiple discriminator models.

[0136] Each discriminator model can be configured to process observations to generate a corresponding predictive output that predicts which latent variable from the possible set of latent variables the action-selecting neural network should conditioned to bring the environment into the state represented by the observations. The predictive output may include, for example, a score distribution over the possible set of latent variables, which defines a corresponding score for each latent variable in the possible set of latent variables.

[0137] The system uses reinforcement learning techniques to train the action selection neural network based on rewards (308). In some implementations, the system may train the action selection neural network based on overall rewards, such as task rewards representing the agent's progress toward completing the goal and rewards based on a measure of inconsistency between the predicted outputs generated by the discriminator model.

[0138] In some implementations, the system can determine a measure of inconsistency between the predicted outputs generated by the ensemble of multiple discriminator models based on the combined score distribution over a possible set of latent variables. For example, the system can determine the combined score distribution for a given time step by combining (e.g., averaging) the respective score distributions generated by each discriminator model for that time step.

[0139] After determining the combined score distribution for that time step, the system can determine the inconsistency metric based on (i) the combined scores and (ii) the corresponding score distribution generated by each discriminator model for that time step. For example, the system can determine the inconsistency metric for that time step by determining the entropy of the combined score distribution. Further, the system can determine the corresponding entropy of the corresponding score distribution generated by each discriminator model.

[0140] The system can then determine an inconsistency metric based on these entropies. As a specific example, the system can determine the difference between (i) the entropy of the combined score distribution for that time step and (ii) the average of the entropies of the score distributions generated by each discriminator model for that time step.

[0141] In some implementations, the system may determine the reward for that time step based on the accuracy of the predicted output. For example, the system may determine the corresponding accuracy of the predicted output generated by each discriminator model for that time step. Further, the system may determine the reward for that time step based at least in part on the accuracy of the predicted output, for example, based at least in part on the average accuracy of the predicted output. In some implementations, the system may determine the reward for that time step based at least in part on a task reward that measures the agent's progress in completing a task in the environment.

[0142] As shown above (refer to the reference) Figure 1 The system described above can train an ensemble of discriminator models. Each discriminator model can have a corresponding set of discriminator model parameters. In some implementations, the values ​​of the corresponding discriminator model parameter sets can be different for each discriminator model. Before training the discriminator models, the system can initialize the values ​​of the discriminator model parameter sets for each discriminator model to be different from the discriminator model parameter sets for each other.

[0143] The system can train each discriminator model on batches of independently sampled training examples from the replay memory. Each training example in the replay memory may include, for example, (i) training observations representing the state of the environment during previous interactions between the agent and the environment, and (ii) a target output defining the action selection neural network as a latent variable to bring the environment into the state represented by the observations.

[0144] In some implementations, after training the action selection neural network, the system can use the neural network to select actions to be performed by a real-world agent interacting with the real-world environment. For example, the agent could be a physical robot in a physical environment, and the system could select actions to perform a specific task, such as moving the robot from a first location to a second location in the physical environment.

[0145] The results of the example experiment will be described in more detail below.

[0146] Figure 4A and Figure 4B The illustration shows the sample experimental results 400 achieved using the action selection system 100. Figure 4A and Figure 4B Each panel in the diagram illustrates a grid world with four rooms. Each grid world has a total of 104 states (e.g., a square in the grid). System 100 initializes the environment's first state to the square in the top left corner.

[0147] After initializing the first state, system 100 can sample latent variables from the possible set of latent variables and use an action selection neural network 120, conditioned on the sampled latent variables, to select actions to be performed by the agent in the grid world over a time-step sequence (e.g., 20 time steps). The time-step sequence is long enough to reach all but one state, allowing a total of 103 distinguishable skills. For example, these actions can include moving left, right, up, down, and staying in place. The set of latent variables includes 128 latent variables.

[0148] Figure 4A The illustration shows the performance of System 100 in the absence of unsupervised rewards generated based on a measure of inconsistency between predicted outputs produced by the ensemble of discriminator models. As shown, the agent enters almost three out of four rooms, thus failing to effectively explore the environment. In this case, the agent manages to learn a total of 30 different skills.

[0149] Figure 4B The performance of System 100 with unsupervised rewards is illustrated. As shown, the agent manages to efficiently explore the environment by entering all four rooms. Furthermore, in this case, the agent learns three times the number of skills, for example, approximately 90 different skills. Therefore, System 100 described in this specification enables the agent to learn more skills and explore new parts of the environment.

[0150] This specification uses the term "configuration" in relation to system and computer program components. Configuring one or more computer systems to perform a specific operation or action means that software, firmware, hardware, or a combination thereof are installed on the system, which in operation causes the system to perform the operation or action. Configuring one or more computer programs to perform a specific operation or action means that one or more programs include instructions that, when executed by a data processing device, cause that device to perform the operation or action.

[0151] The embodiments of the subject matter and functional operation described in this specification can be implemented in digital electronic circuit systems, tangible implementations of computer software or firmware, computer hardware (including structures disclosed in this specification and their structural equivalents), or combinations thereof.

[0152] Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transient storage medium for execution by a data processing apparatus or for controlling the operation of such a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination thereof. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagating signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) generated for encoding information to be transmitted to a suitable receiver device for execution by the data processing apparatus.

[0153] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of devices, apparatuses, and machines used for processing data, including, by example, programmable processors, computers, or multiple processors or computers. The apparatus may also be or further include special-purpose logic circuit systems, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0154] Computer programs (which may also be referred to or described as programs, software, software applications, applications, modules, software modules, scripts, or code) can be written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages); and they can be deployed in any form (including as standalone programs or as modules, components, subroutines, or other units suitable for a computing environment). A program may, but does not necessarily, correspond to a file in a file system. A program may be stored as part of a file containing other programs or data (e.g., one or more scripts stored in a markup language document), as a single file dedicated to the program in question, or as multiple collaborating files (e.g., files storing one or more modules, subroutines, or portions of code). Computer programs can be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected via a data communication network.

[0155] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines may be installed and run on the same one or more computers.

[0156] The processes and logic flows described in this specification can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by a dedicated logic circuit system (such as an FPGA or ASIC) or a combination of a dedicated logic circuit system and one or more programmable computers.

[0157] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The essential components of a computer are the central processing unit for executing or carrying out instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into a special-purpose logic circuit system. Typically, a computer will also include one or more mass storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, or the computer may be operatively coupled to receive data from or transfer data to or both of these mass storage devices. However, a computer does not necessarily need to have such devices. Furthermore, a computer can be embedded in another device, such as, to name just a few, mobile phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, GPS receivers, or portable storage devices (e.g., Universal Serial Bus (USB) flash drives).

[0158] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0159] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having: a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor; and a keyboard and pointing device, such as a mouse or trackball, through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form (including acoustic input, voice input, or tactile input). Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending web pages to a web browser on the user's device in response to a request received from a web browser. Moreover, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone) running a messaging application and receiving response messages from the user as a reply.

[0160] The data processing apparatus used to implement machine learning models may also include, for example, dedicated hardware accelerator units for handling the common compute-intensive parts of machine learning training or production (i.e., inference, workloads).

[0161] Machine learning models can be implemented and deployed using machine learning frameworks, such as the TensorFlow framework.

[0162] Embodiments of the subject matter described in this specification can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., client computers with a graphical user interface, web browser, or application through which users can interact with the implementation of the subject matter described in this specification), or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0163] A computing system may include clients and servers. Clients and servers are typically located remotely to each other and usually interact via a communication network. The client-server relationship is established by means of computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., HTML pages) to a user device, for example, to display data to a user interacting with the device and to receive user input from that user, the device acting as a client. Data generated at the user device (e.g., the result of user interaction) can be received from the device at the server.

[0164] While this specification contains numerous specific implementation details, these details should not be construed as limiting the scope of any invention or the scope of what may be claimed, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in the context of individual embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations, or even as originally claimed, one or more features from a claimed combination may be removed from the combination in some cases, and the claimed combination may involve sub-combinations or variations thereof.

[0165] Similarly, although operations are depicted in a specific order in the accompanying drawings and described in the claims, this should not be construed as requiring such operations to be performed in the specific order shown or in a sequential order, or that all illustrated operations be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0166] Specific embodiments of this subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method performed by one or more data processing apparatus, the method comprising: sampling a latent variable from a set of possible latent variables; selecting, using an action selection neural network conditioned on the sampled latent variable, an action to be performed by an agent to interact with an environment over a sequence of time steps; determining a respective reward received for each time step in the sequence of time steps, including for each of one or more time steps: providing an observation representing a state of the environment at the time step to each discriminator model in an ensemble of a plurality of discriminator models, wherein each discriminator model processes the observation to generate a respective prediction output predicting which latent variable in the set of possible latent variables the action selection neural network was conditioned on to cause the environment to enter the state characterized by the observation; and determining the reward for the time step based at least in part on a measure of inconsistency between the prediction outputs generated by the ensemble of the plurality of discriminator models; training the action selection neural network based on the determined rewards using a reinforcement learning technique, and after training the action selection neural network, using the action selection neural network conditioned on a latent variable sampled from the set of possible latent variables to select actions to be performed by a real-world agent interacting with a real-world environment, wherein, when the action selection neural network is used to select actions performed by the real-world agent: the real-world agent is a mechanical agent interacting with a real-world environment, the action selection neural network conditioned on a latent variable sampled from the set of possible latent variables selects actions based on observations captured while the mechanical agent interacts with the real-world environment, and the selected actions are actions to be taken by the mechanical agent in the real-world environment; the real-world agent is an electronic agent configured to control actions of an electronic and / or mechanical device in a real-world environment, the action selection neural network conditioned on a latent variable sampled from the set of possible latent variables selects actions based on observations comprising electronic signals representing operation data of the electronic and / or mechanical device in the real-world environment, and the selected actions comprise controlling the electronic and / or mechanical device; or the real-world agent is configured to manage distribution of tasks across different computing resources in a real-world environment, and the actions selected by the action selection neural network conditioned on a latent variable sampled from the set of possible latent variables comprise assigning tasks to particular computing resources based on observations comprising electronic signals representing operation of the computing resources in the real-world environment.

2. The method of claim 1, wherein, for each of the one or more time steps, the prediction output generated by each discriminator model for the time step comprises a respective score distribution over the set of possible latent variables, the respective score distribution defining a respective score for each latent variable in the set of possible latent variables.

3. The method of claim 2, further comprising, for each time step of the one or more time steps, determining an inconsistency measure between the predicted outputs generated by the ensemble of the plurality of discriminator models, comprising: determining, for the time step, a combined score distribution over the set of possible latent variables by combining the respective score distributions generated by each discriminator model for the time step; and determining the inconsistency measure based on (i) the combined score distribution for the time step and (ii) the respective score distributions generated by each discriminator model for the time step.

4. The method of claim 3, wherein determining a combined score distribution over the set of possible latent variables for the time step comprises: averaging the score distributions generated by the discriminator models for the time step.

5. The method of claim 3, wherein determining the inconsistency measure based on (i) the combined score distribution for the time step and (ii) the respective score distributions generated by each discriminator model for the time step comprises: determining an entropy of the combined score distribution for the time step; determining respective entropies of the respective score distributions generated by each discriminator model for the time step; and determining the inconsistency measure based on the respective entropies of the combined score distribution for the time step and the respective score distributions generated by each discriminator model for the time step.

6. The method of claim 5, wherein determining the inconsistency measure based on the respective entropies of the combined score distribution for the time step and the respective score distributions generated by each discriminator model for the time step comprises: determining the inconsistency measure as a difference between (i) the entropy of the combined score distribution for the time step and (ii) an average of the entropies of the score distributions generated by each discriminator model for the time step.

7. The method of claim 1, wherein determining, for each time step of the one or more time steps, a reward for the time step comprises: determining respective accuracies of the predicted outputs generated by each discriminator model for the time step; and determining the reward for the time step based at least in part on the accuracies of the predicted outputs generated by the discriminator models for the time step.

8. The method of claim 7, wherein determining the reward for the time step based at least in part on the accuracies of the predicted outputs generated by the discriminator models for the time step comprises: determining the reward based at least in part on an average of the accuracies of the predicted outputs generated by the discriminator models for the time step.

9. The method of claim 1, wherein each discriminator model has a respective set of discriminator model parameters, and wherein respective values of the respective sets of discriminator model parameters are different for each discriminator model.

10. The method of claim 9, wherein each discriminator model is trained on independently sampled batches of training examples from a replay memory.

11. The method of claim 10, wherein each training example in the replay memory comprises: (i) a training observation of the environment state during a previous interaction of the agent with the environment, and (ii) a target output that defines a latent variable for conditioning the action selection neural network to cause the environment to enter the state characterized by the observation.

12. The method of claim 9, wherein, prior to being trained, a respective value of the set of discriminator model parameters of each discriminator model is initialized to be different from the set of discriminator model parameters of each other discriminator model.

13. The method of claim 1, further comprising, for each time step in the sequence of time steps: determining a reward for the time step based at least in part on a task reward that measures progress of the agent in completing a task in the environment.

14. The method of claim 1, wherein the set of possible latent variables includes only a finite number of possible latent variables.

15. The method of claim 1, wherein using the action selection neural network conditioned on the sampled latent variable to select an action to be performed by the agent to interact with the environment over the sequence of time steps comprises, for each time step: processing, using the action selection neural network, an observation characterizing an environment state at the time step and the sampled latent variable to generate an action selection output; and selecting, based on the action selection output, an action to be performed at the time step.

16. The method of claim 15, wherein, for each time step, the action selection output includes a respective score for each action in a set of possible actions.

17. The method of claim 16, wherein, for each time step, selecting, based on the action selection output, an action to be performed at the time step comprises: selecting, from the action selection output, an action having a highest score.

18. A system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the method of any of claims 1-17.

19. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations of the method of any of claims 1-17.

Images