Reinforcement learning system

Abstract

The present disclosure provides methods, systems, and apparatuses, including computer programs encoded on computer storage media, for predicting outcomes related to an environment. In one aspect, a system includes a state representation neural network configured to receive an observation characterizing a state of an environment with which an agent interacts, and to process the observation to generate an internal state representation of the state of the environment; a prediction neural network configured to receive a current internal state representation of a current state of the environment, and to process the current internal state representation to generate a predicted successor state representation of a successor state of the environment and a predicted reward for the successor state; and a value prediction neural network configured to receive the current internal state representation of the current state of the environment, and to process the current internal state representation to generate a value prediction.

Description

[0001] This application is a divisional application of the invention patent application filed on November 4, 2017, with application number 201780078702.3. Background Technology

[0002] This manual relates to predictions using machine learning models.

[0003] Machine learning models receive input and generate outputs, such as predicted outputs, based on the received input. Some machine learning models are parametric models and generate outputs based on the received input and the values ​​of the model's parameters.

[0004] Some machine learning models are deep models, which employ multiple layers to generate outputs for received inputs. For example, deep neural networks are deep machine learning models that include an output layer and one or more hidden layers, where each hidden layer applies a nonlinear transformation to the received input to generate an output. Summary of the Invention

[0005] This specification describes a system implemented as a computer program on one or more computers in one or more locations, the system generating value predictions through a series of internal planning steps to determine an estimate of the aggregate reward from an environment in its initial state.

[0006] According to a first aspect, a system is provided, comprising: a state representation neural network configured to: receive one or more observations representing the state of an environment interacting with an agent, and process the one or more observations to generate an internal state representation of the current environment state; a prediction neural network configured for each of a plurality of internal time steps to: receive an internal state representation for the internal time step; and process the internal state representation for the internal time step to generate: an internal state representation for the next internal time step, and a predicted reward for the next internal time step; and a value prediction neural network configured for each of the plurality of internal time steps to: receive an internal state representation for the internal time step, and... The system comprises: processing the internal state representations for internal time steps to generate a value prediction, which is an estimate of the future cumulative discounted return starting from the next internal time step; and a prediction subsystem configured to: receive one or more observations characterizing the state of the environment; provide the one or more observations as input to a state representation neural network to generate an internal state representation of the current state of the environment; for each of a plurality of internal time steps: using a prediction neural network and a value prediction neural network, and generating from the internal state representations for the internal time steps: an internal state representation for the next internal time step, a predicted return for the next internal time step, and a value prediction; and determining the total return based on the predicted return and the value prediction for the internal time steps.

[0007] In a related aspect, a system implemented by one or more computers is provided, the system comprising: a state representation neural network configured to: receive observations characterizing the state of an environment interacting with an agent, and process the observations to generate an internal state representation of the environment state; a prediction neural network configured to: receive a current internal state representation of the current environment state; and process the current internal state representation to generate: a predicted subsequent state representation of the environment's subsequent states, and a predicted reward for the subsequent states; and a value prediction neural network configured to: receive a current internal state representation of the current environment state, and process the current internal state representation to generate a value prediction, the value prediction being an estimate of a future cumulative discounted reward starting from the current environment state.

[0008] In a preferred embodiment of the relevant aspects, the system includes a prediction subsystem configured to: receive an initial observation characterizing an initial state of the environment; provide the initial observation as input to a state representation neural network to generate an initial internal state representation of the environment state; for each of a plurality of internal time steps: use a prediction neural network and a value prediction neural network and, based on the current state representation, generate a predicted subsequent state representation, a predicted reward, and a value prediction; and determine a total reward based on the predicted reward and value prediction for the time step.

[0009] Therefore, as described herein, the system can integrate the environmental model with the planning model. This is referred to herein as a forecasting system; in some implementations, the forecasting system employs a forecasting subsystem as described above. The forecasting subsystem can also be configured to provide a total return as an estimate of the return derived from the environment in its current state. Internal time steps can be considered as planning steps. Future cumulative discounted returns can include estimates of future returns for multiple future time steps, and thus can be cumulative. Returns can be discounted by assigning weights to returns and distributing them less at later time steps than at earlier time steps.

[0010] In some implementations, the predictive neural network is further configured to generate a predictive discount factor for the next internal time step, and the predictive subsystem is configured to use the predictive discount factor for the internal time step when determining the total return. The return can be discounted by weighting future returns with a product of discount factors, where each discount factor is between 0 and 1, and one discount factor is applied for each consecutive time step. The predictive subsystem can be used to predict the discount factors. As described below, the total return can be determined by an accumulator.

[0011] In some implementations, the system further includes a lambda neural network configured to process the internal state representation for the current internal time step for each of the internal time steps to generate a lambda factor for the next internal time step. The prediction subsystem is configured to determine a return factor for the internal time step when determining the total return, and to use the lambda factor to determine the weights for the return factor. The return factor may include a predicted return for the internal planning time step. This can be determined based on a combination of predicted returns, predicted discount factors, and value predictions; it can be determined for each of k future internal times (i.e., planning steps).

[0012] In some implementations, the state representation neural network is a recurrent neural network.

[0013] In some implementations, the state representation neural network is a feedforward neural network.

[0014] In some implementations, the predictive neural network is a recurrent neural network.

[0015] In some implementations, the predictive neural network is a feedforward neural network, which has different parameter values ​​at each of multiple time steps.

[0016] According to the second aspect, a method is provided, including corresponding operations performed by a prediction subsystem.

[0017] According to a third aspect, a method for training the system is provided, comprising: determining a loss gradient based on the total reward and an estimate of the reward obtained from the environment in the current state; and backpropagating the loss gradient to update the current values ​​of the parameters of a state representation neural network, a prediction neural network, a value prediction neural network, and a λ neural network.

[0018] According to the fourth aspect, a method for training the system is provided, comprising: determining a consistency loss gradient based on the consistency of a return factor determined by a prediction subsystem for an internal time step; and backpropagating the consistency loss gradient to update the current values ​​of the parameters of a state representation neural network, a prediction neural network, a value prediction neural network, and a λ neural network.

[0019] Specific embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. The prediction system described in this specification jointly learns the model of the environment (i.e., the system's state representation neural network and the prediction neural network) and the planning model (i.e., the value prediction neural network, and, in use, a λ neural network), wherein the planning model generates a value function that estimates the cumulative reward. Conventional systems learn the model of the environment and the planning model separately, and therefore, in conventional systems, the model does not perfectly match the planning task. In contrast, for the prediction system described in this specification, the environment model and the planning model are jointly learned, thus enabling the system to generate a value function that helps to estimate the outcome associated with the current state of the environment more accurately than conventional systems.

[0020] Furthermore, unlike conventional systems, the prediction system described in this specification can be trained in part by unsupervised learning methods (i.e., based on observations representing the state of an unknown environment in which the outcome associated with the current state of the environment is unknown). Therefore, due to assisted unsupervised training, the system described in this specification generates a value function that helps to estimate the outcome associated with the current state of the environment more accurately than conventional systems. Moreover, as described in this specification, the prediction system requires less labeled training data than conventional systems because, unlike conventional systems, the prediction system can be trained through assisted unsupervised training.

[0021] Furthermore, the predictive system described in this specification generates output based on a number of planning steps that depend on the adaptive number of the system's internal state representation and internal dynamics. In particular, in some cases, the predictive system can generate output based on fewer planning steps than the total possible number of planning steps, and therefore consumes fewer computational resources (e.g., uses less computational power and time) than conventional systems that generate output by utilizing every planning step in all cases.

[0022] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of this subject matter will become clear from the specification and drawings. Attached Figure Description

[0023] Figure 1 An exemplary prediction system is shown.

[0024] Figure 2 This is a flowchart of an example process for determining the total return output.

[0025] Figure 3 This is a flowchart of an example process for training a prediction system.

[0026] The same reference numerals and names in the various figures denote the same elements. Detailed Implementation

[0027] Figure 1 An exemplary prediction system 100 is illustrated. The prediction system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations, wherein the systems, components and techniques described below are implemented in the computer program.

[0028] System 100 estimates the effect of action 104 performed by agent 102 interacting with environment 106.

[0029] In some implementations, environment 106 is a simulated environment, and agent 102 is implemented as one or more computer programs that interact with the simulated environment. For example, the simulated environment may be a video game, and agent 102 may be a simulated user playing the video game. As another example, the simulated environment may be a motion simulation environment, such as a driving simulator or a flight simulator, and agent 102 may be a simulated vehicle navigating in the motion simulation.

[0030] In some other implementations, environment 106 is a real-world environment, and agent 102 is a mechanical agent that interacts with the real-world environment. For example, agent 102 could be a robot that interacts with the environment to perform a specific task. As another example, agent 102 could be an autonomous or semi-autonomous vehicle that navigates within environment 106.

[0031] System 100 outputs a total return 110 as an estimate of the result 128 associated with the current state of the environment 106 interacting with agent 102. System 100 generates the total return 110 by accumulating the predicted return 116, the predicted discount factor 118, and the sum prediction over multiple internal time steps (referred to as planning steps in this specification).

[0032] Result 128 can encode any event or aspect of the environment 106 that interacts with agent 102. For example, result 128 may include a binary value indicating whether an agent navigating in the environment has arrived at a specific location in the environment from its current state. As another example, result 128 may include a value indicating cumulative rewards received by agent 102 based on the agent 102 completing certain tasks (e.g., arriving at certain locations in environment 106) from its current state in the environment 106.

[0033] Once trained, system 100 can be used, for example, to select action 104 to be performed by agent 102. For example, if result 128 includes a value assessing the success of the interaction between agent 102 and environment 106, such as a value representing the amount of time the agent takes to complete the task from the current state of the environment, then action 104 of agent 102 can be selected as the action predicted by system 100 to optimize the component of result 128 corresponding to that value.

[0034] System 100 includes a predictive neural network 120, which, for each planning step, is configured to process inputs to generate the following as outputs: (i) an internal state representation 114 for the next planning step, i.e., a planning step following the current planning step; (ii) a predicted reward 116 for the next planning step; and (iii) a predicted discount factor 118 for the next planning step. For a first planning step, predictive neural network 120 receives the internal state representation 114 generated by state representation neural network 122 as input, and for subsequent planning steps, predictive neural network 120 receives the internal state representation 114 generated by predictive neural network 120 in a previous planning step as input. The predicted reward 116, predicted discount factor 118, and result 128 can be scalars, vectors, or matrices, and typically all have the same dimension. Typically, entries for the predicted discount factor 118 are values ​​between 0 and 1. The internal state representation 114, predicted reward 116, and predicted discount factor 118 are abstract representations used by the system to facilitate the prediction of the result 128 associated with the current state of environment 106.

[0035] The state representation neural network 122 is configured to receive a sequence of one or more observations 108 of the environment 106 as input, and process the observations according to the values ​​of a set of parameters of the state representation neural network to generate an internal state representation 114 as output for a first planning step. Typically, the dimension of the internal state representation 114 may differ from the dimension of the one or more observations 108 of the environment 106.

[0036] In some implementations, observation 108 may be generated by or derived from the sensors of agent 102. For example, observation 108 may be an image captured by a camera of agent 102. As another example, observation 108 may be derived from data captured by a laser sensor of agent 102. As yet another example, observation 108 may be a hyperspectral image captured by a hyperspectral sensor of agent 102.

[0037] System 100 includes a value prediction neural network 124, which is configured to process an internal state representation 114 for each planning step to generate a value prediction for the next planning step. The value prediction for the planning step is an estimate of the future cumulative discounted returns starting from the next planning step; that is, the value prediction can be an estimate of the following sum rather than a direct calculation:

[0038] v k =r k+1 +γ k+1 r k+2 +γ k+1 γ k+2 r k+3 +…

[0039] Where v k The value of r is predicted in the planning step k. i The predicted return is 116 in step i of the plan, and γ i It is the predictor factor 118 in planning step i.

[0040] The cumulative return 110 is generated by accumulator 112 and is an estimate of the result 128 associated with the current state of environment 106. The total return 110 can be a scalar, vector, or matrix and has the same dimensions as the result 128. In some embodiments, accumulator 112 generates the total return 110 through a process referred to herein as k-step prediction, where k is an integer between 1 and K, and K is the total number of planning steps. In these embodiments, accumulator 112 generates the total return 110 by combining the predicted return 116 and the prediction discount factor 118 used for each of the first k planning steps, along with the value prediction for the k-th planning step, to determine the output referred to herein as k-step return. For k-step prediction, the total return 110 is typically determined as the k-step prediction corresponding to the final planning step K. In some embodiments, accumulator 112 generates the total return 110 through a process referred to herein as λ-weighted prediction. In these embodiments, system 100 includes a λ neural network 126 configured to generate λ factors for each processing internal state representation 114 in the planning step, where the λ factors can be scalars, vectors, or matrices. In some cases, the entries for the λ factors are all values ​​between 0 and 1. In these embodiments, accumulator 112 generates a total return 110 by determining k-step returns for each planning step k and combining them according to weights defined by the λ factors to determine an output referred to herein as λ-weighted returns. Reference Figure 2 Further description is needed to determine the total reward output.

[0041] System 100 is trained by training engine 130 based on a set of training data including observations 108 and corresponding results 128. Specifically, training engine 130 backpropagates based on gradients determined by a loss function (e.g., by stochastic gradient descent) to jointly optimize the set of parameters of the value prediction neural network 124, the state representation neural network 122, the prediction neural network 120, and the λ neural network 126 in the λ-weighted prediction implementation. Training system 100 involves supervised training, and in some cases, assisted unsupervised training.

[0042] In supervised training of system 100, the loss function depends on the result 128 corresponding to the observation 108 provided as input and processed by system 100. For example, in a k-step prediction implementation, the supervised loss function may measure the difference between the result 128 and the k-step return generated by accumulator 112. As another example, in a λ-weighted prediction implementation, the supervised loss function may measure the difference between the result 128 and the λ-weighted return generated by accumulator.

[0043] In the unsupervised training of system 100, the loss function does not depend on the results 128 corresponding to the observations 108 provided as input and processed by system 100. For example, in a λ-weighted prediction implementation, the unsupervised loss function could be a consistency loss function that measures the difference between each k-step return and the λ-weighted return. In this case, unsupervised training jointly adjusts the values ​​of the parameters of the neural network of system 100 to reduce the difference between each k-step return and the λ-weighted return, making the k-step returns self-consistent and thereby increasing the robustness of system 100. (Reference) Figure 3 The training of system 100 by training engine 130 is further described.

[0044] Data structures such as matrices and vectors mentioned in this specification, such as the output of any of the neural networks of System 100, can be represented in any format that allows the data structures to be used in the manner described in the specification (e.g., the output of a neural network described as a matrix can be represented as a vector of matrix entries).

[0045] Figure 2 This is a flowchart of an example process 200 for determining the total return output. For convenience, process 200 will be described as being executed by a system of one or more computers located in one or more locations. For example, a prediction system appropriately programmed according to this specification, such as... Figure 1 The prediction system 100 is capable of executing process 200.

[0046] The system receives one or more observations of the environment in which it interacts with the agent (step 202).

[0047] In some implementations, the environment is a simulated environment, and the agent is implemented as one or more computer programs that interact with the simulated environment. For example, the simulated environment could be a video game, and the agent could be a simulated user playing the video game. As another example, the simulated environment could be a motion simulation environment, such as a driving simulator or a flight simulator, and the agent could be a simulated vehicle navigating in the motion simulation.

[0048] In some other implementations, the environment is a real-world environment, and the agent is a mechanical agent that interacts with the real-world environment. For example, the agent could be a robot that interacts with the environment to perform a specific task. As another example, the agent could be an autonomous or semi-autonomous vehicle that navigates the environment.

[0049] In some implementations, observations can be generated by or derived from the agent's sensors. For example, an observation can be an image captured by the agent's camera. As another example, an observation can be derived from data captured by the agent's laser sensor. As yet another example, an observation can be a hyperspectral image captured by the agent's hyperspectral sensor.

[0050] The state representation neural network receives one or more observations of the environment as input and processes the input according to the values ​​of the set of parameters of the state representation neural network to generate an internal state representation for the first planning step as output (step 204).

[0051] In some implementations, the state representation neural network is a recurrent neural network, and the output of the state representation neural network is the output of the recurrent neural network after each observation has been processed sequentially. In some other implementations, the state representation neural network is a feedforward neural network, and the output of the state representation neural network is the output of the last layer of the feedforward neural network. In implementations where the state representation neural network is a feedforward neural network, the system may concatenate the one or more observations before providing them as input to the state representation neural network 122.

[0052] For each planning step, the predictive neural network processes the input to generate the following as outputs: (i) an internal state representation for the next planning step, (ii) a predicted reward for the next planning step, and (iii) a predicted discount factor for the next planning step (step 206). For the first planning step, the predictive neural network receives the internal state representation generated by the state representation neural network as input, and for subsequent planning steps, the predictive neural network receives the internal state representation generated by the predictive neural network at the previous planning step as input. The predicted reward and the predicted discount factor can be scalars, vectors, or matrices, and typically have the same dimensions as the result. Typically, the entries for the discount factor are values ​​between 0 and 1. The internal state representation for the planning step is an abstract representation of the environment used by the system to facilitate the prediction of the result.

[0053] In some implementations, the predictive neural network is a recurrent neural network. In other implementations, the predictive neural network is a feedforward neural network with different parameter values ​​corresponding to each of the planning steps. In some implementations, the predictive neural network includes sigmoid nonlinear layers so that the values ​​of the discount factor entries are in the range of 0 to 1.

[0054] For each planning step, the value prediction neural network processes the input to generate a value prediction for the next planning step (step 208). For the first planning step, the value prediction neural network receives an internal state representation generated by the state representation neural network as input, and for subsequent planning steps, the value prediction neural network receives an internal state representation generated by the prediction neural network in the previous planning step as input. The value prediction for the planning step is an estimate of the future cumulative discounted return starting from the next internal time step.

[0055] In some implementations, the value prediction neural network shares parameter values ​​with the prediction neural network; that is, the value prediction neural network receives as input the intermediate output of the prediction neural network, which is generated as a result of processing the internal state representation. The intermediate output of the prediction neural network refers to the activation of one or more units in one or more hidden layers of the prediction neural network.

[0056] In the implementation where the accumulator determines the total reward through λ-weighted prediction, the λ neural network processes the input to generate λ factors for the next planning step (step 209). For the first planning step, the λ neural network receives an internal state representation generated by the state representation neural network as input, and for subsequent planning steps, the λ neural network receives an internal state representation generated by the prediction neural network in the previous planning step as input. The λ factors can be scalars, vectors, or matrices, and typically have the same dimensions as the result. In some cases, the values ​​of the λ factor entries are between 0 and 1. In some implementations, the λ neural network includes a sigmoid nonlinear layer so that the values ​​of the λ factor entries are in the range of 0 to 1. In some implementations, the λ neural network shares parameter values ​​with the prediction neural network.

[0057] The system determines whether the current planning step is the final planning step (step 210). In some cases, the current planning step may be the final planning step if it is the last planning step of a predetermined number of planning steps. In the λ-weighted prediction implementation, the current planning step may be the final planning step if the λ factor used for the current planning step is equal to zero (i.e., the λ factor is zero if it is a scalar, or each entry of the λ factor is zero if it is a vector or matrix), as will be further described below. In response to determining that the current planning step is not the final planning step, the system proceeds to the next planning step, returns to step 206, and repeats the previous steps. In response to determining that the current planning step is the final planning step, the accumulator determines the total reward (step 212).

[0058] In some implementations, the accumulator determines the total return through k-step predictions, where k is an integer between 1 and K, and K is the total number of planning steps. In these implementations, the accumulator generates the total return by combining the predicted return and prediction discount factor used for each of the first k planning steps with the value prediction for the k-th planning step, to determine the k-step return as the output. Specifically, the accumulator determines the k-step return as:

[0059] g k = r1+γ1(r2+γ2(…+γ) k-1 (r k +γ k v k )…))

[0060] Where g k It returns after k steps, r i It is the reward for step i in the plan, γ i It is the discount factor for planned step i, and v k This is the predicted value of the planned step k.

[0061] In some other implementations, the accumulator determines the total return through λ-weighted prediction. In these implementations, the accumulator determines k-step returns for each planning step k and combines them according to weights defined by the λ factor to determine the λ-weighted return as output. Specifically, the accumulator can determine the λ-weighted return as:

[0062] in

[0063] Where g λ It is a λ-weighted return, λ k w is the λ factor used for the k-th planning step. k It is a weighting factor, 1 is the identity matrix, that is, a matrix with one on the diagonal and zero elsewhere, and g k It returns after k steps. The accumulator can also return via intermediate steps g. k,λ The λ-weighted return is determined by backward accumulation, where:

[0064] g k,λ =(1-λ) k )v k +λ k (r k+1 +γ k+1 g k+1,λ And g K,λ =v K

[0065] And λ-weighted return g λ Determined as g 0,λ .

[0066] The system can compute the λ-weighted return g based on a sequence of consecutive planning steps that does not include all K planning steps. λ For example, in the previously provided g λ In the example formula, if for planning step k, λ k If g = 0, then g is determined based on the k-step return from the first k planning steps rather than the subsequent planning steps. λ Because for n>k, the weight w n The total reward is zero. Therefore, the system determines the total reward based on the number of planned steps that depend on the number of adaptive steps that depend on the internal state representation and the system's learning dynamics.

[0067] Figure 3This is a flowchart of an example process 300 for training a prediction system. For convenience, process 300 will be described as being executed by an engine comprising one or more computers located in one or more locations. For example, a training engine appropriately programmed according to this specification (e.g., Figure 1 The training engine (130) can execute process 300.

[0068] The engine receives one or more observations of the environment in which it interacts with the agent, and in some cases, receives corresponding results associated with the current state of the environment (step 302).

[0069] The engine provides observations to the system, and the system determines the total reward as an estimate of the outcome. (Reference) Figure 2 Describe an example process used to determine the total return.

[0070] The engine determines the gradient based on the loss function and backpropagates the gradient to jointly update the set of parameters of the system's neural networks (i.e., value prediction neural networks, state representation neural networks, prediction neural networks, and λ neural networks in the λ-weighted prediction implementation). The loss function can be a supervised loss function, i.e., a loss function that depends on the outcome corresponding to the observations provided as input and processed by the system; it can be an unsupervised loss function, i.e., a loss function that does not depend on the outcome; or it can be a combination of supervised and unsupervised loss terms.

[0071] In the k-step prediction implementation, the supervised loss function can be given by the following formula:

[0072]

[0073] Where g is the result. As another example, in a λ-weighted prediction implementation, the supervised loss function used to backpropagate the gradient to the λ neural network can be given by the following equation:

[0074]

[0075] The supervised loss function used to backpropagate gradients into value prediction neural networks, state representation neural networks, and prediction neural networks can be given by the following equation:

[0076]

[0077] Alternatively, it can be given by the following formula:

[0078]

[0079] In the λ-weighted prediction implementation, the unsupervised loss function can be given by the following equation:

[0080]

[0081] Where g λ It is assumed to be fixed, and the gradient is backpropagated to return g for each k steps. k More like g λ Conversely, the same applies. Backpropagation gradients based on the unsupervised loss function reduce the difference between the k-step return and the λ-weighted return, making the k-step return self-consistent and thus increasing the robustness of the system. Furthermore, since the unsupervised loss function does not depend on the outcome corresponding to the observations provided as input and processed by the system, the engine can train the system by backpropagating gradients based on the unsupervised loss function for the sequence of observations in which the corresponding outcome is unknown.

[0082] Given training observations with known results, the engine can update the set of parameters of the system's neural network based on a loss function that combines both supervised and unsupervised loss terms. For example, the loss function could be a weighted linear combination of supervised and unsupervised loss terms.

[0083] This specification uses the term "configured to" in relation to system and computer program components. For a system of one or more computers to be configured to perform a specific operation or action, this means a system on which software, firmware, hardware, or a combination thereof is installed, which, when in operation, causes the system to perform said operation or action. For one or more computer programs to be configured to perform a specific operation or action, this means that one or more programs include instructions that, when executed by a data processing device, cause that device to perform said operation or action.

[0084] Embodiments of the subject matter and functional operation described in this specification can be implemented in digital electronic circuits, in tangibly implemented computer software or firmware, in computer hardware including the structures disclosed in this specification and their equivalents, or in a combination of one or more of these. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium, wherein the computer program instructions are for execution by a data processing apparatus or for controlling the operation of the data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these. Alternatively or additionally, the program instructions can be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus.

[0085] The term "data processing device" refers to data processing hardware and includes all types of means, apparatus, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The device may also be or further include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may also include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0086] A computer program (also referred to or described as a program, software, software application, application, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative languages, or programming languages; it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program can be stored as a portion of a file that holds other programs or data, for example, as one or more scripts in a markup language document, as a single file dedicated to the program in question, or as multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions). A computer program can be deployed to execute on a single computer, or on multiple computers located at a single site or distributed across multiple sites and interconnected via a data communication network.

[0087] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines may be installed and run on the same computer or on multiple computers.

[0088] The processes and logic flows described in this specification can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by special-purpose logic circuitry (e.g., FPGA or ASIC) or a combination of special-purpose logic circuitry and one or more programmable computers.

[0089] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into special-purpose logic circuitry. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, or transfer data to one or more mass storage devices, or both. However, a computer does not need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0090] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0091] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from a web browser. Furthermore, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and, in turn, receiving response messages from the user.

[0092] The data processing apparatus for implementing machine learning models may also include, for example, a dedicated hardware accelerator unit for processing the common and computationally intensive parts of machine learning training or production, namely inference and workloads.

[0093] It is possible to implement and deploy machine learning models using machine learning frameworks, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0094] Embodiments of the subject matter described in this specification can be implemented in a computing system including back-end components (e.g., as a data server), or in a computing system including middleware components (e.g., an application server), or in a computer system including front-end components (e.g., a client computer with a graphical user interface, web browser, or application through which a user interacts with embodiments of the subject matter described in this specification), or in any combination of one or more such back-end components, middleware components, or front-end components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0095] A computing system can include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is established by computer programs running on separate computers and having a client-server relationship with each other. In some embodiments, the server sends data (e.g., HTML pages) to a user device, for example, for the purpose of displaying data to a user interacting with the device acting as a client and receiving user input from that user. It is possible to receive data generated at the user device, such as the result of user interaction, from the user device at the server.

[0096] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases it is possible to remove one or more features from the claimed combination, and the claimed combination may be for sub-combinations or variations thereof.

[0097] Similarly, although the operations are depicted in a specific order in the accompanying drawings and recited in a specific order in the claims, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or to perform all the shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0098] Specific embodiments of this subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in different orders and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method performed by one or more data processing devices for estimating outcomes associated with an environment interacting with an agent to perform a task by aggregating rewards and value predictions over a sequence of planning steps, the method comprising: Receive one or more observations that represent the state of the environment interacting with the agent; The state representation neural network is used to process the one or more observations to generate an internal state representation of the first planning step of the sequence of planning steps; For each planning step in the sequence of planning steps, the internal state representation of the planning step is processed using a predictive neural network to generate: (i) the internal state representation of the next planning step, and (ii) the predicted reward of the next planning step; For each of one or more planning steps in a sequence of planning steps, a value prediction neural network is used to process the internal state representation of the planning step to generate a value prediction, which is an estimate of the future cumulative discounted return received after the planning step; and Estimates of environmentally relevant outcomes are determined based on projected returns and value projections for planned steps. Among them, the results associated with the environment characterize the efficiency of the agent in performing the task, and Each observation, which represents the state of the environment interacting with the agent, includes a corresponding image of the environment.

2. The method according to claim 1, wherein, The agent is a mechanical agent that interacts with the real-world environment.

3. The method according to claim 1, wherein, For each planning step in the sequence of planning steps, the predictive neural network also generates a predictive discount factor for the next planning step, and wherein determining the estimate of the outcome associated with the environment further includes: In addition to the predicted returns and value predictions based on the planning steps, estimates of environmentally relevant outcomes are also determined based on the predicted discount factors of the planning steps.

4. The method according to claim 3, wherein, Determining the estimates of environmentally relevant outcomes involves combining (i) the projected returns and projected discount factors for each planning step, and (ii) the value forecast for the final planning step.

5. The method according to claim 4, wherein, The estimates of environmentally related outcomes satisfy the following: in, It is an estimate of the outcome. It is the number of planning steps in the sequence of planning steps. It is the planning step in the sequence of planning steps. i The predicted returns It is the planning step in the sequence of planning steps. i The predicted discount factor, and This is the predicted value for the final planned step.

6. The method of claim 3, further comprising, for each planning step in the sequence of planning steps, processing the internal state representation of the planning step using a λ neural network to generate a λ factor for the next planning step, wherein, Determining estimates of environmentally relevant outcomes includes: In addition to the forecast discount factor, forecast return, and value forecast based on the planning steps, the estimate of the outcome is also determined based on the λ factor of the planning steps.

7. The method according to claim 6, wherein, The estimates of environmentally related outcomes satisfy the following: in, It is an estimate of the outcome. It is the index of the planning step in the sequence of planning steps. It is the index of the last planning step in the sequence of planning steps. Is with the planned steps The associated weighting factors, which are determined based on the λ factor of the planning steps, Is with the planned steps Related k Step back, the The step-by-step return is based on the predicted return, value prediction, and prediction discount factor determination.

8. The method according to claim 7, wherein, For each , and planned steps Related Step back satisfy: in, It is the planning step in the sequence of planning steps. The predicted returns It is the planning step in the sequence of planning steps. The predicted discount factor, and It is the planning step in the sequence of planning steps. Value prediction, Among them, the 0-step return The value prediction is equal to that of the first planning step in the sequence of planning steps.

9. The method according to claim 7, wherein, For each , and planned steps Associated weighting factors satisfy: in, It is the λ factor of the planned step j.

10. The method according to claim 1, wherein, The state representation neural network includes a feedforward neural network.

11. The method according to claim 1, wherein, The predictive neural network includes a recurrent neural network.

12. The method according to claim 1, wherein, The predictive neural network includes a feedforward neural network, which has different parameter values ​​at each planning step.

13. A system implemented by one or more computers, comprising: One or more computers; and One or more storage devices, communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform an operation by aggregating rewards and value predictions over a sequence of planning steps to estimate an outcome associated with an environment interacting with an agent to perform a task, the operation including: Receive one or more observations that represent the state of the environment interacting with the agent; The state representation neural network is used to process the one or more observations to generate an internal state representation of the first planning step of the sequence of planning steps; For each planning step in the sequence of planning steps, the internal state representation of the planning step is processed using a predictive neural network to generate: (i) the internal state representation of the next planning step, and (ii) the predicted reward of the next planning step; For each of one or more planning steps in a sequence of planning steps, a value prediction neural network is used to process the internal state representation of the planning step to generate a value prediction, which is an estimate of the future cumulative discounted return received after the planning step; and Estimates of environmentally relevant outcomes are determined based on projected returns and value projections for planned steps. Among them, the results associated with the environment characterize the efficiency of the agent in performing the task, and Each observation, which represents the state of the environment interacting with the agent, includes a corresponding image of the environment.

14. The system according to claim 13, wherein, The agent is a mechanical agent that interacts with the real-world environment.

15. The system of claim 13, wherein for each planning step in the sequence of planning steps, the predictive neural network further generates a predictive discount factor for the next planning step, and wherein, The estimation of environmentally relevant outcomes also includes: In addition to the predicted returns and value predictions based on the planning steps, estimates of environmentally relevant outcomes are also determined based on the predicted discount factors of the planning steps.

16. A non-transitory computer storage medium storing one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform operations that estimate the results associated with an environment interacting with an agent to perform a task by aggregating rewards and value predictions over a sequence of planning steps, the operations including: Receive one or more observations that represent the state of the environment interacting with the agent; The state representation neural network is used to process the one or more observations to generate an internal state representation of the first planning step of the sequence of planning steps; For each planning step in the sequence of planning steps, the internal state representation of the planning step is processed using a predictive neural network to generate: (i) the internal state representation of the next planning step, and (ii) the predicted reward of the next planning step; For each of one or more planning steps in a sequence of planning steps, a value prediction neural network is used to process the internal state representation of the planning step to generate a value prediction, which is an estimate of the future cumulative discounted return received after the planning step; and Estimates of environmentally relevant outcomes are determined based on projected returns and value projections for planned steps. Among them, the results associated with the environment characterize the efficiency of the agent in performing the task, and Each observation, which represents the state of the environment interacting with the agent, includes a corresponding image of the environment.

Images