A reinforcement learning system comprising a relational network for generating data encoding relations between entities in an environment

By introducing attention blocks and adaptive networks into the reinforcement learning system, extracting and processing entity data in a multi-entity environment, the problem of difficulty in learning relationships in an existing system is solved, and the task execution of the agent is improved successfully and generalization ability, achieving more efficient training and better performance.

CN112272831BActive Publication Date: 2024-11-08GDM HOLDINGS LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980032397.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2018-05-18
Filing Date
2019-05-20
Publication Date
2024-11-08
Estimated Expiration
2039-05-20

AI Technical Summary

Technical Problem

When existing reinforcement learning systems deal with multi-entity environments, it is difficult to effectively learn the relationship between entities, resulting in insufficient success rate and generalization ability of agents in task execution.

Method used

Using a neural network system containing attention blocks, extract entity data through transformed networks and relational networks, generate modified entity data to select actions, use adaptive networks and convolutional layers to perform non-local calculations, and train the output layer to generate strategies or actions. Relevant independent variable values, identify entities and their relationships directly from the state data.

Benefits of technology

It improves the success rate and generalization of the task execution of the agent in a multi-entity environment, reduces training time and storage requirements, realizes direct learning and action selection of entity relationships, and enhances performance efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112272831B_ABST
    Figure CN112272831B_ABST
Patent Text Reader

Abstract

A neural network system for reinforcement learning is proposed, comprising an input network for extracting corresponding entity data for each of a plurality of entities present or at least potentially present in an environment from state data. The entity data describes the entity. The neural network comprises a relational network for parsing the data, the relational network comprising one or more attention blocks that can be stacked to perform consecutive actions on the entity data. Each attention block comprises a corresponding transformation network for each of the entities. The transformation network of each entity is capable of transforming data received by the transformation network for the entity into modified entity data for the entity based on data of a plurality of other entities. The output network is arranged to receive data output by the relational network and to use the received data to select a corresponding action (e.g., an action from a predefined space of possible actions).
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present specification relates to reinforcement learning. In particular, the present specification relates to a neural network system for selecting actions for an agent to perform in an environment that may contain multiple entities, and to methods performed by the neural network system. Background Art

[0002] In a reinforcement learning system, an agent interacts with an environment by performing actions selected by the reinforcement learning system in response to receiving observations characterizing the current state of the environment.

[0003] Some reinforcement learning systems select actions for the agent to perform in response to receiving a given observation, based on the output of a neural network.

[0004] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict outputs for received inputs. Some neural networks are deep neural networks that include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output from the received input according to the current values ​​of the corresponding set of parameters. Summary of the invention

[0005] This specification generally describes a reinforcement learning system that selects actions to be performed by a reinforcement learning agent that interacts with an environment. In order for the agent to interact with the environment, the system receives data representing the current state of the environment ("state data") and selects actions to be performed by the agent in response to the received data. In this specification, the data representing the state of the environment will be referred to as observations.

[0006] One innovative aspect of the subject matter described in this specification can be represented as a neural network system for reinforcement learning, comprising an input network for extracting corresponding entity data for each of a plurality of entities present or at least potentially present in an environment from state data. The entity data describes the entity. The neural network includes a relational network for parsing this data, the relational network including one or more "attention blocks" that can be stacked to perform continuous actions on the entity data. Each attention block includes a corresponding transformation network for each of the entities. The transformation network for each entity is capable of transforming the data received by the transformation network for the entity into modified entity data for the entity based on data for a plurality of entities (i.e., not only the entity data of the entity corresponding to the transformation network, but also for a plurality of entities, or even all entities). The output network is arranged to receive data output by the relational network and use the received data to select a corresponding action (e.g., an action from a predetermined space of possible actions).

[0007] Therefore, using entity data corresponding to the plurality of entities, the modified entity data generated by the transformation network for the first entity of the entities can represent information in the state data related to the interaction between the first entity and the plurality of entities. This causes an architectural bias when training the neural network system, which causes the relationship network between the entities to learn relationships.

[0008] Each transformation network may include one or more "head sections", and an adaptive network for using the output of the head(s) to generate modified entity data. Each attention block may convert each entity data set into at least one value vector. Optionally, there may be a corresponding value vector for each head (i.e., if the number of heads for each transformation network is h, then h value vectors may be generated). The head of each transformation network may weight the corresponding value vector by a corresponding attention weight and sum them.

[0009] If each transformation network has h heads, then h value networks of the attention block can be used to produce the value vector. Each value network can, for example, apply a linear projection (linear transformation) to any given set of entity data it receives. For a given value network, the linear transformation can be the same regardless of which entity the entity data is associated with, but the value network produces a different corresponding linear transformation.

[0010] The head can generate a corresponding attention weight for each set of entity data by generating corresponding saliency values ​​and then combining the saliency values ​​in a non-linear manner. In this way, it can increase the highest saliency value relative to the other saliency values. The saliency value generated by the head corresponding to a given first entity is a measure of the importance of information in the state data related to both the first and second entities, using the entity data of a given second entity. The attention weight emphasizes the highest of these measures, and therefore emphasizes the corresponding value vector in the output of the head.

[0011] One function that can be used to combine the saliency values ​​is the soft-max function. This has the advantage of being a differentiable function, which can be helpful when training a neural network system. However, after training, it will be possible to replace the differentiable function with another function (e.g., a function that sets all saliency values, but the highest to zero).

[0012] For each entity, the attention block can generate h query vectors by inputting the corresponding entity data into h corresponding query networks. Similarly, for each entity, the attention block can generate h key vectors by inputting the corresponding entity data into h corresponding key networks. To generate saliency values ​​for the corresponding multiple entities, the head for a given entity can multiply the query vector for the given entity with the corresponding key vector. The result can be normalized by a normalization factor that is a function of the number of components in the query vector and the key vector (which are usually the same).

[0013] Each query network and each value network can apply a corresponding linear transformation to the entity data it receives. The linear transformation performed by each query network is the same regardless of which entity the entity data is associated with. Similarly, the linear transformation performed by each key network is the same regardless of which entity the entity data is associated with.

[0014] As described above, the transformation network for a given entity operates on the entity data of multiple entities. The multiple entities may include the entity itself. This is called "self attention" and can be useful, for example, for generating actions that only involve a single entity among the entities.

[0015] As described above, each transformation network may include an adaptive network, such as a multilayer perceptron, for transforming the output of the head to generate output data for the attention block (i.e., modified entity data corresponding to the entity of the transformation network). The output of the head may be concatenated and optionally added to the entity data of the corresponding entity received by the transformation network.

[0016] Certain embodiments of the present invention are particularly useful for tasks in environments where state data is defined by an array of pixels. The state data includes one or more values ​​for each pixel. In this case, the input network of the neural network system may include at least one convolutional layer. In the case of multiple convolutional layers, they may be stacked so that data is passed continuously between them in a certain order of layers.

[0017] The entities may correspond to different respective subsets of the environment (possibly overlapping subsets), which may be spatially displaced relative to each other. For example, one of the convolutional layers (the last in the order) may be arranged to generate at least one feature value for each cell in a cell array. The cell array typically has the same number of dimensions as the pixel array. The cells may correspond to entities. Thus, for each entity, the corresponding entity data may be at least one feature value for the corresponding cell. Note that this means that the head of each entity receives entity data for cells located at different respective positions within the cell array (e.g., all cells of the array). Thus, the head may perform "non-local" computations in which it considers data that depends on state data for substantially the entire pixel array. For many tasks, it is believed that agents controlled based on non-local computations will be better suited for computational relationships between entities than agents that employ only more explicit local computations (such as shift-invariant convolutions). The entity data for a given one of the cells may include data indicating the position of the corresponding cell in the cell array.

[0018] In one form, the convolutional layer(s) (and specifically the last convolutional layer if there are multiple convolutional layers) generates multiple feature values ​​for each unit. Optionally, corresponding feature values ​​for different units in the unit can form corresponding feature value sets (e.g., a first feature value set includes a first feature value from a first unit in the unit, and other feature values ​​of the first feature value set are corresponding first feature values ​​from other units), which can be sent continuously to the relational network. The corresponding outputs of the relational network for the feature value sets can optionally be combined in an output layer (e.g., using a feature-wise max pooling layer).

[0019] For certain tasks, and in particular tasks in which the reward depends on the state of the system many time steps after an action is taken, the neural network system may benefit from including at least one recurrent layer, so that the action selected by the neural network system at any time step depends not only on the most recently received state data, but also on state data from earlier time steps. In one form, the recurrent layer may be a layer of long short-term memory (LSTM) cells ("LSTM layer"); for example, it may be a convolutional LSTM layer.

[0020] Although the output layer can take many forms, it can include a rectified linear unit (reLU). The output layer can be trained to select a single action. Alternatively, it can be configured to generate a policy that defines a distribution of corresponding probability values ​​for each action in the space of possible actions. Training may involve a policy gradient method. Additionally, the output layer can be trained to generate a baseline value. The baseline value is used to modify the cost function used in the policy gradient method during training of the policy network.

[0021] Additionally, the output layer can be trained to generate one or more action-related independent variable values ​​that can be sent to the agent to modify how the agent performs an action selected by the policy network. For example, the action-related independent variable values ​​can include at least one corresponding value for each of a plurality of locations (e.g., regions) within the environment, and in this case, can be used to influence how an action selected based on the policy is performed in that region.

[0022] Note that entities do not have to correspond to corresponding parts of the environment. For example, instead, a neural network system can be used to perform tasks related to responding to commands or questions (e.g., verbal commands / questions) about the environment (e.g., related to objects that may or may not be present in the environment and that may constitute entities). Tasks can be specified using feature planes that encode the commands / questions. If the environment is a home location, the action can include controlling a home robot.

[0023] The neural network system can be implemented in a distributed architecture such as the Imapla architecture.

[0024] Another innovative feature of the present invention involves generating a neural network system by training each of an input layer, a relational network, and an output layer using training data (e.g., using a policy gradient method). The training process can simultaneously train the input layer, the relational network, and the output layer based on a reward generated from state data and indicating the success of the agent in performing a task. Specifically, the training of the relational network involves training a value network, a query network, and a key network, and it has been found that in this process, salient features of the relationships between entities are naturally emphasized.

[0025] The innovative aspects of the present specification may be represented in the following aspects: a neural network system, or a method performed by a neural network system to select an action based on state data, or a method for training a neural network system. Alternatively, the method is represented in terms of a computer system configured to perform one of the methods, or as a computer storage medium storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of one of the methods.

[0026] The subject matter described in this specification can be implemented in specific embodiments to achieve one or more of the following advantages. First, neural networks can control agents to perform tasks with greater success than certain known RL systems. This is particularly true for environments where the state data is pixel data (e.g., intensity data at individual pixels), and / or for tasks that require interaction between multiple entities at correspondingly spaced locations in the environment. From another perspective, this improvement in performance efficiency has the potential to reduce training time and / or memory requirements compared to known systems that perform the same task with the same accuracy.

[0027] Furthermore, during the training of the neural network system, the relational model learns to identify entities directly from the state data, and in particular the relations between entities. This means that there is no need to define the task of the agent to operate on the object by specifying an entity-based language and requiring the neural network system to interpret commands in the language to control the agent. Instead, the neural network system learns the relevant objects and their relations directly from the input space. The relational model is operable to generate data that characterizes the state data in a manner informed by these relations so that the output network can generate actions based on them. We have demonstrated in experiments that this leads to the following neural network system that is operable to perform certain tasks with very high success compared to other techniques, and in some cases, compared to human-controlled agents. Specifically, neural networks, when trained to perform tasks in an environment that includes certain objects, exhibit high generalization capabilities, so that in use they can successfully perform other tasks involving similar objects, including more complex tasks and tasks that include sub-goals that were not used during training.

[0028] In this specification, a system of one or more computers to be configured to perform a specific operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that causes the system to perform the operation or action in operation. A system of one or more computer programs to be configured to perform a specific operation or action means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0029] The details of one or more non-limiting examples of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 An example reinforcement learning system for training an agent to interact with an environment is shown.

[0031] Figure 2 A neural network system according to the present specification is shown.

[0032] Figure 3 shows a relationship network, which is Figure 2 components of the system.

[0033] Figure 4 shows the attention block, which is Figure 3 Components of a relationship network.

[0034] Figure 5 shows a transformation network, which is Figure 4 Components of the attention block.

[0035] Figure 6 The steps of the method according to the present description are shown. DETAILED DESCRIPTION

[0036] The implementations described herein involve a reinforcement learning system.

[0037] Broadly speaking, a reinforcement learning system is a system that selects actions to be performed by a reinforcement learning agent that interacts with an environment. In order for the agent to interact with the environment, the system receives data representing the current state of the environment and selects actions to be performed by the agent in response to the received data. The data representing the state of the environment is referred to as observations in this specification. Optionally, an observation at a time step may include data from a previous time step, for example, an action performed at a previous time step, a reward received at a previous time step, and so on.

[0038] In some implementations, the environment is a real-world environment, and the agent is an electromechanical agent that interacts with the real-world environment. For example, the agent can be a robot or other static or mobile machine that interacts with the environment to complete a specific task (e.g., locating an object of interest in the environment or moving an object of interest to a specified location in the environment or navigating to a specified destination in the environment); or the agent can be an autonomous or semi-autonomous vehicle (e.g., a land, air, or sea vehicle) that navigates in the environment.

[0039] In these implementations, observations may include, for example, one or more of images, object position data, and sensor data to capture observations when the agent interacts with the environment, such as sensor data from images, distance or position sensors, or from actuators. In the case of a robot or other mechanical agent or vehicle, similarly, observations may include one or more of the position, linear or angular velocity, force, torque or acceleration, and global or relative posture of one or more parts of the agent. Observations may be defined in 1, 2, or 3 dimensions and may be absolute and / or relative observations. For example, in the case of a robot, observations may include data characterizing the current state of the robot, such as one or more of the following: joint position, joint velocity, joint force, torque or acceleration, and global or relative posture of a part of the robot (such as an arm and / or an item held by the robot). Observations may also include, for example, sensed electronic signals, such as motor current or temperature signals; and / or, for example, image or video data from a camera or LIDAR sensor, for example, data from a sensor of the agent or data from a sensor located separately from the agent in the environment.

[0040] In these implementations, the action may be a control input for controlling a robot, such as a torque or higher-level control command for a joint of the robot; or controlling an autonomous or semi-autonomous land or air or sea vehicle, such as a torque or higher-level control command for a control surface or other control element of the vehicle; or, for example, motor control data. In other words, the action may include, for example, position, velocity, or force / torque / acceleration data of one or more joints of a robot or a component of another mechanical intelligent body. The action data may include data for these actions and / or electronic control data, such as motor control data, or more generally, data for controlling one or more electronic devices within the environment, the control of which has an effect on the state of the observed environment. For example, in the case of an autonomous or semi-autonomous land or air or sea vehicle, the action may include actions to control navigation, such as steering and movement, such as braking and / or acceleration of the vehicle.

[0041] In some implementations, the environment is a simulated environment and the agents are implemented as one or more computers that interact with the simulated environment.

[0042] For example, the simulation environment can be a simulation of a real-world environment, such as a simulation of a real-world environment including a robot or a vehicle, and the reinforcement learning system can be trained on the simulation. For example, the simulation environment can be a motion simulation environment, such as a driving simulation or a flight simulation, and the agent is a simulated vehicle navigating in the motion simulation. In these implementations, the action can be a control input that controls a simulated user or a simulated vehicle. The simulation environment may be useful for training the reinforcement learning system before using the system in the real world. In another example, the simulation environment can be a video game, and the agent can be a simulated user playing the video game. Typically, in the case of a simulated environment, observations can include simulated versions of one or more previously described observations or types of observations, and actions can include simulated versions of one or more previously described actions or types of actions.

[0043] In another example, the environment can be a protein folding environment, such that each state is a corresponding state of a protein chain, and the agent is a computer system for determining how to fold the protein chain. In this example, the actions are possible folding actions for folding the protein chain, and the results to be achieved can include, for example, folding the protein so that the protein is stable and so that it achieves a specific biological function. As another example, the agent can be a mechanical agent that automatically performs or controls the protein folding actions selected by the system without human intervention. Observations can include direct or indirect observations of the state of the protein and / or can be derived from simulations.

[0044] In a similar manner, the environment can be a drug design environment, such that each state is a corresponding state of a potential pharmaceutical chemical drug, and the agent is a computer system for determining elements of the pharmaceutical chemical drug and / or a synthetic pathway for the pharmaceutical chemical drug. For example, in a simulation, the drug / synthesis can be designed based on a reward derived from the drug's goal. As another example, the agent can be a mechanical agent that performs or controls the synthesis of the drug.

[0045] In some applications, observations can include data from one or more sensors that monitor a part of a plant or service facility, such as current, voltage, power, temperature and other sensors and / or electronic and / or mechanical signals representing the functioning of an electronic and / or mechanical item of equipment. In some applications, an agent can control actions in a real-world environment including items of equipment, for example, in a facility (such as: a data center, a server farm or a main power supply or water distribution system of a power grid), or in a manufacturing plant or service facility. Then, observations can be related to the operation of a plant or facility. For example, as an additional or alternative to those previously described, they can include observations of the use of electricity or water of equipment, or observations of power generation or distribution control, or observations of the use of resources or waste generation. An agent can, for example, control actions in an environment to improve efficiency by reducing resource use, and / or, for example, reduce the environmental impact of operations in an environment by reducing waste. For example, an agent can control the use of electricity or other power consumption or water in a facility, and / or the temperature of a facility and / or an item in a facility. Actions may include actions to control or impose operating conditions on items of equipment of the plant / facility, and / or actions that result in changes in operating settings of the plant / facility, for example to adjust or turn on / off components of the plant / facility.

[0046] In some other applications, the environment is a real-world environment, and the agent manages the allocation of tasks across computing resources, such as on mobile devices and / or in a data center. In these implementations, the action may include allocating the task to a specific computing resource. As a further example, the action may include presenting an advertisement, the observation may include an ad impression or a click count or click rate, and the reward may characterize a prior selection of an item or content by one or more users.

[0047] The reinforcement learning system can be implemented as one or more computer programs on one or more computers in one or more locations where the systems, components, and techniques described herein are implemented.

[0048] Figure 1 An example of a reinforcement learning system is shown. The reinforcement learning system 10 generates output data that controls the agent 100. The generation of the output data can be based on a so-called "policy". Each time an action is determined, the agent 100 performs the action on the environment 101. The action updates the state of the environment 101. The updated state is returned to the reinforcement learning system 10 along with the associated reward for the action. The reinforcement learning system 10 uses these to determine the next action. Typically, the reward is a numerical value. The reward can be based on any event or aspect of the environment 101. For example, the reward can indicate whether the agent 100 has completed a task (e.g., navigated to a target location in the environment 30) or the progress of the agent 100 in completing a task.

[0049] The interaction of the agent 100 with the environment 101 over one or more time steps may be represented by a "trajectory" (i.e., sequence) of experience tuples, where each experience tuple corresponds to a corresponding time step. The experience tuple corresponding to a time step may include: (i) an observation characterizing the state of the environment at the time step, (ii) a selection of actions to be performed by the agent at the time step, (iii) a subsequent observation characterizing the subsequent state of the environment after the agent performs the selected action, (iv) a reward received by the agent after performing the selected action, and (v) a selection of subsequent actions to be performed at subsequent time steps.

[0050] The policy defines how the system performs actions based on the state of the environment. When training the system 10 based on a set of experiences, the policy used to generate output data to generate actions to be followed by the agent 100 is updated by evaluating the value of the action according to an approximate value function or a reward function to improve the expected return from the actions taken by the policy. This is usually achieved by evaluating the success (sometimes called "reward") of the actions performed by the agent through a combination of prediction and control. The reward is calculated based on the reward received after a given action. For example, the reward may be the accumulation of multiple reward values ​​over multiple time steps.

[0051] refer to Figure 2 , shows a reinforcement learning system according to the present specification. The reinforcement learning system can be Figure 1 It is a neural network system 2. The system includes an input network 21, which receives state data and generates a set of entity data describing one or more corresponding entities in the environment. The neural network system 2 also includes a relationship network 22, which modifies the entity data to generate modified entity data. The neural network system 2 also includes an output network 23 for generating output data of the neural network system 2 to be used as control data for controlling an intelligent agent acting on the environment.

[0052] For simplicity, the following case may be considered: where the environment is represented by a two-dimensional grid of points, and the input state data is therefore a two-dimensional array of data. In one form, the input network 21 may include at least one convolutional layer with a stride of 1, such as a 2×2 convolutional layer. In addition, it may include a rectified linear unit (reLU) that receives the output of the convolutional layer.

[0053] In one case, the input network 21 may parse the pixel input into k feature maps of size nxn, where k is the number of output channels of the convolutional layer, and may concatenate x and y coordinates to each k-dimensional pixel feature vector to indicate the location of the pixel in the map. The neural network system 2 is operable to generate n 2 The pixel feature vector is compiled into n 2xk matrix (denoted by matrix E) to view it as a collection of entities. This provides an efficient and flexible way to learn representations of related entities in a way that is agnostic to the entities that may constitute the specific problem at hand. The number of entities (n 2 ) is denoted as N below.

[0054] The output network 23 can take a variety of forms. For example, it can include a fully connected layer followed by a rectified linear unit (reLU). The output layer can be trained to select a single action. Alternatively, it can be configured to generate a policy that defines a distribution of corresponding probability values ​​for each action in the space of possible actions. The output network 23 can be trained using conventional methods. Training can involve a policy gradient method.

[0055] Go to Figure 3 , shows the structure of the relation network 22. It includes one or more attention blocks 31, 32, 33. As described below, a single attention block performs non-local pairwise relation computations. Optionally, multiple attention blocks with shared (circular) or non-shared (deep) parameters may be provided. Although the attention blocks are shown in parallel, alternatively one or more attention blocks may be stacked (i.e., arranged in such a sequence that the output of any block (except the last block) is the input of another in the block) so that information flows along the sequence. In this case, multiple blocks can more easily approximate high-order relations between entities. Our experiments are mainly performed where the relation network contains only a single attention block.

[0056] Go to Figure 4 , showing Figure 3 It includes a projector unit 40 that receives entity data. This can be from Figure 1 Entity data of the input network 21. Alternatively, in the case of an attention block that is part of a stack of attention blocks rather than the first attention block in the stack, the received entity data may be modified entity data generated earlier by another attention block in the stack.

[0057] The number of entities in the environment is represented as N, and the received entity data can be represented as e 1 ,e 2 ,….e N The projector unit 40 transmits the entity data e of each entity (indicated by i) to i Projected to query, key and value representations q respectively i , k i and v i This is done using the query network, key network, and value network respectively. i , k i and v iThe activities of are normalized to have zero mean and unit variance.

[0058] In fact, as described below, the projector unit 40 may include a plurality ("h", where h is an integer greater than one) of query networks, key networks, and value networks, each generating a corresponding query, key, and value representation for each entity. Each of the query networks may be associated with a corresponding one of the value networks and a corresponding one of the key networks as a network set. Thus, there are h network sets. The first network set generates a value The second network set produces the value And so on.

[0059] A plurality of transformation networks 411, 412, ... 41N are provided. Each is used to generate modified entity data for a corresponding one of the entities using data related to other entities. For example, the transformation network 411 generates modified entity data with respect to the first entity For simplicity, we will consider here the case where each transformation network uses data related to all other entities, but in a variant of the example, this is not the case: each transformation network can only use entity data from a corresponding subset of entities. The transformation networks 411, 412, ... 41N collectively generate modified entity data for all entities.

[0060] refer to Figure 5 , shows the structure of the i-th transform block. It includes a number of h (usually greater than one) headers 511, 512, ... 51h. Each generates a corresponding vector by the following process

[0061] First, each q is transformed into i With the key k of all entities 1:N The results are the corresponding unstandardized significance s i , where the vector s i The set representing the saliency [s i,j=1 ,s i,j=2 ,…,s i,j=N ] Normalize the significance to weight w i =softmax(s i ), where w i represents the set of N weights of the i-th entity [w i,j=1 ,w i,j=2 ,…,w i,j = N]. For each entity, the cumulative interaction is calculated by the value vector a of all entities i =∑ j=1:N w i,j v jThis is the output of the head. It can be computed compactly using the following matrix multiplication:

[0062]

[0063] where matrices A, Q, K, and V compile the accumulated interactions, queries, keys, and values ​​into matrices, and d is the dimension of the key vector used as a scaling factor.

[0064] vector are cascaded together and passed to the adaptive network 520. This can be implemented as having i Multilayer Perceptron of the same layer size (2-layer MLP with ReLU nonlinearity). The results can be compared with i The sum is performed (i.e., there is a residual connection) and transformed via layer normalization to produce the output of the i-th transform block.

[0065] Go to Figure 6 , describes Figure 2 Operation of a neural network system. In step 61, the input network 21 receives state data. In step 62, the input network 21 extracts corresponding entity data for each of the entities. In step 63, the transformation network of the relationship network generates corresponding modified entity data for each entity based on the entity data corresponding to the plurality of entities. If the relationship network 22 includes only a single attention block, all transformation networks may operate in parallel. Note that, alternatively, in the presence of a plurality of attention blocks arranged in series, step 61 may include a plurality of sub-steps, i.e., successively for each of the attention blocks. In step 64, the output network uses the modified entity data output by the relationship network to generate output data for controlling the agent to act on the environment.

[0066] In the process of adaptively training each of the h query networks, value networks, and key networks of the relationship network 22, the neural network system is trained. At the same time, the input network 21 and the output network 23 can be trained based on the training data, which can be the above-mentioned Figure 1 The training process can be performed by directly modifying known reinforcement algorithms.

[0067] The embodiments of the subject matter and functional operations described in this specification can be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware including the structure disclosed in this specification and its structural equivalents, or in a combination of one or more of them. The embodiments of the subject matter described in this specification can be implemented as one or more computer programs, that is, one or more modules of computer program instructions encoded on a tangible non-temporary program carrier to be run or control its operation by a data processing device. Alternatively or additionally, program instructions can be encoded on artificially generated propagation signals, such as machine-generated electrical, optical or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device to be run by a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. However, a computer storage medium is not a propagated signal.

[0068] The term "data processing apparatus" covers all kinds of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer, or a plurality of processors or computers. The apparatus may include special-purpose logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may also include code that creates an operating environment for the computer program in question, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0069] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script, or code) may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to run on one computer or on multiple computers located at one site or distributed over multiple sites and interconnected by a communications network.

[0070] As used in this specification, an "engine" or "software engine" refers to a software-implemented input / output system that provides outputs that are different from inputs. An engine can be a coded block of functionality, such as a library, platform, software development kit ("SDK"), or object. Each engine can be implemented on any suitable type of computing device, such as a server, mobile phone, tablet computer, notebook computer, music player, e-book reader, portable or desktop computer, PDA, smart phone, or other fixed or portable device, including one or more processors and computer-readable media. In addition, two or more engines can be implemented on the same computing device or on different computing devices.

[0071] The processes and logic flows described in this specification may be performed by one or more programmable computers running one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuits, and the apparatus may also be implemented as special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits). For example, the processes and logic flows may be performed by a graphics processing unit (GPU) and the apparatus may also be implemented as a graphics processing unit (GPU).

[0072] The computer that is suitable for running a computer program includes a general or special microprocessor or both, or a central processing unit of any other type, for example, can be based on a general or special microprocessor or both, or a central processing unit of any other type. Usually, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic element of a computer is a central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. Usually, a computer will also include or be operably coupled to receive data from one or more large-capacity storage devices (for example, disks, magneto-optical disks or optical disks) for storing data or to transmit data thereto. However, a computer does not necessarily have such a device. In addition, a computer can be embedded in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver or a portable storage device, to name a few, such as a universal serial bus (USB) flash drive.

[0073] Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0074] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0075] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer having a graphical user interface or a web browser by which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), such as the Internet.

[0076] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on the respective computers and having a client-server relationship to each other.

[0077] Although this specification contains many specific implementation details, these should not be interpreted as limitations on any invention or the scope of protection that can be claimed, but rather as descriptions of features that can be specific to specific embodiments of specific inventions. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments or in any suitable sub-combination, respectively. Moreover, although features may be described above as functioning in certain combinations and even initially claimed as such, one or more features in the claimed combination may be cut out of the combination in some cases, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.

[0078] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the specific order shown or in a continuous order, or that all of the operations shown be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0079] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented neural network system for reinforcement learning, wherein: The neural network system is used to select actions to be performed by an agent interacting with an environment to perform a task in an attempt to achieve a specified result, and the system includes: an input for receiving state data representing an environment, wherein the state data comprises image data defined by an array of pixels and comprising one or more values ​​for each pixel; an input network for extracting, from the image data, for each of a plurality of entities potentially present in the environment, corresponding entity data indicating the presence of the entity in the environment, the input network comprising at least one convolutional layer arranged to generate, from the state data, at least one feature value for each cell in an array of cells, the cells corresponding to entities of different corresponding subsets of the environment, the input network being arranged to generate the corresponding entity data for each entity as a data set, the data set comprising at least one feature value for the cell corresponding to the entity and data indicating the position of the cell in the array of cells; a relationship network comprising at least one attention block, each attention block comprising a respective transformation network for each of the entities, each transformation network being arranged to generate respective modified entity data for the corresponding entity, the modified entity data being based on entity data for the corresponding plurality of entities; and The output network is arranged to receive data output by the relational network and use the received data to select a corresponding action.

2. The neural network system according to claim 1, wherein: Each of the transformation networks comprises one or more heads, and an adaptation network for generating modified entity data from outputs of the heads.

3. The neural network system according to claim 2, wherein: Denoting the number of heads in each transformation network as h, each attention block is operable to generate h value vectors for each entity using entity data of the corresponding entity, and each head is operable to form the sum of the value vectors of the corresponding multiple entities weighted by the corresponding attention weights.

4. The neural network system according to claim 3, wherein: The attention block includes h value networks, each value network is used to generate a value vector from entity data.

5. The neural network system according to claim 4, wherein: Each value network produces a value vector by applying a linear transformation to the entity data.

6. The neural network system according to any one of claims 3 to 5, wherein: For each entity, each respective head is arranged to generate an attention weight by generating a respective saliency value for each of the corresponding plurality of entities and combining the saliency values ​​using a non-linear function to form an attention weight.

7. The neural network system according to claim 6, wherein: The non-linear function is a soft-max function.

8. The neural network system according to claim 7, wherein: Denoting the number of heads in each transformation network as h, each attention block includes h query networks for generating a query vector for each entity from the corresponding entity data, and h key networks for generating a key vector for each entity from the corresponding entity data. Each head is arranged to generate a saliency value for each of the plurality of entities using a query vector for the corresponding entity as a dot product of the query vector and the corresponding key vector.

9. The neural network system according to any one of claims 3 to 5, wherein: Each transformation network is arranged to concatenate weighted value vectors, and uses the concatenated weighted value vectors to generate modified entity data.

10. The neural network system according to claim 9, wherein: Each transformation network is arranged to add the concatenated weight value vector to the entity data of the corresponding entity to form a summed vector, and to send the summed vector to the adaptation network.

11. The neural network system according to any one of claims 2 to 5, wherein: The adaptive network includes a multi-layer perceptron.

12. The neural network system according to any one of claims 1 to 5, wherein: For each entity, the corresponding plurality of entities includes the entity itself.

13. The neural network system according to claim 1, wherein: The convolutional layer generates multiple feature values ​​for each unit, and the input network is arranged to send corresponding feature values ​​for different ones of the units together to the relational network, and the output network is operable to combine corresponding outputs of the relational network for different sets of corresponding feature values.

14. The neural network system according to claim 13, wherein: The output network includes a maximum pooling layer, which is used to combine corresponding outputs of the relational model for different sets of corresponding feature values.

15. The neural network system according to any one of claims 1 to 5, comprising at least one recursive layer.

16. The neural network system according to claim 15, wherein: The recurrent layer is an LSTM layer.

17. The neural network system according to claim 16, wherein: The LSTM layer is a convolutional LSTM layer.

18. A neural network system according to any one of claims 1 to 5, arranged to generate actions to be performed by an agent interacting with an environment comprising a two-dimensional space, and the state data comprises at least one value for each of a two-dimensional array of pixels within the two-dimensional space.

19. The neural network system according to any one of claims 1 to 5, wherein: The output network includes a rectified linear unit.

20. The neural network system according to any one of claims 1 to 5, wherein: The output network is configured to generate a baseline value.

21. The neural network system according to any one of claims 1 to 5, wherein: The output network is configured to generate a policy that defines a distribution of corresponding probability values ​​for each action in a space of possible actions, and to randomly select an action using the policy.

22. The neural network system according to claim 21, wherein: The output network is arranged to generate one or more action-related independent variables, so that the agent can perform the selected action based on the action-related independent variables.

23. The neural network system according to claim 22, wherein: The action-related independent variables include a respective value for each of a plurality of locations in an array having the same number of dimensions as the environment.

24. A method for selecting actions to be performed by an agent interacting with an environment to perform a task in an attempt to achieve a specified result, the method comprising: receiving state data characterizing an environment, wherein the state data comprises image data defined by an array of pixels and comprising one or more values ​​for each pixel; extracting, from the image data, corresponding entity data indicating the presence of the entity in the environment for each of a plurality of entities potentially present in the environment, generating, from the state data, at least one feature value for each cell in a cell array, the cells corresponding to entities of different corresponding subsets of the environment, generating, for each entity, corresponding entity data as a data set, the data set comprising at least one feature value for the cell corresponding to the entity and data indicating the position of the cell in the cell array; for each entity, generating modified entity data for the entity based on the entity data for the corresponding plurality of entities using a respective transformation network for each entity, the modified entity data being based on the entity data for the corresponding plurality of entities; and Selects an action based on the modified entity data.

25. A reinforcement learning system comprising one or more computers and one or more storage devices, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the method of claim 24.

26. One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the method of claim 24.