Proxy speech in to facilitate task learning
Through the combination of action selection neural network and image subtitle neural network, the ability of the agent to perform new tasks without direct experience is realized, the problems of high training resources and difficulty in promotion in the existing technology are solved, and the efficiency and resources of the system are improved. use.
Patent Information
- Application Number
- CN202380031694.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-05-19
- Filing Date
- 2023-05-19
- Publication Date
- 2024-11-15
AI Technical Summary
The prior art is difficult to allow agents to learn and perform tasks involving new objects without direct experience, and the process of training image subtitles is time-consuming and expensive.
By using the action selection neural network system to process multimodal demonstration data, generate action selection strategies and language strategy output, combine the image subtitle neural network system to generate natural language subtitles, achieve zero-excitation task execution, and train the system to reduce the required data and resources.
It realizes that the agent performs new tasks without additional training, reduces the memory and computing resources required for training, and improves the system's promotion capabilities and data utilization efficiency.
Smart Images

Figure CN118974738A_ABST
Abstract
Description
Background Art
[0001] This specification generally relates to controlling agents using neural networks.
[0002] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict outputs for received inputs. Some neural networks include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output from the received input based on the current values of the corresponding set of parameters. Summary of the invention
[0003] This specification describes systems and methods implemented as computer programs on one or more computers in one or more locations for learning to control an agent (e.g., an embedded agent) to perform tasks. The described techniques use language, more specifically internal intra-agent speech, when learning, and are therefore able to perform tasks involving new objects without any direct experience interacting with those objects, i.e., zero-shot.
[0004] In one aspect, a computer-implemented method of training an action-selection neural network system to control an agent to select actions to perform a task in an environment is described.
[0005] The method involves acquiring a multimodal demonstration data sequence of demonstration observations and demonstration actions. The demonstration observations include image observations representing the state of the environment when the demonstration agent performs a task in the environment, and one or more natural language observations.
[0006] The action selection neural network system is configured to process the embedding of the image of the environment and the embedding of the natural language input to generate an action selection policy output for selecting an action to be performed by the agent, and a language policy output for generating a natural language output.
[0007] The method involves processing an embedding of a demonstration observation using an action selection neural network system to generate an action selection policy output and a language policy output for the demonstration observation. The action selection neural network system is trained using a natural language output defined by the language policy output such that actions defined by the action selection policy output from the action selection neural network system are encouraged to match actions of the demonstration agent.
[0008] The image observations from the demonstration observations are processed using the image captioning neural network system to generate natural language captions for the image observations. The training of the action selection neural network system involves training the action selection neural network system using the natural language captions.
[0009] The subject matter described in this specification can be implemented in specific embodiments to realize one or more of the following advantages.
[0010] Embodiments of the described systems and methods simulate humans' use of "inner speech," i.e., they generate speech that describes the environment as a task progresses. This can help the agent generalize as it learns to perform a task. As a result, the system can perform new tasks without any additional training.
[0011] For example, after having been trained to move or manipulate some objects, the system can move or manipulate new objects without any prior training involving tasks related to that object, e.g., the new object was not included in any task-related instructions during training. New objects are identified using natural language. The system knows about the new objects because the image captioning neural network system describes the environment and identifies the objects. This natural language information is used when the system learns to perform tasks involving other objects so that it can also perform tasks involving new objects.
[0012] Techniques are also described for learning to generate captions for images using less data than some other methods. These techniques can train an image captioning neural network system by relying on a dataset of images, where only a small fraction of the images have been captioned, for example, by humans. Typically, generating image captions for training an image captioning neural network system is time consuming and expensive. The described techniques can estimate the missing information, thereby efficiently using the available information.
[0013] A general implementation of the described techniques promotes efficient learning by using language to help understand the world, and thus can reduce the memory and computing resources required to train systems used to control agents.
[0014] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 An example of an action selection system is shown.
[0016] Figure 2 A specific example implementation of an action selection neural network system is shown.
[0017] Figure 3 is a flow chart of an example process for training an action selection system.
[0018] Figure 4 Shows Figure 3 An example implementation of the training process of .
[0019] Figure 5 A system for training an image captioning neural network system is shown.
[0020] Figure 6 An example process for training an image captioning neural network system is shown.
[0021] Figure 7 Shows Figure 5 operation of the system.
[0022] Figure 8 Shows Figure 1 Performance of example implementations of systems.
[0023] Fig. 9 An example of an agent system configured to control an agent is shown.
[0024] Fig.10 is a flow chart of an example process for controlling an agent.
[0025] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION
[0026] Figure 1 An example of an action selection system 100 that can be implemented as one or more computer programs on one or more computers in one or more locations is shown. The action selection system 100 is trained using demonstration data, and can then be used to control an agent 102 (e.g., an embedded agent) that interacts with an environment 104 to perform a task based on observations 106 of the environment. The observations include images, and the system uses a captioning neural network system to obtain natural language descriptions of the images. During training, as the system interacts with some objects in the environment, it uses natural language captions describing other objects it sees to help the training generalize. For example, some embodiments of the system can use other "new" objects to perform a task without being trained by interacting with these objects, i.e., the system can exhibit "zero-initiate task acquisition."
[0027] In some embodiments, the action selection neural network system 100 is assisted in generalization by training the action selection neural network system using an auxiliary loss based on natural language subtitles. For example, subtitles can be used as an auxiliary input to the system during training, and / or can be used as a training target. The auxiliary loss can be a subtitle loss or a subtitle matching loss as described below.
[0028] Typically, the action selection system 100 receives language observations (ie, language input) as well as image observations, and in some embodiments also generates language outputs. This allows the system to answer questions, engage in conversations, request clarification, etc.
[0029] Reference again Figure 1 , the action selection system 100 has at least one observation input 108 to receive observations, the observations including image observations (i.e., still or moving image observations) and natural language observations. Typically, image observations characterize the state of the environment; after training, image observations may include observations of the environment 104. As used herein, images may be obtained from a camera or other image sensor, for example, "images" include LIDAR point clouds. The natural language observations input to the system typically relate to the environment, more specifically to tasks performed in the environment, and may include, for example, a description of the environment, or a description of the tasks to be performed by the system in the environment, or dialogue related to these. Typically, observations are obtained for each of a series of time steps; typically, each time step will include an image observation, but there need not be a natural language observation at each time step. Typically, at least one of the natural language observations describes the task being performed. In some embodiments, some natural language observations may be repeated at multiple time steps, for example, at each time step until a new natural language observation is received, i.e., some natural language observations may be "sticky". Observations may include a representation of the history of previous observations.
[0030] During training, image observations and natural language observations may be obtained from a local or remote demonstration data storage device 160. During further training or after training, observations may be obtained from the environment 104. In some embodiments, the system may be trained in part or in whole using data from a simulation of a real-world environment and then used in the real-world environment. Typically, after training, image observations are derived from images of the environment 104. After training, natural language observations may be derived from local or remote text or verbal input, for example, via a local or remote user interface 150.
[0031] The observations are processed by the action selection system 100 to generate an embedding of the observation received at the observation input 108, which includes: an image of the environment (o X )'s embedding (o EX ) and embeddings of natural language input (o EL ). In embodiments, the embeddings of observations, in particular the embeddings of the environment image and the embeddings of the natural language input, are processed by the action selection neural network system 110 according to trainable parameters (such as weights) of the action selection neural network system 110 to generate an action selection policy output 112 for selecting an action 114, and a language policy output 116 for generating a natural language output 118. In some embodiments, the action selection neural network system 110 also generates a combined representation of the natural language input and the image, for example at an intermediate output 119.
[0032] As used herein, an "embedding" of an observation may refer to an observation represented as an ordered set of numerical values, e.g., a vector or matrix of numerical values. An embedding of an observation may be generated, e.g., as an output of a neural network processing data representing an entity, or by multiplying an embedding matrix. The embedding neural network may have any suitable architecture, and may include, for example, one or more feedforward neural network layers, one or more recurrent neural network layers, one or more convolutional neural network layers, one or more attention neural network layers, or one or more normalization layers.
[0033] By way of example only, an embedding of an image of an environment may be generated by processing the image using a residual neural network (ResNet). An embedding of a natural language input may be generated by an embedding matrix, for example, by indexing a predetermined or learnable table of vocabulary of natural language tokens (e.g., words) and their embeddings. In some embodiments, but not essentially, the embedding of the image and the embedding of the natural language input have the same dimensions.
[0034] Typically, observations are obtained for each time step in a series of time steps, and an action selection policy output 112 is generated for each time step in the series of time steps. The action selection policy output 112 is used to select an action 114. The action 114 can be continuous or discrete; it can include a collection of multiple individual or primitive actions to be performed at a time step, such as a mixture of continuous and discrete actions. During training, the selected actions can be compared, for example, with actions stored in a demonstration data storage device 160. During or after training, the selected actions can be performed by the agent 102. The execution of the selected action by the agent 102 typically causes the environment 104 to transition to a new state. By repeatedly causing the agent 102 to act in the environment 104, the action selection system 100 can control the agent 102 to complete a specified task.
[0035] There are many ways in which the action selection policy output 112 can be used to select an action. For example, the action selection policy output 112 can directly define the action, for example, it can include values for a continuous value that defines the action, such as torque or speed. As another example, it can define (e.g., parameterize) a continuous or categorical distribution from which values defining the action can be selected; or it can define a distribution as a set of scores, one score for each action in a set of possible actions. Such a distribution can be used to select an action, for example by sampling from the distribution or by selecting the action with the highest probability. In an embodiment, the action selected by the action selection policy output 112 can be a non-action, that is, defining that the agent does not act at a particular time step, or predicting that there is no action at a time step (during training).
[0036] In general, the neural network outputs described herein may define a distribution, for example as a set of scores or by parameters that define the distribution. References to a neural network output that defines a particular output (such as a particular action or language token) may be interpreted as a neural network output that speculatively defines the particular output, for example by sampling from a distribution; or the particular output may be defined, for example, by the mean or maximum of a distribution.
[0037] The language strategy output 116 may be generated for each of a series of time steps, or at less frequent intervals. The language strategy output may generate language actions in a manner substantially similar to the manner in which the action selection strategy output 112 generates movement actions. A language action may be an action that defines one or more language tokens to be emitted at a time step. For example, it may define a distribution of possible language tokens over a vocabulary and then select a language token from the possible language tokens. A language token may define a word or a portion of a word, or multiple words, such as a sentence or a portion of a sentence. A language token may include a token representing a punctuation mark. In some embodiments, the natural language output is generated one word at a time.
[0038] The techniques described herein are not limited to any particular implementation of the action selection neural network system 110 (although some implementations require the system to generate the previously described combined representations). In general, the action selection neural network system 110 may include, for example, one or more feed-forward neural network layers, one or more recurrent neural network layers, one or more convolutional neural network layers, one or more attention neural network layers, or one or more normalization layers.
[0039] The action selection system 100 includes an image captioning neural network system 120. This is used during training and is not needed after training (although it can be retained if further training is required). The image captioning neural network system 120 is configured to process still or moving image observations ( X ), in particular based on trainable parameters (such as weights) of the image captioning neural network system to generate natural language captions 122 ("captions") for the image observations. The image observations may be part of a demonstration observation retrieved from demonstration data storage 160; it may be derived from one or more frames of a video image of a simulated or real environment.
[0040] Typically, natural language captions describe image observations. XThe natural language captions may describe features of multiple objects in the image observation, such as one or more of the object name, shape, configuration, outline, color, texture, material, etc. The natural language captions may also describe the relative arrangement of two or more objects in the image observation. Typically, the specific content of the captions will depend on the data used to train the image captioning neural network system, for example, on the textual content of the human annotations of the training images.
[0041] In some embodiments, the action selection system 100 also includes a classifier neural network 130. This is used during training and is not needed after training, although it can be retained, for example, in the event that further training is required. The classifier neural network 130 has trainable parameters (such as weights) and is configured (trained) to process a combined representation of a natural language input and an image obtained from the action selection neural network system 110 (e.g., from the intermediate output 119) according to the trainable parameters to generate a classifier output 132.
[0042] More specifically, the classifier neural network 130 is configured to process a combined representation generated by the action selection neural network system 110 by processing the embedding of the image and the embedding of the natural language caption from the image caption neural network system 120. The image can be the image for which the caption was generated, or a different image. The classifier output 132 is configured to predict whether the natural language caption and the image match, i.e., whether the image matches the image for which the caption was generated, e.g., whether the caption and the image were obtained from the same image observation. For example, the classifier output 132 may include a score that predicts whether the natural language caption and the image observation match based on the value of the score. The classifier neural network 130 may also be considered a discriminator neural network.
[0043] As previously mentioned, the action selection system 100 is provided with a demonstration data storage device 160 as a source of multimodal demonstration data for training the system. In an embodiment, the multimodal demonstration data includes multiple task demonstration sequences. Each of these may include a sequence or "track" of demonstration observations and demonstration actions. The demonstration data is multimodal because the demonstration observations include both image observations and language observations.
[0044] Demonstration observations can include image observations that characterize the state of the environment as the demonstration agent interacts with the environment to perform a task. The environment can be the environment in which the agent will act after training, or it can be another similar environment, or it can be a simulation or representation of the environment. Image observations of a simulated environment can characterize the state of the simulated environment. As an example, an agent can be trained using a simulation of a real-world environment (e.g., a type of real-world environment), and then the agent can be used in the real-world environment. In some embodiments, each image observation includes a video frame from a video of the environment.
[0045] The demonstration observation may also include at least one natural language observation related to the performed task (e.g., describing the performed task). For example, such a natural language observation may exist at an initial time step or shortly after the initial time step. In an embodiment, the demonstration action characterizes the action of the demonstration agent in the environment when performing the task. Demonstration observations and actions may be provided for each of a plurality of time steps. The demonstration agent may be, for example, a person or another machine learning system.
[0046] The multimodal demonstration data may be obtained in any convenient manner. For example, in some embodiments, it may be generated by capturing multiple examples of people performing a particular task after having been given natural language instructions describing the task. As a specific example, the multimodal demonstration data may be obtained from a 3D simulated environment in which two embedded avatars act and interact with natural language to collaboratively complete a task, such as one involving object manipulation, navigation, and question answering. The avatars (i.e., agents) that may be controlled by humans receive image observations and produce motor actions, and also receive language input and produce language output. This allows them to answer questions, engage in conversations, ask for clarification, etc., which helps the action selection neural network system 110 learn to perform the same operations.
[0047] Examples of generating data in this way are described in arXiv:2112.03762, Abramson et al., “Creating Multimodal Interactive Agents with Imitation and Self-Supervised Learning,” 2022, which can be referenced for details; additional material can be found in arXiv:2012.05672v2, Interactive Agents Group, 2020. The method of Abramson et al. involves simulated environments, but can also be used in the real world. The simulated environment can be a simulation or representation of a real-world environment, and the captured demonstration data can be used to train an action selection neural network system so that it can then act in the virtually simulated or represented real-world environment. Typically, the techniques described herein are improved with larger training data sets and more model parameters, for example, with more than 10 6 Up to 10 8 parameters and / or for example using a set of more than 10 human participants 3 Up to 10 4 hours of training data.
[0048] The action selection system 100 also includes a training engine 140 that controls the training of the system, as described below. Again, this is not required after training, although it can be retained in the event that further training is required. Typically, training a neural network system as described herein (such as the action selection system 100) includes back-propagating the gradient of an objective function (e.g., a loss function) to update the learnable parameters (e.g., weights) of the neural network system. This can use any suitable gradient descent optimization algorithm, for example, the Adam algorithm or another optimization algorithm.
[0049] Once trained, the action selection neural network system can be used to perform the requested task. This can include receiving a natural language description of the task and providing an embedding of the natural language description of the task to the action selection neural network system 110 as one or more natural language observations (optionally as "sticky" observations). The action selection neural network system 110 is used to process the embedding of the natural language description of the task and the embedding of the observation 106 of the environment 104, e.g., at successive time steps, to control the agent 102, e.g., to select an action at a time step to perform the requested task in the environment. This can also involve the action selection neural network system 110 generating language at the natural language output 118, e.g., for engaging in a conversation with a user about the task.
[0050] As previously described, some embodiments of the system include a local or remote user interface 150. Typically, the user interface 150 is configured to accept natural language input, for example, as typed or spoken text, and can be used to provide natural language observations to the action selection system 100 after training. For example, the user interface 150 can be used to request the performance of a task, for example, by obtaining a natural language description of the task. The user interface 150 can also provide natural language output. Additionally or alternatively, the natural language description of the task to be performed can be retrieved from a memory or obtained via a communication interface (e.g., via a wired or wireless network connection). Natural language output can be provided in a similar manner.
[0051] In some embodiments, but not necessarily, the action selection neural network system 110 includes a transformer neural network. In general, the transformer neural network can be a neural network having a series of self-attention neural network layers. The self-attention neural network layer has an attention layer input for each element of the input, and is configured to apply an attention mechanism on the attention layer input to generate an attention layer output for each element of the input; there are many possible attention mechanisms that can be used. As an example, a query-key-value (QKV) attention operation can be applied, where the input embedding is used to determine a set of query vector and key-value vector pairs, and the updated embedding includes a weighted sum of values weighted by a similarity function of the query for each corresponding key. In embodiments where one or more residual neural network blocks are used, or where an embedding of an image is generated, for example from blocks of the image, the embedding can be flattened to provide input to the transformer neural network.
[0052] The transformer neural network can be coupled to a memory such as a recurrent neural network, such as a neural network having one or more LSTM (long short-term memory) neural network layers. The transformer neural network can generate an embedded combined representation of a demonstration observation that can be provided to such a recurrent neural network. The intermediate output 119 can be an internal representation of the action selection neural network system, such as an output from the transformer neural network. One or more outputs from the recurrent neural network can then be processed to generate an action selection policy output and a language policy output, such as using an action selection policy head and a language policy head.
[0053] Figure 2 Shown suitable for use as Figure 1 A specific example implementation of an action selection neural network system 110. Figure 2In a specific example of , the multimodal transformer neural network 210 processes the image and language embeddings to generate an output that is provided as input to the LSTM memory 220, whose output regulates the action selection strategy head 230 and the language strategy head 240. The language embeddings may include one or more aggregate ("CLS") embeddings that include an aggregate representation of the input sentence. In some embodiments, the image embeddings provide only keys and values to the transformer neural network 210, while the language embeddings provide queries, keys, and values. The output of the multimodal transformer neural network 210 can be used as a combined representation of the natural language captions and image observations of the intermediate output 119.
[0054] Figure 2 The example shown implements hierarchical control where new observations arrive every n internal time steps, and for n internal time steps a second LSTM in the action selection strategy head 230 is unrolled, selecting an action at each of these time steps. Language actions are emitted every n internal time steps, and language actions are generated using a second transformer neural network in the language strategy head 240, e.g., sampling one token at a time using the same vocabulary as the natural language input, up to a maximum number of tokens. Figure 2 In a variation of the implementation of , non-hierarchical control is used.
[0055] Figure 3 is used to train action selection systems (e.g. Figure 1 Flowchart of an example process of the action selection system 100). Figure 3 The process can be implemented by one or more computers in one or more locations. Typically, for each of the multiple training steps, Figure 3 process. Figure 3 The steps need not be performed in the order shown.
[0056] At step 300, the process obtains, for example from demonstration data storage 160: demonstration observations including image observations and natural language observations; and demonstration actions.
[0057] The action selection neural network system 110 processes the embeddings of the image observation input and the embeddings of the natural language observation to generate an action selection policy output 112 and a language policy output 116 for the demonstration observation (step 302).
[0058] Typically, the process trains the action selection neural network system 110 so that actions selected using the action selection policy output 112 are encouraged to match actions of the demonstration agent (step 304). This can be done using imitation learning techniques. For example, training can include one or more of: behavioral cloning, inverse reinforcement learning, and generative adversarial imitation learning (GAIL, arXiv:1606.03476, Ho et al.). In embodiments, training the system so that actions selected using the action selection policy output 112 are encouraged to match actions of the demonstration agent can simply involve encouraging the action selection policy of the action selection neural network system 110 to select environmental states that are similar to environmental states defined by demonstration observations.
[0059] Training can be performed offline, i.e., based solely on demonstration data, or can also be performed online, for example by fine-tuning actions selected using reinforcement learning using the agent 102 to act in the environment 104 and in response receive rewards (e.g., from a reward model based on human feedback) for training the system. The action selection neural network system 110 can be trained to optimize an objective function that depends on the difference between the distribution of actions defined by the action selection policy output 112 and the distribution of actions defined by the actions of the demonstration agent.
[0060] Typically, training also uses language from natural language output 118. As an example, the action selection neural network system 110 can be trained so that the natural language output defined by the language policy output 116 for the demonstration observation is encouraged to match the natural language observation in the demonstration observation (which can include dialogue, such as about the task being performed). This can help to associate the visual and language representations in the action selection neural network system 110. Additionally or alternatively, the action selection neural network system 110 can be trained so that the natural language output defined by the language policy output 116 for the demonstration observation is encouraged to match the natural language caption generated for the image observation of the demonstration observation.
[0061] As a specific example, the action selection neural network system 110 can be trained to minimize the behavior cloning loss L BC The behavior cloning loss L may be evaluated for a batch of B trajectories of demonstration observations sampled from the demonstration data store 160 BC , and wherein the action selection neural network system 110 comprises a recurrent neural network unrolled over K-steps, i.e., with backpropagation through time of K steps. For example, the behavior cloning loss may be determined as:
[0062]
[0063] Here π m(a|o) refers to the action (move) selection policy defined by the action selection policy output 112, which defines the probability a of each move action given observation o m ; and ≤t represents the demonstration observation o leading to and including time t t , (in embodiments including recurrent neural networks, the action at a time step depends on the current observation and the previous observation). BC The second term in depends on the language action definition a selected by the language strategy output 116 l Defined language policy π l (a|o), and also includes L in the case where the system is trained using behavioral cloning based on language observations BC The second item in .
[0064] In general, the process involves training the action selection neural network system 110 using an auxiliary loss based on natural language captions from the image caption neural network system 120 (step 306). The natural language observations in the demonstration observations may be relatively sparse, and the captions may provide an additional language source for training the system. In some embodiments, the image caption neural network system 120 has already been trained. In some other embodiments, the caption neural network system 120 may be trained in parallel with the action selection neural network system 110. Any image caption neural network system may be used; specific examples of image caption neural network systems 120 that may be used are described later.
[0065] The auxiliary loss may include a loss that depends on the difference between the natural language output 118 from the action selection neural network system 110 and the captions generated by the image captioning neural network system 120 ("captioning loss"). Additionally or alternatively, when the action selection neural network system 110 processes the embedding of captions, the auxiliary loss may include a loss that depends on the intermediate output 119 from the action selection neural network system 110 (particularly on the classifier output 132) ("captioning matching loss"). The captioning loss is used to train the system to "speak" what it sees; the captioning matching loss is used to train the system's language encoding through language input.
[0066] In some embodiments, the process generates natural language captions for image observations in the demonstration observations using the image captioning neural network system 120 (step 308). The demonstration observations may, but need not be, the demonstration observations processed at step 302. In embodiments of the process, captions may be generated for more image observations than have corresponding natural language observations, thereby increasing the amount of language training data available to the system.
[0067] The process determines a caption loss that depends on (a measure of) the difference between the natural language captions generated for the image observations and the natural language output from the action selection neural network system 110 for the demonstration observations (step 310). That is, the action selection neural network system 110 processes the image observations of the demonstration observations to generate a language policy output 116, which in turn defines a language output 118 for determining the caption loss. The action selection neural network system 110 is trained using the caption loss (step 312).
[0068] As a specific example, the subtitle loss L C It can be determined as:
[0069]
[0070] in is the bth trajectory o observed at time t and for the demonstration t In an embodiment, the action selection neural network system 110 is processed to generate a natural language caption defining lnπ l (·|o b,≤t ) Language strategy output 116 demonstration observation o t Including image observation and natural language observation. Caption loss L C Can be understood as subtitles and the language policy π defined by the language policy output 116 for image observation l (a l |o) between behavioral cloning losses, where a l is the selected language action. The caption loss encourages the natural language output of the image observation from the action selection neural network system 110 to match the natural language caption of the image observation.
[0071] In such an embodiment, sometimes the natural language output from the action selection neural network system is being trained to match subtitles, i.e., the action selection neural network system 110 is being trained to talk about what it sees, and sometimes the action selection neural network system 110 is being trained to reproduce the natural language input in the demonstration observation, i.e., for the purpose of interactive behavior.
[0072] Optionally, an indicator variable can be used to facilitate the system to distinguish between these tasks, i.e., to indicate whether a subtitle loss is used for training. The action selection neural network system 110 (e.g., the language policy head of the action selection neural network system) can receive as input the corresponding binary indicator or flag embedding. For example, the input of the language policy can be added to the learnable embedding of the indicator variable indicating whether the target is subtitles or language output from a demonstration.
[0073] In some embodiments, training the action selection neural network system using natural language captions also or alternatively involves determining a caption matching loss. This may include processing the embeddings of the natural language captions generated for the image observations of the demonstration observations and the embeddings of the image observations (of the demonstration observations) using the action selection neural network system 110 to generate a combined representation of the natural language captions and the image observations at the intermediate output 119 (step 314).
[0074] The combined representation is processed by the classifier neural network 130 to generate the classifier output 132. This predicts whether the natural language caption and the image observation match (step 316).
[0075] The classifier output 132 is then used to determine a value for a caption matching loss (step 318), and the caption matching loss is used to train the action selection neural network system 110 (step 320). The caption matching loss may include or depend on the classifier output. In some embodiments, the caption matching loss is determined by processing a batch of image observations, where one of the batch of image observations corresponds to a natural language caption. Then, for each image in the batch, the caption matching loss may include the sum of a first term representing the probability that the image observation and the natural language caption are paired and a second term representing the probability that the image observation and the natural language caption are not paired. For example, when the action selection neural network system 110 processes paired (i.e., matching) captions and image observations, the first term may be determined from the classifier output. When the captions and image observations are not paired, i.e., when the captions do not match (i.e., do not correspond to) the image observation, the second term may be determined from the classifier output. The caption matching loss may be used to train the action selection neural network system 110, and in embodiments, the classifier neural network 130 may be trained.
[0076] As a specific example, the subtitle matching loss L CM It can be determined as:
[0077]
[0078] in represents the image observation portion o of the demonstration observation for the bth trajectory in the batch t ,and is a natural language caption as described above. D(·) represents the classifier output 132, which is determined from the intermediate output 119 of the action selection neural network system 110 when the action selection neural network system processes the embedding of the natural language caption and the embedding of the corresponding image observation. CM middle, Representation and Image Observation Unmatched natural language subtitles. In an embodiment represents subtitles from another batch of demonstration observations, for example subtitles selected according to the rolling function roll(b)=(b+1)modB.
[0079] In an embodiment, the classifier neural network and the action selection neural network system 110 are trained in separate training processes. That is, steps 314 to 320 of the process can be performed separately from step 304. In an embodiment, the total effective loss of the system includes the sum of the above losses, namely: L BC +L C +L CM .
[0080] As described above, typically, the image captioning neural network system 120 processes an image observation in one of the demonstration observations to generate a natural language caption for the image observation. In this way, when the action selection neural network system 110 is trained using the demonstration observation, it learns to generalize its training to objects that it has not specifically interacted with but that are present in the image observation and are annotated with the captions, i.e., described as being present in the system. In some embodiments, specific information in the captions can be customized for the task to be performed. For example, if the color of the object is important for the task, the captions can be arranged to describe the color of the object that is present. The captions do not need to describe every object in the image, and typically the image captioning neural network system 120 will generate different captions when reused to process the same or similar image observations. In one example embodiment, the image captioning neural network system 120 describes an average of 1.2 objects out of ~13 objects per image.
[0081] Figure 4 Shows Figure 3 An example implementation of a training process for an image is also shown, and an action selection neural network system for use in inference is also shown. At optional stage 400, the image captioning neural network system 120 is trained to caption images, describing a "new" object, a drum, in the example shown. At stage 402B, the action selection system 100 learns to perform tasks involving interactions with objects (pears) described in one or more natural language observations in the demonstration data. While learning in this manner, the system also learns using self-supervision as described above to generate captions for the image observations in the demonstration data (stage 402A). In this example, the caption for the image observation describes a drum, but there is no mention of a drum in the natural language observations in the demonstration data. Nonetheless, as shown in stage 404, after training as described, the action selection system is able to perform tasks involving drums.
[0082] The image captioning neural network system 120 may be any pre-trained image captioning system, or an image captioning neural network system for use in the system may be conventionally trained, e.g., using supervised training based on labeled training data (i.e., examples of images and their associated captions).
[0083] Conveniently, the image captioning neural network system 120 can be trained using demonstration observations, but there are typically more image observations without corresponding natural language observations than with corresponding natural language observations. Therefore, a technique for semi-supervised training of an image captioning neural network system (such as the image captioning neural network system 120) using a reduced amount of labeled training data is also described.
[0084] Figure 5 A system for training an image captioning neural network system, such as the image captioning neural network system 120, is shown. The system includes an encoder neural network subsystem 502, a decoder neural network subsystem 508, and a language prior neural network 506. After training, the encoder neural network subsystem 502 is used as an image captioning neural network system.
[0085] The encoder neural network subsystem 502 is configured to process image observations to generate image captions. The decoder neural network subsystem 508 is configured to process the image caption input to generate a decoder output 510. In some embodiments, the decoder output 510 defines a reconstruction of the image observation of the corresponding natural language caption from the image, which is used as a discrete latent code 504. In some embodiments, the decoder output 510 includes a value representing the likelihood that a sampled image observation (e.g., sampled from demonstration data) corresponds to the image observation of the corresponding natural language caption. In these latter embodiments, the decoder can operate as a discriminator or classifier that, given a particular caption and an image or set of images, is configured to generate a decoder output value for determining the likelihood that a particular one of the images or set of images corresponds to a particular caption.
[0086] During training, the encoder neural network subsystem 502 and the decoder neural network subsystem 508 may be trained in part using supervised learning and in part using a variational method that involves reconstructing an image from its captions or determining a value of a contrastive loss corresponding to an image reconstruction loss. The variational method uses language as a latent code 504 so that the latent code represents the image as a natural language caption. Thus, training using a variational method also involves encouraging the generated image captions to remain close to a language prior of a language model used as the captions. The language prior may be generated by a pre-trained language prior neural network 506. Because the space of latent codes is large and discrete, in some embodiments, reinforcement learning is used to train the system.
[0087] The encoder neural network subsystem 502, the language prior neural network 506, and the decoder neural network subsystem 508 may each have any suitable neural network architecture, and may include, for example, one or more feed-forward neural network layers, one or more recurrent neural network layers, one or more convolutional neural network layers, one or more attention neural network layers, or one or more normalization layers.
[0088] As an example, the encoder neural network subsystem 502 may include an image embedding neural network, such as ResNet, having an output that regulates a (causal) transformer neural network that is configured to generate one or more output language tokens one at a time. By way of example only, for supervised training language, the encoder may encode the target using sentence bytes (Kudo et al., arXiv:1808.06226, 2018). As an example, the language prior neural network 506 may include a transformer neural network.
[0089] In the case where the decoder neural network subsystem 508 generates an image output from an image caption input, the decoder neural network subsystem can have any generative neural network architecture. As just one example, the decoder neural network subsystem 508 can have a VQ-VAE architecture (van den Oord et al., "Neural discrete representation learning", Advances in Neural Information Processing Systems, 2017) that is conditioned on natural language tokens determined from the image caption input. In such an example, the VQ-VAE can be pre-trained on unlabeled images to define a VQ-VAE codebook. The VQ-VAE tokens can then be autoregressively modeled using a first transformer neural network. A second transformer neural network can be used to process embeddings of language tokens representing the caption input, and the output of this transformer can be cross-attended by the decoder, such as by the first transformer neural network. In the case where the decoder neural network subsystem 508 operates as a discriminator or classifier, it can include an image representation neural network (e.g., an image embedding neural network as described above) and / or a language representation neural network (e.g., a language embedding neural network such as a transformer neural network).
[0090] Figure 6 An example process for training an image captioning neural network system, such as the image captioning neural network system 120, is shown. The training process may be performed independently of the training of the action selection system 100 described previously. Figure 6The process of obtaining a trained image captioning neural network system is configured to process a representation of an image to generate a natural language caption output describing the image. Figure 6 The processes may be implemented by one or more computers in one or more locations. Figure 6 The steps need not be performed in the order shown.
[0091] At step 600, the process obtains image observations, for example from multimodal demonstration data, and obtains image captions for the image observations (where available), thereby obtaining pairs of data items, each pair of data items comprising an image observation and a corresponding image caption. The multimodal demonstration data can be, but need not be, the same data used to train the action selection system 100. For example, in some embodiments, image captions can be generated by asking humans to provide captions for some of the image observations in the multimodal demonstration data. An advantage of the described training technique is that captions only need to be provided for a small portion of the image.
[0092] At step 602, the decoder neural network subsystem 508 is trained using a supervised loss that depends on the corresponding decoder output and a ground truth reference derived from paired data items.
[0093] In the case where the decoder output 510 defines a reconstruction of an image observation, the ground truth reference may be the reconstructed image observation. The decoder output for the paired data items may be generated by processing corresponding natural language captions from the paired data items using the decoder neural network subsystem 508 to generate a decoder output defining a reconstruction of the image observation. The ground truth reference derived from the paired data items may include the image observation for the corresponding image caption.
[0094] Training the decoder neural network subsystem 508 using a supervised loss can include training using a supervised loss function (decoder objective function) that depends on the difference between the reconstruction of the image observation and the image observation corresponding to the image caption. The supervised loss function can be defined as logp θ (x|y), where x represents the reconstructed image, y represents the corresponding natural language caption, and p θ (x|y) represents the decoder neural network subsystem 508 with learnable parameters (e.g., weights θ). In practice, the value of the supervised loss function can be determined as, for example, the squared difference between the image and its reconstruction.
[0095] In the case where the decoder output 510 defines the likelihood of image observations and natural language captions corresponding, the ground truth reference can be a ground truth correspondence between an image and its corresponding image caption. The (ground truth) corresponding image caption can be obtained from a paired data item, or it can be generated by the encoder neural network subsystem 502.
[0096] The paired data items may also be used to train the encoder neural network subsystem 502. For example, the encoder neural network subsystem 502 may be trained using a supervised language loss based on the difference between image captions generated by the encoder neural network subsystem 502 from image observations and image captions corresponding to the image observations in the paired data items.
[0097] At step 604, the encoder neural network subsystem 502 and, in embodiments, the decoder neural network subsystem 508 are (additionally) trained using only image observations, i.e., without relying on corresponding image captions in paired data items.
[0098] Typically, this involves estimating (i.e., inferring) an image caption for an image observation (which may be an image caption that is not paired with a corresponding image caption) by processing the image observation using an encoder neural network subsystem to generate an estimated image caption. The estimated image caption is processed by a decoder neural network subsystem to determine a target value, i.e., the value of an objective function, based on the decoder output of the estimated image caption. This is used to train the encoder neural network subsystem 502, and in embodiments also to train the decoder neural network subsystem 508. For example, the value of the objective function may define a lower bound on the likelihood of a reconstructed image or the likelihood that a particular one of an image observation or a set of image observations corresponds to an estimated image caption. The value of the objective function may then be maximized, or its negative value may be minimized.
[0099] In some embodiments, the objective function has a first term that depends on the likelihood of the decoded image x generated by processing the estimated image caption y, e.g., log p θ (x|y). It may also have a second regularization term that represents the distribution q of the estimated image captions generated by the encoder neural network subsystem 502 ω (y|x) and a fixed (e.g., pre-trained) prior probability distribution p that defines the image captions φ y), where ω represents the learnable parameters of the encoder neural network subsystem and φ represents the learnable parameters of the language prior neural network. The (fixed) distribution defining the prior probabilities can be generated, for example, by training the language prior neural network 506 using natural language observations in the demonstration data. The difference can include a measure of the KL (Kullback-Leibler) divergence, D KL [q ω (y|x)||p φ (y)] and can be estimated randomly using a single value or by sampling multiple (unpaired) image observations x and determining an average. In some embodiments, the objective function J u Can be determined as
[0100] J u =log p θ (x|y)-D KL [q ω (y|x)||p φ (y)]
[0101] In practice, log p θ (x|y) can be determined directly, i.e., as the predicted log-likelihood (e.g., the predicted log-likelihood of discrete tokens generated by a VQ-VAE). The encoder neural network subsystem 502 and optionally the decoder neural network subsystem 508 can be trained to maximize J u .J u Gradients with respect to the learnable parameters of the encoder neural network subsystem 502 It can be determined as:
[0102]
[0103] And can be done, for example, by ω (y|x) is randomly sampled, and an estimate of this is determined by sampling one or more (unpaired) image observations.
[0104] The space of natural language captions (ie, discrete latent codes) is large. In some embodiments, reinforcement learning can be used to maximize the value of the objective function based on a reward corresponding to the value of the objective function.
[0105] Processing image observations using the encoder neural network subsystem 502 to generate estimated image captions implicitly defines a caption determination strategy, e.g., q ω (y|x) can be taken as the subtitle determination policy. Using reinforcement learning to update the subtitle determination policy, i.e., the parameters of the encoder neural network subsystem, can help J u Optimization.
[0106] In embodiments, the reward depends on the accuracy of reconstructing the image observation from the corresponding image caption or the likelihood that the sampled image observation corresponds to the image observation of the corresponding image caption. In embodiments, the reward also depends on the difference between the distribution of the estimated image caption and the distribution defining the prior probability of the image caption (e.g., determined by a metric such as KL divergence). For example, in some embodiments, the reward may depend on the first and second terms described above. In some embodiments, the reward depends on logp θ (x|y)+logp φ (y)-logq ω (y|x).
[0107] Any reinforcement learning technique can be used to learn the caption determination policy, for example, a technique based on temporal difference learning or based on directly learning a policy (e.g., via policy gradients); or another policy optimization technique such as MPO (Maximum A Posterior Policy Optimization, Abdolmaleki et al., 2018) or PPO (arXiv:1707.06347), or a variant thereof such as V-MPO (arXiv:1909.12238 Song et al.).
[0108] The encoder neural network subsystem 502 can be used for the image captioning neural network system 120 after the image captioning neural network system 120 has been trained or while the image captioning neural network system 120 is being trained (step 606). That is, the encoder neural network subsystem can be trained while the action selection system 110 is also being trained.
[0109] In some embodiments, the decoder neural network subsystem 508 includes an image classifier neural network subsystem. The decoder output 510 can then define a value representing the likelihood that a sampled image observation sampled from the multimodal demonstration data corresponds to an image observation of a corresponding image caption. The ground truth reference derived from the paired data items can define when the sampled image observation corresponds to a ground truth, for example, because it is an image observation of a corresponding image caption or an image from which a corresponding image caption was generated using the encoder neural network subsystem. The reconstruction loss log p is generated θ (x|y) is proportional to such a likelihood (up to a constant factor relative to y). Therefore, this likelihood can be used instead of the generative reconstruction loss described previously, e.g., log p θ (x|y). This has the advantage of reducing system complexity, since it can be easier to implement (and train) a classifier to generate image outputs than to generate a neural network.
[0110] Training the image captioning neural network system may then include obtaining a batch of image observations from the multimodal demonstration data, and obtaining captions for selected image observations in the batch from the multimodal demonstration data or by processing the selected image observations using the encoder neural network subsystem. In some implementations, the batch may include captions for each image observation in the batch, but only one image observation-caption pair may be matched.
[0111] The value of the contrastive loss function may then be determined. In an embodiment, the contrastive loss function comprises a combination of the likelihood that the selected image observation corresponds to the obtained caption and the likelihood that other image observations in the batch do not correspond to the obtained caption, such as a cross entropy loss.
[0112] The selected image observations may be processed using an encoder neural network subsystem to generate an estimated image caption. A target value dependent on a decoder output for estimating the image caption may depend on a value of a contrastive loss function.
[0113] In embodiments where a batch includes a caption for each image observation, the contrastive loss function may be symmetrized by combining (e.g., summing) a first loss derived from matching each image in the batch with a corresponding caption in the batch and a second loss derived from matching each caption in the batch with the corresponding image observation.
[0114] As a specific example, for a Given the correct index c=j, the cross entropy loss distinguishes the batch elements y i Is it consistent with batch element x? i Pairwise multi-class classifier, log p θ (x|y) is proportional to up to a constant factor relative to y. So for batch element j, this value can be used instead of log p in the expression θ (x J |y j ), and the gradient with respect to ω remains the same in expectation, i.e. on average.
[0115] The image classifier neural network subsystem may include an image representation neural network, such as a ResNet or other image processing neural network, configured to process a sampled image observation to generate a representation of the sampled image observation, such as an image vector. The image classifier neural network subsystem may also include a subtitle representation neural network, such as a transformer neural network, such as followed by an MLP (multi-layer perceptron), configured to process a corresponding image subtitle to generate a representation of the corresponding image subtitle, such as a subtitle vector. The decoder output may be determined by determining a similarity between a representation of the sampled image observation and a representation of the corresponding image subtitle, such as by determining a dot product or other similarity measure between the image vector and the subtitle vector.
[0116] Continuing with the previous specific example, and denoting the image representation neural network as f(x) and the caption representation neural network as g(y), in some embodiments, the contrastive loss function for the classifier comprises the sum of two losses, one for matching each image to its paired caption in the batch, and one for matching each caption to its paired image:
[0117]
[0118] The image classifier neural network subsystem can be trained by minimizing L, for example, while also training the encoder neural network subsystem 502. In some embodiments, the ω Samplings of (y|x) can be included as positive (i.e., paired) examples for the classifier as well as examples in the multimodal demonstration data.
[0119] Figure 7 Shows Figure 5 A large corpus of unlabeled images together with a relatively small number of images labeled with captions can be used to train the system so that the trained encoder neural network subsystem 502 can generate captions as described. Optionally, in embodiments where the decoder neural network subsystem 508 includes a generative model, the trained decoder neural network subsystem 508 can be used to generate images from captions (i.e., from descriptions of the images to be generated) according to a distribution corresponding to the distribution of the training data.
[0120] Figure 8 Pictured Figure 1 Example implementation of the system after training. Figure 8 Two tasks related to a "drum" object are shown, "lift the drum" and "ask the color drum"; the y-axis represents the relative performance on the task, specifically the reward normalized to human performance. The bar labeled "B" relates to a baseline system with no auxiliary loss and trained on the demonstration data without any drum-based interaction. The bar labeled "D" is for a system trained with drum-based instructions and interactions in the demonstration data (expected upper bound on performance). The bars labeled "C", "M", and "CM" are for implementations of the described system that have a caption loss, a caption matching loss, and both losses, respectively, and are trained on the demonstration data without any drum-based interaction (but some image observations include drums). It can be seen that embodiments of the system can perform tasks involving drums without being explicitly trained to interact with drums, but rather generalizing from other trained tasks.
[0121] Fig. 9 The example embodiment is shown as a control agent 102 (eg, Figure 1 The example of an agent system 900 that selects an action to perform a task in an environment is shown in FIG. 1 . The agent system 900 can be implemented as one or more computer programs on one or more computers in one or more locations. In some implementations, the agent 102 can include the agent system 900.
[0122] The agent system 900 includes an image capture subsystem 902 to capture image observations of an environment at a series of time steps. In a real-world environment, the image capture subsystem can receive signals from one or more image sensors 904 configured to capture image observations of the environment. The agent system 900 includes a local or remote natural language interface 906 to receive natural language input, such as a description of a task to be performed. As an example, a task can involve manipulating or moving an object defined by a natural language description of the task. The natural language interface 906 can also provide natural language output, such as to facilitate a conversation with a user about the task.
[0123] The agent system 900 also includes a trained action selection neural network system 110, for example, an action selection neural network system 110 trained as described above. The action selection neural network system can be local to the mechanical agent or remote from the mechanical agent, for example, onboard the agent or partially or completely implemented on a remote server. The action selection neural network system 110 is configured to process an embedding of an observation input at a time step to generate an action selection policy output 112 for selecting an action 114 at the time step for the agent 102 to perform a task. In an embodiment, the action selection neural network system 110 is also configured to generate a language policy output 116 for defining a natural language output 118 at a time step. The embedding of the observation input at the time step includes an embedding of an image of the environment from the image capture subsystem 902 and an embedding of the natural language input from the natural language interface 906.
[0124] The agent system 900 also includes an agent control system 908 to control the agent to perform the task according to the selected action. The agent control system can be coupled to the action selection strategy output to provide a control signal 910 according to the selected action. For example, where the agent includes a mechanical agent having one or more mechatronic devices to control the movement or motion of the mechanical agent in a real-world environment, the control signal can control the mechatronic device to perform the task.
[0125] Typically, the natural language output at a time step relates to the environment, more specifically to a task performed in the environment, and may include, for example, a natural language description of the environment at the time step, such as a natural language description of the agent or another agent, or a natural language description associated with one or more objects in the environment; or a dialogue associated with the task or environment. In an embodiment, for at least one of the time steps, the embedding of the natural language input includes an embedding of the natural language description of the task.
[0126] Optionally, the agent system 900 may include an image captioning neural network system 120, and optionally also a classifier neural network 130. The agent system 900 may include a training engine, such as a training engine 140, to further train the action selection neural network system 110, such as as previously described and / or using reinforcement learning. For example, it may be useful for the agent to be able to continue training, such as when moving from one environment to another, such as when a home robot is trained in one house and moved to another house.
[0127] Fig.10 An example process for controlling an agent to select actions to perform tasks in an environment, such as using agent system 900, is shown. Fig.10 The processes may be implemented by one or more computers in one or more locations. Fig.10 The steps need not be performed in the order shown.
[0128] In an embodiment, the process obtains a natural language observation including a natural language description of a task to be performed, such as written text or from spoken input processed by a speech recognition system, such as via a local or remote user interface (step 1000).
[0129] At each of a series of time steps, the process may capture an image observation of the environment that characterizes a state of the environment (step 1002).
[0130] At one or more of the time steps, user natural language observations (dialogue inputs) may be received from the user, for example, via a user interface. For example, the user natural language observations may include user turn dialogues related to the task (step 1004).
[0131] At each time step, embeddings of observations including image observations are processed using a trained action selection neural network system (e.g., one trained as described above) to generate an action selection policy output for selecting an action at that time step for the agent to perform the task (step 1006). The trained action selection neural network system may also generate a language policy output for defining a natural language output at the time step. Typically, the natural language output relates to the observation and the task, e.g., it may describe observations relevant to the task, describe an environment relevant to the agent, such as describing the movement of an object or the agent, or it may include dialogue relevant to the task, such as answering a user question.
[0132] The trained action selection neural network system may include a transformer neural network coupled to a recurrent neural network. Processing the embeddings of the image observations and the embeddings of the user natural language observations may include using the transformer neural network to generate a combined representation of the embeddings and then processing the combined representation using the recurrent neural network to generate an action selection policy output and a language policy output.
[0133] For one or more of the time steps, e.g., at an initial time step and optionally at one or more time steps after the initial time step, the observation may include a natural language observation that includes a natural language description of a task to be performed. The embeddings of the image observations and the embeddings of the natural language observations are then processed by the trained action selection neural network system.
[0134] In the case where the user natural language observations include a user turn dialog related to the task, the embedding of the image observations and the embedding of the user natural language observations can be processed by the trained action selection neural network system to generate an action selection policy output and a language policy output. The language policy output can then define a machine turn dialog natural language output in response to the user turn dialog provided to the user.
[0135] A control signal is provided to the agent based on the selected action for controlling the agent to perform the selected action (step 1008). Any type of agent control interface suitable for the agent may be used (an example of an agent is described later). For example, the control signal may include an electrical control signal and / or data for processing by the agent's computer system.
[0136] The natural language output is provided to the user, for example, via a user interface, for example, in text or spoken form (step 1010). This may include providing the machine turn dialog natural language output to the user.
[0137] For illustrative purposes, a few example implementations are described below.
[0138] In some embodiments, the agent is a mechanical agent that acts in a real-world environment to perform a task. The image observations come from one or more image sensors, such as cameras, that sense the real-world environment. The action selection neural network system 110 is configured to process the embeddings of the image observations and the embeddings of the natural language description of the task to generate an action selection policy output for selecting an action for controlling the agent to perform the task.
[0139] In some embodiments, a simulation of a mechanical agent is used in a simulation of a real-world environment to train an action selection neural network system so that the action selection neural network system is then used to control a mechanical agent in the real-world environment. The observations can relate to the real-world environment in the sense that they are observations of the simulation of the real-world environment. The actions can relate to actions to be performed by the mechanical agent acting in the real-world environment to perform a task in the sense that they are simulations of actions that will later be performed in the real-world environment. After the action selection neural network system has been trained to perform a task in a simulation, it can be used to control the mechanical agent to act on the real-world environment to perform that task or another task.
[0140] In more detail, where the environment is a real-world environment, the agent may be a mechanical agent that interacts with the real-world environment, such as a robot or an autonomous or semi-autonomous land, air or sea vehicle that operates in or navigates through the environment, and the action is an action taken by the mechanical agent in the real-world environment to perform a task. For example, the agent may be a robot that interacts with the environment to complete a specific task, such as locating an object of interest in the environment, or manipulating a specified object, or moving a specified object to a specified location in the environment, or navigating to a specified destination in the environment. The object, location or destination may be specified by a natural language instruction as previously described.
[0141] Observations include, for example, image or video data from a camera or other image sensor (e.g., a LIDAR sensor) (herein, "image" includes a point cloud). The camera or other image sensor can be mounted on the agent or positioned separately from the agent in the environment. Observations can also include other sensor data, such as object position data, data from a distance or position sensor, data from an actuator, or sensed electronic signals (such as motor current or temperature signals). In the case of a robot, observations can also include data characterizing the current state of the robot, for example, one or more of the following: joint positions, joint velocities, joint forces, torques, or accelerations, for example, gravity-compensated torque feedback, and a global or relative pose of one or more parts of an item or agent held by the robot. Optionally, in any described embodiment, observations at any given time step can include data from a previous time step, which can be useful in characterizing the environment.
[0142] Actions may include control signals for controlling a robot or other mechanical agent, such as torques for joints of a robot or torques to control surfaces or other control elements (e.g., steering control elements of a vehicle), or higher level control commands. Control signals may include, for example, position, velocity or force, torque or acceleration data of one or more joints of a robot or a component of another mechanical agent. Control signals may also or alternatively include electronic control data, such as motor control data, or, for vehicles, signals for controlling navigation (e.g., steering) and movement (e.g., braking and / or acceleration of a vehicle).
[0143] As previously described, the environment can be a simulation of a specific real-world environment, and the agent can be implemented as one or more computers that interact with the simulated environment. For example, the simulated environment can be a simulation of a robot or vehicle, and the system can be trained on the simulation and then, once trained, used to control a real-world mechanical agent in a specific or similar real-world environment. This can avoid unnecessary wear and damage to the real-world environment or real-world agent, and can allow the training and evaluation of control neural networks in situations that rarely occur or are difficult to recreate in the real-world environment. In some cases, the system can be partially trained using a simulation as described above and then further trained in the real-world environment. Typically, in the case of a simulated environment, observations can include simulated versions of one or more of the previously described observations or observation types, and actions can include simulated versions of one or more of the previously described actions or action types.
[0144] In some applications, the real-world environment is a manufacturing environment for manufacturing products, such as chemical, biological or mechanical products or food products (as used herein, including the manufacture of food products by kitchen robots). The mechanical agent can then be a machine such as a robot that operates to manufacture the product or a part thereof, or a machine that controls the movement of intermediate versions or components of the product between manufacturing units. The task can be, for example, any type of task related to the manufacture of the product or its intermediate versions or components, including control tasks, such as minimizing the use of resources, such as controlling the consumption of electricity or water or the consumption of any materials or consumables used in the manufacturing process, or tasks to optimize product quality. Such an environment can include chemical synthesis, such as a protein or drug synthesis environment, where the product is a chemical, such as a protein or drug, or an intermediate or component thereof.
[0145] In some applications, the real world environment is a facility that generates or uses electricity or water, the mechanical agent includes a machine, and the task is to control the generation or use of electricity or water. For example, the mechanical agent can be a machine that controls the delivery of electricity or a configuration of one or more renewable power generation elements, such as a configuration of wind turbines or solar panels or mirrors, or a configuration of rotating power generating machines.
[0146] In the above applications, the control signal may be any control signal suitable for the type of machine being controlled.
[0147] This specification uses the term "configuration" in conjunction with system and computer program components. For a system of one or more computers configured to perform a particular operation or action, it means that the system has installed thereon software, firmware, hardware, or a combination thereof, which, in operation, causes the system to perform the operation or action. For one or more computer programs configured to perform a particular operation or action, it means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.
[0148] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware (including the structures disclosed in this specification and their structural equivalents), or a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-temporary storage medium, for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or additionally, program instructions may be encoded on an artificially generated propagation signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.
[0149] The term "data processing apparatus" refers to data processing hardware and includes all kinds of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The apparatus may also be or further include special-purpose logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may optionally include code that creates an execution environment for a computer program, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
[0150] A computer program (which may also be referred to or described as a program, software, software application, app, module, software module, script, or code) may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, such as files storing one or more modules, subroutines, or code portions. A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed over multiple sites and interconnected by a data communications network.
[0151] In this specification, the term "database" is used broadly to refer to any collection of data: the data need not be structured in any particular way, or at all, and it may be stored on a storage device in one or more locations. Thus, for example, an index database may include multiple collections of data, each of which may be organized and accessed differently.
[0152] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same or multiple computers.
[0153] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a special purpose logic circuit (e.g., an FPGA or ASIC), or by a combination of a special purpose logic circuit and one or more programmed computers.
[0154] Computers suitable for executing computer programs can be based on general or special microprocessors or both, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The elements of a computer are a central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by a special logic circuit or incorporated into the special logic circuit. Typically, the computer will also include one or more large-capacity storage devices (such as disks, magneto-optical disks or optical disks) for storing data, or be operably coupled to receive data from it or send data to it or both. However, the computer does not necessarily have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.
[0155] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks.
[0156] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by a user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. In addition, a computer may interact with a user by sending a text message or other form of message to a personal device (e.g., a smart phone running a messaging application) and receiving a response message from the user in return.
[0157] The data processing apparatus for implementing a machine learning model may also include, for example, dedicated hardware accelerator units for processing the common and computationally intensive parts of machine learning training or production, i.e., inference, workloads.
[0158] The machine learning model can be implemented and deployed using a machine learning framework (e.g., the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework).
[0159] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.
[0160] A computing system may include a client and a server. The client and the server are usually remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by means of computer programs that run on respective computers and have a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to a user device, for example, for the purpose of displaying data to a user interacting with a device acting as a client and receiving user input therefrom. Data generated at a user device, for example, a result of a user interaction, may be received from the device at the server.
[0161] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or the scope that may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations and even initially claimed as such, one or more features from the claimed combination may be deleted from the combination in some cases, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.
[0162] Similarly, although operations are depicted in the drawings and recited in a particular order in the claims, this should not be understood as requiring that the operations be performed in the particular order shown or in sequence, or that all of the illustrated operations be performed, to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
[0163] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.
Claims
1. A computer-implemented method for training an action-selecting neural network system to control an agent to select actions to perform a task in an environment, in, The action selection neural network system is configured to process embeddings of observations to generate an action selection policy output for selecting an action to be performed by the agent and a language policy output for generating a natural language output, the embeddings of observations comprising an embedding of an image of the environment and an embedding of a natural language input; The method comprises: obtaining multimodal demonstration data including a plurality of task demonstration sequences, each task demonstration sequence including a sequence of demonstration observation and demonstration action, wherein the demonstration observations include image observations characterizing states of the environment as a demonstration agent interacts with the environment to perform a task, and at least one natural language observation describing the task performed, and wherein the demonstration actions characterize actions of the demonstration agent in the environment as the task is performed; processing the embedding of the demonstration observation using the action selection neural network system to generate the action selection policy output and the language policy output for the demonstration observation; and training the action selection neural network system using natural language outputs defined by the language policy outputs such that actions defined by the action selection policy outputs from the action selection neural network system are encouraged to match actions of the demonstration agent; Wherein, training the action selection neural network system further comprises: processing the image observations of the demonstration observations using an image captioning neural network system to generate natural language captions for the image observations; and The action selection neural network system is trained using the natural language captions.
2. The method according to claim 1, comprising: training the action selection neural network system so that a natural language output defined by the language policy output for the demonstration observation is encouraged to match a natural language observation in the demonstration observation; as well as The action selection neural network system is trained using an auxiliary loss based on the natural language captions.
3. The method according to claim 1 or 2, wherein: Using the natural language captions to train the action selection neural network system includes: determining a captioning loss, wherein the captioning loss is dependent on a difference between the natural language captions generated for the image observation for a demonstration observation and the natural language output from the action selection neural network system for the demonstration observation; and The action selection neural network system is trained using the caption loss.
4. The method according to claim 1, 2 or 3, wherein: Using the natural language captions to train the action selection neural network system includes: processing the embedding of the natural language caption generated for the image observation of a demonstration observation and the embedding of the image observation using an action selection neural network system to generate a combined representation of the natural language caption and the image observation; processing the combined representation using a classifier neural network to generate a classifier output predicting whether the natural language caption and the image observation match; Determining a value of a subtitle matching loss according to the classifier output; and The action selection neural network system is trained using the caption matching loss.
5. The method according to any one of claims 1 to 4, wherein: The action selection neural network system comprises a transformer neural network coupled to a memory, and wherein generating the action selection strategy output and the language strategy output comprises: processing the embedding of the demonstration observation by processing the embedding of the image of the demonstration observation and the embedding of the natural language description of the demonstration observation using the same transformer neural network to generate a combined representation of the embedding of the demonstration observation; providing the combined representation to the memory; One or more outputs from the memory are processed to generate the action selection strategy output and the language strategy output.
6. The method according to any one of claims 5, wherein: The memory includes a recurrent neural network.
7. A method according to any preceding claim, wherein training the action selection neural network system so that the actions defined by the action selection policy output from the action selection neural network system match the actions of the demonstration agent includes training the action selection neural network system to optimize an objective function that depends on the difference between the distribution of actions defined by the action selection policy output and the distribution of actions defined by the actions of the demonstration agent.
8. The method of any preceding claim, further comprising training the image captioning neural network system by: retrieving image observations and retrieving image captions for the image observations from the multimodal demonstration data to obtain pairs of data items, each data item comprising an image observation and a corresponding image caption; A decoder neural network subsystem is trained using the paired data items, the decoder neural network subsystem being configured to process the image captions to generate a decoder output by: processing the corresponding image-caption of each pair of data items using a decoder neural network subsystem to generate a respective decoder output, wherein the decoder output defines a reconstruction of the image observation from the corresponding image caption or defines a value representing a likelihood that a sampled image observation sampled from the multimodal demonstration data corresponds to the image observation of the corresponding image caption; as well as training the decoder neural network subsystem using a supervised loss that depends on the corresponding decoder output and a ground truth reference derived from the paired data items; For each of a plurality of image observations from the multimodal demonstration data, an encoder neural network subsystem configured to process the image observation to generate image captions is trained by: processing the image observations using the encoder neural network subsystem to generate imputed image captions; as well as training the encoder neural network subsystem to maximize the likelihood of the image observation given the estimated image caption by maximizing a target value that depends on the decoder output for the estimated image caption; as well as The encoder neural network subsystem is used in the image captioning neural network system.
9. A method for training an image captioning neural network system, comprising: retrieving image observations and image captions of the image observations from the multimodal demonstration data to obtain pairs of data items, each data item comprising an image observation and a corresponding image caption; A decoder neural network subsystem is trained using the paired data items, the decoder neural network subsystem being configured to process the image captions to generate a decoder output by: processing the corresponding image-caption for each of the paired data items using the decoder neural network subsystem to generate a respective decoder output, wherein the decoder output defines a reconstruction of the image observation from the corresponding image-caption or defines a value representing a likelihood that a sampled image observation sampled from the multimodal demonstration data corresponds to the image observation of the corresponding image-caption; as well as training the decoder neural network subsystem using a supervised loss that depends on the corresponding decoder output and a ground truth reference derived from the paired data items; For each of a plurality of image observations from the multimodal demonstration data, an encoder neural network subsystem is trained, the encoder neural network subsystem being configured to process the image observations to generate image captions, by: processing the image observations using the encoder neural network subsystem to generate estimated image captions; as well as training the encoder neural network subsystem to maximize the likelihood of the image observation given the estimated image caption by maximizing a target value that depends on the decoder output for the estimated image caption; as well as The encoder neural network subsystem is used in the image captioning neural network system.
10. The method according to claim 8 or 9, wherein: The decoder output defines a reconstruction of the image observation according to the corresponding image caption, wherein the ground truth reference derived from the paired data items includes the image observation for the corresponding image caption, and wherein training the decoder neural network subsystem using the supervised loss includes training using a supervised loss function that depends on the difference between the reconstruction of the image observation and the image observation for the corresponding image caption.
11. The method according to claim 8, 9 or 10, wherein: The target value depending on the decoder output for the estimated image subtitles includes a first term that depends on the likelihood of a decoded image generated by processing the estimated image subtitles using the decoder neural network subsystem, and a second term that represents the difference between the distribution of the estimated image subtitles and the distribution defining a prior probability of the image subtitles.
12. The method according to any one of claims 8 to 11, wherein: The decoder neural network subsystem comprises an image classifier neural network subsystem; wherein the decoder output defines a value representing the likelihood that a sampled image observation sampled from the multimodal demonstration data corresponds to the image observation of the corresponding image caption; and wherein the ground truth reference derived from the paired data items defines when the sampled image observation is the image observation for the corresponding image caption or one of the images from which the corresponding image caption was generated using the encoder neural network subsystem.
13. The method according to any one of claims 8 to 12, further comprising: obtaining a batch of image observations from the multimodal demonstration data; obtaining captions for selected image observations in the batch from the multimodal demonstration data or by processing the selected image observations using the encoder neural network subsystem; determining a value of a contrastive loss function, wherein the contrastive loss function comprises a combination of a likelihood that the selected image observation corresponds to the obtained caption and a likelihood that other image observations in the batch do not correspond to the obtained caption; wherein the image observation processed using the encoder neural network subsystem to generate the estimated image caption is the selected image observation; and Therein, the target value depending on the decoder output for the estimated image caption depends on the value of the contrastive loss function.
14. The method according to claim 12 or 13, wherein: The image classifier neural network subsystem includes an image representation neural network and a subtitle representation neural network, wherein the image representation neural network is configured to process the sampled image observations to generate representations of the sampled image observations, and the subtitle representation neural network is configured to process the corresponding image subtitles to generate representations of the corresponding image subtitles; the method also includes determining the decoder output by determining a similarity between the representations of the sampled image observations and the representations of the corresponding image subtitles.
15. The method according to any one of claims 8 to 14, wherein: The target value of the decoder output depending on the estimated image subtitles comprises a second term depending on the difference between a distribution representing the estimated image subtitles and a distribution defining a priori probabilities of the image subtitles.
16. The method according to any one of claims 8 to 15, wherein: Training the encoder neural network subsystem by maximizing the target value dependent on the decoder output for the estimated image caption comprises: determining a reward that depends on i) the accuracy of reconstructing the image observation from the corresponding image caption or the likelihood that the sampled image observation corresponds to the image observation of the corresponding image caption; and ii) the difference between the distribution of the estimated image captions and a distribution defining a prior probability of the image captions; and wherein processing the image observations using the encoder neural network subsystem to generate estimated image captions defines a caption determination strategy; The method further comprises: The encoder neural network subsystem is trained using reinforcement learning techniques to update the caption determination policy using the rewards.
17. The method according to any one of claims 1 to 8 and claims 10 to 16 when dependent on claim 8, further comprising: Receive a natural language description of the task; providing an embedding of the natural language description of the task to the action selection neural network system; as well as The agent is controlled using the action selection neural network system to select actions to perform the requested task in the environment.
18. A computer-implemented method of controlling an agent to select an action to perform a task in an environment, the method comprising: Obtain a natural language description of the task to be performed; as well as At each of a series of time steps: capturing an image observation of the environment characterizing a state of the environment; processing embeddings of observations including the image observation using a trained action selection neural network system to generate an action selection policy output for selecting an action at the time step for the agent to perform a task, and generating a language policy output for defining a natural language output at the time step, wherein the natural language output is related to the observation and the task; Providing a control signal to the agent according to the selected action, for controlling the agent to perform the selected action; as well as providing the natural language output to a user; and wherein For at least one of the time steps, the observation comprises a natural language observation comprising a natural language description of the task to be performed, and processing the embedding of the observation comprises processing an embedding of the image observation and an embedding of the natural language observation.
19. The method of claim 18, further comprising, at one or more of the time steps: receiving a user natural language observation from the user, wherein: The user natural language observation includes user turn dialogue related to the task; as well as processing the embeddings of the image observations and the embeddings of the user natural language observations to generate the action selection policy output and the language policy output, wherein the language policy output defines a machine-turn dialog natural language output in response to the user-turn dialog; and Wherein, providing the natural language output to the user includes providing the machine-turn dialogue natural language output to the user.
20. The method according to claim 19, comprising: processing the embedding of the image observation and the embedding of the user natural language observation using a transformer neural network to generate a combined representation of the embeddings; as well as The combined representation is processed using a recurrent neural network to generate the action selection policy output and the language policy output.
21. A method according to any one of claims 1 to 8 and claims 10 to 17 when dependent on claim 8, comprising: The action selection neural network system is trained using a simulation of a mechanical agent in a simulation of a real-world environment to control the mechanical agent in the real-world environment using the action selection neural network system, wherein the observations are related to the real-world environment and wherein the actions are related to actions to be performed by the mechanical agent acting in the real-world environment to perform the task.
22. A method according to any one of claims 1 to 8 and claims 10 to 20 when dependent on claim 8, wherein The agent is a mechanical agent, the environment is a real-world environment, the image observations are from one or more image sensors sensing the real-world environment, and the actions are used to control the mechanical agent acting in the real-world environment to perform the task.
23. An agent system configured to select actions to perform a task in an environment, the agent system comprising: an image capture subsystem for capturing image observations of the environment at a series of time steps; A natural language interface for receiving a natural language description of a task to be performed; a trained action selection neural network system, wherein the action selection neural network system is configured to process embeddings of observations input at a time step to generate an action selection policy output for selecting an action at the time step to cause the agent to perform the task, and to generate a language policy output for defining a natural language output at the time step, wherein the embeddings of the observations at the time step include embeddings of images of the environment from the image capture subsystem and embeddings of natural language input from the natural language interface; and an agent control system for controlling the agent to perform the task according to the selected action; wherein the natural language output at the time step includes natural language associated with the environment at the time step; and Wherein, for at least one of the time steps, the embedding of the natural language input comprises an embedding of the natural language description of the task.
24. An agent system according to claim 23, wherein the agent is a mechanical agent; wherein the environment is a real-world environment; and wherein said image capture subsystem includes one or more image sensors to capture said image observations of said real-world environment; The mechanical agent further includes one or more electromechanical devices to control movement or motion of the mechanical agent in the real-world environment; The agent control system is coupled to the action selection strategy output to provide a control signal according to the selected action to control the one or more electromechanical devices for the agent to perform the task.
25. The agent system according to claim 23 or 24, further comprising: an image captioning neural network system configured to process the image observation to generate a natural language caption for the image observation; as well as A training engine configured to train the action selection neural network system using the natural language captions from the image captioning neural network system.
26. A system comprising one or more computers and one or more storage devices storing instructions, the instructions being operable when executed by the one or more computers to cause the one or more computers to perform the operations of the corresponding method of any one of claims 1-22.
27. A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the corresponding method of any one of claims 1-22.
