Control robot based on free form natural language input

Abstract

A method implemented by a processor is provided. The method includes generating semantic visual data based on processing instances of robot visual data using one or more neural networks, where the robot visual data is generated based on outputs from visual components of the robot, and wherein the semantic vision data comprises natural language tags of the objects captured in at least some of the instances of the robot vision data, and receiving free-form natural language input based on processing of both the natural language tags of the objects and the free-form natural language input using an additional neural network, generating a motion prediction output indicating a robot motion to be performed; and controlling an actuator of the robot based on the motion prediction output, causing the robot to perform the robot motion indicated by the motion prediction output.

Description

[0001] This application is a divisional application of the application with the application date of March 22, 2019 and the application number 201980031823.1, whose invention name is "Controlling a robot based on free-form natural language input". Background Art

[0002] Many robots are programmed to perform certain tasks. For example, a robot on an assembly line can be programmed to recognize specific objects and perform specific actions on those specific objects.

[0003] In addition, some robots can perform specific tasks in response to explicit user interface inputs corresponding to specific tasks. For example, a vacuuming robot can perform a general vacuuming task in response to the voice of "robot, clean". However, generally, the user interface input that causes the robot to perform a specific task must be explicitly mapped to the task. Therefore, the robot may not be able to perform a specific task in response to various free-form natural language inputs from a user trying to control the robot. For example, the robot may not be able to navigate to a target location based on the free-form natural language input provided by the user. For example, the robot may not be able to navigate to a specific location in response to a user request to "go out, turn left and go through the door at the end of the corridor". Summary of the invention

[0004] Embodiments disclosed herein utilize deep reinforcement learning to train a model (e.g., a deep neural network model) that can be used at each of a plurality of time steps to determine a corresponding robot action for completing a robot task. Additionally or alternatively, embodiments relate to utilizing such a model in controlling a robot. The robot action determined at a given time step using such a model can be based on: current sensor data associated with the robot within the given time step, and free-form natural language input provided by a user. The current sensor data associated with the robot can include, for example, a current visual data instance generated based on outputs of one or more visual sensors from a visual component of the robot, wherein the current visual data instance captures at least a portion of the robot's environment. The free-form natural language input provided by the user can, for example, guide the robot to optionally reference one or more intermediate steps (and optionally only reference (one or more) intermediate steps) for completing a specific task to complete the specific task. For example, the free-form natural language input can guide the robot to navigate to a specific landmark with reference to one or more intermediate landmarks that will be encountered when navigating to the specific landmark.

[0005] In some embodiments, one or more state branches of the model are used to process current sensor data within a given time step and generate one or more state representations (e.g., embeddings) about the given time step. For example, one or more visual branches can be used to process visual data instances to generate at least one visual embedding about the given time step. In addition, the natural language branch of the model is used to process free-form natural language input to generate a natural language representation. The natural language representation within a given time step can optionally be based on attention weighting during processing, and the attention weighting is based on one or more of the state representations within a given time step. Further, the natural language representation and (one or more) state representations can be processed on the policy network of the model to determine the robot action to be implemented within a given time step. As described above, deep reinforcement learning can be used to train the policy network, and optionally used to jointly train at least a portion of the state representation and the natural language branch (e.g., attention function). In these and other ways, the trained model can be based on the current state associated with the robot within the time step and based on the free-form natural language input provided by the user, so that the corresponding robot action can be determined and implemented at each time step in multiple time steps. Through the implementation of robot actions over multiple time steps, the robot can complete tasks instructed by free-form natural language input. The trained model can selectively focus on the parts of the natural language input that are relevant to the current visual context when generating action prediction output, which can lead to an increase in the robot's success rate when following natural language instructions.

[0006] In some embodiments, a method implemented by one or more processors is provided. The method includes: generating semantic visual data, wherein generating the semantic visual data is based on processing instances of robot visual data using one or more neural networks, wherein the robot visual data is generated based on output from a visual component of a robot, and wherein the semantic visual data includes natural language labels of objects captured in at least some of the instances of the robot visual data, and receiving free-form natural language input, wherein the free-form natural language input is generated based on user interface input provided by a user via one or more user interface input devices; generating an action prediction output indicating a robot action to be performed based on processing both the natural language label of the object and the free-form natural language input using an additional neural network; and controlling one or more actuators of a robot based on the action prediction output, wherein controlling the one or more actuators of the robot causes the robot to perform the robot action indicated by the action prediction output.

[0007] In some embodiments, a robot is provided. The robot includes: a visual component; an actuator; a memory storing instructions and one or more processors. The one or more processors are operable to execute the instructions to: generate semantic visual data based on processing instances of robot visual data using one or more neural networks, wherein the robot visual data is generated based on output from the visual component, and wherein the semantic visual data includes natural language labels of objects captured in at least some of the instances of robot visual data, and receive free-form natural language input, the free-form natural language input is generated based on user interface input provided by a user via one or more user interface input devices; based on processing both the natural language label of the object and the free-form natural language input using an additional neural network, generate an action prediction output indicating a robot action to be performed; and control one or more of the actuators based on the action prediction output, causing the robot to perform the robot action indicated by the action prediction output.

[0008] In some embodiments, a method implemented by one or more processors is provided and includes receiving a visual data instance. The visual data instance is generated based on output of one or more visual sensors from a visual component of a robot and captures at least a portion of an environment of the robot. The method also includes generating at least one visual embedding based on processing the visual data instance using at least one visual branch of a neural network model. The method also includes receiving a free-form natural language input, the free-form natural language input being generated based on a user interface input provided by a user via one or more user interface input devices. The method also includes generating a natural language embedding based on processing the free-form natural language input using a language branch of the neural network model. The method also includes generating an action prediction output based on processing the at least one visual embedding and the natural language embedding using an action prediction layer of the neural network model. The generated action prediction output indicates a robot action to be performed based on the visual data instance and the free-form natural language input. The method also includes controlling one or more actuators of the robot based on the action prediction output, wherein controlling the one or more actuators of the robot causes the robot to perform the robot action indicated by the action prediction output.

[0009] These and other implementations may include one or more of the following features.

[0010] In some embodiments, the language branch of the neural network model includes a storage network (e.g., a bidirectional storage network) and an attention layer. In some of those embodiments, generating the natural language embedding based on processing the free-form natural language input using the language branch of the neural network model includes: generating a bidirectional storage network output based on processing the free-form natural language input using the bidirectional storage network; generating an attention-weighted bidirectional storage network output based on processing the bidirectional storage network output using an attention layer that is at least partially conditioned on the at least one visual embedding; and generating the natural language embedding based on further processing of the attention-weighted bidirectional storage network output. In some versions of those embodiments, generating the natural language embedding based on further processing of the attention-weighted bidirectional storage network output includes generating the natural language embedding based on processing the attention-weighted bidirectional storage network output on at least one additional feed-forward layer of the language branch of the neural network model. In some additional or alternative versions, after processing the free-form natural language input using the bidirectional storage network, the attention layer is further conditioned on the hidden states of the forward nodes and backward nodes of the bidirectional storage network. The bidirectional memory network output may be, for example, a concatenation of the forward node and the backward node of the bidirectional memory network after processing the free-form natural language input using the bidirectional memory network.

[0011] In some embodiments, generating at least one visual embedding based on processing the visual data instance using the at least one visual branch of the neural network model includes: generating a deep embedding of the at least one visual embedding and generating a semantic embedding of the at least one visual embedding. Generating the deep embedding can be based on processing the depth data of the visual data instance using the deep visual branch of the at least one visual branch and generating the semantic embedding can be based on processing the semantic data of the visual data instance using the semantic visual branch of the at least one visual branch. In some of those embodiments, the depth data includes a corresponding depth metric for each of a plurality of pixel or voxel groups; and the semantic data includes a corresponding semantic identifier for each of the plurality of pixel or voxel groups. Each of the plurality of pixel or voxel groups can include only a corresponding single one of the pixels or voxels, or can include multiple pixels or voxels, such as two or more adjacent pixels or voxels. In some embodiments, the semantic data of the visual data instance can be generated based on a separate classification model that classifies each of a plurality of portions of an image into one or more corresponding semantic categories (e.g., only a single corresponding category per portion).

[0012] In some embodiments, the vision component of the robot is a stereo camera or a LIDAR component.

[0013] In some embodiments, controlling the one or more actuators of the robot includes: selecting a next occupied unit of the robot from a plurality of candidate next occupied units adjacent to a current occupied unit of the robot based on the robot's action; and controlling the one or more actuators to cause the robot to move to the selected next occupied unit.

[0014] In some embodiments, the robot action indicates movement of the robot in a direction, and controlling the one or more actuators of the robot causes movement of the robot in the direction indicated by the robot action.

[0015] In some embodiments, the action prediction output includes a value corresponding to the robot action, and a plurality of additional values ​​corresponding to additional candidate robot actions. In some of those embodiments, the action prediction output indicates the robot action based on the value corresponding to the robot action satisfying a threshold. In some versions of those embodiments, at least one of the additional candidate robot actions indicates a first change in the orientation of the robot, and the robot action indicates a second change in the orientation of the robot or a forward movement of the robot without a change in orientation. In some of those versions, when the robot action indicates the second change in orientation, controlling the one or more actuators of the robot includes controlling the one or more actuators to cause the second change in orientation without causing the robot to move from a currently occupied cell of the robot. In addition, in some of those versions, when the robot action indicates the forward movement of the robot, controlling the one or more actuators of the robot includes controlling the one or more actuators to cause the robot to move from a currently occupied cell of the robot to an adjacent occupied cell of the robot.

[0016] In some implementations, the free-form natural language input is generated based on speech provided by the user and detected via a microphone of the robot.

[0017] In some implementations, the free-form natural language input is generated based on speech-to-text processing of audio data corresponding to the detection of the speech via the microphone of the robot.

[0018] In some embodiments, the method further includes, after controlling one or more actuators of the robot based on the action prediction output: receiving additional visual data instances, the additional visual data instances generated based on additional outputs from one or more visual sensors of a visual component of the robot; generating at least one additional visual embedding based on processing the additional visual data instances using the at least one visual branch of the neural network model; during the processing, generating additional natural language embeddings based on processing the free-form natural language input using the language branch of the neural network model and based on attention weighting, the attention weighting being based on the additional visual embedding; generating additional action prediction outputs based on processing the at least one additional visual embedding and the additional natural language embeddings using the action prediction layer of the neural network model, wherein the additional action prediction outputs indicate additional robot actions to be performed based on the additional visual data instances and the free-form natural language input; and controlling the one or more actuators of the robot based on the additional action prediction outputs, wherein controlling the one or more actuators of the robot causes the robot to perform the additional robot actions indicated by the additional action prediction outputs.

[0019] In some embodiments, a method implemented by one or more processors is provided and includes receiving a free-form natural language input generated based on a user interface input provided by a user via one or more user interface input devices. The method also includes generating a storage network output based on processing the free-form natural language input using a storage network. For each of a plurality of iterations, the method further includes: receiving a corresponding visual data instance, the corresponding visual data instance being generated for the iteration based on corresponding outputs from one or more visual sensors of a visual component of the robot; generating at least one corresponding visual embedding based on processing the corresponding visual data instance using at least one visual branch comprising a plurality of convolutional neural network layers; generating a corresponding natural language embedding based on processing the storage network output using an attention layer at least partially conditioned on the corresponding visual embedding; generating a corresponding action prediction output based on processing the at least one corresponding visual embedding and the corresponding natural language embedding using a plurality of feed-forward layers, wherein the corresponding action prediction output indicates a corresponding robot action among a plurality of candidate robot actions to be performed based on the corresponding visual data instance and the free-form natural language input; and controlling one or more actuators of the robot based on the corresponding action prediction output, wherein controlling the one or more actuators of the robot causes the robot to perform the corresponding robot action indicated by the corresponding action prediction output.

[0020] In some implementations, a method implemented by one or more processors is provided and includes recognizing free-form natural language describing navigation to a target location; and generating a storage network output based on processing the free-form natural language input using a storage network. For each of a plurality of iterations, the method further comprises: receiving an instance of corresponding simulated or real visual data; generating at least one corresponding visual embedding based on processing the instance of the corresponding simulated or real visual data using at least one visual branch comprising a plurality of convolutional neural network layers; generating a corresponding natural language embedding based on processing the stored network output using an attention layer conditioned at least in part on the corresponding visual embedding; generating a corresponding action prediction output based on processing the at least one corresponding visual embedding and the corresponding natural language embedding using a plurality of feed-forward layers that approximate an action-value function, wherein the corresponding action prediction output indicates a corresponding robot action among a plurality of candidate robot actions to be performed based on the corresponding visual data instance and the free-form natural language input; controlling a real or simulated robot based on the corresponding action prediction output; determining whether controlling the real or simulated robot based on the corresponding action prediction output causes the real or simulated robot to reach the target position; and when determining that controlling the real or simulated robot based on the corresponding action prediction output causes the real or simulated robot to reach the target position: using a reward corresponding to reaching the target position in updating at least the feed-forward layer that approximates the action-value function, wherein updating at least the feed-forward layer is a basis.

[0021] In some of those embodiments, the method further includes: identifying at least one intermediate position to be encountered when navigating to the target position and for each of the multiple iterations: determining whether controlling the real or simulated robot based on the corresponding action prediction output causes the real or simulated robot to reach the at least one intermediate position; and when it is determined that controlling the real or simulated robot based on the corresponding action prediction output causes the real or simulated robot to reach the at least one intermediate position: in at least updating the feedforward layer that approximates the action value function, using an intermediate reward corresponding to reaching the at least one intermediate position, wherein the intermediate reward is reduced relative to the reward.

[0022] In some embodiments, a method implemented by one or more processors is provided and includes receiving an instance of robot sensor data generated based on outputs from one or more sensors of a robot, and capturing at least a portion of an environment of the robot. The method also includes generating at least one robot state embedding based on processing the robot sensor data instance using a state branch of a neural network model. The method also includes receiving a free-form natural language input, the free-form natural language input being generated based on a user interface input provided by a user via one or more user interface input devices. The method also includes generating a natural language embedding based on processing the free-form natural language input using a language branch of the neural network model. The method also includes generating an action prediction output based on processing the at least one visual embedding and the natural language embedding using an action prediction layer of the neural network model. The action prediction output indicates a robot action to be performed based on the robot sensor data instance and the free-form natural language input. The method also includes controlling one or more actuators of the robot based on the action prediction output.

[0023] Other embodiments may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., one or more central processing units (CPUs), one or more graphics processing units (GPUs), and / or one or more tensor processing units (TPUs)) to perform a method such as one or more of the methods described above and / or elsewhere herein. Yet another embodiment may include a system of one or more computers and / or one or more robots having one or more processors operable to execute the stored instructions to perform a method such as one or more of the methods described above and / or elsewhere herein.

[0024] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are contemplated as part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are contemplated as part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 An example environment is shown in which embodiments disclosed herein may be implemented.

[0026] Figure 2 is an example method of controlling a robot based on natural language input from a user and based on visual data from vision component(s) of the robot.

[0027] Figure 3is an example method for training a neural network model for controlling a robot based on natural language input from a user and based on visual data.

[0028] Figure 4 Showing an example architecture of a robot. DETAILED DESCRIPTION

[0029] Disclosed below are various embodiments related to training and / or utilizing a neural network model that can be used at each of a plurality of time steps to determine a corresponding robot action for completing a robot task. The robot action determined at a given time step using such a model can be based on: current sensor data associated with the robot at the given time step, and free-form natural language input provided by a user. Although specific examples are described below with respect to current sensor data being visual data (e.g., depth data and / or visual data), it should be understood that in some embodiments, additional and / or alternative sensor data can be utilized, such as radar data, sensor data from sensors corresponding to (one or more) actuators of the robot (e.g., force sensors, torque sensors), and the like. For example, the depth vision branch and / or semantic vision branch described below can be replaced or supplemented with (one or more) branches that process alternative sensor data (such as radar data, force sensor data, and the like).

[0030] Embodiments disclosed herein utilize natural language input, such as free-form natural language input generated based on user interface input provided by a user via a user interface input device. As used herein, free-form natural language input includes natural language input based on a user's voice, a user's typing, or a user's (one or more) other inputs, the other inputs being not limited to a limited set of options presented by a user's selection (e.g., not limited to a set of options presented in a drop-down menu). An instance of free-form natural language input can be based on user input via one or more user interface input devices of a controlled robot, or can be received based on user input via one or more user interface input devices of a separate component (e.g., a smartphone, a tablet computer, a stand-alone assistant speaker, and / or other client computing device) that communicates with the robot over a network. For example, free-form natural language input can be based on a user's voice. For example, free-form natural language input can be generated based on processing of audio data (e.g., speech-to-text processing), where the audio data is generated based on signals received from (one or more) microphones of the robot and / or a separate component. In some of those cases, processing of the audio data may optionally occur only after the user has explicitly requested such processing, such as by speaking a particular invocation phrase (e.g., “Hi, robot”) and / or activating a hardware button, selecting a graphical interface element, and / or other specific user interface input.

[0031] In some embodiments, a storage network is included in a neural network model. For example, a language branch of a neural network model may include a storage network for initially processing free-form natural language input. The storage network includes one or more storage layers, each storage layer including a plurality of storage units. The storage unit may be, for example, a long short-term memory ("LSTM") storage unit, a gated recurrent unit ("GRU"), or other storage unit. In many embodiments, the storage network may be a bidirectional storage network. Generating features using a storage network may capture long-term dependencies in natural language. In some embodiments, various features generated using a storage network may be used to generate a natural language embedding based on a natural language input. Such features include, for example, a final forward state, a final backward state, and / or a hidden state of a node of a bidirectional storage network after processing the natural language input. For example, and as described herein, a natural language embedding may be generated based on the final forward and backward states and based on an attention function (e.g., represented by a trained attention layer), the attention function being based on a hidden state and based on (one or more) visual embeddings (and / or based on (one or more) other embeddings of sensor data indicating the state of the robot). By utilizing the attention function based on (one or more) visual embeddings and / or (one or more) other sensor embeddings, the natural language embedding will change at each step in one or more steps of controlling the robot based on the corresponding instance of the natural language input. In other words, even if the natural language input remains unchanged and the output of the storage layer (processing the natural language input) remains unchanged during the process of controlling the robot based on the natural language input, the natural language embedding will change during the control due to the attention function based on (one or more) visual embeddings and / or (one or more) other sensor embeddings. Therefore, by training the attention layer (representing the attention function) as described herein, the natural language embedding can be biased at a given time step towards the (one or more) terms corresponding to the (one or more) objects and / or (one or more) actions represented by the corresponding visual data in that time step.

[0032] Embodiments of the neural network model disclosed herein can be considered as a policy network for controlling a robot. The observation space of the policy network includes natural language input and robot sensor input (e.g., visual and depth observation) from the robot's vantage point. The output of the policy at each step is the next motion primitive to be executed (e.g., rotate right, rotate right, move forward, and / or other motion primitives). The neural network model includes a language attention mechanism in its language branch that is conditioned on the robot's sensory observations. The language attention mechanism is able to keep track of the instructions of the natural language input and focus on different parts of the natural language input as the robot explores the environment. In addition, the language attention mechanism associates parts of the motion primitives, sensory observations, and natural language input with (one or more) rewards received during training, which enables generalization to new instructions.

[0033] Before turning to the drawings, a specific implementation is described in detail. In the specific implementation, it can be assumed that the robot is a point-mass with three degrees of freedom, navigating in a 2-dimensional grid overlaid on a 3-dimensional indoor environment. However, other implementations can be configured and trained for robots with more degrees of freedom, which can navigate in a grid of 3 or more dimensions. For example, other implementations can increase the action space to additionally capture (one or more) actions along additional degrees of freedom.

[0034] When training a neural network model (representing a policy) in a specific implementation, the task can be formulated as a partially observable Markov decision process (POMDP): tuple of are natural language instructions sampled from a set of user-provided directions to a goal. The location of the goal is unknown to the agent. is the visual input available to the agent, which consists of the robot at time step The image you see. This group of actions, , so that the robot can get into position or move forward one step. System dynamics, , is qualitative, and applies an action to the robot. The robot transitions to the next grid cell, or changes its orientation. Note that the agent does not know where it is in the environment.

[0035] award Reward the agent for reaching the landmarks (waypoints) mentioned in the instructions, with a strong reward (e.g. +1.0) if the waypoint is the final goal location, and a smaller reward (e.g. +0.05) for intermediate waypoints. The agent is rewarded only once for each waypoint in the instructions it reaches, and the scenario terminates when the agent reaches the last waypoint, or after a maximum number of steps.

[0036] The goal during training in a specific embodiment is to learn the action-value function approximated by a deep neural network and trained with DQN The network model allows approximating the action-value function directly from the language and visual input. To simplify the image processing task, a separate preprocessing step can parse the visual input To obtain semantic segmentation and depth map . Semantic Segmentation Each pixel can be assigned a popular semantic class id, for example. Each pixel may for example be assigned a real number corresponding to a distance from the robot.

[0037] The agent obtains the ground truth from its current point of view and , and runs it through a stack of convolutional layers, followed by fully connected layers. It obtains a fixed-length embedding vector from these and , the embedding vector encodes the visual information available to the agent.

[0038] Single-layer bidirectional GRU network utilizes state size and an initial state set to 0 to encode the natural language instruction using the following formula: ; ;; ; ; ;in are the final hidden states of the forward and backward GRU units, respectively, and is the concatenated output of the forward and backward units, corresponding to the embedding representation of each token conditioned on the entire sentence. To enable the agent to focus on different parts of the instruction based on the context, Add a feed-forward attention layer to it:

[0039]

[0040]

[0041]

[0042]

[0043] )

[0044]

[0045] Using a feed-forward attention layer , its As the condition, is the concatenated embedding of the visual and language inputs to obtain an unnormalized score for each token . Use the softmax function to normalize the unnormalized scores Normalize to get the attention score , the attention scores correspond to the relative importance of each token of the instruction at the current time step. The output vector The attention-weighted average of is passed through another feed-forward layer to obtain , which is the final encoding of the natural language instruction. Then from the cascade through the final feed-forward layer Estimate the Q function. During training, actions are sampled from the Q function using an epsilon-greedy strategy to collect experience, and gradient descent is used to update the Q network to minimize the Bellman error over multiple batches of transitions. After training the Q function, the learned , using the geedy strategy Bring the robot to command goals set out in .

[0046] Now turn to Figure 1 , showing an example environment in which the embodiments disclosed herein may be implemented. Figure 1 A neural network model 150 is included, which includes a visual branch 160, wherein the visual branch 160 includes a semantic branch 162 and a depth branch 166. The semantic branch 162 is used to process an instance of semantic visual data 161 to generate a corresponding semantic embedding 163.

[0047] The semantic visual data 161 may be, for example, an indication of one or more corresponding semantic identifiers (and optionally corresponding probabilities of including multiple semantic identifiers for the corresponding groups) for each of a plurality of corresponding groups of pixels or voxels. For example, the semantic visual data 161 may be a corresponding semantic label (e.g., classification) for each of a plurality of pixels. In such an example, each pixel or voxel group includes only a single corresponding pixel, and the semantic visual data has only a single corresponding semantic label. The semantic labels may include a variety of classification labels, such as labels such as: "door", "floor", "sofa", "chair", "plant", "wall", "light switch", "unknown", etc. It should be understood that in the semantic visual data, directly human-interpretable natural language labels themselves may be used, or alternatively non-directly human-interpretable labels may be used (e.g., "85A1" or an identifier may be used instead of "door"). In other embodiments, each group may optionally include multiple semantic identifiers, optionally with probabilities of each of the multiple semantic identifiers. For example, a group of pixels capturing green plants may include a "plant" label and a "green" label. As another example, the semantic visual data may be a corresponding semantic label for each of a plurality of multi-pixel groups, and in such an example, the pixel or voxel groups each include a plurality of corresponding pixels.

[0048] Semantic visual data 161 can be generated, for example, by processing the visual data using one or more classification networks (not shown) and assigning semantic labels to the semantic visual data 161 based on the output generated by the processing. As a non-limiting example, the visual data can be processed using a trained faster-R-RNN model to generate a plurality of bounding boxes and corresponding classifications. The bounding boxes and classifications can then be used to assign corresponding classification labels to corresponding pixels (surrounded by corresponding bounding boxes). The visual data processed to generate semantic visual data can include, for example, 2D RGB images, 2.5D RGBD images, and / or 3D point clouds.

[0049] The depth branch 166 is used to process an instance of the depth visual data 165 to generate a corresponding depth embedding 167. The depth visual data 165 includes a corresponding depth value for each of a plurality of corresponding groups of pixels or voxels. For example, the depth visual data may be a corresponding depth value for each of a plurality of pixels. In such an example, each pixel or voxel group includes only a single corresponding pixel. In other embodiments, the depth visual data may be a corresponding depth value for each of a plurality of multi-pixel groups, and in such an example, each pixel or voxel group includes a plurality of corresponding pixels. The depth visual data 165 may be generated, for example, using a depth value from a 2.5D RGBD image and / or a 3D point cloud. The resolutions of the depth visual data 165 and the semantic visual data 161 may be the same or may be different. Moreover, the depth visual data 165 and the semantic visual data 161 may each be generated based on visual data from a single visual component, or may be generated based on visual data from different components. For example, the depth visual data 165 is generated based on data from a first sensor component and the semantic visual data 161 is generated based on data from a second sensor component.

[0050] The neural network model 150 also includes a language branch 170. The language branch 170 includes a storage network 172, an attention layer 176, and (one or more) additional layers 178. The storage network 172 can be, for example, a bidirectional storage network. The storage network 172 can be used to process the natural language input 171 token by token to generate a bidirectional storage network output 173. After processing the natural language input, the storage network 172 has one or more hidden states 174. For example, in the case of the bidirectional storage network 172, the (one or more) hidden states 174 may include a forward hidden state and a backward hidden state.

[0051] The hidden state(s) 174 and the semantic embedding 163 and the deep embedding 167 are used as the conditioning vector 175 of the attention layer 176. For example, they can be cascaded and used as the conditioning vector 175. Thus, the bidirectional storage network output 173 is further processed using the attention layer 176 and using the hidden state(s) 174 and the embeddings 163 and 167 as values ​​for conditioning the attention. Processing the bidirectional network output 173 using the attention layer 176 produces the attention weighted output 177.

[0052] Note that during a given scenario of attempting to perform a task based on natural language input, the bidirectional memory network output 173 will remain the same and the hidden state 174 will remain the same. However, as new instances of semantic visual data 161 and deep visual data 165 are processed, the attention weighted output 177 will change throughout the scenario. This is due to the inclusion of an adjustment vector 175 based on the values ​​of the semantic embedding 163 and the deep embedding 167, which will change as new instances of semantic visual data 161 and deep visual data 165 are processed. Therefore, the bidirectional memory network output 173 and hidden state 174 that remain the same can be reused in subsequent iterations of the scenario without having to be regenerated, thereby saving various resources. The use of attention layers and adjustment vectors effectively and efficiently adapts the bidirectional memory network output to the current visual data and / or other sensor data.

[0053] The attention-weighted output 177 is optionally further processed using one or more additional layers 178 (e.g., feed-forward layers) of the language branch 170 to generate a natural language embedding 179.

[0054] At each step during the scene, (one or more) action prediction layers 180 process the current semantic embedding 163, the current depth embedding 167, and the current natural language embedding to generate an action prediction output 181. The action prediction output 181 can indicate the motion primitives of the robot, and the robot can be controlled to implement the motion primitives. For example, the action prediction output in a step can be the corresponding probability of each of a plurality of motion primitives, and the highest probability motion primitive can be implemented in the step. Motion primitives can include, for example, "move forward", "move backward", "turn right", "turn left", "move up", "move down" and / or (one or more) other motion primitives (including motion primitives of more or less granularity).

[0055] exist Figure 1 An example robot 100 is shown in FIG. 1 . The robot 100 is an example robot that may incorporate a neural network model 150 (e.g., in a memory) and may include (one or more) processors for processing data using the neural network model 150 to generate (one or more) actions for controlling the robot 100 based on natural language input and visual data (and / or other sensor data). The robot 400 (described below) Figure 4 ) is yet another example robot that may incorporate the neural network model 150 and may include (one or more) processors for processing data using the neural network model 150.

[0056] The robot 100 includes a robot arm 105 having a gripping end effector 106 in the form of a gripper having two opposing actuatable members. The robot 100 also includes a base 109 having wheels 107A and 107B disposed on opposite sides thereof to enable movement of the robot 100. The base 109 may include, for example, one or more motors for driving the corresponding wheels 107A and 107B to achieve a desired direction, speed and / or acceleration of movement of the robot 100.

[0057] The robot 100 also includes a vision component 108. The vision component 108A includes one or more vision sensors and can be, for example, a stereo camera, a single-lens camera, or a laser scanner. The visual data described herein can be generated based on the output from the (one or more) vision sensors of the vision component 108 and / or from (one or more) other vision components (not shown).

[0058] As described herein, the robot 100 can operate autonomously at least part of the time and control its actuators when performing various actions. For example, when performing various actions, one or more processors of the robot 100 can provide control commands to actuators associated with the wheels 107A and / or 107B, the robot arm 105, and / or the end effector 106. Moreover, in various cases, the control commands provided at a given instance can be generated based at least in part on the actions determined using the neural network model as described herein (e.g., actions that implement motion primitives).

[0059] exist Figure 1 Also shown is a training engine 115, which can use deep Q-learning to train as described herein (e.g., in Figure 3 In some embodiments, the training engine 115 may utilize a simulator 117 that simulates various environments and a robotic agent that functions within the various environments in performing some or all of the training. The training engine 115 may operate the virtual agent using the simulator 117 in exploring various environments using the neural network model 150 and natural language input and in updating the neural network model based on rewards determined from output from the simulator 117.

[0060] The simulator 117 may be implemented by one or more computer systems and used to simulate an environment including (one or more) corresponding environmental objects and to simulate a robot operating in the simulated environment. Various simulators may be utilized, such as physics engines that simulate collision detection, soft and rigid body dynamics, etc. A non-limiting example of such a simulator is the BULLET physics engine.

[0061] When operating in the simulated environment during training of the neural network model 150, the simulator 115 can utilize various natural language inputs, such as natural language inputs generated by humans and / or automatically based on room labels in the simulated environment and / or object labels in the simulated environment. In addition, during the scene of navigating the simulated robot in the simulated environment based on the corresponding natural language input, a reward can be generated (or not generated) based on the goal corresponding to the natural language input and optionally based on (one or more) waypoints towards the goal. For example, assume a natural language input of "leave the room, turn left, and go through the hall and enter the door in front of you". For such input, a reward of +1.0 can be generated in response to the simulated robot agent "enter the door in front of you", and (one or more) optional intermediate rewards can be generated for one or more waypoints (such as "hall"). In addition, during the scene, simulated visual data can be generated from the perspective of the simulated robot agent and used to generate instances of semantic visual data and / or depth data. Alternatively, semantic visual data and / or depth data can be generated directly (e.g., using ground truth values ​​directly available from the simulator 117).

[0062] Figure 2 is an example method 200 for controlling a robot based on natural language input from a user and based on visual data from (one or more) visual components of the robot. For convenience, some operations of the method 200 are described with reference to a system that performs the operations. The system may include various components of the robot, such as the robot 100 ( Figure 1 ) and / or Robot 400 ( Figure 4 ) of one or more processors. In addition, although the operations of method 200 are shown in a specific order, this is not meant to be limiting. One or more operations may be performed in parallel, reordered, omitted, or added.

[0063] At block 202, the system receives free-form natural language (NL) input. For example, the free-form natural language input may be text that has been converted from audio data that captures a user's voice command using speech-to-text processing. As an example, a command may be "go through the door, turn left, go to the end of the hallway, and go through the door on the right."

[0064] At block 204, the system generates a storage network output based on processing the natural language input using the storage layer(s) of the language branch of the model (eg, neural network model). The storage layer(s) may be, for example, a bidirectional storage layer.

[0065] At block 206, the system receives an instance of visual data. For example, the instance of visual data may include an instance of semantic visual data and an instance of deep visual data.

[0066] At block 208, the system generates (one or more) visual embeddings based on processing the instance of the visual data using at least one visual branch of the model. For example, the system can generate a semantic embedding based on processing the semantic visual data using the semantic vision branch, and can generate a deep embedding model based on processing the depth data using the deep vision branch.

[0067] At block 210, the system generates an attention weighted output based on using the attention layer and processing the storage network output using the visual embedding(s) and / or the storage network hidden state(s) in conditioning. For example, the system can generate a conditioning vector that is a concatenation (or other combination) of the visual embedding(s) and the hidden state(s) of the storage network (e.g., forward and / or backward hidden states).

[0068] At block 212, the system generates an NL embedding based on further processing of the attention-weighted output. For example, the attention-weighted output can be processed using (one or more) additional feed-forward layers of the model to generate an NL embedding.

[0069] At block 214, the system generates an action prediction output based on processing the NL embedding and the visual embedding(s) using the action prediction layer(s) of the model. In some implementations, the action prediction output indicates which of a plurality of motion primitives should be implemented.

[0070] At block 216, the system controls the robot's actuator(s) to cause the robot to perform the action indicated by the action prediction output. For example, the system may control the actuator(s) to control the motion primitive indicated by the action prediction output.

[0071] At block 218, the system determines whether the goal has been reached (indicated by the action prediction output), or whether other condition(s) have not been met (e.g., the maximum number of iterations has not been performed). The system can optionally determine that the goal has been reached based on the most recent iteration indicating that no action should be taken (in which case block 216 will be skipped in that iteration). If not, the system proceeds back to block 206, receives another instance of visual data (captured after block 216), and then performs another iteration of blocks 208, 210, 212, 214, and 216 based on the additional instance of visual data.

[0072] If at block 218 , the system determines that the destination has been reached, or if other condition(s) have been met, the system proceeds to block 220 , and the method 200 ends.

[0073] Figure 3is an example method for training a neural network model for controlling a robot based on natural language input from a user and based on visual data. For convenience, some operations of method 300 are described with reference to a system that performs the operations. The system may include various components, such as one or more processors of a computer system. In addition, although the operations of method 300 are shown in a specific order, this is not meant to be limiting. One or more operations may be performed in parallel, reordered, omitted, or added.

[0074] At box 302, the system obtains free-form natural language (NL) input, a corresponding target, and optional waypoint(s) to be encountered when reaching the target. For example, the free-form NL input can be manually or automatically generated based on a label of the corresponding environment to be interacted with and in view of the NL input as human-interpretable instructions for performing the target. As an example: the NL input can be "go through the door, turn left, go to the end of the corridor, and then go through the door on the right"; the target can be "go through the door on the right"; and the waypoint(s) can include "corridor" and / or (one or more) other waypoints.

[0075] At block 304, the system generates a storage network output based on processing the natural language input using the storage layer(s) of the language branch of the model (eg, neural network model). The storage layer(s) may be, for example, a bidirectional storage layer.

[0076] At block 306, the system receives an instance of visual data. For example, the instance of visual data may include an instance of semantic visual data and an instance of deep visual data. In various implementations, the instance of visual data is based on data from a simulator utilized during training.

[0077] At block 308, the system generates (one or more) visual embeddings based on processing the instance of the visual data using at least one visual branch of the model. For example, the system can generate a semantic embedding based on processing the semantic visual data using the semantic vision branch, and can generate a deep embedding model based on processing the depth data using the deep vision branch.

[0078] At block 310, the system generates an attention weighted output based on using the attention layer and processing the storage network output using the visual embedding(s) and / or the storage network hidden state(s) in conditioning. For example, the system can generate a conditioning vector that is a concatenation (or other combination) of the visual embedding(s) and the hidden state(s) of the storage network (e.g., forward and / or backward hidden states).

[0079] At block 312, the system generates an NL embedding based on further processing of the attention-weighted output. For example, the attention-weighted output can be processed using (one or more) additional feed-forward layers of the model to generate an NL embedding.

[0080] At block 314, the system generates an action prediction output based on processing the NL embedding and the visual embedding(s) using the action prediction layer(s) of the model. In some implementations, the action prediction output indicates which of a plurality of motion primitives should be implemented.

[0081] At box 316, the system determines whether the waypoint has been reached. If so, the system proceeds to box 320 and applies the waypoint reward, and then proceeds to box 328. If not, the system proceeds to box 322.

[0082] At box 322, the system determines whether the target has been reached. If so, the system proceeds to box 324 and applies the target reward, and then proceeds to box 326. If not, the system proceeds to box 328.

[0083] At box 328, the system determines whether the maximum number of attempts has been performed in the current scene. If not, the system returns to box 306, receives another instance of visual data, and then performs another iteration of boxes 308, 310, 312 and 314 based on the additional instance of visual data.

[0084] If box 326 is reached (via 322 or 328), the system uses the applied reward(s) of the scene when updating the model. For example, the system may update the action prediction layer(s) and / or the attention layer(s) based on the reward(s). Optionally, other layers(s) may not be updated (i.e., remain fixed), such as the storage network layer(s) and / or the visual branch(s).

[0085] After reaching block 326, the system can proceed back to block 302 and perform another iteration of method 300. Through multiple iterations of method 300 (optionally in parallel), exploration can be performed and the model can be iteratively updated. Once sufficient model training has been completed, the model can be deployed on one or more real physical robots for further training and / or use. Sufficient training can be determined based on, for example, the number of training iterations, simulation and / or actual testing of the trained model, and / or other criteria / conditions.

[0086] Figure 4An example architecture of a robot 400 is shown. Robot 400 is another example of a robot that can be controlled according to the technology disclosed herein. Robot 400 can take various forms, including but not limited to telepresence robots, humanoid robots, animal robots, insect robots, aquatic robots, wheeled robots, submersible robots, unmanned aerial vehicles ("UAV") robots, etc. In various embodiments, robot 400 may include one or more processors 402, which can implement all and / or aspects of the various methods disclosed herein. The robot's (one or more) processors 402 can take various forms, such as one or more CPUs, one or more GPUs, one or more field programmable gate arrays ("FPGAs"), and / or one or more application-specific integrated circuits ("ASICs"). In some embodiments, (one or more) processors 402 can be operably connected to memory 403. Memory 403 can take various forms, such as random access memory ("RAM"), dynamic RAM ("DRAM"), read-only memory ("ROM"), magnetoresistive RAM ("MRAM"), resistive RAM ("RRAM"), NAND flash memory, etc.

[0087] In some embodiments, the robot's processor(s) 402 may communicate with one or more actuators 404, for example, via one or more buses. 1-N , at least one end effector 406 and / or one or more sensors 408 1-M The robot 400 may have multiple degrees of freedom, and the actuator 404 1-N Each of the controllers 404 can control the actuation of the robot 400 in one or more of the degrees of freedom in response to control commands. The control commands are generated by one or more processors 402 and provided to the actuators 404. 1-N (e.g., via one or more buses) to control the robot 400. As described herein, various control commands can be generated based on the motion prediction output generated using the neural network model. As used herein, "actuator" includes a mechanical or electrical device (e.g., a motor) that produces motion, in addition to any driver that can be associated with the actuator and converts a received control command into one or more signals for driving the actuator. Therefore, providing a control command to the actuator can include providing the control command to the driver, which converts the control command into an appropriate signal for driving the electrical or mechanical device to produce the desired motion.

[0088] As used herein, "end effector" may refer to various tools that may be operated by a robot in completing various tasks. For example, some robots may be equipped with an end effector that takes the form of a claw having two opposing "fingers" or "fingers". This claw is a type of "gripper" known as an "impact" gripper. Other types of grippers may include, but are not limited to, "invasive" (e.g., using a pin, needle, etc. to physically penetrate an object), "retractive" (e.g., using suction or vacuum to pick up an object), or "continuous" (e.g., using surface tension, freezing, or adhesives to pick up an object). More generally, other types of end effectors may include, but are not limited to, drills, brushes, force-torque sensors, cutting tools, deburring tools, welding torches, containers, pallets, and the like.

[0089] Robot Sensor 4084 1-M Various forms may be employed, including but not limited to vision components (e.g., laser scanners, stereo cameras, single-lens cameras), force sensors, pressure sensors, pressure wave sensors (e.g., microphones), proximity sensors (also referred to as "range sensors"), torque sensors, barcode readers, radio frequency identification ("RFID") readers, accelerometers, gyroscopes, compasses, position sensors (e.g., odometers, global positioning systems), speedometers, edge detectors, etc. Some of the robot's sensors may be integral to the robot (e.g., fixedly or removably coupled to the robot). Some other sensors of the robot may optionally be located external to the robot but may communicate directly or indirectly with the robot.

[0090] The vision component of the robot may include one or more vision sensors, and may be, for example, a stereo camera, a single-lens camera, or a laser scanner. The visual data described herein may be generated based on the output of (one or more) vision sensors from the vision component. For example, the output may be the raw output from (one or more) vision sensors, or the processed output. In some embodiments, the stereo camera includes two or more sensors (e.g., a charge coupled device (CCD)), each sensor being at a different vantage point. Visual data such as two-dimensional and a half ("2.5D") (2D with depth) images may be generated based on sensor data generated by two sensors in a given instance, wherein each of the pixels of the 2.5D image defines the X, Y, and Z coordinates of the surface of the corresponding object, as well as optional color values ​​(e.g., R, G, B values) and / or other parameters for the coordinates of the surface. In some other embodiments, the stereo camera may include only a single sensor and one or more reflectors, which are used to effectively capture sensor data from two different vantage points. The single-lens camera may include a single sensor and capture two-dimensional ("2D") visual data. The laser scanner includes one or more lasers that emit light and one or more sensors that generate visual sensor data related to the reflection of the emitted light. The visual data generated based on the sensor output from the laser scanner can be, for example, 2.5D point cloud data. The laser scanner can be, for example, a time-of-flight laser scanner or a triangulation-based laser scanner, and can include a position sensitive detector (PSD) or other optical position sensor. In some embodiments, the laser scanner can be a LIDAR component.

[0091] Although some or all aspects of the methods disclosed herein can be performed by one or more processors of the robot, some or all aspects can optionally be performed by one or more computing devices separate from the robot. The computing device typically includes at least one processor that communicates with a plurality of peripheral devices via a bus subsystem. These peripheral devices may include a storage subsystem (including, for example, a memory subsystem and a file storage subsystem), a user interface output device, a user interface input device, and a network interface subsystem. These input and output devices allow user interaction with the computing device. The network interface subsystem provides an interface to an external component (e.g., a robot) directly or indirectly (e.g., via one or more networks).

[0092] User interface input devices may include keyboards, pointing devices (such as mice, tracking balls, touch pads, or graphics tablets), scanners, touch screens incorporated into displays, audio input devices (such as voice recognition systems, microphones, and / or other types of input devices). In general, the use of the term "input device" is intended to include all possible types of devices and methods for inputting information into a computing device, a robot, or onto a communication network. User interface output devices may include a display subsystem, a printer, or a non-visual display (such as an audio output device). The display subsystem may include a flat panel device, a projection device, or some other structure for creating a visible image. In general, the use of the term "output device" is intended to include all possible types of devices and methods for outputting information from a computing device to a user or to another machine or computing device.

[0093] The storage subsystem stores programs and data structures that provide some or all of the functions of the modules described herein. For example, the storage subsystem may include logic for executing selected aspects of the methods described herein. The memory used in the storage subsystem may include multiple memories, including: a main random access memory (RAM) for storing instructions and data during program execution and a read-only memory (ROM) in which fixed instructions are stored. The file storage subsystem may provide persistent storage for program and data files, and may include a hard drive, a CD-ROM drive, an optical drive, or a removable media box. Modules that implement the functions of certain embodiments may be stored in the storage subsystem or in other machines accessible to (one or more) processors through the file storage subsystem. The bus subsystem provides a structure that allows the various components and subsystems of the computing device to communicate with each other as expected. The computing device may be of various types, including workstations, servers, computing clusters, blade servers, server farms, etc.

Claims

1. A method implemented by one or more processors, the method comprising: generating semantic visual data, wherein generating the semantic visual data is based on processing instances of the robotic visual data using one or more neural networks, wherein the robot vision data is generated based on output from a vision component of the robot, and wherein the semantic visual data comprises natural language labels for objects captured in at least some of the instances of the robot visual data, and receiving free-form natural language input generated based on user interface input provided by a user via one or more user interface input devices; generating an action prediction output indicative of a robotic action to be performed based on processing both the natural language labels of the objects and the free-form natural language input using an additional neural network; as well as One or more actuators of a robot are controlled based on the motion prediction output, wherein controlling the one or more actuators of the robot causes the robot to perform the robot motion indicated by the motion prediction output.

2. The method of claim 1, wherein generating the action prediction output further comprises: The additional visual data is processed using the additional neural network along with natural language labels for the objects and processing of free-form natural language input.

3. The method of claim 2, wherein the additional vision data comes from an additional vision sensor of an additional vision component of the robot.

4. The method of claim 3, wherein the additional vision component is a lidar component and the vision component is a stereo camera.

5. The method of claim 1, wherein the natural language labels of the objects are directly human interpretable.

6. The method of claim 5, wherein the free-form natural language input is generated based on speech provided by the user and wherein the one or more user interface input devices include a microphone of the robot.

7. The method of claim 1, wherein the additional neural network comprises one or more attention layers.

8. The method of claim 1, wherein the motion prediction output indicates one or more motion primitives of the robot.

9. The method of claim 1, wherein the free-form natural language input is generated based on speech provided by the user and wherein the one or more user interface input devices include a microphone of the robot.

10. A robot comprising: Visual components; Actuator; a memory for storing instructions; One or more processors operable to execute the instructions to: Generate semantic visual data based on processing instances of robot vision data using one or more neural networks, wherein the robot vision data is generated based on output from the vision component, and wherein the semantic visual data comprises natural language labels for objects captured in at least some of the instances of the robot visual data, and receiving free-form natural language input generated based on user interface input provided by a user via one or more user interface input devices; generating an action prediction output indicative of a robotic action to be performed based on processing both the natural language labels of the objects and the free-form natural language input using an additional neural network; as well as One or more of the actuators are controlled based on the motion prediction output, causing the robot to perform the robot motion indicated by the motion prediction output.

11. The robot of claim 10, wherein in generating the action prediction output, one or more processors are further configured to: process additional visual data using the additional neural network in conjunction with processing natural language labels of the object and free-form natural language input.

12. The robot of claim 11, wherein the additional vision data comes from an additional vision sensor of an additional vision component of the robot.

13. The robot of claim 12, wherein the additional vision component is a lidar component and the vision component is a stereo camera.

14. The robot of claim 10, wherein the natural language label of the object is directly human interpretable.

15. The robot of claim 14, wherein the additional neural network comprises one or more attention layers.

16. The robot of claim 15, wherein the free-form natural language input is generated based on speech provided by the user and wherein the one or more user interface input devices comprises a microphone of the robot.

17. The robot of claim 10, wherein the additional neural network comprises one or more attention layers.

18. The robot of claim 10, wherein the motion prediction output indicates one or more motion primitives of the robot.

19. The robot of claim 10, wherein the free-form natural language input is generated based on speech provided by the user and wherein the one or more user interface input devices comprises a microphone of the robot.

Images