Conveyor network for determining robot action

Abstract

A conveyor network that determines robot actions based on sensor feedback can provide efficient autonomous movement for a robot. The conveyor network can exploit spatial symmetries and not require assumptions of object metrics to provide accurate object manipulation instructions. The machine learning model of the conveyor network can also allow for learning of various tasks with fewer training examples than other machine learning models. The machine learning model of the conveyor network can ingest observation data as input and can output actions in response to the processed observation data.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to robotic manipulation. More specifically, the present disclosure relates to determining robot actions using machine learning. BACKGROUND

[0002] The field of robotics includes the design, construction, operation, and application of robots, as well as computer systems for their control, sensory feedback, and information processing. In many cases, a robot’s manipulation or control plan can be formulated as inducing a sequence of spatial displacements: where the space being moved can contain one or more objects or end effectors (e.g., a robot being controlled).

[0003] Machine learning techniques have proven particularly advantageous for robot manipulation or other forms of robot control planning. For example, machine learning techniques, such as end-to-end models that map directly from pixels to robot actions, have the ability to learn complex manipulation skills. However, it is well known that existing models that map directly from pixels to robot actions require large amounts of data, and thus are difficult to apply in situations where little training data is available or it is difficult or expensive to obtain training data.

[0004] As an alternative approach, some machine learning methods for robot manipulation incorporate object-centric assumptions about objects within a scene (e.g., object keypoints, embeddings, or dense descriptors). This object-centric approach has been shown to improve sampling efficiency. However, these object representations often come with a burden of data collection, and still struggle with difficult scenes that have unseen classes of objects, occluded objects, highly deformable objects, or small objects in a pile. The large amount of data collection required can slow processing speed and can be a computational burden. Furthermore, the robustness of the model can be diminished by rigidity of the representation constraints. SUMMARY

[0005] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or can be apparent from the description, or can be learned through practice of the embodiments.

[0006] One example aspect of the present disclosure relates to a computer-implemented method for generating actions of a robot. The computer-implemented method for generating actions of a robot can include obtaining observation data, where the observation data includes data describing an environment. The computer-implemented method can include determining a first action of the robot based at least in part on the observation data and processing the observation data with a first embedding model to generate a first feature embedding. The computer-implemented method can include processing the observation data with a second embedding model to generate a second feature embedding, where the second embedding model conditions on the first action. The computer-implemented method can include determining a second action of the robot based at least in part on a comparison of the first feature embedding and the second feature embedding.

[0007] Another example aspect of the present disclosure relates to a computer system. The computer system can include one or more processors and one or more non-transitory computer-readable media collectively storing instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations can include obtaining observation data, where the observation data includes data describing an environment. The operations can include determining a first action of a robot and processing the observation data with a first embedding model to generate a first feature embedding. In some implementations, the operations can include processing the observation data with a second embedding model to generate a second feature embedding, where the second embedding model conditions on the first action. The operations can include determining a second action of the robot based at least in part on a comparison of the first feature embedding and the second feature embedding. The operations can include evaluating a loss function that compares one or both of the first action and the second action to one or both of a first ground truth action and a second ground truth action and modifying one or more values of one or more parameters of one or both of the first embedding model and the second embedding model based at least in part on the loss function.

[0008] Another example aspect of the present disclosure relates to a robotic device. The robotic device can include one or more non-transitory computer-readable media collectively storing instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations. In some implementations, the operations include obtaining observation data, where the observation data includes data describing an environment. The operations can include determining a first action of the robotic device and processing the observation data with a first embedding model to generate a first feature embedding. The operations can include processing the observation data with a second embedding model to generate a second feature embedding, where the second embedding model conditions on the first action. The operations can include determining a second action of the robotic device based at least in part on a comparison of the first feature embedding and the second feature embedding. The operations can also include performing, by the robotic device, the first action and the second action.

[0009] Other aspects of the disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0010] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and serve to explain the related principles. BRIEF DESCRIPTION OF DRAWINGS

[0011] The detailed discussion of embodiments below with reference to the accompanying drawings is presented to provide a thorough understanding of examples of the application, and is not intended to be complete. Other examples of the present application will be apparent to those of ordinary skill in the art in view of this discussion, from studying the drawings, or from studying the full disclosure of this application.

[0012] Figure 1A A block diagram depicting an example computing system in accordance with example embodiments of the present disclosure.

[0013] Figure 1B A block diagram depicting an example computing device in accordance with example embodiments of the present disclosure.

[0014] Figure 1C A block diagram depicting an example computing system in accordance with example embodiments of the present disclosure.

[0015] Figure 2 A block diagram depicting an example network process for determining actions of a robot in accordance with example embodiments of the present disclosure.

[0016] Figure 3 A block diagram depicting an example network process for training a system to determine actions of a robot in accordance with example embodiments of the present disclosure.

[0017] Figure 4 A block diagram depicting an example network process for determining actions of a robot in accordance with example embodiments of the present disclosure.

[0018] Figure 5 A flow diagram depicting an example method for determining actions of a robot in accordance with example embodiments of the present disclosure.

[0019] Figure 6 A flow diagram depicting an example method for determining actions of a robot in accordance with example embodiments of the present disclosure.

[0020] Figure 7 A flow diagram depicting an example method for training a system to determine actions of a robot in accordance with example embodiments of the present disclosure.

[0021] Reference numerals repeated between the several figures designate the same features in different embodiments. DETAILED DESCRIPTION

[0022] SUMMARY

[0023] Generally, the present disclosure describes systems and methods for determining actions of a robot based on observations of an environment, such as images of the environment. In particular, the systems and methods presented herein can utilize an end-to-end machine learning transport network that can learn to infer robot actions from observation data, such as visual input. In particular, the systems can utilize deep feature template matching to infer spatial displacements from visual input, which can parameterize robot actions. The presented model avoids reliance on assumptions of objectness (e.g., canonical poses, models, or keypoints), leverages spatial symmetries, and has several orders of magnitude higher sample efficiency than alternative solutions to example benchmarks: from stacking cones of blocks to assembling a set of devices with unseen objects; from manipulating deformable ropes to pushing small objects in a pile with closed-loop feedback. The presented systems and methods can represent complex multi-modal policy distributions and can generalize to multi-step sequential tasks as well as 6DoF tasks, such as 6DoF pick-and-place. Experiments on various simulated tasks demonstrate that the presented model learns faster and generalizes better than various end-to-end baselines, including policies that use ground truth object poses.

[0024] Accordingly, aspects of the present disclosure relate to an end-to-end model architecture that preserves the spatial structure of vision-based manipulation without object-centric assumptions (e.g., which can be referred to as a “transporter network”). The presented model and related systems provide improved learning efficiency over existing pixel-based approaches, without any of the restrictions or burdens of explicit object representations.

[0025] In particular, manipulation often involves rearranging things, which can be thought of as performing a sequence of spatial displacements: where the space being moved (e.g., transported) can include objects or end effectors. The presented approach can formulate the visual representation of manipulation as estimating these displacements. The transporter network can directly optimize for this by learning to focus on local regions and predict their target spatial displacements via deep feature template matching - then parameterize robot actions to perform the manipulation. This formulation enables high-level perceptual reasoning about which visual cues are important and how they should be rearranged in the scene, the distribution of which can be learned from demonstrations.

[0026] Example transporter networks can also preserve the 3D spatial structure of visual input. For example, previous end-to-end models often use convolutional architectures that take in raw images, where valuable spatial information can be lost due to perspective distortion. Example implementations of the presented approach use 3D information to project visual data onto a spatially consistent representation, which leverages the equivariance of the inductive bias present in data symmetries for more efficient learning.

[0027] On experiments, the example embodiments of the proposed conveyor networks demonstrate superior sample efficiency on several tabletop manipulation tasks that involve changing the state of the robot’s environment in a purposeful way: from stacking cones of blocks to assembling a set of devices with unseen objects; from manipulating deformable ropes to pushing small objects in a pile with closed-loop feedback. The conveyor networks are also excellent at modeling multi-modal spatial action distributions and, depending on the construction, generalize both rotation and translation of objects. They do not require any prior knowledge of the objects to be manipulated, but operate based on information contained in partial visual data (e.g., RGB-D data) from a few demonstrations, and are able to generalize to new objects and configurations, and for some tasks, to one-shot learning from a single demonstration.

[0028] Accordingly, example aspects of the present disclosure utilize a new perspective on the role of spatial structure and its capabilities to improve end-to-end learning of vision-based manipulation. Example model architectures are provided that learn to focus on local regions and predict their spatial displacement while preserving the spatial structure of the visual input.

[0029] More specifically, example systems and methods for performing robot control can first obtain observation data. The observation data can include data describing an environment. The observation data can be image data (e.g., RGB data, etc.). Additionally or alternatively, the observation data can be light detection and ranging (LiDAR) point cloud data.

[0030] As noted above, some example embodiments can pre-process the observation to ensure spatial consistency. For example, the robot control system can un-project raw observation data into three-dimensional space and then render the un-projected data into an orthographic representation of the environment to generate spatially consistent observation data.

[0031] The robot control system can process the observation data to determine a first action for the robot. For example, determining the first action can include generating a first probability distribution of the first action based on the observation data by using a first machine-learned action value model (e.g., that is part of a larger conveyor network). The determination of the first action can also involve selecting a maximum value of the first probability distribution as the first action. The first action can be a change in a robot pose and / or contact with an object. The first probability distribution can include one or more points indicative of respective probabilities that the one or more points can be a starting location for manipulation of an object.

[0032] The robot control system can also process the observation data with a first embedding model to generate a first feature embedding. For example, the first embedding model can be part of a larger conveyor network. The system can also process the observation data with a second embedding model to generate a second feature embedding (again, the second embedding model can be part of the larger conveyor network).

[0033] In some embodiments, the second embedding model conditions on the first action. Specifically, in some implementations, processing the observation data with the second embedding model can include rotating and / or cropping (e.g., in a local area around a location associated with the first action) the observation data to generate a plurality of rotated-cropped data samples. In such implementations, processing the observation data with the second embedding model can include processing the plurality of rotated-cropped data samples with the second embedding model to respectively generate a plurality of second feature embeddings. In some implementations, each of the first embedding model and the second embedding model can be or include a fully convolutional neural network.

[0034] The robot control system can then determine a second action for the robot. In some implementations, the determination of the second action can depend in part on a comparison of the first feature embedding and the second feature embedding (or multiple second feature embeddings).

[0035] As one example, the comparison can be or include a convolution of the two feature embeddings. The convolution can include convolving each of the second feature embeddings on the first feature embedding to generate a respective second probability distribution.

[0036] As another example, in more complex implementations, rather than simply convolving the first feature embedding and the second feature embedding (or multiple second feature embeddings), the determination of the second action can include processing the first feature embedding and the second feature embedding (or multiple second feature embeddings) with a second machine-learned value-action model (e.g., which can also be part of the larger conveyor network). The second machine-learned value-action model can output one or more second probability distributions based on the feature embedding (or multiple feature embeddings).

[0037] The determination of the second action can include selecting a maximum of the second probability distribution (or multiple second probability distributions) as the second action. In some cases, the second action can be a second pose of the robot and / or releasing contact with the object. Thus, the second probability distribution (or multiple second probability distributions) can include one or more points indicating respective probabilities that the one or more points can be a stopping location for manipulation of the object.

[0038] In some implementations, the robot control system can also obtain a set of target image data, process the target image data with a third embedding model to generate a third feature embedding, and determine the second action and / or the first action based on the feature embeddings from all three embedding models.

[0039] Additional aspects of the present disclosure can include training steps for training a conveyor network (e.g., its constituent models) for action determinations. The training steps can include evaluating a loss function that compares the first and second actions to the first and second ground truth actions. In response to the evaluating step, values of parameters of the embedding model and / or action value model(s) can be modified based on the loss function (e.g., in an end-to-end manner).

[0040] In some implementations, evaluating the loss function can include generating a first one-hot ground truth pixel map based on the first ground truth action and a second one-hot ground truth pixel map based on the second ground truth action. The first one-hot ground truth pixel map can include a first plurality of pixels having respective binary values indicative of a ground truth location of the first action in the environment. Further, the second one-hot ground truth pixel map can include a second plurality of pixels having respective binary values indicative of a ground truth location of the second action in the environment. The first and second actions (e.g., their respective locations) determined by the conveyor network can then be compared to the first and second one-hot ground truth pixel maps, respectively.

[0041] In some implementations, the loss function can be or include a cross-entropy loss. The cross-entropy loss can compare the first and second actions to the first and second one-hot ground truth pixel maps, respectively. In particular, the comparisons can be evaluated on a pixel-by-pixel basis.

[0042] Accordingly, the systems and methods provided herein can be implemented as a simple end-to-end model architecture that preserves the spatial structure of vision-based manipulation without object-centric assumptions. The vision-based manipulation can include performing a series of spatial displacements. The manipulation can move an object or an end effector. The system of networks can involve a system of visual feedback to determine the spatial displacements and the order of the spatial displacements. The system can learn to focus on local regions and predict target spatial displacements through deep feature template matching. The deep feature template matching can use observation data to determine a first pose and a second pose based on processed observation data. The deep feature template matching can then enable determination of a robot action for performing the manipulation.

[0043] The systems and methods of the present disclosure provide a number of technical effects and advantages. As one example, the systems and methods can provide more efficient action determination for robots, and thus, improve the productivity of the robots. In addition, the systems and methods can reduce the burden of data collection and can reduce the computational energy required for computation. Because the systems and methods can make action determinations without requiring object keypoints, embeddings, or dense descriptors, the systems or methods can reduce the data ingestion required. This reduction can reduce the computational energy required by the systems.

[0044] Another technical advantage of the systems and methods of the present disclosure is the ability to provide better performance with only a small number of training examples or demonstrations. The systems and methods can be trained with fewer examples than previous models, which allows the systems and methods to learn more quickly than other models. Faster learning can result in savings of computational resources. In addition, training a model on fewer training examples can actually result in less use of computational resources (e.g., processor memory, etc.), so by enabling a model to learn using fewer training examples, the systems and methods save computational resources.

[0045] In addition, the systems and methods can be iteratively used to complete a series of tasks. The learned model can effectively determine actions to adjust to multi-modal tasks. The ability to effectively handle multi-modal tasks can further improve the productivity of the robots.

[0046] Example embodiments of the present disclosure will now be discussed in further detail with reference to the drawings.

[0047] Example devices and systems

[0048] Figure 1A A block diagram of an example computing system 100 that performs action determination based on sensor feedback in accordance with example embodiments of the present disclosure is depicted. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 communicably coupled over a network 180.

[0049] The user computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a notebook or desktop computer), a mobile computing device (e.g., a smartphone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0050] The user computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 can store data 116 and instructions 118 for execution by the processor(s) 112, wherein the instructions 118 cause the user computing device 102 to perform operations.

[0051] In some implementations, the user computing device 102 can store or include one or more action determination models 120. For example, the action determination models 120 can be or can otherwise include various machine learning models, such as neural networks (e.g., deep neural networks) or other types of machine learning models, including non-linear models and / or linear models. The neural networks can include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Example action determination models 120 will be discussed with reference to Figure 2 、 Figure 4 、 Figure 5 and Figure 6 .

[0052] In some implementations, the one or more action determination models 120 can be received from the server computing system 130 via the network 180, stored in the memory 114 of the user computing device, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing device 102 can implement multiple parallel instances of a single action determination model 120.

[0053] More specifically, the action determination models can ingest sensor data and output actions for the robot. The action determination models can be used to determine actions for the robot, including but not limited to pick-and-place actions, push actions, drag actions, and sequence actions.

[0054] Additionally or alternatively, one or more action determination models 140 can be included in a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship, or otherwise stored in and implemented by the server computing system 130. For example, the action determination models 140 can be implemented by the server computing system 140 as part of a web service (e.g., a robotic manipulation service). Thus, one or more models 120 can be stored and implemented at the user computing device 102, and / or one or more models 140 can be stored and implemented at the server computing system 130.

[0055] The user computing device 102 can also include one or more user input components 122 that receive user input. For example, the user input components 122 can be touch-sensitive components (e.g., a touch-sensitive display screen or a touchpad) that are sensitive to touch by a user input object (e.g., a finger or a stylus). The touch-sensitive components can be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0056] The server computing system 130 includes one or more processors 132 and a memory 134. The one or more processors 132 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 134 can include one or more non-transitory computer-readable media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 can store data 136 and instructions 138, where the instructions 138 are executable by the processor 132 to cause the server computing system 130 to perform operations.

[0057] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. In cases where the server computing system 130 includes multiple server computing devices, such server computing devices can operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0058] As described above, the server computing system 130 can store or otherwise include one or more machine learning action determination models 140. For example, the models 140 can be or can otherwise include various machine learning models. Example machine learning models include neural networks or other multi-layer non-linear models. Example neural networks include feed-forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Example models 140 will be described with reference to Figure 2 、 Figure 4 、 Figure 5 andFigure 6 to be discussed.

[0059] The user computing device 102 and / or the server computing system 130 can train the models 120 and / or 140 via interaction with a training computing system 150 communicably coupled over the network 180. The training computing system 150 can be separate from the server computing system 130, or can be part of the server computing system 130.

[0060] The training computing system 150 includes one or more processors 152 and a memory 154. The one or more processors 152 can be any appropriate processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158 that are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0061] The training computing system 150 can include a model trainer 160 that trains the machine learning models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130 using various training or learning techniques, such as, for example, backpropagation of errors. For example, a loss function can be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on gradients of the loss function). Various loss functions can be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques can be used to iteratively update the parameters over several training iterations.

[0062] In some implementations, performing backpropagation of errors can include performing truncated backpropagation through time. The model trainer 160 can perform several generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization capabilities of the models being trained.

[0063] In particular, the model trainer 160 can train the action determination model 120 and / or 140 based on a set of training data 162. The training data 162 can include, for example, a set of ground truth actions. The training method can include parameterization based on a comparison of the determined actions to the ground truth actions. The ground truth data can be a one-hot ground truth pixel map.

[0064] In some implementations, the training examples can be provided by the user computing device 102 if the user has consented. Thus, in such implementations, the model 120 provided to the user computing device 102 can be trained by the training computing system 150 on user-specific data received from the user computing device 102. In some cases, this process can be referred to as personalizing the model.

[0065] The model trainer 160 includes computer logic in the form of software, hardware, or a combination of software and hardware that provides desired functionality to the computing system 100. The model trainer 160 can be implemented with hardware alone, software alone, or a combination of software and hardware. For example, in some implementations, the model trainer 160 includes program files stored on a storage device that are loaded and executed by one or more processors. In other implementations, the model trainer 160 includes one or more sets of computer-executable instructions that are stored in tangible computer-readable storage media such as RAM hard disk, or optical or magnetic media.

[0066] The network 180 can be any type of communications network, such as a local area network (e.g., intranet), wide area network (e.g., Internet), or some combination thereof, and can include any number of wired or wireless links. In general, communications over the network 180 can be carried out using a variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL) via any type of wired and / or wireless connection.

[0067] Figure 1A An example computing system that can be used to implement the present disclosure is shown. Other computing systems can also be used. For example, in some implementations, the user computing device 102 can include the model trainer 160 and the training data set 162. In such implementations, the model 120 can be trained and used locally at the user computing device 102. In some such implementations, the user computing device 102 can implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0068] Figure 1B A block diagram of an example computing device 10 that is executed in accordance with example embodiments of the present disclosure is depicted. The computing device 10 can be a user computing device or a server computing device.

[0069] The computing device 10 includes several applications (e.g., applications 1 through N). Each application contains its own machine learning library and machine learning model(s). For example, each application can include a machine learning model. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0070] As Figure 1B shown, each application can communicate with several other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0071] Figure 1C A block diagram of an example computing device 50 that is executed in accordance with example embodiments of the present disclosure is depicted. Computing device 50 can be a user computing device or a server computing device.

[0072] Computing device 50 includes several applications (e.g., applications 1 through N). Each application communicates with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and models stored therein) using an API (e.g., an API common to all applications).

[0073] The central intelligence layer includes several machine learning models. For example, as Figure 1C shown, a respective machine learning model (e.g., model ) can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine learning model. For example, in some implementations, the central intelligence layer can provide a single model (e.g., a single model) for all applications. In some implementations, the central intelligence layer is included in or otherwise implemented by an operating system of computing device 50.

[0074] The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized data repository of computing device 50. As Figure 1C shown, the central device data layer can communicate with several other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0075] Example model implementation

[0076] Spatial inaccuracies can occur through data ingestion generating perceptual distortions. A transmitter network system can preserve the three-dimensional spatial structure of visual input. The system can use three-dimensional information to project visual data to a spatially consistent representation. The spatially consistent representation can allow for leveraging the equivariance of inductive bias present in data symmetries for more efficient learning.

[0077] The system can be used to learn various tasks. One such task can be a pick-and-place action (e.g., the example in Figure 2 From the visual observation o t Learning pick-and-place actions a t can be represented as where is the pose of the end-effector for picking up the object, and is the pose of the end-effector for placing the object. The two poses can be defined in two-dimensional or three-dimensional notation, depending on the available degrees of freedom and tasks of the end-effector. The two actions can depend on the observation data processed by the action model and the embedding model. The place action can also be represented as depending on the place action, such that:

[0078]

[0079] In this representation, the first pose of the robot, or the pick pose, depends on the visual observation o t Once the first pose is determined, the second pose of the robot, or the place pose, can be determined based on the visual observation o t and the determined first pose .

[0080] The system explained by the process of learning pick-and-place actions can be understood as: (i) there exists a distribution of successful pick poses 208 from which a pick pose can be sampled, and (ii) for each successful pick pose, there exists a corresponding distribution of successful place poses 220 from which a place pose

[0081] The system can recover the distributions of successful picks and successful places from the visual observation 206 without assuming object metrics. Avoiding object-centric representations can allow the system to have the ability to handle unseen objects, deformable objects, and uncountable piles of small objects.

[0082] The equations can be generalized to other motion primitives beyond pick-and-place actions, which can also be parameterized by two end-effector poses (i.e., an initial action pose and a final action pose). While the above equations focus on the primitive of two poses, the same approach can be generally applied to longer sequences of pose changes In general, there can be multiple pose changes in a sequence.

[0083] Further, the visual observation o t 206 can be a projection of a scene (e.g., in the form of an RGB-D image) defined on a regular grid of pixels {(u, v)} at time step t of the sequence rearrangement task. Through a camera-to-robot calibration, the system can map ot Each pixel in the image corresponds to a picking action at that location: The distribution 208 of successful picks for a pixel can be multimodal in nature. Multimodal distributions can be more prevalent when there are multiple instances of the same object in the scene, or when the shape of the object has symmetries. The system can use various types of models. In some implementations, the system can use a fully convolutional network to model the action value function associated with successful picking:

[0084]

[0085] The selected neural network can provide equivariance such that if an object to be picked in the scene is translated, the picking pose is also translated. The equivariance can be characterized as where g is the translation. The spatial equivariance can improve the learning efficiency of vision-based picking.

[0086] The system or method can convert the RGB-D image into a spatially consistent form by unprojecting to a three-dimensional point cloud. The system can then render the three-dimensional point cloud into an equirectangular projection, where each pixel (u, v) represents a fixed window of three-dimensional space. The system can use this spatially consistent representation for the teleportation operation.

[0087] The spatially consistent visual representation can enable the system to perform visual spatial teleportation. The teleportation can include dense pixel-level features from a partial crop being rigidly transformed and then superimposed over another partial crop o t [Δτ] where o t is the observation before picking and Δτ is the relative transformation from the picking pose. In some implementations, the system can involve a partial crop of the environment being superimposed over a second partial crop to determine the action of the robot.

[0088] For example, can be a partial crop from o t centered at o i . The goal of the system can be to densely teleport the query to a set of poses {Δτ t} to search for the best placement, (i.e., the o i [Δτ t ] with the highest feature correlation). The system can formulate the query as a template matching problem using the cross-correlation of dense feature embeddings ψ(·) and φ(·) (e.g., first and second feature embeddings) from the two deep models:

[0089]

[0090] where, can be an action-value function related to placement success, and Δτ can cover the space of all possible placement poses. In some implementations, the space of all possible placement poses can be equivalent to the space of two-dimensional translations of o t . The structure of the dense features ψ(o t ) and φ(o t ) can be spatially consistent with the spatial consistency of the observed data.

[0091] As long as the cross-correlation from the relative transformation Δτ produces the desired imagined superposition, the system or method can be invariant to . The invariance property can enable learning from a small number of demonstration samples conditioned on picking, without having to train on all possible picks.

[0092] In two-dimensional plane applications, and can represent rotations as well as superposed translations. For a two-dimensional plane example implementation, the system can discretize the space of two-dimensional plane rotations into k dyads, and then can rotate the input visual observation o t for each dyad. Thus, o t can be defined on a grid of voxels {(u, v, w) i}, where w lies on the axis of k-discretization of rotations, and and can be discretely represented on this grid for two-dimensional rigid transformations. The system can implement these steps by running a fully convolutional neural network k times— once for each rotated o t , sharing the model weights across all rotations.

[0093] The system can be applied to three-dimensional rigid transformations. The transfer function can be used to extend the system to three-dimensional interactions. The system can use a multi-stage approach to extend to tasks requiring three-dimensional placements. First, the degrees of freedom of three two-dimensional rigid transformations can be resolved, producing an estimated Then, the system can extend the transfer operation of cross-correlation. The system or method can use three channels on the space of two-dimensional rigid transformations to regress the remaining degrees of freedom of rotation (r x , r y ) and translation (z-height). The system can use two embedding models ψ' and φ' that are the same as the embedding models ψ and φ used for two-dimensional rigid transformation classification, but with three cross-correlations (*3) separated from a subset of the channels, and can add a learnable nonlinearity via a three-headed MLP network after the cross-correlation,

[0094]

[0095] The system's hybrid discrete / continuous approach can provide an accurate representation of complex multi-modal distributions in the two-dimensional rigid transformation space of images, while once conditioned on a multi-modal in-plane distribution, the remaining degrees of freedom out-of-plane of the image can be unimodal. Furthermore, once the approximated two-dimensional rigid transformation arrangement is superimposed via a teleportation operation, the approach can provide an attention mechanism to help the model make an accurate continuous regression.

[0096] An example application of the system is the sequential challenge of a Tower of Hanoi object pick-and-place sequence (e.g., the example embodiment in Figure 3 The challenge of the Tower of Hanoi can require an actor to move three disks from a first tower to a third tower in order, without placing a large disk on a small disk. The disks can be of different colors. Completing the task can require arranging the order of seven pick-and-place actions correctly. The system can adjust the actions of the sequential task based on visual contextual cues (e.g., the next pick-and-place depends on the positioning of an observed object in the scene or is absent due to occlusion). The system or method can be stateless. The system or method can learn the ordering behavior through visual feedback. The system can increase the receptive field of the embedding model to encompass most of the visual observations o t The system can utilize more memory to handle non-Markovian tasks.

[0097] The system or method can be used for tasks other than pick-and-place actions. The system can be applied to other two- pose primitive tasks, including but not limited to: pushing and dragging. For example, the system can learn to rearrange a deformable rope in order, such that the system determines a set of actions that can allow a robot or other actor to connect two endpoints of an incomplete trihedron. Another example application can include using a shovel-shaped end effector to push a pile of small objects in order to a desired set of targets. The system can learn other closed-loop behaviors using visual feedback to complete these tasks. The system can be applicable to rigid displacements and non-rigid placements, such as these examples. The system can be used to knock an object to initiate movement of the object to a given location.

[0098] The system can be used to determine the start and end positions of a slicing or cutting motion. These example motion primitives can also be combined to learn to perform more complex tasks.

[0099] The system can be configured with various network architecture configurations.

[0100] An example embodiment of the system's observation space can include visual observations o t which can be an orthographic top-down view of a 0.5 x 1 m table workspace generated by fusing 480 x 640 RGB-D images captured with a calibrated camera using known intrinsic and extrinsic factors. The top-down image o tmay have a pixel resolution of 160 x 320. The pixels can represent 3.125 x 3.125 mm vertical columns of 3D space in the workspace. The image o t may contain information of both color (RGB) and scalar distance-to-bottom height (H). Access to both color and height values can enable the depth model to learn rich features in both visual texture and geometry. It can be appreciated that other sizes of workspaces, RGB-D images, and top-view images can be used instead.

[0101] An example embodiment of the system's first-pose determination feature used as a pick model can include a single feed-forward fully convolutional neural network that can take as input a visual observation and can output a dense pixel-level value that can be related to pick success: The pick model can be an hourglass encoder-decoder architecture. For example, a 43-layer residual network (ResNet) with 12 residual blocks and 8 strides (3 2-stride convolutions in the encoder and 3 bilinear up-sampling layers in the decoder) followed by an image-level softmax can be used. Each convolutional layer after the first can be equipped with dilation and interleaved with ReLU activations before the last layer. Using 8 strides can strike a balance between maximizing the receptive field coverage of each pixel prediction while minimizing the resolution loss of the network's latent mid-level features. It can be appreciated that other network structures can be used instead.

[0102] An example embodiment of the system's first-pose determination feature used as a place model can use a dual-stream feed-forward fully convolutional neural network that can take as input a visual observation and can output two dense feature maps: a query feature and a key feature where d can be the feature dimension. The place model can share a similar hourglass encoder-decoder architecture as the related pick model. For example, each stream can use an 8-stride 43-layer ResNet. The system can use ResNet in the last layer without a non-linear activation. The portion of the query feature centered around with size c can be transformed by Δτ, which encompasses the space of possible place configurations, and then cross-correlated with the key feature map φ(o t ) to output a dense pixel-level value that can be related to place success: where Δτ can be discretized into k = 36 angles (multiples of 10°). The maximum correlation can give the translation and rotation Δτ that gives the highest correlation to the portion of the query feature In some implementations, the system can implement operations with highly optimized matrix multiplication, where cropping is flattened. In some implementations, the system can implement operations as convolutions, where cropping is a convolution kernel. It can be appreciated that other network structures can be used instead.

[0103] The system can be trained using ground truth data and various loss functions. For example, during training, a sample observation-action pair can be taken uniformly from the dataset Each observation-action pair can be unpacked into two training labels: t and are used to generate binary one-hot pixel maps and The training loss can be the cross-entropy between the one-hot pixel maps and the output of the pick-and-place model, e.g., While each dense probability map training can only have a label for a single pixel, gradients can be passed to all other pixels via an image-level softmax.

[0104] The system or method can be applied or taught using various transformation groups. The system can utilize lie groups, including but not limited to: SE(2), SO(2), and SE(3).

[0105] Figure 2 An illustration depicting an example action determination model 200 in accordance with example embodiments of the present disclosure. In some implementations, the action determination model 200 is trained to receive a set of input data 206 describing an observation region, and provide output data 208 and 220 including a first pose and a second pose as a result of receiving the input data 206. Thus, in some implementations, the action determination model 200 can include an object manipulation model operable to move an object from an initial point to a determined end point.

[0106] Figure 2 An example application of a system or method of action determination is depicted. The depicted action can involve a pick-and-place action, a push action, a drag action, etc. The environment 202 can include a robot with an end effector, an object, and an end point / target point of the object. The object can be of any shape, but in this embodiment, the shape of the object is a capital “L”. The object and the end point can be located in an observation scene inside the environment. The observation scene can be observed by one or more sensors. Further, the one or more sensors can be a camera, a LiDAR sensor, or other forms of measurement sensors. In this embodiment, the sensors take in data on the scene to generate observation data 206. The observation data can be input into the system. The system can have a goal of picking and placing the object with a set end point 204.

[0107] ​In the illustrated example, the observation data 206 is processed by the action model to generate a distribution of successful first actions 208. Based on the distribution of successful first actions 208, the system determines a first pose of the robot.

[0108] The observation data 206 is then processed by the first embedding model 210. In this implementation, the first embedding model 210 outputs one or more first feature embeddings 214.

[0109] In this example, the observation data 206 can also be cropped and rotated to produce a plurality of cropped and rotated data sets. The plurality of cropped and rotated data sets can be processed by the second embedding model 212 to generate a plurality of second feature embeddings 216.

[0110] The one or more first feature embeddings 214 and the plurality of second feature embeddings 216 can be compiled to provide a plurality of second possible poses 218. In the illustrated example, the system uses deep feature template matching to determine success of each possible second pose. The system generates a distribution of successful second poses 220. The system then selects a second pose from the distribution of successful second poses 220.

[0111] The action determination model 200 can then be used to guide a robot of the environment 202 to complete the first pose and the second pose, where the first pose is a pose in which an end effector of the robot is in contact with the object and the second pose is a pose in which the object is placed at the goal location with the aid of the end effector of the robot.

[0112] Figure 3 An illustration of an example action determination model 300 according to example embodiments of the present disclosure is depicted. The action determination model 300 is similar to the action determination model 200 of Figure 2 except that the action determination model 300 also includes a set of ground truth data for evaluating and modifying parameters of the action determination model. Figure 3 An example implementation of training a system is depicted. The depicted implementation can involve seven steps for completing a target task.

[0113] The action determination model 300 can be taught to complete a seven-step task of a “Tower of Hanoi” sequence. The system can have been taught a specific order or specific rules that allow the task to be completed. The action determination model 300 can include an environment 302 with a robot and an observation scene, where a “Tower of Hanoi” set is disposed in the observation scene. One or more sensors can generate observation data 306 based on the scene. The action determination model can then be used to process the observation data 306 to complete the target task 304.

[0114] In the illustrated example, the first step of the action determination model involves obtaining observation data 306. Once the observation data 306 is obtained, the observation data 306 is processed by the action model to generate a distribution of possible first actions that are successful 308. Then, the system selects the argmax of that model to determine the first action 312.

[0115] In this embodiment, the observation data 306 and the first action 312 are then processed by the first embedding model and the second embedding model to generate a first feature embedding 316 and a second feature embedding. The first feature embedding 316 can be related to a plurality of possible translations. The second feature embedding can be related to a plurality of possible rotations. The first feature embedding and the second feature embedding are then convolved and the convolved feature embedding 320 is used for deep feature template matching. Based on the deep feature template matching, the system can determine a distribution of possible second actions that are successful. The system can select one of the possible second actions as the second action of the robot. The action determination model 300 can be iteratively run in order to complete the entire sequence 314.

[0116] The action determination model 300 can be trained using ground truth data 314. The ground truth data can include the starting image 310, the end point image 318, and images of each placement in between. For the seven step “Tower of Hanoi” depicted in the illustrated example, the ground truth data 314 includes eight images. Each step can be processed to generate a one-hot ground truth pixel map for each step. To complete the first step of the “Tower of Hanoi”, one-hot pixel maps are generated for the ground truth first action and the ground truth second action (i.e., the pick up action and the place action). The generated first ground truth pixel map and the second ground truth pixel map can then be compared to the determined first action and the determined second action to evaluate the action determination model. In response to the evaluation, parameters of the action model, the first embedding model, or the second embedding model can be modified. The training process can be iteratively run to provide more accurate results.

[0117] In some implementations, the action determination model 300 can use the ground truth data 314 to help complete the task. The ground truth data 314 can be processed by a third embedding model to generate a third feature embedding. The third feature embedding can be used to provide more accurate results for the deep template matching.

[0118] Figure 4 An example application of the system or method of action determination is depicted. The depicted implementation can involve using a multi-modal strategy to complete different actions for different shapes.

[0119] The action determination model 400 can be used to complete several tasks in the same environment 402 involving different objects located at different locations in the environment 402. In this implementation, the observation region 404 includes five objects of four different shapes. The observation region 404 also includes five destinations of four different shapes corresponding to the objects. The action determination model 400 can be used to pick and place each object to the corresponding destination. The action determination model can be run iteratively to complete the tasks.

[0120] The action determination model can include obtaining observation data 406, which can be processed by the first action model to determine a first action 408 on one of the objects. For each respective object at each respective location on each object, the second action can be different due to the shape of the object and the shape of the destination. For example, if the first action is a pick action on the edge of a circular object 410, the action determination model can produce a distribution of possible second actions, where the distribution is a circular distribution in a circular destination 414. Another example can involve a first action being a pick action on a leg of one of the “E” shaped objects 412. The action determination model can generate a distribution consisting of two points, where each point is located in one of the “E” shaped destinations 416.

[0121] Furthermore, the action determination model can be used to determine a first action (e.g., a pick action or a contact action) and a second action (e.g., a place action or a release action) for each respective object 420.

[0122] Furthermore, Figure 4 A success distribution 418 of the rotated placement of each shape is displayed. The symmetry of each object contributes to the distribution results. For example, a circle selected at the center can have a uniform distribution from 0 to 2p when placed in a circular destination. However, the square and clover shapes can have four separate maxima with almost uniform amplitudes when the selection point is the center. Shapes that lack any form of symmetry can only produce a single maximum with statistical significance when the destination is the same shape.

[0123] Similarly, changing the selection point or the translation point can change the success distributions 422 and 424.

[0124] Example method

[0125] Figure 5 A flowchart depicting an example method performed in accordance with example embodiments of the present disclosure. Although the method 500 is depicted as being performed in a specific order, the method of the present disclosure is not limited to the specific order or arrangement shown. Various steps of the method 500 can be omitted, rearranged, combined, and / or adjusted in various ways without departing from the scope of the present disclosure. Figure 5 The steps are depicted for illustration and discussion as being performed in a specific order, but the method of the present disclosure is not limited to the specific order or arrangement shown. Various steps of the method 500 can be omitted, rearranged, combined, and / or adjusted in various ways without departing from the scope of the present disclosure.

[0126] At 502, the computing system can obtain observation data. The observation data can be image data, LiDAR point cloud data, or another form of data. Further, the data can be obtained from one or more sensors located around an observation scene. The observation data can include data about an environment. Further, the environment can include one or more objects for which an action is to be taken.

[0127] At 504, the computing system can determine a first action. The determination of the first action can include processing the observation data with a first action model. The first action model can use pixel mapping to generate a distribution of possible first actions that are successful. The system can use the distribution to select the first action. In some embodiments, the first action can be an argmax of a model function.

[0128] At 506, the computing system can process the observation data to determine a first feature embedding. The observation data can be processed by a first embedding model to generate the first feature embedding. In some implementations, the first feature embedding can be a plurality of possible robot movement translations. Further, the first embedding model can be a fully convolutional neural network.

[0129] At 508, the computing system can process the observation data to generate a second feature embedding based at least in part on the first action. The observation data can be processed by a second embedding model to generate the second feature embedding. The second feature embedding can describe a plurality of possible robot movement rotations. In some implementations, the second embedding model can be a fully convolutional neural network.

[0130] In some implementations, the first embedding model and the second embedding model can process the observation data for the first action.

[0131] At 510, the computing system can determine a second action. In some implementations, the second action can be determined based at least in part on a convolution of the first feature embedding and the second feature embedding. The determination can include deep feature template matching. The convolution of the first feature embedding and the second feature embedding can be used to generate a distribution of possible second actions, where the second action can be selected from the distribution.

[0132] Figure 6 A flow diagram depicting an example method performed in accordance with example embodiments of the present disclosure is depicted. Although the steps are depicted as occurring in a particular order, the methods of the present disclosure are not limited to the order or arrangement specifically shown. Individual steps of the method 600 can be omitted, rearranged, combined, and / or adjusted in various ways without departing from the scope of the present disclosure. Figure 6 The steps are depicted as being performed in a particular order for the sake of illustration and discussion, but the methods of the present disclosure are not limited to the order or arrangement specifically shown. Individual steps of the method 600 can be omitted, rearranged, combined, and / or adjusted in various ways without departing from the scope of the present disclosure.

[0133] At 602, the computing system can obtain observation data and then process the observation data to generate a distribution of successful initial actions. The observation data can be processed by an action model and the observation data can include information about the environment in which the robot is working. In some implementations, the action model can process the data on a pixel-by-pixel basis.

[0134] At 604, the computing system can determine a first action. The determination can include selecting the first action from the distribution of successful initial actions. The selection can be based on whether a point is a local maximum. Further, the first action can be a pick action, where the action causes an end effector of the robot to contact an item and pick up the item.

[0135] At 606, the computing system can process the observation data to generate a first feature embedding. The processing can be done by a first embedding model and, in some implementations, the first embedding model can be a fully convolutional neural network.

[0136] At 608, the computing system can process the observation data to generate a second feature embedding. The processing can be done by a second embedding model and, in some implementations, the second embedding model can be a fully convolutional neural network.

[0137] At 610, the computing system can determine a second action based at least in part on a comparison of the first feature embedding and the second feature embedding. The determination of the second action can also depend on the first action. Further, the second action can be determined by using deep feature template matching generated from the comparison of the first feature embedding and the second feature embedding. In some implementations, the second action can be a place action, where the action releases the item contacted by the first action. In some implementations, the second action can place the item in a desired location. In some implementations, a machine learning model can be used to generate the second action based on the first feature embedding and the second feature embedding.

[0138] Figure 7 A flowchart depicting an example training method performed in accordance with example embodiments of the present disclosure. Although the steps are depicted as being performed in a particular order, the methods of the present disclosure are not limited to the order or arrangement specifically shown. Individual steps of the method 700 can be omitted, rearranged, combined, and / or adjusted in various ways without departing from the scope of the present disclosure. Figure 7 The steps are depicted as being performed in a particular order for the sake of illustration and discussion, but the methods of the present disclosure are not limited to the order or arrangement specifically shown. Individual steps of the method 700 can be omitted, rearranged, combined, and / or adjusted in various ways without departing from the scope of the present disclosure.

[0139] At 702, the computing system can obtain observation data. The observation data can be a spatially consistent representation to allow for the use of the principle of equalization. The observation data can include data that observes an environment. The environment can include one or more items for manipulation by a robot.

[0140] At 704, the computing system can determine a first action. The determination of the first action can be based on processing the observation data by the action model to generate a distribution of successful initial actions. The distribution of successful pick actions can include one or more maxima, including an argmax of the function of the action model. The first action can be determined by selecting the maxima or even the argmax of the function of the action model.

[0141] At 706, the computing system can process the observation data to generate a first feature embedding. The processing can be done by a first embedding model, where the first embedding model processes the observation data to generate the first feature embedding. The first feature embedding can include possible translations of the object for the first action. The first embedding model can generate multiple first feature embeddings with different possible translations.

[0142] At 708, the computing system can process the observation data to generate a second feature embedding. The processing can be done by a second embedding model, where the second embedding model processes the observation data to generate the second feature embedding. The second feature embedding can include possible rotations of the object for the second action. The second embedding model can generate multiple second feature embeddings with different possible rotations. The processed observation data can be cropped and rotated observation data. The cropped and rotated observation data can be generated by cropping the observation data to be fixed on a set of pixels of interest. The cropped data can then be duplicated and rotated to generate multiple cropped and rotated observation data to be processed.

[0143] At 710, the computing system can determine a second action, for example, based at least in part on a convolution of the first feature embedding and the second feature embedding or based at least in part on processing by a more complex learning model. The determination of the second action can be determined by performing a convolution of the first feature embedding and the second feature embedding and completing a deep template matching to determine which second actions would be successful. The second action can be selected from the determined successful second actions.

[0144] At 712, the computing system can evaluate the first action and the second action with a loss function. The evaluation can include comparing the first action and the second action to a ground truth action using the loss function. The ground truth action can be processed to generate a one-hot ground truth pixel map to compare to the determined actions. For three-dimensional manipulation, the loss function can be a Huber loss.

[0145] At 714, the computing system can modify values of parameters of the embedding models. The modification to the action model, the first embedding model, or the second embedding model can be done in response to the evaluation of the first action and the second action. The evaluation and modification can be used to train the action determination model. The model can be trained by an iterative training technique.

[0146] Additional disclosure

[0147] The technology discussed herein makes reference to servers, databases, software applications, and other computer-based systems, and actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. For instance, processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0148] While the present subject matter has been described in detail with respect to various specific embodiments of the present subject matter, it should be understood that the embodiments are illustrative only and are not restrictive of the present subject matter. Changes, variations, and equivalents can be made to the embodiments by those skilled in the art in light of the foregoing description, which is to be taken in a non-limiting sense. For example, features shown or described as part of one embodiment can be used with another embodiment to yield still a further embodiment. It is therefore intended that the present subject matter be considered as limited only by the true spirit and scope of the disclosure.

Claims

1. A computer-implemented method for generating robot movements, comprising: Obtain observation data, which includes data describing the environment prior to the robot's first action; The robot's first action is determined at least in part based on observational data; The first feature embedding is generated by processing observation data describing the robot's first action using a first embedding model. A second feature embedding is generated by processing observation data describing the robot prior to its first action using a second embedding model, wherein the second embedding model is conditioned on the first action; and The robot's second action is determined at least in part based on a comparison of the first feature embedding and the second feature embedding.

2. The computer-implemented method according to claim 1 further includes: Control the robot to make it perform the first action and the second action.

3. The computer-implemented method according to claim 1, wherein, The first action involves the robot making contact with the object.

4. The computer-implemented method according to claim 1, wherein, The observation data includes image data.

5. The computer-implemented method according to claim 1, wherein, The observation data includes point cloud data from light detection and ranging (LiDAR).

6. The computer-implemented method according to claim 1, wherein, The first action involves changing the robot's posture.

7. The computer-implemented method according to claim 1, wherein, Each of the first and second embedding models includes a fully convolutional neural network.

8. The computer-implemented method according to claim 1, wherein, Determining the robot's first action includes: Based on observed data and using a first machine learning action-value model, a first probability distribution for the first action is generated; and Choose the action that has the maximum value in the first probability distribution as the first action.

9. The computer-implemented method according to claim 8, wherein, The first probability distribution includes one or more points that indicate the probability that one or more points are the starting positions for object manipulation.

10. The computer-implemented method according to claim 1, wherein, Determining the robot's second action based at least in part on a comparison of the first feature embedding and the second feature embedding includes: The second feature embedding is convolved on the first feature embedding to generate a second probability distribution; and Choose the action that has the maximum value in the second probability distribution as the second action.

11. The computer-implemented method according to claim 1, wherein, Using a second embedding model to process observed data to generate a second feature embedding includes: Rotate and crop the observed data to generate multiple rotated and cropped data samples; and Multiple rotated and cropped data samples are processed using a second embedding model to generate multiple second feature embeddings.

12. The computer-implemented method according to claim 1, wherein, The observational data obtained include: Obtain raw observation data; Projecting the original observation data into three-dimensional space; and The deprojected data is rendered into a frontal representation of the environment, where the frontal representation of the environment is used as the observation data.

13. The computer-implemented method according to claim 1, wherein, The observation data, the first feature embedding, and the second feature embedding include three-dimensional data.

14. The computer-implemented method according to claim 1, wherein, Determining the robot's second action involves processing the first and second feature embeddings using a second machine learning value action model.

15. The computer-implemented method according to claim 1, further comprising: Obtain a set of target image data; as well as The third embedding model is used to process the set of target image data to generate a third feature embedding. Determining the robot's second action includes: determining the second action based at least in part on the first feature embedding, the second feature embedding, and the third feature embedding.

16. A computer system, comprising: One or more processors; as well as One or more non-transitory computer-readable media that commonly store instructions that, when executed by one or more processors, cause a computer system to perform operations, said operations including: Obtain observation data, which includes data describing the environment prior to the robot's first action; Determine the robot's first action; The first feature embedding is generated by processing observation data describing the robot's first action using a first embedding model. A second feature embedding is generated by processing observation data describing the robot prior to its first action using a second embedding model, wherein the second embedding model is conditioned on the first action; The robot's second action is determined at least in part based on a comparison between the first feature embedding and the second feature embedding; The evaluation loss function compares one or both of the first and second actions with one or both of the first and second benchmark true actions; and Based at least in part on the loss function, modify one or more values ​​of one or both of the parameters of the first embedding model and the second embedding model.

17. The computer system according to claim 16, wherein, The evaluation loss function includes: A first one-hot reference truth pixel map is generated based on a first reference truth action, and a second one-hot reference truth pixel map is generated based on a second reference truth action. The first one-hot reference truth pixel map includes a first plurality of pixels having corresponding binary values ​​indicating the reference truth position of the first action in the environment. The second one-hot reference truth pixel map includes a second plurality of pixels having corresponding binary values ​​indicating the reference truth position of the second action in the environment. The first action is compared with the first isolated heat reference true pixel map, and the second action is compared with the second isolated heat reference true pixel map.

18. The computer system according to claim 17, wherein, The loss function includes cross-entropy loss, and wherein comparing the first action with a first one-hot reference ground truth pixel map and comparing the second action with a second one-hot reference ground truth pixel map includes: evaluating the cross-entropy loss on a pixel-by-pixel basis.

19. A robotic device comprising one or more non-transitory computer-readable media that collectively store instructions, which, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations including: Obtain observation data, which includes data describing the environment prior to the robot's first action; Determine the first action of the robot equipment; The first feature embedding is generated by processing observation data describing the robot's first action using a first embedding model. A second feature embedding is generated by processing observation data describing the robot prior to its first action using a second embedding model, wherein the second embedding model is conditioned on the first action; The second action of the robotic device is determined, at least in part, based on a comparison of the first feature embedding and the second feature embedding; and The first and second actions are performed by robotic equipment.

20. The robot device according to claim 19, wherein, The observational data includes spatially consistent data.

Images