Controlling agents by tracking points in image

Abstract

Systems and methods for controlling agents using tracking points in an image. For example, a mechanical agent interacting in a real-world environment is controlled by selecting an action to be performed by the agent to perform an instance of a task using images captured while performing the instance of the task.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 535,568, filed August 30, 2023. The disclosure of that earlier application is considered part of the disclosure of this application and is incorporated herein by reference. Background Technology

[0003] This specification relates to the use of neural networks to control intelligent agents.

[0004] A neural network is a machine learning model that uses one or more layers of non-linear units to predict the output from a given input. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer is used as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output from the received input based on the current values ​​of its corresponding set of parameters. Summary of the Invention

[0005] This specification describes a system implemented as a computer program on one or more computers at one or more locations, which controls an agent by selecting an action to be performed by an agent (e.g., a robot) interacting in an environment and then causing the agent to perform the action.

[0006] Specifically, the system uses images captured while the agent is performing an instance of a task to control the instance of the agent performing the task.

[0007] The subject matter described herein may be implemented in specific embodiments in order to achieve one or more of the following advantages.

[0008] Demonstration learning, also known as learning how to perform a task, involves learning from a set of demonstrations of the task being performed, enabling the agent to autonomously execute new instances of the task after learning from the demonstrations. In other words, it's not about manually programming the ability to perform tasks, but rather the agent learning from expert demonstrations—for example, by humans, by fixed policies for the task, or by an agent controlled by a learned policy that has been trained for the task.

[0009] Current approaches to demonstration learning often require task-specific engineering or excessive amounts of demonstration data, preventing demonstration learning from being implemented within the timeframe required for practical use. For example, imitation learning (i.e., training a first system to mimic actions demonstrated by a second, different system, e.g., behavior cloning) and inverse reinforcement learning for image-guided robotic agents are powerful but data- and time-intensive ways of training robotic agents to perform tasks, as they may require hundreds to thousands of task demonstrations across a variety of environments to teach the agent to process images robustly to perform the task.

[0010] One reason for the large data and time requirements is that the input for demonstration learning is typically raw images associated with the task being performed. Because each demonstration provides a wide range of environments and scenarios, the agent may need a large number of demonstrations (and therefore a large amount of data and training time) to learn the appropriate internal representations required to generalize task performance.

[0011] On the other hand, this specification describes tracking points in images as input to allow for faster and more general learning from demonstrations. As described in this specification, by using points in images as input for demonstration learning, the number of demonstrations required to teach the agent to perform a task is reduced by an order of magnitude (and therefore, the amount of data required and training time are reduced by an order of magnitude), while still enabling generalization of task performance by the agent.

[0012] By tracking points in images during a demonstration task, the system can automatically extract individual motions, the relevant points for each motion, the target locations of those points, and generate plans that can be executed by the agent for new instances of the task, without any action supervision, task-specific training, or neural network fine-tuning.

[0013] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description.

[0014] Other features, aspects and advantages of the subject matter will become apparent from the description, drawings and claims. Attached Figure Description

[0015] Figure 1 An example intelligent agent control system is shown.

[0016] Figure 2 An example intelligent agent control system is shown.

[0017] Figure 3 This is a flowchart of an example process for determining relevant points in a task segment.

[0018] Figure 4A sequence of example images depicting an example process for determining relevant points in a task segment involving a robot with a gripper equipped with a camera is shown.

[0019] Figure 5 This is a flowchart illustrating an example process for controlling an intelligent agent using an intelligent agent control system.

[0020] Figure 6 This is a flowchart of an example process for performing a new instance of a task.

[0021] Figure 7 An example of a task performed by a robot with a gripper equipped with a camera using the described technology is shown.

[0022] The same reference numerals and names in the various figures indicate the same elements. Detailed Implementation

[0023] Figure 1 An example intelligent agent control system 100 is shown. The intelligent agent control system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, wherein the systems, components and techniques described below can be implemented.

[0024] The intelligent agent control system 100 controls the intelligent agent by selecting an action to be performed by an intelligent agent (e.g., a robot) interacting in an environment (e.g., a real-world environment) and then causing the intelligent agent to perform that action. Specifically, the system 100 uses images captured while the intelligent agent is performing an instance of a task to control that instance of task performance. That is, the system 100 receives image 106 and generates action 118 for the intelligent agent to perform the task.

[0025] More specifically, system 100 processes multiple demonstration image sequences 102 to generate data representing multiple task segments (e.g., first task segment 104A and second task segment 104B), and for each task segment, generates data representing multiple related points (e.g., related points 108 of the first task segment 104A). System 100 then processes the images 106 and the data generated from the demonstrations 102 to generate an action 118 to be performed by the agent.

[0026] The images constituting the received image 106 and the plurality of demonstration image sequences 102 can be captured by a camera sensor of the robot or by a camera sensor located in the environment. The robot can be a mechanical robot operating in a real-world environment. The camera sensor can capture images of the robot as it performs a task in the environment (e.g., in a real-world environment). As a specific example, the robot can be a robot including a gripper for holding and moving objects in the environment, and the camera sensor can be positioned on the gripper (or mounted in a fixed position and orientation relative to the gripper), i.e., such that the gripper and the object held in the gripper do not move significantly relative to the camera sensor unless the gripper opens or closes.

[0027] Each of the demonstration image sequences 103A-C is, for example, a sequence of images of a corresponding instance of an agent performing a task when the agent is controlled by a human, a fixed policy for the task, or a learned policy that has been trained for the task. Different instances of the task may have different configurations of objects in the environment, but have the same goal, such as aiming to move similar objects to the same location, or aiming to arrange similar objects in a specific configuration, etc.

[0028] Although Figure 1 Only three demonstration image sequences are shown, namely demonstration image sequences 103A-C, but in reality, system 100 can process any number of demonstration image sequences.

[0029] System 100 operates in two modes: extraction mode and action mode.

[0030] During the extraction mode, the system 100 processes multiple demonstration image sequences 102 to generate data representing multiple task segments, and for each task segment, generates data representing multiple related points of that task segment.

[0031] Each task segment in the task segmentation corresponds to a corresponding portion of each of the multiple demonstration image sequences 102.

[0032] Therefore, each task segment has a corresponding portion in each of the demonstration image sequences in the demonstration image sequence 102, wherein the “portion” of the demonstration image sequence includes only a proper subset of the images in the image sequence.

[0033] In some implementations, the portion of each of the multiple demonstration image sequences 102 associated with the task segment may contain a varying number of images.

[0034] System 100 can use any of a variety of methods to determine task segments.

[0035] For example, system 100 can use a set of time-based rules (e.g., a predefined number of images) to generate task segments.

[0036] As another example, system 100 can use events or actions to create task segments, such as segmentation based on robot pose information (e.g., gripper movements and forces). That is, system 100 can segment each sequence of demonstration images based on the position of a specified component of the robot, based on the forces applied to the robot, or both. Specifically, the system can segment each sequence of demonstration images based on one or both of the following: the position of a specified component of the robot; and the forces applied to the robot.

[0037] As a specific example, when a robotic arm agent moves a gripped block to perform a stenciling task, system 100 can extract gripper actuation events by using the gripper's open position and recording the points where that position crosses a selected threshold. These time points are the start or end of the grip and can be used to determine the start and end points of task segments.

[0038] As another specific example, in the case of a robotic arm agent moving a gripped block to perform a matching task, system 100 can extract the start or end of a force phase. To extract these, system 100 tracks the vertical force measured by a torque sensor, smooths the signal, and converts it into a normalized force signal. System 100 then uses selected thresholds to determine the occurrence of force events and uses these points to determine task segments.

[0039] Although Figure 1 Only two task segments are shown, namely task segments 104A and 104B, but in reality, system 100 can generate any number of task segments.

[0040] As used in this specification, a "point" is a point in the corresponding image, that is, a point that specifies a corresponding spatial location (i.e., a corresponding pixel) in the corresponding image. Each pixel may have one or more associated values ​​or attributes (e.g., intensity values). For example, each pixel may include one or more intensity values, each intensity value representing the intensity of the corresponding color (e.g., RGB values). Therefore, the values ​​of pixels in an image can represent the features of the image.

[0041] System 100 uses a point tracker to track a randomly selected set of points across all demonstrations 102 and generates tracking data. Based on this tracking data, system 100 selects a subset of points associated with each task segment as one or more associated points for that task segment. Points can be tracked across an image sequence by determining corresponding locations (points) within the images that are associated with the same features (e.g., the same segment or portion of a scene or environment shown in the image). For example, each point tracked across a sequence could represent the same location on the surface of an object within the environment. As the relative position of the camera and the object moves, the positions of the points within the images may change.

[0042] For example, for a task involving the manipulation of objects, system 100 can select the relevant points as those points on the relevant object being manipulated.

[0043] As another example, system 100 can select points based on a set of rules, such as relevant points must have a certain degree of motion, or relevant points must have a common location across the demonstration.

[0044] The following text is for reference only. Figure 3 and Figure 4 Describe further details of the relevant points for determining the task segments.

[0045] System 100 also maintains corresponding point tracking data for each relevant point in each task segment, which identifies the corresponding spatial location of the point in at least some images in the images corresponding to the task segment, for example, point tracking data 112 for relevant point 108 of the first task segment 104A.

[0046] Therefore, point tracking data can identify the spatial locations of the same point in different images. Point tracking data may also include an occlusion score representing the probability of occlusion, and optionally, an uncertainty score representing the uncertainty in the predicted spatial location. If the spatial location of a point differs across different images, then the point has moved relative to the camera between the different images.

[0047] During the action mode, the system 100 uses the maintained data to execute new instances of the task, that is, by using (i) images captured when the agent executes a new instance of the task and (ii) relevant points of the task segment.

[0048] More specifically, system 100 can perform the following operations at each of multiple time steps during the execution of a new instance of a task.

[0049] At each of the multiple time steps, system 100 acquires an image of the agent (e.g., a robot) at that time step (e.g., captured by the robot's camera sensor at that time step). As a specific example, the acquired image 106 belongs to time step t.

[0050] System 100 identifies the current task segment at this time step.

[0051] For example, as described below, task segments can be determined based on the positions of robot components in a demonstration sequence, or the forces applied to the robot, or both. That is, each task segment begins when the corresponding component is in a first position or a specific force has been applied to the robot (or, for the first task segment, at the start of the task instance) and continues until the corresponding component is in a second position or a specific force has been applied to the robot. When the agent is a robot with a gripper, segments can be based on the forces applied to the gripper or the position of the gripper, i.e., open and closed positions indicating the degree to which the gripper is open or closed.

[0052] Then, system 100 can identify the current task segment by recognizing whether the criteria for terminating the task segment of the previous time step have been met. If the corresponding criteria have not been met, system 100 sets the current task segment as the task segment of the previous time step, and if the corresponding criteria have been met, system 100 sets the current task segment as the next task segment after the task segment of the previous time step.

[0053] System 100 determines one or more target points from relevant points in the current task segment, and determines the corresponding target prediction position in a future image for each target point from the point tracking data of the task segment.

[0054] As a specific example, for image 106 at time step t, system 100 determines one or more target points 110 from relevant points 108 in the current task segment 104A, and determines a corresponding target prediction location 116 in a future image 114 for each target point 110 from point tracking data 112 in task segment 104A. Each target point 110 may be a location (e.g., a pixel) within image 106 that represents (e.g., shown) a feature (e.g., an object or segment of the image) corresponding to one of the relevant points 108 in the current task segment 104A.

[0055] "Future Image" 114 is an image that identifies the corresponding spatial location of relevant points stored in point tracking data 112, from a demonstration image sequence 102, which the system 100 aims to replicate.

[0056] System 100 can determine the future image 114 as the image with the most similar corresponding point to the target point 110 in position relative to the image frame and the image 106 in all maintained demonstration image sequences, and further determine the target prediction position 116 as the corresponding point associated with the future image 114.

[0057] Generally, the target prediction location 116 is the "place" that the system 100 intends to move the target point 110 to reproduce the "future image". That is, the target point 110 identifies "what" points are relevant in the current image 106, the target prediction location 116 determines the "place" where these points should be, and the generated action 118 will determine "how" to make the target point 110 reach the target prediction location 116.

[0058] Then, system 100 causes the agent to perform action 118, which is predicted to move target point 110 to target predicted position 116.

[0059] For example, system 100 may apply a controller (e.g., a visual servo controller or other robot controller) to process one or more target points 110 and a corresponding target prediction position 116 in a future image 114 for each target point 110 to determine an action 118 predicted to move the target point 110 to the corresponding target prediction position 116, and then cause the agent to perform the determined action 118, for example, by applying control input to one or more controllable elements (e.g., joints, actuators, etc.) of the agent.

[0060] The following references are respectively Figure 5 and Figure 6 Further details describe the updated intelligent agent control system 100 and the new instance of performing tasks.

[0061] Figure 2 An example intelligent agent control system 200 is shown. The intelligent agent control system 200 is an example of a system implemented as a computer program on one or more computers at one or more locations, wherein the systems, components and techniques described below can be implemented.

[0062] The intelligent agent control system 200 is a system for controlling a robot that includes a gripper for holding and moving objects in an environment and a camera sensor positioned on the gripper, i.e., such that the gripper and the object held in the gripper do not move significantly relative to the camera sensor unless the gripper opens or closes. Specifically, the system 200 uses images captured when the intelligent agent performs an instance of a task involving moving an object to control the robot to perform that instance of the task. That is, the system 200 receives image 206 and generates action 218 for the robot to perform the task of moving the object.

[0063] During the extraction mode, system 200 processes three demonstration image sequences 203A-C into a set of demonstration image sequences 202. These demonstrations are for the task of clamping an L-shaped block and placing it on an elliptical target, and Figure 2 The image sequence is depicted for each presentation in top-to-bottom order. Although in Figure 2 The example processed three demo image sequences, but it should be understood that the number of demo image sequences can vary. Similarly, the number of demo images within each demo image sequence can vary.

[0064] Figure 2 The generation of the first task segment 204A and the second task segment 204B is also depicted, illustrating a corresponding portion of each of the multiple demonstration image sequences 202 constituting the first task segment 204A and the second task segment 204B. System 200 divides each demonstration image sequence into portions based on the force applied to the gripper and based on the position of the gripper. More specifically, for each demonstration image sequence, the first three images of the sequence constitute the first task segment 204A, the fourth image corresponds to motor primitives (e.g., the gripper closing and moving upwards), and then the next three images correspond to the second task segment 204B.

[0065] exist Figure 2 The text describes all the task segments and the motor primitives of the gripper under the title "Motion Plan".

[0066] System 200 uses a point tracker called “Tracking Any Point with Frame-by-Frame Initialization and Temporal Refinement” (TAPIR) to generate point tracking data. TAPIR is a method for accurately tracking specific points across a sequence of images, as described in ArXiv:2306.08637. This method employs two phases: 1) a matching phase, which independently locates suitable candidate point matches for each specific point on each other image; and 2) a refinement phase, which updates the trajectory based on local correlations across images. Although in Figure 2 In the example, system 200 uses TAPIR, but more generally, system 200 can use any other suitable point tracker capable of generating the desired output, such as BootsTap described in ArXiv:2402.00847 and TAPNet described in ArXiv:2211.03726. That is, any general point tracking method can be used to identify the relative motion of points across various images (frames) in a sequence. Point tracking can determine two pixels in two different images, each pixel representing the same segment or portion of a scene or environment shown within the images (e.g., they are projections of the same point on the same physical surface within the environment). Figure 2Tracking points are illustrated for each illustrated task segment via connecting lines across the illustrated image. For example, tracking points associated with relevant objects in the first task segment 204A are illustrated for each illustrated task segment.

[0067] Then, system 200 selects the relevant points and corresponding point tracking data for each task segment. Figure 2 Point tracking data 212 is depicted and represented as three sets of continuous lines within a single image (one set per demonstration) for relevant points 208 (represented as qt) in the first task segment 204A. Specifically, system 200 uses object discovery and selects L-shaped blocks as relevant objects and selects corresponding points on the relevant objects as relevant points to determine relevant points across the demonstration for the first task segment 204A.

[0068] During the action mode, the system 200 acquires an image 206 at time step t.

[0069] System 200 determines that the current task segment of image 206 and time step t is the first task segment 204A because the gripper has not yet begun to close for the first time, and because the criterion that the L-shaped block is positioned under the gripper has not yet been met, as can be determined after examining the image below the title 'Current Frame'. For image 206 at time step t, system 200 uses a point tracker (e.g., an online version of TAPIR) to determine the target point 210 from the relevant point 208. System 200 then selects future image 214 as the image with the most similar corresponding relevant point from point tracking data 212 in location to the target point 210, and defines the target predicted position 216 as the relevant point of future image 214.

[0070] Then, the system 200 uses a visual servo controller to process the target point 210 and the target predicted position 216 to generate an action 218 for the robot agent.

[0071] A visual servo controller typically refers to a control system that uses visual data (e.g., images from a camera sensor) to control the actions of another system (e.g., a robotic agent) in real time by continuously processing visual data. For example, in the case of a robotic agent, the visual servo controller determines the speed for moving one or more components of the robot such that target point 210 will move towards the target predicted position 216. Similarly, in the case of a robot with a gripper, the visual servo controller determines the speed for moving the gripper such that target point 210 will move towards the target predicted position 216.

[0072] System 200 can typically use any suitable visual servo controller to select the action to be performed by the agent at any given time step. Examples of visual servoing are described in the following: DOI: 10.1109 / 70.954764, Chen, Hanzhi et al., "Texpose: Neural texture learning for self-supervised 6d objectpose estimation", Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2023; and Hill, John, "Real time control of arobot with a mobile camera", Proceedings of the 9th International Symposium on Industrial Robotics, 1979.

[0073] Figure 3 This is a flowchart of an example process 300 for determining relevant points in a task segment. For convenience, process 300 will be described as being executed by a system of one or more computers located in one or more locations. For example, an intelligent agent control system appropriately programmed according to this specification (e.g., Figure 1 The intelligent agent control system 100 can execute process 300.

[0074] The system selects one or more points as initial correlation points (step 302).

[0075] For example, the system can select initial relevant points based on at least two criteria: (i) the proximity of one or more points to each other at the last image in the task segment of each demonstration image sequence, based on point tracking data, and (ii) the degree of stillness of one or more points during the task segment, based on point tracking data.

[0076] Criterion (i) refers to selecting points that end at a common image location across all presentations. That is, the system can calculate the positional variance of the final point location across presentations and select those points associated with positional variances below a threshold. In other words, points with similar final point locations across each presentation image sequence can be selected.

[0077] For example, for a task segment corresponding to a template, i.e., placing an object into its corresponding hole of appropriate shape, the point associated with the object being placed will most likely satisfy criterion (i) because, across the presentation, the object to be placed can begin in various environmental locations, but always ends in the hole of appropriate shape at the end of the task segment. Therefore, the point associated with the block will also begin in various environmental locations, but end near a common location at the end of the task.

[0078] In criterion (ii), stationary refers to a tracking point whose overall motion is less than a threshold during task segmentation. That is, a measure of point movement relative to frames of the image is defined, rather than movement relative to a third-person viewpoint of the scene, and points with associated motion measures above a threshold are selected across the demonstration.

[0079] For example, when a camera is mounted on a robot end effector for a matching task, points associated with the end effector within the image will not be selected because the end effector has not moved within the frame of the image.

[0080] As a specific example of determining relevant points from point tracking data based on criteria (i) and (ii), the system can select points sequentially according to a sequence of parameterized rules corresponding to the criteria. For example, the system can first select points that satisfy a first rule corresponding to criterion (ii), such as “select points whose position variance is greater than or equal to a specific parameter value across all demos during task segmentation.” Then, evaluating only the points that satisfy criterion (ii), the system can select points that satisfy a second rule corresponding to criterion (i), such as “select points whose position variance within the final frame is less than the parameter value across all demos.”

[0081] In some implementations, the system also considers other criteria when selecting the initial point.

[0082] For example, in addition to (i) and (ii) above, the system can also select one or more points based on (iii) whether one or more points are visible in the last image of the task segment in each demonstration image sequence according to point tracking data.

[0083] Criterion (iii) refers to points that are not guaranteed to be visible at the end of a task segment due to inconsistencies caused by occlusion, sensor failure, or inaccurate presentation.

[0084] For example, for a task segment corresponding to a template, the square block to be moved may have a unique marker on a single face, with a corresponding point that is particularly easy to track (assuming the block face is always visible). However, because the uniquely marked block face may not always be visible (due to the marked block face being facing down or away from the camera face), the corresponding point of the unique marker will not be selected for use in criterion (iii).

[0085] As a specific example of determining relevant points from point tracking data based on criteria (i), (ii), and (iii), the system can sequentially select points according to a sequence of parameterized rules corresponding to the criteria. For example, after sequentially selecting points that satisfy criteria (i) and (ii) according to the parameterized rules described above, the system can then select points from the remaining set that satisfy a third rule corresponding to criterion (iii), such as “selecting points whose average visibility (i.e., average occlusion score) is higher than the parameter value across the entire task segment.”

[0086] In some implementations, the system can then use the initial correlation points to generate correlation points for task segments.

[0087] For example, the system can select the initial correlation points that were previously selected to satisfy criteria (i), (ii) and (iii) as correlation points.

[0088] The system uses point tracking data to cluster multiple points to determine multiple clusters (step 304). Multiple points refer to all tracking points, not just those points designated as initial relevant points from step 302.

[0089] The system can use any of a variety of methods to cluster multiple points.

[0090] For example, the system can use a "point clustering based on 3D motion estimation and reprojection" method for clustering. That is, the system assumes that all points belong to one of several approximately rigid objects in the scene, and their motion can be explained by a set of 3D motions followed by reprojection (i.e., projections from the point's 3D position to its 2D position); for each image in each presentation, the 3D positions of the points and their 3D transformations are parameterized; and the reprojection error function is minimized to determine how many clusters a point belongs to and which clusters it belongs to, such as those clusters that minimize the error function. For a gripper performing a matching task, the reprojection error function could be, for example:

[0091] in It is a point i Time during the demonstration t The predicted location (for simplicity) t(Indexing both time and presentation), and It is the probability of occlusion. Threshold version, k This refers to the number of rigid objects in the scene. It is the 3D position of the i-th point in the k-th object. R(x) is the rigid 3D transformation of each object at each time, and R(x) is the reprojection function that projects the 3D point onto the 2D plane: R(x) = [x[0] / x[2], x[1] / x[2]], where x[0], x[1], and x[2] are the x, y, and z coordinates in 3D, and x[0] / x[2] and x[1] / x[2] are the normalized x and y coordinates in the 2D plane.

[0092] For the previous example, and Both can be parameterized using a neural network designed to capture inductive preferences, namely, that nearby points in 2D space and nearby frames in time should have similar 3D configurations. Specifically, ,in Neural network for output matrix Perform parameterization, and ,in It is the time-smooth learning descriptor for image t. It is a neural network that outputs tensors representing rigid transformations. The neural network parameters.

[0093] Furthermore, for the previous example, the optimal number of rigid objects can be determined using the 'recursive split' method. To achieve this, note that it's possible to have only two neural networks. and The final linear projection layer depends on the number of clusters k: for a given number of channels c, the parameters of these layers can be written as matrices. For each such weight matrix, the system creates two new weight matrices. and , of which The new clustering is parameterized, where The The rows have been split into two distinct clusters, referred to as 'bifuctures' of the original weight matrix. The system calculates the loss under each possible split and optimizes the split with the minimum loss. Mathematically, Define a new matrix, where the w-th element has been removed. Okay, and it has been attached. and The first of both Okay. The system can be used. To calculate two new 3D positions and 3D transformations and Then minimize the following loss:

[0094] here, It is the reprojection error function from the previous example. For output and The neural network is parameterized, and includes and 'bifurcation' variable and Both of these. After multiple (e.g., fifty, one hundred, five hundred, or more generally, several hundred) optimization steps, the system will Replace with and create a new 'fork' in the matrix (using (Small perturbation initialization fork). The system starts from... Start by recursively branching and repeat the process until the desired number of objects is reached or the loss is minimized.

[0095] The system uses clustering and initial relevant points to select relevant points (step 306).

[0096] For example, given a cluster, each initial relevant point can vote (e.g., assign) a vote for the cluster, and the cluster with the largest number of votes is merged, and this process is repeated until one or more criteria are met, such as the number of clusters, the number of points in a cluster, etc. The relevant point can then be selected as belonging to one of those clusters that satisfy one or more criteria, such as the cluster with the largest number of initial points, or the cluster whose initial points have undergone the most movement, etc.

[0097] As another example, the system can use the generated motion-based clustering data to perform motion-based object segmentation, select relevant objects from the segmentation, and choose initial relevant points on the selected objects as relevant points. That is, the system can assume the existence of... k An object is generated, and step 304 is parameterized to produce an object that is similar to... k Each object corresponds to k There are 10 clusters. More specifically, to minimize the reprojection error, the number of clusters is fixed at 100. k Alternatively, when using initial points to vote and merge clusters, clusters can be merged or split until a result is obtained. k Each cluster can be used to generate and k Each object corresponds to k Constraints for each cluster.

[0098] The system can use one or more criteria to... k Select relevant objects from the clusters, such as the cluster with the most points that also satisfy the above criteria (i), (ii), or (iii) or any combination of these criteria.

[0099] The system can select points on the selected relevant object being manipulated as multiple relevant points. For example, a relevant point is a point on the relevant object that satisfies criteria (i), (ii), and (iii) and has an occlusion score exceeding a specific threshold.

[0100] Figure 4 Example 400 image sequences are shown, illustrating an example process for determining relevant points for a task segment involving a robot with a gripper equipped with a camera. More specifically, each image in the example 400 image sequence illustrates a tracking point overlaid on the last image of the task segment that moves the gripper onto a cylindrical block.

[0101] Image 402, labeled 'Input', illustrates all tracking points throughout the task segment. Image 404, labeled 'Low Cross-Demonstration Variance', illustrates tracking points that terminate at a common image location across all demos, while image 406, labeled 'Non-stationary', illustrates tracking points whose overall motion exceeds a threshold during the task segment, and image 408, labeled 'Motion Clustering', illustrates tracking points clustered into groups based on objects (i.e., tracking points) in the image according to the '3D Motion Estimation and Reprojection-Based Point Clustering' method described previously for K objects. The tracking points present in images 404-408 are those present in image 402 that satisfy the corresponding criteria for images 404-408. Image 410, labeled 'Output', illustrates the determined relevant points as the intersection of the tracking points present in images 404-408.

[0102] Figure 5 This is a flowchart of an example process 500 for controlling an intelligent agent using an intelligent agent control system. For convenience, process 500 will be described as being executed by a system of one or more computers located in one or more locations. For example, an intelligent agent control system appropriately programmed according to this specification (e.g., Figure 1 The intelligent agent control system 100 can execute process 500.

[0103] The system acquires multiple demonstration image sequences, each of which is an image sequence of a corresponding instance of the agent performing a task (step 502). For example, the system may acquire the demonstration image sequences from a camera directly mounted on the agent (e.g., a robot) to provide a first-person view of the task, or, if the mounted camera is a 360-degree camera, to provide a complete view of the agent's surrounding environment during the task.

[0104] As another example, the system can obtain a sequence of demonstration images from a camera mounted on the end effector (e.g., gripper) of the robotic agent, thereby providing an end effector view of the task.

[0105] In some cases, the system can obtain a sequence of demonstration images from a camera associated with the environment rather than the agent. For example, the system can obtain images from an overhead camera, thus providing a bird's-eye view of the task.

[0106] As another example, the system can obtain a sequence of demonstration images that provide a third-person view of the task. For instance, the system can obtain a sequence of demonstration images from an environment-mounted (e.g., wall-mounted or tripod-mounted) camera to provide a specific third-person view of the task.

[0107] The system generates data that divides the task into multiple task segments, each task segment including a corresponding part of each demonstration image sequence in the demonstration image sequence (step 504).

[0108] The system can use any of a variety of methods to generate task segments. Generally, these methods involve aligning multiple demo image sequences, i.e., synchronizing the demo image sequences such that corresponding events or actions across the sequences occur at matching or nearly matching images (e.g., at matching or nearly matching time or frame numbers), and then dividing each demo image sequence in the demo image sequence into an equal number of task segments.

[0109] As an example of alignment demonstration, the system can use a neural network trained using a self-supervised representation learning method to perform alignment demonstration. An example of such a method is described in ArXiv: 1904.07846.

[0110] As another example, the system can use events or actions to align the demonstration, such as aligning based on robot pose information (e.g., gripper movements and forces).

[0111] As an example of segmentation, once the system is aligned with the demo, it can create task segments based on a fixed number of images. That is, each demo in the demo can be divided into task segments such that each task segment contains an equal number of images.

[0112] As another example, the system can use scene information contained in the images to segment the presentation. That is, one or more neural networks that learn the embedded representations of the presentation image sequence can be used to process the presentation and generate task segments based on significant changes in the scene.

[0113] As another example, the system can use events or actions to create task segments, such as segmentation based on robot pose information (e.g., gripper movements and forces). That is, the system can segment each sequence of demonstration images based on the position of a specified component of the robot, based on the forces applied to the robot, or both. Specifically, the system can segment each sequence of demonstration images based on one or both of the following: the position of a specified component of the robot and the forces applied to the robot.

[0114] For each task segment, the system applies the point tracker to each of a plurality of points in the image of the corresponding part of the image sequence to generate point tracking data for that task segment (step 506).

[0115] The system can use any of a variety of point tracking methods as a point tracker. For example, the system can use a keypoint-based method, which involves defining a set of unique “keypoints” for an object class and identifying keypoints in each image to track random points that happen to be keypoints. Examples of such methods are described in ArXiv:2112.04910 and ArXiv:1806.08756.

[0116] As another example, the system can use optical flow methods (i.e., methods that involve tracking points based on changes in pixel intensity between two consecutive images) to track points.

[0117] As another example, the point tracker can be a neural network-based point tracker. For each of a plurality of points, the neural network-based point tracker can extract the query features of the point, use the query features of the point and the corresponding visual features of each of the multiple spatial locations in the images in the task segment to generate a corresponding initial point tracking prediction for each point in the images in the task segment, and use a temporal refinement subnetwork to refine the corresponding initial point tracking prediction to generate point tracking data.

[0118] Examples of neural network-based point trackers are TAPNet, described in ArXiv:2211.03726, TAPIR, described in ArXiv:2306.08637, and BootsTap, described in ArXiv:2402.00847.

[0119] Using any appropriate point tracker, the system can use the point tracker to generate point tracking data for each task segment. For each tracked point and for each image in the task segment, the point tracking data includes (i) the predicted location of the point in the image and (ii) a predicted occlusion score indicating the probability that the point is occluded in the image.

[0120] The system uses the point tracking data of each task segment to determine multiple relevant points for that task segment (step 508). For example, the system can determine relevant points, such as reference... Figure 3 As described.

[0121] The system receives a request to execute a new instance of the task (step 510). For example, the system may receive a request from a user or from an external system.

[0122] The system uses (i) images captured when the agent performs a new instance of a task and (ii) relevant points of the task segment to control the agent to perform a new instance of a task (step 512).

[0123] For example, at each time point during the execution of a new instance of a task, the system can receive an image captured at that time step; identify the current task segment corresponding to the image; determine the target point from the relevant points of the current task segment; determine the predicted target location associated with completing the task; cause the agent to perform an action predicted to move the target point to the predicted target location; and then, after performing the action, receive and process a new current image to repeat the previous steps until the task is completed.

[0124] The following text is for reference only. Figure 6 Further details describe the new instance of the task being performed.

[0125] Figure 6 This is a flowchart of an example process 600 for performing a task by controlling an agent at each time step in a sequence of time steps during the task. For convenience, process 600 will be described as being performed by a system of one or more computers located in one or more locations. For example, an agent control system appropriately programmed according to this specification (e.g., Figure 1 The intelligent agent control system 100 can execute process 600.

[0126] The system can execute iterations of process 600 at each time step during the execution of a new instance of the task. That is, the system can continue to execute iterations of process 600 until the new instance of the task is completed.

[0127] The system obtains an image of the agent at a time step (step 602). For example, the image may come from the same source as the demonstration image sequence, i.e., the camera associated with the agent performing the task, as previously referenced. Figure 5 As described.

[0128] The system identifies the current task segment of the time step (step 604).

[0129] The system can then identify the current task segment by recognizing whether the criteria for terminating the task segment of the previous time step have been met. If the corresponding criteria have not yet been met, the system can set the current task segment as the task segment of the previous time step; if the corresponding criteria have been met, the system can set the current task segment as the next task segment after the task segment of the previous time step.

[0130] For example, the system can identify the current task segment by recognizing whether the criteria used for task segmentation and time step values ​​(e.g., the occurrence of an event or action, such as robot posture information, such as gripper motion and force) have been met.

[0131] For the first time step, the current task segment can be set as the default task segment, such as the first task segment created by the system during "extraction mode".

[0132] In other cases, for the first time step, the system determines the current task segment by processing the first received image and determining which task segment the image is most likely to be associated with.

[0133] The system determines one or more target points from the relevant points of the current task segment (step 606). In order to determine the target points from the relevant points, the system can use the previous point tracker in an online manner, that is, identify the target points as tracking points as relevant points.

[0134] For example, in order to use a neural network-based point tracker online, the neural network-based point tracker is modified to be causal. That is, the neural network-based point tracker is modified to determine the target point from relevant points and only from the image associated with the current time step and all previous time steps of the task segment.

[0135] As a specific example, the TAPIR point tracker can be used online by modifying its temporal refinement subnetwork to apply causal convolutions instead of temporal convolutions, as described in ArXiv: 2308.15975. More specifically, the temporal refinement of the TAPIR model uses a depthwise convolutional module, where the query point features, x and y positions, occlusion and uncertainty estimates, and score map for each image are concatenated into a single sequence, and the convolutional model outputs updates for position and occlusion. The depthwise convolutional model replaces each depthwise layer in the original model with a causal depthwise convolution; therefore, the resulting model has the same number of parameters as the original TAPIR model, with all hidden layers having the same shape.

[0136] The system determines the corresponding target prediction position in the future image at a future time step for each target point in the current task segment based on the point tracking data (step 608).

[0137] For example, the system can measure the Euclidean distance between the target point and the relevant point in each image of each demonstration image sequence to select the image with the lowest distance (e.g., the lowest average distance or net distance) as the future image. In this case, the target predicted location of each target point in the target point sequence is a relevant point in the future image.

[0138] In some cases, as in the previous example, there exists a threshold distance value such that a future image is selected as the follower image in the corresponding sequence of demonstration images that has the lowest Euclidean distance from the image.

[0139] Furthermore, in some cases, for the same previous example, the target predicted location of each target point in the target points is determined as the average correlation point associated with future images for a future time step demonstration.

[0140] The system causes the agent to perform an action predicted to move one or more target points to their corresponding predicted target locations (step 610). The system can use any of a variety of suitable controllers, i.e., algorithms that process images and points to generate actions for the agent, making the target points more aligned with the predicted target locations.

[0141] Generally, any position-based visual servoing technique (i.e., including techniques that convert image data into real-world 3D pose data to determine actions) or image-based visual servoing (i.e., including techniques that convert image data into image features to determine actions) can be used to enable an agent to perform actions predicted to move one or more target points to corresponding target predicted positions.

[0142] For example, in the case of a robotic arm agent moving a gripped block to perform a matching task, the action corresponding to the arm's movement speed can be determined by a controller that calculates a Jacobian matrix (i.e., an estimate of how the position of the target point corresponds to a change in the robotic arm's movement speed) and uses this Jacobian matrix to calculate the action corresponding to the movement speed that minimizes the squared error between the target point and the predicted target position under a linear approximation.

[0143] As a more specific example, consider the case where a robotic arm agent moves a gripped block to perform a matching task, given a set of target predicted positions. and the corresponding target point In this case, the controller uses mapping of actions to The system calculates the action that minimizes the error by using a linear approximation of a function of change. Then, the system instructs the agent to execute the calculated action. This process can be summarized as follows:

[0144] Where t represents the time step. It is the gripper speed (i.e., the action), and It is the image Jacobian matrix.

[0145] Figure 7 Example 700 of a task performed by a robot with a gripper equipped with a camera using the described technology is shown.

[0146] Specifically, Example 700 shows that the described technology can be used to control a robot to perform any of a variety of tasks, even when relatively few demonstrations of successfully performed tasks are available.

[0147] For example, Example 700 shows that the system can perform a “four-object stack” using only four demos, a significant improvement over the dozens to hundreds of demos that might be required using other methods.

[0148] In this context, intelligent agents can interact, and intelligence is demonstrated in some examples of the environment they follow.

[0149] In some implementations, the environment is a real-world environment, and the agent is a mechanical agent that interacts with the real-world environment. For example, the agent could be a robot that interacts with the environment to accomplish a goal, such as locating an object of interest in the environment, moving the object of interest to a specified location in the environment, physically manipulating the object of interest in the environment in a specified manner, or navigating to a specified destination in the environment; or the agent could be an autonomous or semi-autonomous land, air, or sea vehicle that navigates through the environment to a specified destination in the environment.

[0150] Actions can be control inputs for controlling a robot, such as torques or higher-level control commands for the robot's joints; or they can be control inputs for controlling autonomous or semi-autonomous land, air, or sea vehicles, such as torques or higher-level control commands to the vehicle's control surfaces or other control elements.

[0151] In other words, actions can include, for example, position, velocity, or force / torque / acceleration data of one or more joints of a robot or parts of another mechanical agent. Actions may additionally or alternatively include electronic control data such as motor control data, or more generally data for controlling one or more electronic devices within an environment, the control of which has an impact on the observed state of the environment. For example, in the case of autonomous or semi-autonomous land, air, or sea vehicles, actions can include actions for controlling navigation (e.g., steering) and movement (e.g., braking and / or acceleration of the vehicle).

[0152] In some implementations, the environment is a simulated environment, and the agent is implemented as one or more computer programs that interact with the simulated environment. For example, the environment can be a computer simulation of a real-world environment, and the agent can be a simulated mechanical agent that navigates through the computer simulation.

[0153] For example, the simulated environment can be a motion simulation environment, such as a driving simulation or a flight simulation, and the intelligent agent can be a simulated vehicle navigating through the motion simulation. In these implementations, the action can be a control input controlling the simulated user or the simulated vehicle. As another example, the simulated environment can be a computer simulation of a real-world environment, and the intelligent agent can be a simulated robot interacting with the computer simulation.

[0154] Typically, when the environment is simulated, actions can include simulated versions of one or more of the previously described actions or action types.

[0155] Although this specification generally describes the input as an image, in some cases the input may include additional data in addition to or in lieu of image data, such as proprioceptive data and / or force data characterizing the agent, or other data captured by other sensors of the agent.

[0156] This specification uses the term "configured" in conjunction with system and computer program components. For a system of one or more computers to be configured to perform a specific operation or action, this means that the system has software, firmware, hardware, or a combination thereof installed thereon that causes the system to perform those operations or actions in operation. For one or more computer programs configured to perform a specific operation or action, this means that one or more programs include instructions that, when executed by a data processing device, cause that device to perform that operation or action.

[0157] Embodiments of the subject matter and functional operation described in this specification may be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their equivalents), or in a combination of one or more of these. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these. Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals (e.g., machine-generated electrical, optical, or electromagnetic signals) that are generated to encode information for transmission to a suitable receiver device for execution by the data processing device.

[0158] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, apparatuses, and machines used for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The device may also be or further include a dedicated logic circuit system, such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0159] A computer program (which may also be referred to or described as a program, software, software application, app, module, software module, script, or code) can be written in any form, including compiled or interpreted languages ​​or declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not need to, correspond to a file in a file system. A program may be stored as a part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), as a single file dedicated to the program in question, or as a plurality of coordinated files (e.g., a file storing portions of one or more modules, subroutines, or code). A computer program can be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected via a data communication network.

[0160] In this specification, the term "database" is used broadly to refer to any collection of data: data that does not need to be structured in any particular way, or does not need to be structured at all, and can be stored on storage devices in one or more locations. Thus, for example, an indexed database may include multiple collections of data, each of which can be organized and accessed differently.

[0161] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines may be installed and run on the same one or more computers.

[0162] The processes and logic flows described in this specification can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by manipulating input data and generating outputs. The processes and logic flows can also be executed by a dedicated logic circuit system (e.g., an FPGA or ASIC) or by a combination of a dedicated logic circuit system and one or more programmable computers.

[0163] A computer suitable for executing computer programs may be based on a general-purpose or special-purpose microprocessor or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory or random access memory or both. The basic components of a computer are the central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into a special-purpose logic circuit system. Generally, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from or transfer data to one or more mass storage devices or both. However, a computer need not have such devices. Furthermore, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0164] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD ROMs and DVD-ROMs.

[0165] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from a web browser. Additionally, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and receiving response messages from the user in response.

[0166] Data processing devices used to implement machine learning models may also include, for example, dedicated hardware accelerator units for handling the general and computationally intensive portions of machine learning training or production (i.e., inference, workloads).

[0167] Machine learning models can be implemented and deployed using machine learning frameworks (such as the TensorFlow framework or the Jax framework).

[0168] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a client computer having a graphical user interface, web browser, or app that a user can interact with through an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. Components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0169] A computing system may include clients and servers. Clients and servers are generally geographically separated and typically interact via a communication network. The client-server relationship is established by computer programs executed on respective computers that establish a client-server relationship between them. In some embodiments, the server transmits data (e.g., HTML pages) to a user device, for example, for the purpose of displaying data to a user interacting with the device acting as a client and receiving user input from that user. Data generated at the user device, such as the result of user interaction, may be received at the server from the device.

[0170] While this specification contains numerous details of specific implementations, these details should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features that may be characteristic of particular embodiments of a particular invention. Certain features described in the context of individual embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases one or more features from that claimed combination may be removed, and the claimed combination may involve sub-combinations or variations thereof.

[0171] Similarly, although operations are depicted in the accompanying drawings and described in a specific order in the claims, this should not be construed as requiring such operations to be performed in the specific order shown or in a sequential order, or requiring all shown operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0172] Specific embodiments of this subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions described in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require a specific order or sequence shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method executed by one or more computers, the method comprising: Obtain multiple demonstration image sequences, each of which is a sequence of images of a corresponding instance of the agent performing a task; Generate data that divides the task into multiple task segments, each task segment including a corresponding part of each demonstration image sequence in the demonstration image sequence; For each task segment, a point tracker is applied to each of a plurality of points in the image in the corresponding portion of the image sequence to generate point tracking data for the task segment; For each task segment, the point tracking data of the task segment is used to determine multiple relevant points of the task segment; Receive a request to perform a new instance of the task; as well as The agent is controlled to execute the new instance of the task using (i) the image captured when the agent performs the new instance of the task and (ii) the relevant points of the task segment.

2. The method according to claim 1, wherein the intelligent agent is a robot.

3. The method of claim 2, wherein each image in each of the demonstration image sequences is captured by the robot's camera.

4. The method according to claim 2 or claim 3, wherein generating data that divides the task into multiple task segments comprises: Each sequence of demonstration images is divided into parts based on the location of a specified component of the robot, based on the force applied to the robot, or both.

5. The method of claim 4, wherein the robot has a gripper, and wherein dividing each demonstration image sequence into portions comprises: Each sequence of demonstration images is divided into parts based on the force applied to the gripper, the position of the gripper, or both.

6. The method of claim 5 when subordinate to claim 3, wherein the camera is positioned on the gripper of the robot.

7. The method according to any preceding claim, wherein the point tracking data for each task segment comprises, for each of the plurality of points and for each image of the task segment: (i) The predicted location of the point in the image; as well as (ii) The predicted occlusion score of the point, indicating the probability that the point is occluded in the image.

8. The method according to any preceding claim, wherein using (i) an image captured when the agent performs the new instance of the task and (ii) the relevant points of the task segment to control the agent to perform the new instance of the task comprises, at each of a plurality of time steps: Obtain the image of the agent at the time step; Identify the current task segment at the given time step; Determine one or more target points from the relevant points of the current task segment; The corresponding predicted target position in the future image at a future time step is determined from the point tracking data of the current task segment; as well as The agent is made to perform an action that is predicted to move the one or more target points to the corresponding target prediction position.

9. The method of claim 8, wherein determining the corresponding predicted target location in a future image at a future time step for each of the target points from the point tracking data comprises, at a first time step: The point tracker is applied to the image at the first time step to generate the predicted location of the one or more target points in the image; Based on the predicted positions of the one or more target points in the image, a target image is selected from a specific portion of a specific demonstration image sequence in the current task segment; Based on the point tracking data of one or more target points in subsequent images following the target image in the specific demonstration image sequence, the predicted position is selected as the corresponding target position of the one or more target points.

10. The method of claim 9, wherein determining the corresponding predicted target location in a future image at a future time step for each of the target points from the point tracking data comprises, at a second time step: The point tracker is applied to the agent in the image at the second time step to generate the predicted locations of the one or more target points in the image; and Based at least on the distance between the predicted positions of the one or more target points in the image and the predicted positions based on the point tracking data of the one or more target points in the subsequent image, it is determined whether to update the target image to the subsequent image and select a new target predicted position for each of the target points.

11. The method according to any preceding claim, wherein using the point tracking data of the task segment for each task segment to determine a plurality of relevant points of the task segment comprises: The point tracking data is used to determine the relevant objects being manipulated during the task segmentation; as well as Select points on the relevant object being manipulated as the plurality of relevant points.

12. The method according to any preceding claim, wherein using the point tracking data of the task segment for each task segment to determine a plurality of relevant points of the task segment comprises: One or more points are selected as initial relevant points based at least on: (i) the degree of proximity of the one or more points to each other at the last image of the task segment in each of the demonstration image sequences in the demonstration image sequence, according to the point tracking data, and (ii) the degree of stillness of the one or more points during the task segment, according to the point tracking data; as well as The initial correlation points are used to generate the correlation points.

13. The method of claim 12, wherein selecting one or more points as initial correlation points comprises: The selection of the one or more points is based at least on: (i) the degree of proximity of the one or more points to each other at the last image of the task segment in each of the demonstration image sequences, according to the point tracking data; (ii) the degree of stillness of the one or more points during the task segment, according to the point tracking data; and (iii) whether the one or more points are visible at the last image of the task segment in each of the demonstration image sequences, according to the point tracking data.

14. The method of claim 12 or 13, wherein generating the correlation points using the initial correlation points comprises: The point tracking data is used to cluster the multiple points to determine multiple clusters; as well as The clustering and the initial relevant points are used to select the relevant points.

15. The method according to any preceding claim, wherein the point tracker is a neural network-based point tracker, and for each of the plurality of points, the neural network-based point tracker is configured to: Extract the query features of the points; The query features of the points and the corresponding visual features of each spatial location among multiple spatial locations in the images of the task segment are used to generate corresponding initial point tracking predictions for the points in each image of the images of the task segment. as well as A time-refinement subnetwork is used to refine the corresponding initial point tracking prediction to generate the point tracking data.

16. The method of claim 15, wherein the temporal refinement subnetwork applies causal temporal convolution to refine the corresponding initial point tracking prediction of the point.

17. A system comprising one or more computers and one or more storage devices storing instructions, the instructions, when executed by the one or more computers, causing the one or more computers to perform operations according to any one of claims 1 to 16.

18. A computer storage medium storing one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform the operation of a corresponding method according to any one of claims 1 to 16.

Images