Intention tracking for collaborative robot manipulation

Abstract

A system for human-robot collaboration is provided herein. The system includes a robotic device having at least one mechanism for moving at least a part of the robot, one or more sensors configured to capture environmental data, a processor in communication with the robotic device and the one or more sensors, and a memory in communication with the processor. The memory has instructions that, when executed, cause the processor to capture an image of a scene, detect objects, generate manipulation options, receive a prompt indicating an overall objective, generate a list of task steps, generate a task graph, determine a probability for each task step, determine a predicted next intention step, and execute a manipulation option corresponding to the next intention step via the robotic device.

Description

INTENTION TRACKING FOR COLLABORATIVE ROBOT MANIPULATIONCROSS-REFERENCE TO RELATED APPLICATION(S)

[0001] This application claims priority to and incorporates by reference U.S. provisional patent application no. 63 / 653,561, filed May 30, 2024, the content of which is hereby incorporated by reference in its entirety.STATEMENT OF GOVERNMENT SUPPORT

[0002] This invention was made with government support under 2143435 awarded by the National Science Foundation. The government has certain rights in the invention.TECHNICAL FIELD

[0003] The technology7discussed below relates to collaborative manipulation of robots to perform tasks.BACKGROUND

[0004] Collaborative manipulation is the task of leveraging human and robot capabilities to jointly perform tasks. For example, in an assembly task, humans and robots may be given different subtasks (e g., a robot precisely places two objects while a human checks the robof s work and sorts parts).

[0005] When the philosophy of grounding natural language commands into robot control policies is applied to human-robot collaboration (HRC), the human user may have to have a conversation with the robot at each step of the long-horizon task. This situation rarely happens in human-human collaboration, as a human is able to track the progress on the partner’s side based on their shared knowledge over the task. For example, a worker rarely has to have a conversation with a co-worker in a collaborative assembly task on which they have collaborated many times, and a sous chef rarely has to have a conversation w ith the chef when creating a regular dish together. Therefore, there is a need for systems and methods that model long-term behavior of a human user that can be integrated into an large language model (LLM)-driven collaborative robot framework.SUMMARY

[0006] According to an aspect of the present disclosure, a system for human-robot collaboration is provided. The system includes a robotic device having at least one mechanism for moving at least a part of the robot, one or more sensors configured to capture environmental data, a processor in communication with the robotic device and the one or more sensors, and a memory in communication with the processor and having instructions stored thereon that, when executed, cause the processor to: capture, via the one or more sensors, an image of a scene where the robotic device is located; detect a plurality of objects in the scene based on the image; generate a plurality of manipulation options corresponding to each object in the plurality of objects; receive a prompt indicating an overall objective; generate a list of task steps based on the prompt and the plurality of objects; generate a task graph corresponding to the list of task steps based on a reversibility check between each task step in the list of task steps; determine a probability of each task step in the list of task steps corresponding to a behavior of a human in the scene captured by the sensor; determine a predicted next intention step based on the probability of each task step in the list of task steps; and execute, via the robotic device, a manipulation option from the plurality of manipulation options corresponding to the next intention step.

[0007] According to another aspect of the present disclosure, a method for collaborative task execution is provided. The method includes detecting, by one or more sensors, a plurality of objects in a scene; generating, by a processor, a plurality of manipulation options corresponding to each object in the plurality of objects; receiving, at the processor, a prompt indicating an overall objective; generating, by the processor, a list of task steps based on the prompt and the plurality of objects; generating, by the processor, a task graph corresponding to each task step in the list of task steps; generating, with the processor, a probability of one or more humans performing each task step in the list of task steps when a human behavior in the scene is captured by the sensor; determining, with the processor, a predicted next intention step of the one or more humans based on the probabilities included in each task graph; and executing, by a robot, a manipulation option from the plurality of manipulation options, wherein the executed manipulation option corresponds to the predicted next intention step.BRIEF DESCRIPTION OF THE DRAWINGS

[0008] FIG. 1 is a flow diagram illustrating an example process for planning and controlling the movement of a robot, according to some embodiments.

[0009] FIG. 2 is a block diagram illustrating an example collaborative robot system, according to some embodiments.

[0010] FIG. 3 is a diagram illustrates language-driven intention tracking (LIT) based collaborative robot framework, according to some embodiments.

[0011] FIG. 4 is a graphical model for intention tracking, according to some embodiments.

[0012] FIG. 5 illustrates snapshots and charts associated with different similarity metrics, according to some embodiments.DETAILED DESCRIPTION

[0013] The detailed description set forth below in connection with the appended drawings is intended as a description of various configurations and is not intended to represent the only configurations in which the subject matter described herein may be practiced. The detailed description includes specific details to provide a thorough understanding of various embodiments of the present disclosure. However, it will be apparent to those skilled in the art that the various features, concepts, and embodiments described herein may be implemented and practiced without these specific details. In some instances, well- known structures and components are shown in block diagram form to avoid obscuring such concepts.

[0014] The disclosure provided below will first provide general processes for mapping an unknown environment according to the novel mapping / planning approach herein, for motion planning development, and for runtime processes of using such approaches for robotic navigation and motion. After these general descriptions, a discussion of the inventors’ validation work will be provided. While the validation work regards certain specific attributes or embodiments, it should be recognized that these efforts are also intended to provide a quantification of how the techniques herein improve upon the field of mapping and motion planning. Thus, the Examples and Experiments described herein are not meant to be limiting of the scope of the disclosure, but rather as illustrative examples of real world advantages and benefits of this disclosure.

[0015] The systems and methods described herein are directed to Language-driven Intention Tracking (LIT) to model long-term behavior of the human user, and integrates LIT into an LLM-driven collaborative robot framework. LIT extends intention tracking by applying an LLM to model measurement likelihood and transition probabilities in the probabilistic graphical model of human intentions, which is defined by grounding an overall task prompt (e.g., “make a salad”) with understanding of the scene using LLMs and vision language models (VLMs). By harnessing the power of foundation models such as LLMs and VLMs, the LIT framework can be generalized to any collaborative task. In various embodiments, the LIT framework may be applied to different collaborative domains such as manufacturing, fulfillment, distribution, domestic assistive robotics, cooking, or surgery. In one non-limiting example, the effectiveness of the LIT framework is demonstrated in a scenario where the collaborative robot acts as a sous-chef to assist a human user in cooking. The framework may also adapt to new environments by leveraging the broad knowledge encoded in foundation models.

[0016] The present disclosure thus relates to systems and methods for enhancing human-robot collaboration through proactive assistance. The LIT framework leverages advanced language models to understand and predict human behavior in collaborative tasks. In some cases, the LIT framework may utilize LLMs and VLMs to interpret the overall context of a task and track human intentions over time. This approach may allow a robot to anticipate a human collaborator's needs and take supportive actions without requiring explicit instructions at each step. The LIT framework may also extend beyond traditional intention tracking methods by incorporating natural language processing capabilities. In some implementations, this may enable more flexible and generalizable collaboration across diverse tasks and scenarios.

[0017] By modeling long-term human behavior and predicting future intentions, the disclosed systems and methods may facilitate smoother coordination between humans and robots. This proactive collaboration paradigm may reduce the need for constant verbal communication or manual prompting during joint tasks. Human-robot collaboration enhanced by LIT may aim to mimic the intuitive coordination often seen in human-human teamwork, where partners can anticipate each other’s needs based on shared context and experience. This may lead to more natural and efficient joint task completion in various applications.

[0018] FIG. 1 is a flow diagram illustrating an example process 100 for planning and controlling the movement of a robot, according to some embodiments, in accordance with some aspects of the present disclosure. As described below, a particular implementation can omit some or all illustrated features / steps, may be implemented in some embodiments in a different order, and may not require some illustrated features to implement all embodiments. In some examples, an apparatus, such as a robot, in connection with the system illustrated in FIG. 2 (described below) can be used to perform example process 100. However, it should be appreciated that any suitable apparatus or means for carrying out the operations or features described below may perform process 100.

[0019] At step 105, the process 100 detects a plurality of objects in a scene. In some examples, one or more sensors may be used to capture images, video stream, or other data associated with an environment. In one example, an overhead camera coupled to a robot performing process 100 may capture images of objects within a pre-defined proximity to the robot. In some examples, a VLM, or any other suitable model, may be used to name and describe each object in the scene. For example, the VLM may detect an object name, an object orientation, an object size, object dimensions, object shape, or the like.

[0020] At step 110, the process 100 generates a plurality' of manipulation options corresponding to each object in the plurality of objects. The manipulation options may enable a robot to perform a task within the environment (with or without the use of one or more objects) or provide one or more objects to a collaborating human. In some examples, the manipulation options may include movements such as object grasps, object rotations, object relocations, object motions, etc. In one non-limiting example, the manipulation options may include grasping actions, in which the robot may be capable of grasping objects using different end effector configurations such as parallel jaw grippers, suction cups, or multi-fingered hands. For instance, the grasping actions may include one or more top-down grasp poses a robot arm can execute to interact with any object in the environment. The grasping manipulation option may be adapted based on the size, shape, and material properties of the target object.

[0021] By way of further example, the manipulation options may include pushing actions. For objects that are difficult to grasp directly, the robot may employ pushing motions to reposition items on a surface. This may involve using the side of an end effector or a specialized tool to apply controlled force. As another example, the manipulation options may include lifting actions. Once an object is securely grasped, the robot may lift it to adesired height. The lifting motion may be smooth and controlled to avoid destabilizing the held item. The manipulation options may also include a placing action. In these instances, the robot may place grasped objects at specified locations with precision. This may involve gentle placement to avoid damaging delicate items. As other examples, the manipulation options may also include pouring actions (e.g., for handling liquids or granular materials, the robot may execute pouring motions by tilting containers at controlled angles and rates), stirring actions (e.g., using an appropriate end effector like a spoon or spatula, the robot may perform stirring motions in containers to mix ingredients or materials), cutting actions (e.g., with a suitable cutting tool attachment, the robot may make incisions or slice materials when needed for food preparation or manufacturing tasks), screwing or unscrewing actions (e.g., the robot may manipulate objects with rotational motions, such as tightening or loosening screws and bolts using an appropriate bit or socket), folding actions (e.g., for handling fabrics or flexible materials, the robot may execute folding motions to arrange items neatly), and / or aligning actions (e.g., the robot may adjust the orientation and position of objects to align them precisely with other components or markings). In some cases, these manipulation options may be combined in sequences to accomplish more complex tasks. The specific capabilities may depend on the robot’s design, end effectors, and control algorithms.

[0022] At step 115, the process 100 receives a prompt indicating an overall objective. For example, the prompt may be received from one or more humans working with the same environment of the robot or device performing process 100. In some examples, the overall object may correspond to preparing food, fabricating or constructing an assembly, performing a medical task, a maintenance or cleaning objective, or the like.

[0023] At step 120, the process 100 generates a list of task steps based on the prompt and the plurality of objects. In some examples, a VLM may be used to generate task descriptions corresponding to each task step in the list of task steps based on a captured frame. Moreover, in some examples, the process 100 may use a large language model (LLM) to generate a sequence of the task steps.

[0024] At step 125, the process 100 generates a task graph corresponding to each task step in the list of task steps generated in step 120. For example, each task graph may include a plurality of nodes corresponding to each task step, as well as a plurality of edges corresponding to feasible orders between each task step. In some examples, the LLM maydetermine if further edges can be created, based on a sequence reversibility check of list of tasks generated in step 120.

[0025] At step 130, the process 100 receives an image or video stream that depicts a human in a scene. In some examples, an image of the human is input into a vision language model (VLM) to generate a textual description of the human’s behavior in the image. Moreover, in some examples, the textual description can be used to compare a similarity with task steps in the task graph to compute a measurement likelihood.

[0026] At step 135, process 100 determines the probability of the human performing each task step based on the image or video stream. As illustrated in FIG. 1, steps 130 and 135 may be repeated to continuously update the probability'. In some examples, the likelihood determined at step 130 is used in Bayesian filtering to update the probability of the human performing each task step.

[0027] At step 140, the process 100 determines a predicted next intention step of the one or more humans based on the probabilities included in each task graph. In some examples, one or more of the steps generated in step 120 may be inputted to an LIT system (as described in more detail below). The LIT system may evaluate the probabilities associated with the one or more tasks to output the predicted next intention step.

[0028] At step 145, the process 100 executes a manipulation option from the plurality of manipulation options generated in step 110. The executed manipulation option corresponds to the predicted next intention step determined in step 130. As described above, in some examples, the robot or device may execute step 135 by interacting with one or more objects in the scene, and optionally providing the one or more objects to a collaborating human.

[0029] FIG. 2 shows a block diagram illustrating a system for an example collaborative robot system 200, according to some embodiments. In some examples, a computing device 210 can obtain or receive environmental data 202 including via a communication network 230. In some examples, the computing device 210 may be onboard a robot 240, or in wireless communication with one or more robots 240. While the general term “robot’' is used herein to refer to a device that may be performing some or all of the processes disclosed herein, it should be understood that this term is meant to be encompassing of a variety of devices with mechanisms that can perform collaborative tasks (e.g., robotic arms; aquatic, terrestrial, or aerial drones; autonomous vehicles;autonomous devices; general purpose or purpose-specific robots; autonomous mapping and exploration devices; etc.).

[0030]

[0031] In some examples, the environmental data 202 can include a photo or video of objects surrounding a robot 240, a configuration space, a workspace, obstacles in the workspace, or the like. In some examples, the environmental data 202 may be captured by one or more sensors on or in communication with a robot 240. For example, the environmental data 202 may be obtained using one or more cameras such as a singlephoton avalanche diode (SPAD) sensor, Time of Flight (ToF) camera, stereo vision cameras, a LiDAR sensor, an infrared sensor, a thermal sensor, an event-based camera, or the like.

[0032] In further examples, the computing device 210 can include a processor 212. In some embodiments, the processor 212 can be any suitable hardware processor or combination of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), a microcontroller (MCU), etc.

[0033] In further examples, the computing device 210 can further include a memory 214. The memory 214 can include any suitable storage device or devices that can be used to store suitable data (e.g., environmental data 202, a large language model (LLM), a vision language model (VLM), a neural network model, etc.) and instructions that can be used, for example, by the processor 212. The memory 214 can include any suitable volatile memory, non-volatile memory, storage, or any suitable combination thereof. For example, memory 214 can include random access memory (RAM), read-only memory' (ROM), electronically-erasable programmable read-only memory (EEPROM), other forms of volatile memory, other forms of non-volatile memory, one or more forms of semi-volatile memory, one or more flash drives, one or more hard disks, one or more solid state drives, one or more optical drives, etc. In some embodiments, memory' 214 can have encoded thereon, or otherwise stored therein, a computer program for controlling operation of the computing device 210. In some embodiments, the processor 212 can execute at least a portion of process 100 described above in connection with FIG. 1.

[0034] In further examples, computing device 210 can further include communications system 218. The communications system 218 can include any suitable hardware, firmware, and / or software for communicating information over communication network230 and / or any other suitable communication networks. For example, the communications system 218 can include one or more transceivers, one or more communication chips and / or chip sets, etc. In a more particular example, the communications system 218 can include hardware, firmware and / or software that can be used to establish a Wi-Fi connection, a Bluetooth connection, a cellular connection, an Ethernet connection, etc.

[0035] In further examples, the computing device 210 can receive or transmit information (e.g., environmental data 202, robot manipulation instructions, etc.) and / or any other suitable system over a communication network 230. In some examples, the communication network 230 can be any suitable communication network or combination of communication networks. For example, the communication network 230 can include a Wi-Fi network (which can include one or more wireless routers, one or more switches, etc.), a peer-to-peer network (e.g., a Bluetooth network), a cellular network (e.g., a 3G network, a 4G network, a 5G network, etc., complying with any suitable standard, such as CDMA, GSM, LTE, LTE Advanced, NR, etc.), a wired network, etc. In some embodiments, communication network 230 can be a local area network, a wide area network, a public network (e.g., the Internet), a private or semi-private network (e.g., a corporate or university intranet), any other suitable type of network, or any suitable combination of networks. Communications links shown in FIG. 2 can each be any suitable communications link or combination of communications links, such as wired links, fiber optic links. Wi-Fi links, Bluetooth links, cellular links, etc.

[0036] In further examples, computing device 210 can further include a display 216 and / or one or more inputs 220. In some embodiments, the display 216 can include any suitable display devices, such as a computer monitor, a touchscreen, a television, an infotainment screen, etc. to display a report, a predicted intention step, or any suitable result of the robot manipulation determination. In further embodiments, and / or the input(s) 220 can include any suitable input devices (e.g., a keyboard, a mouse, a touchscreen, a microphone, etc.).

[0037] The collaborative robot system 200 implements an LIT framework, or system, 250 that integrates multiple modules to enable proactive human-robot collaboration. In some cases, the LIT system 250 may implement the Robot Operating System (ROS), which provides a flexible architecture for integrating different components and managing communication between modules.

[0038] The LIT system 250 generally includes an open scene understanding module 252, a task graph reasoning module 254, an LIT module 256, and an intention-grounded planning module 258. The integration and operation of the modules in the LIT system 250 may proceed as follows.

[0039] The LIT system 250 begins by analyzing the visual input, or other environmental data 202, from a data source (e.g., a camera, other sensors) using the open scene understanding module 252. The open scene understanding module 252 provides a foundation for the collaborative robot system 200 to perceive and interpret its environment. By way of example, this module can utilize computer vision and natural language processing techniques to detect objects, generate potential manipulation options, and provide a comprehensive understanding of the scene.

[0040] In some cases, the open scene understanding module 252 may employ a VLM to analyze image data or other environmental data received from a data source, such as a camera that provides a view of a workspace. The VLM may be prompted to identify and describe objects present in the image(s). This approach allows for flexible and adaptable object recognition without requiring pre-defined object categories. In some cases, the object names and descriptions generated by the VLM may then be used as inputs for more specialized object detection and segmentation algorithms. In some implementations, the system may utilize Grounding DINO coupled with Segment Anything to locate and segment individual objects within the scene. Grounding DINO may provide bounding boxes for detected objects, while Segment Anything may generate precise segmentation masks. To ensure robustness, the open scene understanding module 252 may filter out objects that were named by the VLM but detected with low confidence by object detection algorithms. This filtering step helps reduce false positives and improves the overall reliability of the scene understanding.

[0041] In some cases, the open scene understanding module 252 may perform additional analysis on the detected objects to generate potential manipulation options. For example, Principal Component Analysis (PCA) may be applied to the object segmentations to compute object orientations. Based on these orientations, the system may determine corresponding manipulation options that could be used by the robot 240 to manipulate the objects. As one non-limiting example, the manipulation options may include grasp actions and corresponding grasp poses that may be used by the robot 240 to manipulate the objects. The manipulation option generation process may consider factors such as objectshape, size, and spatial relationships to other objects in the scene. For instance, for a cylindrical object like a bottle, the system may generate top-down grasp poses aligned with the principal axis of the object. For flat objects like plates, side grasps may be preferred.

[0042] In some implementations, the open scene understanding module 252 may provide a continuous stream of updated scene information as objects are moved or new objects are introduced. This dynamic understanding allows the collaborative robot system 200 to adapt to changes in the environment in real-time.

[0043] The outputs of the open scene understanding module 252, including detected object names, segmentations, orientations, and potential manipulation options (e.g., including grasp poses), may be made available to other components of the LIT system 250. These outputs may be used for subsequent task planning and robot control modules, enabling intelligent and context-aware collaboration between the robot 240 and human user.

[0044] The task graph reasoning module 254 utilizes an LLM to process user prompts and detected objects in order to generate task steps and construct a task graph. Upon receiving an initial task prompt from the human collaborator, the LLM reasons about the necessary’ steps to complete the task. The LLM may generate a sequence of task steps and identify the objects required for each step. This module plays a role in understanding the overall task structure and potential execution sequences. By leveraging the LLM's ability to understand natural language prompts and reason about task structures, the task graph reasoning module 254 may enable more flexible and adaptable collaborative robot behavior across a wide range of tasks and scenarios.

[0045] In some cases, the task graph reasoning module 254 may receive a user prompt describing a high-level task, such as "make a salad.” The module may also receive information about detected objects in the scene from the open scene understanding module 252. Using this input, the LLM may generate a sequence of task steps required to complete the overall task. For example, given the prompt “make a salad’' and detected objects like tomatoes, cucumbers, and a bowl, the LLM may generate task steps such as:Table 1. Example task steps for a “make the salad” task1. Wash vegetables2. Cut tomatoes3. Cut cucumbers4. Put vegetables in bowl5. Add dressing6. Mix salad

[0046] The task graph reasoning module 254 may then use the LLM to analyze the relationships between these task steps. In some cases, the module may query the LLM to determine if adjacent steps in the sequence can be performed in a different order without affecting the final outcome. This process helps identify which steps have flexible ordering and which steps have strict dependencies. For instance, the LLM may determine that steps 2 and 3 (cutting tomatoes and cutting cucumbers) can be performed in either order without impacting the final result. However, step 4 (putting vegetables in bowl) may need to occur after steps 2 and 3, and before steps 5 and 6.

[0047] Based on this analysis, the task graph reasoning module 254 may construct a directed task graph. In this graph, nodes may represent individual task steps, while edges may indicate valid transitions between steps. The resulting task graph may capture both the necessary sequence of actions and the flexibility in ordering certain steps.

[0048] In some cases, the task graph reasoning module 254 may provide additional information for each task step, such as the objects or tools required for that step. This information may be used by downstream modules to assist with planning and execution.

[0049] The LIT module 256 uses the task graph generated by the task graph reasoning module 254 to build a probabilistic graphical model for intention transition. For instance, using the task graph generated by the task graph reasoning module 254, the LIT module 256 builds a probabilistic graphical model to represent possible human intentions and transitions between them. As the collaboration progresses, the human’s actions are recorded in the environmental data 202. A VLM generates text descriptions of these actions, w hich serve as measurements for the intention tracking algorithm.

[0050] The probabilistic model allow s for tracking and predicting human intentions based on observed behaviors. Human intention, Gt, may be defined as the task step the human intends to work on at time t. The robot 240 uses the measurement history of the human behavior, Xut, to track the human intention, as illustrated by the example graphical model for intention tracking illustrated in FIG. 4. A language probabilistic graphical model(LPGM) may be constructed to described the dynamics of human behavior in FIG. 4, where the value of each node is a natural language sentence.

[0051] In some cases, an LLM may be used to model measurement likelihood and transition probabilities in the probabilistic graphical model of human intentions. The LLM enables more flexible and context-aware modeling compared to traditional fixed probabilistic models. To compute transition probabilities between intentions, the LIT module 256 may apply LLM-based methods to estimate the probability P^9t+i \dt>xt-Tw-.t} , where gt represents the human intention at time t, and xt-Tw:t represents a window of past observations. This approach allows for dynamic updating of transition probabilities based on recent context.

[0052] For measurement likelihood computation, the module may use a VLM to generate text descriptions of human behavior from images (e.g., camera frames, video frames). These text descriptions can be used as measurements for intention tracking. The measurement likelihood P{xt+ 1\xt -Tw . t, gt + 1^ may then be computed using similarity metrics between the VLM-generated descriptions and expected behaviors for each possible intention.

[0053] In some cases, different similarity metrics may be compared for computing measurement likelihood. These metrics may include a BERTScore metric, a BERT-mean- cos metric, and a Word2Vec-mean-cos metric. The BERTScore metric uses contextual embeddings from BERT (Bidirectional Encoder Representations from Transformers) to compute similarity between generated and reference texts. The BERT-mean-cos metric takes the mean of word embeddings from BERT to generate sentence embeddings, then applies cosine similarity. The Word2Vec-mean-cos metric is similar to BERT-mean-cos, but uses Word2Vec embeddings instead of BERT.

[0054] The LIT module 256 may iterate through prediction and update steps of a Bayesian filtering to track human intentions over time. The prediction step may use the computed transition probabilities to estimate the next intention, while the update step incorporates new measurements using the computed measurement likelihoods. In some cases, the module may use a fixed time-window approximation for computing transition probabilities and measurement likelihoods. This approach balances computational efficiency with the ability to capture recent context.

[0055] The LIT module 256 may output a probability distribution over possible human intentions at each time step. This distribution may be used by downstream modules, suchas the intention-grounded planning module 258, to guide robot actions in anticipation of likely human intentions.

[0056] The intention-grounded planning module 258 operates to predict the next human intention and plan corresponding robot actions in a collaborative task scenario. For instance, based on the current estimate of human intention and a prediction of the next likely intention, the intention-grounded planning module 258 plans the actions for the robot 240 to proactively assist the human collaborator. The planning takes into account the objects relevant to the predicted intention and the capabilities of the robot 240.

[0057] The intention-grounded planning module 258 takes as input the current posterior distribution over human intentions from the LIT module 256. In some cases, the intention- grounded planning module 258 may also consider task constraints and safety requirements when planning robot actions. For instance, the module may ensure the robot 240 maintains a safe distance from the human while still positioning objects within easy reach. The intention-grounded planning module 258 may also continuously update its plans as new intention estimates become available. If the predicted human intention changes, the planned actions of the robot 240 may be dynamically adjusted to align with the updated prediction.

[0058] In some cases, the intention-grounded planning module 258 may perform an additional prediction step on the current intention posterior to estimate the most likely- next human intention. For example, if the current posterior indicates a high probability that the human is cutting vegetables, the module may predict that the next likely intention is to place the cut vegetables in a bowl. Based on this predicted next intention, the intention-grounded planning module 258 may identify relevant objects and actions for the robot 240 to manipulate or perform. In the vegetable cutting example, the module may determine that the robot 240 should retrieve a bowl and move it closer to the human's workspace in preparation for the next task step. The module may then generate a sequence of low-level robot control commands to carry out the identified actions. This may involve path planning to avoid obstacles, grasp planning to properly manipulate objects, and motion planning to execute smooth trajectories. The intention-grounded planning module 258 may interface with the low-level control systems of the robot 240 to execute the planned actions. In some cases, this may involve translating high-level action plans into joint-space trajectories or end-effector poses for the robot 240 to follow.

[0059] By way of example, in a collaborative cooking scenario, the intention-grounded planning module 258 may enable a robot sous-chef to proactively assist a human chef. For example, as the human finishes chopping vegetables, the robot 240 may predict the next intention is to saute the vegetables and thus may preemptively retrieve a pan and cooking oil. This proactive assistance may help streamline the cooking process without requiring explicit instructions from the human.

[0060] By leveraging the predictive capabilities of the intention tracking system, the intention-grounded planning module 258 may enable more fluid and natural human-robot collaboration across a variety of task domains. The ability of the intention-grounded planning module 258 to anticipate human needs and plan appropriate robot actions accordingly may enhance the efficiency and intuitiveness of collaborative workflows.

[0061] Once a plan is generated, the LIT system 250 translates high-level actions into specific motion commands for the robot 240. These commands may be executed through the ROS framework, which manages the low-level control of the robot hardware. Throughout the collaboration, the LIT system 250 continuously updates its understanding of the scene, tracks the human’s intentions, and adjusts its plans accordingly. For example, in a salad-making task, the LIT system 250 may initially detect ingredients and utensils on the workspace. When the human reaches for a tomato, the LIT module 256 may predict that the next step is slicing the tomato. The intention-grounded planning module 258 may then instruct the robot to retrieve and offer a knife to the human. If the human instead reaches for a cucumber, the LIT system 250 updates its intention estimate and may adjust its plan to assist with cucumber preparation.

[0062] This integrated approach allows for fluid and adaptive collaboration between the human and robot 240, with the robot 240 proactively offering assistance based on its understanding of the task and the human’s intentions, without requiring explicit commands at each step.

[0063] The collaborative robot system 200 may include a user device 260 that allows a human collaborator to interact with the LIT system 250. In some aspects, the user device 260 may be a smartphone, tablet, laptop, desktop computer, or dedicated control panel. The user device 260 may have an interface (e.g., a graphical user interface (GUI), or the like) that enables the human to input prompts or commands. For example, the interface could include a text entry7field where the user can type in a high-level task descriptionlike “make a salad” or “assemble a chair.” In some cases, the interface may provide predefined options or templates that the user can select from.

[0064] The user device 260 may communicate with the computing device 210 running the LIT system 150 through the communication network 230, such as via a wired or wireless connection. When the user enters a prompt, the user device 260 may transmit this information to the computing device 210 via the communication network 230. Upon receiving the prompt, the processor 212 may parse and interpret the natural language input. The processor 212 may then use this interpreted task description to initialize the task graph reasoning module 254 and begin the collaborative process. The processor 212 may also send confirmation or status updates back to the user device 260.

[0065] In some implementations, the user device 260 may provide additional contextual information along with the prompt. This could include the user’s location, preferences, or relevant environmental data captured by the user device’s own sensors. The LIT system 250 may incorporate this supplementary information to refine its task understanding and planning.

[0066] The user device 260 may also serve as a feedback mechanism, allowing the human collaborator to provide real-time input or corrections as the task progresses. For instance, the user device 260 could display the LIT system’s current understanding of the task and allow the user to make adjustments if needed. In some cases, the user device 260 may implement augmented reality capabilities. This could enable the human to visually indicate objects or locations in the workspace, providing more intuitive and precise communication with the LIT system 250.

[0067] In some embodiments, any suitable computer-readable media can be used for storing instructions for performing the functions and / or processes described herein. For example, in some embodiments, computer-readable media can be transitory or non- transitory. For example, non-transitory computer-readable media can include media such as magnetic media (e.g., hard disks, floppy disks), optical media (e.g., compact discs, digital video discs, Blu-ray discs), semiconductor media (e.g., RAM, flash memory, EPROM, EEPROM), any suitable media that is not fleeting or devoid of any semblance of permanence during transmission, and / or any suitable tangible media. As another example, transitory computer-readable media can include signals on networks, in wires, conductors, optical fibers, circuits, or any suitable media that is fleeting and devoid of any semblance of permanence during transmission, and / or any suitable intangible media.

[0068] As used herein in the context of computer implementation, unless otherwise specified or limited, the terms “component,"’ “system.” “module,"’ “framework,” and the like are intended to encompass part or all of computer-related systems that include hardware, software, a combination of hardware and software, or software in execution. For example, a component may be, but is not limited to being, a processor device, a process being executed (or executable) by a processor device, an object, an executable, a thread of execution, a computer program, or a computer. By way of illustration, both an application running on a computer and the computer can be a component. One or more components (or system, module, and so on) may reside within a process or thread of execution, may be localized on one computer, may be distributed between two or more computers or other processor devices, or may be included within another component (or system, module, and so on).

[0069] In some implementations, devices or systems disclosed herein can be utilized or installed using methods embodying aspects of the disclosure. Correspondingly, description herein of particular features, capabilities, or intended purposes of a device or system is generally intended to inherently include disclosure of a method of using such features for the intended purposes, a method of implementing such capabilities, and a method of installing disclosed (or otherwise known) components to support these purposes or capabilities. Similarly, unless otherwise indicated or limited, discussion herein of any method of manufacturing or using a particular device or system, including installing the device or system, is intended to inherently include disclosure, as embodiments of the disclosure, of the utilized features and implemented capabilities of such device or system.

[0070] FIG. 3 illustrates a diagram for a language-driven intention tracking (LIT) based collaborative robot framework. The open scene understanding module detects objects in the scene and generates potential manipulation options, such as top-down grasp poses and / or other manipulation actions described above. The task graph reasoning module takes the user’s prompt on the overall task and the detected objects as input to generate a list of task steps, which is defined as intention in this work. As some steps of the overall task can switch order without impact on the outcome, the LLM checks on reversibility of sequences of task steps, and builds a task graph. The LIT module uses the task graph to build the probabilistic graphical model for intention transition. The VLM is used to generate text descriptions from frames as measurements. The time-varying transitionprobabilities are computed and prediction steps are made, and measurements are used to compute measurement likelihood and make update steps to track the human intention. The intention-grounded planning module makes an additional prediction step on the current intention posterior, and manipulates the objects relevant to the predicted next intention to proactively collaborate with the human.

[0071] Language-Driven Intention Tracking - Problem Formulation: A human user collaborates with a robot to perform a long-horizon multi-step task. The robot can understand the scene and reason about the task to formulate a directed task graph, where nodes are defined as steps of the task, and edges are defined as the feasible orders between the task steps. The reason a task graph was used instead of a task chain is the relationships between task steps may not always be causal and the order may be reversible, such as "‘cut tomatoes” and "cut cucumbers” for a salad making task. Human intention is defined as Gt as the task step the human intends to work on at time I. The robot uses the measurement history of the human behavior Xi.t to track the human intention Gt. FIG. 4 illustrates a graphical model for intention tracking. The measurement of human behavior is denoted as Xt. and the human intention as Gt.

[0072] Language Probabilistic Graphical Model: Described here is a Language Probabilistic Graphical Model (LPGM) that describes the dynamics of human behavior in FIG. 4, where the value of each node is a natural language sentence. To calculate a conditional probability for example P(A = a\B = b, C = c) an LPGM. an LLM is used, where the prompt has two parts: a conditional part and a query part. The conditional part of the prompt is defined as "We observe {B} is {b}, and {C} is {c}, ”. Three different methods compose the query7part of the prompt and calculate the conditional probability7accordingly.

[0073] The first method is to directly ask for P(a\b, c). The query part of the prompt is formulated as “provide the probability of {A} being {a}.” However, the outputs from the LLM may not be trustworthy if corresponding materials are not covered much by the corpus used to train the LLM.

[0074] The second method is to ask the LLM to generate a value of A. and compare the similarity score such as BERTScore of the generated text with respect to a to quantify P(a\b, c). The query7part of the prompt is formulated as “what do you think {A} would be?”. This method essentially uses the LLM to provide a maximum likelihoodestimate argmaxa P ( / I = a\B = b, C = c) , and uses the distance between this point estimate and the value to compute the conditional probability.

[0075] The third method addresses the case where A is a discrete variable. The LLM is asked to generate a list of values of A with a large length N, compare the values in the list to all possible values of A, aggregate the number of the most similar generated values to each possible value, and form a statistical estimate of the P (a | b, c) . The query' part of the prompt is formulated as "what do you think {A} would be? Provide A different examples.”. This is similar to a Monte Carlo method which approximates the distribution of P(A = a\B = b, C = c) by sampling from the LLM.

[0076] Application in Intention Tracking: To perform language-driven intention tracking, the prediction and update steps of Bayesian filtering are iterated as:St^(.9t+i|xi:t+i) °C P(xt+1\x1.t, gt+1')P(gt+1\x1.t')

[0077] LPGM methods are applied to compute the fixed time-window approximations of intention transition P(jh+i \9t>xt-rw:t)ancl measurement likelihood P(t+1\xt-Tw-.t’ 9t+ )- The third method is applied to compute P(gt+1\gt, xt-Tw..t\ since the human intention Gt is a discrete variable. The second method is applied to compute P(xt+i \xt-Tw:t> 9t+i)-

[0078] Robot Sous-Chef Application - Collaborative Cooking Setup: The human user wants to make a dish, but all the required materials and tools are not reachable by the human - but are by the robot. The robot is tasked with acting as a sous-chef to smoothly coordinate with the human by passing materials and tools at appropriate times while not making the human’s cooking table overly occupied with unnecessary items at the moment. The robot is assumed to only receive the prompt at the beginning on what dish is going to be made, and will not receive prompts during collaboration.

[0079] In one non-limiting example, LLaVA with a 13-billion parameter Vicuna backbone (derived from Llama 2) was selected as the VLM in the system, due to both its open source nature and competitiveness with commercial-grade models such as Gemini. Note that the same model was used as the LLM for consistent performance by inputting the text prompt with a full-black image. The collaborative robot is a UR5e arm equipped with a Robotiq Hand-E Gripper. In one non-limiting example, Intel RealSense RGBDCameras are used to provide a top-do wn view of the robot table with objects on it, and to provide a front view of the human user’s behavior. Moreover, in one example, the Robot Operating System (ROS) is used to build the LIT framework.

[0080] Open Scene Understanding Module: The overhead camera provide a top-down view of the robot workspace with all object reachable by the robot. The VLM is prompted to name and describe the objects in the image frame from the overhead camera, and take these names as input to Grounding DINO coupled with Segment Anything to locate and segment all objects on the frame. Object names listed as present in the scene by the VLM that are detected with low confidence are thrown out. Principal Component Analysis is performed on object segmentations and compute object orientation and corresponding grasp poses. The detected object names and grasp poses are fed into the downstream modules.

[0081] Task Graph Reasoning Module: Given the available objects produced by the open scene understanding module and the general task prompt from the human user, the LLM is queried to output a sequence of task steps in order to achieve the overall task. The LLM is also asked to provide the corresponding objects needed in each task step, which will be used to inform the downstream planning module. The task graph are initialized with the sequence of task steps, and query the LLM whether adjacent steps can be reversible to add new' edges to the task graph.

[0082] Language-Driven Intention Tracking Module: The task steps from the reasoned task graph were used as possible values of intention Gt. The task graph is used to initialize the uniform prior among the first steps the human can start on, and to inform which pairs of (gt, gt+i) may be needed to compute the intention transition P(gt+1\dt>Xt-Tw:t)- During collaboration, a front-view camera was used to collect frames of the human user, and feed the frames to VLM to generate text descriptions of human behavior as measurements Xt. Eq. 1 is used to perform language-driven intention tracking.

[0083] Intention-Grounding Planning Module: To collaborate with the human proactively, the robot predicts the human intention at the next time step by running one prediction step in LIT on the current posterior of the intentions, and outputting the maximum probability intention as the prediction. The intention-grounded planning module performs planning and control to manipulate the objects relevant to the predicted next intention. In our collaborative cooking scenario, the robot sous-chef would pass the objects needed for the next cooking step to the human chef in advance.

[0084] FIG. 5 illustrates language-driven intention tracking with different similarity metrics. The ground truth order of the human intention includes: slicing tomatoes, slicing cucumbers, putting tomatoes and cucumbers in a bowl, putting salad dressing on tomatoes and cucumbers, and stirring and mixing the salad with a spoon. The snapshots show the moment when the intention transition happens. Panel (a) illustrates when the human starts cutting a cucumber after finishing cutting a tomato. Panel (b) illustrates when the human starts putting vegetables into a bowl after cutting the cucumber.

[0085] Preliminary Study: A salad cooking demonstration is collected and a language- driven intention tracking is run to compare how similarity metrics affect tracking performance as presented in FIG. 5. In addition to BERTScore, BERT-mean-cos and Word2Vec-mean-cos are introduced, which take the mean of word embeddings from the corresponding pre-trained model to generate candidate and reference sentence embeddings, and apply cosine similarity’ to generate similarity score. During evaluation, the effect of similarity’ metrics are isolated by computing measurement likelihood with the similarity metrics while using a fixed intention transition matrix based on the task graph. FIG. 5 shows that BERT-mean-cos empirically outperforms BERTScore and Word2Vec-mean-cos for tracking human intentions.

Claims

CLAIMSWHAT IS CLAIMED IS:

1. A system for human-robot collaboration, the system comprising: a robotic device having at least one mechanism for moving at least a part of the robot; one or more sensors configured to capture environmental data; a processor in communication with the robotic device and the one or more sensors; a memory in communication with the processor and having instructions stored thereon that, when executed, cause the processor to: capture, via the one or more sensors, an image of a scene where the robotic device is located; detect a plurality of objects in the scene based on the image; generate a plurality of manipulation options corresponding to each object in the plurality of objects; receive a prompt indicating an overall objective; generate a list of task steps based on the prompt and the plurality of objects; generate a task graph corresponding to the list of task steps based on a reversibility check between each task step in the list of task steps; determine a probability of each task step in the list of task steps corresponding to a behavior of a human in the scene captured by the sensor; determine a predicted next intention step based on the probability7of each task step in the list of task steps; and execute, via the robotic device, a manipulation option from the plurality of manipulation options corresponding to the next intention step.

2. The system of claim 1 , wherein the instructions further cause the processor to: input the image of the scene to a trained visual language model (VLM) and a trained image segmentation foundation model,wherein the trained VLM and the trained image segmentation foundation model detect the plurality of objects in the scene by identify ing at least one of a name, an orientation, or a dimension of each object in the plurality of objects.

3. The system of claim 1, wherein each manipulation option in the plurality7of manipulation options comprises an object grasping action and an object relocation action.

4. The system of claim 1, further comprising: a user device in communication with the processor, wherein the prompt is received via the user device.

5. The system of claim 1 , wherein the instructions further cause the processor to: input the prompt to a trained large language model (LLM); and receive, via the trained LLM. a sequence of task steps to achieve the overall objective, wherein one or more pairs of adjacent task steps are input into the LLM to determine the reversibility check for building a task graph.

6. The system of claim 5, wherein the task graph comprises a plurality of nodes and a plurality of edges. wherein the plurality of nodes represents a plurality of steps associated with each task and the plurality of edges represents the sequence of task.

7. The system of claim 1. wherein each task in the list of tasks steps corresponds to a predicted task by one or more humans operating in the scene.

8. The system of claim 1, wherein executing the manipulation options comprises providing, via the robotic device, one or more objects from the plurality of objects to a human operating in the scene.

9. The system of claim 1, wherein the one or more sensors comprise at least one of a camera, a LiDAR sensor, or an infrared sensor.

10. A method for collaborative task execution, comprising: detecting, by one or more sensors, a plurality of objects in a scene; generating, by a processor, a plurality of manipulation options corresponding to each object in the plurality of objects; receiving, at the processor, a prompt indicating an overall objective; generating, by the processor, a list of task steps based on the prompt and the plurality of objects; generating, by the processor, a task graph corresponding to each task step in the list of task steps; generating, by the processor, a probability of one or more humans performing each task step in the list of task steps when a human behavior in the scene is captured by the sensor; determining, with the processor, a predicted next intention step of the one or more humans based on the probabilities included in each task graph; and executing, by a robot, a manipulation option from the plurality of manipulation options, wherein the executed manipulation option corresponds to the predicted next intention step.

11. The method of claim 10, wherein generating the plurality of manipulation options comprises determining at least one of a grasping action, a pushing action, a lifting action, or a placing action for each object.

12. The method of claim 10, wherein generating the list of task steps comprises using a large language model (LLM) to analyze the prompt and the plurality of objects.

13. The method of claim 12, wherein generating the task graph comprises querying the LLM to determine if adjacent steps in the list of task steps can be performed in a different order.

14. The method of claim 10, wherein determining the predicted next intention step comprises:generating text descriptions of human behavior from environmental data captured by the one or more sensors using a vision language model (VLM); computing a measurement likelihood based on the text descriptions; performing Bayesian filtering using the measurement likelihood to generate the probability of one or more human performing each task step in the list of task steps; and determining the next intention step based on an additional Bayesian filtering prediction step.

15. The method of claim 14, wherein computing the measurement likelihood comprises using a similarity metric.

16. The method of claim 10. wherein executing the manipulation option comprises: identifying an object relevant to the predicted next intention step; and moving the identified object to a location accessible to the one or more humans.

17. The method of claim 16, wherein moving the identified object comprises planning a trajectory that avoids collision with the one or more humans.

18. The method of claim 10, further comprising continuously updating the plurality of task graphs based on new environmental data captured by the one or more sensors.

19. The method of claim 10, further comprising: continuously updating the probability of one or more humans performing each task step by: generating new text descriptions of human behavior from the new environmental data using a vision language model (VLM); computing updated measurement likelihoods based on the new text descriptions; and adjusting the probability of one or more humans performing each task step based on the updated measurement likelihoods by Bayesian filtering and the task graph.

Images