Dynamic collaboration of multiple robotic arms

Abstract

An arrangement for dynamic collaboration of a plurality of robotic arms is provided. Image data of a real-world environment associated with a plurality of tasks to be collaboratively performed by a plurality of robotic arms may be acquired. An object in the image data may be identified and a parameterized representation of the object is generated. A parameterized model of the real-world environment may be generated using the parameterized representation. Based on the parameterized model, a planned trajectory of the first robot arm within a predetermined time span may be determined. The first robotic arm may be controlled to execute the planned trajectory according to the first task, including manipulating the first object to reveal the second object in the real-world environment. The parameterized model may be dynamically updated, and a second robot arm may be controlled to perform a second task on a second object in cooperation with the first task.

Description

[0001] Cross-references to related applications This application requires a document with the subject line " to be filed on May 17, 2023" Devices, Systems, and Methods for Multiple Robotic manipulator Arms to Collaboratively Explore, Map and Exploit Diverse Environments The rights and priorities of U.S. Provisional Patent Application No. 63 / 502783, the entire contents of which are clearly incorporated herein by reference. Technical Field

[0002] The topics discussed in this article generally involve the dynamic coordination of multiple robotic arms. Background Technology

[0003] For many tasks, adding extra robotic arms can make a system more efficient. However, while it is relatively easy for humans to manipulate objects with two arms, doing so is much more difficult for robot programming. For example, increasing the number of robotic arms from one to multiple arms leads to an exponential increase in the computational complexity of task planning. Therefore, it is difficult to plan for high redundancy quickly enough to handle various environments. For instance, it may be difficult to effectively coordinate collaborative tasks involving multiple robotic arms in a shared environment. Often, the dynamic manipulation of one robotic arm is not considered in relation to the other. Summary of the Invention

[0004] Current topics include enabling dynamic collaboration of multiple robotic arms for discovering, exploring, and manipulating objects in an environment with low latency. In one aspect, a method includes: acquiring, from the perspective of a robotic device, image data of a real-world environment associated with multiple tasks to be performed collaboratively by at least a first robotic arm and a second robotic arm of the robotic device; identifying one or more objects in the image data; generating one or more parameterized representations of the one or more objects in the image data; generating a parameterized model of the real-world environment associated with the multiple tasks using the one or more parameterized representations, wherein the parameterized model includes a three-dimensional volumetric mesh composed of voxels, wherein each voxel is assigned a low-dimensional representation of a state of a portion of the real-world environment and semantic information associated with the one or more objects; determining a planned trajectory of the first robotic arm over a predetermined number of time steps based on the parameterized model, including identifying whether an obstacle will intersect the planned trajectory of the first robotic arm over the predetermined number of time steps; controlling the first robotic arm to execute the planned trajectory according to a first task among the multiple tasks, including manipulating the first object among the one or more objects to reveal a second object in the real-world environment; dynamically updating the parameterized model of the real-world environment associated with the multiple tasks; and controlling the second robotic arm to perform a second task among the multiple tasks on the second object based on the updated parameterized model, wherein the second task is performed collaboratively with the first task.

[0005] One or more of the following features can be included in any workable combination. In some variations, dynamically updating the parameterized model of the real-world environment associated with the plurality of tasks can include determining a planned trajectory of the second robotic arm for a predetermined number of time steps.

[0006] In some variations, parameterizing the image data can include converting pixel data of the one or more objects to a compressed representation of the one or more objects.

[0007] In some variations, the second object can include an object at least partially occluded by the first object.

[0008] In some variations, the first object can include at least a portion of the second robotic arm.

[0009] In some variations, the image data can be acquired with an image sensor of the robotic device.

[0010] In some variations, dynamically updating the parameterized model of the real-world environment associated with the plurality of tasks can include making updates to include additional objects identified and parameterized by the image sensor of the robotic device.

[0011] In some variations, manipulating a first object of the one or more objects can include volumetrically deforming or displacing the first object.

[0012] In some variations, the predetermined number of time steps can be three or fewer future time steps.

[0013] In some variations, the planned trajectory of the first robotic arm can be selected from a set of candidate trajectories of the first robotic arm. In some variations, each candidate trajectory of the set of candidate trajectories can include a series of planned states of the first robotic arm occurring for the predetermined number of time steps.

[0014] In some variations, dynamically updating the parameterized model of the real-world environment associated with the plurality of tasks can include cyclically updating in a round-robin schedule for each additional robotic arm used to perform one of the plurality of tasks.

[0015] In some variations, the first robotic arm can include an end effector operably coupled to a distal end of the first robotic arm used to perform the first task on the first object. Further, the end effector can include a flexible paddle.

[0016] In some variations, the second robotic arm can include an end effector operably coupled to a distal end of the second robotic arm used to perform the second task on the second object. Further, the end effector can include an object gripper.

[0017] In some variations, the real-world environment can be an outdoor environment.

[0018] In some variations, the control of the first robotic arm and the control of the second robotic arm can occur without user interaction.

[0019] In another aspect, an apparatus is provided that includes a plurality of robotic arms, one or more image sensors, a processor, and a memory. The memory can store instructions that, when executed by the at least one processor, cause operations. The operations can include: obtaining, from the one or more image sensors, image data of a real-world environment associated with a plurality of tasks to be performed collaboratively by at least a first robotic arm and a second robotic arm of the plurality of robotic arms; identifying one or more objects in the image data; generating one or more parametric representations of the one or more objects in the image data; generating, with the one or more parametric representations, a parametric model of the real-world environment associated with the plurality of tasks, wherein the parametric model comprises a three-dimensional volumetric grid composed of voxels, wherein each voxel is assigned a low-dimensional representation of a state of a portion of the real-world environment and semantic information related to the one or more objects; determining, based on the parametric model, a planned trajectory of the first robotic arm for a predetermined number of time steps, including identifying whether an obstacle will intersect the planned trajectory of the first robotic arm within the predetermined number of time steps; controlling the first robotic arm to perform the planned trajectory in accordance with a first task of the plurality of tasks, including manipulating a first object of the one or more objects to reveal a second object in the real-world environment; dynamically updating the parametric model of the real-world environment associated with the plurality of tasks; and controlling the second robotic arm to perform a second task of the plurality of tasks on the second object based on the updated parametric model, wherein the second task is performed collaboratively with the first task.

[0020] In some variations, one or more features disclosed herein including the following features are optionally incorporated in any operable combination. In some variations, the first robotic arm can include an end effector operably coupled to a distal end of the first robotic arm for performing the first task on the first object. Further, the end effector can include a rigid or flexible paddle.

[0021] The document also describes a non-volatile computer program product (i.e., a physically implemented computer program product) containing instructions that, when executed by one or more data processors of one or more computing systems, cause at least one data processor to perform the operations described herein. Similarly, a computer system is described that may include one or more data processors and memory connected to one or more data processors. This memory may temporarily or permanently store instructions that cause at least one processor to perform one or more operations described herein. Furthermore, these methods can be implemented using one or more data processors distributed across a single computing system or two or more computing systems. Such computing systems can be interconnected and can exchange data and / or commands, or other instructions, through one or more connections (including connections via networks (e.g., the Internet, wireless wide area networks, local area networks, wide area networks, wired networks, etc.)), through direct connections between one or more of the multiple computing systems, etc.

[0022] Details of one or more variations of the subject matter described herein are set forth in the accompanying drawings and the following description. Other features and advantages of the subject matter described herein will become apparent from the specification, the drawings, and the claims. Attached Figure Description

[0023] Figure 1 A computing environment for dynamic collaboration of multiple robotic arms is shown according to some example implementations.

[0024] Figure 2 This is a flowchart illustrating an example implementation of a process for dynamic coordination of multiple robotic arms, based on some example implementations.

[0025] Figure 3 This is a flowchart illustrating a computational process for the dynamic coordination of multiple robotic arms, according to some example implementations.

[0026] Figure 4 This is a block diagram illustrating an example implementation of a computational process for dynamic collaboration of multiple robotic arms, based on some example implementations.

[0027] Figure 5 An example parameterized representation of an observed strawberry is shown according to some example implementations.

[0028] Figure 6 An example parameterized model based on some example implementations is shown.

[0029] Figure 7 Example scenarios before and after interacting with the scene, based on some implementations disclosed herein, are shown.

[0030] Figure 8 An example hierarchical semantic model for a typical scenario based on some implementations disclosed herein is shown.

[0031] Figures 9-11 Example classes of non-grasping motion primitives for interacting with strawberry plants, based on some implementations disclosed herein, are described.

[0032] Figures 12-15 Example variants of end effectors that can be deployed for branch and leaf manipulation according to some implementations disclosed herein are depicted.

[0033] Figure 16 The paper illustrates how some implementations of the current topic, based on some example implementations, enable dynamic replanning in response to unpredictable changes in a scenario.

[0034] Figure 17 Example configurations of a multi-arm harvesting robot based on some example implementations are depicted.

[0035] In the various figures, the same reference numerals denote the same elements. Detailed Implementation

[0036] This invention provides one or more technical solutions to address problems related to the dynamic collaboration of multiple robotic arms. For example, the multiple robotic arms may include arms with N degrees of freedom (N-DOF arms) or delta robots. Other aspects of this invention provide a multi-arm robot planning and control platform that utilizes real-time predictions obtained from parametric representations to plan and act on the world at a speed sufficient to handle various environments, particularly unstructured, unorganized, open, cluttered, and / or dynamic environments. A further aspect of this invention provides a multi-robotic arm arrangement that, based on scene-based collaborative exploration and mapping, increases the affordance of operations available to each individual robotic arm. Yet another aspect of this invention provides multiple robotic arms capable of real-time perception and response to environmental changes, enabling multiple robotic arms to work collaboratively in overlapping workspaces.

[0037] Some implementations of this topic can include multi-robotic arm systems that utilize collaborative exploration and mapping of unstructured scenes, increasing the operational possibilities available to each individual execution unit. An example implementation could include an agricultural environment (including strawberry picking), but the topic can be extended to other applications. In picking applications, maximizing the picking rate (e.g., the number of strawberries picked per minute) may be necessary. This places demands on the high-speed movement of the robotic arms and their ability to perceive and react to environmental changes in real time. Some implementations of this topic can include several complementary elements for addressing this problem; the techniques described herein include lightweight and expressive world models and estimation representations, methods and designs for scene interaction, and low-latency multi-robot planning architectures and methods.

[0038] Regarding lightweight and expressive world models and related estimation representations and methods, some implementations of the current topic may include leveraging world modeling techniques that compress observations from heterogeneous camera arrays (e.g., heterogeneity in location, resolution, spectral density, and mobility) into lightweight, hierarchical, and semantic representations. This can be combined with state estimation techniques that utilize prior knowledge about the geometric and / or phenotypic features of scene elements to improve the stability of pose estimation for scene objects in occlusion and limited field of view.

[0039] Regarding scene interaction methods and designs, some implementations of the current topic may include using the end effector design of a single robotic arm and its corresponding manipulator primitives to actively alter the state of the scene. Some implementations may include algorithms for multiple robotic arms to utilize the operational possibilities provided by scene interaction to detect and locate previously occluded objects of interest.

[0040] Regarding the planning architecture and methodology for low-latency multi-robot systems, some implementations of the current topic may include a complete multi-arm robot planning and control system that utilizes the aforementioned model to plan and act upon the world at a speed sufficient to handle unstructured, unorganized, open, cluttered, and / or dynamic environments. A higher-level scheduler can change or swap the roles assigned to physical execution units within the system, freeing the execution units from their nominal roles. For example, end effectors are not bound by specific roles but can be repurposed or swapped as needed based on the currently assigned task. In some implementations, non-grasping motion can be used for interaction with the environment. For example, instead of end effectors, or in addition to end effectors, the forearm of the robotic arm can also be used for interaction with the environment (e.g., parting branches).

[0041] Some implementations of this topic leverage lightweight and expressive world modeling to accelerate robot reasoning related to its surrounding environment. The modeling techniques described herein can utilize scene interaction techniques, as described below, to alter the state of the scene, facilitating the overall task undertaken by the robot. Furthermore, some implementations of this topic can utilize low-latency multi-robot planning to proactively alter environmental morphology, potentially leading to unexpected and unpredictable changes. In such dynamic environments, the behavior of the arms can be adjusted as changes occur, and their interactions can be resolved in real time. This demands high-speed movement of the robotic arms and the ability to perceive and react to environmental changes in real time. In other approaches, this can be achieved, for example, by considering an invariant environment and jointly planning two arms, or by moving each arm sequentially and treating the arms as obstacles to each other.

[0042] Many tasks are difficult or impossible to perform with a single agent (robotic arm) due to the limited capabilities of a single robotic arm. Examples of this include situations where additional arms are needed to interact with branches and leaves to improve the visibility of the image acquisition system and / or the reachability of the picking arm. Some implementations of the current topic involve adding assisting arms to the system. For some tasks, adding one or more additional robotic arms can make the system more efficient. An example of this implementation is the situation described above, where the assisting arms are idle while the picking arm is busy moving strawberries to the picking box. More picking arms can be added to improve the efficiency of the system by reducing, for example, the idle time of the assisting arms. Furthermore, with an increased number of arms, more cameras can be used to collect images from different viewpoints and angles, resulting in the construction of higher-quality maps.

[0043] When multiple robotic arms are positioned and configured to cooperate and share the same workspace, their movements can take into account both spatial and temporal dimensions to provide safe operation. An example of such a multi-robotic arm system is a fruit-picking robot. The picking machine can include picking units with multiple robotic arms. Example robotic picking machines can also include perception systems, world modeling systems, control systems, and planning systems.

[0044] Figure 17 An example configuration of a multi-arm harvesting robot is depicted. The left side shows an example harvesting unit with two robotic arms (e.g., one for harvesting and one for assisting). The right side shows an example harvesting unit with three robotic arms (e.g., two for harvesting and one for assisting). More specifically, the configuration on the right shows an example where the first harvesting arm is assisted by the assisting arm, while the second harvesting arm delivers some fruit to a shelf.

[0045] One example configuration may include two or more arms with defined roles. With two arms, the first arm can be responsible for picking fruit, while the second arm can assist the picking arm by pushing aside branches and leaves, thus improving visibility and accessibility. Another example configuration may use two or more multi-purpose arms. With two multi-purpose arms, both arms have the ability to push aside branches and leaves and pick fruit.

[0046] The robotic arm can interact with and alter the state of its environment. Examples of grasping manipulation include, but are not limited to, picking and moving fruit, and grasping and moving stems. Examples of non-grasping manipulation include, but are not limited to, moving branches and leaves, and pruning branches. Other examples of interaction include cutting stems.

[0047] In some implementations, frequent interactions may occur between perception tasks (e.g., image acquisition, state estimation) and scene interactions (e.g., moving branches, deformable obstacles, and other movable objects for better visibility or picking pathways). These interactions can occur at different spatial scales. As an example, upon reaching a new section of a plant bed ready for harvesting, a sample harvesting robot can conduct a high-level survey to assess the areas most worthy of exploration. During the exploration of these areas, the sample harvesting robot may detect more target objects, at which point it plans and performs a more detailed scan, narrowing the error range of its pose estimation until it is sufficient to trigger a specific action utilizing the environment (e.g., a picking attempt in the case of strawberries). This process can be iteratively executed until all areas have been explored.

[0048] Figure 1 A computing environment 100 for dynamic collaboration of multiple robotic arms is illustrated according to some example implementations. Reference Figure 1 The computing environment 100 may include one or more computing devices and / or other computing systems. For example, the computing environment 100 may include a multi-arm computing platform 110, a control system 120, a world modeling system 130, and a perception system 140. The multi-arm computing platform 110 may include one or more computing layers configured to perform one or more functions described herein. The control system 120 may regulate and command the functions of the robotic device to achieve desired objectives. The world modeling system 130 may include an end-to-end model for a specific problem domain. The world modeling system 130 may enable predictions of what will happen in the real world due to specific actions. The perception system 140 may include image sensors, etc., that enable the robotic device to collect and process data about its environment.

[0049] More specifically, the multi-robotic arm computing platform 110 (hereinafter referred to as "computing platform 110") may include a high-level (resource) planning layer 110a, a task planning layer 110b, a motion planning layer 110c, an execution layer 110d, and a monitoring layer 110e.

[0050] In the high-level planning layer 110a, based on the current understanding of the world, the computing platform 110 can pre-calculate a predetermined number of time steps before the robot needs to perform the steps or sequences to achieve a specific goal (e.g., a business-level goal). Accordingly, the computing platform 110 can decide which actions should be taken and by which agent (e.g., a robotic arm) will perform them. In an example involving fruit picking, the high-level planning layer 110a could decide to assign one arm to picking while the other interacts with the environment. Examples of resource allocation processes could include: assigning an arm to pick accessible and pickable fruit, assigning an arm to explore the scene and search for fruit if the scene has not yet been explored, or assigning an arm to relocate clustered fruit or move plant branches. These tasks can be planned based on a value function that estimates how to make better use of available resources within the planned timeframe.

[0051] In the task planning layer 110b, the computing platform 110 can define a sequence of motion primitives to be executed by various agents (e.g., robotic arms). These motion primitives can be defined for various roles that the arms may have and can include conventional primitives as well as custom, purpose-specific primitives. Furthermore, collaboration can be defined at the task level rather than the motion planning level, allowing planning to be performed separately for each arm. This decentralization enables the use of multiple robotic arms without increasing planning time.

[0052] Accordingly, tasks can be defined for each robotic arm. In other examples, tasks may include moving to a specific location, pushing aside branches, and picking fruit. Examples of custom, purpose-specific primitives include, but are not limited to: Harvesting: A series of movements that bring the actuators (e.g., grippers) of a robotic arm to the fruit and properly separate it from the plant. For strawberries, this can be done by rapidly rotating the end effector or "breaking" it off the stem; Grazing: A series of movements that cause another actuator (e.g., a gripper of a robotic arm holding a paddle) to move with large motion relative to or along the canopy of branches and leaves to expose the center of the plant below (e.g., to help distinguish individual plants by sequentially exposing the center when it is obscured by the canopy). Pushing aside branches and leaves: A series of movements that cause another actuator (e.g., a gripper of a robotic arm holding a pick) to move branches and leaves aside to expose areas normally covered by the canopy (e.g., pushing the plant to one side to make room for camera exploration and harvesting); and Shearing: A series of movements that cause another actuator (e.g., a gripper of a robotic arm holding a knife or other sharp tool) to cut a stem or creeping plant.

[0053] In the motion planning layer 110c, the computing platform 110 can determine the motions that need to be executed by the agents in a decentralized, agent-by-agent manner. It can also determine the motions and dynamics of other agents within a time window. This is particularly useful in changing environments as described above. An aspect of the invention considers a solution where the motions of each arm can be planned in a decoupled manner using a polling scheduling approach. During the execution of the arm's motion, the system can dynamically plan and adjust its behavior, triggered by real-time monitored environmental changes. The tasks of each robotic arm can be converted into start and target configurations, and motion plans can be calculated separately for each arm. Therefore, the system can handle many types of dynamic events.

[0054] Examples of reactive motion planning may include adjustments to the motion planning of a robotic arm due to the following: the appearance of a deformable or extendable object in the planned motion path of the arm; a deformable or extendable object requiring a change in one or more objectives of the motion planning; the appearance of a moving or deformable part in the path of the arm; an obstacle in the planned motion path of the arm; or a change in environmental conditions.

[0055] In the execution layer 110d, the computing platform can control each robotic arm and execute the motion plan for each arm.

[0056] In the monitoring layer 110e, by processing and updating the model at system speed and utilizing the monitoring agent to monitor task execution, replanning can be performed at each time step in the event of scene changes. For example, discovering a new strawberry during a task of pushing away branches can trigger the assignment of a picking task to the arm to move and pick the strawberry, or, if the strawberry is still partially obscured, the path of the branch-interacting arm can be reactively modified to enable picking. Thus, by modeling and understanding the scene at a sensing rate, the arm's movement can react to these changes.

[0057] An example of this scenario is as follows: a strawberry is located by the harvester's sensing system (e.g., sensing system 140), but the strawberry is partially obscured by surrounding foliage. An initial parameterized model of the strawberry (a low-level representation as further described herein) is recorded by the system. The system recognizes the need to push aside the foliage to better make the strawberry visible and locate it. An assisting arm is assigned to interact with the foliage using actions that can be grasping (e.g., picking and moving fruit, grasping and moving stems) or non-grasping (e.g., moving foliage, tidying foliage). Other example interactions could include cutting stems. A harvesting arm is assigned to follow the assisting arm to locate and harvest the strawberry as quickly as possible. The strawberry is indirectly affected and moves as the assisting arm deforms the plant. The system is able to update the strawberry bed model in real time. Based on the speed of perception and execution, the computing platform can adjust the movements of the assisting arm and the harvesting arm to avoid collisions and create space for harvesting.

[0058] The world modeling system 130 can stabilize the pose estimation of scene objects by allocating them in a hierarchical manner. Unlike simple detection that considers individual elements in isolation, the hierarchical approach allows for estimation of relative parent-child and sibling relationships, improving the consistency of the overall scene estimation. This representation can be highly compressed, thus enabling efficient querying and updating. Figure 8 An example hierarchical semantic model 805 for a typical scenario 810 is shown.

[0059] Map building can be performed by multiple cameras with potentially different resolutions, installation locations (fixed, mobile), spectra, etc. By applying prior knowledge about objects expected to be encountered in the scene, more reliable and faster estimation of their salient attributes can be achieved. In the case of picking strawberries, salient attributes can include: position and orientation, maturity, and other phenotypic features, such as the shape of the tip. Therefore, the example picking machine can adapt the target grasping posture to the specific geometry of each strawberry. In the specific case of strawberry picking, a common strawberry is typically in the form of an elongated cone. When the strawberry is detected using a convolutional neural network in a world modeling system (e.g., world modeling system 130), a canonical model of the world in the estimated pose can be determined. The example world modeling system of the example picking machine can update key parameters, including the length, width, maximum width location, and tip curvature associated with the target strawberry, using a combination of semantic keypoints, contours, and textures. Even if the strawberry is partially occluded, the example picking machine can still update its estimation using the visible part of the strawberry, knowing that the occluded part may have a consistent structure. As for the plants, the fact that they can typically be planted in a regular, staggered grid can be utilized when estimating their observed locations. In both cases, the example world modeling system can perform the estimation of instance-specific values ​​in the parametric representation through the training and detection of semantic keypoints.

[0060] Figure 2 This is a flowchart illustrating an example implementation of a process 200 for dynamic coordination of multiple robotic arms.

[0061] refer to Figure 2 In step 202, the computing platform 110 can acquire image data of a real-world (e.g., physical) environment from the perspective of the robotic device, which is associated with multiple tasks to be performed collaboratively by at least a first and a second robotic arm of the robotic device. For example, an image sensor may be mounted on the frame or robotic arm of the robotic device to acquire image data of the environment (e.g., a scene or workplace).

[0062] In some example embodiments, the first robotic arm may include an end effector operably engaged with the distal end of the first robotic arm for performing a first task on a first object. The second robotic arm may include the same or a different type of end effector operably engaged with the distal end of the second robotic arm for performing a second task on a second object. Example end effectors include rigid or flexible paddles (e.g., waffle paddles, solid boards, rakes), object grippers (e.g., hooks, augers, grippers), etc.

[0063] Environments can include a wide variety of environments, including indoor and outdoor environments, such as agricultural environments or other complex organic environments, construction sites, mining areas, asteroids, lunar landing sites, etc.

[0064] In step 204, computing platform 110 may identify one or more objects in the image data. In step 206, computing platform 110 may generate one or more parameterized representations of the identified objects (e.g., the world, objects of interest, and / or the robotic arm). Parameterizing the image data may include converting pixel data of one or more objects into compressed representations of one or more objects (e.g., parameterizing key features of the planning associated with the robotic arm to the desired extent). Instance-specific values ​​of the parameterized representations may be estimated through training and detection of semantic keypoints and key features.

[0065] in this regard, Figure 5Examples of simplified model construction are illustrated, including the observed strawberry and its corresponding parametric representation (e.g., a conical model or deformation-fitting model). In one example, computational platform 110 would identify the strawberry in the scene and determine its associated radius, length, conical shape, or other aspects. Low-level representations would be stored (e.g., using fewer bytes, such as 8 bytes, instead of representing individual vertices in a 3D model). These simplified models make the computational cost of planning any particular robotic arm lower (e.g., reducing processing and computation time) and provide a high update rate (e.g., 25 Hz or faster, allowing efficient synchronization of planning for individual robotic arms in a multi-arm system). This approach of reducing object complexity to a simplified state can be applied to various key features or objects in the environment.

[0066] Return to Figure 2 In step 208, the computing platform 110 can utilize one or more parameterized representations to generate a parameterized model (e.g., a parameterized world model) of a real-world environment associated with multiple tasks. The parameterized model can be dynamically updated. The parameterized model can include a three-dimensional volumetric mesh composed of voxels (the smallest unit being a voxel). Furthermore, each voxel can be assigned a low-dimensional representation of a state (e.g., occupancy state) or location of a part of the real-world environment, as well as semantic information associated with one or more objects (e.g., occupancy, deformability, geometry, and dynamic properties such as velocity and acceleration associated with one or more objects). For example, the three-dimensional volumetric mesh can represent the visibility level of a region of interest, ranging from highly occluded regions (which have a high potential to reveal occluded target objects) to occluded target regions with sufficient visibility (enough to determine that no target of interest exists). Additionally, semantic information associated with one or more objects (e.g., whether it is a plant, whether it is deformable, etc.) can be assigned to each of the multiple mesh cells. This information can be extrapolated to the future (e.g., "N" time steps forward from the current time). The parameterized model can be used to assist the computing platform 110 in estimating the next optimal scene interaction (e.g., the region most worth exploring to achieve the objective task). Alternatively, the parameterized model can include an octree-based representation that implements a variable 3D volume representation.

[0067] A parametric model of the world can include information about the elasticity and plasticity of objects in the environment, as well as the forces and moments required to deform or displace them. A parametric model with this information can be used to specify the requirements and constraints of a robot planning system.

[0068] in this regard, Figure 6An example parametric model 610 based on some example implementations is shown. A higher-level planning layer 110a can assign values ​​to different cells in the parametric model and identify cells expected to gain more information, for example, during exploration of the search space. Figure 6 As shown, lower values ​​can be assigned to cells of the dynamic occupancy grid corresponding to unoccluded areas of the scene. Similarly, medium values ​​can be assigned to cells of the dynamic occupancy grid corresponding to moderately occluded areas of the scene, and higher values ​​can be assigned to cells of the dynamic occupancy grid corresponding to heavily occluded areas of the scene.

[0069] Back Figure 2 In step 210, based on the parametric model, the computing platform 110 can determine the planned trajectory of the first robotic arm over a predetermined number of future time steps (also referred to herein as "motion planning"). In determining the planned trajectory of the first robotic arm, the computing platform 110 can identify whether obstacles will intersect the planned trajectory of the first robotic arm within a predetermined number of future time steps (e.g., three or fewer time steps ahead). This enables the subsequent movements of each robotic arm to be planned within a limited planning span by leveraging immediate awareness and understanding of the scene.

[0070] In this way, the computing platform 110 can generate one or more instructions to cause one or more robotic arms to manipulate a desired object while avoiding collisions with other objects and robotic arms that may exist in the environment or workspace. Alternatively, there may be situations where contact with obstacles or objects in the environment or with other robotic arms is required. In this case, the computing platform 110 can generate one or more instructions to induce such contact.

[0071] The planned trajectory of the first robotic arm can be selected from a set of candidate trajectories. The selection of the planned trajectory can occur when the current task of the first robotic arm is completed. Each candidate trajectory in the set of candidate trajectories can include a series of planned states of the first robotic arm occurring within a predetermined number of future time steps. Candidate trajectories can be filtered (e.g., based on dynamic constraints) and stacked or sorted, which can help select the planned trajectory from the candidate trajectories.

[0072] In step 212, the computing platform 110 can control the first robotic arm to execute a planned trajectory based on a first task among multiple tasks, for example, selecting from candidate trajectories described previously. In some examples, the planned trajectory may include manipulating a first object among one or more objects to reveal a second object in a real-world environment (e.g., to perform a second task among multiple tasks on the second object). For example, the second object may be an object at least partially or completely occluded by the first object. In examples, the first object may include at least a portion of the second robotic arm. In some examples, manipulating the first object may include temporarily or permanently deforming or displacing the first object volumetrically (e.g., pushing it away). In other examples, manipulating the first object may include picking up, tilting, grasping, pulling, twisting, flipping, or otherwise interacting with or moving the first object. For example, the computing platform 110 may use the first robotic arm to deform a first object in the environment to provide space for the second robotic arm to view, interact with, or otherwise contact another object in the environment (e.g., improve the visibility and accessibility of previously occluded areas or objects) while avoiding collisions between the first and second robotic arms.

[0073] Some implementations may include harvesting crops, which can utilize the ability of one or more robotic arms to manipulate and pass through organic matter (e.g., branches, leaves, etc.) to obtain crops (e.g., peppers, blueberries, stone fruits, avocados, leafy vegetables, etc.) or other objects of interest below. Other implementations may include weeding (e.g., removing normal plants to reach and remove unwanted plants). Another example may include using a first robotic arm to remove cables, thereby deforming one or more cables and making room for another robotic arm (which may interact with another of the one or more cables or perform some other activity).

[0074] Alternatively or concurrently, the manipulated object may include holding or placing the object in an environment using one of the robotic arms, and performing a second activity on the object using another robotic arm. For example, in a food processing task, a first robotic arm may perform an object-holding activity (e.g., holding ingredients), while another robotic arm may manipulate the object in a shared workspace (e.g., chopping ingredients, shaping dough) without damaging the manipulated object. In another example, when handling building materials (e.g., installing timber, pouring concrete, laying bricks, moving cables, etc.), a first robotic arm may be used to stabilize the material, and another robotic arm may be used to tighten or perform a specific activity. Such an activity may require a first robotic arm for the first manipulation (e.g., stabilizing) and a second robotic arm for performing the second manipulation.

[0075] The interaction with the environment described in this paper increases the quantity and quality of operational possibilities that the system can use for subsequent actions (e.g., what the scene can offer as a reward). It makes unknown, exploreable areas visible, thereby discovering potential rewards. In this respect, Figure 7 An example scene is shown before (left) and after (right) interaction with the scene, according to some embodiments disclosed herein, revealing a cluster of previously obscured strawberries.

[0076] Scene interaction is achieved through a combination of end effectors and motion primitives, with the motion primitives moving the end effectors in a useful manner. In the strawberry picking application described above, a situation arises where a robotic arm needs to interact with branches and leaves to improve the visibility and / or accessibility of the "picking" robotic arm within the image acquisition system. Figures 9-11 Example classes of non-grasping motion primitives for interacting with strawberry plants are depicted, represented as tilt ( Figure 9 ), twist ( Figure 10 ) and flipping ( Figure 11 These can serve a dual purpose, including making previously obscured areas visible and clearing the path for harvesting tasks to avoid collisions with plants or robotic arms. More specifically, Figure 9 A first example manipulation element 900 is shown, which may include using a flat-plate end effector to push the upper part of the plant (e.g., branches and leaves) away from the base of the plant. Pushing the plant to one side creates space for inspection (middle) and harvesting (right side) with a camera. Figure 10 A second example of a manipulation element 1000 is shown, in which a robotic arm grasps the stem and twists the plant using a helical end effector. Figure 11 A third example of a manipulation element 1100 is shown, in which a flat-plate end effector moves at the canopy level via a robotic arm, sequentially revealing the center when it is obscured by the canopy, to help distinguish individual plants.

[0077] Figures 12-15 Variations of end effectors for use with robotic arms are depicted, which can be deployed for branch and leaf manipulation, including the Waffle Mesh 1200 ( Figure 12 ), solid board 1300 ( Figure 13 ), hooks and spiral extractors 1400 ( Figure 14 ) and a rake 1500 with connecting teeth ( Figure 15 ).

[0078] Accordingly, in some implementations, a library of descriptive motion types from which the final end effector can manipulate and select can be created as implementable functions, stored in a codebase and callable by higher-level planning modules. Interaction with the scene can increase the quantity and quality of operational possibilities available to the system for subsequent actions (e.g., what the scene can offer as a reward).

[0079] Back Figure 2 In step 214, the computing platform 110 can update the parameterized model of the real-world environment associated with multiple tasks. For example, in the event of scene changes, the update can occur at each time step. Dynamically updating the parameterized model of the real-world environment associated with multiple tasks can include determining the planned trajectory of the second robotic arm over a predetermined number of future time steps. In some examples, the parameterized model of the real-world environment associated with multiple tasks can be updated to include additional objects identified by image sensors on the robotic device and parameterized to a finite set of model parameters.

[0080] In other words, during the execution of the planned trajectory, the computing platform 110 can dynamically plan and adjust its behavior, triggered by real-time monitored environmental changes. For example, discovering a new object during a maneuver in which a robotic arm pushes aside a first object (e.g., discovering a new strawberry during a maneuver in which branches and leaves are pushed aside) can trigger another robotic arm to move and manipulate the newly discovered object (e.g., picking strawberries). Alternatively, if the new object is still partially obscured (e.g., strawberries are still obscured by branches and leaves), the other robotic arm can modify its configuration until the desired task can be performed on the newly discovered object (e.g., until strawberries can be picked).

[0081] In step 216, the computing platform 110 can control the second robotic arm to collaboratively perform the second task among multiple tasks on the second object, working in an overlapping workspace, based on an updated parametric model. The control of the robotic arm can occur automatically without user interaction. Advantageously, the parametric model of the real-world environment associated with the multiple tasks can be cyclically updated for each additional robotic arm used to perform one of the multiple tasks, in a cyclical scheduling manner. Because a simplified model is used instead of real-time raw data, the computing platform 110 can be configured to process information rapidly, thereby performing path planning for the robotic arms (with “N” degrees of freedom) arm-by-arm in this cyclical scheduling manner. This achieves a fast refresh rate without needing to plan for a large number of dimensions. Furthermore, since collaboration is defined at the task level rather than the motion planning level, planning can be performed separately for each arm. This decentralized approach allows for the use of multiple robotic arms without increasing planning time.

[0082] In some implementations, if the computing platform 110 encounters a local minimum or is unable to plan, the system can request a different plan or escalate the plan to a higher level (e.g., to a higher-level planning layer 110a). The higher-level planning layer 110a can identify the problem and trigger fallback or recovery actions (e.g., immediately using an available second plan).

[0083] Figures 3-4 An example implementation of the computational process for dynamically coordinating multiple robotic arms at each time step is shown.

[0084] refer to Figure 3 In step 302, the computing platform 110 can calculate various targets (e.g., business-level targets) for the multiple robotic arms. In step 304, the computing platform 110 can assign individual tasks to each robotic arm. For example, one robotic arm can be assigned to pick fruit observed in the scene (picking arm), while another robotic arm can be assigned to remove branches and leaves to make room for the picking arm. Tasks can include sequences transcribed into various types of movements. For example, the motion planning for one robotic arm can include moving from its current position to the target fruit, while the motion planning for another robotic arm can include moving parallel to the ground and removing branches and leaves.

[0085] In step 306, the computing platform 110 can calculate a set or collection of trajectories with associated costs for each robotic arm. For example, refer to... Figure 4 In section 410, each robotic arm can consider a quasi-static scenario and plan a set of trajectories from the current time step (t) to the future (t, t+1, t+2) within a time window. In section 412, the cost associated with a specific task can be allocated to each trajectory. For example, the next two time steps can be calculated, and different trajectories can be identified based on the corresponding costs.

[0086] Back Figure 3 In step 308, the computing platform 110 can calculate the arm movement priority based on a priority mechanism (e.g., "circular scheduling"). In step 310, considering cost and efficiency, an optimal trajectory can be selected for each robotic arm (e.g., the area most worth exploring to achieve the target task). For example, if the first robotic arm is selected first to choose the trajectory to be executed, it will select the optimal trajectory for its target, and the selected trajectory will be invalid for the second robotic arm. Conversely, if the second robotic arm is selected first to choose the trajectory to be executed, it will select the optimal trajectory for its target, and the selected trajectory will be invalid for the first robotic arm. In both cases, the invalid selected trajectory can force either robotic arm to choose a suboptimal trajectory that is still feasible for the task to be executed.

[0087] In step 312, the trajectory (also known as "motion planning") is sent to the execution level (e.g., execution level 110d) that causes the individual robotic arms to perform the motion. In step 314, the state of the world is monitored while the motion is being performed. If any change in the world model triggers a change, a signal is sent to the previous arbitrary boxes (302-310) to take appropriate action. For example, the position and orientation of the target fruit may change when branches are pushed aside. While the overall goal and the specific tasks assigned to the individual robotic arms may not need to change (the picking arm still needs to pick, and the prying arm still needs to push branches aside), the set of trajectories can be recalculated, and aspects of the invention enable such replanning for one or more robotic arms. In this respect, Figure 4 The time step t+1 in the diagram illustrates the repeated process, as well as the trajectory planned for future time steps (t+1, t+2, t+3) and selected based on which arm has priority.

[0088] Some implementations of the current topic can realize automated systems with human-like behavior that interacts with the environment. Some implementations of the current topic can achieve one or more of the following: simulating human-like movement without human intervention; enabling multiple robotic arms to collaborate in a shared workspace to perform tasks, where their coordination is crucial to the success of the task; not limiting each actuator (e.g., robotic arm) to operating the same object, but improving the quality of scene understanding of each actuator in the process; and enabling the independent movement of each robotic arm to achieve higher-level goals. For example, Figure 16 This illustrates some implementations of the current topic that enable dynamic replanning 1600 caused by unpredictable changes in a scene. For example, when branches are pushed aside, the position and orientation of a strawberry may change with the displacement of the branches. The computing platform 110 can adapt to this change by replanning one or more robotic arms.

[0089] In some implementations, the high-level goals of the computing platform 110 for the robot can be set by an expert user. For example, an expert user can decide the threshold for when to harvest fruit, as environmental conditions will force different aspects to become priorities. For example, if rainfall is forecast, the expert user can lower the ripeness threshold of the fruit to be harvested. In another example, the expert user can make adjustments for specific needs, such as specifying a target area for fruit harvesting while avoiding other areas.

[0090] In the foregoing description and claims, phrases such as “at least one of…” or “one or more of…” may appear after a successive list of elements or features. The term “and / or” may also appear in a list of two or more elements or features. Unless otherwise implicitly or explicitly contradicted by the context, this phrase is intended to mean any element or feature listed individually, or any recounted element or feature combined with any other recounted element or feature. For example, the phrases “at least one of A and B”; “one or more of A and B”; and “A and / or B” are intended to mean “A alone, B alone, or A and B together”, respectively. Similar interpretations are also intended for lists comprising three or more items. For example, the phrases “at least one of A, B, and C”; “one or more of A, B, and C”; and “A, B, and / or C” are intended to mean “A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together”, respectively. Furthermore, the use of the term "based on" in the above and claims is intended to mean "at least partially based on," thereby also allowing for features or elements not described.

[0091] Depending on the desired configuration, the subject matter described herein can be implemented in systems, devices, methods, and / or articles. The embodiments set forth in the foregoing specification do not represent all embodiments consistent with the subject matter described herein. Rather, they are merely examples of aspects consistent with the described subject matter. Although some variations have been described in detail above, other modifications or additions are possible. Specifically, further features and / or variations may be provided in addition to those features and / or variations set forth herein. For example, the foregoing embodiments may be for various combinations and sub-combinations of the disclosed features, and / or combinations and sub-combinations of several further features disclosed above. Furthermore, the logical flows depicted in the appended drawings and / or described herein do not necessarily require the specific order or sequential order shown to achieve the desired results. For example, the logical flow may include operations different from and / or additional to those shown without departing from the scope of the invention. One or more operations of the logical flow may be repeated and / or omitted without departing from the scope of the invention. Other embodiments are within the scope of the appended claims.

Claims

1. A method comprising: From the perspective of the robotic device, acquire image data of the real-world environment associated with multiple tasks to be performed collaboratively by at least the first and second robotic arms of the robotic device; Identify one or more objects in image data; Generate one or more parameterized representations of one or more objects in the image data; A parameterized model of a real-world environment associated with multiple tasks is generated using one or more parameterized representations, wherein the parameterized model comprises a three-dimensional volumetric mesh composed of voxels, wherein each voxel is assigned a low-dimensional representation of the state of a portion of the real-world environment and semantic information associated with one or more objects. Based on the parametric model, the planned trajectory of the first robotic arm within a predetermined number of time steps is determined, including identifying whether obstacles will intersect with the planned trajectory of the first robotic arm within the predetermined number of time steps. Control the first robotic arm to execute a planned trajectory based on the first of multiple tasks, including manipulating the first object of one or more objects to reveal the second object in a real-world environment; Dynamically update the parameterized model of the real-world environment associated with multiple tasks; and Based on the updated parameterized model, the second robotic arm is controlled to perform the second task of a series of tasks on the second object, wherein the second task is performed in coordination with the first task.

2. The method according to claim 1, wherein, Dynamically updating the parameterized model related to multiple tasks includes determining the planned trajectory of the second robotic arm within a predetermined number of time steps.

3. The method according to claim 1, wherein, Parameterizing image data includes converting pixel data of one or more objects into a compressed representation of one or more objects.

4. The method according to claim 1, wherein, The second object is an object that is at least partially obscured by the first object.

5. The method of claim 1, wherein the first object comprises at least a portion of the second robotic arm.

6. The method according to claim 1, wherein, The image data was acquired using the image sensor of the robotic device.

7. The method according to claim 1, wherein, Dynamically updating the parameterized model of the real-world environment associated with multiple tasks includes updating it to include additional, parameterized objects identified by the image sensors of the robotic device.

8. The method according to claim 1, wherein, Manipulating the first object of one or more objects includes: deforming or displacing the first object in volume.

9. The method according to claim 1, wherein, The predetermined number of time steps is three or fewer future time steps.

10. The method according to claim 1, wherein, The planned trajectory of the first robotic arm is selected from a set of candidate trajectories of the first robotic arm, wherein each candidate trajectory in the set of candidate trajectories includes a series of planned states of the first robotic arm that occur within a predetermined number of time steps.

11. The method according to claim 1, wherein, Dynamically updating the parameterized model of the real-world environment associated with multiple tasks involves cyclically updating the model for each additional robotic arm used to perform one of the multiple tasks in a cyclical scheduling manner.

12. The method according to claim 1, wherein, The first robotic arm includes an end effector operatively engageable with the distal end of the first robotic arm for performing a first task on a first object.

13. The method according to claim 12, wherein, The end effector includes a flexible paddle.

14. The method according to claim 1, wherein, The second robotic arm includes an end effector operatively engageable with the distal end of the second robotic arm for performing a second task on a second object.

15. The method according to claim 14, wherein, The end effector includes an object gripper.

16. The method according to claim 1, wherein, The real-world environment is the outdoor environment.

17. The method according to claim 1, wherein, The control of the first robotic arm and the control of the second robotic arm occur without user interaction.

18. An apparatus comprising: Multiple robotic arms; One or more image sensors; processor; as well as A memory that stores instructions, which, when executed by a processor, cause the following operations: Acquire image data of the real-world environment associated with multiple tasks to be performed collaboratively by at least the first and second robotic arms from one or more image sensors; Identify one or more objects in image data; Generate one or more parameterized representations of one or more objects in the image data; A parameterized model of a real-world environment associated with multiple tasks is generated using one or more parameterized representations, wherein the parameterized model comprises a three-dimensional volumetric mesh composed of voxels, wherein each voxel is assigned a low-dimensional representation of the state of a portion of the real-world environment and semantic information associated with one or more objects. Based on the parametric model, the planned trajectory of the first robotic arm within a predetermined number of time steps is determined, including identifying whether obstacles will intersect with the planned trajectory of the first robotic arm within the predetermined number of time steps. Control the first robotic arm to execute a planned trajectory based on the first of multiple tasks, including manipulating the first object of one or more objects to reveal the second object in a real-world environment; Dynamically update the parameterized model of the real-world environment associated with multiple tasks; and Based on the updated parameterized model, the second robotic arm is controlled to perform the second task of a series of tasks on the second object, wherein the second task is performed in coordination with the first task.

19. The apparatus according to claim 18, wherein, The memory further stores instructions that, when executed by the processor, cause operations including the method according to any one of claims 1 to 17.

20. A non-volatile computer-readable medium storing instructions thereon for controlling a robotic device, which, when executed, cause a processor to perform the following operations: From the perspective of the robotic device, acquire image data of the real-world environment associated with multiple tasks to be performed collaboratively by at least the first and second robotic arms of the robotic device; Identify one or more objects in image data; Generate one or more parameterized representations of one or more objects in the image data; Parametric models of real-world environments associated with multiple tasks are generated using one or more parametric representations, where, The parametric model consists of a three-dimensional volumetric mesh composed of voxels, where each voxel is assigned a low-dimensional representation of the state of a portion of a real-world environment and semantic information associated with one or more objects. Based on the parametric model, the planned trajectory of the first robotic arm within a predetermined number of time steps is determined, including identifying whether obstacles will intersect with the planned trajectory of the first robotic arm within the predetermined number of time steps. Control the first robotic arm to execute a planned trajectory based on the first of multiple tasks, including manipulating the first object of one or more objects to reveal the second object in a real-world environment; Dynamically update the parameterized model of the real-world environment associated with multiple tasks; and Based on the updated parameterized model, the second robotic arm is controlled to perform the second task of a series of tasks on the second object, wherein the second task is performed in coordination with the first task.

21. A non-volatile computer-readable medium further stores instructions that, when executed by a processor, cause operation comprising the method according to any one of claims 1 to 17.

Images