Hand-eye reinforcement learner for dynamic grasping with active pose estimation
By using the Hand-Eye Augmented Learner (EARL) system, combined with the coupling of a wrist-mounted camera and a robotic arm, good generalization and robust tracking of complex trajectories never seen before are achieved in dynamic environments. This solves the problem of grasping moving targets in dynamic environments and improves the system's flexibility and grasping success rate.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-05
- Publication Date
- 2026-04-07
AI Technical Summary
Existing technologies struggle to achieve stable tracking and grasping of moving targets in dynamic environments, especially in space-constrained or occluded situations. Furthermore, existing methods typically require extensive training data or are applicable to discrete action sets, making it difficult to generalize to unseen complex trajectories.
Employing a hand-eye reinforcement learner (EARL) system, the system couples a wrist-mounted camera with a robotic arm, utilizing active pose tracking and course-based reinforcement learning to achieve dynamic grasping synthesis with all six degrees of freedom. By combining RGB-D camera input and machine learning feature maps, the system dynamically selects the grasping pose and adjusts the robotic arm's movement in real time.
It achieves good generalization to unseen complex trajectories in both simulated and real-world environments, robustly tracks and successfully grasps moving targets, improving the system's flexibility and grasping success rate in dynamic environments.
Smart Images
Figure CN121816249A_ABST
Abstract
Description
Technical Field
[0001] This invention generally relates to a system and controller that enable an eye-on-hand robotic arm system to perform tracking and grasping of moving target objects. Background Technology
[0002] Robotic manipulation of everyday objects in dynamic environments constitutes a fundamental skill for realizing next-generation advanced robotic systems. Providing robots with six degrees of freedom (DoF) pose tracking and grasping capabilities in unstructured and dynamic environments beyond static desktop scenarios can benefit many automation applications. Examples include human-to-robot object transfer and the assembly of industrial parts.
[0003] Autonomous grasping and manipulation of objects in static environments where the object does not move has been extensively studied. Dynamic environments present several challenges for performing grasping. First, the target object may move with unknown motion, requiring understanding and predicting the object's motion or continuous tracking and active following by the manipulator. Second, computational motion planning may fail, thus dynamic environments require online or rapid replanning. Furthermore, the approach direction for the grasping plan changes with the object's movement, and therefore, stable grasping may become unreachable and unstable as the target object moves. Active perception implies computer vision with a movable camera, which can intelligently change the camera's viewpoint to improve system performance. Dynamic grasping using hand-eye (EoH) systems requires active perception because the robot may lose tracking of the target object due to the robot's movement or when the target moves away from the camera's field of view (FoV).
[0004] The vast majority of current vision-based manipulation systems typically utilize cameras fixed above the workspace. Therefore, the perception subsystem (e.g., an RGB-D camera) and the manipulation system (e.g., a robotic arm) are decoupled. This setup implicitly assumes an ideal viewing distance and angle for focusing on the target object. This fixation may require large gaps above / around the workspace, making the entire system inflexible and unsuitable for some applications, especially when obstruction is unavoidable or space is limited, such as retrieving spice jars from a cabinet or using a snake-like robot to perform inspections in pipes. This decoupling can lead to occlusion or tracking loss in many scenarios, especially when the target object is moving. Therefore, mounting the camera on the wrist of the manipulator can be an alternative. In some cases, vision sensors are mounted on the robotic arm near the end effector; however, past pose tracking often utilized a separate, static workspace camera at a distance, decoupling the perception process from the object manipulation operation.
[0005] Vision-based robotic grasping solutions can be categorized along several dimensions. Model-based methods rely on knowledge about the target object, such as CAD models. Model-free methods directly propose grasping candidates and aim to generalize to new objects. Analytical or geometric methods analyze the shape of the target object for grasping pose recognition. Data-driven grasping synthesis uses learning and has made significant progress due to better learning methods and data availability. Some methods use deep neural networks to sample and rank grasping candidates. Reinforcement learning methods are suitable for grasping synthesis, where appropriate candidates are learned and validated using repeated interactions in simulated or real-world environments. However, many end-to-end RL-based methods for grasping interactions require large amounts of data for training or are only applicable to discrete action sets. Many methods are characterized by constrained state-action spaces, such as up-and-down grasping concentrated in 4-DoF, and are therefore limited in their task scope.
[0006] Grasping in dynamic environments presents additional challenges, requiring adaptive and real-time motion and grasping planning for the robot. Generative grasping convolutional neural networks (CNNs) for fast closed-loop grasping in slightly moving scenes can be used for 4-DoF up-and-down grasping, but they have high requirements for data annotation. A common approach to dynamic grasping involves motion prediction for the target object to improve success. Motion prediction capabilities can help plan grasping and approaching the object. Reachability and motion-aware solutions implement recurrent neural networks (RNNs) for modeling and predicting object motion, which are applicable to linear, sinusoidal, and circular trajectories. However, it may have poor generalization in unseen and complex trajectories (e.g., random movement), and this solution uses a position-fixed camera.
[0007] Some assumptions need to be relaxed and the above challenges addressed by using a "hand-eye" (EoH) system, in which the sensory perception system is coupled with the manipulator, to track the robot's grasping. Summary of the Invention
[0008] Some embodiments of the present invention provide a method that utilizes the continuous motions of a robotic arm generated by a course-trained model-free reinforcement learning strategy to perform dynamic grasping synthesis in full six degrees of freedom (DoF) to grasp target objects from a wider range of unstructured environments beyond a desktop scene, and utilizes a wrist-mounted camera to perform tracking of new objects for dynamic grasping.
[0009] In dynamic grasping problems, a moving target object is grasped and lifted, and the object's pose and motion are not known a priori and must be estimated using visual observation. This invention discloses a novel method and apparatus that endows an eye-hand (EoH) system with the ability to track and grasp moving target objects. This document describes techniques for solving the full 6-DoF dynamic grasping problem of a new object without prior knowledge of its motion properties. In at least one embodiment, the EoH system includes a high-DOF robotic manipulator, an end effector, and a wrist-mounted sensory perception system fixed to the manipulator near the end effector. The coupling between the perception and manipulation subsystems means that such an EoH system can operate without workspace constraints and the visual occlusion faced by most existing systems using external cameras. In at least one embodiment, the workspace is the area accessible to the robotic arm for grasping the desired target object. In at least one embodiment, the system uses an RGB-D camera input as a vision sensor for visual observation. The visual observation includes image data acquired from the camera, and the image data includes one or more channels. In at least one embodiment, the visual observation can be used to learn feature maps representing high-level features of the input image. These feature maps contain an abstract representation of the image and task-related encoded information. In another implementation, visual observations can be encoded as the spatial location of a target object as a six-dimensional (6D) pose in the workspace, indicating the position and orientation of the target object relative to a camera frame, gripper, or base frame of the robot arm. The base frame is attached to the robot arm's base, and the gripper frame is attached to the end effector. A camera coordinate system is attached to the camera, and the transformation from the camera coordinate system to the robot's base frame is determined. This method performs active pose tracking to encode visual observations and is trained through a carefully designed curriculum to overcome the challenges of dynamic grasping with hand-eye systems. The method includes active pose tracking, grasping pose computation and adaptation, and a curriculum-based reinforcement learning strategy. We propose a curriculum-based model-free RL method with a wrist-mounted camera for picking up moving objects through continuous motions. Our proposed method, called Hand-Eye Augmented Learner (EARL), approaches a target while maintaining it within the FoV of a moving camera, despite abrupt changes in distance and viewpoint between the camera and the target. Our method does not limit grasping synthesis to the up-down direction and generalizes well to unseen and complex trajectories. Through extensive evaluation, we demonstrate that EARL robustly and efficiently tracks and approaches moving target objects until successful grasping is achieved. Evaluations of the method are provided in simulated and real-world robotic experiments. In one implementation, the technique described herein is shown to achieve high success rates in dynamic grasping of a variety of commonly used objects across numerous experiments.In at least one implementation, the model is trained purely in simulation and is adapted to the real world by utilizing direct simulation-to-reality adaptation.
[0010] Some embodiments of the present invention provide a controller for using a robotic arm to perform dynamic grasping of a target object based on visual perception input. The controller may include: a data input / output interface configured to receive state measurements of the robotic arm and the target object from sensors arranged on the robotic arm, wherein the robotic arm includes a gripper connected by joints having actuators and encoders and an end effector of the robotic arm, the gripper being configured to grasp the target object in response to robot control signals; wherein the sensors include a vision sensor configured to continuously provide visual observations for tracking the pose of the target object in the workspace and calculating the grasping pose; wherein the vision sensor is mounted on the distal end of the robotic arm adjacent to the gripper; and a memory configured to store hand-eye (EoH) reinforcement learning (EARL) strategies, a physics-based simulator, and the arm itself. A motion generation program; and a processor connected to the memory, configured to perform the following steps: training the hand-eye reinforcement learning policy; tracking the pose of the target object moving in the workspace based on the state measurements; calculating a set of grasping poses on the target object and dynamically selecting a desired grasping pose on the target object moving in the workspace; calculating robotic arm motion commands using the trained hand-eye reinforcement learning policy; generating robot control signals based on the calculated robotic arm motion commands; and transmitting the robot control signals to the actuators and grippers of the joints via the data input / output interface to follow the target object while holding it in the field of view of the vision sensor and grasping the target object in the workspace.
[0011] Furthermore, according to some embodiments of the present invention, a system is provided for performing dynamic grasping of a target object using visual perception input. The system may include a robotic arm comprising links connected by joints having actuators and encoders, and a gripper with an end effector of the robotic arm, the gripper being configured to grasp the target object in response to robot control signals; a vision sensor configured to continuously provide / generate visual observations (images or image signals) for tracking the pose of the target object in a workspace and calculating the grasping pose, wherein the vision sensor is mounted on the distal end of the robotic arm adjacent to the gripper; a memory configured to store hand-eye (EoH) reinforcement learning (EARL) strategies, a physics-based simulator, and an arm motion generation program; and so on. A processor connected to the memory is configured to perform the following steps: training a hand-eye reinforcement learning policy; tracking the pose of a target object moving in the workspace; calculating a set of grasping poses on the target object and dynamically selecting a desired grasping pose on the target object moving in the workspace; calculating robotic arm motion commands using the trained hand-eye reinforcement learning policy; generating robot control signals based on the calculated robotic arm motion commands; and transmitting the robot control signals to the actuators and grippers of the joints to follow the target object while holding it in the field of view of the vision sensor and grasping the target object in the workspace.
[0012] The accompanying drawings, included to provide a further understanding of the invention, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention. The drawings shown are not necessarily drawn to scale, but generally focus on illustrating the principles of the currently disclosed embodiments. Attached Figure Description
[0013] [ Figure 1 ]
[0014] Figure 1 A hand-eye (EoH) robotic system with a robotic manipulator and a wrist-mounted camera is shown.
[0015] [ Figure 2 ]
[0016] Figure 2 A diagram illustrating an example of a system based on the proposed hand-eye reinforcement learner according to an embodiment of the present invention is shown.
[0017] [ Figure 3 ]
[0018] Figure 3 A diagram illustrating the proposed EARL framework according to some embodiments of the present invention is shown.
[0019] [ Figure 4 ]
[0020] Figure 4 An overview of training EARL in a simulation using reinforcement learning with course training according to an embodiment of the present invention is shown.
[0021] [ Figure 5 ]
[0022] Figure 5 An overview of the baseline control strategy according to an embodiment of the present invention is shown.
[0023] [ Figure 6A ]
[0024] Figure 6A A training set object used in a simulation for training strategy is shown according to an embodiment of the present invention.
[0025] [ Figure 6B ]
[0026] Figure 6B A validation set object used in a simulation for training a strategy, according to an embodiment of the present invention, is shown. [ Figure 6C ] Figure 6C A real-world test set object according to an embodiment of the present invention is shown.
[0027] [ Figure 7A ]
[0028] Figure 7A One of four motion modes of the target object considered in the evaluation according to an embodiment of the present invention is shown, and these motions occurring in 3D and 6D are shown in 2D for visualization purposes.
[0029] [ Figure 7B ]
[0030] Figure 7B One of four motion modes of the target object considered in the evaluation according to an embodiment of the present invention is shown, and these motions occurring in 3D and 6D are shown in 2D for visualization purposes.
[0031] [ Figure 7C ]
[0032] Figure 7C One of four motion modes of the target object considered in the evaluation according to an embodiment of the present invention is shown, and these motions occurring in 3D and 6D are shown in 2D for visualization purposes.
[0033] [ Figure 7D ]
[0034] Figure 7D One of four motion modes of the target object considered in the evaluation according to an embodiment of the present invention is shown, and these motions occurring in 3D and 6D are shown in 2D for visualization purposes.
[0035] [ Figure 8A ]
[0036] Figure 8A An experimental task for evaluating EARL in a challenging setting is illustrated according to an embodiment of the present invention.
[0037] [ Figure 8B ]
[0038] Figure 8B An experimental task for evaluating EARL in a challenging setting is illustrated according to an embodiment of the present invention.
[0039] [ Figure 8C ]
[0040] Figure 8B An experimental task for evaluating EARL in a challenging setting is illustrated according to an embodiment of the present invention.
[0041] [ Figure 9 ]
[0042] Figure 9 The simulation results (all values are in percentage) according to an embodiment of the present invention are shown, where AR refers to the algorithm (baseline or EARL) and the robot 6-DoF (R1) or 7-DoF (R2), and M refers to the static (S), linear (L), elliptical, random (R) motion mode type.
[0043] [ Figure 10 ]
[0044] Figure 10 Real-world experimental results (all values are in percentage) according to an embodiment of the invention are shown, wherein 20 trials were performed on a real robot as an EoH system for each test case. Detailed Implementation
[0045] Various embodiments of the invention are described below with reference to the accompanying drawings. It should be noted that the drawings are not drawn to scale, and elements with similar structures or functions are indicated by the same reference numerals throughout the drawings. It should also be noted that the drawings are intended only to facilitate the description of specific embodiments of the invention. They are not intended as an exhaustive description of the invention or as a limitation on its scope. Furthermore, aspects described in connection with specific embodiments of the invention are not necessarily limited to those embodiments and can be practiced in any other embodiment of the invention.
[0046] This work endows the hand-eye (EoH) system with the ability to track and grasp moving target objects. The EoH system comprises a high-DOF robot manipulator, an end effector, and a wrist-mounted sensing system fixed to the manipulator near the end effector. The coupling between the sensing and manipulating subsystems means that this EoH system can operate without the workspace constraints and occlusions faced by most existing systems that use external cameras.
[0047] This invention introduces a high-performance manipulation framework for hand-eye (EoH) robotic systems, which includes the proposed active pose tracking using a moving camera and a course-based reinforcement learning (EARL) method that has simulation-to-reality generalization capability for full 6-DoF dynamic grasping of new objects without prior indication of the object's motion properties.
[0048] Problem Definition: We focus on enabling coupled EoH systems to operate at 6-DoF (i.e., ...). This involves dynamically grasping a moving object with previously unknown motion. We make no assumptions about the object's shape or identity, except that the target object is a rigid body and can be grasped by the end effector. Furthermore, the target object can move freely within the robot's reachable workspace (robot-accessible area). Completing this task using the EoH system requires a subtask of tracking the moving object, utilizing the robot's motion to follow the object and keep it within the FoV (FoV) of the EoH camera as it approaches the target. The task is successful if the robot can grasp and pick up the object.
[0049] Figure 1 A hand-eye (EoH) system 100 is shown, which has a wrist-mounted RGB-D camera as a vision sensor 110 and a robotic arm 120 as a manipulator. Figure 2A diagram illustrating an example of a system based on the proposed hand-eye reinforcement learning (EARL) according to an embodiment of the present invention is shown. The robot system 200 may include a controller (robot controller) 250 and a robotic arm manipulator 220 equipped with sensors (not shown) for state measurement and a computerized instrumentation system for storing data and controlling the manipulator arm 220. The controller 250 may include a data input / output interface (data input / output interface circuitry) 211, a processor, and a memory 212. The memory 212 is configured to store programs 213 and policies 214, including hand-eye (EoH) reinforcement learning (EARL) policies, a physics-based simulator, and an arm motion generation program. The manipulator arm 220 may include several rigid links 201, 202, 203 and joints 204, 205, 206. The robot controller 250 is used to control the manipulator arm 220, and the robot controller 250 generates commands or tasks (robot commands) 215 that can be provided to the system 200 from the outside. Command or task 215 may be control signal 216, which operates the actuator of manipulator arm 220 to track or grasp object 221 using the manipulator's gripper 210. Robot controller 250 sends control signal 216 to manipulator arm 220. Control signal 216 may be a torque, speed, or position command to be applied at each of joints 204, 205, 206 of manipulator arm 220, and the opening / closing of gripper 210. The state of the robot system is measured using sensors. These sensors may include encoders configured to detect joint position, speed, and torque values at joints 204, 205, 206, and a camera (vision sensor) 231 that can observe the robot's environment. In this case, camera 231 may be attached to the end effector of manipulator arm 220. Camera 231 has a field of view (FoV) 232. The EoH system (robot system) 200 is configured to keep the object of interest 221 in the FoV 232 of the camera 231 by controlling the robotic arm as the target object 221 moves within the workspace. State measurements from sensors are sent to a data input / output interface 211, which stores the data received from the sensors. The robot controller 250 is initialized using a control strategy 214 for the desired task. Several episodes of the control strategy 214 are implemented on the robot system 200, and state measurement data from the sensors are collected in the data input / output interface 211. The processor then processes the data using software containing a computer program 212 for updating or executing the current strategy 214 of the robot system. The program 213 for updating the strategy 214 may be an EARL method program for training or calculating parameters for the strategy 214 in one step.The controller 250 then calculates robot commands (strategy commands) 215, generates control signals 216 based on the calculated robot commands 215, and transmits the control signals 216 to the actuators of the robotic arm manipulator 220 using the data input / output interface 211 to perform the desired task.
[0050] EARL framework
[0051] In this section, we describe the various components of the framework. First, we give an overview of our framework. Then, we describe the vision processing unit for real-time tracking of target object poses. Next, we describe the grasp planning pipeline that continuously adapts to grasp selection. Finally, we propose an arm motion generation method based on curriculum-based reinforcement learning (RL).
[0052] Grasping in dynamic environments presents numerous challenges. Stable and reachable grasping can become unreachable and unstable as the target object moves. EoH systems may also lose tracking of dynamic targets. We apply RL to EoH systems, where we learn a control policy (EARL) in simulation for dynamic grasping of arbitrary objects, and we propose utilizing the object's active pose information as visual feedback. Our proposed method enables dynamic grasping using active tracking of new objects without explicit motion prediction requirements. First, we perform active pose estimation of the target using a moving EoH camera to provide visual feedback. In at least one implementation, we opt for a design that continuously tracks the target object in 2D using learned features, and then use a combination of 2D features and depth information to perform pose estimation to accelerate processing and ensure sufficient tracking accuracy.
[0053] In at least one implementation, the control strategy is based on a proximal policy optimization (PPO) RL framework, where we map the object pose difference to the desired robotic arm joint velocities. Since we are working with an EoH system, the robotic arm must continuously adjust the pose of the RGB-D camera to hold the target object in FoV. Simultaneously, the control strategy must guide the end effector to approach and grasp the object. All these requirements can only be achieved naturally and smoothly in real-time through a finely tuned, multi-stage control strategy. In at least one implementation, we perform a grasping plan with multiple grasping proposals on the target object, and the optimal grasping pose is dynamically updated, which may occur because the target object moves continuously in six degrees of freedom (6D). We dynamically select suitable candidates for the bias control strategy actions. The overall framework is... Figure 3 The steps are shown in the diagram and described in detail in the following sections. Figure 3The diagram illustrates the components for initializing object tracking and mask refinement using an RGB-D image input for pose estimation. The pose tracking module provides the 6D pose of the target object. A grasp pose pool is constructed from the grasp proposal network. The grasp planning module calculates the appropriate grasp pose on the target. The control strategy commands the EoH system to use the robotic arm to track and ultimately grasp the target.
[0054] Active object pose tracking
[0055] Active perception refers to computer vision implemented using a movable camera, which can intelligently change the viewpoint to improve system performance. In at least one embodiment, we consider using a moving camera for tracking. Many methods focus on up-and-down grasping, where they constrain the target object within a 2D workspace. This can be relatively simple. In this invention, we aim for 6-DoF grasping synthesis, which requires the robot to know the 3D position and 3D orientation of the target object. In at least one embodiment, we utilize an EoH system to perform 6D active pose estimation for tracking and computing the encoding of the target object from visual observation. Typically, 6D object pose estimation methods assume a known object model and can be categorized as correspondence-based, template-based, and vote-based. In this invention, we make no assumptions about the object's model and motion properties and continuously track the object's pose using a moving camera.
[0056] In at least one implementation, given a target object O The first frame of RGB-D image In the image At any time t Continuous tracking O Relative to the camera's 6D pose. In at least one implementation, we achieve this via a correspondence-based approach, utilizing the BundleTrack method to maintain a keyframe memory pool and performing online pose graph optimization that extracts feature correspondences and dense pixel-wise correspondences from the depth image. To achieve fast and active pose tracking of new moving objects using only RGB-D image sequences, we introduce several enhancements to improve BundleTrack. In time... t The pose tracking process begins with obtaining the bounding box of the target object. We employ a transformer-based 2D object tracker, which can track objects across multiple frames using a series of RGB images. Then, we refine the bounding boxes using an alpha-refine method. The fitting was performed, and the object mask was obtained simultaneously. Alpha-Refine is an augmentation method for extracting and maintaining detailed spatial information. These augmentations enable fast tracking and application to new objects without requiring additional training or object models. We utilize masks... To provide more accurate RGB and depth information as input to BundleTrack, a direct performance gain is obtained for fast 6D pose tracking. A key component of correspondence-based pose estimation is the feature extractor. In at least one implementation, we employ the R2D2 feature detector, which predicts a set of sparse locations as object features in the input image. These object features are repeatable and reliable for local feature matching and therefore outperform the LF-Net detector used in BundleTrack. We note that the color image input to R2D2 only contains features based on… The target object to be clipped. Our method enables robust active pose tracking in real-time robot control.
[0057] Scraping Planning
[0058] In at least one implementation, we use a first RGB-D image to initialize pose tracking and compute grasping proposals. Selecting a good set of candidate grasping poses is important for two reasons. First, on-the-fly computation of grasping poses is time-consuming, making real-time performance difficult to achieve. Second, a desired output needs to be computed based on the graspable poses on the moving object according to the control policy, which drives the end effector to approach the target object. In at least one implementation, for grasping planning, we use machine learning to compute the set of grasping poses. In at least one implementation, the machine learning model is based on a contact grasping net, which uses the original depth image and object mask as input to generate a set of 6-DoF grasping proposals. This approach is adaptable to new objects and environments without requiring fine-tuning.
[0059] Next, we compute a set of transformations from the grasp pose to the 6D pose of the target object and store them in the grasp memory pool. All proposed grasp poses are relative to the pose of the target object, which helps avoid computationally expensive real-time grasp synthesis. In this invention, the system dynamically selects which grasp pose to track from the grasp memory pool at a given time as both the EoH and the object move. In at least one embodiment, the selection of the grasp pose is primarily driven by angular distance. We use quaternion representations to store orientation information. In each control iteration, the orientation of the gripper is determined via the following formula: Orientation of all grasping poses Comparison .
[0060] With minimum The capture pose is set to the current target pose to be tracked.
[0061] Coupling control of EoH system
[0062] Given an active pose tracking and grasping plan for adjusting target grasping of a moving object, the EoH system must then track and follow the target and successfully perform dynamic grasping. The strategy must collectively satisfy two constraints to control the robot's approach to and grasping of the target object: keeping the object within the camera's FoV and avoiding collisions, including self-collisions.
[0063] In at least one implementation, we developed a standard PD controller for the control strategy, driven by an inverse kinematics (IK) solver. We view this solution as a baseline approach for performing dynamic grasping using an EoH system.
[0064] Baseline control strategy
[0065] In at least one implementation, given an end-effector target as the grasping pose target to the EoH system in each frame, using IK (damped least squares), we develop a servoing method that generates the desired joint velocity commands using PD control formulas. At a higher level, based on the servo baseline, the pose of the next target, optimized by the end-effector following a manually designed trajectory, is repeatedly calculated, as shown in... Figure 4 middle.
[0066] The EoH system will hold the object "above" (with pose) the gripper. Navigate to the pose at a certain distance from the object. From this pose, the linear movement of the gripper can obtain the desired grasping pose. In the diagram, when of z Axis and object pose z This stage is achieved when the axis is aligned. To implement this design, the gripper's pose path can be determined. Calculated as position and rotation . posture The next position requires aligning the gripper in the approach direction while maintaining a fixed distance. of z (axis) movement, .
[0067] We use a radius of r This is achieved by moving on the sphere, such as... Figure 4 As shown. If and Between z The axis is well aligned (less than the threshold) Set as If the radius is greater than or equal to radians, then r decreases; otherwise, r increases but does not exceed 1 / 2 radians. (Set to 30 cm).
[0068] Training EARL
[0069] In another embodiment of this invention, we propose a hand-eye reinforcement learning (EARL) as the control policy, which directly maps the desired grasping pose and joint states to the robot's desired joint velocities and gripper movements. Training RL models with many consecutive DoFs is challenging due to the long policy time domain and sparse reward signals. In our case, the strong coupling between the vision and manipulation subsystems exacerbates the challenge.
[0070] In at least one implementation, to efficiently train EARL in simulation, we formulate the task as a single-agent Markov game. In each round, the target object (and possibly other objects) is randomly placed in the workspace. The target object can be randomly translated and rotated in the 3D workspace (gravity is disabled, allowing the object to move in mid-air). The robot's task is to grasp the target object using a specific grasping pose. The environment resets itself upon successful / failed grasping or if object tracking fails (out of view).
[0071] In at least one implementation, we utilize a PPO agent to train an EARL policy according to RL game theory. This PPO agent receives the state variables of the target object and the EoH system and determines which action to take to perform dynamic grasping. In at least one implementation, we employ simulations based on Isaac Gym physics for efficient RL and data acquisition, leveraging PPO's native support for parallel training. In at least one implementation, we perform simultaneous simulations of thousands of independent EoH systems in an Isaac Gym simulator used for model training. Figure 5 An overview of training EARL is shown. In at least one implementation, the MLP for PPO has three hidden layers of size [256, 128, 64].
[0072] For the input, the reinforcement learner requires information about the target object; in addition to the state of the robotic arm, we provide this information as an encoding of the object's pose (translation and orientation). We describe a unique pose representation for the input. Training an RL agent with this parameterization can be difficult because Euler angles have singularities and quaternions are not unique. In at least one implementation, we use four keypoints to represent the translation and orientation of the rigid body. These keypoints can be represented in many ways, and in our setting, we use the corners of a fixed-size square to describe them, where the square is chosen to coincide with the XY plane of the coordinate system of the given object's pose, with its center coinciding with the origin of the pose coordinates. We represent the input as follows:
[0073] in This is the key point of the gripper. Let... If it is the target (grabbing) key point, then It is the error that should be minimized (reached to 0) for successful capture. It refers to the change in target capture key points between the current frame and the previous frame. and These are the joint positions and velocity vectors of the robotic arm, respectively. Scaled to [0, 1] according to rotational constraints. `c` (a Boolean variable) indicates whether the gripper is closed or open. The policy actions, as outputs of the MLP, are the velocity target for each robot joint and the Boolean command for the gripper. We chose joint velocity to control the robot arm because it results in relatively smoother motion than position-based control.
[0074] Reward Adjustment
[0075] With a suitable reward mechanism featuring diverse guidance, the learning process can be significantly faster and more successful. In at least one implementation, we carefully perform reward adjustments to encourage the robot to grasp the target object quickly and stably. At a high level, this problem can be viewed as tracking and grasping. The tracking component moves the end effector closer to the desired grasping pose while maintaining the camera's field of view of the object and avoiding collisions. The grasping component needs to determine the appropriate moment to close the gripper. We consider that the motion of the target object may be unpredictable, therefore the optimal strategy for the robotic arm is to continuously track the target and grasp it at the appropriate moment. We design the reward to have multiple components. .
[0076] Intensive punishment It is the distance between the gripper pose and the grasping pose, which encourages the robot to minimize. . This is a penalty applied when the target pose is outside the field of view. This penalty helps with active pose estimation using a moving camera by keeping the object within the FoV. This is a penalty for any collisions before grasping. Robots can easily keep objects within their field of vision and avoid collisions by moving away from the target object, which is a behavior that must be avoided. Therefore, The agent is penalized when the distance between the gripper and the grasping pose exceeds a threshold. It's an assistant used to regularize arm movements and aid in learning. It consists of three parts: z-axis alignment of the gripper and grasping pose, y-axis alignment of the gripper and grasping pose, and centering of the object within the camera's field of view. The z-axis and y-axis alignment guide the robot towards the target. The closer the gripper is to the grasping pose, the higher the probability that the target object might collide with the robot or move out of the field of view, as the target object may move randomly. If the object's speed is faster than... (set to) ), then introduce Punishment to keep the robot's gripper (Set to 20 cm) Keep away from the object. The reward for disabling the gripper is when the target object has been successfully grasped. We use the contact force of two fingers to determine whether the gripper has grasped the target object in the simulation.
[0077] Curriculum design for learning
[0078] Learning dynamic capture can be challenging for EoH systems, and training such networks from scratch with many consecutive DoFs is particularly challenging. In this invention, we propose a curriculum design for efficiently training EoH systems. In at least one implementation, we use a three-stage curriculum that gradually increases the difficulty of the task and dynamically changes the reward metric for efficient learning.
[0079] In the first phase, the object is randomly placed in the workspace, and the robot's gripper is randomly sampled to a gripping pose closer to the object. The object is moved at a low speed ( The objects are moved randomly. Furthermore, the round does not terminate when the object is out of sight. These relaxed conditions significantly limit the initial search space to aid training. Once the success rate of the initial policy exceeds a threshold, in the second phase of training, the environment is reset and the task is marked as failed once the object is out of sight. The penalties increase, and they will continue to increase to reinforce the robot's obligation to avoid these unwanted scenarios. Low penalties lead to aggressive strategies, resulting in high collision rates or failure to keep objects in sight. On the other hand, high penalties discourage the robot from approaching objects. We dynamically change the penalty scale to encourage different behaviors and achieve high success rates. In the final stage, we make the object move faster (maximum...). Set as Furthermore, the robot gripper is fixed away from the target object at the beginning of each round and works throughout the workspace.
[0080] The RL training parameters are the same for the robot. The only difference is that, for Kinova, due to the robot's kinematics, and The penalty scale is set lower, and Kinova has long links between joints, making it relatively difficult to train.
[0081] After training, our strategy EARL is run in real time on new objects, and the training time is equivalent to only ~4 hours of learning experience.
[0082] Direct simulation to reality adaptation
[0083] The goal of EARL is to train RL agents in simulators directly applicable to real-world EoH systems. In at least one implementation, we develop an efficient technique with independent research value based on the observation that the simulation-to-reality gap is not strongly correlated with the control policy, and we encode visual observations into high-level representations. Instead, the gap is primarily caused by parameter differences between the simulated and real-world systems. Through this observation, we decouple simulation-to-reality gap reduction from EARL training. In at least one implementation, using only baseline PD-based controllers operating in both the simulated and real-world EoH systems, we fine-tune the necessary parameters to reduce the simulation-to-reality gap. For example, because real-world robotic arms are torque-controlled at a lower level, damping parameters must be appropriately adjusted to achieve accurate joint speed control.
[0084] Evaluate
[0085] We evaluated the proposed method in simulation (Issac Gym) and on two robotic arms (General UR-5e 6-DoF(R1) and Kinova Gen 3 7-DoF(R2)) as EoH systems. Both robots were equipped with a two-finger Robotq2F-85 gripper and an Intel RealSense L515 camera as a vision sensor. The workspace used for training was... The cubic region. We evaluated our system on a machine with a single GPU (Nvidia 3090, using 8GB of memory). Our method can handle novel objects and can reasonably track grasp poses for them, which allows us to train using only a few objects (four). We tested in both simulated and real-world scenarios, each utilizing a set of four unseen objects. Objects were selected from the YCB and HOPE datasets, as well as some random household items. Figures 6A-6CThese objects are shown. The pipeline runs at 15 FPS.
[0086] Simulation experiments and results
[0087] The primary objective of the simulation experiments was to evaluate the baseline of the EARL and control policy for a dynamic grasping problem. In the simulation experiments, similar to training, the simulator provides the object's pose, and we perform a grasping action once per trial. To evaluate the policy, we created test cases with four different motions, such as... Figures 7A-7D As shown.
[0088] Static (S): Randomly sampled objects in the workspace.
[0089] Linear (L): A random sampling line path from boundary to boundary.
[0090] Ellipse (O): An elliptical path in a randomly sampled plane, where the object stops moving at random time steps. The z-axis is also pre-sampled randomly.
[0091] Random (R): The object moves randomly for a period of time, then stops moving for five seconds and repeats.
[0092] The object moves randomly along any dimension at its maximum speed for translation and rotation. In the simulation, the target object never stops moving, even when it slows down significantly at random; it will "vibrate" around its center of mass to simulate uncertainty.
[0093] We used a success rate to evaluate our method, where the robot successfully grasps the target object within a given time (timeout). Three failure conditions were recorded: (1) collision, (2) loss of tracking of the target object, and (3) reaching the maximum round length. Since active pose estimation was not performed in the simulation experiments, the object was checked for being out of the field of view by checking that its pose center was beyond a certain threshold (in our experiments) away from the principal axis of the simulated camera. The maximum time for each trial was 35 seconds (timeout). The robot was required to approach the target object and perform a rapid grasp when appropriate.
[0094] We generated one thousand test cases in the simulation across motion types and test objects, and repeated them five times. The EARL control strategy outperformed the baseline method on both R1 and R2 of the robot system, such as... Figure 9As shown, the baseline method achieves an average success rate of 86.83%. EARL exhibits an excellent average success rate of 95.73%. EARL's main failure mode is timeout due to high penalties for collisions and out-of-view issues, which encourages the robot to move conservatively toward the target object. EARL's timeout failures are relatively low in random motion settings, likely due to training only using random object motion. Regardless, EARL adapts to other object motion types. In contrast, the performance of the baseline method significantly decreases with increasing task difficulty.
[0095] Real-world experiments and results
[0096] Real-world experiments used actual perception and employed pose tracking via an EoH system to perform grasping planning. A human operator freely moved objects within the workspace based on the task scenario. We performed four motions (S / L / O / R) from simulation experiments to analyze the performance gap between simulation and reality. The grasping pose pool was limited to one for fair comparison. We also performed two additional motion tasks to test adaptive grasping switching: random and complex. The former involved random movement of the object, forcing a grasping pose switch to achieve success. The complex task involved combinations of motion types. These comprised three challenging tasks, as shown in Figure 8, demonstrating generalization capabilities. Figures 8A-8C Three experimental tasks are shown for evaluating EARL in a challenging setting. Figure 8A The handover / external workspace task is illustrated, demonstrating that the system can operate outside of a trained workspace and perform human-machine handover. Figure 8B The task above and behind the wall is shown, demonstrating that the system can overcome the occlusion faced by typical external camera setups by tracking across barriers. Figure 8C The task is shown placed on an elevated platform box, demonstrating that the system can handle significant changes in height.
[0097] We performed real-world robot trials as an EoH system for each test subject for each method and motion type using random start and end positions (20 trials per method per motion type). We ensured that each trial was consistent across both methods. Figure 10 These results are shown. EARL outperforms the baseline method for all test cases. The average success rate is higher than the baseline of 73.75%. Note that the EARL policy is trained entirely in simulation. Comparing simulation results shows that we achieve a small simulation-to-reality gap (less than 5%) in motion (S / L / O / R) through our simulation-to-reality adaptation. Furthermore, EARL performs well on random motion tasks and complex tasks with adaptive grasping switching, achieving average success rates of 90% and 80%, respectively. Based on our observations, the pose tracking module is the bottleneck for achieving dynamic grasping without explicit modeling of motion prediction. We utilize object-level SLAM to perform active pose tracking by considering texture, geometric information, and robust features; however, RGB images are still affected by motion blur, and depth images are affected by noise reflections and inaccurate measurements, impacting performance. Additionally, object tracking can be affected by complex backgrounds and texture defects on the object itself. Nevertheless, EARL performs well in complex tasks such as human-machine handover, moving above and behind barriers, and overcoming object occlusion limitations typically faced by external camera setups.
[0098] Some embodiments of this invention provide a framework for enabling EoH systems to perform 6-DoF dynamic grasping using visually perceived inputs through active pose tracking and curriculum-trained RL methods. Our method involves sensing for actuation and is applied to complex robotic tasks requiring hand-eye control using visual feedback. We validate our contributions through extensive experiments in simulated and complex real-world tasks, achieving high success rates with previously unseen objects. Our framework is task-general, but it does not consider explicit collision modeling, and the target cannot move faster than the robot. Future work will focus on mechanisms for recovering targets in case of tracking failure and handling more cluttered environments.
[0099] The embodiments of the present invention described above can be implemented in any of a variety of ways. For example, the embodiments can be implemented using hardware, software, or a combination thereof. When implemented in software, the software code can execute on any suitable processor or set of processors (whether located on a single computer or distributed among multiple computers). Such a processor can be implemented as an integrated circuit, wherein one or more processors are within an integrated circuit assembly. However, the processor can be implemented using circuitry of any suitable format.
[0100] Furthermore, embodiments of the present invention can be implemented as methods, examples of which have been provided. Actions performed as part of a method can be ordered in any suitable manner. Therefore, embodiments in which actions are performed in a different order than those shown can be constructed, which may include performing some actions simultaneously, even if they are shown as sequential actions in the illustrative embodiments.
[0101] The use of ordinal terms such as “first” or “second” to modify a claim element in a claim does not imply any priority, order of precedence or sequence of actions of a method relative to another claim element, but is merely a label to distinguish one claim element with a specific name from another element with the same name (other than the use of ordinal terms).
[0102] Although the invention has been described by way of examples of preferred embodiments, it should be understood that various other adjustments and modifications can be made within the spirit and scope of the invention. Therefore, the appended claims are intended to cover all such variations and modifications that fall within the true spirit and scope of the invention.
Claims
1. A controller for performing dynamic grasping of a target object using a robotic arm and visual perception input, the controller comprising: A data input / output interface configured to receive state measurements of the robotic arm and the target object from sensors arranged on the robotic arm, wherein the robotic arm includes a gripper of an end effector of the robotic arm connected by joints having actuators and encoders, the gripper being configured to grasp the target object in response to robot control signals, wherein the sensors include: A vision sensor configured to continuously provide visual observations for tracking the pose of the target object in the workspace and to calculate the grasping pose; A memory configured to store the hand-eye EoH reinforcement learning (EARL) policy, a physics-based simulator, and an arm motion generation program; and A processor, connected to the memory, is configured to perform the following steps: Train the hand-eye reinforcement learning strategy; The pose of the target object moving in the workspace is tracked based on the state measurement. Calculate the set of grasping poses on the target object, and dynamically select the desired grasping pose on the target object moving in the workspace; The trained hand-eye reinforcement learning strategy is used to compute robotic arm motion commands; The robot control signal is generated based on the calculated robotic arm motion commands; and The robot control signals are transmitted to the actuators and grippers of the joints via the data input / output interface to follow the target object while keeping it in the field of view of the vision sensor and to grasp the target object in the workspace.
2. The controller according to claim 1, wherein, The vision sensor is mounted on the distal end of the robotic arm adjacent to the gripper.
3. The controller according to claim 1, wherein, The vision sensor is an airborne camera sensor, wherein the visual observation provided by the airborne camera sensor constitutes image data including information from a depth channel, a first color channel, a second color channel, and a third color channel.
4. The controller according to claim 3, wherein, The visual observation is encoded as a high-level representation or code of the target object.
5. The controller according to claim 3, wherein, The vision sensor provides the visual observation to the processor, wherein the processor calculates the spatial position of the target object as a six-dimensional (6D) pose of the target object in the workspace, the 6D pose indicating the position and orientation of the target object relative to the camera frame of the vision sensor, or the gripper, or the base frame of the robotic arm.
6. The controller according to claim 1, wherein, The policy is generalized to apply to target objects not used to train the policy, where the model or identity of the target object is unknown, and the motion of the target object is not explicitly modeled or known prior for dynamic grasping.
7. The controller according to claim 1, wherein, The dynamic crawling is performed using six degrees of freedom (DoF).
8. The controller according to claim 1, wherein, The target object moves in the workspace in a linear pattern, a circular pattern, or a random pattern that includes random motion.
9. The controller according to claim 1, wherein, The processor calculates a set of grasping poses on the target object relative to the pose of the target object based on the visual observation, and dynamically selects the desired grasping pose position on the target object based on the pose of the target object.
10. The controller according to claim 1, wherein, The processor trains a neural network to learn a control strategy for dynamic grasping, the control strategy mapping the desired grasping pose and joint states to the desired joint velocities or positions of the robotic arm for tracking the target object and gripper actions, the gripper actions including opening and / or closing the gripper to perform the desired grasping of the target object.
11. The controller according to claim 10, wherein, The robot's joint states, provided by one or more sensors, include one or more of joint position, velocity, and torque values.
12. The controller according to claim 10, wherein, The parameters of the neural network are learned using reinforcement learning to execute the control strategy for dynamic grasping.
13. The controller according to claim 10, wherein, The control strategy is learned in simulation, wherein the parallel training of the control strategy is performed in a physics-based simulator using one or more independent hand-eye systems in simultaneous simulation.
14. The controller according to claim 10, wherein, The control strategy for dynamic grasping is learned using one or more real robotic arms.
15. The controller according to claim 10, wherein, The reward function used to train the control strategy includes multiple components with diverse guidance to encourage the robotic arm to safely grasp the target object in the event of a collision while keeping the moving target object within the field of view of the vision sensor.
16. The controller according to claim 10, wherein, The control strategy training was conducted using a curriculum design that featured progressively increasing task difficulty and an adjusted reward function design.
17. The controller according to claim 1, wherein, The learned strategy operates in real time for the continuous degrees of freedom of the hand-eye system.
18. The controller according to claim 1, wherein, Control strategies trained in simulators are applicable to real-world hand-eye systems with parameter tuning.
19. A system for performing dynamic grasping of a target object using visual perception input, the system comprising: A robotic arm configured to include a gripper connected by a joint having an actuator and an encoder, and an end effector of the robotic arm, the gripper being configured to grasp the target object in response to a robot control signal; Sensors arranged on the robotic arm, wherein the sensors are configured to measure the state of the robotic arm and the target object, wherein the sensors include a vision sensor configured to continuously provide visual observations for tracking the pose of the target object in the workspace and calculating the grasping pose; and Controller, the controller includes: A data input / output interface configured to receive state measurements of the robotic arm and the target object from the sensor; A memory configured to store the hand-eye EoH reinforcement learning (EARL) policy, a physics-based simulator, and an arm motion generation program; and A processor, connected to the memory, is configured to perform the following steps: Train the hand-eye reinforcement learning strategy; Track the pose of the target object moving within the workspace; Calculate the set of grasping poses on the target object, and dynamically select the desired grasping pose on the target object moving in the workspace; The trained hand-eye reinforcement learning strategy is used to compute robotic arm motion commands; The robot control signal is generated based on the calculated robotic arm motion commands; and The robot control signals are transmitted to the actuators and grippers of the joints via the data input / output interface to follow the target object while keeping it in the field of view of the vision sensor, and to grasp the target object in the workspace.
20. The system according to claim 19, wherein, The vision sensor is mounted on the distal end of the robotic arm adjacent to the gripper.
21. The system according to claim 19, wherein, The vision sensor is an airborne camera sensor, wherein the visual observation provided by the airborne camera sensor constitutes image data including information from a depth channel, a first color channel, a second color channel, and a third color channel.
22. The system according to claim 21, wherein, The visual observation is encoded as a high-level representation or code of the target object.
23. The system according to claim 21, wherein, The vision sensor provides the visual observation to the processor, wherein the processor calculates the spatial position of the target object as a six-dimensional (6D) pose of the target object in the workspace, the 6D pose indicating the position and orientation of the target object relative to the camera frame of the vision sensor, or the gripper, or the base frame of the robotic arm.
24. The system according to claim 19, wherein, The policy is generalized to apply to target objects not used to train the policy, where the model or identity of the target object is unknown, and the motion of the target object is not explicitly modeled or known prior for dynamic grasping.
25. The system according to claim 19, wherein, The dynamic crawling is performed using six degrees of freedom (DoF).
26. The system according to claim 19, wherein, The target object moves in the workspace in a linear pattern, a circular pattern, or a random pattern that includes random motion.
27. The system according to claim 19, wherein, The processor calculates a set of grasping poses on the target object relative to the pose of the target object based on the visual observation, and dynamically selects the desired grasping pose position on the target object based on the pose of the target object.
28. The system according to claim 19, wherein, The processor trains a neural network to learn a control strategy for dynamic grasping, the control strategy mapping the desired grasping pose and joint states to the desired joint velocities or positions of the robotic arm for tracking the target object and gripper actions, the gripper actions including opening and / or closing the gripper to perform the desired grasping of the target object.
29. The system according to claim 28, wherein, The joint states of the robot, provided by one or more sensors, include one or more of joint position, velocity, and torque values.
30. The system according to claim 28, wherein, The parameters of the neural network are learned using reinforcement learning to execute the control strategy for dynamic grasping.
31. The system according to claim 28, wherein, The control strategy is learned in simulation, wherein the parallel training of the control strategy is performed in a physics-based simulator using one or more independent hand-eye systems in simultaneous simulation.
32. The system according to claim 28, wherein, The control strategy for dynamic grasping is learned using one or more real robotic arms.
33. The system according to claim 28, wherein, The reward function used to train the control strategy includes multiple components with diverse guidance to encourage the robotic arm to safely grasp the target object in the event of a collision while keeping the moving target object within the field of view of the vision sensor.
34. The system according to claim 28, wherein, The control strategy training is conducted using a curriculum design that features progressively increasing task difficulty and an adjusted reward function design.
35. The system according to claim 19, wherein, The learned strategy operates in real time for the continuous degrees of freedom of the hand-eye system.
36. The system according to claim 19, wherein, Control strategies trained in simulators are applicable to real-world hand-eye systems with parameter tuning.