Line-of-sight based robot grasping process end pose adaptive control method

Abstract

The application discloses a kind of robot grasping and putting end pose adaptive control method based on line-of-sight guidance. For the existing eye movement control can only guide the center position of manipulator gripper, it is difficult to combine object shape and size to dynamically adjust the angle of grasping and the position of placing, resulting in the problem of low success rate of grasping and placing in unstructured environment, an improved line-of-sight guidance control strategy is proposed. Under the guidance of operator line-of-sight, the gripper is moved to the vicinity of the target object, point cloud data is obtained by the end depth camera, and multiple feasible grasping poses are generated based on the grasping pose detection (GPD) algorithm. Combined with the line-of-sight attention and dynamic weight mechanism, the pose that best meets the operator's intention is selected. When placing, according to the position relationship between the bottom of the object and the gripper grasped object, the gripper center position guided by the line-of-sight is dynamically compensated to achieve accurate placement. This method significantly improves the success rate of grasping and placing and the efficiency of human-machine cooperation.

Description

Technical Field

[0001] This invention relates to the fields of robot control and human-computer interaction, and in particular to an adaptive control method for the end-effector pose of a robot grasping process based on line-of-sight guidance. Background Technology

[0002] There is a large population of people with upper limb disabilities both domestically and internationally. These individuals often lack self-care abilities and struggle to independently and efficiently perform grasping and placing tasks in daily life. Using collaborative robotic arms to replace manual operations can significantly improve their independent living abilities and quality of life. The key to enabling collaborative robotic arms to autonomously perform grasping, placing, and storing tasks lies in the effectiveness of the grasping model.

[0003] Traditional collaborative robotic arm operation relies primarily on joysticks or teach pendants; however, these methods are inefficient, difficult to operate, and place a significant cognitive load on the operator. In recent years, eye-tracking technology has been widely used in the fields of intelligent robotics and medical assistance, demonstrating a more natural and intuitive interaction method.

[0004] However, existing eye-tracking control strategies still have significant limitations. Current research generally relies on pre-set, programmed task flows, requiring the operator to gaze at a specific area according to a pre-defined logical sequence for more than a fixed time threshold to trigger the robotic arm to perform the corresponding action. These methods are essentially based on highly structured environmental settings; the system can only operate stably when the object type, position, and background are fixed. Once the object's position shifts or the environment changes, the original gaze area and system response no longer match, easily leading to recognition failure and response inefficiency, making them unsuitable for unstructured task environments in real-world scenarios.

[0005] In robot grasping models, 3D pose estimation technology using RGB-D cameras has laid the foundation for this field. The GPD (Grasp Pose Detection) algorithm is an algorithm that generates robot grasping poses by processing RGB-D camera point cloud data, and it is widely used in automated grasping tasks of irregular objects. This algorithm automatically detects grasping points, significantly improving the robot's adaptability in unstructured scenes.

[0006] However, the lack of prediction of human subjective intent means that although such grasping models based on automated strategies can perform well in complex environments, they often fail to adapt to the individual needs of users when faced with more dynamic and diverse tasks. This limitation greatly hinders the application of grasping tasks in human-machine shared control scenarios.

[0007] To solve these problems, a new grasping and placing model is needed that can not only accurately grasp objects by combining human visual intent and the geometric rationality of their movements, but also flexibly adapt to environmental changes and place them autonomously during the placement phase. Summary of the Invention

[0008] Purpose of the invention: To solve the problems mentioned above, and in view of the current grasping and releasing methods of collaborative robotic arms, this invention proposes an adaptive control method for the end-effector pose of a robot grasping process based on line-of-sight guidance.

[0009] Technical solution: The present invention provides an adaptive control method for the end-effector pose of a robot grasping process based on gaze guidance, comprising the following steps:

[0010] (1) The operator wears a binocular vision eye tracker and performs calibration, including calibration of the eye tracker pupil capture camera and chessboard calibration of the eye tracker scene camera; calculates the three-dimensional coordinates of the gaze focus in the eye tracker scene camera coordinate system and the position of the gaze focus;

[0011] (2) Perform chessboard calibration on the global camera and the robotic arm to obtain the transformation matrix between the global camera and the robotic arm's base coordinate system. R T C This allows the object pose captured by the global camera to be converted into the robot arm's base coordinate system, helping the robot arm complete subsequent grasping actions.

[0012] (3) Perform hand-eye calibration on the local camera and the robotic arm to obtain the fixed transformation matrix between the coordinate system of the local camera and the end effector gripper of the robotic arm. E T PC Then, combined with the real-time transformation matrix between the robot arm's base coordinate system and the gripper's coordinate system... R T E The point cloud of the object captured by the local camera is converted into the base coordinate system of the robotic arm to complete the autonomous grasping of the robot with line of sight guidance;

[0013] (4) The robotic arm and the binocular vision eye tracker are synchronously connected through real-time data transmission. The operator's gaze position in the robotic arm's base coordinate system is transmitted to the robotic arm's motion function in real time, enabling the robotic arm to accurately follow the user's gaze point.

[0014] (5) Construct the GD-GPD algorithm and introduce a dynamic weight adjustment mechanism based on gaze attention on the basis of GPD scoring; the operator's gaze guides the robotic arm to move to the object placement area, and the final grasping six-degree-of-freedom posture is generated through the GD-GPD algorithm to drive the end of the robotic arm to complete the grasping.

[0015] (6) After the grabbing is completed, the line of sight guidance point is optimized so that the reference point for placing the grabbed object can more accurately follow the operator's line of sight target, making it easier for the bottom surface of the object to align with the plane of the target placement area.

[0016] (7) After the line of sight guidance point is optimized, the robotic arm carries the target object to the preset placement area and calculates the height difference between the placement reference point of the grasped object and the placement plane in real time. When the height difference is less than the preset error threshold within a preset time, the gripper is opened to perform the placement operation.

[0017] Furthermore, the implementation process of step (1) is as follows:

[0018] After the pupil capture camera is calibrated, the three-dimensional coordinates of the gaze focus in the eye tracker scene camera coordinate system are calculated based on the pupil and corneal reflection positions detected by the pupil capture camera. After the scene camera is calibrated, the positional relationship between the eye tracker scene camera {S} and the QR code calibration board {B} is obtained. B T S The position of the focal point of the line of sight is calculated by the pose of the calibration board; for the QR code positioning board image, the two-dimensional image coordinates of the QR code vertices are obtained by image recognition and corner detection, and the PnP algorithm is used to solve the three-dimensional to two-dimensional point pairs to calculate the pose of the positioning board {B} relative to the global camera {C}. C T B Subsequently, according to B T S Calculate the pose of the scene camera relative to the global camera using an eye tracker. C T S And then according to R T C Calculate the pose of the eye-tracking scene camera relative to the robot's base coordinate system. R T S Then, the three-dimensional coordinates of the gaze focus are transformed from the eye tracker scene camera coordinate system to the robot base coordinate system, and the direction of the three-dimensional gaze vector is calculated.

[0019] Furthermore, the implementation process of step (3) is as follows:

[0020] The host computer obtains the data based on hand-eye calibration. E T PC By obtaining the pose of the local camera {PC} relative to the mechanical gripper {E}, and considering that the robotic arm is running in a ROS environment, subscribing to the real-time ROS-topic node of the robotic arm SDK in the ROS environment will yield the real-time transformation matrix between the robotic arm's base coordinate system and the gripper's coordinate system. R T E This transforms the point cloud coordinate data from the local camera coordinate system to the robot's base coordinate system.

[0021] Furthermore, the implementation process of step (4) is as follows:

[0022] The host computer continuously analyzes the three-dimensional coordinates of the operator's gaze focus in the eye tracker scene camera at a frequency of 50Hz and performs a series of coordinate transformations. The eye gaze coordinates after calibration and transformation are mapped to the robot's base coordinate system and transmitted to the robot's motion function in real time for calculation, so that the center point of the robotic arm's end effector can move accurately along the target path guided by the user's gaze.

[0023] Furthermore, the implementation process of step (5) is as follows:

[0024] Based on the GPD score, a dynamic weight adjustment mechanism based on gaze attention is introduced. First, the host computer analyzes the operator's real-time gaze point and calculates the distance between each candidate grasping pose and the gaze point. It adaptively calculates the decay factor based on the dispersion of the user's eye-tracking data and assigns a gaze probability weight to each grasping point using an exponential decay function. Second, the original GPD score is directly multiplied by the corresponding probability weight to obtain the final score for each candidate grasping pose, thus balancing physical plausibility and human-machine intent. Finally, during the grasping decision process, the score changes of the candidate grasping poses are continuously monitored. When the highest score at a certain moment exceeds the experimentally calibrated empirical threshold, the host computer immediately triggers the grasping operation, sending the six degrees of freedom parameters of the grasping pose with the highest current score to the robotic arm and driving the robotic arm to quickly adjust its pose, ensuring accurate and stable execution of the grasping task. Simultaneously, the final selected grasping pose and the point cloud information of the grasped object are recorded, and the lowest point of the point cloud information is selected as the reference point for placing the grasped object, serving as the benchmark point for subsequent placement stages.

[0025] Furthermore, the implementation process of step (6) is as follows:

[0026] Based on the six-degree-of-freedom grasping pose information, the fixed offset vector between the robotic arm's end effector {e} and the reference point {obp} where the grasped object is placed is calculated after the grasping is completed. During the placement phase, the host computer uses this offset vector to perform reverse compensation for the operator's gaze position. That is, when the operator's gaze is fixed on the target placement point, the user's gaze position is reversed by compensating for the offset vector. The gaze position of the reverse compensation is passed to the robotic arm motion function to realize the change of the guide point.

[0027] Furthermore, the implementation process of step (7) is as follows:

[0028] After grasping, the robotic arm is guided to move the target object to the preset placement area. The host computer reads the position information of the center point of the robotic arm's end effector in the base coordinate system in real time, and combines it with a fixed offset vector. Using a positive compensation method, the real-time height information of the reference point for placing the object is calculated based on the offset and the real-time position information of the end of the robotic arm, and compared with the height of the target placement plane. If the height is within the allowable error range of ±2cm of the placement plane for 2 seconds, the placement condition is determined to be met, the gripper release command is triggered, the robotic arm opens the gripper, and the object is placed securely in the designated area.

[0029] Furthermore, the specific calculation process of the GD-GPD algorithm is as follows:

[0030] Calculate the standard deviation of the average distance from the sampling point of the human eye's gaze position to the candidate grasp position. This reflects the degree of concentration of human eye fixation data.

[0031]

[0032] Among them, in the formula Represents the position coordinates of the i-th capture point. This represents the position coordinates of the j-th eye-tracking data point. N represents the average distance from the sampling point of all human eye gaze position to the grab candidate position, and N represents the number of human eye sampling points;

[0033] Calculate the coefficients of the probability distribution of human eye fixation. :

[0034]

[0035] When the human eye is highly focused on the data Small, As the focal length increases, the capture points near the fixation point score higher; conversely, as the focal length decreases, the weight distribution becomes more even.

[0036] Calculate the cumulative gaze probability for each grasping pose. This represents the degree of matching between the grasping pose and the user's attention:

[0037]

[0038] Cumulative gaze probability for each grasping pose Normalize:

[0039]

[0040] Where k represents the number of candidate positions selected for GPD;

[0041] Finally, the final score for each candidate grasp pose is obtained. :

[0042]

[0043] in, This represents the original score of the GPD algorithm for the i-th candidate pose.

[0044] Beneficial Effects: Compared with existing technologies, the beneficial effects of this invention are as follows: This invention integrates a host computer, a binocular vision eye tracker, and a global camera, improving the positioning accuracy of the gaze focus in the robotic arm's base coordinate system and reducing eye movement feature data errors; for grasping operations, this invention combines the existing GPD (Grasp Pose Detection) algorithm and proposes GD-GPD (Gaze-Directed Grasp Pose)... The detection algorithm uses real-time updated point cloud data as a reference. The decision-making process takes into account both the robot's autonomous perception and human intent, while ensuring the stability of the grasping operation and the correctness of the grasped object. The operator does not need to perform any operations outside of gaze, resulting in a natural, smooth, low-complexity, and highly reliable operation. For the placement operation, this invention proposes an optimized method for the gaze guidance point, changing the gaze guidance point from the end of the robotic arm to the reference point for placing the grasped object. This allows the robotic arm control system to more accurately follow the operator's gaze target, facilitating the alignment of the object's bottom surface with the target placement area and improving placement accuracy and stability. This invention achieves the effect of directly guiding the movement of the robotic arm with the gaze, providing a more convenient and user-friendly interaction method for people with disabilities. It greatly reduces the difficulty and effort required to use the system, significantly enhancing the practical significance of this type of technology. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of an online decision-making system for the autonomous grasping end effector of a robot based on gaze guidance.

[0046] Figure 2 A schematic diagram illustrating the process of guiding a robot's movement using visual cues;

[0047] Figure 3 A flowchart illustrating the autonomous grasping process of a robot based on gaze guidance;

[0048] Figure 4 This is a schematic diagram illustrating different types of objects to be grasped in this invention;

[0049] Figure 5 This diagram illustrates the optimization of robot movement guide points based on line-of-sight guidance after the grasping operation is completed. Detailed Implementation

[0050] The present invention will now be described in further detail with reference to the accompanying drawings:

[0051] like Figure 1The illustrated line-of-sight-guided robotic grasping end-effector posture adaptive control system comprises a collaborative robotic arm (with mechanical grippers), a binocular vision eye tracker, a global camera, local cameras, and a host computer. Specifically, it includes a global camera 1, local cameras 2, a robotic arm 3, mechanical grippers 4, an object to be grasped 5, a host computer 6, pupil-capturing cameras 7, scene cameras 8, and a QR code calibration board 9. During system operation, an operator wears a binocular vision eye tracker, which has one built-in scene camera and two pupil-capturing cameras. The scene camera captures the external environment from a first-person perspective, while the pupil-capturing cameras capture images of the operator's pupils and corneal reflections. The operator gazes at a target on a calibration board or calibration card at a distance of approximately half a meter to complete eye tracker calibration based on their individual pupil data. After calibration, the eye tracker converts the two-dimensional image coordinates of the user's gaze focus into three-dimensional coordinates within the scene, thus providing the three-dimensional position of the gaze focus in the scene camera coordinate system {S}. After system calibration, the eye tracker transmits the 3D coordinates of the eye focus in the scene camera to the host computer in real time via the TCP / IP protocol through the WiFi module. The global camera, an RGB-D depth camera, is fixedly installed and uses a QR code positioning plate to acquire real-time pose information of objects and the robotic arm in the scene. The image data acquired by the global camera is transmitted to the host computer via a USB Type-C data cable using the USB 3.0 protocol. The host computer uses visual recognition technology to analyze the spatial position of the QR code positioning plate, and then calculates the relative position of the scene camera of the binocular vision eye tracker to determine the operator's gaze vector direction and focus position. When the operator gazes at the target object they intend to grasp, the real-time 3D coordinates of the gaze focus are analyzed by the host computer, directly converted to the robotic arm's base coordinate system {R}, and transmitted to the robotic arm's end effector, guiding the end effector to approach the target object. The local camera (RGB-D depth camera) is installed at the robotic arm's gripper to capture real-time environmental information around the robotic arm and adjusts the coordinate transformation relationship through hand-eye calibration to ensure that the acquired point cloud data is converted to the robotic arm's base coordinate system in real time. This invention proposes a GD-GPD (Gaze-Directed Grasp Pose Detection) algorithm, which integrates operator eye-tracking gaze information with machine perception capabilities. It analyzes local scene point cloud data captured by a local camera and the operator's visual attention hotspot in real time, dynamically selecting the optimal grasping pose within a six-degree-of-freedom space (XYZ position and ABC Euler angles). Finally, the high-precision pose parameters are transmitted to the robotic arm via USB 3.0 protocol for autonomous grasping. After grasping, the user gazes at the target placement area, and the host computer analyzes the real-time gaze point position, controlling the robotic arm's end effector to move the object to the target area and place it.During this process, since the system has already obtained the precise spatial pose of the grasped object relative to the end effector through grasping pose calculation, the gaze guidance logic is optimized from "the end effector of the robotic arm tracks the gaze point" to "the reference point of the grasped object is placed to follow the operator's gaze position".

[0052] Specifically, the present invention proposes an end-effector pose adaptive control method based on line-of-sight guidance for robot grasping, the implementation process of which is as follows:

[0053] First, the eye tracker needs to be calibrated, including the calibration of the pupil-capturing camera and the checkerboard calibration of the scene camera. Before actual operation, the operator wears the eye tracker and gazes at the target on a calibration board or calibration card at a distance of about half a meter. The eye tracker's software performs relevant calculations simultaneously. During calibration, the software detects the positions of the pupil center and corneal reflection points, calculates the gaze vector using a geometric optics model, and matches these positions with the known positions of calibration points in the scene camera coordinate system to generate a calibration model. After calibration, based on the pupil and corneal reflection positions detected by the pupil-capturing camera, the software can calculate the three-dimensional coordinates of the gaze focus in the eye tracker's scene camera coordinate system. Simultaneously, the eye tracker's scene camera is calibrated using a checkerboard. More than twenty photos of the checkerboard calibration board in different positions and postures are taken using the eye tracker's scene camera, ensuring sufficient checkerboard calibration points are covered at various angles and positions. These photos are then imported into MATLAB and processed using its built-in computer vision toolbox. The first step is feature point detection: Corner points of the chessboard squares are detected in each chessboard calibration board image; these corner points are used as feature points for subsequent calculations. The second step is camera parameter estimation: Using these feature points, the camera extrinsic parameters (i.e., the rotation and translation matrices of the camera at different shooting positions and angles) are estimated using a camera calibration algorithm. The third step is position transformation matrix calculation: Based on the extrinsic parameters, the relative pose between the eye-tracking scene camera {S} and the QR code calibration board {B} is calculated at each shooting position and angle, resulting in a series of transformation matrices. A unified position transformation matrix is ​​then calculated. B T S Thus, during the actual shared control operation by the operator, the position of the line of sight focus in the base coordinate system of the robotic arm can be calculated by the pose of the calibration plate.

[0054] In the process of calculating the gaze focus, the raw data provided by the eye tracker is the three-dimensional coordinates of the gaze focus in the eye tracker's scene camera. To facilitate the control of the robotic arm's movement using eye-tracking data, a series of coordinate transformations are needed to transform the gaze focus into the robotic arm's coordinate system. To achieve this, the host computer first acquires the image of the QR code positioning board captured by the global camera. Image recognition and corner detection are used to obtain the two-dimensional image coordinates of the QR code vertices. The PnP (Perspective-n-Point) algorithm is then used to solve for the three-dimensional to two-dimensional point pairs. The PnP algorithm is a computer vision method for calculating camera pose. It establishes coordinate transformation equations through a camera projection model and solves a system of algebraic equations to determine the camera's position and pose. Specifically, the global camera captures the QR code positioning board and identifies the four corner points of the QR code. The three-dimensional coordinates of these corner points are known in the QR code positioning board's coordinate system {B}. The global camera projects these 3D coordinate points onto a 2D image plane. Using the PnP algorithm, combined with the camera's intrinsic parameters and these points on the 2D image plane, a system of equations is established and solved to calculate the pose of the QR code positioning plate {B} relative to the global camera {C}. C T B Next, based on the position transformation matrix between the eye tracker scene camera {S} and the QR code calibration board {B} obtained during the calibration process... B T S ,Will C T B The pose of the eye-tracking scene camera {S} relative to the global camera {C} is calculated through a transformation. C T S Furthermore, based on the hand-eye calibration results between the global camera {C} and the robotic arm's base coordinate system {R}, a transformation matrix is ​​used... R T C It can be further C T S The transformation yields the pose of the eye tracker scene camera {S} relative to the robot's base coordinate system {R}. R T S Ultimately, by utilizing R T S The three-dimensional coordinates of the gaze focus are transformed from the eye tracker scene camera coordinate system {S} to the robot arm base coordinate system {R}, and the direction of the three-dimensional gaze vector in the global coordinate system is obtained.

[0055] Secondly, it is necessary to perform chessboard calibration of the global camera and hand-eye calibration of the global camera and robotic arm to obtain the transformation matrix between the global camera {C} and the robotic arm's base coordinate system {R}. R T CIn this process, the chessboard calibration board is placed within the field of view of the global camera, and multiple calibration images are captured at different positions and angles. The global camera uses these images to detect the corner positions of the chessboard squares, generating the 3D pose of the chessboard in the camera coordinate system. Then, the end effector of the robotic arm holds the chessboard calibration board and moves the robotic arm to multiple different positions and postures. At each position and posture, the pose of the chessboard in the robotic arm's base coordinate system is recorded. Combining this pose data from the global camera and the robotic arm's base coordinate system, the transformation matrix from the global camera to the robotic arm's base coordinate system is calculated. R T C This matrix describes the position and orientation relationship between the global camera coordinate system and the robotic arm base coordinate system. It can transform the object pose captured by the global camera into the robotic arm base coordinate system, helping the robotic arm to successfully complete subsequent grasping and other actions.

[0056] Then, hand-eye calibration of the local camera and the robotic arm is performed to obtain the fixed transformation matrix between the local camera {PC} and the coordinate system {E} of the robotic arm's end effector gripper. E T PC During the calibration process, the chessboard calibration plate is fixedly placed within the working area of ​​the robotic arm. The end effector of the robotic arm, carrying a local camera, is controlled to move to multiple different poses, and chessboard images are acquired in each pose to obtain the 3D pose of the chessboard in the camera coordinate system. Simultaneously, the kinematic model of the robotic arm is used to record the pose of the corresponding end effector in the base coordinate system. Multiple relative motion equations are established using the classic hand-eye calibration method AX=XB, where A represents the relative motion transformation matrix of the robotic arm end effector in the base coordinate system, B represents the relative motion transformation matrix of the local camera in the chessboard coordinate system, and X represents the fixed transformation matrix of the local camera relative to the end effector gripper. E T PC The solution is obtained. E T PC Then, by subscribing to the ros-topic real-time node of the robotic arm's ROS SDK, you can obtain the real-time transformation matrix between the robotic arm's base coordinate system and the gripper's coordinate system. R T E Through chain transformation R T PC = R T E* E T PC Map local camera point clouds to the base coordinate system.

[0057] The calculation process for the pose transformation of the local camera point cloud is as follows: The local camera provides the three-dimensional coordinates of the target area point cloud in the local camera coordinate system {PC}. Through the hand-eye calibration process, a fixed transformation matrix is ​​obtained from the local camera coordinate system {PC} to the coordinate system {E} of the robotic arm's end effector gripper. E T PC Using the ROS environment, the real-time transformation matrix from the gripper coordinate system {E} to the robot's base coordinate system {R} is obtained from the real-time node data of the robotic arm. R T E The chain rule can be used to determine the real-time transformation matrix from the local camera coordinate system {PC} to the robot arm's base coordinate system {R}. Finally, using... R T PC The point cloud is transformed from the local camera coordinate system {PC} to the robot arm base coordinate system {R}.

[0058] Furthermore, by using gaze-guided movement of the robotic arm, the operator shifts their gaze to the target area. During this process, the eye tracker continuously captures the operator's eye movement data through a pupil-capturing camera. The host computer continuously analyzes the three-dimensional coordinates of the operator's gaze focus in the eye tracker's scene camera coordinate system {S} at a frequency of 50Hz, performs coordinate transformation, maps the eye gaze coordinates to the robotic arm's base coordinate system {R}, and transmits it in real time to the robot's motion function for inverse kinematics calculation, obtaining the vector velocities of each joint of the robotic arm. This achieves the effect of the robotic arm's end effector precisely moving along the target path guided by the user's gaze. Figure 2 As shown.

[0059] Next, dynamic decision-making and execution for target grasping are performed. The operator gazes at the target object, and the system determines the gaze area based on the operator's focus, synchronously guiding the robotic arm to pre-move towards this area. During this process, a local camera at the end of the robotic arm's gripper captures a point cloud image of the target area, obtaining the point cloud's position coordinates in the local camera coordinate system {PC}, and then transforming it to obtain the point cloud's position coordinates in the local robotic arm base coordinate system {R}. The host computer uses the proposed GD-GPD (Gaze-Directed Grasp Pose Detection) algorithm to generate a large number of grasping poses based on the point cloud data. It then scores each grasping pose by integrating geometric stability and operator gaze intent matching. When the score of a grasping pose exceeds an experimentally calibrated empirical threshold, a grasping operation is triggered. The host computer sends the parameters corresponding to this grasping pose to the robotic arm, driving it to quickly adjust its pose and complete the grasping operation. This process integrates human-machine collaborative intent and physical constraints, ensuring the correctness and efficiency of the grasping operation. Specifically, as follows... Figure 3 As shown.

[0060] The GD-GPD algorithm is an improvement upon the GPD algorithm. The GPD algorithm takes point cloud data from an RGBD camera as input, generating numerous grasping poses in different directions, including gripper position, orientation, and opening / closing width. It then generates a score based on the geometric rationality of each grasping pose, selecting the highest-scoring pose for execution. This invention adds a gaze guidance mechanism, integrating physical constraints and operator intent. Specifically, the system introduces a dynamic weight adjustment mechanism based on gaze attention on top of the GPD scoring. The host computer analyzes the operator's real-time gaze point and calculates the distance between each candidate grasping pose and the gaze point. It adaptively calculates a decay factor based on the dispersion of the user's eye-tracking data and assigns a gaze probability weight to each grasping point using an exponential decay function. The specific calculation process is as follows:

[0061] 1) Calculate the standard deviation of the average distance from the sampling point of the human eye's gaze position to the candidate grasping position using the following formula. This reflects the degree of concentration of human eye fixation data.

[0062]

[0063] Among them, in the formula Represents the position coordinates of the i-th capture point. This represents the position coordinates of the j-th eye-tracking data point. The distance from the sampling point of all human eye gaze position to the candidate position of grasping is represented by N, and N represents the number of human eye sampling points.

[0064] 2) Using the calculated standard deviation Use the following formula to calculate the coefficients of the human eye fixation probability distribution. , with standard deviation The relationship is inversely proportional; when the human eye is highly focused on the data, Small, As the focal length increases, the capture points near the fixation point score higher; conversely, a smaller focal length results in a more even weight distribution.

[0065]

[0066] 3) Calculate the cumulative gaze probability for each grasping pose. This represents the degree of matching between the grasping pose and the user's attention:

[0067]

[0068] 4) Cumulative fixation probability for each grasping pose Normalize:

[0069]

[0070] Where k represents the number of GPD candidate positions selected, and its value should be as large as possible within the limits of the computer's computing power.

[0071] Finally, the final score for each candidate grasp pose is obtained. :

[0072]

[0073] in, This represents the original score of the GPD algorithm for the i-th candidate pose.

[0074] To select the optimal solution from numerous grasping postures, empirical threshold calibration is required. For example... Figure 4 As shown, this invention involves organizing multiple participants to conduct eye-tracking-guided grasping experiments on various target objects, recording the grasping quality score output by the GPD algorithm and the corresponding actual grasping results in real time. The scores are divided into preset intervals (e.g., 300–400, 400–500, 500–600), and the grasping success rate within each interval is statistically analyzed. A curve showing the relationship between the score and the success rate is then fitted. Finally, the score value corresponding to a stable success rate of 95% on this curve is selected as the empirical threshold benchmark for grasping judgment. .

[0075] During the grasping decision-making process, the system continuously monitors the score changes of candidate grasping poses, and the highest score of all candidate poses in a given point cloud frame. Exceeding the experimentally calibrated empirical threshold Upon receiving the data, the host computer immediately triggers the grasping operation, sending the six-degree-of-freedom parameters of the grasping pose with the highest current score to the robotic arm and driving it to quickly adjust its pose to ensure accurate and stable execution of the grasping task. Simultaneously, during the grasping phase, the system records the point cloud information of the object being grasped and selects the lowest point as the reference point for placing the object, which will then serve as the benchmark for subsequent placement phases.

[0076] Next, optimize the line-of-sight guidance points. For example... Figure 5 As shown, the robotic arm has completed the grasping operation and is holding the object to be grasped. The system calculates the offset vector between the center point {e} of the robotic arm's end effector and the reference point {obp} where the object is placed, based on the grasping pose information. This vector is used to modify the line-of-sight guide point from the end of the robotic arm to the object placement reference point:

[0077]

[0078] in, This represents the operator's original line of sight focus in the robot arm's base coordinate system. This point is passed into the robot arm's motion function as the target position before grasping, and the corresponding joint angular velocity vector is obtained after inverse kinematics solution. The optimized line-of-sight guide point represents the reference point position where the operator expects the object to be grasped to be placed. The optimization strategy involves... Alternative As input to the robotic arm's motion function, this offset vector helps avoid deviations between the center point of the robotic arm's end effector and the target placement point. The system uses this offset vector to guide the operator's line of sight from the robotic arm's end effector to the reference point for placing the grasped object, facilitating object placement facing its target area and improving placement accuracy and stability.

[0079] Finally, the placement operation is executed. The system first guides the robotic arm to move the target object to the preset placement area. After reaching the preset area, the host computer begins to use a fixed offset vector... By combining the real-time coordinates of the center point {e} at the end of the robotic arm, the z-coordinate of the reference point {obp} for placing the object is calculated. If the z-coordinate of {obp} remains within the allowable error range of ±2cm for 2 seconds, the system determines that the placement conditions are met, triggers the gripper command, and the robotic arm opens the gripper to securely place the object in the designated position.

[0080] The above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made based on this technical solution fall within the scope of protection of the technical concept proposed in this invention and are included in the claims of this invention.

Claims

1. A method for adaptive pose control of the end effector of a robot grasping process based on gaze guidance, characterized in that, Includes the following steps: (1) The operator wears a binocular vision eye tracker and performs calibration, including calibration of the eye tracker pupil capture camera and calibration of the eye tracker scene camera chessboard. Calculate the three-dimensional coordinates of the gaze focus in the eye tracker scene camera coordinate system and the position of the gaze focus; (2) Perform chessboard calibration on the global camera and the robotic arm to obtain the transformation matrix between the global camera and the robotic arm's base coordinate system. R T C This allows the object pose captured by the global camera to be converted into the robot arm's base coordinate system, helping the robot arm complete subsequent grasping actions. (3) Perform hand-eye calibration on the local camera and the robotic arm to obtain the fixed transformation matrix between the coordinate system of the local camera and the end effector gripper of the robotic arm. E T PC Then, combined with the real-time transformation matrix between the robot arm's base coordinate system and the gripper's coordinate system... R T E The point cloud of the object captured by the local camera is converted into the base coordinate system of the robotic arm to complete the autonomous grasping of the robot with line of sight guidance; (4) The robotic arm and the binocular vision eye tracker are synchronously connected through real-time data transmission. The operator's gaze position in the robotic arm's base coordinate system is transmitted to the robotic arm's motion function in real time, enabling the robotic arm to accurately follow the user's gaze point. (5) Construct the GD-GPD algorithm and introduce a dynamic weight adjustment mechanism based on gaze attention on the basis of GPD scoring; the operator's gaze guides the robotic arm to move to the object placement area, and the final grasping six-degree-of-freedom posture is generated through the GD-GPD algorithm to drive the end of the robotic arm to complete the grasping. (6) After the grabbing is completed, the line of sight guidance point is optimized so that the reference point for placing the grabbed object can more accurately follow the operator's line of sight target, making it easier for the bottom surface of the object to align with the plane of the target placement area. (7) After the line of sight guidance point is optimized, the robotic arm carries the target object to the preset placement area and calculates the height difference between the placement reference point of the grasped object and the placement plane in real time. When the height difference is less than the preset error threshold within a preset time, the gripper is opened to perform the placement operation. The implementation process of step (5) is as follows: Based on the GPD score, a dynamic weight adjustment mechanism based on gaze attention is introduced. First, the host computer analyzes the operator's real-time gaze point and calculates the distance between each candidate grasping pose and the gaze point. It adaptively calculates the decay factor based on the dispersion of the user's eye-tracking data and assigns a gaze probability weight to each grasping point using an exponential decay function. Second, the original GPD score is directly multiplied by the corresponding probability weight to obtain the final score for each candidate grasping pose, thus balancing physical plausibility and human-machine intent. Finally, during the grasping decision process, the score changes of the candidate grasping poses are continuously monitored. When the highest score at a certain moment exceeds the experimentally calibrated empirical threshold, the host computer immediately triggers the grasping operation, sending the six degrees of freedom parameters of the grasping pose with the highest current score to the robotic arm and driving the robotic arm to quickly adjust its pose, ensuring accurate and stable execution of the grasping task. Simultaneously, the final selected grasping pose and the point cloud information of the grasped object are recorded, and the lowest point of the point cloud information is selected as the reference point for placing the grasped object, serving as the benchmark point for subsequent placement stages.

2. The end-effector pose adaptive control method for robot grasping process based on gaze guidance according to claim 1, characterized in that, The implementation process of step (1) is as follows: After the pupil capture camera is calibrated, the three-dimensional coordinates of the gaze focus in the eye tracker scene camera coordinate system are calculated based on the pupil and corneal reflection positions detected by the pupil capture camera. After the scene camera is calibrated, the positional relationship between the eye tracker scene camera {S} and the QR code calibration board {B} is obtained. B T S The position of the focal point of the line of sight is calculated by the pose of the calibration board; for the QR code positioning board image, the two-dimensional image coordinates of the QR code vertices are obtained by image recognition and corner detection, and the PnP algorithm is used to solve the three-dimensional to two-dimensional point pairs to calculate the pose of the positioning board {B} relative to the global camera {C}. C T B Subsequently, according to B T S Calculate the pose of the scene camera relative to the global camera using an eye tracker. C T S And then according to R T C Calculate the pose of the eye-tracking scene camera relative to the robot's base coordinate system. R T S Then, the three-dimensional coordinates of the gaze focus are transformed from the eye tracker scene camera coordinate system to the robot base coordinate system, and the direction of the three-dimensional gaze vector is calculated.

3. The end-effector pose adaptive control method for robot grasping process based on gaze guidance according to claim 1, characterized in that, The implementation process of step (3) is as follows: The host computer obtains the data based on hand-eye calibration. E T PC By obtaining the pose of the local camera {PC} relative to the mechanical gripper {E}, and considering that the robotic arm is running in a ROS environment, subscribing to the real-time ROS-topic node of the robotic arm SDK in the ROS environment will yield the real-time transformation matrix between the robotic arm's base coordinate system and the gripper's coordinate system. R T E This transforms the point cloud coordinate data from the local camera coordinate system to the robot's base coordinate system.

4. The line-of-sight guided adaptive pose control method for the end-effector of a robot grasping process according to claim 1, characterized in that, The implementation process of step (4) is as follows: The host computer continuously analyzes the three-dimensional coordinates of the operator's gaze focus in the eye tracker scene camera at a frequency of 50Hz and performs a series of coordinate transformations. The eye gaze coordinates after calibration and transformation are mapped to the robot's base coordinate system and transmitted to the robot's motion function in real time for calculation, so that the center point of the robotic arm's end effector can move accurately along the target path guided by the user's gaze.

5. The end-effector pose adaptive control method for robot grasping process based on gaze guidance according to claim 1, characterized in that, The implementation process of step (6) is as follows: Based on the six-degree-of-freedom grasping pose information, the fixed offset vector between the robotic arm's end effector {e} and the reference point {obp} where the grasped object is placed is calculated after the grasping is completed. ; During the placement preparation phase, the host computer uses this offset vector to perform reverse compensation for the operator's gaze position. That is, when the operator's gaze is fixed on the target placement point, the user's gaze position is reversed to compensate for the offset vector. The gaze position of the reverse compensation is passed to the motion function of the robotic arm, so that the gaze guidance point is changed from the center point of the robotic arm end to the reference point where the object being grasped is placed.

6. The end-effector pose adaptive control method for robot grasping process based on gaze guidance according to claim 1, characterized in that, The implementation process of step (7) is as follows: After grasping, the robotic arm is guided to move the target object to the preset placement area. The host computer reads the position information of the center point of the robotic arm's end effector in the base coordinate system in real time, and combines it with a fixed offset vector. Using a positive compensation method, the real-time height information of the reference point for placing the object is calculated based on the offset vector and the real-time position information of the end of the robotic arm, and compared with the height of the target placement plane. If the height is within the allowable error range of ±2cm of the placement plane for 2 seconds, the placement condition is determined to be met, triggering the gripper release command. The robotic arm opens the gripper and securely places the object in the designated area.

7. The end-effector pose adaptive control method for robot grasping process based on gaze guidance according to claim 1, characterized in that, The specific calculation process of the GD-GPD algorithm is as follows: Calculate the standard deviation of the average distance from the sampling point of the human eye's gaze position to the candidate grasp position. This reflects the degree of concentration of human eye fixation data. Among them, in the formula Represents the position coordinates of the i-th capture point. This represents the position coordinates of the j-th eye-tracking data point. N represents the average distance from the sampling point of all human eye gaze position to the grab candidate position, and N represents the number of human eye sampling points; Calculate the coefficients of the probability distribution of human eye fixation. : When the human eye is highly focused on the data Small, As the focal length increases, the capture points near the fixation point score higher; conversely, as the focal length decreases, the weight distribution becomes more even. Calculate the cumulative gaze probability for each grasping pose. This represents the degree of matching between the grasping pose and the user's attention: Cumulative gaze probability for each grasping pose Normalize: Where k represents the number of candidate positions selected for GPD; Finally, the final score for each candidate grasp pose is obtained. : in, This represents the original score of the GPD algorithm for the i-th candidate pose.

Images