Parameter calibration method and system of hand-eye coordination grabbing device

Abstract

The application belongs to the field of hand-eye coordination grasping, and particularly discloses a parameter calibration method and system of a hand-eye coordination grasping device, which comprises the following steps: acquiring n groups of measurement samples to represent a homogeneous transformation matrix of a first pose sensor coordinate system to a second pose sensor coordinate system and a homogeneous transformation matrix of a foreground camera coordinate system to a two-dimensional code coordinate system; taking a to-be-solved variable as a variable, constructing a target function G, taking as a homogeneous transformation matrix of the second pose sensor coordinate system to the two-dimensional code coordinate system, and taking as a homogeneous transformation matrix of the first pose sensor coordinate system to the foreground camera coordinate system; then, the G is transformed into a function about a rotation term; the minimum value of the target function G is solved by using the Newton method, and a transformation matrix of a robot hand base coordinate system to an object coordinate system is obtained. The application can fuse multiple groups of sample data to calibrate parameters of a three-dimensional eye tracker, reduce calibration errors, and improve reliability.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of hand-eye coordination grasping, and more particularly relates to a parameter calibration method and system of a hand-eye coordination grasping device. BACKGROUND

[0002] The rich eye movement of human beings can not only collect information about the surrounding environment, but also convey subjective operation intentions. In combination with the spatial position and posture of a human being, the gaze direction of the eye is judged, the object of interest of the human being and the position of the object in the three-dimensional space are determined by an eye tracker, and the information is provided to a robot to carry out corresponding operation tasks, which is a new type of intuitive and promising human-machine interaction. Especially for groups with defects in hand and foot movement functions, this technology helps them regain certain self-care ability, greatly reducing the burden of nursing personnel.

[0003] Most of the currently designed eye trackers are applied to interact with elements in a virtual reality scene, such as a head-mounted eye tracker composed of a data analysis device, an information collection frame, and other connection units mentioned in patent CN219143421U. However, for the eye tracker to analyze the gaze point of the human eye in the real world, judge the real object target of interest, and assist the mechanical arm to realize a series of operation activities, the design principle of the functional structure of the eye tracker device still needs to be established to provide basic criteria for real object design. In order to measure the pose relationship between the eye tracker and other modules and realize the function of robot-assisted operation, various types of sensors should be distributed on the eye tracker. In order to calibrate the transformation parameters between heterogeneous sensors, most studies use the method of solving a well-posed linear equation set, such as the idea of solving the hand-eye calibration problem AX=XB with only one set of samples in patent CN115272245A. However, considering the measurement noise of the sensor, this calibration method has strong accidental factors, poor reliability, and large calibration error. SUMMARY

[0004] In view of the above defects or improvement needs of the prior art, the application provides a parameter calibration method and system of a hand-eye coordination grasping device, which aims to fuse multiple sets of sample data to calibrate the parameters of a three-dimensional eye tracker, reduce the calibration error, and improve the reliability.

[0005] To achieve the above-mentioned purpose, according to the first aspect of the application, a parameter calibration method of a three-dimensional eye tracker is provided, which comprises the following steps:

[0006] The hand-eye coordination grasping device includes a head-mounted three-dimensional eye tracker and a mechanical arm hand. A first pose sensor is fixed on the head-mounted three-dimensional eye tracker, a second pose sensor is fixed on the base of the mechanical arm hand, and a two-dimensional code is attached to the object to be grasped.

[0007] Change the spatial pose of a head-mounted three-dimensional eye tracker to obtain n sets of measurement samples denotes a homogeneous transformation matrix from the first pose sensor coordinate system to the second pose sensor coordinate system in the i-th set of measurement samples, denotes a homogeneous transformation matrix from the foreground camera coordinate system to the two-dimensional code coordinate system in the i-th set of measurement samples;

[0008] Taking as the variable to be solved, the objective function G is constructed:

[0009]

[0010] wherein, is a homogeneous transformation matrix from the second pose sensor coordinate system to the two-dimensional code coordinate system, is a homogeneous transformation matrix from the first pose sensor coordinate system to the foreground camera coordinate system;

[0011] The in the objective function G is expressed by a translation term and a rotation term , and the objective function G is further transformed into a function of ;

[0012] The minimum value of the objective function G is solved by Newton's method to obtain the optimal solution of , which is converted into the corresponding , and the transformation matrix of the robot hand base coordinate system to the object coordinate system is obtained according to , and the parameter calibration of the eye tracker is completed.

[0013] As a further optimization, the objective function G is transformed into a function of , including the following steps:

[0014] The rotation term is regarded as a general constant, and the translation term is regarded as a variable. The corresponding variable when the objective function G takes the minimum value, i.e. the minimum point, is determined by the convex optimization theory; the minimum point is brought into the objective function G, so that is expressed by , so that the objective function G is transformed into a function of .

[0015] As a further optimization, when solving the objective function G by Newton's method, the variable needs to be iterated. The variable is in the k-th iteration. It is expressed as the iteration point x k , and the formula for updating the iteration point is:

[0016] x k+1 = x k + dir k · s k

[0017]

[0018] wherein dir k is the search direction, and k is the gradient vector of the function G at x k , is the Hessian matrix of the function G at x k ; s k is the search step determined according to the search direction.

[0019] As a further optimization, at each iteration point x k , a small perturbation is superimposed on the variable by Lie algebra.

[0020] As a further optimization, when determining the search step s , the following optimization function is established:

[0021]

[0022] wherein dir k (a:b) represents a column vector consisting of the elements from the a-th to the b-th in the search direction dir k , and s is the search step variable to be solved.

[0023] The above optimization problem is solved to obtain the optimal search step.

[0024] As a further optimization, when iterating the variable, the method for determining the iteration initial value of the variable is as follows:

[0025] A calibration plate is prepared, which includes a third pose sensor and a chessboard fixed together, so as to obtain a transformation matrix from the third pose sensor coordinate system to the chessboard coordinate system, and then the iteration initial value

[0026] As a further optimization, when iterating the variable, the iteration initial value of the variable is determined by a random global optimization method.

[0027] As a further optimization, when obtaining the measurement sample, the transformation matrix from the first pose sensor coordinate system to the second pose sensor coordinate system is determined based on a lighthouse laser positioning method, specifically as follows:

[0028] Two linear infrared light emitters with two rotation axes are arranged inside the lighthouse, the lighthouse performs infrared light scanning cycle on the visual field where the first pose sensor and the second pose sensor are located, then the position of the first pose sensor and the second pose sensor in the lighthouse coordinate system is determined according to the pulse signals generated after the first pose sensor and the second pose sensor receive the infrared light, and the transformation relationship of the first pose sensor coordinate system and the second pose sensor coordinate system relative to the lighthouse coordinate system is determined, and then the transformation matrix from the first pose sensor coordinate system to the second pose sensor coordinate system is obtained.

[0029] According to a second aspect of the present application, a parameter calibration system of a three-dimensional eye tracker is provided, comprising a processor configured to execute the parameter calibration method of the three-dimensional eye tracker.

[0030] According to a third aspect of the present application, a computer readable storage medium having a computer program stored thereon is provided, wherein the computer program, when executed by a processor, implements the parameter calibration method of the hand-eye coordination grasping device.

[0031] Overall, compared with the prior art, the above technical solutions conceived by the present application mainly have the following technical advantages:

[0032] 1. The calibration method of the present application is based on the constructed least squares problem, and multiple groups of easily observed sample data are fused, thereby overcoming the problem of large calibration deviation and poor reliability caused by random errors due to sensor measurement noise when using a limited group of samples to construct a well-posed linear equation system.

[0033] 2. When the Newton method is used to solve the objective function, the rotation matrix after superimposing a small perturbation variable at each iteration point is represented by using Lie algebra, thereby eliminating the constraints between the elements in the rotation matrix and converting it into an unconstrained optimization problem, and the solving accuracy is improved.

[0034] 3. Due to the non-convexity of the calibration problem, the present application innovatively proposes an iterative initial value calculation method based on a "Tracker-chessboard" calibration board to accurately solve the global minimum value of the objective function, and effectively avoids the problem of falling into a local minimum. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 The structure block diagram of the embodiment of the present application is shown in the figure;

[0036] Figure 2 The coordinate system distribution diagram corresponding to each functional module of the head-mounted three-dimensional eye tracker of the embodiment of the present application is shown in the figure;

[0037] Figure 3 The mechanical structure diagram of the head-mounted three-dimensional eye tracker case of the embodiment of the present application is shown in the figure;

[0038] Figure 4 Schematic diagram of robot-assisted operation based on human eye interaction for embodiments of the present application;

[0039] Figure 5 Principle diagram of heterogeneous sensor pose transformation parameter calibration on eye tracker for embodiments of the present application;

[0040] Figure 6 Schematic diagram of solving initial values based on “Tracker-chessboard” calibration board for embodiments of the present application;

[0041] Figure 7 Principle diagram of heterogeneous sensor pose transformation parameter calibration based on “Tracker-chessboard” calibration board for embodiments of the present application.

[0042] In all the drawings, the same reference signs are used to represent the same elements or structures, wherein: 101-right pupil camera, 102-left pupil camera, 103-right foreground camera, 104-left foreground camera, 105-inner ring, 106-rotation shaft, 107-outer ring, 108-first pose sensor, 109-adjusting and locking device, 201-second pose sensor, 202-robotic hand, 3-two-dimensional code, 4-lighthouse, 501-third pose sensor, 502-chessboard. DETAILED DESCRIPTION

[0043] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0044] The embodiments of the present application provide a parameter calibration method for a hand-eye coordinated grasping device, as shown in Figure 1 , which includes the following contents.

[0045] 1. Design principle of head-mounted three-dimensional eye tracker and hand-eye coordinated grasping device

[0046] In order to enable the eye tracker to capture the intention of the human eye to command the robotic hand to carry out corresponding operation activities, it should be equipped with three main functional modules: human eye pupil gaze direction monitoring module, foreground visual field acquisition module and eye tracker six-dimensional pose measurement module, as shown in Figure 2 .

[0047] The gaze direction of the human eye pupil is calculated by image processing. The contour of the pupil is extracted from the image, combined with auxiliary information such as reflected light spots and the characteristics of multiple refractions of light in the human eye, to determine the gaze direction of the left and right eyes of the human in the coordinate systems 10101 and 10201 of the human eye pupil gaze direction monitoring module (pupil camera) (i.e. the coordinate systems 10101 and 10201 are the right pupil camera coordinate system and the left pupil camera coordinate system, respectively). In order to avoid blocking the line of sight of the human, the coordinate systems 10101 and 10201 should be arranged below the front side of the two eyes.

[0048] After obtaining the gaze direction of the human eye, the approximate region where the two line vectors intersect is calculated by representing the two line vectors in the coordinate systems 10301 and 10401 of the foreground field of view acquisition module (foreground camera) (i.e. the coordinate systems 10301 and 10401 are the right foreground camera coordinate system and the left foreground camera coordinate system, respectively) to determine the object of interest of the human and its spatial pose that the human wants to operate. In order to ensure the openness of the foreground field of view, the coordinate systems 10301 and 10401 need to be arranged above the coordinate systems 10101 and 10201.

[0049] After obtaining the object pose represented in the coordinate systems 10301 or 10401 of the foreground field of view acquisition device, the eye tracker needs to inform the computer of the pose of the object to be operated in the world coordinate system in order to guide the end of the robotic arm to reach the specified position. Before that, the eye tracker needs a six-dimensional pose measurement module to determine its spatial position and attitude at all times, i.e. the transformation of the coordinate system 10801 (i.e. the first pose sensor coordinate system) relative to the world coordinate system. In order to balance the mass distribution of the eye tracker, the module corresponding to the coordinate system 10801 should be symmetrically distributed on both sides of the human sagittal plane, and can be selected on the back side of the human head.

[0050] In order to make the eye tracker have corresponding functions and be suitable for the diverse head shapes of users, the mechanical structure and joint configuration of the eye tracker need to follow the following specifications:

[0051] 1) The three functional modules should be fixed to the same rigid body to have a determined coordinate transformation relationship between the corresponding five coordinate systems. Otherwise, the transformation relationship between the five coordinate systems needs to be constantly calibrated in real time, which is not only complex but also lacks feasibility in actual application.

[0052] 2) The eye tracker should include a fixed band with stretchability to fit different sizes of head shapes. For this purpose, the fixed band should have a certain elastic deformation ability, and a sliding joint and a locking device are also needed to make the two ends of the fixed band have relative motion and the closed loop formed thereby maintain a constant circumference.

[0053] 3) The rigid body that the functional module is located in should be equipped with a rotational freedom in the sagittal plane to achieve accurate adjustment of the pupil monitoring module position. When wearing, the person can ensure the centration of the eye tracker by adjusting the position of the eye tracker, but the pupil monitoring module position may be too high or too low, and the person's eye part may not be included in the collected image. Therefore, the rigid body that the functional module is located in should have the ability to rotate relative to the fixed band, and the rotation axis is perpendicular to the sagittal plane of the human body.

[0054] In this embodiment, as shown in Figure 3 , the three-dimensional eye tracker functions by the right pupil camera 101 and the left pupil camera 102 to monitor the gaze direction of the human eye pupil, the right foreground camera 103 and the left foreground camera 104 are used to collect the foreground view of the eye tracker, and the first pose sensor 108 is used to measure the position and attitude of the eye tracker in the lighthouse coordinate system in real time. In terms of mechanical structure, the above five functional components are fixedly connected to the rigid outer ring 107 of the eye tracker to ensure a certain transformation relationship. In addition, this eye tracker example is equipped with an inner ring 105, which changes the size of the inner ring through the adjusting device 109 containing a gear and rack mechanism and a ratchet mechanism to adapt to different head sizes. A rotary joint 106 is arranged between the outer ring and the inner ring to implement the relative rotational motion in the sagittal plane described in the structural design principle.

[0055] As shown in Figure 4 , the mechanical arm hand 202 is used in cooperation with the three-dimensional eye tracker as a mechanical device for performing operation tasks. The arm of the mechanical arm hand 202 has seven degrees of freedom, the shoulder, elbow and wrist have three, two and two degrees of freedom respectively, and except for a single degree of freedom joint for simulating shoulder rotation, the remaining six degrees of freedom are realized by three two-degree-of-freedom bevel gear differential mechanisms. The hand of the mechanical arm hand 202 adopts an underactuated lightweight design, which only uses four motors to generate diversified human-like functional movements.

[0056] When in use, when the person's eyes are staring at the object to be operated (the bottle in the figure) Figure 4 , the pupil camera calculates the line vectors of the left and right pupil lines of sight according to the human eye image information obtained by shooting, and represents them under the foreground camera according to the coordinate transformation relationship, and calculates the approximate region where the two line vectors intersect in the foreground camera. When there are multiple two-dimensional codes with recorded three-dimensional model information of the object in the foreground camera, according to the degree of adjacency to the line of sight intersection region, it is judged which two-dimensional code belongs to the object to be grabbed. Then, the eye tracker transmits the object pose information to the mechanical arm hand through the computer, and the mechanical arm hand moves to the specified position according to the instruction and controls the opening and closing of the mechanical hand through Bluetooth communication to realize the grabbing operation of the object (the bottle in the figure).

[0057] 2. Calculation of coordinate transformation in hand-eye coordination grabbing equipment

[0058] To inform the robot hand 202 of the position of the object to be manipulated to generate the grasping motion, the homogeneous transformation from the robot hand base coordinate frame to the object coordinate frame needs to be calculated, as shown in Figure 5 i.e. the homogeneous transformation matrix from coordinate frame 20101 to the coordinate frame 301 of the checkerboard 3 According to the coordinate transformation relationship, we have:

[0059]

[0060] where the homogeneous transformation from coordinate frame 10801 to 20101 is which is calculated by:

[0061]

[0062] and are the homogeneous transformations from the lighthouse coordinate frame 401 to the coordinate frames 10801 and 20101 of the first and second pose sensors, respectively. is the homogeneous transformation from the left foreground camera coordinate frame 10401 to the coordinate frame 301 of the 2D code. These two transformations can be directly measured by the method described in 2.1 below.

[0063] Therefore, we want to find Fundamentally, we need to obtain the transformation relationship between the coordinate frame 10801 of the first pose sensor 108 and the coordinate frame 10401 of the left foreground camera 104 This transformation relationship is calibrated by the method described in 2.2 below.

[0064] 2.1, Measurement of time-varying transformation

[0065] The first pose sensor 108 and the second pose sensor 201 are both six-dimensional pose sensors tracker, which adopt lighthouse laser positioning technology, and can be regarded as infrared receivers. The second pose sensor 201 is fixed at the base coordinate system of the mechanical arm hand 202. There are two linear infrared light emitters in the lighthouse 4, which are orthogonal to the rotation axis, and can sequentially perform a round of scanning in the horizontal direction and the vertical direction, and the two are combined as one infrared light scanning cycle. Before each cycle, the lighthouse will perform a long-time global exposure on the space, and the first pose sensor 108 and the second pose sensor 201 will generate a long-time pulse signal accordingly. After that, when the horizontal or vertical infrared light scans the infrared receiver on the first pose sensor 108 and the second pose sensor 201, the receiver will generate a short-time pulse signal. According to the lag time of the pulse signal relative to the start of the scanning, combined with the rotation speed of the infrared light emitter, the position of the infrared receiver in the lighthouse coordinate system can be located, and then the three-dimensional information of the first pose sensor 108 and the second pose sensor 201 is reconstructed through the double lighthouse, and the transformation relationship of the coordinate systems 10801 and 20101 relative to the lighthouse coordinate system 401 is calculated.

[0066] The left foreground camera 104 (any foreground camera can be selected) calculates the extrinsic matrix [R t] representing the transformation relationship between the camera coordinate system 10401 and the two-dimensional code coordinate system 301 according to the image information of the two-dimensional code on the object, where R ∈ SO(3), (SO(3) is a three-dimensional special orthogonal group, which is defined as a three-dimensional vector in the real number field). Specifically, the intrinsic matrix representing the inherent parameters of the foreground camera is calibrated in advance through a checkerboard, which includes the focal length f of the camera, the scale dX and dY of a single pixel along the two axes of the pixel coordinate system, the horizontal and vertical angle θ of the photosensitive plate, and the offset information u0 and v0 of the image coordinate system relative to the pixel coordinate system. The homogeneous coordinates [u, v, 1] T of a pixel point in the camera in the pixel coordinate system and the homogeneous coordinates [x, y, z, 1] T of the corresponding three-dimensional space point in the two-dimensional code coordinate system are as follows (the superscript T represents the transpose of the matrix):

[0067]

[0068] The extrinsic matrix can be calculated through the equation, where Z is a scale factor, and the two-dimensional code coordinate system is generally set to make z = 0.

[0069] 2.2, Calibration of constant transformation between heterogeneous sensors

[0070] Construct the transformation equation:

[0071]

[0072] in, All are known quantities that have been measured. For unknown quantities, SE(3) is a three-dimensional special Euclidean group. Due to the fixed connection characteristics of the functional modules on the eye tracker, It is always a constant matrix.

[0073] To solve the two unknown homogeneous transformation matrices in the above transformation equation and By keeping the robotic arm's hand coordinate system and the QR code coordinate system stationary, and only changing the spatial pose of the eye tracker, the transformation can be achieved. and Collect n groups of samples Given i = 1, 2, ..., n, solve the system of linear equations. Theoretically, this can be calculated from three sets of samples. The solution is unique. However, considering the actual measurement noise present in the sensor, this amplifies the randomness of the calculation results, causing significant deviations and reducing reliability.

[0074] Therefore, for the obtained multiple sets of measurement samples, the following least squares criterion is established to eliminate the randomness of single-set data in the measurement, and the homogeneous transformation matrix is ​​calculated accordingly. and The optimal solution.

[0075]

[0076]

[0077] Where G is the objective function, For and G is represented by unknown parameters. and That is, the matrix when G is at its minimum value. and (In this invention, A* represents the optimal solution of variable A).

[0078] At the same time, define the operation:

[0079]

[0080]

[0081] Among them, ||·|| F Let ||·||2 be the Fibonacci norm, and ||·||2 be the 2-norm of the vector. ( (representing the positive real number field) is the weighting coefficient of the translation term.

[0082] In the least squares problem expression In this problem, because the range of values ​​for the nine unknown parameters of the rotation matrix is ​​constrained by the property of a special orthogonal group, the domain of the optimization problem is not a convex set, and therefore it may have multiple local minima. When approximate initial values ​​for iteration cannot be obtained, it is necessary to... The domain SE(3)×SE(3) (Cartesian product of two special three-dimensional Euclidean groups) is sampled to conduct stochastic global optimization. Considering the unboundedness of the translation term in SE(3), a step-by-step optimization approach is adopted to transform the domain into SO(3)×SO(3) (Cartesian product of two special three-dimensional orthogonal groups). The steps are as follows:

[0083] 1) Calculate the original optimization problem relative to the variables and The minimum point t corresponding to the minimum value. * ;

[0084] 2) t * Substitute into the original optimization problem;

[0085] 3) Change G to vectorized form and express.

[0086] Specifically, according to the defined operation method, Transform into Substituting these values ​​into the expression for the objective function G, we get:

[0087]

[0088]

[0089]

[0090] Where vec is the matrix vectorization operator, H 11 H 12 H 22 f1, f2, and c are all parameters that are generated naturally after being substituted.

[0091] Since the original optimization problem G is relative to the variable and It is expressed as a convex function on a convex set, therefore, when... and When treated as a general constant, the function The minimum point t corresponding to the minimum value. * This can be directly obtained through convex optimization theory:

[0092]

[0093] t * Substitute the known quantity In and After the two, the objective function G can be changed to use the vectorization of . Therefore, the objective function G can be written as:

[0094]

[0095]

[0096] Then, the parameter H m is eigenvalue decomposition to rewrite G(x1) as There are:

[0097]

[0098] Where ξ is a matrix composed of eigenvectors, and Λ is a diagonal matrix composed of 18 eigenvalues; if λ j is the jth eigenvalue of H m , then is its corresponding eigenvector; in addition, f m is regarded as Therefore, can be written as:

[0099]

[0100]

[0101] The converted objective function is solved by Newton method, and the steepest descent direction dir k at each iteration point x k : is:

[0102]

[0103] Where, and are Hamiltonian and Laplacian operators respectively. In order to eliminate the constraints between the elements in the rotation matrix, it is converted into an unconstrained optimization problem, and the Lie algebra representation is used to superimpose the small perturbation variables δω1 and δω2 after the rotation matrix at each iteration point x k :

[0104]

[0105]

[0106] Where I is a three-order unit matrix, and [P] represents the skew-symmetric matrix of vector P. Further, the rotation matrix at iteration point x k ​: At point G, find the function G(δω1,δω2) of G with respect to (δω1,δω2), and solve for its gradient vector and Hessian matrix:

[0107]

[0108]

[0109] At this point, the search direction dir at each iteration point can be determined. k .

[0110] Based on this, in order to determine the optimal search step size, the following optimization problem is established:

[0111]

[0112] Where, dir k (a:b) represents the search direction vector dir k Given a column vector consisting of elements from the a-th to the b-th element, the problem can ultimately be transformed into finding the roots of a multivariate polynomial, making it easy to find the optimal step size s. k This will not be elaborated upon here.

[0113] Then the formula for updating the iteration point during the solution is: x k+1 =x k +dir k ·s k .

[0114] In addition, initial values ​​need to be selected for the iteration process. Considering the non-convexity of the optimization problem, there are three methods to determine the initial values ​​for iteration, which can yield the global minimum: 1) uniform sampling method based on stochastic global optimization algorithm; 2) calculating the initial values ​​for iteration directly by solving well-posed linear equations based on sufficient and sufficient sample size; 3) obtaining the initial values ​​for iteration based on a specially manufactured "Tracker-chessboard" calibration board.

[0115] The stochastic global optimization algorithm performs uniform random sampling on SO(3)×SO(3) to obtain initial value of iteration Based on different initial values ​​for iteration, this method can eventually obtain a certain number of local minima. The sampling density is gradually increased until the following Bayesian optimal termination criterion is met. Then, the smallest value is selected from the dataset of local minima and considered as the obtained global minimum.

[0116]

[0117] In the above formula, w is the number of local minima discovered, N is the initial sample size used for calculation, and ε and δ are self-selected iteration termination parameters.

[0118] For the direct calculation of the iteration initial value based on the sufficient sample size by solving a well-posed linear equation set, it is easy to cause solving error when the sensor measurement noise is large and the local minimum value distribution is too dense.

[0119] The method for obtaining the iteration initial value based on the specially manufactured "Tracker-chessboard" calibration board is shown in Figure 6 and Figure 7 . The third pose sensor 501 and the chessboard 502 are fixed together according to the predetermined pose parameters by using a specially processed rigid resin board, so that the homogeneous transformation between the coordinate systems 50101 and 50201 corresponding to the two is a relatively accurate known quantity. And the corresponding is calculated by the equation:

[0120]

[0121] to obtain the corresponding which is used as the subsequent iteration initial value of for subsequent iteration calculation.

[0122] The optimal solution finally obtained by solving the converted target function by the Newton method is On this basis, according to the foregoing equation

[0123]

[0124] the translation term can be calculated, and the homogeneous transformation matrix between the coordinate system 10801 of the first pose sensor 108 on the eye tracker and the coordinate system 10401 of the left front camera 104 is finally calibrated.

[0125]

[0126] Based on the above parameter calibration, when performing hand-eye coordination grasping: the gaze direction of the human eye is captured by the pupil camera, the object to be grasped in the field of view of the front camera is determined, the pose of the object in the front camera coordinate system is recognized, and the pose is converted to the robot hand coordinate system through the first pose sensor coordinate system; then, based on the transformation matrix of the robot hand coordinate system to the object coordinate system, a motion instruction is sent to the robot hand to reach the object position, and the opening and closing of the robot hand is controlled through Bluetooth communication, so that the robot hand accurately grasps the target object.

[0127] Those skilled in the art will readily understand that the above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for parameter calibration of a hand-eye coordination grasping device, characterized in that, The method comprises the following steps: The hand-eye coordination grabbing device comprises a head-mounted three-dimensional eye tracker and a mechanical arm hand, a first pose sensor is fixed on the head-mounted three-dimensional eye tracker, a second pose sensor is fixed on the base of the mechanical arm hand, and a two-dimensional code is attached to the object to be grabbed; Change the spatial pose of the head-mounted three-dimensional eye tracker, obtain n measurement samples , represent the first i measurement samples from the first pose sensor coordinate system to the second pose sensor coordinate system, represent the first i measurement samples from the foreground camera coordinate system to the two-dimensional code coordinate system; With as the variable to be solved, the objective function is constructed G : wherein, is a homogeneous transformation matrix from the second pose sensor coordinate system to the two-dimensional code coordinate system, is a homogeneous transformation matrix from the first pose sensor coordinate system to the foreground camera coordinate system; The objective function G In By translation terms and rotation items This indicates that the objective function will then be... G Transformation to about The function includes the following steps: The rotation term is regarded as a general constant, and the translation term is regarded as a variable, and the objective function is determined by convex optimization theory G The variable corresponding to the minimum value, i.e. the minimum point; the minimum point is brought into the objective function G , so that By representation, the objective function G is transformed into a function about ; The minimum value of the objective function is solved by Newton method G , and the optimal solution of is obtained, which is converted into the corresponding , and then according to , the transformation matrix of the hand base coordinate system of the mechanical arm to the object coordinate system is obtained, and the parameter calibration of the eye tracker is completed. 2.The method of calibrating parameters of a hand-eye coordination grasping device according to claim 1, wherein, Solve the objective function with Newton method G , the variable needs to be iterated, the variable is k , , ) in the first iteration, which is expressed as iteration point x k The formula for updating the iteration point is: wherein is a search direction, is a function G of the gradient vector at x k is a Hessian matrix of the function G at x k is a search step, which is determined depending on the search direction.​​ 3. The method of claim 2, wherein, When performing the iterations, at each iteration point x k is superimposed by a small perturbation in terms of the Lie algebra variables , . 4.The method of calibrating parameters of a hand-eye coordination grasping device according to claim 2, wherein, When the search step is determined, the following optimization function is established: wherein, represents a search direction a column vector consisting of the first a to the first b element, and s is a search step variable to be solved. The optimal search step is obtained by solving the optimization function. 5.The method of calibrating parameters of a hand-eye coordination grasping device according to claim 2, wherein, When the variable is iterated, the variable iteration initial value is determined by the following method: A calibration board is prepared, which includes a third pose sensor and a chessboard fixed together, so as to obtain a transformation matrix of the third pose sensor coordinate system to the chessboard coordinate system, and then the initial value of iteration is calculated . 6.The method of calibrating parameters of a hand-eye coordination grasping device according to claim 2, wherein, When the variable is iterated, the variable iteration initial value is determined by the following method:

7. The method of parameter calibration of a hand-eye coordination grasping device according to any one of claims 1-6, wherein, When the measurement sample is obtained, the transformation matrix of the first pose sensor coordinate system to the second pose sensor coordinate system is determined based on the lighthouse laser positioning method, and specifically comprises the following steps: There are two linear infrared light emitters in the lighthouse, which are orthogonal to the rotation axis. The lighthouse performs infrared light scanning cycles on the field of view of the first pose sensor and the second pose sensor, and then determines the orientation of the first pose sensor and the second pose sensor in the lighthouse coordinate system according to the pulse signals generated after the first pose sensor and the second pose sensor receive the infrared light. Thus, the transformation relationship of the first pose sensor coordinate system and the second pose sensor coordinate system relative to the lighthouse coordinate system is determined, and then the transformation matrix of the first pose sensor coordinate system to the second pose sensor coordinate system is obtained.

8. A parameter calibration system for a hand-eye coordination grasping device, characterized in that, The processor is configured to execute the parameter calibration method of the hand-eye coordination grabbing device according to any one of claims 1-7.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the parameter calibration method of the hand-eye coordination grabbing device according to any one of claims 1-7.

Images