Mechanical arm hand-eye calibration method based on standard model and mark points

Abstract

The invention discloses a mechanical arm hand-eye calibration method based on a standard model and mark points, and belongs to the technical field of visual control, and the method comprises the following steps: based on a URDF standard model, determining a plurality of key points of a mechanical arm and local three-dimensional coordinates of the key points in each connecting rod coordinate system; for each moving image, identifying two-dimensional coordinates containing key points; determining global three-dimensional coordinates of the key points in a base coordinate system; calculating candidate calibration parameters according to the two-dimensional coordinates and the global three-dimensional coordinates of the key points, and fitting the candidate calibration parameters of all the moving images to obtain initial hand-eye calibration parameters; analyzing the actually measured poses of the mark points from the detection image; calculating theoretical poses of the mark points; and generating the hand-eye calibration parameters according to the actually measured poses and the theoretical poses of the mark points and the initial hand-eye calibration parameters. According to the invention, the problem that the hand-eye calibration of the robotic arm is not accurate enough in the prior art can be solved.

Description

Technical Field

[0001] This invention relates to the field of vision control technology, specifically to a hand-eye calibration method for a robotic arm based on a standard model and marker points. Background Technology

[0002] With the development of intelligent manufacturing technology, vision-guided robotic arms are increasingly widely used in automated grasping, precision assembly, and welding. Hand-eye calibration, as a crucial link between the vision system and the mechanical execution system, has the core task of accurately calculating the spatial rigid body transformation relationship between the camera coordinate system and the robotic arm base coordinate system. The accuracy and stability of the calibration results directly determine the accuracy of the robotic arm in converting visual information into motion commands, and are the prerequisite and foundation for achieving high-precision, high-reliability hand-eye coordinated operations.

[0003] However, existing hand-eye calibration technologies are often inaccurate in practical applications. Traditional methods generally rely on specific external auxiliary calibration objects, such as checkerboard patterns or dot arrays, which require cumbersome manual adjustments during calibration. This is difficult to implement in complex working environments with limited space or unattended operation, resulting in inaccurate hand-eye calibration of the robotic arm. More importantly, most existing solutions are one-time static calibrations, meaning that parameters are calculated and fixed before the start of the operation. However, as the robotic arm enters a long-term operation phase, factors such as minor deformations of the mechanical structure, joint wear, changes in ambient light, or camera micro-vibrations can cause accumulated errors in the initial calibration parameters. Due to the lack of a mechanism that can both quickly initialize without relying on complex external calibration objects and utilize feature information during operation for real-time closed-loop correction, existing technologies struggle to maintain high-precision positioning requirements during long-term continuous operation. Summary of the Invention

[0004] This invention provides a method, apparatus, electronic device, and storage medium for hand-eye calibration of a robotic arm based on a standard model and marker points, which can solve the problem of inaccurate hand-eye calibration of robotic arms in the prior art.

[0005] An embodiment of the present invention provides a hand-eye calibration method for a robotic arm based on a standard model and marker points, comprising: The process involves acquiring a sequence of motion images of the robotic arm during the calibration phase, the first joint angle value corresponding to each motion image in the sequence, detection images of the robotic arm during the operation phase, the second joint angle value corresponding to each detection image, and the URDF standard model of the robotic arm. The detection images are images acquired from marker points set on the end effector of the robotic arm. The robotic arm consists of links, a base, joints, and an end effector. Based on the URDF standard model, multiple key points of the robotic arm are determined, as well as the local three-dimensional coordinates of each key point in the corresponding link coordinate system; wherein, the link coordinate system is constructed with the center of the link where the key point is located as the origin of the coordinate system; For each motion image, identify the two-dimensional coordinates of the key points contained in the current motion image; Based on the first joint angle value of the current motion image and the URDF standard model, determine the global three-dimensional coordinates of the key points contained in the current motion image in the base coordinate system of the robotic arm; wherein, the base coordinate system is constructed with the base of the robotic arm as the origin of the coordinate system; Based on the two-dimensional coordinates of the key points contained in the current motion image and the global three-dimensional coordinates of the key points contained in the current motion image, candidate calibration parameters for the current motion image are calculated and generated. Initial hand-eye calibration parameters are generated by fitting candidate calibration parameters from each motion image. The measured pose of the marker point is parsed from the detected image; the theoretical pose of the marker point is determined based on the initial hand-eye calibration parameters, the URDF standard model, and the second joint angle value. Based on the measured pose, the theoretical pose, and the initial hand-eye calibration parameters, the final hand-eye calibration parameters are determined.

[0006] Furthermore, based on the aforementioned URDF standard model, multiple key points of the robotic arm are determined, along with the local three-dimensional coordinates of each key point in its corresponding link coordinate system, including: Based on the URDF standard model, the three-dimensional geometric mesh of each link of the robotic arm is extracted; wherein, the three-dimensional geometric mesh is composed of several triangular facets; each triangular facet contains three vertices; For each vertex, select the triangle facet that shares a common vertex with the current vertex as the adjacent triangle facet of the current vertex; calculate and generate the discrete curvature of the current vertex based on the adjacent triangle facets of the current vertex, and select vertices with discrete curvature greater than a preset geometric threshold as candidate key points. Based on the URDF standard model, the motion sensitivity of each candidate key point is calculated and generated; Candidate key points with motion sensitivity greater than a preset motion threshold are selected as key points, and the local three-dimensional coordinates of each key point in the corresponding link coordinate system are extracted from the URDF standard model.

[0007] Furthermore, based on the aforementioned URDF standard model, the motion sensitivity of each candidate keypoint is calculated and generated, including: From the URDF standard model, the parent-child topological connection relationship between the base of the robotic arm, each link and joint, and the geometric transformation parameters used to characterize the relative pose of each joint coordinate system and the corresponding link coordinate system are analyzed. Based on the parent-child topological connection relationship and the geometric transformation parameters, a sequence of homogeneous transformation matrices is constructed to characterize the coordinate transformation relationship between adjacent links; Based on the homogeneous transformation matrix sequence, the serial kinematics chain of the robotic arm is constructed; The joint angles in the tandem kinematic chain are set to preset extension posture values; For each candidate key point, a forward kinematic derivation is performed based on the cascaded kinematic chain to calculate and generate the global position vector of the current candidate key point in the coordinate system of the robot arm base. Based on the parent-child topology connection relationship, the joint located before the link where the current candidate key point is located is taken as the driving joint of the current candidate key point; Based on the homogeneous transformation matrix sequence, the rotation axis unit vector and joint center position vector of each driving joint of the current candidate key point in the coordinate system of the robot arm base are extracted. For each drive joint, calculate the difference between the global position vector of the current candidate key point and the joint center position vector of the current drive joint, and generate the lever arm vector of the current drive joint. The Jacobian matrix of the current candidate key point is calculated and generated based on the unit vector of the rotation axis of each drive joint and the lever arm vector of each drive joint. The motion sensitivity of each candidate keypoint is calculated based on the Jacobian matrix of each candidate keypoint.

[0008] Furthermore, before identifying the two-dimensional coordinates of key points contained in the current motion image for each motion image, the process also includes: Based on the URDF standard model, multi-view rendering of each key point is performed in a preset virtual simulation environment to generate virtual projection images of each key point. Feature extraction is performed on each virtual projection image to generate reference feature descriptors for each key point; A key point feature template library is constructed based on the reference feature descriptors of each key point.

[0009] Furthermore, for each motion image, the two-dimensional coordinates of key points contained in the current motion image are identified, including: For each motion image, a global feature scan is performed on the current motion image to extract several candidate feature points in the current motion image, as well as the pixel coordinates of each candidate feature point; Based on the current motion image and the pixel coordinates of each candidate feature point, calculate and generate the real-time feature descriptor for each candidate feature point. For each candidate feature point, the current candidate feature point is combined with each key point in the key point feature template library to generate candidate matching pairs for the current candidate feature point. For each candidate matching pair, the vector similarity of the current candidate matching pair is calculated based on the real-time feature descriptor of the current candidate feature point and the reference feature descriptor in the corresponding key point feature template library; based on the vector similarity of the current candidate matching pair, it is determined whether the current candidate matching pair is successfully matched. Each successfully matched candidate pair is taken as the target matching pair for the current motion image; For each target matching pair, the key points in the key point feature template library contained in the current target matching pair are taken as target key points, and the pixel coordinates of the candidate feature points contained in the current target matching pair are taken as the initial pixel coordinates of the corresponding target key points. For each target key point, obtain the grayscale distribution data of neighboring pixels within the preset range of the initial pixel coordinates of the current target key point; based on the grayscale distribution data of the neighboring pixels, calculate and generate the brightness geometric center of the current target key point based on the sub-pixel positioning algorithm; and use the brightness geometric center of the current target key point as the two-dimensional coordinates of the current target key point. Each target key point is used as a key point contained in the current motion image, and the two-dimensional coordinates of each target key point are used as the two-dimensional coordinates of the key points contained in the current motion image.

[0010] Furthermore, based on the first joint angle value of the current motion image and the URDF standard model, the global three-dimensional coordinates of the key points contained in the current motion image in the base coordinate system of the robotic arm are determined, including: Set the joint angles in the cascaded kinematic chain to the first joint angle value corresponding to the current motion image; For each key point contained in the current motion image, a forward kinematic derivation is performed based on the cascaded kinematic chain to calculate and generate the global position vector of the current key point in the coordinate system of the robot arm base. The global position vector is used as the global three-dimensional coordinates of the current key point in the coordinate system of the robot arm base.

[0011] Furthermore, based on the two-dimensional coordinates of the keypoints contained in the current moving image and the global three-dimensional coordinates of the keypoints contained in the current moving image, candidate calibration parameters for the current moving image are calculated and generated, including: Obtain the intrinsic parameter matrix of the camera used to acquire the motion image sequence; Associate the global 3D coordinates of the key points contained in the current motion image with their corresponding 2D coordinates to construct several pairs of spatially corresponding points in the current motion image; Based on the preset reprojection error calculation model and the corresponding spatial point pairs, a reprojection error function for the current motion image is constructed; wherein, the reprojection error function is used to characterize the distance deviation between the global three-dimensional coordinates in the corresponding spatial point pairs, the coordinates of the theoretical projection point formed after being projected onto the preset image plane, and the two-dimensional coordinates. Based on a nonlinear iterative optimization algorithm, with the goal of minimizing the distance deviation, the reprojection error function is solved to generate the camera pose transformation matrix; The camera pose transformation matrix is ​​determined as a candidate calibration parameter for the current moving image.

[0012] Furthermore, based on the candidate calibration parameters of each motion image, initial hand-eye calibration parameters are fitted and generated, including: Extract the rotation matrix components and translation vector components of each candidate calibration parameter from each candidate calibration parameter; Calculate the mean value of the rotation matrix components based on the rotation matrix components in each candidate calibration parameter; Calculate the average value of the translation vector components based on the translation vector components in each candidate calibration parameter; For each candidate calibration parameter, calculate the rotation matrix component of the current candidate calibration parameter and the degree of deviation of the rotation matrix component from the mean of the rotation matrix component to generate the rotation deviation of the current candidate calibration parameter; and calculate the translation vector component of the current candidate calibration parameter and the degree of deviation of the translation vector component from the mean of the translation vector component to generate the translation deviation of the current candidate calibration parameter. The effective set of calibration parameters is determined based on each rotational deviation, each translational deviation, and the preset statistical tolerance threshold. The average rotation matrix is ​​calculated and generated based on each rotation matrix component in the set of valid calibration parameters. The average translation vector is generated based on each translation vector component in the set of valid calibration parameters. Initial hand-eye calibration parameters are constructed based on the average rotation matrix and the average translation vector.

[0013] Furthermore, based on the initial hand-eye calibration parameters, the URDF standard model, and the second joint angle value, the theoretical pose of the marker point is determined, including: From the URDF standard model, the fixed installation transformation parameters of the marker point relative to the end effector of the robotic arm, as well as the serial kinematic chain structure of the robotic arm, are extracted; Based on the second joint angle value, the forward kinematic recursive calculation is performed using the serial kinematic chain structure to generate the real-time spatial pose of the end effector of the robotic arm in the coordinate system of the robotic arm base. Based on the real-time spatial pose of the end effector and the fixed installation transformation parameters, the global reference pose of the marker point in the coordinate system of the robot arm base is calculated and generated. Based on the initial hand-eye calibration parameters, a spatial coordinate transformation matrix is ​​constructed to characterize the transformation from the robotic arm base coordinate system to the camera coordinate system. Based on the spatial coordinate transformation matrix, the global reference pose of the marker point is transformed by coordinate space mapping, and the theoretical pose of the marker point in the camera coordinate system is calculated and generated.

[0014] Furthermore, based on the measured pose, the theoretical pose, and the initial hand-eye calibration parameters, the final hand-eye calibration parameters are determined, including: Based on the measured pose and the theoretical pose, calculate and generate the Euclidean distance deviation between the measured pose and the theoretical pose in the spatial position component, and the rotation angle deviation between the measured pose and the theoretical pose in the spatial attitude component. Based on the Euclidean distance deviation and the theoretical pose, a comprehensive pose residual is generated; With minimizing the combined pose residual as the optimization objective, a parameter correction objective function based on nonlinear optimization is constructed; Using the initial hand-eye calibration parameters as initial values, the parameter correction objective function is solved based on an iterative optimization algorithm to calculate and generate the calibration parameter correction amount; The initial hand-eye calibration parameters are corrected based on the calibration parameter correction amount to determine the final hand-eye calibration parameters.

[0015] Compared with the prior art, the present invention has the following beneficial effects: This invention provides a hand-eye calibration method for a robotic arm based on a standard model and marker points. The method acquires a sequence of motion images of the robotic arm during the calibration phase, the first joint angle values ​​corresponding to each motion image, detection images during the operation phase, the second joint angle values ​​corresponding to each detection image, and the URDF standard model of the robotic arm; wherein, the detection images are images acquired from marker points placed on the end effector. Based on the URDF standard model, multiple key points of the robotic arm and their local three-dimensional coordinates in each link coordinate system are determined. For each motion image, the two-dimensional coordinates of the key points contained in the current motion image are identified; and based on the first joint angle values ​​corresponding to the current motion image and the URDF standard model, the global three-dimensional coordinates of the key points in the base coordinate system are determined. Based on the two-dimensional coordinates of the key points and their global three-dimensional coordinates, corresponding candidate calibration parameters are calculated and generated, and the candidate calibration parameters of all motion images are fitted to obtain initial hand-eye calibration parameters. Based on this, the measured pose of the marker points is analyzed from the detected image; the theoretical pose of the marker points is calculated according to the initial hand-eye calibration parameters, the URDF standard model, and the second joint angle value; finally, the final hand-eye calibration parameters are obtained by optimizing the measured pose, theoretical pose, and the initial hand-eye calibration parameters.

[0016] This invention first determines the key points and their local 3D coordinates of the robotic arm by analyzing its own URDF standard model. Then, it uses motion image sequences acquired during the calibration phase and synchronized first joint angle values ​​to perform 2D-3D spatial matching to solve for initial parameters. This achieves automated initial calibration without relying on external specialized calibration objects such as checkerboard patterns, solving the problems of cumbersome manual deployment and production interruptions required by traditional methods. Furthermore, addressing the issue of decreased calibration accuracy due to mechanical wear or environmental changes during long-term operation, this invention utilizes markers installed on the end effector to acquire and analyze measured poses in real time during the operation phase. Subsequently, the measured pose is compared with the theoretical pose calculated by combining the initial calibration parameters and real-time joint angle values, and the initial parameters are corrected online using a closed-loop correction. This invention effectively overcomes the limitation of static calibration in eliminating dynamic cumulative errors without the need for any external high-precision measuring equipment, ensuring high accuracy during long-term operation. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a hand-eye calibration method for a robotic arm based on a standard model and marker points, provided by an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] like Figure 1 As shown, to address the problem of inaccurate hand-eye calibration of robotic arms in existing technologies, an embodiment of the present invention provides a hand-eye calibration method for robotic arms based on a standard model and marker points, comprising at least the following steps: Step S1: Acquire the motion image sequence of the robotic arm during the calibration phase, the first joint angle value corresponding to each motion image in the motion image sequence, the detection image of the robotic arm during the operation phase, the second joint angle value corresponding to the detection image, and the URDF standard model of the robotic arm; wherein, the detection image is an image acquired from the marker points set on the end effector of the robotic arm; the robotic arm consists of a link, a base, joints, and an end effector; Specifically, it is necessary to clarify the hardware configuration and digital description model of the robotic arm used for calibration in this embodiment. The robotic arm physically consists of a base, several links, joints connecting adjacent links, and an end effector located at the very end of the robotic arm, connected in series. The base serves as the fixed reference end of the robotic arm, the links are the moving arms of the robotic arm, the joints are the driving units that drive the links to generate relative motion, and the end effector is the end tool used for grasping, welding, or, in this embodiment, carrying marker points. To achieve self-calibration without an external calibration plate, the system pre-acquires the Unified Robot Description Format (URDF) standard model of the robotic arm. This URDF standard model is an XML-based robot description file that not only contains the visual geometric mesh data of the robotic arm but also precisely defines the kinematic structure of the robotic arm, including the physical dimensions of each link, the type of each joint (such as rotary or prismatic joints), the installation position and orientation of each joint relative to its parent link, and the limit range of joint movement. By analyzing the URDF standard model, the system can construct the complete kinematic tree structure of the robotic arm, thus providing a unique geometric reference for subsequently determining the position of key points in the local coordinate system of the link.

[0020] After model loading is complete, the system enters the data acquisition process during calibration. In this stage, the robotic arm is controlled to execute a pre-designed motion trajectory covering the main range of motion in the joint space, ensuring that the camera can capture the features of each link from different angles. During the robotic arm's movement, the camera continuously acquires images at a preset frame rate, forming a motion image sequence. Simultaneously, the system synchronously records the joint encoder values ​​of each joint of the robotic arm at the moment each frame of motion image is acquired, using hardware triggering or timestamp alignment. This set of synchronously recorded joint encoder values ​​is defined as the first joint angle value. The first joint angle value is a set of vector data, with dimensions consistent with the number of degrees of freedom of the robotic arm, accurately reflecting the actual rotation angle of each joint of the robotic arm at the instant the corresponding motion image is captured. This strict time synchronization mechanism ensures a one-to-one correspondence between the two-dimensional image information acquired by the vision sensor and the three-dimensional physical state information acquired by the joint sensor in the time dimension, thereby eliminating time delay errors caused by dynamic motion.

[0021] Subsequently, to achieve online error correction, the system needs to acquire relevant data from the operational phase. The operational phase typically refers to the period after the robotic arm completes initial calibration and is put into actual work, or the phase during each restart for self-testing. In this phase, the system controls the robotic arm to move to a specific monitoring posture, ensuring that the marker point mounted on the end effector is within the camera's field of view. This marker point is an artificial identifier with significant visual characteristics, such as a QR code with a specific coding pattern, an AprilTag tag, or a high-contrast geometric shape. Its physical position is rigidly fixed to the end effector and is fixed relative to the end effector's mounting position. The system controls the camera to acquire the marker point, obtaining a detection image, and simultaneously reads the encoder values ​​of each joint of the robotic arm at this time, defining them as the second joint angle values. The detection image reflects the visual position of the marker point in the current actual physical environment, while the second joint angle values ​​are used to subsequently calculate the theoretical position of the marker point. Through the above steps, this embodiment completes the comprehensive acquisition of data from static model description to dynamic vision and kinematics, and constructs a multimodal dataset containing the robot arm's ontological features and end-effector marker features, providing accurate and complete data support for subsequent coarse calibration based on the ontological model and fine closed-loop correction based on marker points.

[0022] Step S2: Based on the URDF standard model, determine multiple key points of the robotic arm, and the local three-dimensional coordinates of each key point in the corresponding link coordinate system; wherein, the link coordinate system is constructed with the center of the link where the key point is located as the origin of the coordinate system; In a preferred embodiment, based on the URDF standard model, multiple key points of the robotic arm, and the local three-dimensional coordinates of each key point in the corresponding link coordinate system, are determined, including: Based on the URDF standard model, the three-dimensional geometric mesh of each link of the robotic arm is extracted; wherein, the three-dimensional geometric mesh is composed of several triangular facets; each triangular facet contains three vertices; For each vertex, select the triangle facet that shares a common vertex with the current vertex as the adjacent triangle facet of the current vertex; calculate and generate the discrete curvature of the current vertex based on the adjacent triangle facets of the current vertex, and select vertices with discrete curvature greater than a preset geometric threshold as candidate key points. Based on the URDF standard model, the motion sensitivity of each candidate key point is calculated and generated; Candidate key points with motion sensitivity greater than a preset motion threshold are selected as key points, and the local three-dimensional coordinates of each key point in the corresponding link coordinate system are extracted from the URDF standard model.

[0023] In a preferred embodiment, based on the URDF standard model, the motion sensitivity of each candidate keypoint is calculated and generated, including: From the URDF standard model, the parent-child topological connection relationship between the base of the robotic arm, each link and joint, and the geometric transformation parameters used to characterize the relative pose of each joint coordinate system and the corresponding link coordinate system are analyzed. Based on the parent-child topological connection relationship and the geometric transformation parameters, a sequence of homogeneous transformation matrices is constructed to characterize the coordinate transformation relationship between adjacent links; Based on the homogeneous transformation matrix sequence, the serial kinematics chain of the robotic arm is constructed; The joint angles in the tandem kinematic chain are set to preset extension posture values; For each candidate key point, a forward kinematic derivation is performed based on the cascaded kinematic chain to calculate and generate the global position vector of the current candidate key point in the coordinate system of the robot arm base. Based on the parent-child topology connection relationship, the joint located before the link where the current candidate key point is located is taken as the driving joint of the current candidate key point; Based on the homogeneous transformation matrix sequence, the rotation axis unit vector and joint center position vector of each driving joint of the current candidate key point in the coordinate system of the robot arm base are extracted. For each drive joint, calculate the difference between the global position vector of the current candidate key point and the joint center position vector of the current drive joint, and generate the lever arm vector of the current drive joint. The Jacobian matrix of the current candidate key point is calculated and generated based on the unit vector of the rotation axis of each drive joint and the lever arm vector of each drive joint. The motion sensitivity of each candidate keypoint is calculated based on the Jacobian matrix of each candidate keypoint.

[0024] Specifically, the system performs a comprehensive digital reconstruction of the robotic arm based on a pre-defined Unified Robot Description Format (URDF) standard model. The URDF standard model stores the robotic arm's geometric mesh data and kinematic parameters. Based on this, the system determines multiple key points of the robotic arm and obtains the local three-dimensional coordinates of each key point in its corresponding link coordinate system. This link coordinate system is a local reference system fixed to each link of the robotic arm. It is constructed with the geometric center of the link containing the key point or the intersection of the joint axes as its origin, ensuring that the local coordinates of the key points remain constant during the robotic arm's movement, providing a static reference for subsequent rigid body transformations.

[0025] To ensure that the selected key points are easily identifiable and accurately located in the visual image, this embodiment first filters candidate points from a geometric perspective. The system extracts the three-dimensional geometric mesh of each link of the robotic arm based on the URDF standard model. This three-dimensional geometric mesh is a digital representation of the robotic arm's outer surface, and its microstructure consists of several triangular facets, each bounded by three spatial vertices. The system traverses each vertex in the mesh and selects all triangular facets sharing the current vertex as a common vertex through topological search, defining them as adjacent triangular facets of the current vertex. Subsequently, based on the geometric relationships of the adjacent triangular facets, the system calculates the discrete curvature of the current vertex using the angle loss method. The formula for calculating the discrete curvature is as follows: In the formula, Represents the discrete curvature of the current vertex; This represents the number of adjacent triangle faces that share the current vertex as a common vertex. Indicates the first The interior angle of each adjacent triangle face at the current vertex.

[0026] The physical meaning of this formula is that if a vertex is on a flat surface, the sum of the interior angles of all its adjacent triangles should be close to 2π (i.e., 360 degrees), at which point the discrete curvature is close to zero. Conversely, if the vertex is located at an edge, tip, or hole, the sum of the interior angles will be significantly less than 2π, leading to an increase in the discrete curvature value. The system compares the calculated discrete curvature with a preset geometric threshold, and selects vertices with discrete curvature greater than the preset geometric threshold as candidate keypoints. This step effectively eliminates points located on the smooth surface of the robotic arm, while retaining feature points with rich texture and gradient information, such as corner points and edge points, thus ensuring the robustness of the visual detection algorithm.

[0027] Building upon geometric screening, to ensure that key points provide sufficient constraint during hand-eye calibration calculations and prevent equation non-convergence due to insufficient motion amplitude, the system further calculates and generates the motion sensitivity of each candidate key point based on the aforementioned URDF standard model. Specifically, the system first parses the parent-child topological connections between the robot arm's base, links, and joints, as well as the geometric transformation parameters characterizing the relative pose between each joint coordinate system and the corresponding link coordinate system, from the URDF standard model. The parent-child topological connections define the motion transmission path of the robot arm, while the geometric transformation parameters (including translation vectors and rotation quaternions or Euler angles) define the static structural dimensions of the robot arm. Based on this information, the system constructs a sequence of homogeneous transformation matrices characterizing the coordinate transformation relationship between adjacent links, and accordingly constructs the serial kinematics chain of the robot arm.

[0028] To simulate the most realistic motion, the system sets the joint angles in the cascaded kinematics chain to preset extension posture values. These extension posture values ​​are a specific set of joint angle vectors designed to position the robotic arm in a non-singular configuration with all links fully extended, maximizing the potential displacement capability of key points. For each candidate key point, the system performs forward kinematic derivation using the cascaded kinematics chain, which involves cascading and multiplying all homogeneous transformation matrices from the base to the link corresponding to the key point, calculating and generating the global position vector of the current candidate key point in the robotic arm base coordinate system.

[0029] Subsequently, based on the parent-child topological connection relationship, the system traces the kinematic chain and identifies all joints located before the link where the current candidate key point is located as the driving joints of the current candidate key point. According to the homogeneous transformation matrix sequence, the system extracts the unit vector of the rotation axis and the joint center position vector of each driving joint of the current candidate key point in the robot arm base coordinate system. For each driving joint, the system calculates the difference between the global position vector of the current candidate key point and the joint center position vector of the current driving joint, generating the lever arm vector of the current driving joint. Then, using the construction principle of the geometric Jacobian matrix, the system calculates the cross product of the unit vector of the rotation axis and the lever arm vector of each driving joint, and uses the resulting cross product vector as a column of the Jacobian matrix, thereby constructing the Jacobian matrix of the current candidate key point. The formula for calculating the column vector of the Jacobian matrix is ​​as follows: In the formula, Represents the corresponding number in the Jacobian matrix. A column vector of driving joints; Indicates the first The rotation axis unit vector of each driving joint in the base coordinate system; This represents the global position vector of the current candidate keypoint in the base coordinate system; Indicates the first The joint center position vector of each driving joint in the base coordinate system; This represents the vector cross product operation.

[0030] After obtaining the Jacobian matrix, the system calculates the motion sensitivity of each candidate keypoint based on the Jacobian matrix. Motion sensitivity is a quantitative indicator that measures the degree of displacement change of a keypoint with joint movement, and its calculation formula is as follows: In the formula, This indicates the motion sensitivity of the current candidate keypoint; The Jacobian matrix represents the current candidate keypoints; The matrix representing the transpose of the Jacobian matrix; Represents the calculation of the determinant of a matrix; This represents matrix multiplication.

[0031] This index reflects the operable ellipsoidal volume of the keypoint in Cartesian space. A larger value indicates stronger mobility of the point in all directions and greater sensitivity to changes in joint angles. Finally, the system selects candidate keypoints with motion sensitivity greater than a preset motion threshold as the final determined keypoints, and extracts the local three-dimensional coordinates of each keypoint in the corresponding link coordinate system from the URDF standard model. Through this dual geometric and kinematic screening mechanism, the present invention can automatically select feature points that are easily recognized by the vision system and can provide high signal-to-noise ratio data for the calibration algorithm, thereby significantly improving the convergence speed and final accuracy of hand-eye calibration parameter calculation.

[0032] Step S3: For each motion image, identify the two-dimensional coordinates of the key points contained in the current motion image. In a preferred embodiment, before identifying the two-dimensional coordinates of key points contained in the current motion image for each motion image, the method further includes: Based on the URDF standard model, multi-view rendering of each key point is performed in a preset virtual simulation environment to generate virtual projection images of each key point. Feature extraction is performed on each virtual projection image to generate reference feature descriptors for each key point; A key point feature template library is constructed based on the reference feature descriptors of each key point.

[0033] In a preferred embodiment, for each motion image, identifying the two-dimensional coordinates of key points contained in the current motion image includes: For each motion image, a global feature scan is performed on the current motion image to extract several candidate feature points in the current motion image, as well as the pixel coordinates of each candidate feature point; Based on the current motion image and the pixel coordinates of each candidate feature point, calculate and generate the real-time feature descriptor for each candidate feature point. For each candidate feature point, the current candidate feature point is combined with each key point in the key point feature template library to generate candidate matching pairs for the current candidate feature point. For each candidate matching pair, the vector similarity of the current candidate matching pair is calculated based on the real-time feature descriptor of the current candidate feature point and the reference feature descriptor in the corresponding key point feature template library; based on the vector similarity of the current candidate matching pair, it is determined whether the current candidate matching pair is successfully matched. Each successfully matched candidate pair is taken as the target matching pair for the current motion image; For each target matching pair, the key points in the key point feature template library contained in the current target matching pair are taken as target key points, and the pixel coordinates of the candidate feature points contained in the current target matching pair are taken as the initial pixel coordinates of the corresponding target key points. For each target key point, obtain the grayscale distribution data of neighboring pixels within the preset range of the initial pixel coordinates of the current target key point; based on the grayscale distribution data of the neighboring pixels, calculate and generate the brightness geometric center of the current target key point based on the sub-pixel positioning algorithm; and use the brightness geometric center of the current target key point as the two-dimensional coordinates of the current target key point. Each target key point is used as a key point contained in the current motion image, and the two-dimensional coordinates of each target key point are used as the two-dimensional coordinates of the key points contained in the current motion image.

[0034] Specifically, due to changes in perspective, lighting fluctuations, and background interference during the movement of the robotic arm, direct feature matching is prone to misidentification. Therefore, this embodiment employs a two-stage processing strategy of "offline learning-online matching." Before identifying the two-dimensional coordinates of key points contained in the current motion image, the system first performs an offline preprocessing step. Based on the aforementioned URDF standard model, the system constructs a digital twin model with the same geometric appearance as the real robotic arm in a preset virtual simulation environment. In the virtual environment, the virtual camera is controlled to render each key point on the model from multiple preset perspectives and distances, generating a series of virtual projection images covering different pose changes. Subsequently, the system extracts features from each virtual projection image. The extraction algorithm focuses on visual information with rotation invariance and scale invariance, such as texture gradients and edge directions within the key points and their neighborhoods, encoding them into high-dimensional feature vectors, i.e., reference feature descriptors for each key point. The system associates and stores the unique identifier of each key point with its corresponding multiple sets of reference feature descriptors, constructing a key point feature template library, thereby providing a priori standard reference for subsequent real-time matching.

[0035] In the online processing phase, for each frame of the acquired moving image, the system first performs a global feature scan. This scan uses a feature point detection algorithm to traverse the entire image, searching for regions with drastic pixel grayscale changes or significant corner features, extracting several candidate feature points, and simultaneously recording the integer pixel coordinates of each candidate feature point in the image coordinate system. Next, using each candidate feature point as the center and combining it with the local texture information of the current moving image, the system calculates and generates a real-time feature descriptor for each candidate feature point. The mathematical dimensions and structure of this real-time feature descriptor must be consistent with the reference feature descriptor in the keypoint feature template library to facilitate vector operations.

[0036] To determine the true identity of candidate feature points, the system executes a combination matching logic. For each candidate feature point, it iterates through and combines it with each keypoint in the keypoint feature template library to generate candidate matching pairs for the current candidate feature point. For each candidate matching pair, the system calculates the vector similarity between the real-time feature descriptor of the current candidate feature point and the corresponding reference feature descriptor in the keypoint feature template library. Vector similarity is typically measured using Euclidean distance.

[0037] The system determines whether a candidate matching pair is successful based on the calculated vector similarity. Typically, a similarity threshold is set; when the distance is less than this threshold, a successful match is determined, and each successfully matched candidate matching pair is taken as the target matching pair for the current motion image. For each target matching pair, the system identifies keypoints from the template library within the pair as target keypoints, thus giving the image feature points a clear physical semantic meaning (i.e., determining which part of the robotic arm they belong to). Simultaneously, the pixel coordinates of the candidate feature points in the matching pair are used as the initial pixel coordinates of the corresponding target keypoint.

[0038] Since initial pixel coordinates are typically limited by the physical resolution of the image sensor, achieving only integer-level accuracy, the system further performs sub-pixel localization to meet the requirements of precise hand-eye calibration. For each target key point, the system uses its initial pixel coordinates as the center and extracts a preset range (e.g., ...). or Within an image region (pixel window), the grayscale distribution data of neighboring pixels in that region is obtained. Based on the grayscale distribution data of the neighboring pixels, a sub-pixel localization algorithm is used to calculate and generate the luminance geometric center of the current target key point. In this embodiment, the grayscale centroid method is used to calculate the luminance geometric center, and the calculation formula is as follows: In the formula, Represents the calculated geometric center coordinates of the brightness with sub-pixel precision; and These represent the width and height of the neighboring window, respectively. and Represents the horizontal and vertical coordinates of local pixels within the neighborhood window; Indicates local coordinates The pixel grayscale value at that location.

[0039] Finally, the system will calculate the geometric center of brightness. The initial pixel coordinates are added to obtain the final two-dimensional coordinates of the current target key point. Through this strategy of coarse matching followed by fine positioning, the present invention can automatically and accurately identify key point locations from complex moving images without manual intervention, improving positioning accuracy from pixel-level to sub-pixel-level and significantly reducing the impact of quantization errors on the calibration results.

[0040] Step S4: Based on the first joint angle value of the current motion image and the URDF standard model, determine the global three-dimensional coordinates of the key points contained in the current motion image in the base coordinate system of the robotic arm; wherein, the base coordinate system is constructed with the base of the robotic arm as the origin of the coordinate system; In a preferred embodiment, based on the first joint angle value of the current motion image and the URDF standard model, the global three-dimensional coordinates of the key points contained in the current motion image in the base coordinate system of the robotic arm are determined, including: Set the joint angles in the cascaded kinematic chain to the first joint angle value corresponding to the current motion image; For each key point contained in the current motion image, a forward kinematic derivation is performed based on the cascaded kinematic chain to calculate and generate the global position vector of the current key point in the coordinate system of the robot arm base. The global position vector is used as the global three-dimensional coordinates of the current key point in the coordinate system of the robot arm base.

[0041] Specifically, since a robotic arm is a complex multi-degree-of-freedom system composed of multiple rigid links connected by joints, its key points, while stationary in the local coordinate system of their respective links, change in real time in global space with the movement of the joints. Therefore, determining the global three-dimensional coordinates of the key points contained in the current motion image in the robotic arm's base coordinate system is essentially a coordinate transformation process based on a kinematic model. The base coordinate system is a Cartesian coordinate system constructed with the fixed base of the robotic arm as the reference origin. It is the absolute reference system throughout the calibration process and does not change with the movement of the robotic arm.

[0042] To achieve this coordinate transformation, the system first needs to update the state of the kinematic model. When acquiring the current motion image, the system simultaneously records the first joint angle value. This first joint angle value is a vector data containing the rotation angle or displacement of all joints of the robotic arm (from the base to the end effector) at the current moment. The system reads this first joint angle value and sets the joint variables in the cascaded kinematic chain to the first joint angle value corresponding to the current motion image. This step is equivalent to aligning the digitized virtual robotic arm into a posture completely consistent with that of the physical robotic arm at the moment of capture. The cascaded kinematic chain is constructed based on the URDF standard model. It describes the hierarchical transformation relationship from the base to the end effector through a series of homogeneous transformation matrices. Once the joint angles are assigned values, the variable parts in these transformation matrices are instantiated into specific numerical values.

[0043] For each keypoint in the current motion image, the system performs a forward kinematics derivation based on the cascaded kinematics chain. The core of the forward kinematics derivation lies in calculating the cumulative transformation matrix from the robot arm base to the link to which the keypoint belongs. Based on the parent-child topology defined in the URDF model, the system determines the path from the base to the link to which the keypoint belongs, and sequentially right-multiplies (cascaded multiplication) the homogeneous transformation matrices between all adjacent links on this path to obtain the global pose transformation matrix of the link relative to the base. Subsequently, the system reads the local 3D coordinates of the keypoint in the coordinate system of its link, converts them to homogeneous coordinates, and left-multiplies them by the aforementioned global pose transformation matrix to calculate the global position vector of the current keypoint in the robot arm base coordinate system. The mathematical expression of this forward kinematics derivation process is as follows: In the formula, This represents the calculated global position vector of the keypoint in the robot arm's base coordinate system, typically expressed as: ; This indicates the number of joints traversed by the link from the base to the critical point; Indicates the first The joint coordinate system relative to the first joint coordinate system The homogeneous transformation matrix of the coordinate system of each joint (or link), which consists of the static geometric parameters in the URDF and the current joint angles. Joint decision; This indicates the value of the first joint angle corresponding to the first joint angle. The rotation angle of each joint; The local three-dimensional coordinates of a key point in its respective link coordinate system are represented as follows: .

[0044] Through the above calculations, the system fuses the static local data in the virtual model with the dynamic joint data collected by the sensors, and finally extracts the first three-dimensional components of the global position vector as the global three-dimensional coordinates of the current key point in the coordinate system of the robotic arm base. By implementing this step, the present invention can calculate the absolute position of each visual key point in physical space in real time and accurately, without the need for external measuring equipment (such as a laser tracker), relying solely on the kinematic feedback of the robotic arm itself. This establishes a precise correspondence between "3D spatial coordinates and 2D image coordinates," providing a solid and high-precision spatial data foundation for subsequent calculation of camera extrinsic parameters using the PnP algorithm.

[0045] Step S5: Calculate and generate candidate calibration parameters for the current motion image based on the two-dimensional coordinates of the key points contained in the current motion image and the global three-dimensional coordinates of the key points contained in the current motion image. In a preferred embodiment, candidate calibration parameters for the current motion image are calculated and generated based on the two-dimensional coordinates of key points contained in the current motion image and the global three-dimensional coordinates of key points contained in the current motion image, including: Obtain the intrinsic parameter matrix of the camera used to acquire the motion image sequence; Associate the global 3D coordinates of the key points contained in the current motion image with their corresponding 2D coordinates to construct several pairs of spatially corresponding points in the current motion image; Based on the preset reprojection error calculation model and the corresponding spatial point pairs, a reprojection error function for the current motion image is constructed; wherein, the reprojection error function is used to characterize the distance deviation between the global three-dimensional coordinates in the corresponding spatial point pairs, the coordinates of the theoretical projection point formed after being projected onto the preset image plane, and the two-dimensional coordinates. Based on a nonlinear iterative optimization algorithm, with the goal of minimizing the distance deviation, the reprojection error function is solved to generate the camera pose transformation matrix; The camera pose transformation matrix is ​​determined as a candidate calibration parameter for the current moving image.

[0046] Specifically, while the relative positions of the camera and the robot arm base (for eye-to-hand scenarios) or the camera and the end effector flange (for eye-in-hand scenarios) are theoretically fixed during the robot arm's movement, they fluctuate due to image noise during single-frame calculations. Therefore, the results calculated in this step are defined as "candidate calibration parameters." First, the system acquires the intrinsic parameter matrix of the camera used to acquire the motion image sequence. This intrinsic parameter matrix is ​​a set of parameters describing the camera's internal optical imaging characteristics. It is typically calibrated at the camera's factory or pre-obtained using the Zhang Zhengyou calibration method. It includes inherent attributes such as the camera's focal length, principal point coordinates, and distortion coefficients, used to map three-dimensional points in the camera coordinate system to a two-dimensional pixel coordinate system.

[0047] Next, the system associates the global 3D coordinates of keypoints contained in the current moving image with their corresponding 2D coordinates, constructing several pairs of spatially corresponding points in the current moving image. Since each keypoint has been assigned a unique feature identifier (ID) in the previous steps, the system uses this ID to pair the sub-pixel-level 2D coordinates identified in step S3 with the global 3D coordinates calculated in step S4, forming the dataset required to solve the Perspective-n-Point (PnP) problem. Each pair of spatially corresponding points represents the dual representation of the same physical feature point in both "world space" and "image plane".

[0048] Subsequently, based on a preset reprojection error calculation model and the corresponding spatial point pairs, the system constructs a reprojection error function for the current moving image. The reprojection error calculation model, based on the pinhole camera imaging principle, describes the mathematical process of transforming three-dimensional spatial points into the camera coordinate system through rigid body transformation (rotation and translation), and then into the image pixel plane through perspective projection. The reprojection error function characterizes the distance deviation between the global three-dimensional coordinates of the corresponding spatial point pairs, the coordinates of the theoretically projected points formed after projection onto the preset image plane, and the actually identified two-dimensional coordinates. The mathematical expression of this function is as follows: In the formula, This represents the reprojection error function, which is the sum of squared reprojection errors of all matching points; This indicates the total number of keypoints contained in the current motion image; Indicates the first The two-dimensional pixel coordinate vectors of the key points actually identified in the image, denoted as [missing information]. ; Indicates the first The global three-dimensional coordinate vectors of the key points in the coordinate system of the robotic arm base are denoted as follows: ; This represents the intrinsic parameter matrix of the camera; Let represent the rotation matrix of the camera relative to the robot arm base; Let represent the translation vector of the camera relative to the robot arm base; Represents a mapping function that includes coordinate system transformation and perspective projection; This represents the Euclidean norm (L2 norm).

[0049] After constructing the objective function, the system uses a nonlinear iterative optimization algorithm to solve for the reprojection error function, aiming to minimize the distance deviation (i.e., the reprojection error function value). Since this projection relationship involves nonlinear constraints on the rotation matrix, the Levenberg-Marquardt (LM) algorithm or the Gauss-Newton method is typically used for iterative optimization. The optimization process starts from an initial estimate and continuously adjusts the rotation matrix. Translation vector The value of the value makes the theoretical projection point gradually approach the actual observation point until the error converges or reaches the preset number of iterations, thereby generating the camera pose transformation matrix that minimizes the reprojection error.

[0050] Finally, the system determines the camera pose transformation matrix as candidate calibration parameters for the current moving image. These candidate calibration parameters include the solved rotation matrix components and translation vector components, representing the camera's spatial position and pose inferred from the current frame of image data. By implementing this step, the present invention can fully utilize the rich geometric constraint information contained in each frame of moving image, aligning abstract image features with the physical model, and laying a solid mathematical foundation for subsequent multi-frame data fusion to eliminate random errors and obtain high-precision initial calibration parameters.

[0051] Step S6: Based on the candidate calibration parameters of each motion image, fit and generate the initial hand-eye calibration parameters; In a preferred embodiment, initial hand-eye calibration parameters are fitted and generated based on candidate calibration parameters for each motion image, including: Extract the rotation matrix components and translation vector components of each candidate calibration parameter from each candidate calibration parameter; Calculate the mean value of the rotation matrix components based on the rotation matrix components in each candidate calibration parameter; Calculate the average value of the translation vector components based on the translation vector components in each candidate calibration parameter; For each candidate calibration parameter, calculate the rotation matrix component of the current candidate calibration parameter and the degree of deviation of the rotation matrix component from the mean of the rotation matrix component to generate the rotation deviation of the current candidate calibration parameter; and calculate the translation vector component of the current candidate calibration parameter and the degree of deviation of the translation vector component from the mean of the translation vector component to generate the translation deviation of the current candidate calibration parameter. The effective set of calibration parameters is determined based on each rotational deviation, each translational deviation, and the preset statistical tolerance threshold. The average rotation matrix is ​​calculated and generated based on each rotation matrix component in the set of valid calibration parameters. The average translation vector is generated based on each translation vector component in the set of valid calibration parameters. Initial hand-eye calibration parameters are constructed based on the average rotation matrix and the average translation vector.

[0052] Specifically, since the robotic arm acquires multiple frames of motion images during the calibration phase, the candidate calibration parameters calculated independently for each frame are often subject to fluctuations due to factors such as image noise, feature point extraction accuracy, and illumination variations. Therefore, the result of a single frame cannot be simply selected as the final calibration value; instead, a more robust initial solution needs to be generated through a fitting algorithm. First, the system iterates through all candidate calibration parameters corresponding to all motion images, extracting the rotation matrix component describing the attitude and the translation vector component describing the position from each candidate calibration parameter that serves as a homogeneous transformation matrix.

[0053] To establish a statistical benchmark for screening outliers, the system first performs a preliminary mean calculation on all extracted components. For translation vector components, the system uses an arithmetic mean method, summing the translation vectors in all candidate calibration parameters and dividing by the total number of samples to calculate the mean of the translation vector component. For rotation matrix components, since the rotation matrix is ​​located in a non-Euclidean space (SO(3) manifold), a simple arithmetic mean may result in a non-orthogonal matrix. Therefore, in this embodiment, a method based on quaternion averaging or Lie algebra averaging is used to calculate the mean of the rotation matrix component that can represent the geometric center of all candidate rotational attitudes. This step establishes the data distribution center of the current dataset.

[0054] Subsequently, the system enters the outlier detection phase. For each candidate calibration parameter, the system calculates its deviation from the aforementioned statistical benchmark. For the spatial position dimension, the system calculates the Euclidean distance between the translation vector component and the average value of the translation vector components of the current candidate calibration parameter, generating the translation deviation of the current candidate calibration parameter. For the spatial attitude dimension, the system calculates the angular difference between the rotation matrix component and the average value of the rotation matrix components of the current candidate calibration parameter, generating the rotation deviation of the current candidate calibration parameter. The calculation formulas for the translation deviation and rotation deviation are as follows: in, Indicates the first Translational deviation of each candidate calibration parameter; Indicates the first Translation vector components of each candidate calibration parameter; This represents the average value of the calculated translation vector components; This represents the L2 norm of a vector. Indicates the first The rotation deviation of each candidate calibration parameter is physically represented as the geodesic distance (i.e., the difference in rotation angle) between two rotation matrices. Indicates the first Rotation matrix components of each candidate calibration parameter; This represents the mean of the components of the calculated rotation matrix; Represents the trace operation of a matrix; This represents the matrix transpose operation.

[0055] After obtaining the quantified deviation data, the system determines the validity of each candidate calibration parameter based on a preset statistical tolerance threshold (e.g., a 3-standard-deviation criterion or a fixed error limit). If the rotational and translational deviations of a candidate calibration parameter are both less than the statistical tolerance threshold, it is determined to be valid data; conversely, if any deviation exceeds the threshold, the parameter is determined to be affected by significant noise or is an outlier due to mismatch, and it is removed from the dataset. Through this screening process, the system determines the set of valid calibration parameters after removing outliers.

[0056] Finally, the system performs a final fitting calculation based on this effective set of calibration parameters. The system performs weighted averaging and orthogonalization on all retained rotation matrix components in the set to generate an average rotation matrix; simultaneously, it generates an average translation vector for all retained translation vector components. The system combines this average rotation matrix and the average translation vector to construct a standard 4 × 4 homogeneous transformation matrix, thus constructing the initial hand-eye calibration parameters. By implementing this step, the present invention can automatically identify and eliminate poor calibration data caused by local illumination reflection, motion blur, or keypoint occlusion, ensuring that the initial hand-eye calibration parameters have extremely high reliability and stability, providing accurate initial values ​​for subsequent fine-tuning based on marker points.

[0057] Step S7: Parse the measured pose of the marker point from the detected image; determine the theoretical pose of the marker point based on the initial hand-eye calibration parameters, the URDF standard model, and the second joint angle value; In a preferred embodiment, determining the theoretical pose of the marker point based on the initial hand-eye calibration parameters, the URDF standard model, and the second joint angle value includes: From the URDF standard model, the fixed installation transformation parameters of the marker point relative to the end effector of the robotic arm, as well as the serial kinematic chain structure of the robotic arm, are extracted; Based on the second joint angle value, the forward kinematic recursive calculation is performed using the serial kinematic chain structure to generate the real-time spatial pose of the end effector of the robotic arm in the coordinate system of the robotic arm base. Based on the real-time spatial pose of the end effector and the fixed installation transformation parameters, the global reference pose of the marker point in the coordinate system of the robot arm base is calculated and generated. Based on the initial hand-eye calibration parameters, a spatial coordinate transformation matrix is ​​constructed to characterize the transformation from the robotic arm base coordinate system to the camera coordinate system. Based on the spatial coordinate transformation matrix, the global reference pose of the marker point is transformed by coordinate space mapping, and the theoretical pose of the marker point in the camera coordinate system is calculated and generated.

[0058] Specifically, the system needs to determine the relative positional relationship of the marker points within the kinematic chain of the robotic arm. To this end, the system parses the fixed installation transformation parameters of the marker points relative to the end effector of the robotic arm, as well as the serial kinematic chain structure of the robotic arm, from the URDF standard model. The fixed installation transformation parameters are static data describing the rigid spatial transformation relationship between the origin of the marker point's coordinate system and the origin of the end effector's (usually the flange center) coordinate system. These parameters are typically determined by mechanical design drawings and verified in URDF. <joint>Tag <origin>The properties are defined and remain constant during the operation of the robotic arm. Simultaneously, the cascade kinematic chain structure defines the link sequence from the base to the end effector and the axis of joint movement.

[0059] Next, based on the second joint angle values ​​synchronously acquired during the operation phase, the system performs forward kinematic recursive calculations using the aforementioned cascaded kinematic chain structure. The second joint angle values ​​reflect the actual physical state of each joint of the robotic arm at the instant the detection image is acquired. The system substitutes these angle values ​​into the kinematic equations and generates the real-time spatial pose of the robotic arm's end effector in the robotic arm's base coordinate system through cascaded multiplication of homogeneous transformation matrices. This real-time spatial pose represents the absolute position and orientation of the robotic arm's end flange center in global space.

[0060] After obtaining the position of the end effector, the system further calculates the global coordinates of the marker point by combining the previously analyzed rigid installation relationship. Based on the real-time spatial pose of the end effector and the fixed installation transformation parameters, the system generates the global reference pose of the marker point in the robot arm base coordinate system through matrix multiplication. This process essentially transforms the local coordinates of the marker point from the end effector coordinate system to the base coordinate system, establishing the absolute true value of the marker point in the physical world (the true value calculated based on the model). This calculation process can be expressed by the following formula: In the formula, This represents the global reference pose matrix of the generated marker points in the coordinate system of the robot arm base; This represents the input second joint angle value; This represents the pose matrix of the end effector relative to the base, calculated using forward kinematics based on the second joint angle values. This represents the fixed-mount transformation parameter matrix of the marker points obtained from the analysis relative to the end effector; This represents matrix multiplication.

[0061] Finally, to map the coordinates of the physical world to the coordinate system of the vision sensor, the system needs to access the initial calibration results. Based on the initial hand-eye calibration parameters, the system constructs a spatial coordinate transformation matrix to characterize the transformation from the robotic arm base coordinate system to the camera coordinate system. This transformation matrix includes rotation and translation components, describing the camera's mounting position relative to the robotic arm base. Based on this spatial coordinate transformation matrix, the system performs a coordinate space mapping transformation on the global reference pose of the marker point, that is, multiplying the global coordinates of the marker point on the left by the transformation matrix, thereby calculating the theoretical pose of the marker point in the camera coordinate system. The formula for calculating the theoretical pose is as follows: In the formula, This represents the theoretical pose matrix of the generated marker points in the camera coordinate system. This represents the spatial coordinate transformation matrix from the robot arm base coordinate system to the camera coordinate system, constructed based on the initial hand-eye calibration parameters. This represents the global reference pose matrix of the marker points calculated in the previous step.

[0062] By implementing this step, the system predicts the theoretical state of the marker point in the camera coordinate system based solely on sensor data (joint angles) and prior knowledge (URDF model and initial calibration parameters) without relying on image recognition results. This theoretical state will serve as the "standard answer" for subsequent comparison with the measured state, thereby enabling the system to quantify the degree of deviation of the current calibration parameters in the actual working environment.

[0063] Step S8: Determine the final hand-eye calibration parameters based on the measured pose, the theoretical pose, and the initial hand-eye calibration parameters.

[0064] In a preferred embodiment, determining the final hand-eye calibration parameters based on the measured pose, the theoretical pose, and the initial hand-eye calibration parameters includes: Based on the measured pose and the theoretical pose, calculate and generate the Euclidean distance deviation between the measured pose and the theoretical pose in the spatial position component, and the rotation angle deviation between the measured pose and the theoretical pose in the spatial attitude component. Based on the Euclidean distance deviation and the theoretical pose, a comprehensive pose residual is generated; With minimizing the combined pose residual as the optimization objective, a parameter correction objective function based on nonlinear optimization is constructed; Using the initial hand-eye calibration parameters as initial values, the parameter correction objective function is solved based on an iterative optimization algorithm to calculate and generate the calibration parameter correction amount; The initial hand-eye calibration parameters are corrected based on the calibration parameter correction amount to determine the final hand-eye calibration parameters.

[0065] Specifically, the system needs to quantify the difference between visual observation and model deduction. Based on the measured pose and the theoretical pose, the system extracts the translation vector in the spatial position component and the rotation matrix in the spatial attitude component. For the spatial position component, the system calculates the Euclidean distance between the measured and theoretical translation vectors; this distance is the Euclidean distance deviation, which directly reflects the linear error of the marker point's position in the camera's optical center coordinate system. For the spatial attitude component, the system calculates the rotation angle deviation between the measured and theoretical rotation matrices. Since the rotation matrix resides in a nonlinear Lie group space, this deviation is usually characterized by calculating the angle of relative rotation between the two rotation matrices (i.e., the rotation angle in axis-angle notation). The formulas for calculating the Euclidean distance deviation and the rotation angle deviation are as follows: In the formula, This represents the Euclidean distance deviation in the spatial position component; This represents the translation vector extracted from the measured pose; This represents the translation vector extracted from the theoretical pose; The L2 norm of a vector; This represents the rotation angle deviation on the spatial attitude component; This represents the rotation matrix extracted from the measured pose; This represents the rotation matrix extracted from the theoretical pose; Represents the trace operation of a matrix; This represents the matrix transpose operation.

[0066] After obtaining the independent deviations of position and attitude, the system needs to construct a unified evaluation index. Based on the Euclidean distance deviation and the theoretical pose (used to determine the weight distribution under different poses), the system generates a comprehensive pose residual. To address the inconsistency in the dimensions of position error (in length) and attitude error (in radians), the system typically introduces a weighting factor to fuse the two into a scalar residual value. Furthermore, the system constructs a parameter correction objective function based on nonlinear optimization, with the goal of minimizing the comprehensive pose residual. This objective function describes the mapping relationship between small perturbations in hand-eye calibration parameters and the final pose residual, and its mathematical expression is as follows: In the formula, This represents the objective function for parameter correction. This represents the correction amount for the calibration parameters to be solved, which typically includes rotation and translation corrections; The weighting coefficient representing the position error; The weighting coefficients represent the attitude error.

[0067] After constructing the objective function, the system enters the solution phase. Since the objective function exhibits non-convex and non-linear characteristics, direct analytical solution is difficult; therefore, the system employs a numerical iterative method. The system uses the initial hand-eye calibration parameters as initial values ​​(i.e., assuming the initial correction value is zero or an identity matrix) and solves the parameter correction objective function based on an iterative optimization algorithm (such as the Levenberg-Marquardt algorithm or the Gauss-Newton method). During the iteration process, the algorithm calculates the Jacobian matrix (gradient) of the objective function relative to the calibration parameters and continuously updates the parameters along the gradient descent direction until the combined pose residual converges to its minimum value. The corresponding parameter variable at this point is the calculated calibration parameter correction.

[0068] Finally, the system uses the calibration parameter correction to compensate and update the initial hand-eye calibration parameters. This process typically involves left or right matrix multiplication, superimposing the correction matrix onto the initial matrix to determine the final hand-eye calibration parameters. By implementing this step, the present invention can effectively eliminate systematic deviations introduced by errors in the robotic arm kinematic model, camera installation errors, and environmental interference, significantly improving the accuracy and robustness of the hand-eye calibration matrix and achieving high-precision vision-guided operations.

[0069] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0070] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.< / origin> < / joint>

Claims

1. A hand-eye calibration method for a robotic arm based on a standard model and marker points, characterized in that, The method comprises the following steps: acquiring a motion image sequence of a robot arm in a calibration stage, a first joint angle value corresponding to each motion image in the motion image sequence, a detection image of the robot arm in a working stage, a second joint angle value corresponding to the detection image, and a URDF standard model of the robot arm; wherein the detection image is an image collected by a marker point arranged on an end effector of the robot arm; the robot arm is composed of a connecting rod, a base, a joint, and an end effector; determining a plurality of key points of the robot arm and local three-dimensional coordinates of each key point in a corresponding connecting rod coordinate system based on the URDF standard model; wherein the connecting rod coordinate system is constructed with the center of the connecting rod where the key point is located as the coordinate system origin; for each motion image, identifying two-dimensional coordinates of the key points contained in the current motion image from the current motion image; determining global three-dimensional coordinates of the key points contained in the current motion image in a base coordinate system of the robot arm according to the first joint angle value of the current motion image and the URDF standard model; wherein the base coordinate system is constructed with the base of the robot arm as the coordinate system origin; generating candidate calibration parameters of the current motion image according to the two-dimensional coordinates of the key points contained in the current motion image and the global three-dimensional coordinates of the key points contained in the current motion image; generating initial hand-eye calibration parameters by fitting according to the candidate calibration parameters of each motion image; parsing a measured pose of the marker point from the detection image; determining a theoretical pose of the marker point according to the initial hand-eye calibration parameters, the URDF standard model, and the second joint angle value; determining final hand-eye calibration parameters according to the measured pose, the theoretical pose, and the initial hand-eye calibration parameters.

2. The standard model and marker point based hand-eye calibration method of the robot arm according to claim 1, wherein, determining a plurality of key points of the robot arm and local three-dimensional coordinates of each key point in a corresponding connecting rod coordinate system based on the URDF standard model, comprising: extracting a three-dimensional geometric mesh of each connecting rod of the robot arm according to the URDF standard model; wherein the three-dimensional geometric mesh is composed of a plurality of triangular facets; the triangular facet contains three vertices; for each vertex, selecting a triangular facet with the current vertex as a common vertex as an adjacent triangular facet of the current vertex; generating a discrete curvature of the current vertex according to the adjacent triangular facet of the current vertex, and taking a vertex with a discrete curvature greater than a preset geometric threshold as a candidate key point; calculating motion sensitivity of each candidate key point based on the URDF standard model; selecting a candidate key point with motion sensitivity greater than a preset motion threshold as a key point, and extracting local three-dimensional coordinates of each key point in a corresponding connecting rod coordinate system from the URDF standard model.

3. The standard model and marker point based hand-eye calibration method of the robot arm according to claim 2, wherein, calculating motion sensitivity of each candidate key point based on the URDF standard model, comprising: parsing a parent-child topological connection relationship between the base, each connecting rod, and each joint of the robot arm, and a geometric transformation parameter for representing the relative pose of each joint coordinate system and the corresponding connecting rod coordinate system from the URDF standard model; Based on the parent-child topology connection relationship and the geometric transformation parameter, a sequence of homogeneous transformation matrices for representing the coordinate transformation relationship between adjacent links is constructed; According to the sequence of homogeneous transformation matrices, a serial kinematic chain of the robot arm is constructed; The joint angle in the serial kinematic chain is set to a preset stretch posture value; For each candidate key point, based on the serial kinematic chain, forward kinematics is deduced to calculate a global position vector of the current candidate key point in the robot arm base coordinate system; According to the parent-child topology connection relationship, the joints located before the link where the current candidate key point is located are taken as the driving joints of the current candidate key point; According to the sequence of homogeneous transformation matrices, the rotation axis unit vector and the joint center position vector of each driving joint of the current candidate key point in the robot arm base coordinate system are extracted; For each driving joint, the difference between the global position vector of the current candidate key point and the joint center position vector of the current driving joint is calculated to generate a force arm vector of the current driving joint; According to the rotation axis unit vector of each driving joint and the force arm vector of each driving joint, a Jacobian matrix of the current candidate key point is calculated and generated; According to the Jacobian matrix of each candidate key point, the motion sensitivity of each candidate key point is calculated and generated.

4. The hand-eye calibration method of a standard model and marker point-based robot arm according to claim 3, wherein, Before identifying the two-dimensional coordinates of the key points contained in the current motion image from the current motion image for each motion image, the method further includes: Based on the URDF standard model, multi-view rendering of each key point is performed in a preset virtual simulation environment to generate a virtual projection image of each key point; Feature extraction is performed on each virtual projection image to generate a reference feature descriptor of each key point; According to the reference feature descriptor of each key point, a key point feature template library is constructed.

5. The standard model and marker point based hand-eye calibration method of the robot arm according to claim 4, wherein, For each motion image, the two-dimensional coordinates of the key points contained in the current motion image are identified from the current motion image, including: For each motion image, global feature scanning is performed on the current motion image to extract a plurality of candidate feature points in the current motion image and the pixel coordinates of each candidate feature point; According to the current motion image and the pixel coordinates of each candidate feature point, a real-time feature descriptor of each candidate feature point is calculated and generated; For each candidate feature point, the current candidate feature point is combined with each key point in the key point feature template library one by one to generate each candidate matching pair of the current candidate feature point; For each candidate matching pair, according to the real-time feature descriptor of the current candidate feature point and the reference feature descriptor in the corresponding key point feature template library, a vector similarity of the current candidate matching pair is calculated and generated; according to the vector similarity of the current candidate matching pair, it is determined whether the current candidate matching pair is matched successfully; Each candidate matching pair that is matched successfully is taken as a target matching pair of the current motion image; For each target matching pair, the key point in the key point feature template library contained in the current target matching pair is taken as a target key point, and the pixel coordinates of the candidate feature point contained in the current target matching pair are taken as the initial pixel coordinates of the corresponding target key point; For each target key point, obtain neighborhood pixel gray scale distribution data within a preset range of initial pixel coordinates of the current target key point; according to the neighborhood pixel gray scale distribution data, generate a luminance geometric center of the current target key point based on a sub-pixel positioning algorithm; and take the luminance geometric center of the current target key point as a two-dimensional coordinate of the current target key point. Take each target key point as a key point contained in the current motion image, and take the two-dimensional coordinate of each target key point as a two-dimensional coordinate of the key point contained in the current motion image.

6. The hand-eye calibration method of a standard model and marker point-based robot arm according to claim 5, wherein, According to the first joint angle value of the current motion image and the URDF standard model, determine global three-dimensional coordinates of the key point contained in the current motion image in the base coordinate system of the robot arm, including: Set the joint angles in the serial kinematic chain to the first joint angle values corresponding to the current motion image; For each key point contained in the current motion image, perform forward kinematics derivation based on the serial kinematic chain to generate a global position vector of the current key point in the base coordinate system of the robot arm; Take the global position vector as the global three-dimensional coordinates of the current key point in the base coordinate system of the robot arm.

7. The method of claim 6, wherein the method further comprises: According to the two-dimensional coordinates of the key point contained in the current motion image and the global three-dimensional coordinates of the key point contained in the current motion image, generate candidate calibration parameters of the current motion image, including: Obtain an intrinsic matrix of a camera used to collect the motion image sequence; Correlate the global three-dimensional coordinates of the key point contained in the current motion image with the corresponding two-dimensional coordinates to construct a plurality of groups of spatial corresponding point pairs of the current motion image; Based on a preset re-projection error calculation model and the spatial corresponding point pairs, construct a re-projection error function of the current motion image; wherein the re-projection error function is used to represent the distance deviation between the coordinates of the theoretical projection point formed after projecting the global three-dimensional coordinates in the spatial corresponding point pairs to a preset image plane and the two-dimensional coordinates; Based on a nonlinear iterative optimization algorithm, solve the re-projection error function to minimize the distance deviation to generate a camera pose transformation matrix; Determine the camera pose transformation matrix as the candidate calibration parameters of the current motion image.

8. The hand-eye calibration method of a standard model and marker point-based robot arm according to claim 7, wherein, According to the candidate calibration parameters of each motion image, fit to generate initial hand-eye calibration parameters, including: Extract the rotation matrix component and the translation vector component of each candidate calibration parameter from each candidate calibration parameter; According to the rotation matrix components in each candidate calibration parameter, calculate to generate a mean value of the rotation matrix components; According to the translation vector components in each candidate calibration parameter, calculate to generate a mean value of the translation vector components; For each candidate calibration parameter, calculate the deviation degree of the rotation matrix component of the current candidate calibration parameter relative to the mean value of the rotation matrix component to generate a rotation deviation of the current candidate calibration parameter; and calculate the deviation degree of the translation vector component of the current candidate calibration parameter relative to the mean value of the translation vector component to generate a translation deviation of the current candidate calibration parameter; According to each rotation deviation, each translation deviation, and a preset statistical tolerance threshold, determine an effective calibration parameter set; According to each rotation matrix component in the effective calibration parameter set, an average rotation matrix is calculated and generated; According to each translation vector component in the effective calibration parameter set, an average translation vector is calculated and generated; According to the average rotation matrix and the average translation vector, an initial hand-eye calibration parameter is constructed.

9. The method of claim 8, wherein, According to the initial hand-eye calibration parameter, the URDF standard model and the second joint angle value, a theoretical pose of a marker point is determined, including: From the URDF standard model, a fixed installation transformation parameter of the marker point relative to the end effector of the robot arm and a serial kinematics chain structure of the robot arm are parsed out; According to the second joint angle value, the serial kinematics chain structure is used for forward kinematics recursive calculation to generate a real-time spatial pose of the end effector of the robot arm in the robot arm base coordinate system; According to the real-time spatial pose of the end effector and the fixed installation transformation parameter, a global reference pose of the marker point in the robot arm base coordinate system is calculated and generated; According to the initial hand-eye calibration parameter, a spatial coordinate transformation matrix for representing the conversion from the robot arm base coordinate system to the camera coordinate system is constructed; Based on the spatial coordinate transformation matrix, a coordinate space mapping transformation is performed on the global reference pose of the marker point to calculate and generate a theoretical pose of the marker point in the camera coordinate system.

10. The standard model and marker point based hand-eye calibration method of the robot arm according to claim 9, wherein, According to the measured pose, the theoretical pose and the initial hand-eye calibration parameter, a final hand-eye calibration parameter is determined, including: According to the measured pose and the theoretical pose, an Euclidean distance deviation of the measured pose and the theoretical pose in the spatial position component and a rotation angle deviation of the measured pose and the theoretical pose in the spatial attitude component are calculated and generated; According to the Euclidean distance deviation and the theoretical pose, a comprehensive pose residual error is generated; Taking the minimization of the comprehensive pose residual error as an optimization objective, a parameter correction objective function based on nonlinear optimization is constructed; Taking the initial hand-eye calibration parameter as an initial value, the parameter correction objective function is solved based on an iterative optimization algorithm to calculate and generate a calibration parameter correction amount; According to the calibration parameter correction amount, the initial hand-eye calibration parameter is corrected to determine the final hand-eye calibration parameter.

Images