A method, apparatus and device for calibrating robot extrinsic parameters

Abstract

This application provides a method, apparatus, and device for calibrating robot extrinsic parameters. The method includes acquiring multiple time-aligned captured images and corresponding end-effector pose data; determining the initial extrinsic parameter matrix of the camera relative to the robot's base coordinate system based on the end-effector pose data and captured images of each frame; determining the three-dimensional coordinates of each gripper tip on the end effector in the robot's base coordinate system based on the end-effector pose data and gripper opening of each frame; projecting the three-dimensional coordinates of each gripper tip onto the captured image of the corresponding frame based on the camera's intrinsic parameter matrix and the camera's initial extrinsic parameter matrix to obtain the projected two-dimensional coordinates of the gripper tip in the corresponding frame; and calibrating the camera's extrinsic parameters based on the frame index of the multiple captured images, the three-dimensional coordinates of each gripper tip in the corresponding frame, and the projected two-dimensional coordinates to determine the target extrinsic parameter matrix. By using the robot's own gripper tips as calibration feature points, autonomous calibration of the camera's extrinsic parameters relative to the robot's base coordinate system is achieved.

Description

Technical Field

[0001] This invention relates to the field of robotics, and more specifically, to a method, apparatus, and device for calibrating robot extrinsic parameters. Background Technology

[0002] In the field of high-precision robotic operations, hand-eye calibration is a crucial step in enabling the collaborative work of the robot's end effector and vision sensor. Its core objective is to solve for the extrinsic parameter matrix between the vision sensor and the robot's end effector, establishing their coordinate mapping relationship. Existing hand-eye calibration methods generally rely on the controllable motion of the robot's end effector and the observation data from the vision sensor. This involves controlling the robot's end effector to drive the sensor or calibration target through a series of preset movements, simultaneously acquiring the robot's end effector pose data and the sensor's observation images, and then solving for the extrinsic parameter matrix based on a pre-defined calibration model.

[0003] Among them, the closed-form solution method is currently the most widely used initial value solution method. This method has high computational efficiency, simple process, and can quickly obtain the initial values ​​of extrinsic parameters. However, due to the mechanical characteristics of the robot body, such as backlash and flexibility, its absolute positioning accuracy is limited and cannot provide completely accurate end-effector pose data. At the same time, visual sensors inevitably have observation noise during the observation process, and the feature point extraction process in the calibration scene will also produce sub-pixel level errors. These errors will be directly transmitted to the closed-form solution solution process, resulting in limited accuracy of the obtained initial values ​​of extrinsic parameters. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of the prior art by providing a robot extrinsic parameter calibration method, apparatus, and device. This method involves first acquiring multiple frames of images and corresponding end-effector pose data to determine the initial extrinsic parameter matrix of the camera. Then, based on the gripper opening and end-effector pose data, the three-dimensional coordinates of each gripper tip in the robot's base coordinate system are reconstructed. The corresponding projected two-dimensional coordinates are obtained through projection. Finally, the frame index, the three-dimensional coordinates of each gripper tip, and the projected two-dimensional coordinates are combined to complete the camera extrinsic parameter calibration and obtain the target extrinsic parameter matrix, thereby improving the accuracy of the target extrinsic parameter matrix.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, embodiments of this application provide a method for calibrating robot extrinsic parameters, the method comprising: Acquire time-aligned multi-frame images and corresponding end-effector pose data. The acquired images are images captured by the camera on the robot of the robotic arm holding a calibration plate at the end. The end-effector pose data are the pose data of the end effector of the robotic arm. Based on the end pose data of each frame and the acquired image, determine the initial extrinsic parameter matrix of the camera relative to the robot's base coordinate system; Based on the end-effector pose data of each frame and the gripper opening of the end effector, determine the three-dimensional coordinates of the tips of each gripper on the end effector in the robot base coordinate system; Based on the camera's intrinsic parameter matrix and the camera's initial extrinsic parameter matrix, the three-dimensional coordinates of each gripper tip are projected onto the acquired image of the corresponding frame to obtain the projected two-dimensional coordinates of each gripper tip in the corresponding frame. The camera's extrinsic parameters are calibrated based on the frame index of the acquired images, the three-dimensional coordinates of the gripper tips of each corresponding frame, and the projected two-dimensional coordinates, to determine the target extrinsic parameter matrix.

[0006] In an optional implementation, determining the three-dimensional coordinates of each gripper tip on the end effector in the robot base coordinate system based on the end effector pose data of each frame and the gripper opening of the end effector includes: Based on the end-effector pose data of each frame, the gripper opening of the end effector, and the fixed offset of the preset gripper center relative to the end effector coordinate system, calculate the three-dimensional coordinates of each gripper tip in the end effector coordinate system. The three-dimensional coordinates of each gripper tip in the end effector coordinate system are transformed to the robot base coordinate system to obtain the three-dimensional coordinates of each gripper tip in the robot base coordinate system.

[0007] In an optional implementation, the method further includes projecting the three-dimensional coordinates of each gripper tip onto the acquired image of the corresponding frame based on the camera's intrinsic parameter matrix and the camera's initial extrinsic parameter matrix, to obtain the projected two-dimensional coordinates of the gripper tip in the corresponding frame. Based on the camera's intrinsic parameter matrix, the camera's preset distortion coefficients, and the preset image size, the acquired images of each frame are subjected to distortion removal processing to obtain the distortion-removed images of each frame. The region of interest is cropped from the distortion-free image of each frame to obtain the corrected image of each frame; The step of projecting the three-dimensional coordinates of each gripper tip onto the acquired image of the corresponding frame based on the camera's intrinsic parameter matrix and the camera's initial extrinsic parameter matrix to obtain the projected two-dimensional coordinates of each gripper tip in the corresponding frame includes: Based on the camera's intrinsic parameter matrix and the camera's initial extrinsic parameter matrix, the three-dimensional coordinates of each gripper tip are projected onto the corrected image of the corresponding frame to obtain the projected two-dimensional coordinates of each gripper tip in the corresponding frame.

[0008] In an optional implementation, before performing extrinsic parameter calibration on the camera based on the frame index of multiple acquired images, the three-dimensional coordinates of the gripper tips of each corresponding frame, and the projected two-dimensional coordinates, and determining the target extrinsic parameter matrix, the method further includes: The two-dimensional coordinates of the projection of the gripper tip in the corresponding frame are visualized and drawn in the acquired image of the corresponding frame. In response to a click operation on the tip of the target gripper in any frame of the acquired image, the two-dimensional coordinates of the projected target gripper tip in that frame are updated.

[0009] In an optional implementation, the step of calibrating the camera's extrinsic parameters and determining the target extrinsic parameter matrix based on the frame index of the acquired images, the three-dimensional coordinates of the gripper tips of each corresponding frame, and the projected two-dimensional coordinates includes: The annotation data for the multiple frames is determined based on the frame index of the acquired images, the three-dimensional coordinates of the gripper tips of the corresponding frames, and the projected two-dimensional coordinates. Based on the Lupin kernel function, construct an objective optimization function with the goal of minimizing the reprojection error; Based on the labeled data from multiple frames, the target optimization function is solved to obtain the target extrinsic parameter matrix.

[0010] In an optional implementation, the step of solving the target optimization function based on multi-frame labeled data to obtain the target extrinsic parameter matrix includes: The robustness loss of the robotic arm is calculated based on the labeled data from multiple frames. Based on the robustness loss of the robotic arm, a variety of preset optimization algorithms are used to solve the objective optimization function, resulting in multiple extrinsic parameter matrices and corresponding objective function values. The extrinsic parameter matrix that minimizes the objective function value is selected as the objective extrinsic parameter matrix.

[0011] In an optional implementation, before solving the objective optimization function using multiple preset optimization algorithms based on the robustness loss of the robotic arm to obtain multiple extrinsic parameter matrices and corresponding objective function values, the method further includes: Based on the labeled data of each frame, calculate the initial reprojection error of each frame. The initial reprojection error statistical parameters are obtained by statistically analyzing the initial reprojection error of multiple frames. Based on the initial reprojection error statistics parameters, calculate the adaptive threshold of the robust kernel function in the objective optimization function.

[0012] In an optional implementation, before solving the objective optimization function using multiple preset optimization algorithms based on the robustness loss of the robotic arm to obtain multiple extrinsic parameter matrices and corresponding objective function values, the method includes: The total robustness loss is calculated based on the robustness loss of the multiple robotic arms and the number of annotation points of each robotic arm in the multi-frame annotation data. Based on the robustness loss of the robotic arm, multiple preset optimization algorithms are used to solve the objective optimization function, resulting in multiple extrinsic parameter matrices and corresponding objective function values, including: Based on the total robust loss, multiple preset optimization algorithms are used to solve the objective optimization function, resulting in multiple extrinsic parameter matrices and corresponding objective function values.

[0013] Secondly, embodiments of this application also provide a robot extrinsic parameter calibration device, the device comprising: The acquisition module is used to acquire time-aligned multi-frame captured images and corresponding end-effector pose data. The captured images are images captured by the camera on the robot of the robotic arm holding a calibration plate at the end, and the end-effector pose data are the pose data of the end effector of the robotic arm. The determination module is used to determine the initial extrinsic parameter matrix of the camera relative to the robot base coordinate system based on the end pose data of each frame and the acquired image; The determining module is further configured to determine the three-dimensional coordinates of the tips of each gripper on the end effector in the robot base coordinate system based on the end effector pose data of each frame and the gripper opening of the end effector. The projection module is used to project the three-dimensional coordinates of the tips of each gripper onto the acquired image of the corresponding frame according to the intrinsic parameter matrix of the camera and the initial extrinsic parameter matrix of the camera, so as to obtain the projected two-dimensional coordinates of the tips of each gripper in the corresponding frame. The calibration module is used to calibrate the camera's extrinsic parameters based on the frame index of the acquired images, the three-dimensional coordinates of the gripper tips of the corresponding frames, and the projected two-dimensional coordinates, and to determine the target extrinsic parameter matrix.

[0014] Thirdly, embodiments of this application also provide a control device, including: a processor, a storage medium, and a bus. The storage medium stores program instructions executable by the processor. When the control device is running, the processor communicates with the storage medium via the bus, and the processor executes the program instructions to perform the steps of the robot extrinsic parameter calibration method as described in any of the first aspects.

[0015] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the robot extrinsic parameter calibration method as described in any of the first aspects.

[0016] The beneficial effects of this application are: This application provides a robot extrinsic parameter calibration method, apparatus, and device. The method includes: acquiring time-aligned multi-frame acquired images and corresponding end-effector pose data, wherein the acquired images are images captured by a camera on the robot of a robotic arm holding a calibration plate at the end, and the end-effector pose data are pose data of the end effector of the robotic arm; determining the initial extrinsic parameter matrix of the camera relative to the robot's base coordinate system based on the end-effector pose data of each frame and the acquired images; determining the three-dimensional coordinates of the tips of each gripper on the end effector in the robot's base coordinate system based on the end-effector pose data of each frame and the gripper opening of the end effector; projecting the three-dimensional coordinates of each gripper tip onto the acquired image of the corresponding frame based on the camera's intrinsic parameter matrix and the camera's initial extrinsic parameter matrix to obtain the projected two-dimensional coordinates of the gripper tip in the corresponding frame; and calibrating the camera's extrinsic parameters based on the frame index of the multi-frame acquired images, the three-dimensional coordinates of each gripper tip in the corresponding frame, and the projected two-dimensional coordinates to determine the target extrinsic parameter matrix.

[0017] The method of this application first acquires multiple frames of images and corresponding end-effector pose data to determine the initial extrinsic parameter matrix of the camera. Then, it reconstructs the three-dimensional coordinates of each gripper tip in the robot's base coordinate system based on the gripper opening and end-effector pose data. The corresponding projected two-dimensional coordinates are obtained by projection. Finally, the camera extrinsic parameter calibration is completed by combining the frame index, the three-dimensional coordinates of each gripper tip, and the projected two-dimensional coordinates. It does not rely on single-point constraints of high-precision calibration board features. It uses the robot's own gripper tips as calibration feature points to achieve autonomous calibration of the extrinsic parameters of the camera and the robot's base coordinate system. This improves the automation level of robot vision extrinsic parameter calibration and the accuracy of the target extrinsic parameter matrix, and is suitable for calibration scenarios with multiple postures and positions. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 One of the flowcharts for a robot extrinsic parameter calibration method provided in this application embodiment; Figure 2 A second schematic flowchart illustrating a robot extrinsic parameter calibration method provided in this application embodiment; Figure 3 The third flowchart illustrating a robot extrinsic parameter calibration method provided in this application embodiment; Figure 4 A fourth schematic flowchart illustrating a robot extrinsic parameter calibration method provided in this application embodiment; Figure 5 Fifth of a flowchart illustrating a robot extrinsic parameter calibration method provided in this application embodiment; Figure 6 A flowchart illustrating a robot extrinsic parameter calibration method provided in this application embodiment is shown in Figure 6. Figure 7 The seventh flowchart illustrating a robot extrinsic parameter calibration method provided in this application embodiment; Figure 8 This is the eighth flowchart illustrating a robot extrinsic parameter calibration method provided in this application embodiment; Figure 9 This is a schematic diagram of the functional modules of a robot extrinsic parameter calibration device provided in an embodiment of this application; Figure 10 This is a schematic diagram of a control device provided in an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0021] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0022] In the description of this application, it should be noted that if the terms "upper", "lower", etc. appear to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship that the product of this application is usually placed in, it is only for the convenience of describing this application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0023] Furthermore, the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Additionally, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] It should be noted that, where there is no conflict, the features in the embodiments of this application can be combined with each other.

[0025] To obtain more accurate extrinsic parameters of the robot's camera, this application provides a robot extrinsic parameter calibration method. This method first acquires multiple frames of images and corresponding end-effector pose data to determine the initial extrinsic parameter matrix of the camera. Then, it reconstructs the three-dimensional coordinates of each gripper tip in the robot's base coordinate system based on the gripper opening and end-effector pose data. These coordinates are then projected to obtain the corresponding projected two-dimensional coordinates. Finally, the frame index, the three-dimensional coordinates of each gripper tip, and the projected two-dimensional coordinates are combined to complete the camera extrinsic parameter calibration and obtain the target extrinsic parameter matrix. This method eliminates the need for single-point constraints on high-precision calibration boards, using the robot's own gripper tips as calibration feature points to achieve autonomous calibration of the camera and robot base coordinate system extrinsic parameters, thereby improving the accuracy of the target extrinsic parameter matrix.

[0026] The robot extrinsic parameter calibration method provided in this application will be explained in detail below with reference to the accompanying drawings and specific examples. Figure 1 This is one of the flowcharts illustrating a robot extrinsic parameter calibration method provided in an embodiment of this application; as shown below. Figure 1 As shown, the method includes: S101. Acquire time-aligned multi-frame images and corresponding end-effector pose data.

[0027] The acquired images are images captured by the camera on the robot, showing the robotic arm holding the calibration plate at its end effector. The end effector pose data are the pose data of the end effector of the robotic arm.

[0028] In this embodiment, the operator pre-fixes the checkerboard calibration plate onto the gripper of the robot's end effector to ensure that the checkerboard calibration plate is located at the center of the gripper when the gripper is closed, and that the plane of the calibration plate is parallel to the plane of the gripper, and to ensure that the checkerboard calibration plate remains stable during the robot's movement.

[0029] Then, the operator uses a remote control device to control one of the robot's robotic arms to perform multi-pose movements within the field of view of the head camera. Specifically, this includes translational movements along the X, Y, and Z axes (i.e., translational movements in the up, down, left, right, and forward / backward directions) and rotational movements around the X, Y, and Z axes in the robot's base coordinate system. During the movement, the operator ensures that the chessboard calibration board remains within the camera's field of view, thereby collecting comprehensive data for one robotic arm. Then, the operator controls the other robotic arm to perform multi-pose movements within the field of view of the head camera, collecting comprehensive data for the other robotic arm. The robot's base coordinate system is a Cartesian coordinate system established with the robot's mounting base as the spatial reference.

[0030] The robot's sensor data acquisition module synchronously acquires data at a preset frequency. For example, the end pose data of the robot's end effector is acquired at a frequency of 100Hz. The end pose data includes position vectors and attitude rotation matrices. Multiple frames of image data from the head camera are acquired at a frame rate of 30fps. At the same time, a precise timestamp is recorded for each end pose data and each frame of acquired image.

[0031] Then, the robot's data synchronization module calls the timestamp matching algorithm to find the robot's end-effector pose data that is closest to the timestamp of each frame of acquired images, based on the timestamp of the acquired images. This completes the time alignment of multiple frames of acquired images and end-effector pose data, and finally obtains the acquired images and end-effector pose data that correspond one-to-one with the timestamps of multiple frames.

[0032] It should be noted that if the robot's head camera is a monocular camera, then multiple frames of images are acquired using the monocular camera. If the robot's head camera is a binocular camera, then multiple frames of images are acquired using the binocular camera. This results in multiple frames of images acquired by the left camera and multiple frames of images acquired by the right camera. Subsequently, the target extrinsic parameter matrix of the left camera is determined based on the multiple frames of images acquired by the left camera and the corresponding end-effector pose data, and the target extrinsic parameter matrix of the right camera is determined based on the multiple frames of images acquired by the right camera and the corresponding end-effector pose data.

[0033] S102. Based on the end pose data and acquired images of each frame, determine the initial extrinsic parameter matrix of the camera relative to the robot's base coordinate system.

[0034] Specifically, the hand-eye calibration equation is used to calculate the end pose data of each frame and the corresponding acquired image to determine the initial extrinsic parameter matrix of the camera relative to the robot's base coordinate system.

[0035] Specifically, the hand-eye calibration equation is expressed as AX = XB, where A represents the homogeneous matrix of the camera between two adjacent frames, B represents the homogeneous matrix of the robot's end effector between two adjacent frames, and X represents the extrinsic parameter matrix of the camera relative to the robot's base coordinate system. The corner features of the checkerboard calibration board in each frame of the acquired image are combined with the end effector pose data of the corresponding frame and substituted into the AX = XB equation. The initial extrinsic parameter matrix of the camera relative to the robot's base coordinate system is then obtained using the least squares method. If the camera is a stereo camera, the same method is used to obtain the initial extrinsic parameter matrices of the left and right cameras.

[0036] S103. Based on the end-effector pose data of each frame and the gripper opening of the end effector, determine the three-dimensional coordinates of the tips of each gripper on the end effector in the robot base coordinate system.

[0037] Specifically, based on the end-effector pose data of each frame and the gripper opening of the end effector, the coordinates of the gripper tips on the end effector are transformed to obtain the three-dimensional coordinates of each gripper tip in the robot base coordinate system.

[0038] S104. Based on the camera's intrinsic parameter matrix and the camera's initial extrinsic parameter matrix, project the three-dimensional coordinates of each gripper tip onto the acquired image of the corresponding frame to obtain the projected two-dimensional coordinates of each gripper tip in the corresponding frame.

[0039] Specifically, the camera intrinsic parameter matrix K and distortion coefficients D, which are pre-stored in the camera's corresponding calibration file, are obtained. The camera intrinsic parameter matrix K includes the camera's focal length parameter and image center coordinate parameter, and the distortion coefficients D include radial distortion coefficients and tangential distortion coefficients.

[0040] Based on the camera intrinsic parameter matrix, the camera's initial extrinsic parameter matrix, and the three-dimensional coordinates of each gripper tip, a preset projection transformation formula is used to perform projection calculations on the three-dimensional coordinates of each gripper tip in the corresponding frame, thereby obtaining the projected two-dimensional coordinates of each gripper tip in the corresponding frame.

[0041] The projection transformation formula is expressed as follows: , Represented as a camera projection function, This is represented as the initial extrinsic parameter matrix of the camera. This is represented as the 3D coordinates of the gripper tip. The initial extrinsic parameter matrix of the camera transforms the 3D coordinates of the gripper tip from the robot base coordinate system to the camera coordinate system. The camera projection function performs perspective projection on the 3D coordinates in the camera coordinate system using the intrinsic parameter matrix K and distortion coefficients D, ultimately outputting the 2D pixel coordinates of the gripper tip in the acquired image. That is, projected two-dimensional coordinates.

[0042] S105. Based on the frame index of the multi-frame acquired images, the three-dimensional coordinates of the gripper tips of the corresponding frames, and the projected two-dimensional coordinates, the camera's extrinsic parameters are calibrated to determine the target extrinsic parameter matrix.

[0043] Specifically, by using the objective function of minimizing reprojection error, the camera's extrinsic parameters are calibrated using the three-dimensional coordinates of the gripper tips and the two-dimensional coordinates of the projection for each corresponding frame, thus determining the camera's target extrinsic parameter matrix.

[0044] In summary, this application provides a robot extrinsic parameter calibration method, which includes: acquiring time-aligned multi-frame acquired images and corresponding end-effector pose data, wherein the acquired images are images captured by a camera on the robot of a robotic arm holding a calibration plate at the end, and the end-effector pose data are the pose data of the end effector of the robotic arm; determining the initial extrinsic parameter matrix of the camera relative to the robot's base coordinate system based on the end-effector pose data of each frame and the acquired images; determining the three-dimensional coordinates of each gripper tip on the end effector in the robot's base coordinate system based on the end-effector pose data of each frame and the gripper opening of the end effector; projecting the three-dimensional coordinates of each gripper tip onto the acquired image of the corresponding frame based on the camera's intrinsic parameter matrix and the camera's initial extrinsic parameter matrix to obtain the projected two-dimensional coordinates of the gripper tip in the corresponding frame; and performing extrinsic parameter calibration on the camera based on the frame index of the multi-frame acquired images, the three-dimensional coordinates of each gripper tip in the corresponding frame, and the projected two-dimensional coordinates to determine the target extrinsic parameter matrix.

[0045] The method of this application first acquires multiple frames of images and corresponding end-effector pose data to determine the initial extrinsic parameter matrix of the camera. Then, it reconstructs the three-dimensional coordinates of each gripper tip in the robot's base coordinate system based on the gripper opening and end-effector pose data. The corresponding projected two-dimensional coordinates are obtained by projection. Finally, the camera extrinsic parameter calibration is completed by combining the frame index, the three-dimensional coordinates of each gripper tip, and the projected two-dimensional coordinates. It does not rely on single-point constraints of high-precision calibration board features. It uses the robot's own gripper tips as calibration feature points to achieve autonomous calibration of the extrinsic parameters of the camera and the robot's base coordinate system. This improves the automation level of robot vision extrinsic parameter calibration and the accuracy of the target extrinsic parameter matrix, and is suitable for calibration scenarios with multiple postures and positions.

[0046] Based on the above embodiments, this application also provides another possible implementation of the robot extrinsic parameter calibration method. Figure 2 This is a second flowchart illustrating a robot extrinsic parameter calibration method provided in an embodiment of this application, as shown below. Figure 2 As shown, based on the end effector pose data of each frame and the gripper opening of the end effector, the three-dimensional coordinates of each gripper tip on the end effector in the robot base coordinate system are determined, including: S201. Based on the end-effector pose data of each frame, the gripper opening of the end effector, and the fixed offset of the preset gripper center relative to the end effector coordinate system, calculate the three-dimensional coordinates of each gripper tip in the end effector coordinate system.

[0047] In this embodiment, each end effector of the robotic arm has at least two gripper tips for grasping objects. Taking the two gripper tips of the robot's left arm as an example, namely the left tip point and the right tip point of the left arm, the three-dimensional coordinates of the left tip point and the right tip point of the left arm in the end effector coordinate system are calculated based on the end pose data of each frame, the gripper opening of the left arm end effector, and the fixed offset of the preset gripper center relative to the end effector coordinate system of the left arm.

[0048] Among them, the fixed offset of the center of the pre-set gripper relative to the coordinate system of the end effector It is determined by the mechanical design parameters of the grippers, for example, a fixed offset. =[0.08,0,0] indicates that the gripper center is in the positive X-axis direction of the end effector coordinate system, 0.08 meters away from the origin of the left arm end effector, and there is no offset in the Y-axis and Z-axis directions.

[0049] Specifically, for each frame of data, the end effector pose data for that frame is obtained, including the position vector of the left arm end effector in its own coordinate system. and attitude rotation matrix Simultaneously, the gripper opening of the left arm end effector in that frame is obtained. Using a preset formula, the three-dimensional coordinates of the left tip and the right tip of the left arm in the end effector coordinate system are calculated respectively.

[0050] The preset formula corresponding to the left tip of the left arm is expressed as follows: The preset formula corresponding to the left tip of the right arm is expressed as: The position vector of this frame and attitude rotation matrix The gripper opening of the left arm end effector in this frame. Fixed offset Substituting these values ​​into the aforementioned preset formulas, the three-dimensional coordinates of the left tip of the left arm in the end effector coordinate system are calculated. The three-dimensional coordinates of the right tip of the left arm in the end effector coordinate system .

[0051] Similarly, if the end-effector pose data is the end-effector pose data of the right arm, then substitute the gripper opening of the right arm end effector. The three-dimensional coordinates of the left tip of the right arm in the end effector coordinate system were calculated. The three-dimensional coordinates of the right tip of the right arm in the end effector coordinate system .

[0052] S202. Transform the three-dimensional coordinates of each gripper tip in the end effector coordinate system to the robot base coordinate system to obtain the three-dimensional coordinates of each gripper tip in the robot base coordinate system.

[0053] Specifically, the pose transformation matrix of the robot end effector relative to the robot base coordinate system is obtained. This pose transformation matrix is ​​converted from the end effector pose data and includes the translation and rotation components of the end effector relative to the base coordinate system.

[0054] Then, based on the pose transformation matrix, the three-dimensional coordinates of each gripper tip in the end effector coordinate system are transformed to the robot base coordinate system, thus obtaining the three-dimensional coordinates of each gripper tip in the robot base coordinate system.

[0055] Taking the tips of the two grippers on the robot's left arm as an example, the conversion formula corresponding to the left tip of the left arm is expressed as follows: The conversion formula corresponding to the right tip of the left arm is expressed as: , This is represented by the pose transformation matrix of the left arm end effector relative to the robot's base coordinate system, thus obtaining the three-dimensional coordinates of the left tip of the left arm in the robot's base coordinate system. The three-dimensional coordinates of the right tip of the left arm in the robot's base coordinate system .

[0056] Similarly, if the end-effector pose data is the end-effector pose data of the right arm, the same method is used to obtain the three-dimensional coordinates of the left tip of the right arm in the end-effector coordinate system. The three-dimensional coordinates of the right tip of the right arm in the end effector coordinate system .

[0057] The method provided in this application first calculates the three-dimensional coordinates of each gripper tip in the end effector coordinate system by combining the end effector pose data, gripper opening, and fixed offset. Then, it transforms the coordinates to the robot base coordinate system to obtain the three-dimensional coordinates of each gripper tip in the robot base coordinate system. A complete coordinate mapping relationship between the gripper tips and the end effector coordinate system is established. By utilizing the fixed offset characteristics of the gripper mechanical structure and real-time opening and closing parameters, and closely matching the actual robotic arm installation layout, the accuracy of the three-dimensional coordinate calculation for each gripper tip is ensured.

[0058] This application also provides another possible implementation of the robot extrinsic parameter calibration method. Figure 3 This is the third flowchart illustrating a robot extrinsic parameter calibration method provided in this application embodiment, as shown below. Figure 3As shown, based on the camera's intrinsic parameter matrix and initial extrinsic parameter matrix, the three-dimensional coordinates of each gripper tip are projected onto the acquired image of the corresponding frame to obtain the projected two-dimensional coordinates of the gripper tip in the corresponding frame. This method further includes: S301. Based on the camera's intrinsic parameter matrix, the camera's preset distortion coefficients, and the preset image size, perform distortion correction processing on the acquired images of each frame to obtain the distortion-corrected images of each frame.

[0059] In this embodiment, the camera intrinsic parameter matrix K, preset distortion coefficient D, and preset image size are obtained from the camera calibration file. Based on the camera intrinsic parameter matrix K, preset distortion coefficient D, and preset image size, distortion correction is performed on each pixel in each frame of the acquired image. For pixels with radial distortion, their coordinates are corrected using the radial distortion formula; for pixels with tangential distortion, their coordinates are corrected using the tangential distortion formula. Finally, the distortion-free image is output.

[0060] S302. Cropping the region of interest from the distortion-free image of each frame to obtain the corrected image of each frame.

[0061] The region of interest is determined based on the calibration scene to ensure that it includes the gripper and calibration plate, while removing irrelevant background areas such as the robot body and environmental clutter from the image.

[0062] For example, if the region of interest is a rectangular area, an image cropping algorithm is used to crop the corresponding area in the distorted image based on the coordinate range of the region of interest, and remove the background pixels outside the area to obtain the corrected image for each frame.

[0063] Based on the above, using the camera's intrinsic parameter matrix and initial extrinsic parameter matrix, the three-dimensional coordinates of each gripper tip are projected onto the acquired image of the corresponding frame, obtaining the projected two-dimensional coordinates of each gripper tip in the corresponding frame, including: S303. Based on the camera's intrinsic parameter matrix and the camera's initial extrinsic parameter matrix, project the three-dimensional coordinates of each gripper tip onto the corrected image of the corresponding frame to obtain the projected two-dimensional coordinates of each gripper tip in the corresponding frame.

[0064] Specifically, based on the camera intrinsic parameter matrix, the camera's initial extrinsic parameter matrix, and the three-dimensional coordinates of each gripper tip, a preset projection transformation formula is used to perform projection calculations on the three-dimensional coordinates of each gripper tip in the corrected image of the corresponding frame, thereby obtaining the projected two-dimensional coordinates of each gripper tip in the corrected image of the corresponding frame.

[0065] The method provided in this application first corrects distortion of the acquired image, then crops the region of interest to remove invalid background interference, and finally projects the three-dimensional coordinates of each gripper tip onto the corrected image. This effectively eliminates image distortion errors caused by radial and tangential distortion of the camera, while cropping redundant background reduces computational load, focuses the effective area of ​​the gripper and calibration plate, and allows the projection process to be based on the corrected standard image, improving the accuracy of the two-dimensional coordinates of the gripper tip projection and avoiding interference from image distortion and irrelevant background on the calibration projection results.

[0066] This application also provides another possible implementation of the robot extrinsic parameter calibration method. Figure 4 This is the fourth flowchart illustrating a robot extrinsic parameter calibration method provided in this application embodiment, as shown below. Figure 4 As shown, before determining the target extrinsic parameter matrix by calibrating the camera's extrinsic parameters based on the frame index of multiple acquired images, the three-dimensional coordinates of each gripper tip in the corresponding frame, and the projected two-dimensional coordinates, the method further includes: S401. Visualize and draw the two-dimensional coordinates of the claw tip projected onto the corresponding frame in the acquired image of the corresponding frame.

[0067] In this embodiment, the visualization interface generation module calls the image drawing function to obtain the corrected images of each frame and the two-dimensional coordinates of the projection of each gripper tip. The module covers the corresponding two-dimensional coordinates of the projection of each gripper tip in the corrected images of each frame with color marks, but does not obscure the key features of the grippers and calibration plate. At the same time, the frame index is marked in the upper left corner of the image to facilitate the operator to view it frame by frame.

[0068] After the drawing is completed, an interactive annotation interface containing the corrected image is generated. The center of the interface displays the corrected projected image and the color marks of each gripper tip. Operation buttons, such as previous frame, next frame, and save annotation, are displayed at the top of the interface, allowing operators to zoom and pan the image for viewing.

[0069] S402. In response to a click operation on the tip of the target gripper in any frame of the acquired image, update the two-dimensional coordinates of the projected tip of the target gripper in any frame.

[0070] Specifically, operators use an interactive annotation interface to view the corrected projected images frame by frame, focusing on whether the projected coordinates of each gripper tip match the actual gripper tip position in the image. When an operator finds a significant deviation between the projected coordinates of the target gripper tip and the actual gripper tip position in a frame, they click on the actual gripper tip position in the image with the mouse. The annotation interface responds to this click, acquires the two-dimensional coordinates of the mouse click, and uses these coordinates as the true two-dimensional coordinates of the target gripper tip. The projected two-dimensional coordinates of the target gripper tip are then updated, and the actual marked position of the target gripper tip is marked in a different color in the acquired image.

[0071] After the annotation is completed, click the "Save Annotation" button in the interface to associate and store the updated projected 2D coordinates of the gripper tip, the unupdated projected 2D coordinates of the gripper tip, the frame index, and the 3D coordinates of the gripper tip to obtain the annotation file.

[0072] The method provided in this application embodiment visualizes and differentiates the projection tip point, and supports interactive clicking by operators to correct projection deviation coordinates. This quickly corrects the projection offset caused by inaccurate initial extrinsic parameters, generates accurate and reliable manually labeled true two-dimensional coordinates, provides high-quality true data for subsequent nonlinear optimization, and effectively reduces the impact of labeling errors on calibration results.

[0073] This application also provides another possible implementation of the robot extrinsic parameter calibration method. Figure 5 This is the fifth flowchart illustrating a robot extrinsic parameter calibration method provided in this application embodiment, as shown below. Figure 5 As shown, the camera's extrinsic parameters are calibrated based on the frame index of multiple acquired images, the three-dimensional coordinates of the gripper tips in the corresponding frames, and the projected two-dimensional coordinates, to determine the target extrinsic parameter matrix, including: S501. Based on the frame index of the multi-frame acquired images, the three-dimensional coordinates of the tips of each gripper in the corresponding frame, and the projected two-dimensional coordinates, determine the annotation data of the multi-frame images respectively.

[0074] In this embodiment, based on the frame index of the multi-frame acquired images, the three-dimensional coordinates and projected two-dimensional coordinates of each gripper tip corresponding to the multi-frame acquired images are organized to determine the complete annotation data of the multi-frame. The annotation data of each frame includes: frame index, identifiers of two gripper tip points, three-dimensional coordinates of two gripper tip points in the robot base coordinate system, and projected two-dimensional coordinates of two gripper tip points.

[0075] If the projected two-dimensional coordinates of the gripper tip are not updated, the projected two-dimensional coordinates in the annotation data of multiple frames are the two-dimensional coordinates obtained by projection calculation. If the projected two-dimensional coordinates of the gripper tip are updated, the projected two-dimensional coordinates in the annotation data of multiple frames are the updated two-dimensional coordinates.

[0076] S502. Based on the Lupin kernel function, construct an objective optimization function with the goal of minimizing the reprojection error.

[0077] S503. Based on the labeled data of multiple frames, solve the target optimization function to obtain the target extrinsic parameter matrix.

[0078] Among them, the robust kernel function Huber is selected as the error suppression function. This function can effectively suppress the influence of abnormal annotation points on the optimization results. Based on the robust kernel function, an objective optimization function is constructed with the goal of minimizing the reprojection error.

[0079] The objective optimization function is expressed as:

[0080]

[0081] in, This represents the sequence number of the frame index, with a value ranging from 1 to N; Indicates the first The identifier at the tip of the gripper in the frame has a value ranging from 1 to... , Indicates the first Frame number The three-dimensional coordinates of the tips of the grippers; Indicates the first Frame number The projected two-dimensional coordinates of the tips of the grippers; Represents the camera projection function; This represents the Huber robust kernel function; For Euclidean distance, This represents the extrinsic parameter matrix.

[0082] Then, the labeled data from multiple frames are substituted into the objective optimization function, and the objective optimization function is solved to obtain the objective extrinsic parameter matrix.

[0083] In the method provided in this application embodiment, multi-frame annotation data is standardized and organized, a robust kernel function is introduced to construct an optimization objective function that minimizes reprojection error, and then the target extrinsic parameter matrix is ​​obtained by solving based on the annotation data. The reprojection error is used as an optimization constraint, and the robust kernel function is combined to suppress the interference of abnormal annotation points and noisy data, thus getting rid of the defects of traditional least squares optimization that are easily affected by outliers, improving the anti-interference ability and calibration robustness of extrinsic parameter solution, and obtaining a more accurate optimal extrinsic parameter matrix.

[0084] This application also provides another possible implementation of the robot extrinsic parameter calibration method. Figure 6 This is the sixth flowchart illustrating a robot extrinsic parameter calibration method provided in this application embodiment, as shown below. Figure 6 As shown, based on the labeled data from multiple frames, the objective optimization function is solved to obtain the objective extrinsic parameter matrix, including: S601. Calculate the robustness loss of the robotic arm based on the labeled data of multiple frames.

[0085] In this embodiment, the left and right arms of the robot are used as separate calculation objects. The labeled data of multiple frames corresponding to each robotic arm is read, and the labeled data of multiple frames are substituted into the target optimization function to calculate the robust loss of each robotic arm.

[0086] S602. Based on the robust loss of the robotic arm, a variety of preset optimization algorithms are used to solve the objective optimization function, resulting in multiple extrinsic parameter matrices and the corresponding objective function values.

[0087] S603. Select the extrinsic parameter matrix with the smallest objective function value as the objective extrinsic parameter matrix.

[0088] The pre-defined optimization algorithms include: Limited-memory Broyden–Fletcher–Goldfarb–Shanno with Bounds (L-BFGS-B), Truncated Newton Constrained (TNC), and Sequential Least Squares Programming (SLSQP). All three algorithms are used to solve nonlinear least squares problems and are adapted to the objective optimization function. The objective optimization function is solved using these pre-defined algorithms to obtain the corresponding objective function values. and the extrinsic parameter matrix corresponding to the objective function value. .

[0089] Then, the multiple objective function values ​​are sorted, and the extrinsic parameter matrix with the smallest objective function value is determined as the target extrinsic parameter matrix. Subsequently, the target extrinsic parameter matrix is ​​transformed into a homogeneous transformation matrix and written into a calibration file, so that the robot can load the calibration file in subsequent operations to achieve accurate coordinate transformation from the image pixel coordinates acquired by the camera to the robot's base coordinate system.

[0090] In the method provided in this application embodiment, the robust loss of each robotic arm is calculated separately, and multiple optimization algorithms are used in parallel to solve the problem and select the extrinsic parameter matrix with the smallest objective function value. By utilizing the solution advantages of different optimization algorithms, the optimal solution is selected through comparison and screening, further avoiding the local convergence problem of optimization solution and ensuring the optimality of the calibration extrinsic parameter matrix.

[0091] This application also provides another possible implementation of the robot extrinsic parameter calibration method. Figure 7 This is the seventh flowchart illustrating a robot extrinsic parameter calibration method provided in this application embodiment, as shown below. Figure 7 As shown, based on the robustness loss of the robotic arm, various preset optimization algorithms are used to solve the objective optimization function. Before obtaining multiple extrinsic parameter matrices and the corresponding objective function values, the method also includes: S701. Calculate the initial reprojection error of each frame based on the annotation data of each frame.

[0092] S702. Statistical parameters of the initial reprojection error are obtained by statistically analyzing the initial reprojection error of multiple frames.

[0093] S703. Based on the initial reprojection error statistical parameters, calculate the adaptive threshold of the robust kernel function in the objective optimization function.

[0094] In this embodiment, the initial reprojection error of each frame is calculated based on the three-dimensional coordinates of each gripper tip and the projected two-dimensional coordinates in the annotation data of each frame. Then, the initial reprojection errors of multiple frames are statistically analyzed to obtain the initial reprojection error statistical parameter RMSE.

[0095] Substituting the initial reprojection error statistical parameter RMSE into the adaptive threshold calculation formula, we obtain the adaptive threshold of the robust kernel function in the objective optimization function. .

[0096] Among them, adaptive threshold The calculation formula is expressed as follows:

[0097] in, This represents the base threshold, for example, if the base threshold... Given a resolution of 30 pixels, the initial reprojection error statistical parameter RMSE is calculated to be 45.3 pixels. Substituting this into the adaptive threshold calculation formula, the result is:

[0098] Thus, an adaptive threshold is obtained. The calculated adaptive threshold is set to 30 pixels and stored in the optimization parameter file for use in constructing the objective optimization function and calculating the robust loss.

[0099] The method provided in this application calculates the initial reprojection error frame by frame and statistically analyzes the error distribution parameters to adaptively solve the robust kernel function threshold. It can dynamically adapt the threshold size according to the actual error distribution of the field calibration data, accurately distinguish between normal errors and abnormal outliers, and adaptively match the robust constraint requirements under different calibration scenarios and different camera conditions, thereby improving the adaptive adaptability of the target optimization algorithm to complex conditions.

[0100] This application also provides another possible implementation of the robot extrinsic parameter calibration method. Figure 8 This is the eighth flowchart illustrating a robot extrinsic parameter calibration method provided in this application embodiment, as shown below. Figure 8 As shown, based on the robustness loss of the robotic arm, various preset optimization algorithms are used to solve the objective optimization function, and before obtaining multiple extrinsic parameter matrices and the corresponding objective function values, the methods include: S801. Calculate the total robustness loss based on the robustness loss of multiple robotic arms and the number of annotation points for each robotic arm in the multi-frame annotation data.

[0101] In this embodiment, the robot includes two robotic arms, left and right. Based on the multi-frame labeled data of the left and right robotic arms, the objective optimization function is solved to obtain the robust loss of the left robotic arm and the robust loss of the right robotic arm. At the same time, the number of labeled points of each robotic arm in the multi-frame labeled data is read.

[0102] A weighted average loss strategy is adopted, which calculates the total robust loss based on the robust loss of multiple robotic arms and the number of annotation points of each robotic arm in the multi-frame annotation data. This strategy can avoid the robotic arm with a large number of annotation points from having an excessive impact on the total loss and ensure that the loss contribution of the left and right arms is balanced.

[0103] The formula for calculating the weighted average loss is as follows:

[0104] in, This is represented as the robustness loss of the left robotic arm. This is represented as the robustness loss of the right robotic arm. This represents the number of markers on the left robotic arm. This is represented by the number of labeled points on the right robotic arm. The total robustness loss is then calculated. And it serves as the optimization objective of the objective optimization function.

[0105] Based on the robust loss of the robotic arm described above, various preset optimization algorithms are used to solve the objective optimization function, resulting in multiple extrinsic parameter matrices and corresponding objective function values, including: S802. Based on the total robust loss, various preset optimization algorithms are used to solve the objective optimization function, resulting in multiple extrinsic parameter matrices and the corresponding objective function values.

[0106] Specifically, the extrinsic parameter matrix T to be optimized is converted into a 6-dimensional vector, with the first three terms being translation vectors and the last three terms being rotational components represented by Euler angles. Parameter boundaries are set, including the ranges of the translation vectors and rotational components. Then, a weighted strategy based on arm-average loss is used to calculate the total robust loss. Multiple preset optimization algorithms are sequentially called to solve the problem, yielding the extrinsic parameter matrices and objective function values ​​corresponding to each algorithm.

[0107] The method provided in this application combines the robust loss of multiple robotic arms with the number of labeled points to calculate the total robust loss by average weighting, and then optimizes the solution based on the total robust loss using various preset optimization algorithms; it balances the differences in the number of labeled data of multiple robotic arms with the proportion of loss contribution, avoids the problem of single-arm data dominating the optimization result due to uneven number of labeled points, and achieves balanced constraints on the calibration loss of multiple robotic arms.

[0108] The robot extrinsic calibration device and control equipment provided in any of the above embodiments of this application will be explained below. The specific implementation process and the resulting technical effects are the same as those in the corresponding method embodiments. For the sake of brevity, parts not mentioned in this embodiment can be referred to the corresponding content in the method embodiments.

[0109] Figure 9 This is a schematic diagram of the functional modules of a robot extrinsic parameter calibration device provided in an embodiment of this application. Figure 9 As shown, the robot extrinsic calibration device 100 includes: The acquisition module 110 is used to acquire time-aligned multi-frame acquisition images and corresponding end-effector pose data. The acquisition images are images captured by the camera on the robot of the robotic arm holding the calibration plate at the end, and the end-effector pose data are the pose data of the end effector of the robotic arm. The determination module 120 is used to determine the initial extrinsic parameter matrix of the camera relative to the robot's base coordinate system based on the end pose data and acquired images of each frame. The determination module 120 is also used to determine the three-dimensional coordinates of the tips of each gripper on the end effector in the robot base coordinate system based on the end pose data of each frame and the gripper opening of the end effector. The projection module 130 is used to project the three-dimensional coordinates of each gripper tip onto the acquired image of the corresponding frame based on the camera's intrinsic parameter matrix and the camera's initial extrinsic parameter matrix, so as to obtain the projected two-dimensional coordinates of each gripper tip in the corresponding frame. The calibration module 140 is used to calibrate the camera's extrinsic parameters based on the frame index of the multi-frame acquired images, the three-dimensional coordinates of the gripper tips of the corresponding frames, and the projected two-dimensional coordinates, and to determine the target extrinsic parameter matrix.

[0110] Optionally, the determining module 120 is further configured to calculate the three-dimensional coordinates of each gripper tip in the end effector coordinate system based on the end effector pose data of each frame, the gripper opening of the end effector, and the fixed offset of the preset gripper center relative to the end effector coordinate system; and to transform the three-dimensional coordinates of each gripper tip in the end effector coordinate system to the robot base coordinate system to obtain the three-dimensional coordinates of each gripper tip in the robot base coordinate system.

[0111] Optionally, the device further includes: The processing module is used to perform distortion correction processing on the acquired images of each frame based on the camera's intrinsic parameter matrix, the camera's preset distortion coefficients, and the preset image size, so as to obtain the distortion correction images of each frame. The cropping module is used to crop the region of interest in each frame of the distortion-free image to obtain the corrected image of each frame; The projection module 130 is also used to project the three-dimensional coordinates of each gripper tip onto the corrected image of the corresponding frame based on the camera's intrinsic parameter matrix and the camera's initial extrinsic parameter matrix, so as to obtain the projected two-dimensional coordinates of each gripper tip in the corresponding frame.

[0112] Optionally, the device further includes: The drawing module is used to visualize and draw the two-dimensional coordinates of the jaw tip projected onto the corresponding frame in the captured image of the corresponding frame. The update module is used to respond to a click operation on the tip of the target gripper in any frame of the acquired image and update the two-dimensional coordinates of the projected target gripper tip in any frame.

[0113] Optionally, the determining module 120 is also used to determine the annotation data of the multiple frames based on the frame index of the multiple acquired images, the three-dimensional coordinates of each gripper tip of the corresponding frame, and the two-dimensional coordinates of the projection; construct a target optimization function with the goal of minimizing the reprojection error based on the Lupin kernel function; and solve the target optimization function based on the annotation data of the multiple frames to obtain the target extrinsic parameter matrix.

[0114] Optionally, the determination module 120 is also used to calculate the robustness loss of the robotic arm based on the labeled data of multiple frames; based on the robustness loss of the robotic arm, a variety of preset optimization algorithms are used to solve the target optimization function to obtain multiple extrinsic parameter matrices and corresponding target function values; and the extrinsic parameter matrix with the smallest target function value is selected as the target extrinsic parameter matrix.

[0115] Optionally, the device further includes: The calculation module is used to calculate the initial reprojection error of each frame based on the labeled data of each frame; to perform statistics on the initial reprojection errors of multiple frames to obtain the initial reprojection error statistical parameters; and to calculate the adaptive threshold of the robust kernel function in the objective optimization function based on the initial reprojection error statistical parameters.

[0116] Optionally, the calculation module is also used to calculate the total robust loss based on the robust loss of multiple robotic arms and the number of annotation points of each robotic arm in the multi-frame annotation data; based on the total robust loss, a variety of preset optimization algorithms are used to solve the objective optimization function to obtain multiple extrinsic parameter matrices and the corresponding objective function values.

[0117] The above-described device is used to execute the method provided in the foregoing embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0118] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more microprocessors, or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).

[0119] Figure 10 This is a schematic diagram of a control device provided in an embodiment of this application. This control device can be used for robot extrinsic parameter calibration. Figure 10 As shown, the control device includes: a processor 210, a storage medium 220, and a bus 230.

[0120] Storage medium 220 stores machine-readable instructions executable by processor 210. When the control device is running, processor 210 communicates with storage medium 220 via bus 230, and processor 210 executes the machine-readable instructions to perform the steps of the above method embodiment. The specific implementation and technical effects are similar and will not be described in detail here.

[0121] Optionally, this application also provides a storage medium 220, on which a computer program is stored. When the computer program is run by a processor, it executes the steps of the above-described method embodiments. The specific implementation and technical effects are similar, and will not be repeated here.

[0122] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0123] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0124] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0125] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0126] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A robot extrinsic parameter calibration method, characterized in that, The method includes: Acquire time-aligned multi-frame images and corresponding end-effector pose data. The acquired images are images captured by the camera on the robot of the robotic arm holding a calibration plate at the end. The end-effector pose data are the pose data of the end effector of the robotic arm. Based on the end pose data of each frame and the acquired image, determine the initial extrinsic parameter matrix of the camera relative to the robot's base coordinate system; Based on the end-effector pose data of each frame and the gripper opening of the end effector, determine the three-dimensional coordinates of the gripper tips on the end effector in the robot base coordinate system; Based on the camera's intrinsic parameter matrix and the camera's initial extrinsic parameter matrix, the three-dimensional coordinates of each gripper tip are projected onto the acquired image of the corresponding frame to obtain the projected two-dimensional coordinates of each gripper tip in the corresponding frame. The camera's extrinsic parameters are calibrated based on the frame index of the acquired images, the three-dimensional coordinates of the gripper tips of each corresponding frame, and the projected two-dimensional coordinates, to determine the target extrinsic parameter matrix.

2. The method of claim 1, wherein, The step of determining the three-dimensional coordinates of each gripper tip on the end effector in the robot base coordinate system based on the end effector pose data of each frame and the gripper opening of the end effector includes: Based on the end-effector pose data of each frame, the gripper opening of the end effector, and the fixed offset of the preset gripper center relative to the end effector coordinate system, calculate the three-dimensional coordinates of each gripper tip in the end effector coordinate system. The three-dimensional coordinates of each gripper tip in the end effector coordinate system are transformed to the robot base coordinate system to obtain the three-dimensional coordinates of each gripper tip in the robot base coordinate system.

3. The method of claim 1, wherein, The method further includes projecting the three-dimensional coordinates of each gripper tip onto the acquired image of the corresponding frame based on the camera's intrinsic parameter matrix and the camera's initial extrinsic parameter matrix, thereby obtaining the two-dimensional coordinates of the gripper tip in the corresponding frame's projection. Based on the camera's intrinsic parameter matrix, the camera's preset distortion coefficients, and the preset image size, the acquired images of each frame are subjected to distortion removal processing to obtain the distortion-removed images of each frame. The region of interest is cropped from the distortion-free image of each frame to obtain the corrected image of each frame; The step of projecting the three-dimensional coordinates of each gripper tip onto the acquired image of the corresponding frame based on the camera's intrinsic parameter matrix and the camera's initial extrinsic parameter matrix to obtain the projected two-dimensional coordinates of each gripper tip in the corresponding frame includes: Based on the camera's intrinsic parameter matrix and the camera's initial extrinsic parameter matrix, the three-dimensional coordinates of each gripper tip are projected onto the corrected image of the corresponding frame to obtain the projected two-dimensional coordinates of each gripper tip in the corresponding frame.

4. The method of claim 1, wherein, Before determining the target extrinsic parameter matrix by calibrating the camera's extrinsic parameters based on the frame index of multiple acquired images, the three-dimensional coordinates of the gripper tips of each corresponding frame, and the projected two-dimensional coordinates, the method further includes: The two-dimensional coordinates of the projection of the gripper tip in the corresponding frame are visualized and drawn in the acquired image of the corresponding frame. In response to a click operation on the tip of the target gripper in any frame of the acquired image, the two-dimensional coordinates of the projected target gripper tip in that frame are updated.

5. The method according to claim 1, characterized in that, The step of calibrating the camera's extrinsic parameters and determining the target extrinsic parameter matrix based on the frame index of multiple acquired images, the three-dimensional coordinates of the gripper tips of each corresponding frame, and the projected two-dimensional coordinates includes: The annotation data for the multiple frames is determined based on the frame index of the acquired images, the three-dimensional coordinates of the gripper tips of the corresponding frames, and the projected two-dimensional coordinates. Based on the Lupin kernel function, construct an objective optimization function with the goal of minimizing the reprojection error; Based on the labeled data from multiple frames, the target optimization function is solved to obtain the target extrinsic parameter matrix.

6. The method of claim 5, wherein, The step of solving the target optimization function based on multi-frame labeled data to obtain the target extrinsic parameter matrix includes: The robustness loss of the robotic arm is calculated based on the labeled data from multiple frames. Based on the robustness loss of the robotic arm, a variety of preset optimization algorithms are used to solve the objective optimization function, resulting in multiple extrinsic parameter matrices and corresponding objective function values. The extrinsic parameter matrix that minimizes the objective function value is selected as the objective extrinsic parameter matrix.

7. The method of claim 6, wherein, Before solving the objective optimization function using multiple preset optimization algorithms based on the robustness loss of the robotic arm to obtain multiple extrinsic parameter matrices and corresponding objective function values, the method further includes: Based on the labeled data of each frame, calculate the initial reprojection error of each frame. The initial reprojection error statistical parameters are obtained by statistically analyzing the initial reprojection error of multiple frames. Based on the initial reprojection error statistics parameters, calculate the adaptive threshold of the robust kernel function in the objective optimization function.

8. The method of claim 6, wherein, Before solving the objective optimization function using multiple preset optimization algorithms based on the robustness loss of the robotic arm to obtain multiple extrinsic parameter matrices and corresponding objective function values, the method includes: The total robustness loss is calculated based on the robustness loss of the multiple robotic arms and the number of annotation points of each robotic arm in the multi-frame annotation data. Based on the robustness loss of the robotic arm, multiple preset optimization algorithms are used to solve the objective optimization function, resulting in multiple extrinsic parameter matrices and corresponding objective function values, including: Based on the total robust loss, multiple preset optimization algorithms are used to solve the objective optimization function, resulting in multiple extrinsic parameter matrices and corresponding objective function values.

9. A robot extrinsic parameter calibration apparatus, characterized by, The device includes: The acquisition module is used to acquire time-aligned multi-frame images and corresponding end-effector pose data. The acquired images are images captured by the camera on the robot of the robotic arm holding a calibration plate at the end, and the end-effector pose data are the pose data of the end effector of the robotic arm. The determination module is used to determine the initial extrinsic parameter matrix of the camera relative to the robot base coordinate system based on the end pose data of each frame and the acquired image; The determining module is further configured to determine the three-dimensional coordinates of the tips of each gripper on the end effector in the robot base coordinate system based on the end pose data of each frame and the gripper opening of the end effector. The projection module is used to project the three-dimensional coordinates of the tips of each gripper onto the acquired image of the corresponding frame according to the intrinsic parameter matrix of the camera and the initial extrinsic parameter matrix of the camera, so as to obtain the projected two-dimensional coordinates of the tips of each gripper in the corresponding frame. The calibration module is used to calibrate the camera's extrinsic parameters based on the frame index of the acquired images, the three-dimensional coordinates of the gripper tips of the corresponding frames, and the projected two-dimensional coordinates, and to determine the target extrinsic parameter matrix.

10. A control device, characterized by include: The device includes a processor, a storage medium, and a bus. The storage medium stores program instructions executable by the processor. When the control device is running, the processor communicates with the storage medium via the bus, and the processor executes the program instructions to perform the steps of the robot extrinsic parameter calibration method as described in any one of claims 1 to 8.

Images