A dual-arm robot calibration method, device and equipment

Abstract

The double-arm robot calibration method provided in the application is suitable for a double-arm robot comprising a first mechanical arm and a second mechanical arm, the end of the first mechanical arm is provided with a camera, and the end of the second mechanical arm is fixed with at least one label with a mark point. The method comprises the following steps: acquiring a first type of image set and a second type of image set through the camera; for each image, identifying the label from the image, and determining the mark point position information of the mark point in a first base coordinate system of the first mechanical arm based on the identified label; obtaining a rotation matrix between the two base coordinate systems based on the first type of image set; obtaining a translation matrix between the two base coordinate systems based on the second type of image set, and then determining the conversion relationship from the first base coordinate system to the second base coordinate system. The double-arm robot calibration method, device and equipment provided in the application can quickly and accurately calibrate the pose transformation relationship between the two base coordinate systems in the double-arm robot system.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a calibration method, apparatus and equipment for a dual-arm robot. Background Technology

[0002] With the development of intelligent manufacturing, dual-arm robots, due to their advantages such as collaborative operation and flexibility, have been widely used in various fields including industrial manufacturing, medical assistance, education and scientific research, and service robots. Dual-arm robots often need to collaboratively complete precision tasks, such as material handling, bi-arm assembly, and cooperative tightening, which places high demands on the spatial coordination accuracy between the two robots. Therefore, accurately establishing the spatial pose transformation relationship between the base coordinate systems of dual-arm robots is a key prerequisite for achieving high-precision collaborative control and path planning.

[0003] In existing technologies, the calibration methods for dual-robot systems typically rely on manual teaching, where the end effectors of the two robotic arms are aligned or brought into contact manually to calculate the transformation relationship between their base coordinate systems. While the principle behind this method is relatively intuitive, the operation is cumbersome, susceptible to human error, and time-consuming, making it difficult to meet the dual requirements of efficiency and accuracy in industrial scenarios. Summary of the Invention

[0004] In view of this, this application provides a method, apparatus and device for calibrating a dual-arm robot, which can quickly and accurately calibrate the pose transformation relationship between two base coordinate systems in a dual-arm robot system.

[0005] Specifically, this application is implemented through the following technical solution: The first aspect of this application provides a calibration method for a dual-arm robot, the method being applied to a dual-arm robot comprising a first robotic arm and a second robotic arm, wherein a camera is mounted at the end of the first robotic arm, and at least one tag is fixed to the end of the second robotic arm, the at least one tag having marking points; the method includes: The camera acquires a first type of image set and a second type of image set. The first type of image set consists of at least three images obtained by taking pictures of at least one label when the end-effector pose of the first robotic arm remains unchanged and the end-effector pose of the second robotic arm remains unchanged but the position changes. The second type of image set consists of at least three images obtained by taking pictures of at least one label when the end-effector pose of the first robotic arm remains unchanged and the end-effector pose of the second robotic arm changes but the position changes. For each image, a label is identified from the image, and the position information of the marker point in the first base coordinate system of the first robotic arm is determined based on the identified label; Based on the end pose information of the second robotic arm corresponding to each image in the first type of image set, and the position information of the marker points corresponding to each image, the rotation matrix from the first base coordinate system to the second base coordinate system is determined. Based on the end pose information of the second robotic arm corresponding to each image in the second type of image set, and the position information of the marker points corresponding to each image, the translation matrix from the first base coordinate system to the second base coordinate system is determined. Based on the rotation and translation matrices, determine the transformation relationship from the first base coordinate system to the second base coordinate system.

[0006] A second aspect of this application provides a calibration device for a dual-arm robot. The device is applied to a dual-arm robot comprising a first robotic arm and a second robotic arm. A camera is mounted at the end of the first robotic arm, and at least one tag is fixed to the end of the second robotic arm. The at least one tag has marking points. The device includes an acquisition module, an identification module, and a processing module. The acquisition module is used to acquire a first type of image set and a second type of image set using a camera. The first type of image set consists of at least three images obtained by the camera taking pictures of at least one label while keeping the end-effector pose of the first robotic arm constant and the end-effector pose of the second robotic arm constant but changing position. The second type of image set consists of at least three images obtained by the camera taking pictures of at least one label while keeping the end-effector pose of the first robotic arm constant and the end-effector pose of the second robotic arm changing but changing position. The recognition module is used to identify labels from each image and determine the position information of the marker points in the first base coordinate system of the first robotic arm based on the identified labels. The processing module is used to determine the rotation matrix from the first base coordinate system to the second base coordinate system based on the end pose information of the second robotic arm corresponding to each image in the first type of image set and the position information of the marker points corresponding to each image. The processing module is used to determine the translation matrix from the first base coordinate system to the second base coordinate system based on the end pose information of the second robotic arm corresponding to each image in the second type of image set and the position information of the marker points corresponding to each image. The processing module is used to determine the transformation relationship from the first base coordinate system to the second base coordinate system based on the rotation matrix and the translation matrix.

[0007] A third aspect of this application provides a dual-arm robot calibration device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods provided in the first aspect of this application.

[0008] The dual-arm robot calibration method, apparatus, and equipment provided in this application are applicable to dual-arm robot systems comprising a first robotic arm and a second robotic arm, wherein a camera is mounted at the end of the first robotic arm and at least one label with marking points is fixed at the end of the second robotic arm. Furthermore, firstly, image data is acquired using a camera. The first image set consists of at least three images obtained by photographing at least one label while keeping the end-effector pose of the first robotic arm constant and the end-effector pose of the second robotic arm constant but changing position. The second image set consists of at least three images obtained by photographing at least one label while keeping the end-effector pose of the first robotic arm constant and changing position. Then, the label is identified from each image, and the position information of the marker point in the first base coordinate system of the first robotic arm is determined based on the identified label. Next, based on the end-effector pose information of the second robotic arm corresponding to each image in the first image set and the marker point position information corresponding to each image, a rotation matrix from the first base coordinate system to the second base coordinate system is determined. Then, based on the end-effector pose information of the second robotic arm corresponding to each image in the second image set and the marker point position information corresponding to each image, a translation matrix from the first base coordinate system to the second base coordinate system is determined. Finally, based on the rotation and translation matrices, the transformation relationship from the first base coordinate system to the second base coordinate system is determined. This method achieves the calibration process through the movement of the robotic arm and camera acquisition, eliminating the need for manual teaching or additional external measuring devices, thus avoiding the measurement difficulties caused by the closed and invisible base coordinates in traditional methods. By acquiring images in groups and solving the rotation and translation parameters in stages, both the algorithm complexity and the stability of the solution are reduced. Attached Figure Description

[0009] Figure 1 A flowchart of an embodiment of the dual-arm robot calibration method provided in this application;

[0010] Figure 2 This is a schematic diagram of the structure of Embodiment 1 of the dual-arm robot calibration device provided in this application;

[0011] Figure 3 This is a hardware structure diagram of the dual-arm robot calibration device provided in this application. Detailed Implementation

[0012] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0013] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0014] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0015] The following specific embodiments are given to illustrate the technical solution of this application in detail.

[0016] Figure 1 This is a flowchart illustrating an embodiment of the dual-arm robot calibration method provided in this application. The dual-arm robot calibration method provided in this embodiment is applied to a dual-arm robot comprising a first robotic arm and a second robotic arm. A camera is mounted at the end of the first robotic arm, and at least one tag is fixed to the end of the second robotic arm. The at least one tag has marking points. Please refer to... Figure 1 The dual-arm robot calibration method provided in this embodiment may include:

[0017] S101. Use a camera to acquire a first type of image set and a second type of image set; the first type of image set is at least three images obtained by the camera taking pictures of at least one label when the end pose of the first robotic arm remains unchanged and the end pose of the second robotic arm remains unchanged but the position changes; the second type of image set is at least three images obtained by the camera taking pictures of at least one label when the end pose of the first robotic arm remains unchanged and the end pose of the second robotic arm changes but the position changes.

[0018] It should be noted that the first and second robotic arms of the dual-arm robot are mounted on the same base, forming a compact dual-arm system with strong cooperative motion capabilities. Each robotic arm has multi-degree-of-freedom motion capabilities and is connected to the end effector via a flange structure at its end.

[0019] Furthermore, a camera is mounted at the end of the first robotic arm. This camera can be an industrial camera, a structured light camera, an RGB-D camera, or other vision sensors, used to acquire images of the label at the end of the second robotic arm during the calibration process. The camera is rigidly mounted on the flange at the end of the first robotic arm, and its posture changes with the movement of the first robotic arm, forming a flexible image acquisition system.

[0020] Furthermore, at least one label, which can be a QR code label, is fixed to the end of the second robotic arm. For example, in one possible implementation, a QR code label is fixed to the end of the second robotic arm, with the marking point being the center point of the QR code label.

[0021] It should be noted that, in some embodiments, the end effector of the second robotic arm can be fixed with multiple QR code tags to enhance the robustness and redundancy of visual recognition. The spatial relationship between the multiple QR code tags is known, and the marker point is the center point of a designated QR code tag among the multiple tags. Thus, even if some tags are obscured, the position of the marker point can be deduced from other visible tags, improving the reliability and flexibility of the system.

[0022] Specifically, in this step, a first type of image set and a second type of image set are acquired using a camera. When acquiring the first type of image set, the pose of the end effector of the first robotic arm must be kept constant; that is, the position and orientation of the camera mounted on the end effector of the first robotic arm must remain fixed in space. Under this premise, the end effector of the second robotic arm is moved, but its pose (or orientation) remains unchanged. At this time, the QR code label shifts in the camera's field of view as the position of the end effector of the second robotic arm changes, but its orientation remains unchanged. The camera at the end effector of the first robotic arm captures at least three images, forming the first type of image set.

[0023] Furthermore, when acquiring the second type of image set, while keeping the position of the camera at the end of the first robotic arm unchanged, the second robotic arm is controlled to change its end-effector orientation (orientation), but the end-effector position remains unchanged. At least three images are obtained by taking pictures of at least one tag using the camera, forming the second type of image set.

[0024] It should be noted that during image acquisition, it is recommended that the QR code label be kept within the camera's field of view as much as possible. Clear label information in the image helps improve the stability and accuracy of calibration. By adjusting the motion trajectory of the second robotic arm, image data covering different spatial positions and postures can be acquired, thereby enhancing data diversity and algorithm robustness. Even if the label is not fully visible in some images, the system still possesses a certain degree of fault tolerance.

[0025] In practice, for example, when acquiring the first type of image set, the pose of the first robotic arm and the pose of the second robotic arm are kept fixed. The position of the second robotic arm is changed, and the camera takes a picture of the QR code label, obtaining one image. Further, the above steps are repeated, with the second robotic arm changing position more than three times, obtaining at least three images to form the first type of image set. Similarly, when acquiring the second type of image set, the pose of the first robotic arm is kept fixed, and the pose of the second robotic arm is changed. The camera takes a picture of the QR code label, obtaining one image. Further, the above steps are repeated, with the second robotic arm changing position more than three times, obtaining at least three images to form the second type of image set.

[0026] S102. For each image, identify the label from the image, and determine the label position information of the marker point in the first base coordinate system of the first robotic arm based on the identified label.

[0027] Specifically, the original image is first processed using image processing techniques, such as grayscale conversion, noise reduction, and image enhancement, to improve the detectability of the labels. Next, an object detection algorithm is used to identify the labels in the image. As described above, labels can be QR codes, barcodes, graphic symbols of specific colors or shapes, or structured artificial visual markers (such as dot arrays or crosshairs).

[0028] Optionally, in one possible implementation, the label is a QR code label; the step of identifying the label from the image includes:

[0029] (1) Preprocess the image to obtain a preprocessed image; the preprocessing includes grayscale processing, noise reduction processing and enhancement processing.

[0030] Specifically, the image is first processed to grayscale. Then, median filtering or Gaussian filtering is used to denoise the image, removing random noise and slight blurring interference. Other denoising methods include bilateral filtering, non-local mean denoising, and guided filtering. The appropriate denoising algorithm can be flexibly selected based on the actual image characteristics. Finally, the image is enhanced using techniques such as image sharpening, contrast enhancement, histogram equalization, and adaptive histogram equalization to improve the edge contrast between the QR code and the background, thereby enhancing the recognizability of the QR code boundary. After these processes, a preprocessed image suitable for edge detection is obtained.

[0031] (2) Extract the outline of the QR code label based on the edge processing algorithm.

[0032] Specifically, the system employs edge processing algorithms to extract edges from the preprocessed image. For example, edge processing algorithms such as the Canny edge detection algorithm, Sobel operator, Prewitt operator, Laplacian operator, and Scharr operator are used to extract the contours of the QR code labels. In this embodiment, the Canny edge detection algorithm is used to identify edge regions with significant grayscale changes in the image through gradient calculation, thereby extracting the closed contours that may be the borders of the QR code labels. Furthermore, to reduce the false detection rate, preliminary screening can be performed by combining geometric features such as contour area, edge continuity, and rectangular approximation, retaining only contours with rectangular boundary features as candidate regions.

[0033] (3) Based on the extracted contour, filter out the QR code candidate region and perform perspective transformation correction on the QR code candidate region to identify the region where the QR code label is located.

[0034] Specifically, from the contours extracted in the previous step, contour regions with a side length ratio close to 1:1 and an angle close to 90°, or those approaching a rectangular shape within a certain tolerance range, are selected as candidate QR code regions. Subsequently, perspective transformation is performed on these candidate regions to eliminate viewpoint distortion and restore the QR code region under a standard viewpoint. Further, QR code decoding is performed on the corrected image region. If decoding is successful, the region is identified as a valid QR code label; if decoding fails, the system ignores the candidate region and attempts to identify other candidate regions in the image. If no region in the image successfully identifies a QR code label, the image is not included in the subsequent marker location information extraction process.

[0035] Furthermore, after identifying the label, the system extracts key feature points of the label, such as the center point, corner points, or predefined reference points. Then, based on the camera's intrinsic parameters (such as focal length, principal point position, and distortion coefficient) and the spatial transformation relationship between the camera and the first robotic arm, the pixel coordinates of the marker points are converted into three-dimensional spatial coordinates, and further transformed to the base coordinate system of the first robotic arm, thereby obtaining the position information of the marker points corresponding to the image in the first base coordinate system.

[0036] It should be noted that the specific implementation principle of determining the position information of the marker point in the first base coordinate system of the first robotic arm based on the identified label will be described in detail in the following embodiments, and will not be repeated here.

[0037] S103. Based on the end pose information of the second robotic arm corresponding to each image in the first type of image set, and the position information of the marker point corresponding to each image, determine the rotation matrix from the first base coordinate system to the second base coordinate system.

[0038] It should be noted that the end-effector pose information of the second robotic arm is used to characterize the position and orientation relationship between the end-effector coordinate system of the second robotic arm and the second base coordinate system. When acquiring each image, the position and orientation information of the end-effector coordinate system of the second robotic arm relative to the second base coordinate system of the second robotic arm can be directly read. For ease of distinction, the position and orientation information of the end-effector coordinate system of the second robotic arm relative to the second base coordinate system of the second robotic arm when acquiring each image will be recorded as the position and orientation information of the end-effector coordinate system of the second robotic arm relative to the second base coordinate system of the second robotic arm for each image.

[0039] Optionally, in one possible implementation, determining the rotation matrix from the first base coordinate system to the second base coordinate system based on the end-effector pose information of the second robotic arm corresponding to each image in the first type of image set, and the marker point position information corresponding to each image, may include:

[0040] (1) For each image, based on the pose transformation relationship from the end coordinate system of the second robotic arm to the second base coordinate system of the second robotic arm and the position information of the marker point corresponding to the image, construct the first type of coordinate transformation equation corresponding to the image; wherein, the first type of coordinate transformation equation is used to describe the correspondence between the position information of the marker point and the rotation relationship from the first base coordinate system to the second base coordinate system.

[0041] Specifically, in order to determine the first base coordinate system (denoted as...) ) and the second base coordinate system (denoted as The rotation relationship between ) is called the rotation matrix. The calibration is performed by constructing a coordinate mapping model based on multiple sets of observation points with consistent attitudes but different positions.

[0042] Referring to the preceding description, regarding the first From the first image, the position information of the corresponding marker point in the first base coordinate system can be obtained, as well as the position information of the second image in the first base coordinate system. The position and attitude information of the second robotic arm's end effector coordinate system to the second base coordinate system of the second robotic arm corresponding to the image are used to construct the first... The first type of coordinate transformation equation corresponding to the image:

[0043] ;

[0044] in, Let be the rotation matrix from the first base coordinate system to the second base coordinate system. Let be the translation matrix from the first base coordinate system to the second base coordinate system. This indicates the position information of point A in the coordinate system of the second robotic arm's end effector. For the first The position information of the marker points corresponding to each image in the first base coordinate system. For the first The image corresponds to the attitude information from the end-effector coordinate system of the second robotic arm to the second base coordinate system of the second robotic arm. For the first The image corresponds to the position information of the second robotic arm's end-effector coordinate system to the second base coordinate system of the second robotic arm.

[0045] As can be understood from the preceding description, Take 1 to 3.

[0046] (2) Transform the first type of coordinate transformation equations corresponding to all images in the first type of image set into a first type of linear equation system with respect to the rotation matrix.

[0047] Furthermore, a difference process is performed on all first-kind coordinate transformation equations, that is, two images (the first and second images) are selected. Zhang Yudi By subtracting the first-type coordinate transformation equations corresponding to the two images (e.g., Zhang) respectively, we obtain the difference expression:

[0048] ;

[0049] Separately =1 and =2, the first kind of linear equation system is:

[0050] ;

[0051] ;

[0052] (3) Solve the first type of linear equations to obtain the rotation matrix.

[0053] As described above, it is understandable that the first type of linear equation system, Rx = y, can be solved using the Umeyama algorithm to obtain the rotation matrix. .

[0054] Through the above fitting process, the rotation matrix from the first base coordinate system to the second base coordinate system can be obtained, thereby completing the accurate calibration of the attitude between the two coordinate systems.

[0055] S104. Based on the end pose information of the second robotic arm corresponding to each image in the second type of image set, and the position information of the marker point corresponding to each image, determine the translation matrix from the first base coordinate system to the second base coordinate system.

[0056] Optionally, in one possible implementation, determining the translation matrix from the first base coordinate system to the second base coordinate system based on the end-effector pose information of the second robotic arm corresponding to each image in the second type of image set, and the marker point position information corresponding to each image, may include:

[0057] (1) For each image, based on the pose transformation relationship from the end coordinate system of the second robotic arm to the second base coordinate system of the second robotic arm and the position information of the marker point corresponding to the image, construct the second type of coordinate transformation equation corresponding to the image; wherein, the second type of coordinate transformation equation is used to describe the correspondence between the position information of the marker point and the translation relationship between the first base coordinate system and the second base coordinate system.

[0058] Specifically, as described above, for each image in the second type of image set... This allows us to obtain the position information of marker point A corresponding to the image in the first base coordinate system. and the The position information of the second robotic arm's end effector relative to the second coordinate system corresponding to the image. and attitude information Combining this data, construct the first... The second type of coordinate transformation equation corresponding to the image:

[0059]

[0060] in, Let be the rotation matrix from the first base coordinate system to the second base coordinate system. For the first The location information of the marker points corresponding to each image. Let be the translation matrix from the first base coordinate system to the second base coordinate system. This refers to the position information of the marker point in the end-effector coordinate system of the second robotic arm; Take 1 to 3.

[0061] (2) Transform the second type of coordinate transformation equations corresponding to all images in the second type of image set into a system of second type linear equations about the translation matrix.

[0062] Specifically, having already obtained the rotation matrix based on the first type of image set, the second type of coordinate transformation equations corresponding to all images in the second type of image set are processed. This is to eliminate unknowns. The second type of image set contains two adjacent images. and Taking the difference between the equations, we obtain the difference equation:

[0063] ;

[0064] When acquiring three images, respectively set... =1 and =2, thus obtaining the second type of linear equation system:

[0065] ;

[0066] ;

[0067] As can be understood from the preceding description, the second type of linear equation system is: The problem is to solve the second type of linear equations numerically using methods such as least squares, matrix pseudo-inverse (Moore-Penrose inverse), QR decomposition, or singular value decomposition (SVD) given the rotation matrix, to obtain the position information of the marker point in the coordinate system of the second robotic arm end effector. .

[0068] Furthermore, the rotation matrix and the position information of the marked points in the coordinate system of the second robotic arm's end effector are... Substituting into the equation of the second kind of coordinate transformation, we can obtain the translation matrix from the first base coordinate system to the second base coordinate system. .

[0069] (3) Solve the second type of linear equations based on the rotation matrix to obtain the translation matrix.

[0070] Specifically, the obtained Substituting any one of the original equations, we obtain the translation matrix. This allows us to determine the translation matrix from the first base coordinate system to the second base coordinate system. Specifically, the solution for the translation matrix can be transformed into the following formula:

[0071] ;

[0072] Furthermore, to improve accuracy, the above results can be calculated for all images and the average taken. In another implementation, the average can be... and As joint optimization variables, a nonlinear least squares problem is constructed based on the original equation, and the overall residual is minimized using the Gauss-Newton method or the Levenberg-Marquardt algorithm, thereby obtaining results with higher accuracy.

[0073] Referring to the preceding description, it can be understood that, in one possible implementation, solving the second type of linear equations based on the rotation matrix to obtain the translation matrix may include:

[0074] The rotation-related parameter terms in the second type of linear equation system are determined based on the rotation matrix, and the second type of linear equation system is solved using the least squares method to obtain the translation matrix.

[0075] Specifically, the position information of marker point A in the coordinate system of the second robotic arm end effector is obtained by solving the least squares method. Then obtain Substituting the second type of coordinate transformation equation corresponding to any image, the translation matrix is ​​calculated. .

[0076] S105. Determine the transformation relationship from the first base coordinate system to the second base coordinate system based on the rotation matrix and translation matrix.

[0077] Specifically, based on the obtained rotation matrix Translation matrix This allows us to determine the transformation relationship from the first base coordinate system to the second base coordinate system. This transformation relationship can be represented as a homogeneous transformation matrix, as follows:

[0078] ;

[0079] in, For rotation matrix, This is a translation matrix. Using this transformation relationship, the coordinates of any point in the first base coordinate system can be accurately converted to coordinates in the second base coordinate system, thus achieving robot calibration.

[0080] The method provided in this embodiment combines multi-angle images acquired by the end-effector camera of a dual-arm robot with marker point position data, and utilizes multi-view data fusion and mathematical optimization algorithms to achieve high-precision calibration of rotation and translation relationships between base coordinate systems, which has the following significant advantages:

[0081] (1) Reduce manual intervention and errors: The calibration process is completed by using automated visual label recognition technology and mathematical optimization algorithm, which avoids the tedious operation and human error caused by traditional manual teaching, making the calibration process more stable and reliable. At the same time, it reduces the dependence on professional operators and improves the ease of use of the system.

[0082] (2) Improve calibration accuracy: This method constructs coordinate transformation equations based on multiple images and adopts a joint solution strategy of rotation matrix and translation matrix to accurately calculate the spatial transformation relationship between the two robot arm base coordinate systems, effectively reducing error accumulation, ensuring the accuracy and stability of robot cooperative motion, and meeting the high requirements for positioning accuracy in complex industrial environments.

[0083] (3) Improve calibration efficiency: This method supports the rapid acquisition of multiple sets of image data from the first type of image set and the second type of image set, and achieves rapid solution by establishing an efficient mathematical model, thereby realizing a high degree of automation and intelligence in the calibration process, greatly shortening the calibration time, suitable for the real-time calibration needs of the production site, and improving the overall production efficiency.

[0084] (4) Strong adaptability: This method flexibly uses a variety of numerical solutions and optimization strategies, which can adapt to different robot arm posture changes and complex environmental conditions, ensuring the stability and robustness of calibration results under various working conditions, and effectively coping with the ever-changing actual application environment in industrial sites.

[0085] (5) Easy system integration: Relying on the camera and QR code label at the end of the robot body, it does not need to rely on expensive external measurement equipment, realizes seamless integration of visual and motion data, reduces system integration cost, simplifies hardware layout, and is easy to promote and apply in various dual-arm robot systems, and has good engineering practical value.

[0086] The dual-arm robot calibration method, apparatus, and device provided in this application are applicable to dual-arm robot systems comprising a first robotic arm and a second robotic arm. The first robotic arm has a camera mounted at its end cap, and the second robotic arm has at least one tag with marked points fixed at its end cap. Further, image data is first acquired using the camera. A first set of images consists of at least three images obtained by photographing at least one tag while keeping the end cap pose of the first robotic arm constant and the end cap pose of the second robotic arm constant but changing position. A second set of images consists of at least three images obtained by photographing at least one tag while keeping the end cap pose of the first robotic arm constant and the end cap pose of the second robotic arm changing position. Then, each image... The method involves identifying labels in images and determining the position information of marker points in the first coordinate system of the first robotic arm based on the identified labels, thus obtaining the marker point position information corresponding to that image. Next, based on the end-effector pose information of the second robotic arm corresponding to each image in the first image set and the marker point position information for each image, a rotation matrix from the first coordinate system to the second coordinate system is determined. Then, based on the end-effector pose information of the second robotic arm corresponding to each image in the second image set and the marker point position information for each image, a translation matrix from the first coordinate system to the second coordinate system is determined. Finally, based on the rotation and translation matrices, the transformation relationship from the first coordinate system to the second coordinate system is determined. This method achieves the calibration process through the robotic arm's own motion and camera image acquisition, eliminating the need for manual teaching or additional external measuring devices, thus avoiding the measurement difficulties caused by the closed and invisible base coordinates in traditional methods. By grouping images for acquisition and solving the rotation and translation parameters in stages, both the algorithm complexity and the solution stability are reduced.

[0087] In a preferred implementation, the end effector of the second robotic arm is fixed with multiple QR code labels, and the marker point is the center point of a specific QR code label among the multiple QR code labels; the step of identifying the label from the image and determining the marker point position information in the first base coordinate system of the first robotic arm based on the identified label may include:

[0088] (1) Identify all visible QR code labels in the image and obtain the location information of each visible QR code label in the image.

[0089] Specifically, image recognition technology is used to process the acquired images to obtain the two-dimensional pixel coordinates of each visible QR code label in the image. This provides the basic data for subsequent calculations.

[0090] (2) If the currently identified visible QR code label contains the specified QR code label, then based on the position information of the specified QR code label in the image, as well as the pre-calibrated camera intrinsic parameters and the transformation relationship between the pre-calibrated camera coordinate system and the first base coordinate system of the first robotic arm, the position information of the marker point in the first base coordinate system of the first robotic arm is determined.

[0091] Specifically, if the currently identified visible QR code label contains the specified QR code label, then based on the position information of the specified QR code label in the image, combined with the pre-calibrated camera intrinsic parameter matrix K and the pre-calibrated pose transformation matrix from the camera coordinate system to the first base coordinate system of the first robotic arm, Determine the three-dimensional position information of the marker point in the first base coordinate system of the first robotic arm. The calculation process can be expressed as follows:

[0092] ;

[0093] in, Z represents the pixel coordinates of the center point of the specified QR code label, Z is the depth value of that point in the camera coordinate system, and K is the camera intrinsic parameter matrix. It is the inverse of the camera intrinsic parameter matrix. Let be the transformation matrix from the camera coordinate system to the first base coordinate system of the first robotic arm. This refers to the position information of point A in the first base coordinate system of the first robotic arm.

[0094] (3) If the currently identified visible QR code label does not contain the specified QR code label, then the pixel coordinates of the marker point are determined based on the position information of the currently identified visible QR code label in the image and the known positional relationship between the visible QR code label and the marker point.

[0095] Specifically, if the currently identified visible QR code label does not contain the specified QR code label, then the pixel coordinates of the currently identified visible QR code label will be used. Based on the actual position of the marker on the calibration board and the previously known fixed geometric positional relationship between the visible QR code label and the marker point, the pixel coordinates of the marker point are calculated. This geometric positional relationship can be obtained through precise measurement before system installation and stored in the calibration parameter library.

[0096] (4) Based on the pixel coordinates of the marker point, the pre-calibrated camera intrinsic parameters, and the transformation relationship between the pre-calibrated camera coordinate system and the first base coordinate system of the first robotic arm, determine the marker point position information in the first base coordinate system of the first robotic arm.

[0097] Specifically, based on the pixel coordinates of the marker points obtained in step (3) The depth value Z, the camera intrinsic parameter matrix K, and the transformation matrix from the camera coordinate system to the first base coordinate system of the first robotic arm. Using the same calculation formula as in step (2), the three-dimensional position information of marker point A in the first base coordinate system of the first robotic arm is finally obtained. In summary, even if the target QR code is obscured or unrecognized, the location of the marker point can still be deduced based on other visible QR code labels, thus ensuring the continuity and robustness of the calibration process.

[0098] In the preferred implementation described above, by fixing multiple QR code labels to the end of the second robotic arm and designating the center point of one of the QR codes as a marker point, even if the designated QR code is partially obscured or unrecognized from certain viewpoints, the pixel coordinates of the marker point can be indirectly calculated using the fixed geometric relationship between other visible QR code labels and the marker point. In this way, the system can complete marker point localization without relying on a single label, effectively avoiding calibration interruptions caused by occlusion, lighting conditions, or recognition failures. Furthermore, this method ensures the continuity and robustness of the calibration process while improving the success rate and reliability of data acquisition, thereby reducing manual intervention in practical applications and improving the automated calibration efficiency and accuracy of the dual-arm robot system.

[0099] Optionally, in one possible implementation, a QR code label is fixed to the end of the second robotic arm, and the marker point is the center point of the QR code label; the step of identifying the label from the image and determining the marker point position information in the first base coordinate system of the first robotic arm based on the identified label may include:

[0100] (1) Use image recognition technology to identify the QR code label from the image and obtain the location information of the QR code label in the image.

[0101] Specifically, the system uses visual marker detection algorithms (such as ArUco and AprilTag) to identify individual QR code tags in the images captured by the camera, and outputs the image pixel information of the QR code tag. The output includes the pixel coordinates of the four corner points of the QR code. , , , .

[0102] (2) Based on the position information of the QR code label in the image, determine the position information of the center point of the QR code label, and determine the position information as the pixel coordinates of the marker point.

[0103] Specifically, based on the obtained pixel coordinates of the four corner points , , , The geometric center method is used to calculate the pixel coordinates of the center point of the QR code as the pixel coordinates of the marker point. The calculation formula is as follows:

[0104] , ;

[0105] in, The pixel coordinates of the center of the QR code (marker point A) ( ) represents the pixel coordinates of the four corner points identified in step (1).

[0106] It should be noted that if the detection algorithm has already directly returned the pixel coordinates of the label center, the return value can also be used directly. .

[0107] (3) Determine the three-dimensional coordinates of the marker point in the camera coordinate system based on the pixel coordinates of the marker point and the pre-calibrated camera intrinsic parameters.

[0108] It should be noted that the calculation of the 3D coordinates of the marker points can be selected based on different scene requirements and sensor conditions. When the system is equipped with a depth camera and can directly acquire the depth information of the corresponding pixels, the 3D coordinates of the marker points can be calculated using a back-projection calculation method based on the depth values. When the system only uses a regular RGB camera and cannot directly acquire depth values, the 3D coordinates of the marker points can be calculated using a 3D coordinate calculation method that solves for known physical dimensions or known geometric constraints.

[0109] Specifically, in the first method, based on monocular PnP (Programmable Noise Proof-of-Practice), it is recommended when the physical dimensions of the QR code label or the 3D coordinates of its corners are known. Specifically, the known 3D coordinates of the QR code label's physical corners and their corresponding pixel corners are first used to solve for the QR code label's pose relative to the camera using PnP (e.g., OpenCV's solvePnP), thus obtaining the rotation vector. Translation vector If the origin is defined as the center point of the QR code label in the coordinate system, then the coordinates of the center of the QR code label in the camera coordinate system are directly the translation vector:

[0110] ;

[0111] in, Let A be the three-dimensional coordinates of point A in the camera coordinate system. This is the translation vector from the label coordinate system to the camera coordinate system obtained by solvePnP.

[0112] Furthermore, in the second approach, based on pixel depth or a depth camera, when the system can directly obtain the pixel depth Z (e.g., an RGB-D sensor), the pixel coordinates can be back-projected into three-dimensional coordinates in the camera coordinate system using the camera intrinsic parameter matrix K.

[0113] ;

[0114] ;

[0115] in, For the camera intrinsic parameter matrix, , Focal length (in pixels). Principal point coordinates for The inverse matrix, For pixels This corresponds to the depth value (units consistent with the camera coordinate system). The calculated 3D coordinate vector of the camera system.

[0116] (4) Based on the three-dimensional coordinates of the marker point in the camera coordinate system and the transformation relationship between the pre-calibrated camera coordinate system and the first base coordinate system of the first robotic arm, determine the marker point position information in the first base coordinate system of the first robotic arm.

[0117] Specifically, the homogeneous transformation matrix from the pre-calibrated camera coordinate system to the first base coordinate system is used. Perform homogeneous coordinate transformation:

[0118] ;

[0119] ;

[0120] in, for The homogeneous transformation matrix describes the pose and position relationship of the camera coordinate system relative to the first base coordinate system. Let be the rotation matrix from the camera coordinate system to the first base coordinate system. This provides the position information of the camera origin in the first base coordinate system. To mark the three-dimensional coordinates of the point in the camera frame, The final three-dimensional coordinates of the marked point A in the first base coordinate system.

[0121] The method for obtaining marker point location information provided in this embodiment simplifies the system's hardware configuration and reduces installation and maintenance costs by fixing only a QR code label to the end of the second robotic arm and using the center point of the QR code as the marker point. In this implementation, the system can quickly identify the label using a mature QR code detection algorithm and directly determine the pixel coordinates of the marker point through the geometric center method or the algorithm's return value; the calculation process is simple and efficient. Furthermore, by combining different depth acquisition methods in different scenarios (such as direct measurement by a depth camera or calculation from known physical dimensions based on the PnP method), this scheme can flexibly adapt to different types of vision sensors, ensuring stable calculation of the marker point's three-dimensional coordinates. Finally, by using it in conjunction with pre-calibrated camera extrinsic parameters, the marker point position can be accurately mapped to the first base coordinate system, thereby achieving high-precision calibration of the dual-arm robot system. In summary, this method reduces hardware complexity while still ensuring positioning accuracy and adaptability, possessing technical advantages such as low implementation cost, flexible deployment, and reliable accuracy.

[0122] Corresponding to the aforementioned embodiment of a dual-arm robot calibration method, this application also provides an embodiment of a dual-arm robot calibration device.

[0123] Figure 2 This is a schematic diagram of the structure of Embodiment 1 of the dual-arm robot calibration device provided in this application. Please refer to... Figure 2 The device provided in this embodiment is applied to a dual-arm robot including a first robotic arm and a second robotic arm. A camera is mounted at the end of the first robotic arm, and at least one tag is fixed to the end of the second robotic arm, with marking points on the tag. The device includes an acquisition module 201, an identification module 202, and a processing module 203.

[0124] The acquisition module 201 is used to acquire a first type of image set and a second type of image set using a camera. The first type of image set consists of at least three images obtained by taking pictures of at least one label when the end-effector pose of the first robotic arm remains unchanged and the end-effector pose of the second robotic arm remains unchanged but the position changes. The second type of image set consists of at least three images obtained by taking pictures of at least one label when the end-effector pose of the first robotic arm remains unchanged and the end-effector pose of the second robotic arm changes but the position changes.

[0125] The recognition module 202 is used to identify a label from each image and determine the position information of the marker point in the first base coordinate system of the first robotic arm based on the identified label.

[0126] The processing module 203 is used to determine the rotation matrix from the first base coordinate system to the second base coordinate system based on the end pose information of the second robotic arm corresponding to each image in the first type of image set and the position information of the marker point corresponding to each image.

[0127] The processing module 203 is used to determine the translation matrix from the first base coordinate system to the second base coordinate system based on the end pose information of the second robotic arm corresponding to each image in the second type of image set and the position information of the marker points corresponding to each image.

[0128] The processing module 203 is used to determine the transformation relationship from the first base coordinate system to the second base coordinate system based on the rotation matrix and the translation matrix.

[0129] The apparatus of this embodiment can be used to perform... Figure 1 The steps of the method embodiment shown are similar in principle and process, and will not be repeated here.

[0130] Please continue to refer to Figure 3 This application also provides a dual-arm robot calibration device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods provided in the first aspect of this application.

[0131] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0132] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0133] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A calibration method for a dual-arm robot, characterized in that, The method is applied to a dual-arm robot comprising a first robotic arm and a second robotic arm, wherein a camera is mounted at the end of the first robotic arm, and at least one tag is fixed to the end of the second robotic arm, and the at least one tag has marking points set on it; the method includes: The camera acquires a first type of image set and a second type of image set. The first type of image set consists of at least three images obtained by taking pictures of at least one label when the end-effector pose of the first robotic arm remains unchanged and the end-effector pose of the second robotic arm remains unchanged but the position changes. The second type of image set consists of at least three images obtained by taking pictures of at least one label when the end-effector pose of the first robotic arm remains unchanged and the end-effector pose of the second robotic arm changes but the position changes. For each image, a label is identified from the image, and the position information of the marker point in the first base coordinate system of the first robotic arm is determined based on the identified label; Based on the end pose information of the second robotic arm corresponding to each image in the first type of image set, and the position information of the marker points corresponding to each image, the rotation matrix from the first base coordinate system to the second base coordinate system is determined. Based on the end pose information of the second robotic arm corresponding to each image in the second type of image set, and the position information of the marker points corresponding to each image, the translation matrix from the first base coordinate system to the second base coordinate system is determined. Based on the rotation and translation matrices, determine the transformation relationship from the first base coordinate system to the second base coordinate system; The step of determining the rotation matrix from the first base coordinate system to the second base coordinate system includes: Based on the end pose information and marker point position information of the second robotic arm corresponding to different images in the first type of image set, the corresponding first type of coordinate transformation equations are constructed, and the first type of coordinate transformation equations corresponding to different images are subjected to difference processing to obtain the first type of linear equation system about the rotation matrix. Solve the first type of linear equations to obtain the rotation matrix; Determining the translation matrix from the first base coordinate system to the second base coordinate system includes: Given the rotation matrix, based on the end pose information and marker position information of the second robotic arm corresponding to different images in the second type of image set, the corresponding second type of coordinate transformation equations are constructed, and the second type of coordinate transformation equations corresponding to different images are subjected to differential processing to obtain the second type of linear equation system. Solve the second type of linear equations based on the rotation matrix to obtain the position information of the marker point in the coordinate system of the second robotic arm end effector. The translation matrix is ​​determined based on the rotation matrix and the position information of the marked point in the coordinate system of the second robotic arm end.

2. The method according to claim 1, characterized in that, The end of the second robotic arm is fixed with multiple QR code labels, and the marking point is the center point of a designated QR code label among the multiple QR code labels; The step of identifying the label from the image and determining the position information of the marker point in the first base coordinate system of the first robotic arm based on the identified label includes: Identify all visible QR code labels in the image and obtain the location information of each visible QR code label in the image; If the currently identified visible QR code label contains the specified QR code label, then based on the position information of the specified QR code label in the image, as well as the pre-calibrated camera intrinsic parameters and the transformation relationship between the pre-calibrated camera coordinate system and the first base coordinate system of the first robotic arm, the position information of the marker point in the first base coordinate system of the first robotic arm is determined. If the currently identified visible QR code label does not contain the specified QR code label, then the pixel coordinates of the marker point are determined based on the position information of the currently identified visible QR code label in the image and the known positional relationship between the visible QR code label and the marker point. Based on the pixel coordinates of the marker point, the pre-calibrated camera intrinsic parameters, and the transformation relationship between the pre-calibrated camera coordinate system and the first base coordinate system of the first robotic arm, the marker point position information in the first base coordinate system of the first robotic arm is determined.

3. The method according to claim 1, characterized in that, A QR code label is fixed to the end of the second robotic arm, and the marker point is the center point of the QR code label; the step of identifying the label from the image and determining the marker point position information in the first base coordinate system of the first robotic arm based on the identified label includes: The QR code label is identified from the image using image recognition technology to obtain the location information of the QR code label in the image; Based on the position information of the QR code label in the image, the position information of the center point of the QR code label is determined, and the position information is determined as the pixel coordinates of the marker point; Based on the pixel coordinates of the marker point and the pre-calibrated camera intrinsic parameters, determine the three-dimensional coordinates of the marker point in the camera coordinate system; Based on the three-dimensional coordinates of the marker point in the camera coordinate system and the pre-calibrated transformation relationship between the camera coordinate system and the first base coordinate system of the first robotic arm, the position information of the marker point in the first base coordinate system of the first robotic arm is determined.

4. The method according to claim 1, characterized in that, Based on the end-effector pose information of the second robotic arm corresponding to each image in the first image set, and the marker point position information corresponding to each image, the rotation matrix from the first base coordinate system to the second base coordinate system is determined, including: For each image, based on the pose transformation relationship from the end coordinate system of the second robotic arm to the second base coordinate system of the second robotic arm corresponding to the image, and the position information of the marker point corresponding to the image, a first type of coordinate transformation equation is constructed for the image; wherein, the first type of coordinate transformation equation is used to describe the correspondence between the position information of the marker point and the rotation relationship from the first base coordinate system to the second base coordinate system; Transform the first type of coordinate transformation equations corresponding to all images in the first type of image set into a system of first type linear equations about the rotation matrix; Solve the first type of linear equations to obtain the rotation matrix.

5. The method according to claim 1, characterized in that, Based on the end-effector pose information of the second robotic arm corresponding to each image in the second type of image set, and the marker point position information corresponding to each image, the translation matrix from the first base coordinate system to the second base coordinate system is determined, including: For each image, based on the pose transformation relationship from the end coordinate system of the second robotic arm to the second base coordinate system of the second robotic arm corresponding to the image, and the position information of the marker points corresponding to the image, a second type of coordinate transformation equation is constructed for the image; wherein, the second type of coordinate transformation equation is used to describe the correspondence between the position information of the marker points and the translation relationship between the first base coordinate system and the second base coordinate system; Transform the second type of coordinate transformation equations corresponding to all images in the second type of image set into a system of second type linear equations about the translation matrix; Solving the second type of linear equations based on the rotation matrix yields the translation matrix.

6. The method according to claim 1, characterized in that, The label is a QR code label; the process of identifying the label from the image includes: The image is preprocessed to obtain a preprocessed image; the preprocessing includes grayscale processing, noise reduction processing, and enhancement processing. The outline of the QR code label is extracted based on the edge processing algorithm; Based on the extracted contours, candidate regions for QR codes are selected, and perspective transformation correction is performed on the candidate regions to identify the area where the QR code label is located.

7. The method according to claim 5, characterized in that, The step of solving the second type of linear equations based on the rotation matrix to obtain the translation matrix includes: The rotation-related parameter terms in the second type of linear equation system are determined based on the rotation matrix, and the second type of linear equation system is solved using the least squares method to obtain the translation matrix.

8. A calibration device for a dual-arm robot, characterized in that, The device is applied to a dual-arm robot comprising a first robotic arm and a second robotic arm. A camera is mounted at the end of the first robotic arm, and at least one QR code label is fixed to the end of the second robotic arm. The QR code label has marking points. The device includes an acquisition module, an identification module, and a processing module. The acquisition module is used to acquire a first type of image set and a second type of image set using a camera. The first type of image set consists of at least three images obtained by the camera taking pictures of at least one QR code label while keeping the end-effector pose of the first robotic arm constant and the end-effector pose of the second robotic arm constant but changing position. The second type of image set consists of at least three images obtained by the camera taking pictures of at least one QR code label while keeping the end-effector pose of the first robotic arm constant and the end-effector pose of the second robotic arm changing but changing position. The recognition module is used to identify a QR code label from each image and determine the position information of the marker point in the first base coordinate system of the first robotic arm based on the identified QR code label. The processing module is used to determine the rotation matrix from the first base coordinate system to the second base coordinate system based on the end pose information of the second robotic arm corresponding to each image in the first type of image set and the position information of the marker points corresponding to each image. The processing module is used to determine the translation matrix from the first base coordinate system to the second base coordinate system based on the end pose information of the second robotic arm corresponding to each image in the second type of image set and the position information of the marker points corresponding to each image. The processing module is used to determine the transformation relationship from the first base coordinate system to the second base coordinate system based on the rotation matrix and the translation matrix; The step of determining the rotation matrix from the first base coordinate system to the second base coordinate system includes: Based on the end pose information and marker point position information of the second robotic arm corresponding to different images in the first type of image set, the corresponding first type of coordinate transformation equations are constructed, and the first type of coordinate transformation equations corresponding to different images are subjected to difference processing to obtain the first type of linear equation system about the rotation matrix. Solve the first type of linear equations to obtain the rotation matrix; Determining the translation matrix from the first base coordinate system to the second base coordinate system includes: Given the rotation matrix, based on the end pose information and marker position information of the second robotic arm corresponding to different images in the second type of image set, the corresponding second type of coordinate transformation equations are constructed, and the second type of coordinate transformation equations corresponding to different images are subjected to differential processing to obtain the second type of linear equation system. Solve the second type of linear equations based on the rotation matrix to obtain the position information of the marker point in the coordinate system of the second robotic arm end effector. The translation matrix is ​​determined based on the rotation matrix and the position information of the marked point in the coordinate system of the second robotic arm end.

9. The apparatus according to claim 8, characterized in that, The identification module is specifically used for: The image is preprocessed to obtain a preprocessed image; the preprocessing includes grayscale processing, noise reduction processing, and enhancement processing. The outline of the QR code label is extracted based on the edge processing algorithm; Based on the extracted contours, candidate regions for QR codes are selected, and perspective transformation correction is performed on the candidate regions to identify the area where the QR code label is located.

10. A calibration device for a dual-arm robot, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the method according to any one of claims 1-7.

Images