Hand-eye calibration method, device, equipment and storage medium for robotic arm

Abstract

The present invention discloses a method, device, equipment and storage medium for hand-eye calibration of a robotic arm. The method includes: fixing a calibration plate at the end of the robotic arm, controlling the robotic arm to move along a preset trajectory so that the calibration plate is always in the field of view of the camera; collecting the calibration plate image under at least three different postures of the robotic arm, and recording the position and posture of the calibration plate at the end of the robotic arm in the robotic arm coordinate system; calculating the position and posture of the calibration plate in the camera coordinate system through the calibration plate corner points in the calibration plate image; calculating the rotation matrix and translation vector between the robotic arm coordinate system and the camera coordinate system based on the corresponding position and posture of the calibration plate in the robotic arm coordinate system and the position and posture in the camera coordinate system to complete the hand-eye calibration. The technical solution of the present application realizes high-precision hand-eye calibration and significantly improves the stability of robot automation applications.

Description

Technical Field

[0001] The present invention relates to the technical field of robot control, and in particular to a hand-eye calibration method, device, equipment and storage medium for a robotic arm. Background Art

[0002] At present, the application of robotic automation systems in industries such as industry, logistics, and services is becoming increasingly widespread. As the core technology for realizing robot vision guidance and precise operation, the main purpose of robot hand-eye calibration is to establish an accurate mapping relationship between the camera coordinate system and the robot coordinate system. Traditional hand-eye calibration methods usually use a camera fixed to the head of the robot and a calibration plate fixed to the end of the robotic arm. The calibration plate image is collected through a preset motion trajectory, and the image processing algorithm is used to calculate the position of the calibration plate in the camera coordinate system, and then the rigid transformation parameters between the camera and robot coordinate systems are solved. However, due to the dynamic deviations in the movement of the robotic arm, the camera internal parameter errors, and the influence of environmental factors such as lighting and noise, traditional methods often have problems with low measurement accuracy and insufficient robustness in practical applications, which limits the application of robots in high-precision operations. How to properly solve the above problems has become a topic that needs to be urgently addressed in the industry. Summary of the Invention

[0003] The present invention provides a hand-eye calibration method, device, equipment and storage medium for a robotic arm, which are used to fix a calibration plate and multimodal acquisition at the end of the arm, thereby improving the accuracy of hand-eye calibration and enhancing the stability of robot operation.

[0004] According to a first aspect of the present invention, a hand-eye calibration method for a robotic arm is provided. The hand-eye calibration method for a robotic arm is applied to a robot with a robotic arm, comprising:

[0005] Fixing a calibration plate at the end of the robotic arm and controlling the robotic arm to move along a preset trajectory so that the calibration plate is always within the field of view of the camera;

[0006] Capturing the calibration plate image in at least three different postures of the robotic arm, and simultaneously recording the position and posture of the calibration plate at the end of the robotic arm in the robotic arm coordinate system;

[0007] Calculating the position and orientation of the calibration plate in the camera coordinate system using the calibration plate corner points in the calibration plate image;

[0008] According to the position and posture of the corresponding calibration plate in the robot coordinate system and the camera coordinate system, the rotation matrix and translation vector between the robot coordinate system and the camera coordinate system are calculated to complete the hand-eye calibration.

[0009] In one embodiment, it further includes:

[0010] Acquire the color image and depth image of the calibration plate simultaneously by a depth camera;

[0011] The three-dimensional position and orientation of the calibration plate in the camera coordinate system is calculated according to the three-dimensional coordinates of the corner points of the calibration plate in the calibration plate image with depth information.

[0012] In one embodiment, it further includes:

[0013] Presetting a plurality of positioning marks with different geometric shapes and optical features on the calibration plate to form a multimodal calibration feature together with the pattern of the calibration plate;

[0014] Extracting multimodal calibration features from the calibration plate image captured by the camera, the multimodal calibration features including the calibration plate corner points and the positioning marks;

[0015] The position and orientation of the calibration plate in the camera coordinate system are calculated based on the multimodal calibration features.

[0016] In one embodiment, it further includes:

[0017] Analyze the correction rotation angle and displacement compensation parameters based on the systematic deviations caused by camera hardware, ambient lighting, and robotic arm status;

[0018] The rotation matrix and translation vector are modified by the correction rotation angle and displacement compensation parameters to improve the accuracy of hand-eye calibration.

[0019] In one embodiment, it further includes:

[0020] Through the prior training data set, the neural network algorithm is used to train the preset machine learning model to obtain the error compensation of hand-eye calibration;

[0021] The rotation matrix and translation vector are corrected by the error compensation amount to improve the accuracy of hand-eye calibration.

[0022] In one embodiment, it further includes:

[0023] Analyzing the overlap area between the camera field of view and the motion area of the robotic arm, and dividing the overlap area into at least four overlap sub-areas;

[0024] The motion trajectory of the robot arm is optimized so that the motion trajectory can cover all overlapping sub-areas without repeatedly covering them.

[0025] According to a second aspect of the present invention, a hand-eye calibration device for a robotic arm is provided, which is applied to a robot with a robotic arm, comprising:

[0026] A control module is used to fix a calibration plate at the end of the robotic arm and control the robotic arm to move along a preset trajectory so that the calibration plate is always within the field of view of the camera;

[0027] An acquisition module is used to acquire the image of the calibration plate in at least three different postures of the robotic arm, and simultaneously record the position and posture of the calibration plate at the end of the robotic arm in the robotic arm coordinate system;

[0028] A calculation module, configured to calculate the position and posture of the calibration plate in the camera coordinate system using the calibration plate corner points in the calibration plate image;

[0029] The calibration module is used to calculate the rotation matrix and translation vector between the manipulator coordinate system and the camera coordinate system according to the position of the corresponding calibration plate in the manipulator coordinate system and the position of the camera coordinate system to complete the hand-eye calibration.

[0030] In one embodiment, the control module, the acquisition module, the calculation module and the calibration module are controlled to execute any one of the above-mentioned hand-eye calibration methods for the robotic arm.

[0031] According to a third aspect of the present invention, there is provided an electronic device, the electronic device comprising: a communication interface, a processor, and a memory;

[0032] Wherein, the memory is used to store program instructions, and when the program instructions are executed by the processor that is communicatively connected to the memory through the communication interface, any of the above-mentioned hand-eye calibration methods for the robotic arm is implemented.

[0033] According to a fourth aspect of the present invention, there is provided a robot with a robotic arm, which can implement any of the above-mentioned hand-eye calibration methods for the robotic arm when executed.

[0034] In summary, the present invention provides a method and device for hand-eye calibration of a robotic arm, the method comprising: fixing a calibration plate at the end of the robotic arm, controlling the robotic arm to move along a preset trajectory so that the calibration plate is always within the field of view of the camera; capturing images of the calibration plate in at least three different postures of the robotic arm, and simultaneously recording the position and posture of the calibration plate at the end of the robotic arm in the robotic arm coordinate system; calculating the position and posture of the calibration plate in the camera coordinate system through the corner points of the calibration plate in the calibration plate image; calculating the rotation matrix and translation vector between the robotic arm coordinate system and the camera coordinate system according to the corresponding position and posture of the calibration plate in the robotic arm coordinate system and the camera coordinate system to complete hand-eye calibration. The technical solution of the present application fixes a calibration plate at the end of the robotic arm, controls the robotic arm to move along a preset trajectory, captures camera images in multiple different postures, and calculates the position and posture of the calibration plate in the camera coordinate system, thereby achieving high-precision hand-eye calibration. In particular, through depth cameras, multimodal calibration plate design, error compensation and AI correction technology, the system error is effectively reduced, and the stability of robotic automation applications is significantly improved.

[0035] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0036] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 A flowchart of a hand-eye calibration method for a robotic arm provided by an embodiment of the present invention;

[0039] Figure 2 A flowchart of another method for hand-eye calibration of a robotic arm provided by an embodiment of the present invention;

[0040] Figure 3 A flowchart of another method for hand-eye calibration of a robotic arm provided by an embodiment of the present invention;

[0041] Figure 4 A flowchart of another method for hand-eye calibration of a robotic arm provided by an embodiment of the present invention;

[0042] Figure 5 A flowchart of another method for hand-eye calibration of a robotic arm provided by an embodiment of the present invention;

[0043] Figure 6 A flowchart of another method for hand-eye calibration of a robotic arm provided by an embodiment of the present invention;

[0044] Figure 7 An image captured by the left robotic arm camera provided in an embodiment of the present invention;

[0045] Figure 8 An image captured by the right robotic arm camera provided in an embodiment of the present invention;

[0046] Figure 9 A structural diagram of a hand-eye calibration device for a robotic arm provided by an embodiment of the present invention;

[0047] Figure 10 A structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present application by illustrating the examples of the present application.

[0049] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, the elements defined by the phrase "comprising..." do not exclude the presence of other identical elements in the process, method, article, or device comprising the elements.

[0050] like Figure 1 As shown, the present invention provides a hand-eye calibration method for a robotic arm, which is applied to a robot with a robotic arm. The hand-eye calibration method for the robotic arm includes:

[0051] In step S11, a calibration plate is fixed at the end of the robotic arm, and the robotic arm is controlled to move along a preset trajectory so that the calibration plate is always within the field of view of the camera;

[0052] In step S12, the calibration plate image is captured in at least three different postures of the robotic arm, and the position and posture of the calibration plate at the end of the robotic arm in the robotic arm coordinate system are recorded;

[0053] In step S13, the position and posture of the calibration plate in the camera coordinate system are calculated using the calibration plate corner points in the calibration plate image;

[0054] In step S14, according to the position of the corresponding calibration plate in the robot coordinate system and the position in the camera coordinate system, the rotation matrix and translation vector between the robot coordinate system and the camera coordinate system are calculated to complete the hand-eye calibration.

[0055] In one embodiment, this embodiment provides a method for hand-eye calibration of a robot arm. A calibration plate is fixed at the end of the robot arm. By controlling the robot arm to move along a preset trajectory, the calibration plate is always within the camera's field of view. The camera is preferably installed on the robot's head. In a preferred embodiment, the robot arm can be in the form of a left or right robot arm, as shown in the attached figure. Figure 7 and attached Figure 8 In multiple different postures, the calibration plate image is collected and the position of the calibration plate at the end of the manipulator in the manipulator coordinate system is recorded. Then, an image processing algorithm is used to extract the corner points of the calibration plate in the image, and the position of the calibration plate in the camera coordinate system is calculated. The rotation matrix and translation vector between the camera coordinate system and the manipulator coordinate system are then solved to complete the hand-eye calibration.

[0056] A prefabricated standardized calibration plate is fixed to the end effector of the robotic arm. The calibration plate can adopt a 6×9 checkerboard pattern, but is not limited to a 6×9 checkerboard pattern, and has high-precision printing and good flatness. After fixation, the robotic arm is preset for motion planning through the robot control system. The planning path can adopt a spiral or grid trajectory to ensure that the calibration plate always remains within the field of view of the robot head camera during the movement of the robotic arm. To avoid image blurring caused by motion acceleration, the robotic arm can pause for a period of time after reaching each predetermined position to stabilize the posture. For example, in a calibration process, the robotic arm starts from the initial position and moves to 10 different positions along the predetermined path. Each position corresponds to a posture, and each posture has different translation and rotation angles, thereby ensuring that the collected data covers a larger workspace.

[0057] While the robotic arm is moving along a preset trajectory, the camera simultaneously captures the calibration plate in real time. Whenever the robotic arm reaches a predetermined posture and stabilizes, the calibration plate image is automatically captured, and the posture data of the calibration plate at the end of the robotic arm in the robotic arm coordinate system is recorded. The posture data is obtained by the robotic arm joint encoder and kinematic calculation. In order to improve the accuracy of data acquisition, at least 3 data in different postures are collected, and in actual applications, 10 times or more can be collected. For example, in a certain experiment, a total of 15 sets of image data were collected, each set of data containing the calibration plate image and the corresponding posture of the robotic arm end, providing a rich sample basis for subsequent hand-eye calibration.

[0058] After performing image enhancement and denoising on the captured calibration plate image, the calibration plate's corners are extracted using a processing algorithm (such as the findChessboardCorners function in the OpenCV library). Sub-pixel optimization (the cornerSubPix function) further improves corner detection accuracy, obtaining highly accurate corner pixel coordinates. These corner coordinates are then combined with the camera's intrinsic parameters and the calibrateCamera function is used to calculate the calibration plate's position in the camera coordinate system, yielding the rotation matrix and translation vector.

[0059] Using the position of the calibration plate at the end of the robotic arm in the robotic arm coordinate system and the position of the calibration plate in the camera coordinate system, a hand-eye calibration algorithm (such as the calibrateHandEye function) is used to solve the rigid transformation parameters between the camera coordinate system and the robotic arm coordinate system, namely the rotation matrix and translation vector. By fitting multiple sets of data and using mathematical methods such as least squares, the optimal matching solution is obtained, thus achieving the purpose of hand-eye calibration. For example, in one experiment, after multiple data fittings, the rotation matrix and translation vector obtained reduced the overall reprojection error of the system to within a preset threshold, indicating that the hand-eye calibration results have high accuracy and repeatability.

[0060] After completing hand-eye calibration, its accuracy is verified through practical applications. For example, the calibrated hand-eye parameters are applied to robotic grasping or assembly tasks to detect deviations between the actual operation and the expected target. If the verification results meet the requirements, the rigid transformation parameters between the camera and the robotic arm coordinate system are output and used as the basis for the robot's subsequent visual guidance and autonomous operation.

[0061] To further improve calibration accuracy and robustness, this embodiment, based on the above-mentioned basic method, proposes an extended solution for optimizing the data acquisition strategy and improving the calibration plate design. In terms of data acquisition strategy, the overlap between the camera field of view and the robot arm's motion area is analyzed, and the overlapping area is divided into several sub-areas (at least four). Then, by optimizing the robot arm's motion trajectory, it is ensured that each sub-area is fully covered and not repeatedly covered during the acquisition process, so that the collected calibration data is evenly distributed throughout the entire workspace, improving the generalization ability of the calculated hand-eye calibration parameters. For example, in one optimization, after the overlapping area division and path optimization, every corner of the camera's field of view can be covered, eliminating the problem of local errors caused by insufficient data in some areas.

[0062] In terms of calibration plate design, in addition to the traditional checkerboard pattern, multiple auxiliary positioning markers with different geometric shapes and optical characteristics are pre-set. A multi-template matching algorithm is used to simultaneously extract information from the checkerboard corners and auxiliary markers to form multimodal calibration features, further improving the recognition rate and stability of features in the calibration plate image. This not only enhances robustness under varying lighting conditions and partial occlusion, but also provides richer feature information for subsequent pose calculations.

[0063] The technical solution in this embodiment utilizes a calibration plate and camera fixed to the end of the robotic arm, combined with a preset motion trajectory, to capture data in multiple different postures. Image processing and mathematical algorithms are then used to accurately determine the transformation parameters between the camera and robotic arm coordinate systems. By expanding the data acquisition strategy and improving the calibration plate design, the system's adaptability and robustness to environmental changes are enhanced while ensuring high-precision hand-eye calibration. This provides reliable technical support for robotic vision guidance and precision operations.

[0064] In one embodiment, Figure 2 As shown, the following steps S21-S22 are also included:

[0065] In step S21, a depth camera is used to simultaneously capture a color image and a depth image of the calibration plate;

[0066] In step S22 , the three-dimensional position and posture of the calibration plate in the camera coordinate system is calculated according to the three-dimensional coordinates of the corner points of the calibration plate in the calibration plate image with depth information.

[0067] In one embodiment, it involves depth camera acquisition and three-dimensional pose calculation. The depth camera simultaneously acquires the color image and depth image of the calibration plate, and calculates the three-dimensional pose of the calibration plate in the camera coordinate system based on the three-dimensional coordinates of the corner points of the calibration plate extracted from the image. During the implementation process, the depth camera is installed in the robot system, and the internal parameters and distortion parameters of the depth camera are calibrated in advance to ensure that there is an accurate correspondence between the acquired color image and the depth image. The depth camera simultaneously acquires the color image and depth image of the calibration plate, so that in each frame of the image, clear visual information of the calibration plate pattern can be obtained, and the distance information of each pixel can be obtained through the depth map, thereby realizing the conversion of two-dimensional image to three-dimensional point cloud data.

[0068] After image acquisition, the image processing module preprocesses the captured color image, including denoising and contrast enhancement. A classic checkerboard corner detection algorithm (such as the findChessboardCorners function in OpenCV) is then used to extract the corner locations in the calibration plate image. Subsequently, the 2D pixel coordinates of the corners are converted to 3D coordinates in the camera coordinate system using the depth values of the corresponding pixel locations in the depth image. To improve corner extraction accuracy, sub-pixel optimization can be performed on the extracted corners (for example, using the cornerSubPix function) to ensure higher accuracy in the converted 3D coordinate data. For example, in one calibration experiment, the image processing module extracted 15 sets of corner data in different poses. The reprojection errors between the calibration plate pose and the actual measurement results were all below a preset threshold, validating the high accuracy and robustness of this method. Furthermore, to further improve computational accuracy, the acquired data can be repeatedly acquired and averaged, or the initial calculation results can be iteratively corrected using nonlinear optimization methods. In this way, not only can the data error caused by the inherent noise of the depth sensor be compensated, but the error caused by factors such as ambient lighting and motion blur can also be effectively reduced.

[0069] This embodiment uses a depth camera to simultaneously capture color images and depth images, converts the corner points of the calibration plate into three-dimensional coordinates, and calculates the three-dimensional pose of the calibration plate in the camera coordinate system based on this, providing reliable and accurate basic data for robot hand-eye calibration, significantly improving the overall calibration accuracy and robustness of the system.

[0070] In one embodiment, Figure 3 As shown, the following steps S31-S33 are also included:

[0071] In step S31, a plurality of positioning marks with different geometric shapes and optical features are preset on the calibration plate, and these marks and the pattern of the calibration plate form a multimodal calibration feature;

[0072] In step S32, multimodal calibration features are extracted from the calibration plate image captured by the camera, where the multimodal calibration features include the calibration plate corner points and the positioning marks;

[0073] In step S33, the position and posture of the calibration plate in the camera coordinate system is calculated based on the multimodal calibration features.

[0074] In one embodiment, the traditional hand-eye calibration method is improved by pre-setting multiple positioning markers with different geometric shapes and optical characteristics on a calibration plate. These markers, together with the traditional checkerboard pattern, form a multimodal calibration feature, thereby improving the accuracy and robustness of feature point extraction in the image. In a specific implementation, positioning markers with different shapes (e.g., circles, triangles, rectangles, etc.) and different colors or optical reflectivity are pre-designed and printed on the calibration plate. These positioning markers complement the checkerboard pattern to form a set of multimodal calibration features. This not only provides the image processing algorithm with more auxiliary information when extracting traditional checkerboard corner points, but also allows the other positioning markers to be effectively identified even when parts of the area are obscured or the lighting changes.

[0075] The camera captures the calibration plate image in real time. The preprocessing module performs denoising, brightness adjustment, and edge enhancement on the calibration plate image to improve subsequent feature extraction. A checkerboard corner detection algorithm (such as the findChessboardCorners and cornerSubPix functions in OpenCV) is used to extract the checkerboard corners. A multi-template matching algorithm is used to identify and locate the positioning markers. This technique uses pre-built positioning marker templates to scan the image template by template, using correlation metrics to determine the degree of matching, thereby accurately locating the position and outline of each positioning marker. Through this dual extraction method, the multimodal calibration features include both traditional checkerboard corners and positioning markers. Data redundancy ensures that even if a feature is missing in certain situations, accurate calibration can still be performed using other features. For example, in one experiment, the system designed 10 different positioning markers on the same calibration plate and, combined with the checkerboard corners, extracted over 60 feature points. Through the above method, after calculation using the PnP (Perspective-n-Point) algorithm, the pose parameters of the calibration plate in the camera coordinate system are obtained, and its reprojection error is lower than the preset threshold.

[0076] This embodiment uses multimodal calibration features to accurately calculate the position and posture of the calibration plate, which not only enhances the reliability of traditional checkerboard features in the image, but also supplements additional information through positioning marks, thereby improving the calibration accuracy under complex lighting and partial occlusion conditions.

[0077] In one embodiment, Figure 4As shown, the following steps S41-S42 are also included:

[0078] In step S41, the correction rotation angle and displacement compensation parameters are analyzed based on the systematic deviations caused by the camera hardware, ambient lighting, and the state of the robotic arm;

[0079] In step S42, the rotation matrix and translation vector are corrected by using the correction rotation angle and displacement compensation parameters to improve the accuracy of hand-eye calibration.

[0080] In one embodiment, a technical solution for correcting rotation angle and displacement compensation parameters is proposed to address systematic deviations caused by factors such as camera hardware characteristics, ambient lighting variations, and fluctuations in the robotic arm's state. This method corrects the initially calculated rotation matrix and translation vector, improving the overall accuracy of hand-eye calibration. By collecting multiple sets of calibration data, the systemic errors caused by camera intrinsic parameter deviations, lens distortion, uneven lighting, and changes in the robotic arm's operating state (such as joint wear and motion instability) are analyzed. These errors manifest as systematic shifts in rotation angle and displacement under multiple different postures.

[0081] During the data acquisition phase, the robot control system records the calibration plate image and the end-of-arm pose captured along a preset trajectory. Using image processing algorithms and traditional checkerboard corner detection techniques, the initial pose of the calibration plate in the camera coordinate system is calculated. This is then compared with the theoretical pose recorded in the robotic arm coordinate system to determine the reprojection error or pose error for each data set. Analysis results show that the errors often exhibit certain regularities. For example, in most collected data, the rotation angle exhibits a fixed deviation, and the translation generally exhibits a certain amount of offset in a certain direction.

[0082] Based on the above analysis, an error model is further constructed to quantitatively describe the systematic deviation. The error distribution of all collected data is statistically processed to extract the mean deviation and variance, and then the correction rotation angle and displacement compensation parameters are determined. The correction rotation angle is the angle adjustment that minimizes the error after rotation matrix compensation, while the displacement compensation parameter represents the offset that needs to be added or subtracted from the translation vector. To further improve the correction effect, this embodiment uses a nonlinear least squares method, and through iterative optimization, it is solved so that the compensation parameters can reduce the reprojection error of all collected data to below a preset threshold.

[0083] For example, in one experiment, after preliminary calibration, it was found that the average deviation of the calculated rotation angle relative to the theoretical pose was positive 5°, and the translation vector had a positive offset of 2 mm in the X-axis direction. Through nonlinear optimization calculations, the correction rotation angle was obtained to be -5° and the X-axis displacement compensation parameter was -2 mm. Subsequently, the compensation parameters were applied to the rotation matrix and translation vector obtained from the preliminary calculation to generate the corrected hand-eye calibration parameters. After verifying the corrected parameters, the results showed that the reprojection error of each set of data was greatly reduced, and the overall calibration accuracy was significantly improved.

[0084] Furthermore, extended compensation measures for systematic errors are provided. While collecting data, the system simultaneously monitors the camera hardware status (such as focal length and lens temperature), ambient light intensity, and the real-time motion parameters of the robotic arm. This auxiliary data is used as input to further refine the error model. By introducing these additional variables, it is possible to more accurately capture error trends caused by environmental or hardware fluctuations, allowing for real-time adjustments to the correction rotation angle and displacement compensation parameters, ensuring that the hand-eye calibration process maintains high accuracy under various working conditions.

[0085] This implementation comprehensively analyzes systematic deviations and uses nonlinear optimization methods to calculate the corrected rotation angle and displacement compensation parameters, thereby correcting the rotation matrix and translation vector obtained through preliminary calibration. This not only effectively reduces systematic errors caused by camera hardware, ambient lighting, and robotic arm status, but also further improves correction accuracy by introducing additional variables.

[0086] In one embodiment, Figure 5 As shown, the following steps S51-S52 are also included:

[0087] In step S51, a preset machine learning model is trained using a neural network algorithm using a priori training data sets to obtain an error compensation amount for hand-eye calibration;

[0088] In step S52, the rotation matrix and translation vector are corrected by using the error compensation amount to improve the accuracy of hand-eye calibration.

[0089] In one embodiment, to address systematic errors introduced by various factors during robot hand-eye calibration, such as the camera system, environmental factors, and the robot arm's state, a machine learning model is constructed using a priori training datasets and a neural network algorithm to predict error compensation. This compensation is then applied to the rotation matrix and translation vector, further improving hand-eye calibration accuracy. Multiple calibration experiments are conducted within the robot system to collect a large number of hand-eye calibration data samples. Each data sample includes a calibration plate image captured by the camera, the robot arm's end-of-arm pose data, and the rotation matrix and translation vector calculated by the preliminary calibration algorithm. Deviations measured during actual operation (such as reprojection error or positioning error) are also recorded. To ensure data representativeness and diversity, data collection is performed under varying ambient lighting conditions, robot arm speeds, and camera temperatures. The collected data is then preprocessed, such as normalization and noise filtering, to construct a priori training dataset. Based on the training dataset, a machine learning model is constructed and trained using a neural network algorithm. The neural network model's inputs include characteristic parameters of the collected data, such as the rotation matrix and translation vector obtained through preliminary calibration, camera image quality indicators, ambient lighting values, and the dynamic state parameters of the robotic arm. Its output is the predicted error compensation, which includes both rotation angle deviation compensation and displacement compensation. During training, supervised learning methods are used to minimize the mean squared error between the model output and the actual calibration error, continuously adjusting the network weights until the model achieves high accuracy in predicting the calibration error under different environments. To enhance the model's robustness and prevent overfitting, techniques such as cross-validation, regularization, and data augmentation can also be used.

[0090] After model training is complete, the pre-set machine learning model is integrated into the robot calibration system. During the actual hand-eye calibration process, a preliminary rotation matrix and translation vector are obtained using traditional methods. Real-time environmental parameters and preliminary calibration results are then collected as input and fed into the neural network model. Based on the input features, the model outputs an error compensation factor, representing systematic errors caused by factors such as camera intrinsic parameter deviations, ambient lighting changes, or dynamic instability of the robotic arm. Based on the error compensation factor, the pre-calculated rotation matrix is subjected to a corresponding rotation correction, and the translation vector is subjected to a translation correction, resulting in the final hand-eye calibration parameters. For example, in one experiment, preliminary calibration results showed a rotation matrix deviation of approximately 3° and a translation vector offset of approximately 1.5 mm on the X-axis. The trained neural network model predicted the output error compensation factors: a rotation correction of -3° and a displacement compensation of -1.5 mm. After applying this compensation factor, the system's reprojection error was significantly reduced, and the final calibration accuracy was significantly improved.

[0091] This embodiment constructs an error compensation model by utilizing prior training data and a neural network algorithm to achieve intelligent correction of the preliminary calibration results, significantly reducing systematic errors and improving the accuracy and robustness of hand-eye calibration.

[0092] In one embodiment, Figure 6 As shown, the following steps S61-S62 are also included:

[0093] In step S61, the overlapping area between the camera field of view and the motion area of the robot arm is analyzed, and the overlapping area is divided into at least four overlapping sub-areas;

[0094] In step S62 , the motion trajectory of the robot arm is optimized so that the motion trajectory can cover all overlapping sub-regions without repeatedly covering them.

[0095] In one embodiment, a motion trajectory optimization method based on the overlap of the camera field of view and the robot arm motion area is proposed to address the data collection problem of the robot arm and the camera during the calibration process. By geometrically modeling the camera installation position, its field of view, and the robot arm workspace, the overlap area between the two is determined, that is, the area where the camera can stably collect the image of the calibration plate at the end of the robot arm. The overlapping area is divided and evenly divided into at least four sub-areas, and each sub-area is ensured to contain sufficient calibration plate image acquisition information. For example, if the camera field of view is rectangular, it can be divided into four parts in the horizontal and vertical directions, thereby forming sixteen sub-areas to ensure that representative image data is collected in each sub-area.

[0096] After the division is completed, the motion planning algorithm is used to optimize the motion trajectory of the robot arm so that the calibration plate can cover all overlapping sub-areas during the movement process, and the data collection between the sub-areas is non-repetitive. In specific implementation, the geometric boundary of the overlapping area is calculated through the camera internal parameters and the robot arm kinematic parameters, and then the overlapping area is divided into several sub-areas according to the boundary coordinates. According to the position and area of each sub-area, a genetic algorithm or a greedy algorithm is used to design an optimal motion path. This path not only meets the shortest stroke requirement, but also ensures that the robot arm moves smoothly without drastic acceleration and deceleration, thereby reducing image blur or data inconsistency caused by dynamic changes during the acquisition process.

[0097] For example, in one experiment, through path optimization, the calibration plate at the end of the robotic arm passed through 25 sub-areas in sequence, and each sub-area collected at least two sets of image data with different postures, ensuring the uniform distribution of the collected data. After optimization, the overlapping data collection of each sub-area avoided local over-sampling or data redundancy, thereby significantly improving the accuracy of posture calculation based on image features during hand-eye calibration. In order to further enhance the robustness of the system, this embodiment can also monitor the quality of the collected images in each sub-area in real time during the movement process. When it is detected that the image quality of a sub-area is poor, the robotic arm movement trajectory is automatically adjusted to re-cover the sub-area to ensure that the final collected data is complete and balanced.

[0098] This embodiment divides the overlapping area between the camera field of view and the robot arm's motion area in detail, and adopts a motion trajectory optimization algorithm to ensure that the robot arm's motion path can cover all sub-areas without repeating them. This not only improves the representativeness of the collected data, but also provides a reliable data basis for subsequent hand-eye calibration based on image processing.

[0099] In one embodiment, Figure 9 FIG. 1 is a block diagram of a hand-eye calibration device for a robotic arm according to an exemplary embodiment. Figure 9 As shown, the hand-eye calibration device for a robotic arm includes a control module 91, an acquisition module 92, a calculation module 93 and a calibration module 94, and is applied to a robot with a robotic arm.

[0100] The control module 91 is used to fix the calibration plate at the end of the robotic arm and control the robotic arm to move along a preset trajectory so that the calibration plate is always within the field of view of the camera;

[0101] The acquisition module 92 is used to acquire the calibration plate image in at least three different postures of the robotic arm, and simultaneously record the position and posture of the calibration plate at the end of the robotic arm in the robotic arm coordinate system;

[0102] The calculation module 93 is used to calculate the position and posture of the calibration plate in the camera coordinate system using the calibration plate corner points in the calibration plate image;

[0103] The calibration module 94 is used to calculate the rotation matrix and translation vector between the manipulator coordinate system and the camera coordinate system according to the position of the corresponding calibration plate in the manipulator coordinate system and the position in the camera coordinate system to complete the hand-eye calibration.

[0104] The control module 91, the acquisition module 92, the calculation module 93 and the calibration module 94 included in the block diagram of the hand-eye calibration device for a robotic arm are controlled to execute the hand-eye calibration method for the robotic arm described in any of the above embodiments.

[0105] like Figure 10As shown, the present invention provides an electronic device 1000, which includes: a communication interface, a processor 1001, and a memory 1002;

[0106] Among them, the memory 1002 is used to store program instructions. When the program instructions are executed by the processor 1001 that is communicatively connected to the memory 1002 through the communication interface, a calibration plate is fixed at the end of the robotic arm, and the robotic arm is controlled to move along a preset trajectory so that the calibration plate is always in the field of view of the camera; the calibration plate image is collected in at least three different postures of the robotic arm, and the position of the calibration plate at the end of the robotic arm in the robotic arm coordinate system is recorded at the same time; the position of the calibration plate in the camera coordinate system is calculated through the corner points of the calibration plate in the calibration plate image; according to the corresponding position of the calibration plate in the robotic arm coordinate system and the position in the camera coordinate system, the rotation matrix and translation vector between the robotic arm coordinate system and the camera coordinate system are calculated to complete hand-eye calibration.

[0107] The present invention provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, a calibration plate is fixed at the end of a robotic arm, and the robotic arm is controlled to move along a preset trajectory so that the calibration plate is always within the field of view of a camera; an image of the calibration plate is captured in at least three different postures of the robotic arm, and the position and orientation of the calibration plate at the end of the robotic arm in the robotic arm coordinate system is recorded; the position and orientation of the calibration plate in the camera coordinate system is calculated through the corner points of the calibration plate in the calibration plate image; and a rotation matrix and a translation vector between the robotic arm coordinate system and the camera coordinate system are calculated according to the corresponding position and orientation of the calibration plate in the robotic arm coordinate system and the camera coordinate system to complete hand-eye calibration.

[0108] It should be understood that the specific features, operations and details described herein above with respect to the method of the present invention may also be similarly applied to the apparatus and system of the present invention, or vice versa. In addition, each step of the method of the present invention described above may be performed by the corresponding components or units of the apparatus or system of the present invention.

[0109] It should be understood that the various modules / units of the apparatus of the present invention may be implemented in whole or in part by software, hardware, firmware, or a combination thereof. Each module / unit may be embedded in a processor of a computer device in the form of hardware or firmware or may be independent of the processor, or may be stored in a memory of a computer device in the form of software for the processor to call to execute the operations of each module / unit. Each module / unit may be implemented as an independent component or module, or two or more modules / units may be implemented as a single component or module.

[0110] In one embodiment, a computer device is provided, comprising a memory and a processor. The memory stores computer instructions executable by the processor, which, when executed by the processor, instruct the processor to perform the steps of the method according to an embodiment of the present invention. The computer device can be broadly defined as a server, a terminal, or any other electronic device with the necessary computing and / or processing capabilities. In one embodiment, the computer device can include a processor, memory, a network interface, a communication interface, etc. connected via a system bus. The processor of the computer device can be used to provide the necessary computing, processing, and / or control capabilities. The memory of the computer device can include a non-volatile storage medium and an internal memory. An operating system, a computer program, etc. can be stored in or on the non-volatile storage medium. The internal memory can provide an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface and communication interface of the computer device can be used to connect to and communicate with external devices via a network. When the computer program is executed by the processor, the steps of the method according to the present invention are performed.

[0111] The present invention can be implemented as a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the steps of the method of an embodiment of the present invention to be performed. In one embodiment, the computer program is distributed on a plurality of computer devices or processors coupled to a network so that the computer program is stored, accessed, and executed in a distributed manner by one or more computer devices or processors. A single method step / operation, or two or more method steps / operations, can be performed by a single computer device or processor or by two or more computer devices or processors. One or more method steps / operations can be performed by one or more computer devices or processors, and one or more other method steps / operations can be performed by one or more other computer devices or processors. One or more computer devices or processors can perform a single method step / operation, or perform two or more method steps / operations.

[0112] Those skilled in the art will appreciate that the method steps of the present invention can be performed by instructing related hardware, such as a computer device or processor, through a computer program. The computer program can be stored in a non-transitory computer-readable storage medium, and when the computer program is executed, the steps of the present invention are performed. Depending on the circumstances, any reference herein to memory, storage, database, or other media may include non-volatile and / or volatile memory. Examples of non-volatile memory include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state disk, etc. Examples of volatile memory include random access memory (RAM), external cache memory, etc.

[0113] The various technical features described above can be combined arbitrarily. Although not all possible combinations of these technical features are described, any combination of these technical features should be considered to be covered by this specification as long as such combination does not conflict.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hand-eye calibration method for a robotic arm, characterized in that: Applications to robots with robotic arms include: Fixing a calibration plate at the end of the robotic arm and controlling the robotic arm to move along a preset trajectory so that the calibration plate is always within the field of view of the camera; Capturing the calibration plate image in at least three different postures of the robotic arm, and simultaneously recording the position and posture of the calibration plate at the end of the robotic arm in the robotic arm coordinate system; Calculating the position and orientation of the calibration plate in the camera coordinate system using the calibration plate corner points in the calibration plate image; According to the position and posture of the corresponding calibration plate in the manipulator coordinate system and the camera coordinate system, the rotation matrix and translation vector between the manipulator coordinate system and the camera coordinate system are calculated to complete the hand-eye calibration; Also includes: Through the prior training data set, the neural network algorithm is used to train the preset machine learning model to obtain the error compensation of hand-eye calibration; The input of the neural network algorithm includes the characteristic parameters of the collected data, specifically the rotation matrix and translation vector obtained by preliminary calibration, the camera image quality index, the ambient light value, and the dynamic state parameters of the robotic arm; The output of the neural network algorithm includes the predicted error compensation amount, specifically the rotation angle deviation compensation amount and the displacement compensation amount; The rotation matrix and translation vector are corrected by the error compensation amount to improve the accuracy of hand-eye calibration.

2. The hand-eye calibration method for a robotic arm according to claim 1, wherein: Also includes: Acquire the color image and depth image of the calibration plate simultaneously by a depth camera; The three-dimensional position and orientation of the calibration plate in the camera coordinate system is calculated according to the three-dimensional coordinates of the corner points of the calibration plate in the calibration plate image with depth information.

3. The hand-eye calibration method for a robotic arm according to claim 1, wherein: Also includes: Presetting a plurality of positioning marks with different geometric shapes and optical features on the calibration plate to form a multimodal calibration feature together with the pattern of the calibration plate; Extracting multimodal calibration features from the calibration plate image captured by the camera, the multimodal calibration features including the calibration plate corner points and the positioning marks; The position and orientation of the calibration plate in the camera coordinate system are calculated based on the multimodal calibration features.

4. The hand-eye calibration method for a robotic arm according to claim 1, wherein: Also includes: Analyzing the overlap area between the camera field of view and the motion area of the robotic arm, and dividing the overlap area into at least four overlap sub-areas; The motion trajectory of the robot arm is optimized so that the motion trajectory can cover all overlapping sub-areas without repeatedly covering them.

5. A hand-eye calibration device for a robotic arm, characterized in that: Applications to robots with robotic arms include: A control module is used to fix a calibration plate at the end of the robotic arm and control the robotic arm to move along a preset trajectory so that the calibration plate is always within the field of view of the camera; An acquisition module is used to acquire the image of the calibration plate in at least three different postures of the robotic arm, and simultaneously record the position and posture of the calibration plate at the end of the robotic arm in the robotic arm coordinate system; A calculation module, configured to calculate the position and posture of the calibration plate in the camera coordinate system using the calibration plate corner points in the calibration plate image; A calibration module is used to calculate the rotation matrix and translation vector between the manipulator coordinate system and the camera coordinate system according to the position of the corresponding calibration plate in the manipulator coordinate system and the position of the camera coordinate system to complete the hand-eye calibration; The device is also used to train a preset machine learning model using a neural network algorithm through a priori training data sets to obtain error compensation for hand-eye calibration; the input of the neural network algorithm includes characteristic parameters of the collected data, specifically including the rotation matrix and translation vector obtained through preliminary calibration, camera image quality indicators, ambient light values, and dynamic state parameters of the robotic arm; the output of the neural network algorithm includes predicted error compensation, specifically including rotation angle deviation compensation and displacement compensation; the rotation matrix and translation vector are corrected using the error compensation to improve the accuracy of hand-eye calibration.

6. An electronic device, characterized in that: include: Communication interface, processor, memory; Wherein, the memory is used to store program instructions, and when the program instructions are executed by the processor that is communicatively connected to the memory through the communication interface, the electronic device implements the hand-eye calibration method of the robotic arm described in any one of claims 1 to 4.

7. A robot with a robotic arm, capable of executing the method according to any one of claims 1 to 4, or comprising the apparatus according to claim 5, or having the electronic device according to claim 6.

Images