Hand-eye calibration method, device and equipment for mechanical arm and storage medium

Abstract

The invention discloses a hand-eye calibration method, device and equipment for a mechanical arm and a storage medium. The method comprises the steps that a calibration plate is fixed to the tail end of a mechanical arm, and the mechanical arm is controlled to move along a preset track, so that the calibration plate is always located in the view field of a camera; under at least three different postures of the mechanical arm, the calibration plate image is collected, and meanwhile the pose of the calibration plate at the tail end of the mechanical arm under a mechanical arm coordinate system is recorded; calculating the pose of the calibration plate in the camera coordinate system through calibration plate angular points in the calibration plate image; and according to the pose of the corresponding calibration plate in the mechanical arm coordinate system and the pose of the corresponding calibration plate in the camera coordinate system, calculating a rotation matrix and a translation vector between the mechanical arm coordinate system and the camera coordinate system so as to complete hand-eye calibration. According to the technical scheme, high-precision hand-eye calibration is achieved, and the stability of automatic application of the robot is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of robot control, and in particular to a hand-eye calibration method, device, equipment and storage medium for a robot arm. Background Art

[0002] At present, the application of robot automation systems in industries such as industry, logistics and services is becoming more and more widespread. As the core technology for realizing robot vision guidance and precise operation, the main purpose of robot hand-eye calibration is to establish an accurate mapping relationship between the camera coordinate system and the robot coordinate system. The traditional hand-eye calibration method usually uses a camera fixed on the head of the robot and a calibration plate fixed on the end of the robot arm. The calibration plate image is collected through a preset motion trajectory, and the position and posture of the calibration plate in the camera coordinate system are calculated using an image processing algorithm, and then the rigid transformation parameters between the camera and robot coordinate systems are solved. However, due to the dynamic deviation, camera internal parameter error, and environmental factors such as light and noise in the process of robot arm movement, the traditional method often has problems of low measurement accuracy and insufficient robustness in practical applications, which limits the application of robots in high-precision operations. How to properly solve the above problems has become a topic that needs to be solved urgently in the industry. Summary of the invention

[0003] The present invention provides a hand-eye calibration method, device, equipment and storage medium for a robotic arm, which are used to fix a calibration plate and multi-modal acquisition at the end of the arm, thereby improving the accuracy of hand-eye calibration and enhancing the operation stability of the robot.

[0004] According to a first aspect of the present invention, a hand-eye calibration method for a robotic arm is provided. The hand-eye calibration method for a robotic arm is applied to a robot with a robotic arm, comprising: Fixing a calibration plate at the end of the robotic arm, and controlling the robotic arm to move along a preset trajectory so that the calibration plate is always within the field of view of the camera; The calibration plate image is collected in at least three different postures of the robotic arm, and the position and posture of the calibration plate at the end of the robotic arm in the robotic arm coordinate system are recorded simultaneously; Calculating the position and posture of the calibration plate in the camera coordinate system through the calibration plate corner points in the calibration plate image; According to the position and posture of the corresponding calibration plate in the robot arm coordinate system and the camera coordinate system, the rotation matrix and the translation vector between the robot arm coordinate system and the camera coordinate system are calculated to complete the hand-eye calibration.

[0005] In one embodiment, it further includes: Acquire a color image and a depth image of the calibration plate simultaneously by using a depth camera; The three-dimensional position and posture of the calibration plate in the camera coordinate system is calculated according to the three-dimensional coordinates of the corner points of the calibration plate in the calibration plate image with depth information.

[0006] In one embodiment, it further includes: Presetting a plurality of positioning marks with different geometric shapes and optical features on the calibration plate to form a multi-modal calibration feature with the pattern of the calibration plate; Extracting multimodal calibration features from the calibration plate image captured by the camera, wherein the multimodal calibration features include corner points of the calibration plate and the positioning marks; The position and posture of the calibration plate in the camera coordinate system are calculated according to the multimodal calibration features.

[0007] In one embodiment, it further includes: Analyze the correction rotation angle and displacement compensation parameters based on the systematic deviations caused by the camera hardware, ambient lighting, and robot arm status; The rotation matrix and the translation vector are corrected by the correction rotation angle and the displacement compensation parameter to improve the accuracy of the hand-eye calibration.

[0008] In one embodiment, it further includes: Through the prior training data set, the preset machine learning model is trained using the neural network algorithm to obtain the error compensation amount of hand-eye calibration; The rotation matrix and the translation vector are corrected by the error compensation amount to improve the accuracy of hand-eye calibration.

[0009] In one embodiment, it further includes: Analyze the overlap area between the camera field of view and the motion area of ​​the robot arm, and divide the overlap area into at least four overlap sub-areas; The motion trajectory of the robot arm is optimized so that the motion trajectory can cover all overlapping sub-areas without repeated coverage.

[0010] According to a second aspect of the present invention, a hand-eye calibration device for a robotic arm is provided, which is applied to a robot with a robotic arm, comprising: A control module is used to fix a calibration plate at the end of the robotic arm and control the robotic arm to move along a preset trajectory so that the calibration plate is always within the field of view of the camera; An acquisition module, used to acquire the calibration plate image in at least three different postures of the robotic arm, and simultaneously record the position and posture of the calibration plate at the end of the robotic arm in the robotic arm coordinate system; A calculation module, used to calculate the position and posture of the calibration plate in the camera coordinate system through the calibration plate corner points in the calibration plate image; The calibration module is used to calculate the rotation matrix and translation vector between the robotic arm coordinate system and the camera coordinate system according to the position and posture of the corresponding calibration plate in the robotic arm coordinate system and the camera coordinate system to complete the hand-eye calibration.

[0011] In one embodiment, the control module, the acquisition module, the calculation module and the calibration module are controlled to execute any one of the above-mentioned hand-eye calibration methods for the robotic arm.

[0012] According to a third aspect of the present invention, there is provided an electronic device, the electronic device comprising: a communication interface, a processor, and a memory; Wherein, the memory is used to store program instructions, and when the program instructions are executed by the processor that is communicatively connected to the memory through the communication interface, any of the above-mentioned hand-eye calibration methods for the robotic arm is implemented.

[0013] According to a fourth aspect of the present invention, there is provided a robot with a robotic arm, which can be executed to implement any of the above-mentioned hand-eye calibration methods for the robotic arm.

[0014] In summary, the present invention provides a method and device for hand-eye calibration of a robotic arm, the method comprising: fixing a calibration plate at the end of the robotic arm, controlling the robotic arm to move along a preset trajectory, so that the calibration plate is always in the field of view of the camera; collecting the image of the calibration plate under at least three different postures of the robotic arm, and recording the position and posture of the calibration plate at the end of the robotic arm in the robotic arm coordinate system; calculating the position and posture of the calibration plate in the camera coordinate system through the corner points of the calibration plate in the calibration plate image; calculating the rotation matrix and translation vector between the robotic arm coordinate system and the camera coordinate system according to the corresponding position and posture of the calibration plate in the robotic arm coordinate system and the camera coordinate system, so as to complete the hand-eye calibration. The technical solution of the present application fixes the calibration plate at the end of the robotic arm, controls the robotic arm to move along a preset trajectory, collects camera images under multiple different postures, and calculates the position and posture of the calibration plate in the camera coordinate system, thereby realizing high-precision hand-eye calibration, especially through the depth camera, multi-modal calibration plate design, error compensation and AI correction technology, effectively reducing system errors and significantly improving the stability of robot automation applications.

[0015] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0016] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A flow chart of a hand-eye calibration method for a robotic arm provided by an embodiment of the present invention; Figure 2 A flow chart of another method for hand-eye calibration of a robotic arm provided by an embodiment of the present invention; Figure 3 A flowchart of another method for hand-eye calibration of a robotic arm provided by an embodiment of the present invention; Figure 4 A flowchart of another method for hand-eye calibration of a robotic arm provided by an embodiment of the present invention; Figure 5 A flowchart of another method for hand-eye calibration of a robotic arm provided by an embodiment of the present invention; Figure 6 A flowchart of another method for hand-eye calibration of a robotic arm provided by an embodiment of the present invention; Figure 7 An image captured by a left robotic arm camera provided by an embodiment of the present invention; Figure 8 An image captured by a camera of the right robotic arm provided in an embodiment of the present invention; Fig. 9 A structural diagram of a hand-eye calibration device for a robotic arm provided by an embodiment of the present invention; Fig.10 A structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0020] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0021] like Figure 1 As shown, the present invention provides a hand-eye calibration method for a robotic arm, which is applied to a robot with a robotic arm. The hand-eye calibration method for the robotic arm includes: In step S11, a calibration plate is fixed at the end of the robotic arm, and the robotic arm is controlled to move along a preset trajectory so that the calibration plate is always within the field of view of the camera; In step S12, the calibration plate image is collected in at least three different postures of the robotic arm, and the position and posture of the calibration plate at the end of the robotic arm in the robotic arm coordinate system are recorded; In step S13, the position and posture of the calibration plate in the camera coordinate system are calculated by using the calibration plate corner points in the calibration plate image; In step S14, according to the position and posture of the corresponding calibration plate in the robot coordinate system and the camera coordinate system, the rotation matrix and the translation vector between the robot coordinate system and the camera coordinate system are calculated to complete the hand-eye calibration.

[0022] In one embodiment, this embodiment provides a robot hand-eye calibration method, wherein a calibration plate is fixed at the end of the robot arm, and the robot arm is controlled to move along a preset trajectory so that the calibration plate is always within the camera field of view, and the camera is preferably installed on the robot head. In a preferred embodiment, the robot arm can be in the form of a left or right robot arm, as shown in the attached figure. Figure 7 and attached Figure 8 In multiple different postures, the calibration plate image is collected and the position and posture of the calibration plate at the end of the manipulator in the manipulator coordinate system are recorded. Then, the image processing algorithm is used to extract the corner points of the calibration plate in the image, and the position and posture of the calibration plate in the camera coordinate system are calculated. Then, the rotation matrix and translation vector between the camera coordinate system and the manipulator coordinate system are solved to complete the hand-eye calibration.

[0023] The prefabricated standardized calibration plate is fixed to the end effector of the robot arm. The calibration plate can adopt a 6×9 checkerboard pattern, but is not limited to a 6×9 checkerboard pattern, and has high-precision printing and good flatness. After fixing, the robot control system performs preset motion planning on the robot arm, and the planning path can adopt a spiral or grid trajectory to ensure that the calibration plate always remains within the field of view of the robot head camera during the movement of the robot arm. To avoid image blur caused by motion acceleration, the robot arm can pause for a period of time to stabilize the posture after reaching each predetermined position. For example, in a calibration process, the robot arm starts from the initial position and moves to 10 different positions along the predetermined path. Each position corresponds to a posture, and each posture has different translation and rotation angles, thereby ensuring that the collected data covers a larger workspace.

[0024] While the robotic arm is moving along a preset trajectory, the camera simultaneously takes real-time photos of the calibration plate. Whenever the robotic arm reaches a predetermined posture and is stable, the calibration plate image will be automatically captured, and the pose data of the calibration plate at the end of the robotic arm in the robotic arm coordinate system will be recorded. The pose data is obtained by the robotic arm joint encoder and kinematic calculation. In order to improve the accuracy of data acquisition, at least 3 data in different postures are collected, and 10 times or more can be collected in actual applications. For example, in a certain experiment, a total of 15 sets of image data were collected, each set of data contained the calibration plate image and the corresponding pose of the end of the robotic arm, providing a rich sample basis for subsequent hand-eye calibration.

[0025] After image enhancement and denoising, the calibration plate corners in the image are extracted through a processing algorithm (such as the findChessboardCorners function in the OpenCV library). The sub-pixel optimization algorithm (cornerSubPix function) is used to further improve the accuracy of corner detection and obtain high-precision corner pixel coordinates. Then, the corner coordinates are combined with the camera intrinsic parameters, and the calibration plate position in the camera coordinate system is calculated through the calibrateCamera function, that is, the rotation matrix and translation vector are obtained.

[0026] The position and posture of the calibration plate at the end of the robot arm in the robot arm coordinate system and the position and posture of the calibration plate in the camera coordinate system are used to solve the rigid transformation parameters between the camera coordinate system and the robot arm coordinate system, that is, the rotation matrix and the translation vector. Through fitting multiple sets of data, mathematical methods such as the least squares method are used to obtain the best matching solution, thereby achieving the purpose of hand-eye calibration. For example, in one experiment, after multiple data fittings, the obtained rotation matrix and translation vector reduced the overall reprojection error of the system to within the preset threshold, indicating that the hand-eye calibration results have high accuracy and repeatability.

[0027] After completing the hand-eye calibration, verify its accuracy through practical applications, such as using the calibrated hand-eye parameters for robot grasping or assembly tasks to detect the deviation between the actual operation and the expected target. If the verification result meets the requirements, the rigid transformation parameters between the camera and the robot arm coordinate system are output as the basic data for the robot's subsequent visual guidance and autonomous operation.

[0028] In order to further improve the calibration accuracy and robustness, this embodiment proposes an extended solution for optimizing the data acquisition strategy and improving the design of the calibration board based on the above basic method. In terms of data acquisition strategy, the overlap between the camera field of view and the robot arm motion area is analyzed, and the overlapping area is divided into several sub-areas (at least four). Then, by optimizing the motion trajectory of the robot arm, it is ensured that each sub-area is fully covered and not repeatedly covered during the acquisition process, so that the collected calibration data is evenly distributed in the entire workspace, thereby improving the generalization ability of the calculated hand-eye calibration parameters. For example, in one optimization, after the overlapping area division and path optimization, every corner of the camera field of view can be covered, eliminating the problem of local errors caused by insufficient data in some areas.

[0029] In terms of calibration plate design, in addition to the traditional checkerboard pattern, the calibration plate also presets multiple auxiliary positioning marks with different geometric shapes and optical features. Using a multi-template matching algorithm, the checkerboard corner points and auxiliary mark information are extracted simultaneously to form multimodal calibration features, further improving the recognition rate and stability of features in the calibration plate image. This not only enhances the robustness in the case of changing lighting conditions or partial occlusion, but also provides richer feature information for subsequent pose calculations.

[0030] The technical solution in this embodiment realizes data collection in multiple different postures by combining the calibration plate and camera fixed at the end of the robotic arm with the preset motion trajectory, and uses image processing and mathematical algorithms to accurately solve the transformation parameters between the camera and the robotic arm coordinate system. By expanding the data collection strategy and improving the calibration plate design, while ensuring high-precision hand-eye calibration, the system's adaptability and robustness to environmental changes are enhanced, providing reliable technical support for robot vision guidance and precision operations.

[0031] In one embodiment, Figure 2 As shown, the following steps S21-S22 are also included: In step S21, a color image and a depth image of the calibration plate are simultaneously acquired by a depth camera; In step S22, the three-dimensional position and posture of the calibration plate in the camera coordinate system is calculated according to the three-dimensional coordinates of the corner points of the calibration plate in the calibration plate image with depth information.

[0032] In one embodiment, it involves depth camera acquisition and three-dimensional pose calculation. The color image and depth image of the calibration plate are simultaneously acquired by the depth camera, and the three-dimensional pose of the calibration plate in the camera coordinate system is calculated based on the three-dimensional coordinates of the corner points of the calibration plate extracted from the image. During the implementation process, the depth camera is installed in the robot system, and the internal parameters and distortion parameters of the depth camera are calibrated in advance to ensure that there is an accurate correspondence between the acquired color image and the depth image. The color image and depth image of the calibration plate are acquired simultaneously by the depth camera, so that in each frame of the image, clear visual information of the calibration plate pattern can be obtained, and the distance information of each pixel can be obtained through the depth map, thereby realizing the conversion of two-dimensional image to three-dimensional point cloud data.

[0033] After image acquisition, the image processing module is used to preprocess the acquired color image, including denoising, contrast enhancement, etc., and the classic chessboard corner detection algorithm (such as the findChessboardCorners function in OpenCV) is used to extract the corner position in the calibration plate image. Subsequently, the two-dimensional pixel coordinates of the corner point are converted into three-dimensional coordinates in the camera coordinate system by using the depth value of the corresponding pixel position in the depth image. In order to improve the accuracy of corner point extraction, the extracted corner points can also be optimized at the sub-pixel level (for example, using the cornerSubPix function) to ensure that the converted three-dimensional coordinate data has higher accuracy. For example, in a calibration experiment, 15 groups of corner point data in different postures were extracted by the image processing module, and the reprojection error between the calibration plate posture and the actual measurement result was lower than the preset threshold, which verified the high accuracy and robustness of the method. In addition, in order to further improve the calculation accuracy, the acquired data can be repeatedly collected and averaged for multiple times, or the preliminary calculation results can be iteratively corrected using a nonlinear optimization method. In this way, not only can the data error caused by the inherent noise of the depth sensor be compensated, but the error caused by factors such as ambient lighting and motion blur can also be effectively reduced.

[0034] This embodiment uses a depth camera to simultaneously collect color images and depth images, converts the corner points of the calibration plate into three-dimensional coordinates, and calculates the three-dimensional pose of the calibration plate in the camera coordinate system based on this, providing reliable and accurate basic data for robot hand-eye calibration, and significantly improving the overall calibration accuracy and robustness of the system.

[0035] In one embodiment, Figure 3 As shown, the following steps S31-S33 are also included: In step S31, a plurality of positioning marks with different geometric shapes and optical features are preset on the calibration plate, and form a multi-modal calibration feature with the pattern of the calibration plate; In step S32, in the calibration plate image acquired by the camera, a multimodal calibration feature is extracted, wherein the multimodal calibration feature includes the calibration plate corner points and the positioning marks; In step S33, the position and posture of the calibration plate in the camera coordinate system is calculated according to the multimodal calibration features.

[0036] In one embodiment, the traditional hand-eye calibration method is improved, and multiple positioning marks with different geometric shapes and optical features are preset on the calibration board, which together with the traditional checkerboard pattern constitute a multimodal calibration feature, thereby improving the accuracy and robustness of feature point extraction in the image. In the specific implementation process, positioning marks with different shapes (such as circles, triangles, rectangles, etc.) and different colors or optical reflection characteristics are pre-designed and printed on the calibration board. These positioning marks complement the checkerboard pattern to form a set of multimodal calibration features, which not only allows the image processing algorithm to have more auxiliary information when extracting the traditional checkerboard corner points, but also when there is occlusion or illumination changes in some areas, other positioning marks can still be effectively identified.

[0037] The camera collects the calibration plate image in real time, and performs denoising, brightness adjustment, edge enhancement and other processing on the calibration plate image through the preprocessing module to improve the effect of subsequent feature extraction. The checkerboard corner point detection algorithm (such as findChessboardCorners and cornerSubPix functions in OpenCV) is used to extract the checkerboard corner points. The positioning mark is identified and located using a multi-template matching algorithm. The multi-template matching technology uses the pre-built positioning mark templates to scan each template in the image, and uses the correlation index to judge the matching degree, so as to accurately locate the position and contour of each positioning mark. Through the above-mentioned dual extraction method, the multimodal calibration features include traditional checkerboard corner points and positioning marks. Data redundancy ensures that when a feature is missing in a specific case, it can still be accurately calibrated through other features. For example, in one experiment, the system designed 10 different positioning marks on the same calibration plate, and combined with the checkerboard corner points, a total of more than 60 feature points were extracted. Through the above method, after calculation by the PnP (Perspective-n-Point) algorithm, the pose parameters of the calibration plate in the camera coordinate system are obtained, and its reprojection error is lower than the preset threshold.

[0038] This embodiment uses multimodal calibration features to achieve accurate calculation of the calibration plate's posture, which not only enhances the reliability of traditional checkerboard features in the image, but also supplements additional information through positioning marks, thereby improving the calibration accuracy under complex lighting and partial occlusion conditions.

[0039] In one embodiment, Figure 4 As shown, the following steps S41-S42 are also included: In step S41, the correction rotation angle and displacement compensation parameters are analyzed according to the systematic deviations caused by the camera hardware, ambient lighting and the state of the robot arm; In step S42, the rotation matrix and the translation vector are corrected by using the correction rotation angle and the displacement compensation parameters to improve the accuracy of hand-eye calibration.

[0040] In one embodiment, a technical solution for correcting the rotation angle and displacement compensation parameters is proposed to address the systematic deviations caused by factors such as camera hardware characteristics, changes in ambient light, and fluctuations in the state of the robotic arm, so as to correct the rotation matrix and translation vector obtained by the preliminary calculation and improve the overall accuracy of hand-eye calibration. By collecting multiple sets of calibration data, the systematic errors caused by camera intrinsic parameter deviations, lens distortion, uneven lighting, and changes in the working state of the robotic arm (such as joint wear, motion instability, etc.) are analyzed. These errors will manifest as systematic deviations in rotation angles and displacements in multiple different postures.

[0041] In the data collection stage, the robot control system records the calibration plate image and the end position of the robotic arm collected under the preset trajectory. Through the image processing algorithm, the initial position of the calibration plate in the camera coordinate system is calculated using the traditional checkerboard corner detection technology, and compared with the theoretical position recorded in the robotic arm coordinate system, so as to obtain the reprojection error or position error of each set of data. The analysis results show that the errors often show a certain regularity. For example, in most of the collected data, there is a fixed deviation value for the rotation angle, and there is generally a certain value offset in the translation amount in a certain direction.

[0042] Based on the above analysis, an error model is further constructed to quantitatively describe the systematic deviation. The error distribution of all collected data is statistically processed, and the average deviation and variance are extracted to determine the correction rotation angle and displacement compensation parameters. The correction rotation angle is the angle adjustment amount that minimizes the error after rotation matrix compensation, and the displacement compensation parameter represents the offset that needs to be added or subtracted on the translation vector. In order to further improve the correction effect, this embodiment adopts a nonlinear least squares method, and through iterative optimization and solution, the compensation parameters can reduce the reprojection error of all collected data to below the preset threshold.

[0043] For example, in one experiment, after preliminary calibration, it was found that the average deviation of the calculated rotation angle was positive 5° relative to the theoretical posture, and the translation vector had a positive offset of 2 mm in the X-axis direction. Through nonlinear optimization calculations, it was found that the correction rotation angle was -5° and the X-axis displacement compensation parameter was -2 mm. Subsequently, the compensation parameters were applied to the rotation matrix and translation vector obtained by the preliminary calculation to generate the corrected hand-eye calibration parameters. After verifying the corrected parameters, the results showed that the reprojection error of each set of data was greatly reduced, and the overall calibration accuracy was significantly improved.

[0044] In addition, extended compensation measures for systematic errors are also provided. When collecting data, the system simultaneously monitors the camera hardware status (such as focal length, lens temperature), ambient light intensity, and real-time motion parameters of the robot arm, and uses these auxiliary data as input to further refine the error model. By introducing these additional variables, the error change trend caused by environmental or hardware fluctuations can be captured more accurately, so that the correction rotation angle and displacement compensation parameters can be adjusted in real time, so that the hand-eye calibration process can maintain high accuracy under different working conditions.

[0045] This embodiment performs a comprehensive analysis of the systematic deviations and uses a nonlinear optimization method to calculate the correction rotation angle and displacement compensation parameters, and then corrects the rotation matrix and translation vector obtained by the preliminary calibration. This not only effectively reduces the systematic errors caused by the camera hardware, ambient lighting, and the state of the robotic arm, but also further improves the correction accuracy by introducing additional variables.

[0046] In one embodiment, Figure 5 As shown, the following steps S51-S52 are also included: In step S51, a preset machine learning model is trained using a neural network algorithm through a priori training data set to obtain an error compensation amount for hand-eye calibration; In step S52, the rotation matrix and the translation vector are corrected by using the error compensation amount to improve the accuracy of hand-eye calibration.

[0047] In one embodiment, in view of the systematic errors introduced by various factors such as the camera system, environmental factors, and the state of the robotic arm during the robot hand-eye calibration process, it is proposed to use a priori training data set and a neural network algorithm to build a machine learning model to predict the error compensation amount, and apply the compensation amount to the rotation matrix and translation vector to further improve the accuracy of hand-eye calibration. A large number of hand-eye calibration data samples are collected by conducting multiple calibration experiments in the robot system. Each data sample includes the calibration plate image collected by the camera, the position data of the end of the robotic arm, and the rotation matrix and translation vector calculated by the preliminary calibration algorithm, and the deviation value measured in the actual operation (such as reprojection error or positioning error) is recorded. In order to ensure the representativeness and diversity of the data, the data is collected under different ambient lighting, different robotic arm movement speeds, and different camera temperature conditions during the data collection process, and the collected data is preprocessed, such as normalization and noise filtering, to construct a priori training data set. Based on the training data set, a machine learning model is constructed and trained by a neural network algorithm. The input of the neural network model includes the characteristic parameters of the collected data, such as the rotation matrix and translation vector obtained by preliminary calibration, the camera image quality index, the ambient light value, and the dynamic state parameters of the robotic arm; the output is the predicted error compensation, which includes the rotation angle deviation compensation and the displacement compensation. During the training process, the supervised learning method is used to minimize the mean square error between the model output and the actual calibration error, and the network weights are continuously adjusted until the model achieves a high accuracy in predicting the calibration error under different environments. In order to enhance the robustness of the model, cross-validation, regularization, and data augmentation techniques can also be used to prevent overfitting.

[0048] After the model training is completed, the preset machine learning model is integrated into the robot calibration system. In the actual hand-eye calibration process, the preliminary rotation matrix and translation vector are obtained according to the traditional method, and then the real-time environmental parameters and the preliminary calibration results are collected as input and sent to the neural network model. The model outputs the error compensation amount according to the input features, indicating the systematic error caused by factors such as camera intrinsic parameter deviation, ambient light change or dynamic instability of the robot arm. According to the error compensation amount, the preliminary calculated rotation matrix is ​​rotated and corrected accordingly, and the translation vector is translated to obtain the final hand-eye calibration parameters. For example, in an experiment, the preliminary calibration results showed that the rotation matrix had a deviation of about 3° and the translation vector had a positive offset of about 1.5 mm on the X-axis. After prediction by the trained neural network model, the output error compensation amount was -3° for the corrected rotation angle and -1.5 mm for the displacement compensation. After applying this compensation, the reprojection error of the verified system was significantly reduced, and the final calibration accuracy was significantly improved.

[0049] This embodiment uses prior training data and a neural network algorithm to build an error compensation model to achieve intelligent correction of the preliminary calibration results, significantly reducing systematic errors and improving the accuracy and robustness of hand-eye calibration.

[0050] In one embodiment, Figure 6 As shown, the following steps S61-S62 are also included: In step S61, the overlapping area between the camera field of view and the motion area of ​​the robot arm is analyzed, and the overlapping area is divided into at least four overlapping sub-areas; In step S62, the motion trajectory of the robot arm is optimized so that the motion trajectory can cover all overlapping sub-areas without repeated coverage.

[0051] In one embodiment, in order to solve the data collection problem of the robot arm and the camera during the calibration process, a motion trajectory optimization method based on the overlap area between the camera field of view and the robot arm motion area is proposed. By geometrically modeling the camera installation position and its field of view and the robot arm workspace, the overlap area between the two is determined, that is, the area where the camera can stably collect the image of the calibration plate at the end of the robot arm. The overlapping area is divided and evenly divided into at least four sub-areas, and each sub-area is ensured to contain sufficient calibration plate image acquisition information. For example, if the camera field of view is rectangular, it can be divided into four parts in the horizontal and vertical directions, thereby forming sixteen sub-areas, ensuring that representative image data is collected in each sub-area.

[0052] After the division is completed, the motion planning algorithm is used to optimize the motion trajectory of the robot arm, so that the calibration plate can cover all overlapping sub-areas during the movement, and the data collection between each sub-area is not repeated. In the specific implementation, the geometric boundary of the overlapping area is calculated through the camera internal parameters and the robot arm kinematic parameters, and then the overlapping area is divided into several sub-areas according to the boundary coordinates. According to the position and area of ​​each sub-area, a genetic algorithm or a greedy algorithm is used to design an optimal motion path. This path not only meets the shortest stroke requirements, but also ensures that the robot arm moves smoothly without drastic acceleration and deceleration, thereby reducing image blur or data inconsistency caused by dynamic changes during the acquisition process.

[0053] For example, in one experiment, through path optimization, the calibration plate at the end of the robot arm passed through twenty-five sub-areas in turn, and each sub-area collected at least two sets of image data with different postures, ensuring the uniform distribution of the collected data. After optimization, the overlapping data collection of each sub-area avoids local over-sampling or data redundancy, thereby significantly improving the accuracy of posture calculation based on image features during hand-eye calibration. In order to further enhance the robustness of the system, this embodiment can also monitor the quality of the collected images in each sub-area in real time during the movement. When it is detected that the image quality of a sub-area is low, the robot arm movement trajectory is automatically adjusted to re-cover the sub-area to ensure that the final collected data is complete and balanced.

[0054] This embodiment divides the overlapping area between the camera field of view and the robot arm movement area in detail, and adopts a motion trajectory optimization algorithm, so that the robot arm movement path can cover all sub-areas without repeating the coverage, which not only improves the representativeness of the collected data, but also provides a reliable data basis for subsequent hand-eye calibration based on image processing.

[0055] In one embodiment, Fig. 9 FIG. 1 is a block diagram of a hand-eye calibration device for a robotic arm according to an exemplary embodiment. Fig. 9 As shown, the hand-eye calibration device for the robotic arm includes a control module 91, a collection module 92, a calculation module 93 and a calibration module 94, and is applied to a robot with a robotic arm.

[0056] The control module 91 is used to fix the calibration plate at the end of the robot arm and control the robot arm to move along a preset trajectory so that the calibration plate is always within the field of view of the camera; The acquisition module 92 is used to acquire the calibration plate image in at least three different postures of the robotic arm, and simultaneously record the position and posture of the calibration plate at the end of the robotic arm in the robotic arm coordinate system; The calculation module 93 is used to calculate the position and posture of the calibration plate in the camera coordinate system through the calibration plate corner points in the calibration plate image; The calibration module 94 is used to calculate the rotation matrix and translation vector between the robot coordinate system and the camera coordinate system according to the position and posture of the corresponding calibration plate in the robot coordinate system and the camera coordinate system to complete the hand-eye calibration.

[0057] The control module 91, the acquisition module 92, the calculation module 93 and the calibration module 94 included in the block diagram of the hand-eye calibration device for a robotic arm are controlled to execute the hand-eye calibration method for the robotic arm described in any of the above embodiments.

[0058] like Fig.10As shown, the present invention provides an electronic device 1000, which includes: a communication interface, a processor 1001, and a memory 1002; Among them, the memory 1002 is used to store program instructions. When the program instructions are executed by the processor 1001 that is communicatively connected to the memory 1002 through the communication interface, a calibration plate is fixed at the end of the robotic arm, and the robotic arm is controlled to move along a preset trajectory so that the calibration plate is always in the field of view of the camera; the calibration plate image is collected in at least three different postures of the robotic arm, and the position and posture of the calibration plate at the end of the robotic arm in the robotic arm coordinate system are recorded at the same time; the position and posture of the calibration plate in the camera coordinate system are calculated through the corner points of the calibration plate in the calibration plate image; according to the corresponding position and posture of the calibration plate in the robotic arm coordinate system and the position and posture of the calibration plate in the camera coordinate system, the rotation matrix and translation vector between the robotic arm coordinate system and the camera coordinate system are calculated to complete hand-eye calibration.

[0059] The present invention provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, a calibration plate is fixed at the end of a robotic arm, and the robotic arm is controlled to move along a preset trajectory so that the calibration plate is always within the field of view of a camera; the image of the calibration plate is captured under at least three different postures of the robotic arm, and the position and posture of the calibration plate at the end of the robotic arm in the robotic arm coordinate system are recorded simultaneously; the position and posture of the calibration plate in the camera coordinate system are calculated through the corner points of the calibration plate in the calibration plate image; and the rotation matrix and translation vector between the robotic arm coordinate system and the camera coordinate system are calculated according to the corresponding position and posture of the calibration plate in the robotic arm coordinate system and the position and posture of the calibration plate in the camera coordinate system to complete hand-eye calibration.

[0060] It should be understood that the specific features, operations and details described hereinabove about the method of the present invention may also be similarly applied to the device and system of the present invention, or, vice versa. In addition, each step of the method of the present invention described above may be performed by the corresponding parts or units of the device or system of the present invention.

[0061] It should be understood that each module / unit of the device of the present invention can be implemented in whole or in part by software, hardware, firmware or a combination thereof. Each module / unit can be embedded in the processor of the computer device in the form of hardware or firmware or independent of the processor, or can be stored in the memory of the computer device in the form of software for the processor to call to perform the operation of each module / unit. Each module / unit can be implemented as an independent component or module, or two or more modules / units can be implemented as a single component or module.

[0062] In one embodiment, a computer device is provided, which includes a memory and a processor, and the memory stores computer instructions executable by the processor, and the computer instructions instruct the processor to execute each step of the method of the embodiment of the present invention when executed by the processor. The computer device can be a server, a terminal, or any other electronic device with necessary computing and / or processing capabilities in a broad sense. In one embodiment, the computer device may include a processor, a memory, a network interface, a communication interface, etc. connected through a system bus. The processor of the computer device can be used to provide necessary computing, processing and / or control capabilities. The memory of the computer device may include a non-volatile storage medium and an internal memory. An operating system, a computer program, etc. may be stored in or on the non-volatile storage medium. The internal memory can provide an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface and the communication interface of the computer device can be used to connect and communicate with external devices through a network. The steps of the method of the present invention are executed by the processor.

[0063] The present invention may be implemented as a computer-readable storage medium having a computer program stored thereon, which causes the steps of the method of an embodiment of the present invention to be executed when executed by a processor. In one embodiment, the computer program is distributed on a plurality of computer devices or processors coupled to a network so that the computer program is stored, accessed, and executed in a distributed manner by one or more computer devices or processors. A single method step / operation, or two or more method steps / operations, may be performed by a single computer device or processor or by two or more computer devices or processors. One or more method steps / operations may be performed by one or more computer devices or processors, and one or more other method steps / operations may be performed by one or more other computer devices or processors. One or more computer devices or processors may perform a single method step / operation, or perform two or more method steps / operations.

[0064] It will be appreciated by those skilled in the art that the method steps of the present invention may be performed by instructing related hardware such as a computer device or a processor through a computer program, and the computer program may be stored in a non-temporary computer-readable storage medium, which causes the steps of the present invention to be performed when the computer program is executed. Depending on the circumstances, any reference to memory, storage, database, or other media herein may include non-volatile and / or volatile memory. Examples of non-volatile memory include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, etc. Examples of volatile memory include random access memory (RAM), external cache memory, etc.

[0065] The various technical features described above can be combined arbitrarily. Although all possible combinations of these technical features are not described, any combination of these technical features should be considered to be covered by this specification as long as there is no contradiction in such combination.

[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hand-eye calibration method for a robotic arm, characterized in that: Applications to robots with robotic arms, including: Fixing a calibration plate at the end of the robotic arm, and controlling the robotic arm to move along a preset trajectory so that the calibration plate is always within the field of view of the camera; Under at least three different postures of the robotic arm, the calibration plate image is collected, and the position and posture of the calibration plate at the end of the robotic arm in the robotic arm coordinate system are recorded; Calculating the position and posture of the calibration plate in the camera coordinate system through the calibration plate corner points in the calibration plate image; According to the position and posture of the corresponding calibration plate in the robot arm coordinate system and the camera coordinate system, the rotation matrix and the translation vector between the robot arm coordinate system and the camera coordinate system are calculated to complete the hand-eye calibration.

2. The hand-eye calibration method of a robotic arm as claimed in claim 1, characterized in that: Also includes: Acquire a color image and a depth image of the calibration plate simultaneously by using a depth camera; The three-dimensional position and posture of the calibration plate in the camera coordinate system is calculated according to the three-dimensional coordinates of the corner points of the calibration plate in the calibration plate image with depth information.

3. The hand-eye calibration method of a robotic arm as claimed in claim 1, characterized in that: Also includes: Presetting a plurality of positioning marks with different geometric shapes and optical features on the calibration plate to form a multi-modal calibration feature with the pattern of the calibration plate; Extracting multimodal calibration features from the calibration plate image captured by the camera, wherein the multimodal calibration features include corner points of the calibration plate and the positioning marks; The position and posture of the calibration plate in the camera coordinate system are calculated according to the multimodal calibration features.

4. The hand-eye calibration method of a robotic arm as claimed in claim 1, characterized in that: Also includes: Analyze the correction rotation angle and displacement compensation parameters based on the systematic deviations caused by the camera hardware, ambient lighting, and robot arm status; The rotation matrix and the translation vector are corrected by the correction rotation angle and the displacement compensation parameter to improve the accuracy of the hand-eye calibration.

5. The hand-eye calibration method of a robotic arm as claimed in claim 1, characterized in that: Also includes: Through the prior training data set, the preset machine learning model is trained using the neural network algorithm to obtain the error compensation amount of hand-eye calibration; The rotation matrix and the translation vector are corrected by the error compensation amount to improve the accuracy of hand-eye calibration.

6. The hand-eye calibration method of a robot arm as claimed in claim 1, characterized in that: Also includes: Analyze the overlap area between the camera field of view and the motion area of ​​the robot arm, and divide the overlap area into at least four overlap sub-areas; The motion trajectory of the robot arm is optimized so that the motion trajectory can cover all overlapping sub-areas without repeated coverage.

7. A hand-eye calibration device for a robotic arm, characterized in that: Applications to robots with robotic arms, including: A control module is used to fix a calibration plate at the end of the robotic arm and control the robotic arm to move along a preset trajectory so that the calibration plate is always within the field of view of the camera; An acquisition module, used to acquire the calibration plate image in at least three different postures of the robotic arm, and simultaneously record the position and posture of the calibration plate at the end of the robotic arm in the robotic arm coordinate system; A calculation module, used to calculate the position and posture of the calibration plate in the camera coordinate system through the calibration plate corner points in the calibration plate image; The calibration module is used to calculate the rotation matrix and translation vector between the robotic arm coordinate system and the camera coordinate system according to the position and posture of the corresponding calibration plate in the robotic arm coordinate system and the camera coordinate system to complete the hand-eye calibration.

8. The hand-eye calibration device for a robotic arm as claimed in claim 7, characterized in that: The control module, the acquisition module, the calculation module and the calibration module are controlled to execute the hand-eye calibration method of the robotic arm as described in any one of claims 1-6.

9. An electronic device, characterized in that: include: Communication interface, processor, memory; Wherein, the memory is used to store program instructions, and when the program instructions are executed by the processor that is communicatively connected to the memory through the communication interface, the electronic device implements the hand-eye calibration method for the robotic arm as described in any one of claims 1 to 6.

10. A robot with a mechanical arm, which can execute the method according to any one of claims 1 to 6, or comprises the device according to claims 7 to 8, or has the electronic device according to claim 9.

Images