A robot online hand-eye calibration method based on multi-frame pseudo-label data enhancement
The robot online hand-eye calibration method addresses the limitations of existing methods by using multi-frame pseudo-label data enhancement to improve model generalization and reduce annotation costs, enabling efficient and accurate calibration for industrial applications.
Patent Information
- Application Number
- CN202210607581.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-31
- Publication Date
- 2024-12-17
- Estimated Expiration
- 2042-05-31
AI Technical Summary
Existing hand-eye calibration methods for robots face challenges with poor generalization of models trained using synthetic data and high real data annotation costs, making them cumbersome and difficult to deploy in industrial production.
A robot online hand-eye calibration method based on multi-frame pseudo-label data enhancement, utilizing a semi-supervised learning approach that generates synthetic data through domain randomization, preprocesses data to enhance key point detection, and adjusts the model with enhanced pseudo-labels from real scene data to improve accuracy.
The method enables efficient online hand-eye calibration with improved accuracy and reduced real data annotation costs, allowing for real-time deployment and enhanced performance in real-world scenarios.
Smart Images
Figure CN115008454B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image data processing technology, and in particular to a robot online hand-eye calibration method, system, terminal and computer-readable storage medium based on multi-frame pseudo-label data enhancement. Background Art
[0002] Giving robots vision can give them the flexibility to adapt to complex and novel tasks. However, how to obtain the posture relationship between the robot and an external camera is a fundamental problem in robot vision research. This problem is also called the hand-eye calibration problem (for example, the robot is regarded as a hand and the external camera is regarded as an eye).
[0003] For the hand-eye calibration problem, taking the eye to hand as an example, the traditional solution is to first fix the position of the robot base and the camera in the world coordinate system, and then change the robot end posture. Through the invariance between the robot base and the camera position, an equation of the form AX = BX is established and solved, where: Indicates the transformation relationship from the base to the end when in posture 1. Indicates the transformation relationship from the end to the base when in posture 2. Indicates the transformation relationship from the calibration plate to the camera when in posture 1. Indicates the transformation relationship from the camera to the calibration plate when in posture 2. Represents the transformation relationship from camera to base.
[0004] The final solution X is the posture relationship between the robot base and the camera, that is, the hand-eye relationship. For the solution of AX=BX, the specific solution methods include separation solution method, synchronous solution method, and iterative solution method. Although the hand-eye relationship solved by the above method can meet the basic robot operation application, its calibration process is very cumbersome and needs to be performed offline. There is no doubt that it is difficult to quickly deploy robots in industrial production by relying on traditional methods. Therefore, it is necessary to explore the online hand-eye calibration method of robots.
[0005] In recent years, with the rise of deep neural networks and researchers' continuous in-depth research on online hand-eye calibration methods, many online hand-eye calibration methods based on deep neural networks have emerged to address the cumbersome steps of traditional hand-eye calibration algorithms. These methods usually train a key point detection network and combine robot forward kinematics with the PnP algorithm (Perspective-n-Point is a method for solving 3D to 2D point pair motion. When n 3D space points and their projection positions are known, the camera's position is estimated. The 3D position of the feature point can be determined by triangulation or the depth map of the RGB-D camera. Therefore, in the binocular or RGB-D visual odometer, PnP can be used directly to estimate the camera motion) to complete the hand-eye calibration. However, the current hand-eye calibration algorithm based on deep learning has the problems of poor generalization of the model trained only using synthetic data and high real data annotation cost.
[0006] Therefore, the prior art still needs to be improved and developed. Summary of the invention
[0007] The main purpose of the present invention is to provide a robot online hand-eye calibration method, system, terminal and computer-readable storage medium based on multi-frame pseudo-label data enhancement, aiming to solve the problems of poor generalization of the model trained by only using synthetic data and high real data annotation cost in the hand-eye calibration algorithm based on deep learning in the prior art.
[0008] To achieve the above object, the present invention provides a robot online hand-eye calibration method based on multi-frame pseudo-label data enhancement, and the robot online hand-eye calibration method based on multi-frame pseudo-label data enhancement comprises the following steps:
[0009] Generate multiple robot model animation sequences, randomize the robot model animation sequences by domain, and output multiple synthetic robot images and key point labels as synthetic data sets;
[0010] Preprocessing the synthetic data set, using the preprocessed synthetic data set to train a key point detection model to obtain a pre-trained key point detection model, extracting feature information of the image based on an encoder of the pre-trained key point detection model to obtain a feature map, then using a decoder to map the feature map into multiple confidence maps, and obtaining key point coordinates according to the confidence map;
[0011] Collect robot key point images in real scenes, use the pre-trained key point detection model to predict the collected robot key point images, use the prediction results as original key point pseudo labels, use the EPnP algorithm to solve the hand-eye relationship by combining the original key point pseudo labels of multiple frames of images, and then use the hand-eye relationship to reproject the key points to obtain enhanced key point pseudo label data, and use the enhanced key point pseudo label data to adjust the pre-trained key point detection model to obtain the final robot key point detection model;
[0012] A three-dimensional coordinate set of key points, a two-dimensional key point set and a camera intrinsic parameter are calculated, and online hand-eye calibration is completed based on the three-dimensional coordinate set of key points, the two-dimensional key point set and the camera intrinsic parameter and the robot key point detection model.
[0013] Optionally, in the robot online hand-eye calibration method based on multi-frame pseudo-label data enhancement, the step of generating multiple robot model animation sequences specifically includes:
[0014] Import the STL 3D model of the robot into the blender simulation software that supports the STL format, set the key points whose coordinates need to be recorded on the robot model in the blender simulation software to complete the key point pre-definition, and establish the connection relationship between the connecting rods so that the robot joints can rotate arbitrarily;
[0015] The Python interface in the Blender simulation software is used to automatically generate a preset number of robot model animation sequences with arbitrary rotations of each joint.
[0016] Optionally, the robot online hand-eye calibration method based on multi-frame pseudo-label data enhancement, wherein the robot model animation sequence is domain randomized and multiple synthetic robot images and key point labels are output as a synthetic data set, specifically comprising:
[0017] The robot model animation sequence generated by the blender simulation software is imported into the UE4 game engine in the fbx format, and field randomization is performed in the UE4 game engine; the field randomization includes randomizing the robot joint angles, randomizing the shooting background, randomizing the colors of various parts of the robot, and randomizing the relative posture of the robot and the camera;
[0018] Use the virtual camera of the UE4 game engine to take pictures, and use the NDDS plug-in to record the key point coordinates of each frame of the picture, and output a preset number of synthetic robot pictures and key point two-dimensional coordinate labels as a synthetic data set.
[0019] Optionally, the robot online hand-eye calibration method based on multi-frame pseudo-label data enhancement, wherein the synthetic data set is preprocessed, the key point detection model is trained using the preprocessed synthetic data set to obtain a pre-trained key point detection model, an encoder based on the pre-trained key point detection model extracts feature information of the image to obtain a feature map, and then a decoder is used to map the feature map into multiple confidence maps, and the key point coordinates are obtained according to the confidence map, specifically including:
[0020] Using the synthetic data set as a training set, and preprocessing the training set;
[0021] Using the preprocessed training set to train the key point detection model to obtain a pre-trained key point detection model;
[0022] The encoder based on the pre-trained key point detection model extracts feature information of the image to obtain a feature map, and then uses a decoder to map the feature map into multiple confidence maps;
[0023] After denoising each confidence map based on Gaussian smoothing, the coordinates of the peak are taken to obtain the 2D coordinates of each key point of the robot in the image.
[0024] Optionally, in the robot online hand-eye calibration method based on multi-frame pseudo-label data enhancement, the number of channels of the confidence map is the number of key points, and the value of the confidence map is the probability that the pixel is a key point.
[0025] Optionally, the robot online hand-eye calibration method based on multi-frame pseudo-label data enhancement, wherein the robot key point image is collected in a real scene, the collected robot key point image is predicted using the pre-trained key point detection model, the prediction result is used as the original key point pseudo label, the hand-eye relationship is solved by using the EPnP algorithm by combining the original key point pseudo labels of multiple frames of images, and the key points are reprojected using the hand-eye relationship to obtain enhanced key point pseudo-label data, and the enhanced key point pseudo-label data is used to adjust the pre-trained key point detection model to obtain the final robot key point detection model, specifically including:
[0026] In a real scene, while keeping the relative posture between the robot base and the camera unchanged, change the robot joint angle and collect multiple frames of images as a set of data, then change the camera angle and repeatedly collect multiple sets of robot key point images;
[0027] The pre-trained key point detection model is used to predict the collected key point images of the robot, and the prediction results are used as the original key point pseudo labels. The specific description is as follows:
[0028]
[0029]
[0030] Among them, I represents the robot image, θ represents the model parameters, and F represents the model detection process. Represents the pixel coordinates of the nth key point on the image, represents the original pseudo labels of all key points on the mth image, L ori Represents the set of all original keypoint pseudo labels on a set of data;
[0031] The set L of all the original keypoint pseudo labels on a set of data ori The three-dimensional coordinate point set P corresponding to the pseudo label label And the camera internal parameter K, using the EPnP algorithm to solve, get the original hand-eye relationship pass and K transform the three-dimensional coordinate point set P label Reprojection is performed to obtain the initially optimized pseudo label set L rep , and calculate L ori With L rep The Euclidean distance of each element in is expressed as follows:
[0032]
[0033]
[0034]
[0035]
[0036] Among them, m is the number of pictures, n is the number of key points, Represents the reprojected pseudo labels of all key points on the i-th image of this group of data, represents the reprojected pseudo-label of the jth key point on the image, and W is the Euclidean distance between the original pseudo-label and the reprojected pseudo-label of a key point;
[0037] Set the threshold, and according to the W of each key point, convert the set L of the original key point pseudo labels ori The outliers whose Euclidean distance between the original pseudo-label and the reprojected pseudo-label is greater than the threshold are removed, and the new hand-eye relationship is calculated using the EPnP algorithm. use The three-dimensional coordinate set of key points in the robot base coordinate system b The enhanced pseudo-label data of robot key points is obtained by reprojecting P, which is specifically described as follows:
[0038]
[0039] Among them, Lopt Represents the enhanced pseudo-label data of robot key points;
[0040] The enhanced robot key point pseudo-label data is added to the training set, and the pre-trained key point detection model is adjusted according to the updated training set to obtain the final robot key point detection model.
[0041] Optionally, the robot online hand-eye calibration method based on multi-frame pseudo-label data enhancement, wherein the calculation obtains a three-dimensional coordinate set of key points, a two-dimensional key point set and a camera intrinsic parameter, and completes the online hand-eye calibration based on the three-dimensional coordinate set of key points, the two-dimensional key point set and the camera intrinsic parameter, and based on the robot key point detection model, specifically includes:
[0042] According to the angle sensor data and positive kinematics of each joint of the robot, the three-dimensional coordinate set of key points in the predefined n robot base coordinate system is solved. b P, as follows:
[0043]
[0044] The camera is calibrated using Zhang Zhengyou's camera calibration method to obtain the camera intrinsic parameter K. The complete matrix of the camera intrinsic parameter K is expressed as follows:
[0045]
[0046] Among them, f x and f y represents the focal length of the camera described in pixels, c x and c y Indicates the offset of the origin of the physical imaging plane on the u-axis and v-axis;
[0047] Use a camera to take pictures of the robot, and use the final robot key point detection model to predict the two-dimensional coordinates of the key points in the RGB image, as shown below:
[0048]
[0049] Among them, p represents a set of n two-dimensional key points in the image, p i Represents the position coordinates of the i-th key point in the image;
[0050] The three-dimensional coordinates of the key points are set b P, a set of two-dimensional key points p and the camera intrinsic parameter K are used as input, and the EPnP algorithm is used to solve the hand-eye relationship Online hand-eye calibration using a single image.
[0051] In addition, to achieve the above-mentioned purpose, the present invention also provides a robot online hand-eye calibration system based on multi-frame pseudo-label data enhancement, wherein the robot online hand-eye calibration system based on multi-frame pseudo-label data enhancement comprises:
[0052] A synthetic data set generation module, used to generate multiple robot model animation sequences, randomize the robot model animation sequences in different fields, and output multiple synthetic robot images and key point labels as synthetic data sets;
[0053] A feature extraction module, used to preprocess the synthetic data set, train a key point detection model using the preprocessed synthetic data set to obtain a pre-trained key point detection model, extract feature information of the image based on the encoder of the pre-trained key point detection model to obtain a feature map, then use a decoder to map the feature map into multiple confidence maps, and obtain key point coordinates according to the confidence map;
[0054] A data enhancement and fine-tuning module is used to collect robot key point images in real scenes, use the pre-trained key point detection model to predict the collected robot key point images, use the prediction results as original key point pseudo labels, use the EPnP algorithm to solve the hand-eye relationship by combining the original key point pseudo labels of multiple frames of images, and then use the hand-eye relationship to reproject the key points to obtain enhanced key point pseudo label data, and use the enhanced key point pseudo label data to adjust the pre-trained key point detection model to obtain the final robot key point detection model;
[0055] The online hand-eye calibration module is used to calculate the three-dimensional coordinate set of key points, the two-dimensional key point set and the camera intrinsic parameters, and complete the online hand-eye calibration based on the three-dimensional coordinate set of key points, the two-dimensional key point set and the camera intrinsic parameters and the robot key point detection model.
[0056] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a robot online hand-eye calibration program based on multi-frame pseudo-label data enhancement stored in the memory and run on the processor, and when the robot online hand-eye calibration program based on multi-frame pseudo-label data enhancement is executed by the processor, the steps of the robot online hand-eye calibration method based on multi-frame pseudo-label data enhancement as described above are implemented.
[0057] In addition, to achieve the above-mentioned objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a robot online hand-eye calibration program based on multi-frame pseudo-label data enhancement, and when the robot online hand-eye calibration program based on multi-frame pseudo-label data enhancement is executed by a processor, the steps of the robot online hand-eye calibration method based on multi-frame pseudo-label data enhancement as described above are implemented.
[0058] The present invention adopts the pseudo-label idea in semi-supervised learning to mark the key points in the real robot image without the key point two-dimensional coordinate label, and in view of the problem that the pseudo-label often contains wrong labels, a method of joint enhancement of multi-frame pseudo-label data is proposed to improve the accuracy of pseudo-label, thereby improving the performance of the system's online hand-eye calibration. The robot key point pseudo-label processing method adopted by the present invention, especially the process of combining multi-frame robot image information, jointly estimating the hand-eye relationship, and then reprojecting to obtain the label, is a processing method selected through theory and practice. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] Figure 1 It is a flow chart of a preferred embodiment of the robot online hand-eye calibration method based on multi-frame pseudo-label data enhancement of the present invention;
[0060] Figure 2 It is a schematic diagram of predefining key points and establishing connection relationships between connecting rods in a preferred embodiment of the robot online hand-eye calibration method based on multi-frame pseudo-label data enhancement of the present invention;
[0061] Figure 3 It is a schematic diagram of joint enhancement of multiple frames of pseudo-label data in a preferred embodiment of the robot online hand-eye calibration method based on multi-frame pseudo-label data enhancement of the present invention;
[0062] Figure 4 It is a schematic diagram of online hand-eye calibration in a preferred embodiment of the robot online hand-eye calibration method based on multi-frame pseudo-label data enhancement of the present invention;
[0063] Figure 5 It is a schematic diagram of the principle of a preferred embodiment of the robot online hand-eye calibration system based on multi-frame pseudo-label data enhancement of the present invention;
[0064] Figure 6 Schematic diagram of the operating environment of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION
[0065] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0066] The robot online hand-eye calibration method based on multi-frame pseudo-label data enhancement described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the robot online hand-eye calibration method based on multi-frame pseudo-label data enhancement includes the following steps:
[0067] Step S10: generate a plurality of robot model animation sequences, perform domain randomization on the robot model animation sequences, and output a plurality of synthetic robot images and key point labels as synthetic data sets.
[0068] Specifically, the STL three-dimensional model of the robot is imported into the blender simulation software that supports the STL format, and the key points that need to record the coordinates are set on the robot model in the blender simulation software (three-dimensional graphics and image software, providing a series of animation short film production solutions from modeling, animation, materials, rendering, to audio processing, video editing, etc.) to complete the key point pre-definition and establish the connection relationship between the connecting rods (i.e., the parent-child relationship, such as Figure 2 As shown, there are 8 key points, that is, each picture has 8 key points), so that each joint of the robot can rotate arbitrarily; the Python interface in the blender simulation software is used to automatically generate a preset number (for example, 20,000) of robot model animation sequences with arbitrary rotation of each joint.
[0069] Then, the robot model animation sequence generated by the blender simulation software is imported into the UE4 (Unreal Engine 4) game engine in the fbx format, and domain randomization is performed in the UE4 game engine; the domain randomization includes randomizing the robot joint angles, randomizing the shooting background, randomizing the colors of each part of the robot, and randomizing the relative posture of the robot and the camera; finally, the virtual camera of the UE4 game engine is used to take pictures, and the NDDS plug-in is used to record the key point coordinates of each frame of the picture, and finally a preset number (for example, 20,000) of synthetic robot pictures and key point two-dimensional coordinate labels are output as a synthetic data set.
[0070] Step S20, preprocessing the synthetic data set, using the preprocessed synthetic data set to train a key point detection model to obtain a pre-trained key point detection model, extracting feature information of the image based on the encoder of the pre-trained key point detection model to obtain a feature map, and then using a decoder to map the feature map into multiple confidence maps, and obtaining key point coordinates based on the confidence map.
[0071] Specifically, the synthetic data set is used as a training set, for example, the generated 20,000 synthetic data sets are used as training sets. Before network training, the training set is preprocessed, mainly including cropping and rotating the pictures to different degrees, so as to enhance the generalization ability of the model; the preprocessed training set is used to train the key point detection model (Simple A pre-trained key point detection model is obtained by training a baseline key point detection model, which belongs to an encoding-decoding architecture, so that the trained network has a certain robot key point detection capability; due to the domain gap between synthetic scenes and real scenes, the model trained on the synthetic dataset is not effective when applied to real scenes, so the model trained on the synthetic dataset is used as a pre-trained key point detection model, and fine-tuning is subsequently performed on the basis of the pre-trained key point detection model; the encoder based on the pre-trained key point detection model extracts feature information of the image to obtain a feature map, and then uses a decoder to map the feature map into multiple confidence maps, the number of channels of the confidence map is the number of key points, and the value of the confidence map is the probability that the pixel is a key point; after denoising each confidence map based on Gaussian smoothing (reducing the influence of noise), the coordinates of the peak are taken to obtain the 2D coordinates of each key point of the robot in the image.
[0072] Step S30, collecting robot key point images in real scenes, using the pre-trained key point detection model to predict the collected robot key point images, using the prediction results as original key point pseudo labels, solving the hand-eye relationship by combining the original key point pseudo labels of multiple frames of images using the EPnP algorithm, and then using the hand-eye relationship to reproject the key points to obtain enhanced key point pseudo-label data, and using the enhanced key point pseudo-label data to adjust the pre-trained key point detection model to obtain the final robot key point detection model.
[0073] Specifically, Figure 3 As shown in the figure, real scene data collection: in the real scene, set up a camera in the laboratory, keep the relative posture of the robot base and the camera unchanged, change the robot joint angle and collect multiple frames of images as a set of data, then change the camera angle, repeatedly collect multiple groups of robot key point images, so as to provide more perspective information for the subsequent fine-tuning model.
[0074] Generate original pseudo labels: Use the pre-trained key point detection model to predict the collected robot key point images, and use the prediction results as the original key point pseudo labels. The specific description is as follows:
[0075]
[0076]
[0077] Among them, I represents the robot image, θ represents the model parameters, and F represents the model detection process. Represents the pixel coordinates of the nth key point on the image, represents the original pseudo labels of all key points on the mth image, L ori Represents the set of all original keypoint pseudo labels on a set of data.
[0078] Joint enhancement of multi-frame pseudo-label data: The set L of all original key point pseudo-labels on a set of data ori The three-dimensional coordinate point set P corresponding to the pseudo label label And the camera internal parameter K, using the EPnP algorithm to solve, get the original hand-eye relationship like Figure 3 As shown, through and K transform the three-dimensional coordinate point set P label Reproject the pseudo label set L rep (like Figure 3 As shown, in practice, red can be used to represent L ori , yellow indicates L rep ), and calculate L ori With L rep The Euclidean distance of each element in is expressed as follows:
[0079]
[0080]
[0081]
[0082]
[0083] Among them, m is the number of pictures, n is the number of key points, Represents the reprojected pseudo labels of all key points on the i-th image of this group of data, represents the reprojected pseudo-label of the jth key point on the image, and W is the Euclidean distance between the original pseudo-label and the reprojected pseudo-label of a key point;
[0084] like Figure 3 As shown, set the threshold, according to each key point W, the set of original key point pseudo labels L ori The outliers (significant misdetection points) whose Euclidean distance between the original pseudo-label and the reprojected pseudo-label is greater than the threshold are removed, and the new hand-eye relationship is calculated using the EPnP algorithm. Effectively reduce the impact of pseudo labels with large errors from real labels on EPnP solution, improve the accuracy of hand-eye relationship solution, and thus improve the accuracy of pseudo labels. The three-dimensional coordinate set of key points in the robot base coordinate system b The enhanced pseudo-label data of robot key points is obtained by reprojecting P, which is specifically described as follows:
[0085]
[0086] Among them, L opt Represents the enhanced robot key point pseudo-label data.
[0087] By combining multi-frame pseudo-labels with data enhancement, we can not only improve the accuracy of the pose solution results, but also provide pseudo-labels for missed points. Finally, we can fine-tune the model using the enhanced pseudo-label data to enhance the performance of the model in real scenes.
[0088] like Figure 3 As shown, the enhanced robot key point pseudo-label data is added to the training set, and the pre-trained key point detection model is adjusted according to the updated training set (i.e., training is continued on the basis of the pre-trained key point detection model to fine-tune the model). In the process of fine-tuning the model, the model is trained to convergence and then the multi-frame pseudo-label data is jointly enhanced again, and then the model is fine-tuned again. The process of pseudo-label enhancement and fine-tuning the model is iterated until the model converges, and the final robot key point detection model can be obtained.
[0089] Step S40, calculate and obtain a three-dimensional coordinate set of key points, a two-dimensional key point set and a camera intrinsic parameter, and complete online hand-eye calibration based on the three-dimensional coordinate set of key points, the two-dimensional key point set and the camera intrinsic parameter and the robot key point detection model.
[0090] Specifically, Figure 4 As shown in the figure, according to the angle sensor data and positive kinematics of each joint of the robot, the three-dimensional coordinate set of key points in the predefined n robot base coordinate system is solved. b P, as follows:
[0091]
[0092] like Figure 4 As shown, the camera is calibrated using Zhang Zhengyou's camera calibration method to obtain the camera intrinsic parameter K. The complete matrix of the camera intrinsic parameter K is expressed as follows:
[0093]
[0094] Among them, f x and f y represents the focal length of the camera described in pixels, c x and c y Indicates the offset of the origin of the physical imaging plane on the u-axis and v-axis.
[0095] like Figure 4 As shown, a camera is used to take pictures of the robot, and the final robot key point detection model is used to predict the two-dimensional coordinates of the key points in the RGB image, which is expressed as follows:
[0096]
[0097] Among them, p represents a set of n two-dimensional key points in the image, p i Represents the position coordinates of the i-th key point in the image;
[0098] like Figure 4 As shown, the three-dimensional coordinate set of key points b P, a set of two-dimensional key points p and the camera intrinsic parameter K are used as input, and the EPnP algorithm is used to solve the hand-eye relationship Use a single image to complete the online hand-eye calibration (that is, only a single image is used to complete the online hand-eye calibration).
[0099] The present invention discloses an online hand-eye calibration method for a robot based on multi-frame pseudo-label data enhancement. Firstly, the blender simulation software and the UE4 game engine are used to predefine key points and perform various field randomization processing operations, and a synthetic data set is output from the UE4 game engine; then, a key point detection model based on an encoder-decoder network is pre-trained using the synthetic data set to obtain a key point detection model pre-trained on the synthetic data set; then, the pre-trained key point detection model is used to detect the key points of the robot collected in a real scene to obtain the original pseudo-label of each robot image, and the key point pseudo-labels of multiple frame images are combined to obtain a key point pseudo-label set L of the multiple frame images. ori , using the pseudo-label set L of multiple frames ori Perform EPnP posture estimation to obtain the initial hand-eye relationship, and then use the hand-eye relationship to reproject the key points onto the image to obtain the preliminary optimized pseudo label L rep , calculate L ori and L rep The Euclidean distance of each element in is used to set the threshold to remove the original pseudo-label set L ori The significant outliers in the image are then re-estimated by EPnP to obtain a more reliable hand-eye relationship. The hand-eye relationship is used to reproject the key points onto the image to obtain the enhanced robot key point pseudo-label data L opt ; Finally, the enhanced pseudo-label data L opt The training set is added to fine-tune the model to improve the model's key point detection ability in real scenes, thereby improving the system's hand-eye calibration performance. The final robot key point detection model after fine-tuning is combined with forward kinematics and EPnP algorithm to complete the online hand-eye calibration task.
[0100] Beneficial effects:
[0101] (1) Compared with the traditional hand-eye calibration algorithm, the present invention can realize hand-eye calibration with only one image, and has strong real-time performance;
[0102] (2) Compared with the existing online hand-eye calibration algorithm based on deep learning, the present invention adopts the idea of semi-supervised algorithm and introduces high-quality real scene pseudo labels to solve the problems of poor generalization of the model trained by synthetic data and high cost of real data annotation;
[0103] (3) Compared with the existing robot pseudo-label generation method, the present invention uses a multi-frame pseudo-label joint enhancement method to improve the quality of pseudo-labels, thereby improving the accuracy of online hand-eye calibration.
[0104] The solution of the present invention has been tested in practice. When only one picture is used for hand-eye calibration, the error is 4.65cm. It can complete the grasping task in scenes without high precision requirements, and the position of the camera can be changed arbitrarily in vision-based grasping tasks. The present invention can obtain high-quality pseudo-label data to improve the effect of the robot key point detection model, and build a system to complete online hand-eye calibration.
[0105] Furthermore, if Figure 5 As shown, based on the above-mentioned robot online hand-eye calibration method based on multi-frame pseudo-label data enhancement, the present invention also provides a robot online hand-eye calibration system based on multi-frame pseudo-label data enhancement, wherein the robot online hand-eye calibration system based on multi-frame pseudo-label data enhancement includes:
[0106] A synthetic data set generation module 51 is used to generate a plurality of robot model animation sequences, perform domain randomization on the robot model animation sequences, and output a plurality of synthetic robot images and key point labels as synthetic data sets;
[0107] A feature extraction module 52 is used to preprocess the synthetic data set, train a key point detection model using the preprocessed synthetic data set to obtain a pre-trained key point detection model, extract feature information of the image based on an encoder of the pre-trained key point detection model to obtain a feature map, and then use a decoder to map the feature map into multiple confidence maps, and obtain key point coordinates according to the confidence map;
[0108] The data enhancement and fine-tuning module 53 is used to collect robot key point images in real scenes, use the pre-trained key point detection model to predict the collected robot key point images, use the prediction results as original key point pseudo labels, use the EPnP algorithm to solve the hand-eye relationship by combining the original key point pseudo labels of multiple frames of images, and then use the hand-eye relationship to reproject the key points to obtain enhanced key point pseudo label data, and use the enhanced key point pseudo label data to adjust the pre-trained key point detection model to obtain the final robot key point detection model;
[0109] The online hand-eye calibration module 54 is used to calculate the three-dimensional coordinate set of key points, the two-dimensional key point set and the camera intrinsic parameters, and complete the online hand-eye calibration based on the three-dimensional coordinate set of key points, the two-dimensional key point set and the camera intrinsic parameters and based on the robot key point detection model.
[0110] Furthermore, if Figure 6 As shown, based on the above-mentioned robot online hand-eye calibration method and system based on multi-frame pseudo-label data enhancement, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Figure 6 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.
[0111] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a robot online hand-eye calibration program 40 based on multi-frame pseudo-label data enhancement is stored on the memory 20, and the robot online hand-eye calibration program 40 based on multi-frame pseudo-label data enhancement can be executed by the processor 10, thereby realizing the robot online hand-eye calibration method based on multi-frame pseudo-label data enhancement in the present application.
[0112] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the robot online hand-eye calibration method based on multi-frame pseudo-label data enhancement.
[0113] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.
[0114] In one embodiment, when the processor 10 executes the robot online hand-eye calibration program 40 based on multi-frame pseudo-label data enhancement in the memory 20, the following steps are implemented:
[0115] Generate multiple robot model animation sequences, randomize the robot model animation sequences by domain, and output multiple synthetic robot images and key point labels as synthetic data sets;
[0116] Preprocessing the synthetic data set, using the preprocessed synthetic data set to train a key point detection model to obtain a pre-trained key point detection model, extracting feature information of the image based on an encoder of the pre-trained key point detection model to obtain a feature map, then using a decoder to map the feature map into multiple confidence maps, and obtaining key point coordinates according to the confidence map;
[0117] Collect robot key point images in real scenes, use the pre-trained key point detection model to predict the collected robot key point images, use the prediction results as original key point pseudo labels, use the EPnP algorithm to solve the hand-eye relationship by combining the original key point pseudo labels of multiple frames of images, and then use the hand-eye relationship to reproject the key points to obtain enhanced key point pseudo label data, and use the enhanced key point pseudo label data to adjust the pre-trained key point detection model to obtain the final robot key point detection model;
[0118] A three-dimensional coordinate set of key points, a two-dimensional key point set and a camera intrinsic parameter are calculated, and online hand-eye calibration is completed based on the three-dimensional coordinate set of key points, the two-dimensional key point set and the camera intrinsic parameter and the robot key point detection model.
[0119] The step of generating multiple robot model animation sequences specifically includes:
[0120] Import the STL 3D model of the robot into the blender simulation software that supports the STL format, set the key points whose coordinates need to be recorded on the robot model in the blender simulation software to complete the key point pre-definition, and establish the connection relationship between the connecting rods so that the robot joints can rotate arbitrarily;
[0121] The Python interface in the Blender simulation software is used to automatically generate a preset number of robot model animation sequences with arbitrary rotations of each joint.
[0122] The process of randomizing the robot model animation sequence in a domain and outputting a plurality of synthetic robot images and key point labels as a synthetic data set specifically includes:
[0123] The robot model animation sequence generated by the blender simulation software is imported into the UE4 game engine in the fbx format, and field randomization is performed in the UE4 game engine; the field randomization includes randomizing the robot joint angles, randomizing the shooting background, randomizing the colors of various parts of the robot, and randomizing the relative posture of the robot and the camera;
[0124] Use the virtual camera of the UE4 game engine to take pictures, and use the NDDS plug-in to record the key point coordinates of each frame of the picture, and output a preset number of synthetic robot pictures and key point two-dimensional coordinate labels as a synthetic data set.
[0125] The method comprises preprocessing the synthetic data set, training a key point detection model using the preprocessed synthetic data set to obtain a pretrained key point detection model, extracting feature information of a picture based on an encoder of the pretrained key point detection model to obtain a feature map, and then using a decoder to map the feature map into multiple confidence maps, and obtaining key point coordinates according to the confidence map, specifically including:
[0126] Using the synthetic data set as a training set, and preprocessing the training set;
[0127] Using the preprocessed training set to train the key point detection model to obtain a pre-trained key point detection model;
[0128] The encoder based on the pre-trained key point detection model extracts feature information of the image to obtain a feature map, and then uses a decoder to map the feature map into multiple confidence maps;
[0129] After denoising each confidence map based on Gaussian smoothing, the coordinates of the peak are taken to obtain the 2D coordinates of each key point of the robot in the image.
[0130] The number of channels of the confidence map is the number of key points, and the value of the confidence map is the probability that the pixel is a key point.
[0131] The method comprises collecting robot key point images in a real scene, using the pre-trained key point detection model to predict the collected robot key point images, using the prediction results as original key point pseudo labels, solving the hand-eye relationship by combining the original key point pseudo labels of multiple frames of images using the EPnP algorithm, and then using the hand-eye relationship to reproject the key points to obtain enhanced key point pseudo label data, and using the enhanced key point pseudo label data to adjust the pre-trained key point detection model to obtain the final robot key point detection model, which specifically includes:
[0132] In a real scene, while keeping the relative posture between the robot base and the camera unchanged, change the robot joint angle and collect multiple frames of images as a set of data, then change the camera angle and repeatedly collect multiple sets of robot key point images;
[0133] The pre-trained key point detection model is used to predict the collected key point images of the robot, and the prediction results are used as the original key point pseudo labels. The specific description is as follows:
[0134]
[0135]
[0136] Among them, I represents the robot image, θ represents the model parameters, and F represents the model detection process. Represents the pixel coordinates of the nth key point on the image, represents the original pseudo labels of all key points on the mth image, L ori Represents the set of all original keypoint pseudo labels on a set of data;
[0137] The set L of all the original keypoint pseudo labels on a set of data ori The three-dimensional coordinate point set P corresponding to the pseudo label label And the camera internal parameter K, using the EPnP algorithm to solve, get the original hand-eye relationship pass and K transform the three-dimensional coordinate point set P label Reprojection is performed to obtain the initially optimized pseudo label set L rep , and calculate L ori With L rep The Euclidean distance of each element in is expressed as follows:
[0138]
[0139]
[0140]
[0141]
[0142] Among them, m is the number of pictures, n is the number of key points, Represents the reprojected pseudo labels of all key points on the i-th image of this group of data, represents the reprojected pseudo-label of the jth key point on the image, and W is the Euclidean distance between the original pseudo-label and the reprojected pseudo-label of a key point;
[0143] Set the threshold, and according to the W of each key point, convert the set L of the original key point pseudo labels ori The outliers whose Euclidean distance between the original pseudo-label and the reprojected pseudo-label is greater than the threshold are removed, and the new hand-eye relationship is calculated using the EPnP algorithm. use The three-dimensional coordinate set of key points in the robot base coordinate system b The enhanced pseudo-label data of robot key points is obtained by reprojecting P, which is specifically described as follows:
[0144]
[0145] Among them, L opt Represents the enhanced pseudo-label data of robot key points;
[0146] The enhanced robot key point pseudo-label data is added to the training set, and the pre-trained key point detection model is adjusted according to the updated training set to obtain the final robot key point detection model.
[0147] The calculation obtains a three-dimensional coordinate set of key points, a two-dimensional key point set and a camera intrinsic parameter, and completes online hand-eye calibration based on the robot key point detection model according to the three-dimensional coordinate set of key points, the two-dimensional key point set and the camera intrinsic parameter, specifically including:
[0148] According to the angle sensor data and positive kinematics of each joint of the robot, the three-dimensional coordinate set of key points in the predefined n robot base coordinate system is solved. b P, as follows:
[0149]
[0150] The camera is calibrated using Zhang Zhengyou's camera calibration method to obtain the camera intrinsic parameter K. The complete matrix of the camera intrinsic parameter K is expressed as follows:
[0151]
[0152] Among them, f x and f y represents the focal length of the camera described in pixels, c x and c yIndicates the offset of the origin of the physical imaging plane on the u-axis and v-axis;
[0153] Use a camera to take pictures of the robot, and use the final robot key point detection model to predict the two-dimensional coordinates of the key points in the RGB image, as shown below:
[0154]
[0155] Among them, p represents a set of n two-dimensional key points in the image, p i Represents the position coordinates of the i-th key point in the image;
[0156] The three-dimensional coordinates of the key points are set b P, a set of two-dimensional key points p and the camera intrinsic parameter K are used as input, and the EPnP algorithm is used to solve the hand-eye relationship Online hand-eye calibration using a single image.
[0157] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a robot online hand-eye calibration program based on multi-frame pseudo-label data enhancement, and when the robot online hand-eye calibration program based on multi-frame pseudo-label data enhancement is executed by a processor, the steps of the robot online hand-eye calibration method based on multi-frame pseudo-label data enhancement as described above are implemented.
[0158] In summary, the present invention provides a robot online hand-eye calibration method and related equipment based on multi-frame pseudo-label data enhancement, the method comprising: generating multiple robot model animation sequences, randomizing the robot model animation sequences in a domain, and outputting multiple synthetic robot images and key point labels as a synthetic data set; preprocessing the synthetic data set, using the preprocessed synthetic data set to train a key point detection model to obtain a pre-trained key point detection model, extracting feature information of the image based on an encoder of the pre-trained key point detection model to obtain a feature map, and then using a decoder to map the feature map into multiple confidence maps, and obtaining key point coordinates according to the confidence map; collecting robot key point images in real scenes, and using the pre-trained key point detection model to collect the collected The robot key point image is predicted, and the prediction result is used as the original key point pseudo label. The hand-eye relationship is solved by using the EPnP algorithm by combining the original key point pseudo labels of multiple frames of images, effectively reducing the influence of pseudo labels with too large errors with the real labels on the EPnP solution, and improving the accuracy of the hand-eye relationship solution. The key points are reprojected using the hand-eye relationship to obtain enhanced key point pseudo label data, thereby improving the accuracy of the pseudo label, and the enhanced pseudo label data is used to adjust the pre-trained key point detection model to obtain the final robot key point detection model; the key point three-dimensional coordinate set, the two-dimensional key point set and the camera internal parameters are calculated, and the online hand-eye calibration is completed based on the key point three-dimensional coordinate set, the two-dimensional key point set and the camera internal parameters, and based on the robot key point detection model. The present invention combines multiple frames of robot image information, jointly estimates the hand-eye relationship, and then reprojects to obtain the label process, which improves the accuracy of the pseudo label, thereby improving the performance of the system's online hand-eye calibration.
[0159] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or terminal including the element.
[0160] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.
[0161] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.
Claims
1. A robot online hand-eye calibration method based on multi-frame pseudo-label data enhancement, characterized in that: The robot online hand-eye calibration method based on multi-frame pseudo-label data enhancement includes: Generate multiple robot model animation sequences, randomize the robot model animation sequences by domain, and output multiple synthetic robot images and key point labels as synthetic data sets; Preprocessing the synthetic data set, using the preprocessed synthetic data set to train a key point detection model to obtain a pre-trained key point detection model, extracting feature information of the image based on an encoder of the pre-trained key point detection model to obtain a feature map, then using a decoder to map the feature map into multiple confidence maps, and obtaining key point coordinates according to the confidence map; Collect robot key point images in real scenes, use the pre-trained key point detection model to predict the collected robot key point images, use the prediction results as original key point pseudo labels, use the EPnP algorithm to solve the hand-eye relationship by combining the original key point pseudo labels of multiple frames of images, and then use the hand-eye relationship to reproject the key points to obtain enhanced key point pseudo label data, and use the enhanced key point pseudo label data to adjust the pre-trained key point detection model to obtain the final robot key point detection model; Calculate and obtain a three-dimensional coordinate set of key points, a two-dimensional key point set and a camera intrinsic parameter, and complete online hand-eye calibration based on the three-dimensional coordinate set of key points, the two-dimensional key point set and the camera intrinsic parameter and the robot key point detection model; The method includes collecting robot key point images in a real scene, using the pre-trained key point detection model to predict the collected robot key point images, using the prediction results as original key point pseudo labels, solving the hand-eye relationship by combining the original key point pseudo labels of multiple frames of images using the EPnP algorithm, and then using the hand-eye relationship to reproject the key points to obtain enhanced key point pseudo label data, and using the enhanced key point pseudo label data to adjust the pre-trained key point detection model to obtain the final robot key point detection model, specifically including: In a real scene, while keeping the relative posture between the robot base and the camera unchanged, change the robot joint angle and collect multiple frames of images as a set of data, then change the camera angle and repeatedly collect multiple sets of robot key point images; The pre-trained key point detection model is used to predict the collected key point images of the robot, and the prediction results are used as the original key point pseudo labels. The specific description is as follows: ； ； in, Represents a robot picture, represents the model parameters, F represents the model detection process, Represents the pixel coordinates of the nth key point on the image, represents the original pseudo labels of all key points on the mth image, Represents the set of all original keypoint pseudo labels on a set of data; The set of all the original keypoint pseudo labels on a set of data The set of three-dimensional coordinate points corresponding to the pseudo labels and camera intrinsics , use the EPnP algorithm to solve and get the original hand-eye relationship ,pass and The three-dimensional coordinate point set Reprojection is performed to obtain a preliminary optimized pseudo-label set , and calculate and The Euclidean distance of each element in is expressed as follows: ； ； ； ； Among them, m is the number of pictures, n is the number of key points, Represents the reprojected pseudo labels of all key points on the i-th image of this group of data, [ ] represents the reprojected pseudo-label of the jth key point on the image, is the Euclidean distance between the original pseudo label and the reprojected pseudo label of a key point; Set the threshold according to the key points , the set of original key point pseudo labels The outliers whose Euclidean distance between the original pseudo-label and the reprojected pseudo-label is greater than the threshold are removed, and the new hand-eye relationship is calculated using the EPnP algorithm. ,use The three-dimensional coordinate set of key points in the robot base coordinate system The enhanced pseudo-label data of robot key points after reprojection is described as follows: ； in, Represents the enhanced pseudo-label data of robot key points; The enhanced robot key point pseudo-label data is added to the training set, and the pre-trained key point detection model is adjusted according to the updated training set to obtain the final robot key point detection model.
2. According to the method for online hand-eye calibration of a robot based on multi-frame pseudo-label data enhancement according to claim 1, the step of generating multiple robot model animation sequences specifically comprises: Import the STL 3D model of the robot into the blender simulation software that supports the STL format, set the key points whose coordinates need to be recorded on the robot model in the blender simulation software to complete the key point pre-definition, and establish the connection relationship between the connecting rods so that the robot joints can rotate arbitrarily; The Python interface in the Blender simulation software is used to automatically generate a preset number of robot model animation sequences with arbitrary rotations of each joint.
3. According to the method for online hand-eye calibration of a robot based on multi-frame pseudo-label data enhancement according to claim 2, the robot model animation sequence is subjected to domain randomization, and a plurality of synthetic robot images and key point labels are output as a synthetic data set, specifically comprising: The robot model animation sequence generated by the blender simulation software is imported into the UE4 game engine in the fbx format, and field randomization is performed in the UE4 game engine; the field randomization includes randomizing the robot joint angles, randomizing the shooting background, randomizing the colors of various parts of the robot, and randomizing the relative posture of the robot and the camera; Use the virtual camera of the UE4 game engine to take pictures, and use the NDDS plug-in to record the key point coordinates of each frame of the picture, and output a preset number of synthetic robot pictures and key point two-dimensional coordinate labels as a synthetic data set.
4. According to the method for online hand-eye calibration of a robot based on multi-frame pseudo-label data enhancement in claim 3, the method comprises preprocessing the synthetic data set, using the preprocessed synthetic data set to train a key point detection model to obtain a pretrained key point detection model, extracting feature information of an image based on an encoder of the pretrained key point detection model to obtain a feature map, and then using a decoder to map the feature map into multiple confidence maps, and obtaining key point coordinates according to the confidence map, specifically comprising: Using the synthetic data set as a training set, and preprocessing the training set; Using the preprocessed training set to train the key point detection model to obtain a pre-trained key point detection model; The encoder based on the pre-trained key point detection model extracts feature information of the image to obtain a feature map, and then uses a decoder to map the feature map into multiple confidence maps; After denoising each confidence map based on Gaussian smoothing, the coordinates of the peak are taken to obtain the 2D coordinates of each key point of the robot in the image.
5. According to the robot online hand-eye calibration method based on multi-frame pseudo-label data enhancement according to claim 4, the number of channels of the confidence map is the number of key points, and the value of the confidence map is the probability that the pixel of the confidence map is a key point.
6. According to the method for online hand-eye calibration of a robot based on multi-frame pseudo-label data enhancement according to claim 1, the calculation obtains a three-dimensional coordinate set of key points, a two-dimensional key point set and a camera intrinsic parameter, and the online hand-eye calibration is completed based on the three-dimensional coordinate set of key points, the two-dimensional key point set and the camera intrinsic parameter, and based on the robot key point detection model, specifically comprising: According to the angle sensor data and positive kinematics of each joint of the robot, the three-dimensional coordinate set of key points in the predefined n robot base coordinate system is solved. ,as follows: ； Use Zhang Zhengyou's camera calibration method to calibrate the camera and obtain the camera internal parameters , camera internal parameters The complete matrix of is as follows: ； in, and represents the focal length of the camera described in pixels, and The origin of the physical imaging plane is Axis and Offset on the axis; Use a camera to take pictures of the robot, and use the final robot key point detection model to predict the two-dimensional coordinates of the key points in the RGB image, as shown below: ； in, Represents a set of n two-dimensional key points in the image, Represents the position coordinates of the i-th key point in the image; The three-dimensional coordinates of the key points are set , 2D key point set and camera intrinsics As input, the EPnP algorithm is used to solve the hand-eye relationship ,Online hand-eye calibration is completed using a single image.
7. A robot online hand-eye calibration system based on multi-frame pseudo-label data enhancement, used to implement the robot online hand-eye calibration method based on multi-frame pseudo-label data enhancement according to any one of claims 1 to 6, characterized in that: The robot online hand-eye calibration system based on multi-frame pseudo-label data enhancement includes: A synthetic data set generation module, used to generate multiple robot model animation sequences, randomize the robot model animation sequences in different fields, and output multiple synthetic robot images and key point labels as synthetic data sets; A feature extraction module, used to preprocess the synthetic data set, train a key point detection model using the preprocessed synthetic data set to obtain a pre-trained key point detection model, extract feature information of the image based on the encoder of the pre-trained key point detection model to obtain a feature map, then use a decoder to map the feature map into multiple confidence maps, and obtain key point coordinates according to the confidence map; A data enhancement and fine-tuning module is used to collect robot key point images in real scenes, use the pre-trained key point detection model to predict the collected robot key point images, use the prediction results as original key point pseudo labels, use the EPnP algorithm to solve the hand-eye relationship by combining the original key point pseudo labels of multiple frames of images, and then use the hand-eye relationship to reproject the key points to obtain enhanced key point pseudo label data, and use the enhanced key point pseudo label data to adjust the pre-trained key point detection model to obtain the final robot key point detection model; The online hand-eye calibration module is used to calculate the three-dimensional coordinate set of key points, the two-dimensional key point set and the camera intrinsic parameters, and complete the online hand-eye calibration based on the three-dimensional coordinate set of key points, the two-dimensional key point set and the camera intrinsic parameters and the robot key point detection model.
8. A terminal, characterized in that: The terminal includes: a memory, a processor, and a robot online hand-eye calibration program based on multi-frame pseudo-label data enhancement stored in the memory and runnable on the processor. When the robot online hand-eye calibration program based on multi-frame pseudo-label data enhancement is executed by the processor, the steps of the robot online hand-eye calibration method based on multi-frame pseudo-label data enhancement as described in any one of claims 1-6 are implemented.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a robot online hand-eye calibration program based on multi-frame pseudo-label data enhancement. When the robot online hand-eye calibration program based on multi-frame pseudo-label data enhancement is executed by a processor, the steps of the robot online hand-eye calibration method based on multi-frame pseudo-label data enhancement as described in any one of claims 1-6 are implemented.