Collaborative robot hand-eye calibration method and device, and hand-eye calibration model training method

Abstract

This invention belongs to the field of machine vision calibration technology, and relates to a collaborative robot hand-eye calibration method, device, and hand-eye calibration model training method. The collaborative robot hand-eye calibration method includes: acquiring multi-frame calibration board image data obtained by a camera module capturing images of a calibration board; determining the three-dimensional coordinate data of the corner points of the calibration board relative to the camera coordinate system and the six-dimensional pose data relative to the base coordinate system based on the calibration board image data; performing steady-state discrimination and anomaly detection on the three-dimensional coordinate data and six-dimensional pose data corresponding to the multi-frame calibration board image data to obtain steady-state data; performing dimensionality upscaling and condition label embedding on the steady-state data to obtain high-dimensional fusion features; and calculating the high-dimensional fusion features using a hand-eye calibration model to obtain a hand-eye calibration matrix. This invention satisfies targeted data augmentation, filtering, and multi-dimensional information mining based on a small number of calibration points, and uses a lightweight neural network to achieve accurate conversion between the camera and robot coordinate systems based on a small number of calibration points.

Description

Technical Field

[0001] This invention belongs to the field of machine vision calibration technology, specifically relating to a collaborative robot hand-eye calibration method, device, and hand-eye calibration model training method. Background Technology

[0002] Eye-to-hand visual calibration is a core technology in the vision application system of collaborative robots. Its core objective is to solve the hand-eye matrix to establish a rigid transformation relationship between the camera coordinate system and the robot tool coordinate system. This ensures that the coordinates of the calibration points collected by the vision system can be accurately converted into motion control coordinates that the robot can execute, which is a fundamental prerequisite for realizing robot vision-guided operations.

[0003] Traditional eye-to-hand calibration methods, represented by the Tsai-Lenz algorithm, require collecting multiple sets of calibration point data from different angles and solving for the hand-eye matrix through complex matrix operations. These methods suffer from several technical drawbacks in practical industrial applications. First, the required amount of calibration point data is large, the collection process is cumbersome, and the requirements for collection accuracy are stringent. Slight deviations in the calibration point data can be amplified through matrix operations, leading to errors in the final calibration results. Second, it lacks robustness. In complex industrial environments with dust, changes in lighting, and mechanical vibration, factors such as calibration point extraction errors, sensor acquisition noise, and robot motion jitter can seriously interfere with the calibration results, making it difficult to guarantee calibration stability. Third, the data processing method is crude, only performing simple splicing and calculation on the raw data, without dynamic / steady-state data discrimination or abnormal data filtering mechanism, and low-quality data can easily become the source of calibration error; Fourth, the algorithm has high computational complexity, making it difficult to adapt to industrial needs that require frequent calibration, such as robot model changes, camera disassembly and assembly, and changes in work scenarios.

[0004] In existing technologies, artificial neural networks have been gradually applied to machine vision scenarios such as feature extraction and image recognition. Some technologies have also attempted to combine neural networks with robot hand-eye calibration, but the relevant solutions have the following obvious shortcomings: First, there is no targeted data augmentation processing scheme for calibration points, resulting in insufficient mining of low-dimensional feature information; second, there is no lightweight neural network architecture adapted to hand-eye calibration scenarios, resulting in a large number of model parameters and slow inference speed, making it difficult to implement in industrial settings; third, the matrix prediction process for small sample data is vague, making it impossible to achieve high-precision calibration of a small number of calibration points; finally, there is a lack of a full-dimensional error preprocessing framework covering acquisition, features, and labels, and various noises and systematic errors still affect the model training and calibration accuracy.

[0005] In summary, how to achieve targeted data augmentation, filtering, and multidimensional information mining based on a small number of calibration points, and how to use lightweight neural networks to achieve accurate conversion between the camera coordinate system and the robot tool coordinate system based on a small number of calibration points, has become an important technical problem that urgently needs to be solved in the field of visual calibration. Summary of the Invention

[0006] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a collaborative robot hand-eye calibration method, system, and diffusion model training method, which is used to meet targeted data augmentation, filtering, and multi-dimensional information mining based on a small number of calibration points, and to use a lightweight neural network to achieve accurate conversion between the camera coordinate system and the robot tool coordinate system based on a small number of calibration points.

[0007] This invention provides a method for hand-eye calibration of a collaborative robot, comprising: Acquire multi-frame image data of the calibration board obtained by the camera module installed on the collaborative robot; The three-dimensional coordinate data of the corner points of the calibration board relative to the camera coordinate system and the six-dimensional attitude data relative to the base coordinate system of the collaborative robot are determined based on the calibration board image data. Steady-state discrimination and anomaly detection are performed on the three-dimensional coordinate data and six-dimensional attitude data corresponding to the multiple frames of calibration board image data to obtain steady-state data; The steady-state data is upgraded and operating condition labels are embedded to obtain high-dimensional fusion features; The high-dimensional fusion features are calculated using a pre-constructed hand-eye calibration model to obtain the hand-eye calibration matrix.

[0008] In one embodiment of the present invention, the step of performing steady-state discrimination and anomaly detection on the three-dimensional coordinate data and six-dimensional attitude data corresponding to multiple frames of calibration board image data to obtain steady-state data includes: Calculate the gradient change value between two adjacent frames based on the three-dimensional coordinate data and six-dimensional attitude data corresponding to the calibration board image data of multiple frames; When the gradient change value does not exceed the preset judgment threshold, the error is calculated on the three-dimensional coordinate data and six-dimensional attitude data corresponding to the calibration board image data of multiple frames using a preset sliding window. When the error calculation results meet the preset error condition range, the corresponding three-dimensional coordinate data and six-dimensional attitude data are used as steady-state data.

[0009] In one embodiment of the present invention, the step of upscaling and embedding condition labels into the steady-state data to obtain high-dimensional fusion features includes: The basic features are obtained by feature stitching of the three-dimensional coordinate data and the six-dimensional attitude data in the steady-state data; The target features are obtained by performing data filtering and dimensional normalization on the basic features; The target features are subjected to principal component dimensionality upscaling to obtain orthogonal features; A working condition label is generated based on the shooting angle and shooting distance when the calibration board image data is captured. The operating condition labels are embedded into the orthogonal features and normalized to obtain high-dimensional fused features.

[0010] In one embodiment of the present invention, the step of performing principal component dimensionality upscaling on the target features to obtain orthogonal features includes: Principal component analysis is performed on the target features to obtain orthogonal principal component features; Based on the orthogonal principal component features, feature mapping is performed to obtain orthogonal mapping features; The orthogonal principal component features and the orthogonal mapping features are concatenated to obtain orthogonal features.

[0011] In one embodiment of the present invention, the step of calculating the high-dimensional fusion features using a pre-constructed hand-eye calibration model to obtain a hand-eye calibration matrix includes: The high-dimensional fusion features are calculated by using the multi-layer network and corresponding activation function in the pre-constructed hand-eye calibration model to obtain the initial hand-eye calibration features. The initial hand-eye calibration features are denormalized to obtain standard hand-eye calibration features; A hand-eye calibration matrix is ​​constructed based on the standard hand-eye calibration features.

[0012] In one embodiment of the present invention, it further includes: The real hand-eye calibration matrix is ​​calculated based on the three-dimensional coordinate data and six-dimensional posture data corresponding to the calibration board image data under different shooting angles and shooting distances. Calculate the relative error between the hand-eye calibration matrix and the actual hand-eye calibration matrix; When the relative error is less than a preset error threshold, the hand-eye calibration matrix is ​​determined to be correct and the hand-eye calibration model is determined to be reliable.

[0013] In one embodiment of the present invention, it further includes: Based on the hand-eye calibration matrix, the collaborative robot is controlled to move and transform the camera module to obtain the position to be verified. The distance deviation is calculated based on the actual position of the calibration board when the camera module captures the image and the position to be verified. Orthogonality is checked based on the determinant of the hand-eye calibration matrix; When the distance deviation is less than the preset position deviation threshold and the orthogonality verification result meets the preset orthogonality range, the hand-eye calibration matrix is ​​determined to be correct and the hand-eye calibration model is determined to be reliable.

[0014] This invention provides a collaborative robot hand-eye calibration device, comprising: a collaborative robot, a calibration board, a camera module, a control module, and a data processing module; The camera module is installed at the end of the collaborative robot and is used to photograph the calibration board to obtain calibration board image data; The control module is used to control the collaborative robot to move and change, so that the camera module can take pictures of the calibration board from different shooting angles and shooting distances; The data processing module is used to send control signals to the control module to control the collaborative robot to perform movement transformations and to execute the collaborative robot hand-eye calibration method.

[0015] This invention provides a collaborative robot hand-eye calibration device, applied to a collaborative robot hand-eye calibration method, comprising: Acquire multiple sets of calibration board training image data and corresponding real hand-eye verification matrices obtained by the camera module installed on the collaborative robot taking pictures of the calibration board; The multiple sets of calibration board training image data are divided into training set data and validation set data; The training set data is calculated using a preset initial hand-eye calibration model to obtain a predicted hand-eye matrix; The loss value is calculated based on the predicted hand-eye matrix and the real hand-eye verification matrix, and the loss value is used to optimize the initial hand-eye calibration model; The optimized initial hand-eye calibration model is used to calculate the validation set data to obtain the validation hand-eye matrix; Calculate the mean square error based on the verified hand-eye matrix and the actual hand-eye verification matrix; Until the mean square error is less than or equal to the preset convergence value, the optimized initial hand-eye calibration model is used as the hand-eye calibration model.

[0016] In one embodiment of the present invention, acquiring multiple sets of calibration board training image data and the corresponding real hand-eye verification matrix obtained by the camera module installed on the collaborative robot capturing images of the calibration board includes: Acquire multiple sets of initial training image data of the calibration board obtained by the camera module installed on the collaborative robot; The three-dimensional coordinate training data of the corner points of the calibration board relative to the camera coordinate system and the six-dimensional posture training data relative to the base coordinate system of the collaborative robot are determined based on the initial training image data of the calibration board. Steady-state discrimination and anomaly detection are performed on the three-dimensional coordinate training data and six-dimensional attitude training data corresponding to the multi-frame calibration board training image data to obtain steady-state training data; The steady-state training data is upgraded in dimensionality and embedded with working condition labels to obtain calibration board training image data; The real hand-eye verification matrix is ​​calculated based on the three-dimensional coordinate data and six-dimensional posture data corresponding to the initial training image data of the calibration board under different shooting angles and shooting distances.

[0017] The beneficial effects of this invention are as follows: By processing three-dimensional coordinate data and six-dimensional posture data through steady-state discrimination, anomaly detection, and error detection, this invention achieves accurate component analysis and anomaly filtering of the data, eliminating various noises and errors in the data acquisition and processing process, and removing low-quality data, enabling the hand-eye calibration method to maintain high robustness in complex industrial environments. Through operations such as data dimensionality enhancement and condition label embedding, this invention enhances data features and fully mines the hidden information of low-dimensional features and their correlation with the operating conditions by combining feature labels. This eliminates feature redundancy and correlation while retaining the geometric features and calibration condition features of the calibration data, allowing the hand-eye calibration model to learn more comprehensive features and the mapping rules of the hand-eye matrix. This invention can achieve high-precision hand-eye calibration matrix prediction with a small amount of calibration data, making it suitable for industrial scenarios with frequent calibration, such as robot model changes and camera assembly / disassembly. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0019] Figure 1 This is a schematic diagram of a collaborative robot hand-eye calibration device provided in one embodiment of the present invention; Figure 2 This is a flowchart illustrating a collaborative robot hand-eye calibration method provided in one embodiment of the present invention; Figure 3 This is a flowchart illustrating a diffusion model training method provided in one embodiment of the present invention; Among them, 1 is the collaborative robot, 2 is the calibration board, 3 is the camera module, 4 is the control module, and 5 is the data processing module. Detailed Implementation

[0020] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings, which illustrate embodiments of the present application. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of this application will be thorough and complete.

[0021] Example 1 Please see Figure 1 As shown in the figure, an embodiment of the present invention provides a collaborative robot hand-eye calibration device, including: a collaborative robot 1, a calibration board 2, a camera module 3, a control module 4, and a data processing module 5.

[0022] In this embodiment of the invention, the collaborative robot 1 can be an industrial collaborative robot with collision detection function, and its motion repeatability positioning accuracy is ≤ ±0.02mm. Serving as the motion carrier of the camera module 3, it drives the camera module 3 to capture images of the calibration board 2 from multiple different angles and distances within the robot's workspace, completing multi-view, multi-condition image data acquisition of the calibration board, ensuring the safety and positional accuracy of the acquisition process.

[0023] In this embodiment of the invention, the camera module 3, i.e., the 3D vision module, can be a structured light 3D camera or an RGB-D depth camera with a resolution ≥640×480 and a depth accuracy ≤±2%. It can be mounted on the end of the collaborative robot 1 module via a customized bracket, using an eye-on-hand mounting mode. The lens faces the calibration board 2 to capture and collect color and depth images of the calibration board 2 as calibration board image data, thereby obtaining image information and three-dimensional coordinate data of the corner points of the calibration board 2, providing a raw data source for subsequent data processing.

[0024] In this embodiment of the invention, the control module 4 is used to control the collaborative robot 1 to perform movement transformations so that the camera module 3 can capture images of the calibration board 2 from different shooting angles and shooting distances. It can have a built-in lightweight hand-eye calibration model that has been trained, input 64-dimensional high-dimensional fusion features that have undergone data augmentation, output a 16-dimensional vector and reconstruct it into a 4×4 hand-eye calibration matrix, and output the hand-eye calibration matrix to the collaborative robot 1. This establishes a transformation relationship between the camera coordinate system of the 3D vision module and the tool coordinate system of the collaborative robot 1, and completes the accurate transformation between the camera coordinate system and the robot's tool coordinate system.

[0025] In this embodiment of the invention, the data processing module 5 is used to send control signals to the control module 4 to control the collaborative robot 1 to perform movement transformations. It synchronously records the six-dimensional posture data of the base coordinate system of the collaborative robot 1 and the three-dimensional coordinate data of the corner points of the calibration board 2 collected by the 3D vision module. Through data enhancement technologies such as data type discrimination, multi-scale anomaly detection, full-dimensional error preprocessing, PCA processing, and feature mapping, the collected data is filtered, purified, and feature-enhanced to finally generate high-quality high-dimensional fusion feature data.

[0026] Specifically, it includes a multi-dimensional data processing module 5, an error preprocessing unit, a condition-based data storage unit, and a PCA (Principal Component Analysis) processing unit: The multi-dimensional data processing module 5 is used to perform steady-state discrimination, anomaly detection, and condition label fusion on the three-dimensional coordinate data and six-dimensional posture data obtained by the collaborative robot 1 and the camera module 3, adapting to low-sample data processing scenarios. The error preprocessing unit is used to perform operations such as feature splicing and data filtering to eliminate various noises and system errors; The working condition data storage unit is divided into a steady-state sub-library and a dynamic sub-library based on the steady-state discrimination result. It is archived in intervals according to the shooting angle and distance of the collaborative robot 1, supports timely data filtering, and provides a uniformly distributed data source that is adapted to the working conditions for the hand-eye calibration model processing. The PCA processing unit is used to perform principal component analysis, which processes the 9-dimensional basic features into the largest linearly independent principal components, and then extends them to 64-dimensional orthogonal features through a feature mapping algorithm. This preserves the effective information of the original data to achieve data augmentation and ensure data stability.

[0027] It should be noted that the base coordinate system is an absolute reference system fixed on the mounting base of the collaborative robot 1; the tool coordinate system is a moving coordinate system attached to the end effector of the collaborative robot 1, which changes position as the collaborative robot 1 moves; and the camera coordinate system is a coordinate system fixed on the 3D vision module, which moves as the camera module 3 moves.

[0028] Specifically, camera module 3 can observe the transformation matrix of calibration plate 2 relative to the camera coordinate system. The robot forward kinematics can derive the transformation matrix of the end effector of collaborative robot 1 relative to the base coordinate system. The hand-eye calibration matrix can be obtained through the collaborative robot hand-eye calibration method, which represents the transformation matrix of camera module 3 relative to the tool coordinate system corresponding to collaborative robot 1. Based on the above three transformation matrices, the transformation relationship between the camera coordinate system and the tool coordinate system can be established, enabling camera module 3 and collaborative robot 1 to work collaboratively in different coordinate systems.

[0029] The collaborative robot hand-eye calibration device of the present invention optimizes the internal architecture of the data acquisition enhancement part based on the traditional hand-eye calibration device, and adds functional units such as multi-dimensional data processing and error preprocessing. It does not require replacement of core hardware and has low modification cost. At the same time, the device can be adapted to different types of structured light 3D cameras, RGB-D depth cameras and industrial collaborative robots. By adjusting the data processing threshold and working condition classification criteria, it can be adapted to different industrial calibration scenarios, which has strong versatility.

[0030] Example 2 Please see Figure 2 As shown in the figure, the present invention provides a collaborative robot hand-eye calibration method. By designing a feature enhancement scheme for multi-dimensional processing of calibration board image data, it proposes a lightweight hand-eye calibration network model architecture and precise parameter configuration adapted to industrial scenarios. It constructs a full-dimensional data preprocessing framework covering raw data acquisition, feature fusion, and training label embedding, and formulates a standardized matrix prediction process for small sample data (such as 3-4 sets of data). It solves the technical problems of traditional eye-on-hand calibration methods, such as computational complexity, insufficient robustness, large calibration point requirements, and coarse data processing. It achieves accurate and rapid conversion between the camera coordinate system and the robot tool coordinate system with a small number of calibration points, improves the robustness and efficiency of calibration, and adapts to the needs of frequent calibration in industrial settings.

[0031] Specifically, the collaborative robot hand-eye calibration method provided in this embodiment of the invention includes the following steps: S11. Obtain multi-frame image data of the calibration board obtained by the camera module installed on the collaborative robot.

[0032] In this embodiment of the invention, the checkerboard calibration board is fixed in a stable position within the robot's workspace. The collaborative robot drives the camera module to continuously photograph the calibration board from multiple different angles and distances within the robot's workspace, with each angle and distance corresponding to multiple frames of calibration board image data.

[0033] S12. Determine the three-dimensional coordinate data of the corner points of the calibration board relative to the camera coordinate system, and the six-dimensional attitude data relative to the base coordinate system of the collaborative robot, based on the calibration board image data.

[0034] In this embodiment of the invention, the camera module or data processing module 5 can extract the 3D coordinates of the corner points in the camera coordinate system (i.e., the three-dimensional spatial coordinates of each inner corner point on the checkerboard calibration board in the camera coordinate system) based on the calibration board image data to obtain the three-dimensional coordinate data corresponding to the calibration board image data; the data processing module 5 synchronously records the six-dimensional attitude data of the collaborative robot in the base coordinate system under the corresponding shooting angle and distance, including three-dimensional position parameters (X coordinate, Y coordinate, Z coordinate) and three attitude parameters (α roll angle, β pitch angle, γ yaw angle), constituting a complete degree of freedom description in three-dimensional space.

[0035] S13. Perform steady-state discrimination and anomaly detection on the three-dimensional coordinate data and six-dimensional attitude data corresponding to the multi-frame calibration board image data to obtain steady-state data.

[0036] In this embodiment of the invention, the step of performing steady-state discrimination and anomaly detection on the three-dimensional coordinate data and six-dimensional attitude data corresponding to multiple frames of calibration board image data to obtain steady-state data includes: Calculate the gradient change value between two adjacent frames based on the three-dimensional coordinate data and six-dimensional attitude data corresponding to the calibration board image data of multiple frames; When the gradient change value does not exceed the preset judgment threshold, the error is calculated on the three-dimensional coordinate data and six-dimensional attitude data corresponding to the calibration board image data of multiple frames using a preset sliding window. When the error calculation results meet the preset error condition range, the corresponding three-dimensional coordinate data and six-dimensional attitude data are used as steady-state data.

[0037] Specifically, the gradient change value of the first and last segments can be calculated based on the single set of time-series data of multi-frame 3D coordinate data and 6D attitude data. The mean difference of multi-frame data under each calculation index and the time sequence length (number of frames) are calculated respectively. The gradient change value is obtained by calculating the ratio of the mean difference and the time sequence length.

[0038] For example, for X-axis dimension data in multi-frame 3D coordinate data, the difference of the mean values ​​of the data under the X-axis is calculated. The X-axis gradient change value can be obtained by calculating the ratio of the difference of the mean values ​​to the time sequence length corresponding to the multiple frames. The judgment threshold can be dynamically set to 0.03mm / frame. If the X-axis gradient change value exceeds the judgment threshold, it is judged as dynamic data and pushed to the dynamic sub-library of the working condition data storage unit. If it does not exceed the judgment threshold, it is judged as non-dynamic data.

[0039] Furthermore, the sliding window sequentially extracts fixed-length time segments from the temporal data corresponding to continuous multi-frame 3D coordinate data and 6D attitude data for local stability analysis, or combines it with global range analysis. Through window scanning, the sliding window can accurately locate which time periods are stable and extract only truly stable data segments for subsequent processing, avoiding interference from low-quality data.

[0040] In a practical application scenario of this invention, the sliding window size can be set to 10. The variance of the three-dimensional coordinate data and the six-dimensional attitude data can be calculated. If the variance of each data parameter within the sliding window is less than 5%, the corresponding three-dimensional coordinate data and six-dimensional attitude data are determined to be steady-state data. Then, they are averaged to eliminate minor noise interference and pushed to the steady-state sub-database of the working condition data storage unit. If the relative fluctuation ratio of any data parameter exceeds 5%, the corresponding three-dimensional coordinate data and six-dimensional attitude data are determined to be abnormal data and are automatically removed.

[0041] S14. Upgrade the dimensionality of the steady-state data and embed the operating condition labels to obtain high-dimensional fusion features.

[0042] In this embodiment of the invention, the step of upscaling the steady-state data and embedding operating condition labels to obtain high-dimensional fusion features includes: The basic features are obtained by feature stitching of the three-dimensional coordinate data and the six-dimensional attitude data in the steady-state data; The target features are obtained by performing data filtering and dimensional normalization on the basic features; The target features are subjected to principal component dimensionality upscaling to obtain orthogonal features; A working condition label is generated based on the shooting angle and shooting distance when the calibration board image data is captured. The operating condition labels are embedded into the orthogonal features and normalized to obtain high-dimensional fused features.

[0043] Specifically, the three-dimensional coordinate data of the calibration points in the effective data of the steady-state sub-library are concatenated with the six-dimensional posture data of the robot to form a nine-dimensional basic feature vector. The basic feature vector is then subjected to a 2×2 Gaussian filter, with the dispersion degree of the Gaussian distribution (normal distribution) set to σ=0.5, to filter coordinate noise and remove outliers in posture data, eliminating noise interference and system errors caused by collaborative robots, data transmission, matrix calculations, etc. Then, minimum / maximum normalization is performed to map to the [0,1] interval to achieve dimensional normalization, transforming data of different dimensions and orders of magnitude to the same scale range, eliminating the influence of dimensions, and making different features comparable.

[0044] In practical applications of this invention, based on the actual working conditions of the camera module during data acquisition, such as shooting angle (0-30° / 30-60° / 60-90°) and shooting distance (200-300mm / 300-400mm / 400-500mm), labeling information is added to the 64-dimensional orthogonal features. The working condition labels can help the neural network understand the background of the data source and improve the adaptability of the hand-eye calibration model to different working conditions.

[0045] Next, L2 normalization (Euclidean normalization) is performed on the 64-dimensional orthogonal features. Specifically, the L2 norm of the 64-dimensional orthogonal features is first calculated, and then each component in the orthogonal features is divided by the L2 norm so that the length of the normalized feature vector is 1, eliminating the influence of the orthogonal feature scale and ensuring that all features are numerically on the same order of magnitude.

[0046] Furthermore, the principal component upscaling process performed on the target features to obtain orthogonal features includes: Principal component analysis is performed on the target features to obtain orthogonal principal component features; Based on the orthogonal principal component features, feature mapping is performed to obtain orthogonal mapping features; The orthogonal principal component features and the orthogonal mapping features are concatenated to obtain orthogonal features.

[0047] In this embodiment of the invention, PCA is first used to transform the 9-dimensional target features into a 9-dimensional orthogonal principal component space. At this time, each dimension is completely independent, eliminating the correlation between target features. However, the 9-dimensional principal components still only have 9 directions, which has limited expressive power.

[0048] Therefore, QR decomposition can be used to project the 9-dimensional features onto 55 new orthogonal directions, generating 55 new features, namely orthogonal mapping features. QR decomposition is an important matrix decomposition method that can decompose any matrix into the product of an orthogonal matrix Q and an upper triangular matrix R. Finally, the original 9-dimensional orthogonal principal component features are concatenated with these 55-dimensional orthogonal mapping features to obtain 64-dimensional fused orthogonal features. The 64-dimensional orthogonal features not only retain the core information of the target feature principal components, but also expand the expressive power of the features through multi-directional projection.

[0049] S15. Calculate the high-dimensional fusion features using a pre-constructed hand-eye calibration model to obtain the hand-eye calibration matrix.

[0050] In this embodiment of the invention, the step of calculating the high-dimensional fusion features using a pre-constructed hand-eye calibration model to obtain the hand-eye calibration matrix includes: The high-dimensional fusion features are calculated by using the multi-layer network and corresponding activation function in the pre-constructed hand-eye calibration model to obtain the initial hand-eye calibration features. The initial hand-eye calibration features are denormalized to obtain standard hand-eye calibration features; A hand-eye calibration matrix is ​​constructed based on the standard hand-eye calibration features.

[0051] The process involves inputting 64-dimensional high-dimensional fused features into a pre-constructed lightweight hand-eye calibration model. The model sequentially performs inference calculations across the input layer, hidden layer, and output layer, calling activation functions, batch normalization, and regularization functions to complete feature transformation, outputting a 16-dimensional continuous numerical vector, i.e., the standard hand-eye calibration features. Inverse normalization is then performed on the standard hand-eye calibration features to restore them to the true magnitude range of the hand-eye matrix elements, eliminating the magnitude bias introduced by normalization during training. Furthermore, the inverse-normalized 16-dimensional standard hand-eye calibration features are then sequentially filled into a 4×4 matrix in row-major order to obtain the hand-eye calibration matrix.

[0052] In this embodiment of the invention, error analysis can also be performed based on the hand-eye calibration matrix and the actual hand-eye calibration matrix to verify the accuracy of the hand-eye calibration model. Specifically, this includes the following steps: The real hand-eye calibration matrix is ​​calculated based on the three-dimensional coordinate data and six-dimensional posture data corresponding to the calibration board image data under different shooting angles and shooting distances. Calculate the relative error between the hand-eye calibration matrix and the actual hand-eye calibration matrix; When the relative error is less than a preset error threshold, the hand-eye calibration matrix is ​​determined to be correct and the hand-eye calibration model is determined to be reliable.

[0053] This invention can solve for the true hand-eye calibration matrix using the Tsai-Lenz algorithm. Based on 3D coordinate data and 6D posture data under different shooting angles and distances, the same calibration board is photographed by a collaborative robot in multiple different poses. In each pose, the transformation matrix from the end effector of the collaborative robot to the base (given by robot kinematics) and the 3D spatial coordinates of the calibration board in the camera coordinate system can be obtained. The fixed transformation relationship between the camera module and the end effector of the collaborative robot can be solved, i.e., the true hand-eye calibration matrix. This true hand-eye calibration matrix can be used to verify the accuracy of the hand-eye calibration model.

[0054] In this embodiment of the invention, the hand-eye calibration matrix is ​​a 4×4 homogeneous transformation matrix, comprising a rotation component (3×3 matrix) and a translation component (3×1 vector). Therefore, the relative error between the hand-eye calibration matrix and the true hand-eye calibration matrix can be comprehensively represented by calculating the rotation error and translation error separately. The rotation error and translation error can be compared with their corresponding error thresholds to determine whether the hand-eye calibration matrix is ​​correct and the hand-eye calibration model is reliable.

[0055] Furthermore, the rotation error and translation error can be weighted and combined into a percentage index. When this index is less than a preset error threshold, the hand-eye calibration matrix is ​​determined to be correct and the hand-eye calibration model is considered reliable.

[0056] It should be noted that for the hand-eye calibration matrix and / or the real hand-eye calibration matrix, the translation component can be normalized to the [0,1] interval to maintain the consistency of the dimensions with the rotation component; orthogonalization correction is performed on the rotation component to eliminate the orthogonality bias in the matrix calculation; mean centering is performed on all elements of the corrected matrix to reduce the calculation bias of the loss function caused by the difference in the magnitude of the matrix elements.

[0057] In this embodiment of the invention, a dual verification mechanism of position deviation and matrix orthogonality can also be adopted to ensure that the predicted hand-eye calibration matrix meets both the position transformation accuracy requirements and conforms to the physical laws of robot motion. Specifically, this includes the following steps: Based on the hand-eye calibration matrix, the collaborative robot is controlled to move and transform the camera module to obtain the position to be verified. The distance deviation is calculated based on the actual position of the calibration board when the camera module captures the image and the position to be verified. Orthogonality is checked based on the determinant of the hand-eye calibration matrix; When the distance deviation is less than the preset position deviation threshold and the orthogonality verification result meets the preset orthogonality range, the hand-eye calibration matrix is ​​determined to be correct and the hand-eye calibration model is determined to be reliable.

[0058] Specifically, the camera coordinate system coordinates of the corresponding calibration board image data are converted into robot tool coordinate system coordinates through matrix transformation using the hand-eye calibration matrix. The collaborative robot drives the end-effector to move to the converted tool coordinate system coordinates. The three-dimensional Euclidean distance deviation between the actual position of the tool and the corner point of the calibration board is measured using a high-precision laser rangefinder (accuracy ±0.01mm).

[0059] Furthermore, the orthogonality of the rotation components of the hand-eye calibration matrix is ​​verified, and it is confirmed whether the determinant of the hand-eye calibration matrix satisfies 0.99≤|R|≤1.01 to achieve orthogonality verification.

[0060] If the distance deviation is ≤ ±2.0mm and the matrix orthogonality check passes, the hand-eye calibration matrix is ​​determined to be correct and can be used for the conversion between the camera coordinate system and the robot tool coordinate system; if either the position check or the orthogonality check fails, the hand-eye calibration matrix is ​​determined to be inaccurate.

[0061] Example 3 Please see Figure 1 As shown in the figure, the diffusion model training method provided by this invention, applied to the hand-eye calibration method of collaborative robots, includes the following steps: S21. Obtain multiple sets of calibration board training image data and corresponding real hand-eye verification matrix obtained by the camera module installed on the collaborative robot taking pictures of the calibration board. S22. Divide the multiple sets of calibration board training image data into training set data and validation set data; S23. Calculate the training set data using a preset initial hand-eye calibration model to obtain the predicted hand-eye matrix; S24. Calculate the loss value based on the predicted hand-eye matrix and the real hand-eye verification matrix, and optimize the initial hand-eye calibration model using the loss value; S25. The optimized initial hand-eye calibration model is used to calculate the validation set data to obtain the validation hand-eye matrix; S26. Calculate the mean square error based on the verified hand-eye matrix and the actual hand-eye verification matrix; S27. Until the mean square error is less than or equal to the preset convergence value, the optimized initial hand-eye calibration model is used as the hand-eye calibration model.

[0062] The step of acquiring multiple sets of calibration board training image data and the corresponding real hand-eye verification matrix obtained by the camera module installed on the collaborative robot capturing images of the calibration board includes: Acquire multiple sets of initial training image data of the calibration board obtained by the camera module installed on the collaborative robot; The three-dimensional coordinate training data of the corner points of the calibration board relative to the camera coordinate system and the six-dimensional posture training data relative to the base coordinate system of the collaborative robot are determined based on the initial training image data of the calibration board. Steady-state discrimination and anomaly detection are performed on the three-dimensional coordinate training data and six-dimensional attitude training data corresponding to the multi-frame calibration board training image data to obtain steady-state training data; The steady-state training data is upgraded in dimensionality and embedded with working condition labels to obtain calibration board training image data; The real hand-eye verification matrix is ​​calculated based on the three-dimensional coordinate data and six-dimensional posture data corresponding to the initial training image data of the calibration board under different shooting angles and shooting distances.

[0063] This invention can collect 3-4 sets of calibration board image data as calibration board training image data, and perform dynamic / steady-state judgment, full-dimensional error preprocessing, and 9-dimensional basic feature extraction operations on each set of data to strictly remove abnormal data; if a set of data is judged to be abnormal, one set of data is automatically added to ensure that the final input effective data volume is 3-4 sets.

[0064] The present invention divides multiple sets of calibration board training image data (coordinate data of calibration board images) into training set data and validation set data. The training set data is used for iterative optimization of model parameters, and the validation set data is used to monitor the generalization ability of the model.

[0065] In this embodiment of the invention, the specific architecture of the initial hand-eye calibration model can be an input layer, two hidden layers, and an output layer; the model input is a 64-dimensional condition-based fusion feature vector, and the model output is a 16-dimensional vector reconstructed into a 4×4 predicted hand-eye matrix; For example, the hidden layer uses the ReLU activation function, and the output layer uses the Sigmoid activation function, constraining the output to the [0,1] interval to improve stability. The Dropout regularization rate is 0.05, the AdamW optimizer is used, the adaptive learning rate is initially set to 1e-3, and cosine annealing decay is used with a decay coefficient of 0.95, adjusted every 20 epochs. The batch size is adapted to the number of training samples, the maximum number of training epochs is 200, and the mean squared error (MSE) between the predicted hand-eye matrix and the true hand-eye matrix is ​​used as the loss function. The validation set monitors the MSE value in real time during training. If the MSE ≤ 0.002 and there is no upward trend for 10 consecutive rounds, the initial hand-eye calibration model is determined to have converged and the optimal parameters are saved, thus obtaining the trained lightweight hand-eye calibration model.

[0066] It should be noted that after the MSE value of the validation set meets the convergence condition and the optimal parameters of the hand-eye calibration model are saved, the hand-eye calibration model can be further validated using a dual validation mechanism of positional deviation and matrix orthogonality. If the dual validation passes, the optimized initial hand-eye calibration model is determined as the hand-eye calibration model. If either the positional validation or the orthogonality validation fails, 1-2 sets of calibration board training image data are automatically added, and after basic processing, they are fused with the original data. The hand-eye calibration matrix prediction process is then re-executed until both validations pass.

[0067] This invention employs a dual verification mechanism of position deviation and matrix orthogonality to verify the predicted hand-eye calibration matrix. This not only ensures the transformation accuracy between the camera coordinate system and the robot tool coordinate system, but also ensures the orthogonality of the rotation components of the hand-eye matrix, conforming to the physical constraints of robot motion. It avoids the matrix invalidity problem caused by single position verification, making the calibration results more reliable.

[0068] In summary, this invention achieves accurate component analysis and anomaly filtering of data by performing steady-state discrimination, anomaly detection, and error detection on 3D coordinate data and 6D posture data. This eliminates various noises and errors in the data acquisition and processing processes, removes low-quality data, and ensures that the hand-eye calibration method maintains high robustness in complex industrial environments. Through data dimensionality enhancement and condition label embedding, the invention enhances data features and fully mines the hidden information of low-dimensional features and their correlation with operating conditions by combining feature labels. This eliminates feature redundancy and correlation while preserving the geometric features and calibration condition features of the calibration data, enabling the hand-eye calibration model to learn a more comprehensive mapping pattern between features and the hand-eye matrix. This invention can achieve high-precision hand-eye calibration matrix prediction with a small amount of calibration data, making it suitable for industrial scenarios with frequent calibration, such as robot changeovers and camera disassembly / assembly.

[0069] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0070] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0071] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0072] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0073] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in the system claims may also be implemented by a single unit or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for hand-eye calibration of a collaborative robot, characterized in that, include: Acquire multi-frame image data of the calibration board obtained by the camera module installed on the collaborative robot; The three-dimensional coordinate data of the corner points of the calibration board relative to the camera coordinate system and the six-dimensional attitude data relative to the base coordinate system of the collaborative robot are determined based on the calibration board image data. Steady-state discrimination and anomaly detection are performed on the three-dimensional coordinate data and six-dimensional attitude data corresponding to the multiple frames of calibration board image data to obtain steady-state data; The steady-state data is upgraded and operating condition labels are embedded to obtain high-dimensional fusion features; The high-dimensional fusion features are calculated using a pre-constructed hand-eye calibration model to obtain the hand-eye calibration matrix.

2. The collaborative robot hand-eye calibration method according to claim 1, characterized in that, The step of performing steady-state discrimination and anomaly detection on the three-dimensional coordinate data and six-dimensional attitude data corresponding to multiple frames of calibration board image data to obtain steady-state data includes: Calculate the gradient change value between two adjacent frames based on the three-dimensional coordinate data and six-dimensional attitude data corresponding to the calibration board image data of multiple frames; When the gradient change value does not exceed the preset judgment threshold, the error is calculated on the three-dimensional coordinate data and six-dimensional attitude data corresponding to the calibration board image data of multiple frames using a preset sliding window. When the error calculation results meet the preset error condition range, the corresponding three-dimensional coordinate data and six-dimensional attitude data are used as steady-state data.

3. The collaborative robot hand-eye calibration method according to claim 1, characterized in that, The process of upscaling the steady-state data and embedding operating condition labels to obtain high-dimensional fusion features includes: The basic features are obtained by feature stitching of the three-dimensional coordinate data and the six-dimensional attitude data in the steady-state data; The target features are obtained by performing data filtering and dimensional normalization on the basic features; The target features are subjected to principal component dimensionality upscaling to obtain orthogonal features; A working condition label is generated based on the shooting angle and shooting distance when the calibration board image data is captured. The operating condition labels are embedded into the orthogonal features and normalized to obtain high-dimensional fused features.

4. The collaborative robot hand-eye calibration method according to claim 3, characterized in that, The principal component upsizing process performed on the target features to obtain orthogonal features includes: Principal component analysis is performed on the target features to obtain orthogonal principal component features; Based on the orthogonal principal component features, feature mapping is performed to obtain orthogonal mapping features; The orthogonal principal component features and the orthogonal mapping features are concatenated to obtain orthogonal features.

5. The collaborative robot hand-eye calibration method according to claim 1, characterized in that, The step of calculating the high-dimensional fusion features using a pre-constructed hand-eye calibration model to obtain the hand-eye calibration matrix includes: The high-dimensional fusion features are calculated by using the multi-layer network and corresponding activation function in the pre-constructed hand-eye calibration model to obtain the initial hand-eye calibration features. The initial hand-eye calibration features are denormalized to obtain standard hand-eye calibration features; A hand-eye calibration matrix is ​​constructed based on the standard hand-eye calibration features.

6. The collaborative robot hand-eye calibration method according to claim 1, characterized in that, Also includes: The real hand-eye calibration matrix is ​​calculated based on the three-dimensional coordinate data and six-dimensional posture data corresponding to the calibration board image data under different shooting angles and shooting distances. Calculate the relative error between the hand-eye calibration matrix and the actual hand-eye calibration matrix; When the relative error is less than a preset error threshold, the hand-eye calibration matrix is ​​determined to be correct and the hand-eye calibration model is determined to be reliable.

7. The collaborative robot hand-eye calibration method according to claim 1, characterized in that, Also includes: Based on the hand-eye calibration matrix, the collaborative robot is controlled to move and transform the camera module to obtain the position to be verified. The distance deviation is calculated based on the actual position of the calibration board when the camera module captures the image and the position to be verified. Orthogonality is checked based on the determinant of the hand-eye calibration matrix; When the distance deviation is less than the preset position deviation threshold and the orthogonality verification result meets the preset orthogonality range, the hand-eye calibration matrix is ​​determined to be correct and the hand-eye calibration model is determined to be reliable.

8. A collaborative robot hand-eye calibration device, characterized in that, include: Collaborative robot, calibration board, camera module, control module, and data processing module; The camera module is installed at the end of the collaborative robot and is used to photograph the calibration board to obtain calibration board image data; The control module is used to control the collaborative robot to move and change, so that the camera module can take pictures of the calibration board from different shooting angles and shooting distances; The data processing module is used to send control signals to the control module to control the collaborative robot to perform movement transformations and to execute the collaborative robot hand-eye calibration method as described in claim 1.

9. A hand-eye calibration model training method, applied to the hand-eye calibration method of collaborative robots, characterized in that, include: Acquire multiple sets of calibration board training image data and corresponding real hand-eye verification matrices obtained by the camera module installed on the collaborative robot taking pictures of the calibration board; The multiple sets of calibration board training image data are divided into training set data and validation set data; The training set data is calculated using a preset initial hand-eye calibration model to obtain a predicted hand-eye matrix; The loss value is calculated based on the predicted hand-eye matrix and the real hand-eye verification matrix, and the loss value is used to optimize the initial hand-eye calibration model; The optimized initial hand-eye calibration model is used to calculate the validation set data to obtain the validation hand-eye matrix; Calculate the mean square error based on the verified hand-eye matrix and the actual hand-eye verification matrix; Until the mean square error is less than or equal to the preset convergence value, the optimized initial hand-eye calibration model is used as the hand-eye calibration model.

10. The hand-eye calibration model training method according to claim 9, characterized in that, The acquisition of multiple sets of calibration board training image data and the corresponding real hand-eye verification matrix obtained by the camera module installed on the collaborative robot capturing images of the calibration board includes: Acquire multiple sets of initial training image data of the calibration board obtained by the camera module installed on the collaborative robot; The three-dimensional coordinate training data of the corner points of the calibration board relative to the camera coordinate system and the six-dimensional posture training data relative to the base coordinate system of the collaborative robot are determined based on the initial training image data of the calibration board. Steady-state discrimination and anomaly detection are performed on the three-dimensional coordinate training data and six-dimensional attitude training data corresponding to the multi-frame calibration board training image data to obtain steady-state training data; The steady-state training data is upgraded in dimensionality and embedded with working condition labels to obtain calibration board training image data; The real hand-eye verification matrix is ​​calculated based on the three-dimensional coordinate data and six-dimensional posture data corresponding to the initial training image data of the calibration board under different shooting angles and shooting distances.

Images