A hand-eye calibration compensation algorithm based on cross-attention mechanism

Abstract

This invention relates to the field of machine vision, specifically addressing the hand-eye calibration problem. The invention proposes a hand-eye calibration compensation algorithm based on a cross-attention mechanism. By employing illumination correction and calibration error compensation modules, it enhances the robustness and accuracy of calibration. This algorithm not only focuses on the correlations between multimodal data during feature extraction but also improves the model's adaptability and generalization ability in complex environments, especially those with poor lighting conditions. Specifically, the method utilizes a cross-attention mechanism to learn information from both image and pose modalities: first, it corrects the image under poor lighting conditions to eliminate the impact of poor lighting on calibration accuracy. The proposed hand-eye calibration compensation algorithm offers the advantage of high calibration accuracy while exhibiting strong robustness in scenarios with poor lighting conditions, thus better meeting the needs of practical applications.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically, it relates to a hand-eye calibration compensation algorithm based on a cross-attention mechanism. Background Technology

[0002] Vision systems are a core element in realizing the intelligence and automation of industrial robots, and hand-eye calibration algorithms have become a hot research topic for many scholars. Hand-eye calibration, that is, determining the transformation relationship between the coordinate system of the robotic arm's end effector and the camera coordinate system, ensures the accuracy of the robotic arm's "hand-eye" coordinated operation, and its accuracy determines the stability of the vision system.

[0003] Although hand-eye calibration has been extensively studied, with the development of industrial automation and intelligence, traditional hand-eye calibration methods face greater challenges in terms of accuracy and robustness. The principles of hand-eye calibration determine its following characteristics:

[0004] (1) High requirements for the calibration environment. The camera needs to capture a complete image of the calibration board in a bright and undisturbed environment; otherwise, the calibration accuracy will be greatly affected.

[0005] (2) Narrow scope of application. The parameters of the hand-eye calibration mathematical model are strongly correlated, making it difficult to adapt to changes in different cameras, application scenarios and lighting conditions.

[0006] Currently, numerous publications have introduced relatively complete hand-eye calibration algorithms. Some papers propose hand-eye calibration algorithms based on mathematical modeling, obtaining the hand-eye matrix by solving the homogeneous transformation equation stepwise or simultaneously. However, these papers are all based on precise mathematical models, so uncertainties in intrinsic and extrinsic parameters can cause estimation errors in the system, failing to accurately reflect nonlinear mapping relationships. Furthermore, some papers introduce hand-eye calibration methods based on minimizing reprojection errors, but they do not fully discuss the impact of poor lighting conditions. This method has high requirements for the quality of the calibration object image and feature extraction, and is significantly affected by factors such as lighting. Therefore, existing hand-eye calibration algorithms often suffer from insufficient accuracy and weak robustness, failing to meet the application needs of real-world automated factory scenarios. A hand-eye calibration algorithm that maintains high accuracy and robustness even under poor lighting conditions is needed.

[0007] In view of this, the present invention is proposed. Summary of the Invention

[0008] To address the aforementioned technical problems, the basic concept of this invention is as follows: Given the characteristics of hand-eye calibration principles and the shortcomings of existing technologies, we aim to obtain a hand-eye matrix with the highest possible accuracy under poor lighting conditions, thereby improving the robustness of hand-eye calibration and ensuring broad applicability. Therefore, this invention proposes a hand-eye calibration compensation algorithm based on a cross-attention mechanism under poor lighting conditions. This algorithm uses a cross-attention mechanism to fit a nonlinear function relationship, handling the correlation between rotation matrix and translation vector information in greater detail. This significantly improves the accuracy of hand-eye calibration, especially in scenarios with poor lighting conditions, where the model demonstrates strong adaptability and generalization ability. The specific technical solution adopted in this invention is as follows:

[0009] This invention proposes a hand-eye calibration compensation algorithm based on a cross-attention mechanism, specifically designed for practical applications of hand-eye calibration under poor lighting conditions. The method includes the following steps:

[0010] Image datasets containing calibration objects were acquired under three different lighting conditions (insufficient light, overexposed light, and normal light), and the robot arm pose corresponding to the acquisition of calibration object images was recorded.

[0011] Feature extraction is performed on the poorly lit calibration object images and robotic arm poses in the training set so that subsequent models can process them in a lower-dimensional data space.

[0012] A lighting correction module based on the cross-attention mechanism is constructed for image restoration. The low-dimensional features extracted from the input image are fed into the cross-attention module for calculation, and a 3×3 color correction matrix and a brightness correction value γ are output to restore a calibration board image that is closer to that taken under normal lighting conditions.

[0013] A calibration and correction module based on the cross-attention mechanism is constructed to compensate for the hand-eye matrix. Image features and pose features are concatenated and then fed into the cross-attention module for calculation. The module outputs a 3×3 rotation matrix bias and a 1×3 translation vector bias, which are used to correct the error of the traditional hand-eye calibration method.

[0014] The illumination correction module and the calibration correction module are unified into a complete hand-eye calibration process, forming a hand-eye calibration algorithm that uses a cross-attention mechanism for compensation.

[0015] The hand-eye calibration compensation model based on the cross-attention mechanism was used to process the data in the test set to obtain the hand-eye matrix results.

[0016] Compared with the prior art, the present invention has the following advantages and effects: The algorithm of the present invention has the following advantages and effects:

[0017] A cross-attention mechanism-based approach is adopted for hand-eye calibration. An error correction module applicable to both illumination and calibration is designed. By learning information from two modalities—the calibration board image and the robot arm pose—error correction is performed on the calibration algorithm to achieve high-precision and robust hand-eye calibration.

[0018] Through objective evaluation and analysis of the algorithm compared with traditional hand-eye calibration algorithms, the experimental results show that the algorithm can effectively improve the accuracy and robustness of hand-eye calibration under poor lighting conditions. It can obtain a high-precision hand-eye matrix with fewer calibration angles, greatly saving manpower and resources and improving the automation performance of hand-eye calibration.

[0019] The specific embodiments of the present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0020] In the attached diagram:

[0021] Figure 1 This is a flowchart of the implementation method of the present invention;

[0022] Figure 2 This is a performance comparison chart of the rotation matrix accuracy of the present invention with that of existing hand-eye calibration methods when the number of calibration viewpoints is 10.

[0023] Figure 3 This is a performance comparison chart of the translation vector accuracy of the present invention with existing hand-eye calibration methods when the number of calibration viewpoints is 10.

[0024] Figure 4 This is a performance comparison chart of the rotation matrix accuracy of the present invention with existing hand-eye calibration methods when the number of calibration viewpoints is 20.

[0025] Figure 5 This is a performance comparison chart of the translation vector accuracy of the present invention with existing hand-eye calibration methods when the number of calibration viewpoints is 20. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments will be clearly and completely described below with reference to the accompanying drawings. The following embodiments are used to illustrate the present invention.

[0027] like Figures 1 to 3 As shown in the embodiment of the present invention, the construction of the illumination correction module for image restoration based on the cross-attention mechanism specifically includes the following steps:

[0028] Correcting poorly lit images requires processing by an Image Signal Processor (ISP). The process in the ISP involves complex nonlinear operations. An encoder-decoder structure is used to describe the ISP process, as shown below:

[0029]

[0030] Among them, I t Image of the target taken under normal lighting conditions; I i Input images taken at the same angle under poor lighting conditions; It is a 3×3 color transformation matrix; γ represents the gamma correction parameter. ε is set to 1e-8, which helps stabilize training;

[0031] Cross-attention refers to the model's ability to capture element-level dependencies between two input sequences with different distributions, learning the associations between different sequences and better handling data of different modalities. Cross-attention effectively models the positional relationships of data by progressively calculating the dot product of the input data, thereby enabling learning and inference queries for specific tasks. Its working principle is as follows: The input consists of two sequences (Input Arrays) S1 and S2 with different modalities. Values ​​(V) and keys (K) are extracted from input sequence S1, and a query (Q) is extracted from input sequence S2. For each position of query Q, the degree of association between it and all positions in key K is calculated, resulting in a similarity matrix. After normalizing the similarity matrix, an attention weight matrix is ​​obtained, representing the degree of association between query Q and each position in key K. This is then multiplied by the value V and weighted summed to obtain the output sequence (Output Array). Equation 2 is the calculation formula for cross-attention:

[0032]

[0033] In the formula, Q is the query matrix, K is the key matrix, V is the value matrix, and Softmax() represents the Softmax function, used to convert attention weights into a probability distribution. is a scaling factor used to control the size of the attention weights, where C is the dimension of the key and h is the number of attention heads;

[0034] To restore images captured in poor lighting conditions to images captured in normal lighting conditions, images captured in normal lighting conditions at the same angle are used as supervision to train the cross-attention module in a supervised manner.

[0035] During the training phase, the query is dynamically updated in each iteration to match the target image I. t;

[0036] Ten queries were set up to retrieve color and brightness information from the input data, and the data was reorganized into a 3×3 color matrix and a 1×1 brightness correction value γ.

[0037] According to an embodiment of the present invention, the construction of a calibration and correction module based on a cross-attention mechanism for compensating the hand-eye matrix specifically includes the following steps:

[0038] After feature extraction, the calibrated object image and the robotic arm pose matrix are stitched together in the same dimension, and their relationship is learned and modeled through a cross-attention module.

[0039] To retrieve the required error correction information from the input multimodal information, 12 queries are set to generate 3×3 rotation matrix corrections and 1×3 translation vector corrections, and nonlinear perturbations are added through convolutional layers.

[0040] In order to enable the model to correct errors for hand-eye calibration algorithms, the calibration results of traditional algorithms are used as a basis, and the correction values ​​generated by the query are used for correction. This process is carried out under the supervision of the true values ​​of the hand-eye matrix.

[0041] According to an embodiment of the present invention, unifying the illumination correction module and the calibration correction module into a complete hand-eye calibration process specifically includes the following steps:

[0042] To achieve the respective functions of the three modules—feature extraction, illumination correction, and hand-eye calibration error correction—in a single end-to-end training, this invention employs a weighted combined loss function. The L1 loss function is used to measure the differences between the restored image and the real image, between the rotation vector calibration value and the real value, and between the translation vector calibration value and the real value, respectively. The contribution of each loss is measured by λ. The calculation process of the L1 loss function is shown in Equation 3:

[0043]

[0044] Where n is the number of samples, y i It is the true value of the i-th sample. This is the predicted value of the i-th sample. Meanwhile, noting that traditional methods have significant errors in calculating the translation vector, and these errors are further amplified when the images repaired by the model are unstable in the early stages of training, a threshold value 's' is added to the loss function for the translation vector to prevent the adverse effects of extreme outliers on training. The specific loss function is shown in Equation 4:

[0045] Loss=λ img L1 img +λ rot L1rot +λ trans min(L1 trans ,s) (4)

[0046] Where, λ img , λ rot , λ trans These are the loss weight coefficients for the image, rotation matrix, and translation vector, respectively, used to adjust the contribution of each part to the total loss. s is the maximum threshold for the translation vector loss. In this way, the L1 loss of the translation vector part is compared before being added to the total loss to ensure that its value does not exceed s, thereby avoiding excessive translation loss from having a large impact on the total loss.

[0047] The complete process of the hand-eye calibration compensation algorithm based on the cross-attention mechanism is as follows:

[0048] The input consists of N chessboard images for calibration. and robotic arm pose matrix

[0049] After feature extraction via convolutional layers, the image features F i The multimodal features F after splicing the pose features of the robotic arm CAT The signals are input in parallel to the illumination correction module and the hand-eye calibration error correction module, respectively.

[0050] Color matrix output by illumination correction module The luminance value γ was used to reconstruct images that more closely resemble those taken under normal lighting conditions.

[0051] This batch of images I fix The hand-eye matrix is ​​obtained by using traditional algorithms for calibration.

[0052] The rotation matrix correction and translation vector correction generated by the hand-eye calibration error correction module are also concatenated into a correction matrix.

[0053] M bias With M ill The final hand-eye calibration matrix M is obtained by adding them together. corr .

[0054] This invention uses a checkerboard pattern as a calibration object, fixing the checkerboard at a fixed position on the ground and ensuring it is completely within the camera's field of view to avoid the inability to extract corner information. The robotic arm is moved to 100 different poses, and the relevant parameters of the robotic arm's base coordinates are obtained through a teach pendant, recording 100 sets of pose matrices. Simultaneously, one set of checkerboard images is captured under each pose, with indoor lighting adjusted during shooting. Each set includes one image under normal lighting, three images under insufficient lighting, and three overexposed images. The training and test datasets are split according to a certain ratio. In this experiment, an 8:2 ratio is chosen, i.e., 80 sets of robotic arm pose-checkerboard images are used for training, and 20 sets are used for testing. The training set contains 560 checkerboard images taken under different lighting conditions, and the test set contains 140 checkerboard images taken under different lighting conditions. Table 1 briefly describes the composition of the dataset.

[0055] Table 1 Dataset for Hand-Eye Labeling Compensation Algorithm Based on Cross-Attention

[0056]

[0057] Hyperparameter settings

[0058] In this invention, the proposed model is implemented in the PyTorch framework and trained and evaluated on a GeForce RTX 3090 GPU. The hyperparameters used during model training are set as follows: the AdamW optimizer is used to train the model, the initial learning rate is 1e-4, the weight decay is 1e-3, the batch size is set to 10, and the training lasts for 300 epochs.

[0059] Algorithm evaluation metrics

[0060] (1) Matrix prediction accuracy: Hand-eye calibration was performed on 20 chessboard images taken under normal lighting conditions in the test set to obtain the hand-eye matrix X. gt As the true value, the hand-eye matrix X is calculated under three conditions: low lighting, overexposure, and poor lighting (a mixed scene of low lighting and overexposure). i , with X i Relative to X gt The accuracy of the hand-eye calibration algorithm is measured by the similarity between the predicted and actual hand-eye matrix values. A similarity score closer to 1 indicates a closer resemblance between the calculated matrix and the true values, suggesting a more reliable result. Euclidean distance is used to calculate the similarity between the predicted and actual hand-eye matrix values, representing the algorithm's accuracy. The hand-eye matrix is ​​divided into a rotation matrix and a translation vector; the accuracy of these two parts is used to evaluate the algorithm's performance, as shown in the following expression:

[0061]

[0062] Among them, X i X represents the hand-eye matrix predicted under poor lighting conditions. gt This represents the true value of the hand-eye matrix calculated under normal lighting conditions.

[0063] The experiment was conducted in three scenarios: insufficient lighting, overexposure, and poor lighting (including a combination of insufficient and overexposure). The experimental results comparing the proposed algorithm with the traditional hand-eye calibration algorithm are shown in Tables 2 and 3.

[0064] Table 2 shows the comparison results between the proposed algorithm and existing methods when the number of calibration viewpoints is 10.

[0065]

[0066] Table 3 shows the comparison results between the proposed algorithm and existing methods when the number of calibration viewpoints is 20.

[0067]

[0068]

Claims

1. A hand-eye calibration compensation algorithm based on a cross-attention mechanism, characterized in that, include: S1: Acquire image datasets containing calibration objects under three different lighting conditions, including insufficient light, overexposed light, and normal light, and simultaneously record the robot arm pose corresponding to the acquisition of calibration object images. S2: Extract features from poorly lit calibration images and robotic arm poses in the training set so that subsequent models can process them in a lower-dimensional data space; S3: Construct an illumination correction module based on the cross-attention mechanism for image restoration. The low-dimensional features extracted from the input image are fed into the cross-attention module for calculation, and output a 3×3 color correction matrix and a brightness correction value γ to restore a calibration board image that is closer to that taken under normal lighting conditions. S4: Construct a calibration and correction module based on the cross-attention mechanism to compensate for the hand-eye matrix. The image features and pose features are concatenated and then fed into the cross-attention module for calculation. The output is a 3×3 rotation matrix bias and a 1×3 translation vector bias, which are used to correct the traditional hand-eye calibration method. S5: Unify the illumination correction module and the calibration correction module into a complete hand-eye calibration process to form a hand-eye calibration algorithm that uses a cross-attention mechanism for compensation; S6: Use a hand-eye calibration compensation model based on cross-attention mechanism to process the data in the test set and obtain the hand-eye matrix results.

2. The hand-eye calibration compensation algorithm based on cross-attention mechanism according to claim 1, characterized in that, In step S1, the calibration object is placed under different lighting conditions and photographed from different angles to obtain training and testing images, and the calibration object area in each training image is marked using an image annotation tool.

3. The hand-eye calibration compensation algorithm based on cross-attention mechanism according to claim 1, characterized in that, In step S2, convolutional layers are used to extract features from the poorly captured input image and pose matrix so that the model can process the data in a lower-dimensional data space. By concatenating image features and pose features along the same dimension, it is easier to effectively integrate information from different modalities, thereby improving the overall learning performance of the model.

4. The hand-eye calibration compensation algorithm based on cross-attention mechanism according to claim 1, characterized in that, In step S3, the image signal processor process is described using an encoder-decoder structure, and its expression is as follows: Among them, I t For target images taken under normal lighting conditions, I i The input images are taken from the same angle under poor lighting conditions. It is a 3×3 color transformation matrix, where γ represents the brightness correction value and ε is set to 1e-8; Ten queries are set to retrieve color and brightness information from the input data, and then reorganized into a 3×3 color matrix and a 1×1 brightness correction value γ, which is used to restore images taken in poor lighting conditions to images taken in normal lighting conditions.

5. The hand-eye calibration compensation algorithm based on cross-attention mechanism according to claim 1, characterized in that, In step S4, the required error correction information is queried from the input multimodal information, and 12 queries are set to generate 3×3 rotation matrix correction amount and 1×3 translation vector correction amount.

6. The hand-eye calibration compensation algorithm based on cross-attention mechanism according to claim 1, characterized in that, In step S5, a deep learning model for hand-eye calibration compensation is constructed based on the cross-attention mechanism, including: Model input: N chessboard images from the training set used for calibration. and robotic arm pose matrix Model output: Hand-eye calibration matrix M after illumination correction and hand-eye calibration error correction. corr ; The network is trained using a weighted L1 loss function, as follows: Loss=λ img L 1img +λ rot L 1rot +λ trans min(L 1trans (s) Where n is the number of samples, y i It is the true value of the i-th sample. λ is the predicted value of the i-th sample. img , λ rot , λ trans These are the loss weight coefficients for the image, rotation matrix, and translation vector, respectively, used to adjust the contribution of each part to the total loss; s is the maximum value threshold of the translation vector loss. The L1 loss of the translation vector part is compared before being added to the total loss to ensure that its value does not exceed s, thereby avoiding excessive translation loss from having a large impact on the total loss.

7. The hand-eye calibration compensation algorithm based on cross-attention mechanism according to claim 1, characterized in that, In step S6, the collected data is divided into a training set and a validation set. The hand-eye calibration compensation model based on the cross-attention mechanism is trained using the data in the training set, and validated using the data in the validation set.

8. An electronic device, comprising: One or more processors; A storage device for storing one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors perform the steps according to any one of claims 1-7.

9. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the process to perform the steps according to any one of claims 1-7.

Images