An automatic hand-eye calibration method based on neural architecture search calibration board detection
By using a calibration board detection model based on neural architecture search, the calibration board pose is automatically determined and image samples are collected, which solves the problem of large sample collection workload in robotic arm hand-eye calibration and achieves efficient and accurate automated calibration.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUILIN UNIVERSITY OF TECHNOLOGY
- Filing Date
- 2022-11-01
- Publication Date
- 2026-06-09
AI Technical Summary
Existing technologies involve a large workload in sample collection during robotic arm hand-eye calibration, which can easily lead to errors, requiring restarts, and also result in low calibration efficiency.
A calibration board detection model based on neural architecture search is adopted. By moving the robotic arm and adjusting the camera viewpoint, the calibration board pose is automatically determined, image samples are collected, and the hand-eye matrix is calculated, reducing manual intervention and improving calibration efficiency.
It achieves efficient automated hand-eye calibration, reduces sample collection workload, improves calibration accuracy and efficiency, and simplifies the operation process.
Smart Images
Figure CN115641384B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of automated hand-eye calibration technology, and more specifically, to an automated hand-eye calibration method based on neural architecture search calibration board detection. Background Technology
[0002] Robotic arms are widely used in industry to replace human labor in tasks such as welding, sorting, and material handling. The movement trajectories of these robotic arms require careful design, which limits their efficiency. Therefore, when the target location is unknown, machine vision technology is often used to guide the robotic arm's movement. An end effector tool needs to be mounted at the flange center of the robotic arm to manipulate the designated target. After the vision system obtains the target coordinates in the camera coordinate system, it is then converted to the tool coordinates at the robotic arm's end effector using a hand-eye matrix. There are generally two camera mounting methods: eye-in-hand and eye-to-hand. In eye-in-hand, the camera is mounted at the end effector of the robotic arm, and its position relative to the flange center is fixed. In eye-to-hand, the camera is mounted at a fixed position in the robotic arm's base coordinate system, and its position relative to the base is fixed.
[0003] The hand-eye matrix is a homogeneous transformation matrix, consisting of a 3x3 rotation matrix and a 3x1 translation matrix. Due to the clearance between the gears in the robotic arm causing backlash error, when acquiring images of different calibration plates during the calibration process, efforts are made to ensure that each axis of the robotic arm has a certain amount of motion to account for backlash error in the calculations. Additionally, the camera lens can cause tangential and radial distortion in the image; therefore, the camera should be positioned facing the calibration plate from various directions during image acquisition to reduce the impact of distortion on the calculations. This results in a significant workload for sample acquisition, and this large workload is prone to errors, requiring the acquisition process to be restarted.
[0004] Existing technology discloses a robot hand-eye calibration method based on neural networks. This method determines the NDI coordinate system, NDI tool coordinate system, and robot coordinate system. Based on the NDI and tool coordinate systems, the position of the tool tip at the end effector (TCP) of the robotic arm is calibrated to obtain the tool tip position. m point sets are collected, each including the coordinate position of the tool tip in the NDI coordinate system, the position of the tool in the robot coordinate system, and the rotation matrix. The transformation matrix from the robot coordinate system to the NDI coordinate system is transformed using the Rodrigues rotation formula, and a forward propagation network is constructed. Based on the forward propagation network, a backward propagation network is generated to obtain the partial derivatives of the parameters. Newton's gradient descent is used to calculate the hand-eye calibration matrix. This invention is convenient, quick, easy to implement, and provides accurate calibration. During the calibration process, the robotic arm does not need to rotate around a single point, and postures can be collected arbitrarily in space. However, this scheme still suffers from the problem of excessive workload in sample collection. Summary of the Invention
[0005] This invention provides an automatic hand-eye calibration method based on neural architecture search calibration board detection, which improves calibration efficiency.
[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:
[0007] An automated hand-eye calibration method based on neural architecture search calibration board detection includes the following steps:
[0008] S1: Calibrate camera internal parameters;
[0009] S2: Deploy the calibration board detection model, which employs neural architecture search;
[0010] S3: The robotic arm moves to the initial point, and the camera looks down at the calibration plate;
[0011] S4: Use the calibration board detection model to determine whether the pose of the calibration board is correct;
[0012] S5: If the pose of the calibration plate is correct, then acquire image samples; if it is incorrect, then estimate the next moving pose of the robotic arm so that the angle of view of the calibration plate under the camera is different, and return to step S4.
[0013] S6: Determine whether enough image samples have been collected. If yes, calculate the hand-eye matrix. If not, estimate the next moving pose of the robotic arm so that the calibration plate has a different angle of view from the camera. Return to step S4.
[0014] S7: Determine whether the calculated hand-eye matrix has reached the expected accuracy. If yes, complete the calibration. If not, estimate the next moving pose of the robotic arm so that the calibration plate has a different angle of view from the camera, and return to step S4.
[0015] Preferably, the calibration board detection model in step S2 includes a backbone network and a detection network. The backbone network adopts a YOLOv4 with a custom backbone, wherein the custom backbone is automatically searched using DARTS. The input of the backbone is a scaled image captured by the camera, and the output is three feature maps of different shapes. The output feature maps are further input into the detection network for decoding and extracting the calibration board detection boxes. The detection network is the Neck in YOLOv4. Finally, non-maximum suppression is used to remove unnecessary detection boxes to obtain the calibration board detection result.
[0016] Preferably, the backbone network includes three searchable modules and three convolutional layers, wherein:
[0017] The scaled image captured by the camera is input to the first convolutional layer. The output of the first convolutional layer is input to the first searchable module and the second searchable module. The output of the first searchable module is input to the second convolutional layer, the second searchable module, and the third searchable module. The second convolutional layer outputs a first feature map to the detection network. The output of the second searchable module is input to the third convolutional layer and the third searchable module. The third convolutional layer outputs a second feature map to the detection network. The third searchable module outputs a third feature map to the detection network.
[0018] Preferably, the first searchable module and the second searchable module are DARTS's Reduction modules, and the third searchable module is the Normal module.
[0019] Preferably, the searchable module has two preprocessing nodes s0 and s1, which receive the outputs of the previous level module and the previous level module, respectively, and four intermediate nodes node1, node2, node3 and node4. The intermediate nodes use the add operation to fuse the feature maps of the input. There are candidate operations between s0 and node1, node2, node3 and node4, between s1 and node1, node2, node3 and node4, between node1 and node2, node3 and node4, between node2 and node3 and node4, and between node3 and node4. After the input passes through the two preprocessing nodes and the four intermediate nodes, the outputs of the four intermediate nodes are fused using the concat operation and then output.
[0020] Preferably, after the searchable module finishes its search, only two of the four intermediate nodes retain the two inputs with the highest weight, and only one candidate operation with the highest weight is retained in each input.
[0021] Preferably, the candidate operations include:
[0022] None indicates that the output is a tensor containing all zeros;
[0023] skip_connect means that the output is the input, and no processing is done on the input.
[0024] avg_pool_3x3 indicates average pooling with a kernel size of 3×3;
[0025] max_pool_3x3 indicates max pooling with a kernel size of 3x3;
[0026] sep_conv_3x3 means four consecutive ordinary convolutions with a kernel size of 5x5;
[0027] sep_conv_7x7 means four consecutive ordinary convolutions with a kernel size of 7x7;
[0028] dil_conv_3x3 represents a dilated convolution with a kernel size of 3x3;
[0029] dil_conv_5x5 represents a dilated convolution with a kernel size of 5x5;
[0030] conv_7x1_1x7 represents two consecutive ordinary convolutions with kernel sizes of 7x1 and 1x7 respectively.
[0031] Preferably, the pose of the calibration plate in step S4 is divided into four types: up, down, left, and right, based on the region of the calibration plate in the sample image.
[0032] Preferably, the sample image acquired by the camera is divided into four regions. The left and right rectangular regions have equal width and height. The middle region between the left and right rectangular regions is divided into upper and lower rectangular regions with equal height and adjustable width.
[0033] Preferably, the calibration board detection model detects which region of the sample image the center of gravity of the calibration board is located in, and obtains the corresponding label.
[0034] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:
[0035] The present invention is an automated hand-eye calibration method based on neural network architecture search. It uses a gradient-based strategy to search a neural network architecture to obtain the backbone of the calibration board detection model. Finally, it determines the state of the calibration board in the camera's view based on the position of the calibration board, and guides the robotic arm to complete the acquisition of the calibration board image, thereby freeing up manual labor and improving calibration efficiency. Attached Figure Description
[0036] Figure 1 This is a schematic diagram of the method flow of the present invention.
[0037] Figure 2 This is a schematic diagram of the neural network structure of the calibration board detection model provided in the embodiment.
[0038] Figure 3 This is a schematic diagram of the searchable module structure in the calibration board detection model provided in the embodiment.
[0039] Figure 4 The example provides a set of calibration board posture labels R, meaning the calibration board is on the right side of the image. Figure a is a top-down view, Figure b is a top-down view to the right, Figure c is a top-down view to the bottom, and Figure d is a top-down view to the top.
[0040] Figure 5 This is a schematic diagram of sample image region division provided for an embodiment.
[0041] Figure 6 The search results for the searchable modules provided in the embodiment are shown in Figure a, which represents the Reduction module, and Figure b, which represents the Normal module.
[0042] Figure 7 A schematic diagram illustrating the change in loss during the training process, provided for an example.
[0043] Figure 8 A schematic diagram of the classification results provided for an example.
[0044] Figure 9 This is a schematic diagram of the detected category samples provided for an example. Detailed Implementation
[0045] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.
[0046] To better illustrate this embodiment, some parts in the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions;
[0047] It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings.
[0048] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.
[0049] Example 1
[0050] This embodiment provides an automatic hand-eye calibration method based on neural architecture search calibration board detection, such as... Figure 1 As shown, it includes the following steps:
[0051] S1: Calibrate camera internal parameters;
[0052] S2: Deploy the calibration board detection model, which employs neural architecture search;
[0053] S3: The robotic arm moves to the initial point, and the camera looks down at the calibration plate;
[0054] S4: Use the calibration board detection model to determine whether the pose of the calibration board is correct;
[0055] S5: If the pose of the calibration plate is correct, then acquire image samples; if it is incorrect, then estimate the next moving pose of the robotic arm so that the angle of view of the calibration plate under the camera is different, and return to step S4.
[0056] S6: Determine whether enough image samples have been collected. If yes, calculate the hand-eye matrix. If not, estimate the next moving pose of the robotic arm so that the calibration plate has a different angle of view from the camera. Return to step S4.
[0057] S7: Determine whether the calculated hand-eye matrix has reached the expected accuracy. If yes, complete the calibration. If not, estimate the next moving pose of the robotic arm so that the calibration plate has a different angle of view from the camera, and return to step S4.
[0058] Example 2
[0059] This embodiment, based on Embodiment 1, continues to disclose the following content:
[0060] The calibration plate detection model described in step S2 is as follows: Figure 2 As shown, the system includes a backbone network and a detection network. The backbone network uses a custom-designed YOLOv4 backbone, which is automatically searched using DARTS. The backbone input is a scaled image captured by the camera, and the output is three feature maps of different shapes. These output feature maps are further input to the detection network for decoding and extracting calibration board detection boxes. The detection network is the Neck in YOLOv4, where the Neck consists of three feature maps of specific shapes: 1×256×52×52, 1×512×26×26, and 1×1024×13×13. Finally, non-maximum suppression is used to remove unnecessary detection boxes, yielding the calibration board detection results.
[0061] The backbone network comprises three searchable modules and three convolutional layers, wherein:
[0062] The scaled image captured by the camera is input to the first convolutional layer. The output of the first convolutional layer is input to the first searchable module and the second searchable module. The output of the first searchable module is input to the second convolutional layer, the second searchable module, and the third searchable module. The second convolutional layer outputs a first feature map to the detection network. The output of the second searchable module is input to the third convolutional layer and the third searchable module. The third convolutional layer outputs a second feature map to the detection network. The third searchable module outputs a third feature map to the detection network.
[0063] The first and second searchable modules are DARTS's Reduction modules, and the third searchable module is the Normal module.
[0064] The searchable module is as follows Figure 3As shown, there are two preprocessing nodes, s0 and s1, which receive the outputs of the previous level module and the module above it, respectively. There are four intermediate nodes, node1, node2, node3 and node4. The intermediate nodes use the add operation to fuse the feature maps of the input. There are candidate operations between s0 and node1, node2, node3 and node4, between s1 and node1, node2, node3 and node4, between node1 and node2, node3 and node4, between node2 and node3 and node4, and between node3 and node4. After the input passes through the two preprocessing nodes and the four intermediate nodes, the concat operation is used to fuse the outputs of the four intermediate nodes and output the result.
[0065] After the search module finishes its search, only two of the four intermediate nodes retain the two inputs with the highest weight, and only one candidate operation with the highest weight is retained in each input.
[0066] The candidate operations and DARTS
[13] The same includes:
[0067] None indicates that the output is a tensor containing all zeros;
[0068] skip_connect means that the output is the input, and no processing is done on the input.
[0069] avg_pool_3x3 indicates average pooling with a kernel size of 3×3;
[0070] max_pool_3x3 indicates max pooling with a kernel size of 3x3;
[0071] sep_conv_3x3 means four consecutive ordinary convolutions with a kernel size of 5x5;
[0072] sep_conv_7x7 means four consecutive ordinary convolutions with a kernel size of 7x7;
[0073] dil_conv_3x3 represents a dilated convolution with a kernel size of 3x3;
[0074] dil_conv_5x5 represents a dilated convolution with a kernel size of 5x5;
[0075] conv_7x1_1x7 represents two consecutive ordinary convolutions with kernel sizes of 7x1 and 1x7 respectively.
[0076] Example 3
[0077] Based on Examples 1 and 2, this embodiment continues to disclose the following content:
[0078] In step S4, the pose of the calibration board is divided into four types: top (T), bottom (B), left (L), and right (R), based on the region of the calibration board in the sample image. Figure 4 This is a set of sample images with the calibration plate in pose R. Since each sample acquisition requires a certain amount of displacement along each axis of the robotic arm, the position of the calibration plate in the images will vary.
[0079] The sample image acquired by the camera is divided into four regions. The left and right rectangular regions have equal width and height. The area between the left and right rectangular regions is further divided into upper and lower rectangular regions with equal height. The width of the upper and lower rectangular regions is adjustable; adjusting the width changes the extent of all four regions. Figure 5 As shown.
[0080] The calibration board detection model detects which region of the sample image the center of gravity of the calibration board is located in, and obtains the corresponding label.
[0081] In step S6, the number of calibration board images n≥12 is a sufficient number of calibration board images.
[0082] When calculating the hand-eye matrix, if the difference is too large, all samples are discarded and images are reacquired.
[0083] In the specific implementation, the model training environment consisted of Ubuntu 20.04, PyTorch 1.11, an Intel i5 CPU, an NVIDIA RTX3090 GPU, and 16GB of DDR4 memory. The model deployment environment was an NVIDIA Jetson Xavier NX intelligent edge computing platform and a ZED 2 stereo camera. Two sets of samples were manually captured, one for training and one for testing, with mAP used as the evaluation metric.
[0084] In this embodiment, two sets of samples were collected under the same lighting environment and device, serving as the training set and the test set, respectively. Each set of samples contained 16 images, including 4 images for each of the 4 poses, with an image resolution of 1920x1080. Two enhancement methods, rotation and brightness variation, were further used for sample enhancement. The rotation angle was between -10° and 10°, and the gamma value used for brightness enhancement was between -0.3 and 0.3. The number of samples in the training set and the test set is shown in Table 1.
[0085] Table 1 shows the original sample distribution and the augmented sample distribution of the dataset.
[0086]
[0087] The Adam optimizer was used to search for the network structure during the search process, and the SGD optimizer was used to optimize the network weights. The parameters of the optimizers are shown in Table 2.
[0088] The table shows the results. The search process used 50 iterations, and the modules found are as follows: Figure 6 As shown. The searched Reduction module prefers to use max pooling with a kernel size of 3x3 during downsampling, and the intermediate nodes discard most of the input from s0 and s1 to keep the deep feature mapping undisturbed. The searched Normal module uses more dilated convolutions and continuous convolutions to obtain feature mappings, and there is no input or output between intermediate nodes, so they do not interfere with each other.
[0089] Table 2. Optimizer parameters used during search and training.
[0090]
[0091] After obtaining the backbone, 100 optimized backbone weights are trained using the Adam optimizer, while freezing other YOLOv4 weights during the training process. The loss during training is as follows: Figure 7 As shown, the loss curve gradually converges, and the mAP obtained in the 10th epoch is as high as 0.98, and remains stable thereafter.
[0092] After obtaining the trained detection model, the pose discrimination method described in Section 3.3 is used to classify the samples in the test set based on the obtained target locations. The test set contains 250 samples. The AP for each class is as follows: Figure 8 As shown, AP = 100% and mAP = 100% for each class. The detected class samples are as follows: Figure 9 As shown, all samples were classified into the correct categories.
[0093] The same or similar labels correspond to the same or similar parts;
[0094] The terms used to describe positional relationships in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent.
[0095] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.
Claims
1. An automatic hand-eye calibration method based on neural architecture search calibration board detection, characterized in that, Includes the following steps: S1: Calibrate camera internal parameters; S2: Deploy the calibration board detection model, which employs neural architecture search; S3: The robotic arm moves to the initial point, and the camera looks down at the calibration plate; S4: Use the calibration board detection model to determine whether the pose of the calibration board is correct; S5: If the pose of the calibration plate is correct, then acquire image samples; if it is incorrect, then estimate the next moving pose of the robotic arm so that the angle of view of the calibration plate under the camera is different, and return to step S4. S6: Determine whether enough image samples have been collected. If yes, calculate the hand-eye matrix. If not, estimate the next moving pose of the robotic arm so that the calibration plate has a different angle of view from the camera. Return to step S4. S7: Determine whether the calculated hand-eye matrix has reached the expected accuracy. If yes, complete the calibration. If not, estimate the next moving pose of the robotic arm so that the calibration plate has a different angle of view under the camera, and return to step S4. The calibration board detection model in step S2 includes a backbone network and a detection network. The backbone network adopts a YOLOv4 with a custom backbone, which is automatically searched using DARTS. The input of the backbone is a scaled image captured by the camera, and the output is three feature maps of different shapes. The output feature maps are further input into the detection network for decoding and extracting the calibration board detection boxes. The detection network is the Neck in YOLOv4. Finally, non-maximum suppression is used to remove unnecessary detection boxes to obtain the calibration board detection results.
2. The automatic hand-eye calibration method based on neural architecture search calibration board detection according to claim 1, characterized in that, The backbone network comprises three searchable modules and three convolutional layers, wherein: The scaled image captured by the camera is input to the first convolutional layer. The output of the first convolutional layer is input to the first searchable module and the second searchable module. The output of the first searchable module is input to the second convolutional layer, the second searchable module, and the third searchable module. The second convolutional layer outputs a first feature map to the detection network. The output of the second searchable module is input to the third convolutional layer and the third searchable module. The third convolutional layer outputs a second feature map to the detection network. The third searchable module outputs a third feature map to the detection network.
3. The automatic hand-eye calibration method based on neural architecture search calibration board detection according to claim 2, characterized in that, The first and second searchable modules are DARTS's Reduction modules, and the third searchable module is the Normal module.
4. The automatic hand-eye calibration method based on neural architecture search calibration board detection according to claim 3, characterized in that, The searchable module has two preprocessing nodes, s0 and s1, which receive the outputs of the previous level module and the module above it, respectively, and four intermediate nodes, node1, node2, node3, and node4. The intermediate nodes use the add operation to fuse the feature maps of the input. There are candidate operations between s0 and node1, node2, node3, and node4; between s1 and node1, node2, node3, and node4; between node1 and node2, node3, and node4; between node2 and node3, node4; and between node3 and node4. After the input passes through the two preprocessing nodes and the four intermediate nodes, the outputs of the four intermediate nodes are fused using the concat operation and then output.
5. The automatic hand-eye calibration method based on neural architecture search calibration board detection according to claim 4, characterized in that, After the search module finishes its search, only two of the four intermediate nodes retain the two inputs with the highest weight, and only one candidate operation with the highest weight is retained in each input.
6. The automatic hand-eye calibration method based on neural architecture search calibration board detection according to claim 5, characterized in that, The candidate operations include: None indicates that the output is a tensor containing all zeros; skip_connect means that the output is the input, and no processing is done on the input. avg_pool_3x3 indicates average pooling with a kernel size of 3×3; max_pool_3x3 indicates max pooling with a kernel size of 3x3; sep_conv_3x3 means four consecutive ordinary convolutions with a kernel size of 5x5; sep_conv_7x7 means four consecutive ordinary convolutions with a kernel size of 7x7; dil_conv_3x3 represents a dilated convolution with a kernel size of 3x3; dil_conv_5x5 represents a dilated convolution with a kernel size of 5x5; conv_7x1_1x7 represents two consecutive ordinary convolutions with kernel sizes of 7x1 and 1x7 respectively.
7. The automatic hand-eye calibration method based on neural architecture search calibration board detection according to claim 6, characterized in that, In step S4, the pose of the calibration plate is divided into four types: up, down, left, and right, based on the region of the calibration plate in the sample image.
8. The automatic hand-eye calibration method based on neural architecture search calibration board detection according to claim 7, characterized in that, The sample image acquired by the camera is divided into 4 regions. The left and right rectangular regions have equal width and height. The middle region between the left and right rectangular regions is divided into upper and lower rectangular regions with equal height and adjustable width.
9. The automatic hand-eye calibration method based on neural architecture search calibration board detection according to claim 8, characterized in that, The calibration board detection model detects which region of the sample image the center of gravity of the calibration board is located in, and obtains the corresponding label.