Multi-pin circular connector positioning and grabbing method based on RT-DETR

Abstract

The invention discloses a multi-pin circular connector positioning and grabbing method based on RT-DETR. The multi-pin circular connector positioning and grabbing method comprises the following steps: (1) detecting a circular connector; (2) coordinate calibration and mapping: mapping the image coordinates of the circular connector to a world coordinate system to obtain the image center coordinates of the circular connector; (3) estimating a yaw angle: converting the circular connector image from an RGB color space to an HSV space, obtaining a feature region, and calculating a centroid coordinate and the yaw angle of the circular connector image; and (4) robot grabbing and preliminary assembling are conducted, specifically, according to the center coordinates obtained in the step (2) and the yaw angle obtained in the step (3), mechanical arm movement is planned, the circular connector is grabbed from the material panel to the assembling panel, and preliminary buckling is achieved. According to the method, contact force alignment is not needed, the pin safety is high, the system building cost is low, the method is suitable for various circular connectors and high-density multi-hole assembly requirements, and the assembly efficiency and reliability can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of industrial automation and robot vision, and particularly relates to the problems of high-precision target detection and pose estimation in the assembly task of multi-pin circular connectors. Specifically, it is a method for positioning and grasping multi-pin circular connectors based on RT-DETR. Background Technique

[0002] Multi-pin circular connectors are widely used in aviation, aerospace, and precision instruments. Due to their compact size and small pin gaps, they have extremely high requirements for assembly accuracy. Although traditional mechanical guiding methods can achieve basic alignment of pins and sockets, they often face problems of insufficient assembly efficiency and accuracy in multi-pin scenarios and when there is line-of-sight occlusion. At the same time, force feedback control or impedance strategies are prone to cause irreversible damage in the case of easily deformable small pins.

[0003] With the development of computer vision and deep learning, the application of high-precision target detection in complex scenarios has significantly broadened the scope of assembly tasks that robots can handle. As a real-time detection algorithm based on the Transformer concept, RT-DETR has a good balance of speed and accuracy in multi-object detection and is suitable for quickly locating targets with variable shapes and severe local occlusion. However, in the assembly of circular connectors, in addition to detection, it is also necessary to accurately estimate its yaw angle. Factors such as line-of-sight occlusion, compact size, and dense pins greatly increase the assembly difficulty. Therefore, how to combine advanced detection algorithms such as RT-DETR with geometric calibration and image processing methods has become the key technical route to solve the problem of grasping and assembling multi-pin circular connectors. Summary of the Invention

[0004] The purpose of the present invention is to propose a method for positioning and grasping multi-pin circular connectors based on RT-DETR to address the challenges brought about by high-precision requirements and visual occlusion. This method does not require contact force alignment, has high safety for pins, low system construction cost, is applicable to various circular connectors and high-density multi-hole assembly requirements, and can improve assembly efficiency and reliability.

[0005] The technical solution for achieving the purpose of the present invention is as follows:

[0006] A method for positioning and grasping multi-pin circular connectors based on RT-DETR includes the following steps:

[0007] (1) Circular connector detection: Collect an image of the scene where the circular connector is located through an RGB camera, and input the image into the RT-DETR model to locate and crop the bounding box of the circular connector;

[0008] (2) Coordinate calibration and mapping: Using the hand-eye calibration technology and the Homography matrix method, map the image coordinates of the circular connector to the world coordinate system to obtain the image center coordinates of the circular connector;

[0009] (3) Yaw angle estimation: Convert the cropped circular connector image in step (1) from the RGB color space to the HSV space, and obtain the feature region through threshold segmentation, morphological opening operation and Canny edge detection to calculate the centroid coordinates and yaw angle of the circular connector image;

[0010] (4) Robot grasping and preliminary assembly: Plan the movement of the robotic arm according to the center coordinates obtained in step (2) and the yaw angle obtained in step (3), grasp the circular connector from the material panel to the assembly panel and achieve preliminary fastening.

[0011] The specific steps of step (1) are as follows:

[0012] Set the image resolution of the image of the scene where the circular connector is collected by the RGB camera to 640×480 or other preset resolutions to ensure a balance between real-time detection and high-precision recognition by the RT-DETR model; Take the RGB image I rgb as the input and output the bounding box of the circular connector in the image, and use the pre-trained neural network weights θ to process the input image I rgb , to generate the bounding box (x min , x max , y min , y max ), where x min , x max , y min , y max are the pixel coordinates of the bounding box. After obtaining the bounding box of the circular connector, due to the symmetry of the rectangle, calculate the image coordinates of the center position of the circular connector through the following formula (1)

[0013]

[0014] The specific steps of step (2) are as follows:

[0015] Convert the image coordinates of the circular connector to world coordinates: Let M on the M panel be M=(x i , y i , z i ), where 1≤i≤9; Then, detect the edges in the image through the canny edge detection algorithm and calculate the average position P=(u i , v i ), where 1≤i≤9. Since on the M panel (x i , yi , z i ) has a fixed value, where z i represents the height of the M panel in world coordinates. During the position calculation process, the height of the circular connector on the M panel remains unchanged. Therefore, z i is regarded as a constant and is achieved through the Homography matrix. Formula (2) describes the homography calibration method, which converts the point set P in the image captured by the RGB camera into the point set R in the three-dimensional operation of the robot, and uses a 3×3 Homography matrix H to calibrate the world coordinates.

[0016] PH = R, where in Formula (2), the Homography H contains 9 variables. When m≥4, the solution of H is obtained; in this way, the image coordinates of the circular connector are calibrated as world coordinates.

[0017] The specific steps of step (3) are as follows:

[0018] First, convert the cropped circular connector image in step (1) from the RGB color space to the HSV color space, determine the HSV threshold through debugging, and create a binary mask to retain the HSV region. Then, use morphological opening operation to reduce image noise and remove isolated pixels, and use Canny edge detection to detect the contour; draw the contours of the three HSV regions with lines in the image, and then draw the centroid T through Formula (3). j (c xj , c yj ), 1≤j≤3. Assume that T1, T2, and T3 are the contours of the three HSV regions, where M 00 is the area of the contour region, M 10 and M 01 are the first-order moments in the x and y directions respectively.

[0019]

[0020] The angle α between T1T2 and the x-axis of the image is the radian that the robotic arm needs to adjust. Determine whether to perform a 180° rotation adjustment on the image according to the relative position of T3 on the side T1T2 in the image through Formula (4). Then θ is the yaw angle of the robotic arm when grasping the female head of the circular connector.

[0021]

[0022] The obtained world coordinates of the circular connector and the yaw angle θ.

[0023] The specific steps of step (4) are as follows: The robotic arm obtains the center coordinates obtained in step (2) and the yaw angle obtained in step (3), and completes the grasping of the circular connector through a motion control algorithm. After randomly placing it on the material panel, it performs grasping and preliminary alignment. The yaw angle error is assumed to be negligible during the grasping process. If the grasping fails, the detection in step (1) and the calibration process in step (2) are repeated.

[0024] Compared with the traditional mechanical guidance or force control strategy, the method of this technical solution is not only more adaptable to the complex shape of the multi-pin circular connector, but also has the advantages of no need for contact force alignment, high safety for pins, low system construction cost, being applicable to various circular connectors and high-density multi-hole assembly requirements, and can improve the assembly efficiency and reliability. Brief Description of the Drawings

[0025] Figure 1 It is a schematic diagram of the yaw angle estimation process of the circular connector image in the embodiment;

[0026] Figure 2 It is a diagram of the RT-DETR training result. Detailed Embodiment

[0027] The content of the present invention will be further described below in conjunction with the drawings and embodiments, but it is not a limitation of the present invention.

[0028] Embodiment:

[0029] In this example, the circular connector is randomly placed on the material panel, and RT-DETR-R18 inference is performed using a computer with an Intel(R) Core(TM) i9-9900K CPU and 16GB of memory, paired with an NVIDIA GeForce GTX 1080 GPU.

[0030] A method for positioning and grasping a multi-pin circular connector based on RT-DETR includes the following steps:

[0031] (1) Detection of the circular connector: The image of the scene where the circular connector is located is collected by an RGB camera, and the image is input into the RT-DETR-R18 model to locate and crop the bounding box of the circular connector, as Figure 1 (a), Figure 1 (b) shown;

[0032] (2) Coordinate calibration and mapping: Using the hand-eye calibration technology and the Homography matrix method, the image coordinates of the circular connector are mapped to the world coordinate system to obtain the image center coordinates of the circular connector;

[0033] (3) Yaw angle estimation: Convert the cropped circular connector image in step (1) from the RGB color space to the HSV space, and obtain the feature region through threshold segmentation, morphological opening operation, and Canny edge detection to calculate the centroid coordinates and yaw angle of the circular connector image;

[0034] (4) Robot grasping and preliminary assembly: Plan the movement of the robotic arm according to the center coordinates obtained in step (2) and the yaw angle obtained in step (3), and grasp the circular connector from the material panel to the assembly panel and achieve preliminary fastening.

[0035] The specific steps of step (1) are as follows:

[0036] Set the image resolution of the image collected by the RGB camera of the scene where the circular connector is located to 640×480 or other preset resolutions to ensure a balance between real-time detection and high-precision recognition of the RT-DETR-R18 model; Take the RGB image I rgb as the input, and output the bounding box of the circular connector in the image, and use the pre-trained neural network weights θ to process the input image I rgb , to generate the bounding box (x min , x max , y min , y max ), where x min , x max , y min , y max are the pixel coordinates of the bounding box. After obtaining the bounding box of the circular connector, due to the symmetry of the rectangle, calculate the image coordinates of the center position of the circular connector through the following formula (1)

[0037]

[0038] The specific steps of step (2) are as follows:

[0039] Convert the image coordinates of the circular connector to world coordinates: Let M on the M panel be M=(x i , y i , z i ), where 1≤i≤9; then, detect the edges in the image through the canny edge detection algorithm, and calculate the average position P=(u i , v i ), where 1≤i≤9. Since (x i , y i , z i ) values on the M panel are fixed, where z iRepresents the height of the M panel in world coordinates. During the position calculation process, the height of the circular connector on the M panel remains unchanged. Therefore, z i is regarded as a constant and is achieved through the Homography matrix. Formula (2) describes the homography calibration method, which converts the point set P (image coordinates) in the image captured by the RGB camera into the point set R (world coordinates) in the three-dimensional operation of the robot. A 3×3 Homography matrix H is used to calibrate the world coordinates,

[0040] PH = R, where In formula (2), the Homography H contains 9 variables. When m≥4, the solution of H is obtained; in this way, the image coordinates of the circular connector are calibrated as world coordinates.

[0041] The specific steps of step (3) are as follows:

[0042] As Figure 1 shown, first convert the cropped circular connector image in step (1) from the RGB color space to the HSV color space. Determine the HSV threshold of yellow through debugging and create a binary mask to retain the yellow HSV region. Then use morphological opening operation to reduce image noise and remove isolated pixels, and use Canny edge detection to detect the contour; draw the contours of the three yellow HSV regions with green lines in the image, and then draw the centroid T through formula (3) j (c xj ,c yj ), 1≤j≤3. Assume that T1, T2, and T3 are the contours of the three HSV regions, where M 00 is the area of the contour region, M 10 and M 01 are the first-order moments in the x and y directions respectively;

[0043]

[0044] As Figure 1 (c) shown, the angle α between T1T2 and the x-axis of the image is the radian that the robotic arm needs to adjust. Determine whether to perform a 180° rotation adjustment on the image according to the relative position of T3 on the side T1T2 in the image through formula (4), and θ is the yaw angle of the robotic arm when grasping the female head of the circular connector;

[0045]

[0046] The obtained world coordinates of the circular connector and the yaw angle θ.

[0047] The specific steps of step (4) are as follows: The robotic arm obtains the center coordinates obtained in step (2) and the yaw angle obtained in step (3), and completes the grasping of the circular connector through a motion control algorithm. In this example, a UR5 robotic arm and a Robotiq 2F-85 gripper are selected, and grasping and preliminary alignment are performed after being randomly placed on the material panel. The yaw angle error is assumed to be negligible during the grasping process. If the grasping fails, the detection in step (1) and the calibration process in step (2) are repeated.

[0048] The training results are as Figure 2 shown, Figure 2 which is the performance index change curve during the training process of the RT-DETR-R18 model, including the change trends of training / validation loss, Precision, Recall, and mAP with the number of epochs, and is used to evaluate the convergence and detection performance of the RT-DETR-R18 model in the circular connector detection task. Figure 2 The upper half in it is various losses and performance indexes in the training stage, and the lower half is the corresponding indexes in the validation stage. It can be seen from Figure 2 that the loss function of the RT-DETR-R18 model converges rapidly and tends to be stable in both the training stage and the validation stage, showing high detection accuracy and stability. The average detection accuracy of the RT-DETR-R18 model under the mAP@0.5 index can reach 99%, proving its high-quality fitting ability for target regression and classification, and also indicating that the model has good robustness in diverse scenarios.

[0049] The experimental results are shown in Table 1. Five arbitrary positions are selected on the material panel for the grasping experiment. Through multiple experimental tests, when the target detection is accurate, the success rate of the grasping experiment exceeds 99%.

[0050] Table 1

[0051]

Claims

1. A positioning and grasping method for multi-pin circular connectors based on RT-DETR, characterized in that, It includes the following steps: (1) Circular connector detection: Collect an image of the scene where the circular connector is located through an RGB camera, and input the image into the RT-DETR model to locate and crop the bounding box of the circular connector; (2) Coordinate calibration and mapping: Use the hand-eye calibration technique and the Homography matrix method to map the image coordinates of the circular connector to the world coordinate system to obtain the image center coordinates of the circular connector; (3) Yaw angle estimation: Convert the cropped circular connector image in step (1) from the RGB color space to the HSV space, and obtain the feature region through threshold segmentation, morphological opening operation, and Canny edge detection to calculate the centroid coordinates and yaw angle of the circular connector image; (4) Robot grasping and preliminary assembly: Plan the movement of the robotic arm according to the center coordinates obtained in step (2) and the yaw angle obtained in step (3), grasp the circular connector from the material panel to the assembly panel and achieve a preliminary snap fit.

2. The method for positioning and grasping a multi-pin circular connector based on RT-DETR according to claim 1, wherein The specific steps of step (1) are: Set the image resolution of the RGB camera for capturing the image of the scene where the circular connector is located to 640×480 or other preset resolutions to ensure a balance between real-time detection and high-precision recognition of the RT-DETR model; take the RGB image I rgb as the input and output the bounding box of the circular connector in the image. Use the pre-trained neural network weights θ to process the input image I rgb , to generate the bounding box (x min , x max , y min , y max ), where x min , x max , y min , y max are the pixel coordinates of the bounding box. After obtaining the bounding box of the circular connector, due to the symmetry of the rectangle, calculate the image coordinates of the center position of the circular connector through the following formula (1) 3. The multi-pin circular connector positioning and grasping method based on RT-DETR according to claim 1, wherein, The specific steps of step (2) are: Image coordinates of the circular connector are converted to world coordinates: Let M on the M panel be M=(x i , y i , z i ), where 1 ≤ i ≤ 9; then, edges in the image are detected by the canny edge detection algorithm, and the average position P=(u i , v i ) of these edge points is calculated, where 1 ≤ i ≤ 9. Since the values of (x i , y i , z i ) on the M panel are fixed, where z i represents the height of the M panel in world coordinates, and the height of the circular connector on the M panel remains unchanged during the position calculation. Therefore, z i is regarded as a constant. It is achieved through the Homography matrix. Formula (2) describes the homography calibration method, which converts the point set P in the image captured by the RGB camera to the point set R in the three-dimensional operation of the robot. A 3×3 Homography matrix H is used to calibrate the world coordinates. PH = R, where In formula (2), the Homography H contains nine variables. When m ≥ 4, the solution of H is obtained; in this way, the image coordinates of the circular connector are calibrated to the world coordinates.

4. The method for positioning and grasping a multi-pin circular connector based on RT-DETR according to claim 1, characterized in that The specific steps of step (3) are: First, convert the cropped circular connector image in step (1) from the RGB color space to the HSV color space. Determine the HSV threshold through debugging and create a binary mask to retain the HSV region. Then, use morphological opening operations to reduce image noise and remove isolated pixels, and use Canny edge detection to detect the contours. Draw the contours of the three HSV regions with lines in the image, and then draw the centroid T through formula (3). j (c xj ,c yj ), 1 ≤ j ≤ 3. Assume that T1, T2, and T3 are the contours of the three HSV regions, where M 00 is the area of the contour region, and M 10 and M 01 are the first-order moments in the x and y directions, respectively. The angle α between T1T2 and the x-axis of the image is the radian that the robotic arm needs to adjust. Determine whether to perform a 180° rotation adjustment on the image based on the relative position of T3 on the side T1T2 in the image through formula (4). Then θ is the yaw angle of the robotic arm grasping the female head of the circular connector; Obtain the world coordinates of the circular connector and the yaw angle θ.

5. The method for positioning and grasping a multi-pin circular connector based on RT-DETR according to claim 1, wherein The specific steps of step (4) are: The robotic arm completes the grasping of the circular connector according to the center coordinates obtained in step (2) and the yaw angle obtained in step (3). After randomly placing it on the material panel, perform grasping and preliminary alignment. It is assumed that the yaw angle error is negligible during the grasping process. If the grasping fails, repeat the detection in step (1) and the calibration process in step (2).

Images