A Visual Servo Bolt Alignment Method Based on Real-Time Depth Estimation

Abstract

This invention discloses an image-based visual servoing bolt alignment method based on real-time depth estimation, comprising: S1, acquiring RGB images of the component through a camera and performing image processing to obtain the pixel coordinates of the bolt corner points; S2, calculating the homography matrix using a direct linear transformation algorithm; S3, calculating the truncated extrinsic parameter matrix of the camera by combining the homography matrix and the camera intrinsic parameter matrix; S4, reconstructing the complete extrinsic parameter matrix from the bolt coordinate system to the camera coordinate system through rotation matrix orthogonalization and singular value decomposition; S5, iteratively optimizing the complete extrinsic parameter matrix using virtual visual servoing; S6, calculating the depth values ​​of each corner point of the bolt relative to the camera by combining the optimized complete extrinsic parameter matrix and the bolt size; S7, aligning the bolt using an image-based visual servoing control robotic arm. This invention relies only on accurate RGB data and does not employ learning-based or adaptive techniques, thus not placing excessive demands on vision equipment and computing power.

Description

Technical Field

[0001] This invention relates to the field of robotics, and more particularly to an image-based visual servo bolt alignment method based on real-time depth estimation. Background Technology

[0002] With the rapid development of urban rail transit and high-speed rail networks, the maintenance and repair of overhead contact lines have become increasingly crucial. The overhead contact line is a vital component of power supply, and its safety and stability directly affect the normal operation of railway transportation. Among the maintenance aspects of the overhead contact line, bolt tightening is of paramount importance. As a crucial connecting component of the overhead contact line structure, the tightness of the bolts is related to the overall safety of the overhead contact line.

[0003] Traditional manual inspection methods are not only inefficient but also pose numerous safety hazards. Using robots for bolt tightening can significantly improve work efficiency, ensure higher tightening quality, and promote the automation and intelligent development of overhead contact line maintenance. A crucial prerequisite for robots to successfully tighten bolts is ensuring accurate alignment with the bolts. However, due to factors such as geographical environment and manual installation, even the overhead contact line structure along the same line can vary considerably. This presents a significant challenge to the robot's bolt identification and positioning.

[0004] To address the above issues, a technical solution using visual servoing to control a robot for bolt alignment is proposed. Visual servoing is a closed-loop control process that utilizes visual sensors to perceive external information and guide a robot to complete a specific task. Its core is the integration of a vision system with the robot's control system, continuously adjusting the robot's behavior through visual feedback to minimize the error between the current visual features and the desired visual features, thereby achieving the desired control objective. Using visual servoing technology can significantly improve the accuracy, flexibility, and intelligence of robots performing tasks in unstructured scenarios. Based on the different visual feedback information, visual servoing can be divided into two main categories: position-based visual servoing (PBVS) and image-based visual servoing (IBVS). Compared to PBVS, IBVS is less dependent on camera calibration accuracy and does not require prior geometric knowledge of the target object. Two important prerequisites for using IBVS are obtaining accurate image feature point coordinates and corresponding depth information. In particular, depth information significantly affects the convergence speed of the servo control process; inaccurate depth values ​​can even lead to servo failure. However, obtaining accurate depth values ​​is very difficult; currently, deep learning models or advanced vision devices are typically used, but this significantly increases hardware costs. Summary of the Invention

[0005] To address the challenges of obtaining accurate depth values ​​and the high cost of visual servo hardware in existing technologies, this invention proposes an image-based visual servo bolt alignment method based on real-time depth estimation. This method enables a robotic arm to accurately align bolts in unstructured scenes while minimizing hardware costs, thus resolving the aforementioned issues.

[0006] This application discloses an image visual servo bolt alignment method based on real-time depth estimation, characterized by the following steps:

[0007] S1. Acquire RGB images of the components using a camera, and perform image processing using the YOLOv7 deep learning model to obtain the pixel coordinates of the bolt corner points;

[0008] S2. Based on the pixel coordinates of the bolt corner points and the actual size of the bolt, the homography matrix is ​​calculated using the direct linear transformation algorithm. The homography matrix describes the transformation relationship from the bolt coordinate system to the camera coordinate system and the transformation relationship from the camera coordinate system to the pixel coordinate system.

[0009] S3. Combine the homography matrix and the camera intrinsic parameter matrix to calculate the truncated extrinsic parameter matrix of the camera. The camera intrinsic parameter matrix describes the transformation relationship from the camera coordinate system to the pixel coordinate system.

[0010] S4. By orthogonalizing the rotation matrix and reconstructing the singular value decomposition, the complete extrinsic parameter matrix from the bolt coordinate system to the camera coordinate system is restored. The extrinsic parameter matrix describes the transformation relationship from the bolt coordinate system to the camera coordinate system.

[0011] S5. Use virtual visual servoing to iteratively optimize the complete extrinsic parameter matrix;

[0012] S6. Combine the optimized complete extrinsic parameter matrix with the bolt size to calculate the depth values ​​of each corner point of the bolt relative to the camera;

[0013] S7. Using the obtained bolt corner coordinates and depth values, use an image-based visual servoing method to control the robotic arm to align with the bolt.

[0014] Preferably, the image processing in S1 includes object detection, semantic segmentation, and line fitting operations.

[0015] Preferably, the formula for calculating the homography matrix in S2 is as follows:

[0016] The mapping relationship between the pixel coordinates of the bolt corner and the corner coordinates in the bolt coordinate system is as follows:

[0017]

[0018] in, These are the coordinates of the corner points in the bolt coordinate system. Here are the corner coordinates in pixel coordinates, and H is the homography matrix, specifically represented as follows:

[0019]

[0020] Based on the correspondence between the pixel coordinates of the four sets of bolt corner points and the two-dimensional homogeneous coordinates in the bolt coordinate system, the homography matrix H can be calculated.

[0021] Preferably, step S3 includes the following steps:

[0022] The homography matrix is ​​split into a 3×4 camera intrinsic parameter matrix and a 4×3 truncated extrinsic parameter matrix, as shown in the following formula:

[0023]

[0024] Where K is the camera's intrinsic parameter matrix, E is the truncated extrinsic parameter matrix with the original third column removed, and f x f y u0, v0 are the four intrinsic parameters of K, and the complete extrinsic parameter matrix [R,T] describes the bolt coordinate system O. b -X b Y b Z b To camera coordinate system O c -X c Y c Z c The transformation relationship, where R is the rotation matrix and T is the translation vector, is expressed as:

[0025] Expanding equation (4) and solving it yields the elements of the truncated extrinsic parameter matrix E:

[0026]

[0027] The obtained truncated extrinsic parameter matrix E is multiplied by a scaling factor s and then processed.

[0028]

[0029] The scaling factor s has a size of vector [R1, R4, R7]. T and [R2,R5,R8] T The reciprocal of the geometric mean of the length, with the sign according to T. Z The symbol is determined.

[0030] Preferably, step S4 includes the following steps:

[0031] By truncating the extrinsic parameter matrix E, the complete displacement vector T is obtained, and the missing column vectors of the rotation matrix R are recovered through cross product operation:

[0032]

[0033] Singular value decomposition is used to reconstruct the strictly identity orthogonal rotation matrix of R:

[0034] R = USV T (8)

[0035] Where U and V are 3×3 orthogonal matrices, S is a 3×3 non-negative diagonal matrix, and the reconstructed rotation matrix R is the product of matrices U and V;

[0036] This yields the complete extrinsic parameter matrix Q = [R, T]. Further processing of the complete extrinsic parameter matrix is ​​then performed:

[0037]

[0038] Where Q multiplied by F on the left represents the result of rotating the extrinsic parameter matrix Q by 180° around the X-axis of the real camera coordinate system. Bolt coordinate system O b -X b Y b Z b Relative to camera coordinate system O c -X c Y c Z c The complete extrinsic parameter matrix.

[0039] Preferably, step S5 includes the following steps:

[0040] Corner point in bolt coordinate system O b -X b Y b Z b The homogeneous coordinates below are b P = [ b P X , b P Y [,0,1] T Combined with the complete extrinsic parameter matrix Obtain the corner point in the camera coordinate system O c -X c Y c Z c homogeneous coordinates c P = [ c P X , c P Y , c P Z ,1] T The calculation formula is:

[0041]

[0042] in, c P Z The bolt corner point relative to the camera coordinate system O c -X c Y c Z c The depth value;

[0043] Suppose there is a bolt corner point P i In the bolt coordinate system O b -X b Y b Z b The homogeneous coordinates below are The pixel coordinates obtained from image detection are [u i ,v i ] T Based on the principle of perspective modeling, the pixel coordinates [u] are expressed by the following formula. i ,v i ] T Projected onto the image coordinate system O-xy:

[0044]

[0045] Using the complete external parameter matrix P is obtained by forward projection i The projection point on the image coordinate system O-xy is Based on the principle of perspective model:

[0046]

[0047] Set the desired features of the virtual visual servo to The current feature is set to Servo error set to Expanding equation (10) yields the depth values ​​of the feature points:

[0048]

[0049] The formula for calculating the Jacobian matrix of feature points in an image is as follows:

[0050]

[0051] Each corner point All of them are superimposed to obtain the virtual visual servo error e v Then, each corner point All of them are superimposed to obtain the image Jacobian matrix L of virtual visual servoing. v ;

[0052] Visual servo control is performed using the following formula:

[0053]

[0054] Where, λ v For virtual visual servo gain, For L v Moore-Penrose pseudo-reverse, v vc For the spatial velocity of the virtual camera;

[0055] Update the complete extrinsic parameter matrix using T. The calculation formula is as follows:

[0056]

[0057] Where k is the number of iterations, and the updated complete extrinsic parameter matrix is... It is used for forward projection in the next iteration;

[0058] Define an error When err k+1 -err k <10 -8 The virtual vision servo iteration optimization has ended.

[0059] Preferably, step S6 includes the following steps:

[0060] Will b P i =[ b P Xi , b P Yi [,0,1] T Combined with equation (10), the bolt corner point P is obtained. i accurate depth value c P Z .

[0061] Preferably, step S7 includes the following steps:

[0062] Corner pixel coordinates [u] obtained from image detection i ,v i ] T After transformation to the image coordinate system using equation (11), [x] i ,y i ] T , will [x i ,y i ] T With corresponding depth value c P Zi Substituting into equation (14) yields the image Jacobian matrix L corresponding to each corner point. i ;

[0063] Set the desired features of image-based visual servoing as The current feature is set to s i =[x i ,y i ] T The error is set to e at each point i The servo error e is obtained by superimposing the values ​​of each point L. i The images are superimposed to obtain the Jacobian matrix L, and the camera velocity v is obtained through the following formula. c :

[0064] v c =-λL + e; (17)

[0065] Where λ is the image visual servoing gain, L + For the Moore-Penrose pseudoinverse of L;

[0066] Use v c Perform motion control on the robotic arm, and repeat steps S1 to S7.

[0067] Define the total error of an image-based visual servoing. When Err < 0.00005, image-based visual servoing converges, completing bolt alignment.

[0068] The beneficial effects of this invention are:

[0069] (1) This invention relies only on accurate RGB data and does not employ learning-based or adaptive techniques, thus it does not place excessive demands on visual devices and computer computing power.

[0070] (2) The depth estimation algorithm proposed in this invention does not use any related models of the kinematics and dynamics of the robotic arm, which improves the versatility of the algorithm. Finally, by using the obtained bolt corner coordinates and depth values ​​and combining them with IBVS, the robot can be controlled to accurately align with the bolt. Attached Figure Description

[0071] Figure 1 This is a flowchart of the image visual servo bolt alignment method based on real-time depth estimation according to an embodiment of the present invention;

[0072] Figure 2 This is a simulation diagram of the experimental scenario in an embodiment of the present invention;

[0073] Figure 3 The RGB image captured by the camera in this embodiment of the invention;

[0074] Figure 4 This is an image showing the result of the bolt corner point extraction algorithm in an embodiment of the present invention;

[0075] Figure 5 This is a diagram of the bolt coordinate system according to an embodiment of the present invention;

[0076] Figure 6 This is an optical perspective projection model diagram of the camera according to an embodiment of the present invention;

[0077] Figure 7 This is a corner depth estimation curve diagram according to an embodiment of the present invention;

[0078] Figure 8 This is a servo total error curve diagram of an embodiment of the present invention. Detailed Implementation

[0079] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided with reference to the accompanying drawings and embodiments.

[0080] This application discloses an image-based visual servo bolt alignment method based on real-time depth estimation, the process of which is as follows: Figure 1 As shown.

[0081] This embodiment uses an M12 bolt installed on a cantilever arm component as an example, set up in a laboratory environment for overhead contact lines. Figure 2 The method steps proposed in this embodiment are described in detail below. This embodiment uses the UR16e collaborative robot, and the vision device is an Intel Realsense D435 camera. The camera resolution is set to 1920×1080, the frame rate is 30fps, and it is mounted on the end effector of the robotic arm via an eye-in-hand method. The movement of the robotic arm is controlled using the officially provided robot control and data exchange protocol RTDE (Real-Time Data Exchange). The entire process runs on the Ubuntu 20.04LTS operating system, and the software architecture is based on the Visual Servo Platform (VISP) and the Robot Operating System (ROS). The program algorithm is designed using C++ and Python languages. VISP is an open-source visual servo platform developed and designed by the IRISA-Inria team. Specifically, it includes the following steps:

[0082] S1. Acquire RGB images of the components using a camera, and perform image processing using the YOLOv7 deep learning model to obtain the pixel coordinates of the bolt corner points.

[0083] Use a D435 camera to acquire a single frame of RGB image of the entire component, such as Figure 3 As shown. Select the bottom M12 bolt as the target bolt. Using the YOLOv7 deep learning model for image processing, including object detection, semantic segmentation, and line fitting, the pixel coordinates of the bolt corner points can be extracted. The extracted bolt corner points are shown in the image. Figure 4 As shown.

[0084] S2. Based on the pixel coordinates of the bolt corner points and the actual dimensions of the bolt, a direct linear transformation algorithm is used to calculate the homography matrix. The homography matrix describes the transformation relationship from the bolt coordinate system to the camera coordinate system, and the transformation relationship from the camera coordinate system to the pixel coordinate system.

[0085] Because the selected bolt type is M12, its hexagonal surface has a side-to-side distance of 18mm, such as Figure 5 As shown. Assuming in Figure 5 Establish a bolt coordinate system O at the center of the bolt plane. b -X b Y b Z b Currently using [X,Y,1] T To represent a corner point in the bolt coordinate system, a three-dimensional coordinate representation would be used. However, because the Z-coordinates on the bolt plane are all 0, the corner point is rewritten as a two-dimensional homogeneous form with implicitly zero Z-coordinates. This bolt corner point corresponds to... Figure 6 The coordinates in the pixel coordinate system O′-uv are [u,v,1]. T The mapping relationship between the two is unique and can be represented by a homography matrix. This mapping relationship can be obtained by the Direct Linear Transform (DLT) algorithm. The mapping relationship between the pixel coordinates of the bolt corner and the corner coordinates in the bolt coordinate system is as follows:

[0086]

[0087] in, These are the coordinates of the corner points in the bolt coordinate system. Here, H represents the corner coordinates in the corresponding pixel coordinate system, and H is the homography matrix, as shown below:

[0088]

[0089] Based on the correspondence between the pixel coordinates of the four sets of bolt corner points and the two-dimensional implicit homogeneous coordinates in the bolt coordinate system, the homography matrix H can be calculated.

[0090] Because image-based visual servoing typically selects 4 point features, therefore... Figure 4 Points 1, 2, 4, 5 and Figure 5 Points 1, 2, 3, and 4 are used as examples. Substituting these four sets of corresponding points into equation (1) yields the homography matrix H. Expanding the matrix, we get the following equation:

[0091]

[0092] Equation (3) is solved using Gaussian elimination, and the element h9 in H is normalized to 1. Finally, the homography matrix H is obtained.

[0093] S3. Combine the homography matrix and the camera intrinsic parameter matrix to calculate the truncated extrinsic parameter matrix of the camera. The camera intrinsic parameter matrix describes the transformation relationship from the camera coordinate system to the pixel coordinate system.

[0094] The homography matrix is ​​split into a 3×4 camera intrinsic parameter matrix and a 4×3 truncated extrinsic parameter matrix, as shown in the following formula:

[0095]

[0096] Figure 6 This is an optical perspective projection model used in vision devices. The homography matrix H corresponds to the projection transformation and describes the bolt coordinate system O. b -X b Y b Z b The transformation relationship to the pixel coordinate system O′-uv. K is the camera's intrinsic parameter matrix, describing the camera coordinate system O′-uv. c -X c Y c Z c The transformation relationship to the pixel coordinate system O′-uv, and the four intrinsic parameters f in K. x f y u0 and v0 can be obtained beforehand from camera intrinsic parameter calibration. E is the truncated extrinsic parameter matrix with the original third column removed. The reason for the truncation of E is that a two-dimensional homogeneous point with implicitly zero Z was used to calculate matrix H. The complete extrinsic parameter matrix [R,T] describes the bolt coordinate system O. b -X b Y b Z b To camera coordinate system O c -X c Y c Z c The transformation relationship, where R is the rotation matrix and T is the translation vector, is expressed as:

[0097] Expanding equation (4) and solving it yields the elements of the truncated extrinsic parameter matrix E:

[0098]

[0099] Since the column vectors of the rotation matrix are all unit vectors, the obtained truncated extrinsic parameter matrix E needs to be multiplied by a scaling factor s for further processing.

[0100]

[0101] The scaling factor s has a size of vector [R1, R4, R7]. T and [R2,R5,R8] TThe reciprocal of the geometric mean of the length, with the sign according to T. Z The sign is determined. If T Z If the value is greater than 0, then the scaling factor s is negative because... Figure 6 The imaging plane is a virtual plane; the real imaging plane is located behind the camera lens, and its Z-axis is in the real camera coordinate system. C Y C It can also reverse, so T Z The correct sign must remain negative.

[0102] This yields the truncated extrinsic parameter matrix E.

[0103] S4. By orthogonalizing the rotation matrix and reconstructing using singular value decomposition, the complete extrinsic parameter matrix from the bolt coordinate system to the camera coordinate system is restored. The extrinsic parameter matrix describes the transformation relationship from the bolt coordinate system to the camera coordinate system.

[0104] By truncating the extrinsic parameter matrix E, the complete displacement vector T = [T] has been obtained. X ,T Y ,T Z ] T However, the rotation matrix R is still incomplete. Based on the fact that the column vectors of the rotation matrix are orthogonal, the missing third column vector of R can be reconstructed by taking the cross product of the first two column vectors, as shown in the following formula:

[0105]

[0106] The above equation yields the complete rotation matrix R, but it is not strictly orthogonal, so singular value decomposition is still required for reconstruction.

[0107] R = USV T (8)

[0108] Where U and V are 3×3 orthogonal matrices, and S is a 3×3 non-negative diagonal matrix. The reconstructed rotation matrix R is the product of matrices U and V.

[0109] This yields the complete extrinsic parameter matrix Q = [R,T]. However, since the actual imaging plane is located behind the camera lens, the resulting image will be upside down, so the Y-axis of the actual camera coordinate system will be different. C Z C Compared to Figure 6 A reversal occurs, and the subsequent image-based visual servoing in this embodiment uses... Figure 6 Since the camera coordinate system is used, Q needs further processing:

[0110]

[0111] Where Q multiplied by F on the left represents the result of rotating the extrinsic parameter matrix Q by 180° around the X-axis of the real camera coordinate system. Bolt coordinate system O b -X b Y b Z b Relative to camera coordinate system O c -X c Y c Z c The complete extrinsic parameter matrix.

[0112] S5. Use Virtual Visual Servo (VVS) to iteratively optimize the complete extrinsic matrix.

[0113] If the actual dimensions of the bolt are known in advance, the position of the corner point in the bolt coordinate system O can be obtained. b -X b Y b Z b The homogeneous coordinates below are b P = [ b P X , b P Y [,0,1] T Combined with the complete extrinsic parameter matrix Obtain the corner point in the camera coordinate system O c -X c Y c Z c homogeneous coordinates c P = [ c P X , c P Y , c P Z ,1] T The calculation formula is:

[0114]

[0115] in, c P Z The bolt corner point relative to the camera coordinate system O c -X c Y c Z c The depth value. This depth will be used for subsequent image-based visual servoing to control the robot's movement. However, the final depth value obtained through the DLT algorithm... This is inaccurate because the DLT algorithm cannot account for nonlinear distortions. This means the estimated corner depth c P Z This is inaccurate. Therefore, virtual visual servoing is needed next to optimize the iteration matrix.

[0116] Virtual visual servoing treats the pose matrix optimization problem as a visual servoing task based on a 2D image. There is an error between the bolt corner points obtained from image detection and the corresponding corner points obtained by forward projection through the matrix. A virtual camera is defined, and image-based visual servoing is used to minimize this error. When the error is minimized, the pose of the virtual camera is the same as the pose of the real camera.

[0117] Suppose there is a bolt corner point P I In the bolt coordinate system O b -X B Y B Z b The homogeneous coordinates below are The pixel coordinates obtained from image detection are [u i ,v i ] T Based on the principle of perspective modeling, the pixel coordinates [u] are expressed by the following formula. i ,v i ] T Projected to Figure 6 The image coordinate system is O-xy.

[0118]

[0119] Then use the complete extrinsic matrix P is obtained by forward projection i The projection point on the image coordinate system O-xy is Based on the principle of perspective model:

[0120]

[0121] Set the desired features of the virtual visual servo to The current feature is set to Servo error set to Using virtual vision servoing also requires obtaining the depth values ​​of feature points, set as follows: Expanding equation (10) yields the depth values ​​of the feature points:

[0122]

[0123] The formula for calculating the Jacobian matrix of feature points in an image is as follows:

[0124]

[0125] Because there are four feature points, the four corner points are... The virtual visual servo error e is obtained by superimposing all the data. v Then, the four corner points The image Jacobian matrix L is obtained by superimposing the images to obtain virtual visual servoing.v ;

[0126] Visual servo control is performed using the following formula:

[0127]

[0128] Where, λ v It is a manually set virtual visual servo gain. For L v Moore-Penrose pseudo-reverse, v vc The spatial velocity of the virtual camera.

[0129] Because v vc Since ∈se(3), its exponential mapping corresponds to the displacement transformation matrix T∈SE(3), which represents the displacement of the virtual camera per second. The complete extrinsic parameter matrix is ​​updated using T. The calculation formula is as follows:

[0130]

[0131] Where k is the number of iterations, and the updated complete extrinsic parameter matrix is... It is used for forward projection in the next iteration;

[0132] Define an error When err k+1 -err k <10 -8 The virtual vision servoing iteration optimization has ended. It is accurate.

[0133] S6. Combining the optimized complete extrinsic parameter matrix with the bolt size, the depth values ​​of each corner point of the bolt relative to the camera are calculated. Accurate values ​​are obtained. Afterwards, knowing the actual dimensions of the bolt, [the following is done]: b P i =[ b P Xi , b P Yi [,0,1] T Combined with equation (10), the bolt corner point P is obtained. i accurate depth value c P Z .

[0134] S7. Using the obtained bolt corner coordinates and depth values, the image-based visual servoing (IBVS) method is used to control the robotic arm to align with the bolt.

[0135] Corner pixel coordinates [u] obtained from image detection i ,v i ]T After transformation to the image coordinate system using equation (11), [x] i ,y i ] T , will [x i ,y i ] T With corresponding depth value c P Zi Substituting into equation (14) yields the image Jacobian matrix L corresponding to each corner point. i .

[0136] Set the desired features of image-based visual servoing as The current feature is set to s i =[x i ,y i ] T The error is set to e at the four corner points i L i The servo error e and the image Jacobian matrix L are obtained by superimposing them separately. The camera speed v is then obtained by the following formula. c :

[0137] v c =-λL + e; (17)

[0138] Where λ is the manually set image visual servoing gain, L + For L, the Moore-Penrose pseudoinverse.

[0139] Use v c Perform motion control on the robotic arm, and repeat steps S1 to S7.

[0140] Define the total error of an image-based visual servoing. When Err < 0.00005, the image-based visual servoing converges, at which point the bolt alignment is just completed.

[0141] The depth estimation curves of the four corner points of the bolt in a certain control process of IBVS are as follows: Figure 7 As shown, the total error curve of the servo is as follows: Figure 8 As shown, the depth setting before IBVS servo alignment was 40cm, and the expected servo convergence depth was set to 20cm, meaning that the camera plane would be perfectly aligned with the bolt plane when it was 20cm away from the bolt plane. To verify its effectiveness, 10 verification runs were conducted. The results show that the average convergence time for the robot to align with the bolt using IBVS control was 8.723s, and the depth estimation accuracy remained consistently within 2cm, meeting the requirements for automated operations and improving maintenance efficiency.

[0142] In summary, this application aims to enable a robot to quickly align a target bolt in unstructured scenarios. To achieve closed-loop control, visual feedback information is introduced, and an image-based visual servoing method (IBVS) is used to control the robot arm's movement. This method relies on obtaining accurate bolt corner coordinates and their depth values ​​relative to the camera. To obtain accurate bolt corner coordinates in unstructured scenarios, deep learning methods, such as the YOLOv7 model, are necessary. To obtain accurate bolt corner depth values, this application proposes a real-time depth estimation algorithm for coplanar feature points based on the homography transformation principle. This algorithm relies only on accurate RGB data and does not employ learning-based or adaptive methods, thus placing low demands on vision equipment and computing power. Furthermore, the depth estimation algorithm does not use any models related to the robot arm's kinematics and dynamics, improving its versatility. Finally, by combining the obtained bolt corner coordinates and depth values ​​with IBVS, the robot can be controlled to accurately align the bolt.

[0143] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A method for image visual servo bolt alignment based on real-time depth estimation, characterized in that, Includes the following steps: S1. Acquire RGB images of the components using a camera, and perform image processing using the YOLOv7 deep learning model to obtain the pixel coordinates of the bolt corner points; S2. Based on the pixel coordinates of the bolt corner points and the actual dimensions of the bolt, the homography matrix is ​​calculated using a direct linear transformation algorithm. This includes calculating the homography matrix based on the correspondence between the pixel coordinates of the four bolt corner points and the two-dimensional homogeneous coordinates in the bolt coordinate system. ; S3. Combine the homography matrix and the camera intrinsic parameter matrix to calculate the truncated extrinsic parameter matrix of the camera; including the following steps: Decompose the homography matrix into Camera intrinsic parameter matrix and The formula for truncating the extrinsic parameter matrix is ​​as follows: in, This is the intrinsic parameter matrix of the camera. To remove the original truncated extrinsic parameter matrix in the third column, , , , for The four intrinsic parameters, and the complete extrinsic parameter matrix. Describes the bolt coordinate system To the camera coordinate system The transformation relationship, where For rotation matrix, Let be the displacement vector, and its expression is: , ; The obtained truncated extrinsic parameter matrix Multiply by a scaling factor Processing: scaling factor The size is a vector and The reciprocal of the geometric mean of length, the sign is based on The sign is determined; S4. Reconstruct the complete extrinsic parameter matrix from the bolt coordinate system to the camera coordinate system through rotation matrix orthogonalization and singular value decomposition; including the following steps: By truncating the extrinsic parameter matrix Obtain the complete displacement vector The rotation matrix is ​​then obtained by performing a cross product operation. The missing column vector; right Singular value decomposition is used to reconstruct a strictly identity orthogonal rotation matrix, thus yielding the complete extrinsic parameter matrix. Then, the complete extrinsic parameter matrix is ​​further processed to obtain the bolt coordinate system. Relative to camera coordinate system Complete extrinsic parameter matrix ; S5. Use virtual visual servoing to iteratively optimize the complete extrinsic parameter matrix; S6. Combining the optimized complete extrinsic parameter matrix with the bolt size, calculate the depth values ​​of each corner point of the bolt relative to the camera; including the following steps: Corner point in bolt coordinate system The homogeneous coordinates below are Combined with the complete extrinsic parameter matrix The corner points are obtained in the camera coordinate system. homogeneous coordinates The calculation formula is: in, Bolt corner points relative to the camera coordinate system The depth value; Will The bolt corner point is obtained by combining formula (10). accurate depth value ; S7. Using the obtained bolt corner coordinates and depth values, use an image-based visual servoing method to control the robotic arm to align with the bolt.

2. The image visual servo bolt alignment method based on real-time depth estimation according to claim 1, characterized in that, The image processing in S1 includes object detection, semantic segmentation, and line fitting operations.

3. The image visual servo bolt alignment method based on real-time depth estimation according to claim 2, characterized in that, The formula for calculating the homography matrix in S2 is as follows: The mapping relationship between the pixel coordinates of the bolt corner and the corner coordinates in the bolt coordinate system is as follows: The homography matrix is ​​represented as follows: in, These are the coordinates of the corner point in the bolt coordinate system. These are the corner coordinates in the pixel coordinate system.

4. The image visual servo bolt alignment method based on real-time depth estimation according to claim 3, characterized in that, S3 includes the following steps: The above equation represents the truncated extrinsic parameter matrix obtained by expanding and solving equation (4). Each element in the middle.

5. The image visual servo bolt alignment method based on real-time depth estimation according to claim 4, characterized in that, rotation matrix in S4 The missing column vector is: The strictly identity orthogonal rotation matrix is: in, and for Orthogonal matrix for Non-negative diagonal matrix, reconstructed rotation matrix It is a matrix and The product; The complete extrinsic parameter matrix is ​​as follows: in, Left multiplication Representing the extrinsic parameter matrix Rotation around the X-axis of the real camera coordinate system .

6. The image visual servo bolt alignment method based on real-time depth estimation according to claim 5, characterized in that, S5 includes the following steps: Assume there is a bolt corner point In the bolt coordinate system The homogeneous coordinates below are The pixel coordinates obtained from image detection are Based on the principle of perspective modeling, the pixel coordinates are expressed by the following formula. Projected onto the image coordinate system Down: Using the complete extrinsic matrix Obtain by forward projection In the image coordinate system The projection point on is This is derived from the principle of perspective modeling: Set the desired features of the virtual visual servo to The current feature is set to Servo error set to Expanding equation (10) yields the depth values ​​of the feature points: The formula for calculating the Jacobian matrix of feature points in an image is as follows: Each corner point All of them are superimposed to obtain the virtual visual servo error. Then, each corner point All of them are superimposed to obtain the image Jacobian matrix of virtual visual servoing. ; Visual servo control is performed using the following formula: in, For virtual visual servo gain, for Moore-Penrose pseudo-reverse, For the spatial velocity of the virtual camera; use Update the complete extrinsic parameter matrix The calculation formula is as follows: in, The updated complete extrinsic parameter matrix is ​​used to optimize the number of iterations. It is used for forward projection in the next iteration; Define an error ,when The virtual vision servo iteration optimization has ended.

7. The image visual servo bolt alignment method based on real-time depth estimation according to claim 6, characterized in that, S7 includes the following steps: Corner pixel coordinates obtained from image detection Transform it to the image coordinate system using equation (11). ,Will With corresponding depth value Substituting into equation (14) yields the image Jacobian matrix corresponding to each corner point. ; Set the desired features of image-based visual servoing as The current feature is set to The error is set to , to each point The servo error is obtained by superimposing the results. , to each point The image Jacobian matrix is ​​obtained by superimposing the data. The camera speed is obtained using the following formula. : in, For image visual servo gain, for Moore-Penrose pseudo-inverse; use Perform motion control on the robotic arm, and repeat steps S1 to S7. Define the total error of an image-based visual servoing. ,when The bolt alignment is completed by visual servo convergence based on the image.

Images