Bolt and nut pose estimation method based on image and point cloud

Abstract

The invention discloses a bolt and nut pose estimation method based on an image and a point cloud, and the method comprises the steps: carrying out the systematic collection of a bolt and nut image through a mechanical arm, constructing a high-quality data set, carrying out the training based on a YOLOv8 algorithm, and achieving the quick positioning of a bolt and a nut in a two-dimensional image; secondly, a target in the two-dimensional image is mapped to a three-dimensional point cloud through a camera internal reference matrix, point cloud data of the bolt and the nut are segmented, and outliers are removed through statistical filtering; aiming at the pose estimation of the bolt, innovatively using an RANSAC three-dimensional circle fitting algorithm to calculate the circle center and the normal vector of the upper surface of the bolt; for the nut, the plane of the nut is recognized through RANSAC plane fitting, and the pose of the nut is obtained in combination with a three-dimensional circle fitting algorithm; according to the method, the intelligent level and efficiency of overhead line system maintenance are remarkably improved, the risk and labor intensity of manual maintenance are reduced, and technical support is provided for intelligent operation and maintenance of electrified railways.

Description

Technical Field

[0001] The present invention relates to the field of catenary boom maintenance, and particularly to a method for estimating the pose of bolts and nuts based on images and point clouds. Background Art

[0002] The railway catenary is a key component for power supply in electrified railways. The catenary system is responsible for providing power to trains, being an important part of the electrified railway system and the only energy source for electrified railways. The purpose of the operation and maintenance of electrified railways is to ensure the stable operation of the traction power supply system, mainly relying on the daily maintenance of the catenary. The bolt tightening condition of the catenary boom is the main task of catenary maintenance. Currently, the catenary maintenance mode is manual inspection of the bolt status. Although the maintenance tasks can be basically completed, there are problems such as the need to invest a large amount of human resources, high operation intensity, high risk, harsh environment, low operation quality standard, and lack of management and analysis of operation data.

[0003] In the field of catenary maintenance, there are the following technical drawbacks in the current identification of bolts of catenary boom connectors: both the bolt nut and the connector are metals, with high similarity in color and material, making it difficult to identify; the installation position of the nut is on the bolt, resulting in the screw rod blocking the nut, causing the lack of nut point cloud information, and the length of the screw rod exceeding the nut part is random, increasing the difficulty of nut pose identification; the gap between the bolt tightening sleeve and the bolt nut is about 1 mm, so a high precision requirement for bolt pose estimation is needed, with an error within 1 mm, posing high requirements for the resolution of the three-dimensional point cloud and the accuracy of the algorithm. Summary of the Invention

[0004] In order to overcome the disadvantages and deficiencies of the prior art, the present invention provides a method for estimating the pose of bolts and nuts based on images and point clouds.

[0005] A method for estimating the pose of bolts and nuts based on images and point clouds, the method comprising:

[0006] Step S1: Using a catenary maintenance robotic arm equipped with a depth point cloud camera, obtaining bolt image data under different angles, lighting conditions, and backgrounds through the control variable method, and collecting a catenary boom bolt nut data set;

[0007] Step S2: Using the catenary boom bolt nut data set collected in Step S1, selecting appropriate model parameters according to the performance of the computing platform, training the YOLOv8 algorithm, and obtaining a trained YOLOv8 model;

[0008] Step S3: After the catenary maintenance robot equipped with a depth point cloud camera moves in front of the boom connector, taking a static shot to obtain a two-dimensional grayscale image and a three-dimensional point cloud image of the connector;

[0009] Step S4: locating the bolts and nuts based on the YOLOv8 deep learning target detection algorithm;

[0010] Step S5: Segment the bolt and nut point cloud according to the camera intrinsic parameter matrix;

[0011] Step S6: The method of segmenting the point cloud by a rectangular range may result in outliers that are far away in the z-axis direction being included, which need to be removed by a point cloud statistical filtering method;

[0012] Step S7: Calculate the bolt pose based on the RANSAC three-dimensional circle fitting algorithm. The circle parameters in the three-dimensional space are defined as: center c, normal vector n and radius r. The three-dimensional circle can be regarded as a two-dimensional circle on a plane. The geometric constraints are that all points are located in the plane where the circle is located and the distance from the point to the center of the circle is less than the radius. RANSAC performs plane fitting through four steps: random sampling, model assumption, interior point verification and iterative optimization.

[0013] Step S8: for the inner points of the three-dimensional circle obtained in step S7, the circle fitting is optimized using the Levenberg-Marquardt algorithm;

[0014] Step S9: Identify the position and posture of the nut by identifying multiple planes, including the upper plane of the screw end, the nut plane, and the bottom connecting piece plane.

[0015] Furthermore, the step S1 comprises:

[0016] Step S11: In the x-axis direction, 7 sample points are collected at intervals of 5 cm, in the z-axis direction, 6 sample points are collected at intervals of 4.5 cm, and in the y-axis direction, based on the point cloud collection range of the depth point cloud camera, 3 sample points are collected at intervals of 10 cm starting from 30 cm;

[0017] Step S12: changing the shooting angle, shooting object and lighting conditions, and repeating step S11 to obtain multiple images of contact network bolts and nuts;

[0018] Step S13: Screening and sorting the images collected in step S12, removing incomplete, overexposed or dark images of bolts, and constructing a bolt image dataset;

[0019] Step S14: annotate the collected image and select the bolts and nuts in the image with a rectangular frame.

[0020] Furthermore, the step S4 comprises:

[0021] Step S41: Load the trained YOLOv8 target detection model, perform target detection processing on the image, and obtain the confidence of each detection part;

[0022] Step S42: Select the top five detection targets with confidence. Use the K-means clustering algorithm to divide the data into high-value and low-value data, and eliminate the low-value data;

[0023] Step S43: According to the screening in Step S42, obtain the image border coordinates of the bolt and nut.

[0024] Further, the said Step S5 includes:

[0025] Step S51: The camera internal parameter matrix K, as the core parameter of the camera imaging model, defines the projection transformation relationship from the three-dimensional space point to the image pixel coordinate system. The expression is:

[0026]

[0027] where f x , f y are the focal lengths of the x and y axes, with the unit of pixel, and c x , c y are the pixel coordinates of the optical center in the image.

[0028] For the three-dimensional point P=(X, Y, Z) in the camera coordinate system, assuming the depth direction of the camera is the z-axis, the pixel coordinates (u, v) projected onto the image coordinate system are:

[0029]

[0030] Step S52: According to the correspondence between the three-dimensional points and two-dimensional pixels obtained in Step S51, calculate the point cloud corresponding to the pixels within the bolt and nut rectangular frame in Step S43, segment it, and eliminate the background invalid point cloud information to obtain the point cloud of the bolt and nut.

[0031] Further, the said Step S6 includes:

[0032] Step S61: For each point p i in the point cloud, calculate its neighborhood N i . Use K-nearest neighbor or radius search to determine the neighborhood range of each point. K-nearest neighbor selects the K points closest to the target point as the neighborhood, and radius search determines the neighborhood by setting a fixed radius;

[0033] Step S62: For each point p i , calculate its average distance d i to the neighborhood points N i :

[0034]

[0035] where p j belongs to the neighborhood Ni The points in

[0036] Step S63: Calculate the mean μ and standard deviation σ of the average distances of all points:

[0037]

[0038] where N is the total number of points in the point cloud, and i represents the index value from 1 to N.

[0039] Step S64: Set the threshold range [μ - ασ, μ + ασ], where α is for each point p i , if its average distance d i exceeds the threshold range, it is determined as an outlier point, and the outlier points need to be removed.

[0040] Furthermore, the said step S7 includes:

[0041] Step S71: Three-dimensional circle fitting requires at least 3 non-collinear points. In each iteration, the algorithm randomly selects 3 points as the initial hypothesis. If the 3 points are collinear, that is, the vectors formed by the three points are linearly dependent, then the unique plane cannot be determined, and resampling is required;

[0042] Step S72: Based on the 3 sampled points P1(x1, x1, x1), P2(x2, x2, x2), P3(x3, x3, x3), calculate the three-dimensional circle parameters:

[0043] Determine the plane where the circle is located and calculate the plane normal vector n:

[0044] n = (P2 - P1) × (P3 - P1);

[0045] Define the points on the plane as P(x, y, z), and obtain the plane equation: n(P - P1) = 0. Project the three sampled points onto the plane coordinate system to obtain the two-dimensional coordinates Q1, Q2, Q3, and solve the center (a, b) and radius r of the two-dimensional circle by algebraic method:

[0046] (Q ix - a) 2 + (Q iy - b) 2 = r 2 (i = 1, 2, 3);

[0047] Then back-project the two-dimensional center (a, b) into three-dimensional space to obtain the three-dimensional center c.

[0048] Step S73: For all points P in the point cloud i , calculate the distance from it to the three-dimensional center, which includes two parts. One is the distance from the point to the plane:

[0049] dp = |n(P i - c)|;

[0050] One is the distance from a point in a plane to the center of the circle:

[0051] d c = ||P i - c|| - r;

[0052] The distance from a point to a three - dimensional circle is defined as If the distance is less than a preset threshold, it is determined as an inlier. The threshold is usually set according to the sensor noise characteristics;

[0053] Step S74: Dynamically adjust the maximum number of iterations according to the current optimal inlier ratio :

[0054]

[0055] where p is the confidence level, which is used to ensure that the probability of at least one sampling being all inliers reaches p;

[0056] Step S75: By iterating steps S71 - S74, retain the three - dimensional circle parameters with the largest number of inliers, and the set of inliers belonging to this three - dimensional circle.

[0057] Furthermore, the said step S8 includes:

[0058] Step S81: Define the error function e i as the difference between the distance from the inlier p i (x i , y i , z i ) of the three - dimensional circle obtained in step S7 to the circle and its radius r:

[0059] e i (c, r)= ||p i - c|| - r;

[0060] where ||p i - c|| is the Euclidean distance from the point p i to the center of the circle c(c x , c y , c z );

[0061]

[0062] The goal is to minimize the sum of the squared errors E of all points:

[0063]

[0064] Step S82: Use the center and radius obtained in Step S7 as the initial center c0 and the initial radius r0. Define the number of iterations as k, then the center and radius for each iteration are c k and r k . Initialize the damping factor λ to 0.01.

[0065] Step S83: Calculate the residual vector e(c k , r k ), where each element is:

[0066] e i = ||p i - c k || - r k ;

[0067] Step S84: Calculate the Jacobian matrix J, where each row corresponds to the partial derivative of the error function of a point with respect to the parameters:

[0068]

[0069] Step S85: Calculate the update amount Δθ (where θ = [c x , c y , c z , r] T ):

[0070] (J T J + λI)Δθ = -J T e;

[0071] where I is the identity matrix.

[0072] Step S86: Update the parameters:

[0073] θ k+1 = θ k + Δθ;

[0074] Step S87: Calculate the new error E(θ k+1 ). If the error decreases, accept the update and decrease the damping factor; if the error increases, reject the update and increase the damping factor.

[0075] Step S88: Iteratively update Steps S83 - S87 and stop the iteration when one of the following conditions is met: the error E(θ k ) is less than the preset threshold; the parameter update amount ||Δθ|| is less than the preset threshold; the maximum number of iterations is reached.

[0076] Furthermore, Step S9 includes:

[0077] Step S91: Calculate all planes in the overall point cloud of the nut using the RANSAC plane fitting algorithm, and select the three planes with the top three inlier counts. Their inlier sets are respectively defined as N1, N2, and N3.

[0078] Step S92: Calculate the average distance d i from the origin (0, 0, 0) of the camera coordinate system to the inlier set N i of the plane:

[0079]

[0080] where N is the number of inliers in the plane, and p i is an inlier in the plane.

[0081] Step S93: According to the geometric relationship of the three planes, it can be known that the nut plane is located between the upper plane of the screw end and the connecting piece plane. Sort d i from large to small, and select the second one and mark it as the nut plane.

[0082] Step S94: Perform the three-dimensional circle fitting of steps S7 - S8 on the nut plane to obtain the pose of the nut.

[0083] Beneficial effects:

[0084] The present invention proposes a method for estimating the poses of bolts and nuts based on images and point clouds, which has the following effects and advantages:

[0085] 1. The present invention uses the YOLOv8 algorithm to identify bolts in two-dimensional images. First, for the acquisition of the bolt data set, the present invention uses a robotic arm to carry a camera to collect bolt images in a matrix form, and changes lighting conditions, tilt angles, etc., which can quickly obtain a large number of rich data sets, improving the accuracy and robustness of algorithm recognition.

[0086] 2. The present invention uses the internal parameter matrix of the depth point cloud camera to obtain the correspondence between the two-dimensional image and the three-dimensional point cloud, and can map the bolt and nut rectangular frames recognized by YOLOv8 into the three-dimensional point cloud, so as to achieve the purpose of segmenting the bolt and nut point clouds. This method utilizes the accuracy and rapidity of the two-dimensional image deep learning object recognition algorithm, solves the difficulty of object recognition in three-dimensional point clouds, eliminates a large number of background invalid point clouds, and provides a solid foundation for subsequent pose detection.

[0087] 3. The present invention uses the statistical filtering method to eliminate the miscellaneous points far from the bolt and nut point clouds, improving the efficiency of subsequent pose detection.

[0088] 4. For the pose detection of bolts, based on the structural characteristic that the upper surface of the bolt is circular, the RANSAC three-dimensional circle fitting algorithm is innovatively used to calculate the center and normal vector of the upper surface of the bolt, thereby obtaining the pose information of the bolt. For nuts with a more complex structure, based on the structural relationship among the nut, screw, and connecting piece, first, the RANSAC plane detection is used to obtain the inliers of the upper plane of the nut, the end plane of the screw, and the bottom connecting piece plane, and then the distances from them to the origin are calculated to determine which one is the nut plane. Then, the RANSAC three-dimensional circle fitting algorithm is used for the inliers of the nut plane to obtain the nut pose. Description of the Drawings

[0089] Figure 1 is the overall step flow chart of the present invention;

[0090] Figure 2 are the two-dimensional grayscale image and three-dimensional point cloud image of the catenary boom connecting piece;

[0091] Figure 3 is the schematic diagram of the YOLOv8 algorithm for identifying bolts and nuts;

[0092] Figure 4 is the schematic diagram of using the bolt and nut detection frame to segment the point cloud;

[0093] Figure 5 is the result diagram of identifying the bolt pose;

[0094] Figure 6 is the result diagram of identifying the nut pose;

[0095] Figure 7 is the display schematic diagram of all bolts and nuts in the entire point cloud. Detailed Embodiment

[0096] It should be noted that, without conflict, the embodiments and the features in the embodiments in this application can be combined with each other. The following further describes this application in detail with reference to the drawings and specific embodiments.

[0097] As Figure 1 shown, a method for estimating the poses of bolts and nuts based on images and point clouds, the method includes:

[0098] Step S1: Use the catenary maintenance robotic arm to carry a high-precision depth point cloud camera, and obtain bolt image data under different angles, lighting conditions, and backgrounds through the control variable method, and collect the catenary boom bolt and nut data set.

[0099] Specifically, first is the installation method of the depth camera. The depth camera should be installed at the end of the robotic arm in the "eye-in-hand" manner. Then, set the movement range of the robotic arm according to the camera's field of view. On the basis of ensuring that the camera can capture the complete catenary wrist arm connector, set the movement step size of the robotic arm in the x, y, and z directions, and then move the robotic arm for matrix shooting. Then, change the shooting angle and lighting conditions and repeat the matrix shooting. Finally, screen and label the captured images to obtain a rich dataset of catenary bolts and nuts.

[0100] Step S11: In the x-axis (catenary lateral distance) direction, collect 7 sample points at intervals of 5 cm; in the z-axis (catenary height) direction, collect 6 sample points at intervals of 4.5 cm; in the y-axis (distance between the camera and the catenary) direction, based on the effective point cloud acquisition range of the depth point cloud camera, starting from 30 cm, collect 3 sample points at intervals of 10 cm.

[0101] Step S12: Change the shooting angle, shooting object, and lighting conditions, and repeat Step S11 to obtain multiple images of catenary bolts and nuts. Through this systematic sampling method, the diversity and representativeness of the dataset are ensured.

[0102] Step S13: Strictly screen and sort the images collected in Step S12, eliminate the images with incomplete bolts, overexposure, or underexposure, and construct a dataset of high-quality catenary bolt images to lay a reliable data foundation for subsequent model training.

[0103] Step S14: Label the collected images and use a rectangular box to select the bolts and nuts in the images.

[0104] Step S2: Use the dataset of catenary wrist arm bolts and nuts collected in Step 1, select appropriate model parameters according to the performance of the computing platform, and train the YOLOv8 algorithm to obtain a trained YOLOv8 model.

[0105] Specifically, after obtaining the dataset through step S1, it is necessary to train the YOLOv8 model. First, the training parameters of the YOLOv8 model are systematically configured: the detection type is set to single-class detection (type = 1), the number of training epochs is set to 200, the batch size is configured to 8, the number of parallel training threads is set to 2, and the intersection over union (IoU) threshold is determined to be 0.7. The experimental platform uses high-performance computing devices, and the specific configuration is as follows: the CPU is Intel i5 12900, the graphics card is equipped with NVIDIA GeForce RTX 4090, and the system memory is 32GB. Given the excellent hardware performance of the experimental equipment, and the single detection target category and relatively stable environmental conditions, this study selects the pre-trained YOLOv8n model weights as the initial parameters. This parameter configuration scheme not only ensures the training efficiency of the model but also fully considers the optimized utilization of hardware resources.

[0106] Step S3: After the catenary inspection robot equipped with a depth point cloud camera moves in front of the wrist arm connector, it takes a static shot to obtain a two-dimensional grayscale image and a three-dimensional point cloud image of the connector.

[0107] Specifically, by manually operating the lifting and rotation of the vehicle-mounted mobile platform, the robotic arm is moved in front of the connector to be inspected. Then, the robotic arm is controlled to expand from the retracted posture to the working posture. Then, according to the pose of the connector scanned by the radar in advance, it is converted to the base coordinate system of the robotic arm. Finally, the robotic arm is moved directly in front of the wrist arm connector, and the depth point cloud camera carried by the end of the robotic arm takes a shot to obtain a two-dimensional grayscale image and a three-dimensional point cloud image of the wrist arm connector, as Figure 2 shown.

[0108] Step S4: Locate the bolts and nuts based on the YOLOv8 deep learning object detection algorithm.

[0109] Specifically, the trained YOLOv8 model is used to identify the bolts and nuts in the two-dimensional image, and multiple recognition targets are obtained. Then, the top five are selected according to the confidence levels of the recognition targets, and these five recognition results are classified into high values and low values by K-nearest neighbor. After eliminating the low values, the rectangular frames of the bolts and nuts in the two-dimensional image are obtained, as Figure 3 shown.

[0110] Step S41: Load the trained YOLOv8 object detection model, perform object detection processing on the image, and obtain the confidence levels of each detection part.

[0111] Step S42: The shapes of the catenary mast arm connectors are different, and the number of bolts and nuts on them is uncertain, but there are at most 5. Therefore, select the top five detection targets with confidence. There is an obvious gap between the data with higher confidence and the lower ones. Use the K-means clustering algorithm to divide the data into high-value and low-value, and eliminate the low-value data.

[0112] Step S43: According to the screening in Step S42, obtain the image border coordinates of the bolts and nuts.

[0113] Step S5: Segment the bolt and nut point clouds according to the camera internal parameter matrix.

[0114] Specifically, according to the internal parameter matrix of the depth camera, calculate the correspondence between the pixels in the two-dimensional image and the point cloud sequence numbers, map the bolt and nut rectangular frames obtained in Step S3 to the three-dimensional point cloud, and then segment out the bolt and nut point clouds, as Figure 4 shown.

[0115] Step S51: The camera internal parameter matrix K, as the core parameter of the camera imaging model, defines the projection transformation relationship from the three-dimensional space point to the image pixel coordinate system. The expression is:

[0116]

[0117] where f x , f y are the focal lengths of the x and y axes, with the unit of pixel, and c x , c y are the pixel coordinates of the optical center in the image.

[0118] For the three-dimensional point P=(X, Y, Z) in the camera coordinate system, assuming the depth direction of the camera is the z-axis, the pixel coordinates (u, v) projected onto the image coordinate system are:

[0119]

[0120] Step S52: According to the correspondence between the three-dimensional points and two-dimensional pixels obtained in Step S51, calculate the point cloud corresponding to the pixels within the bolt and nut rectangular frames in Step S43, segment it out, and eliminate the background invalid point cloud information to obtain the bolt and nut point clouds.

[0121] Step S6: The method of segmenting the point cloud through a rectangular range may include outlier points with a relatively large distance in the z-axis direction, which need to be removed by the method of point cloud statistical filtering.

[0122] Specifically, calculate the neighborhood range for each point in the segmented point cloud, which can be determined by K-nearest neighbor or radius search; then calculate the average distance from each point to its neighborhood points, and then calculate the mean and standard deviation of the average distances of all points, and set a dynamic threshold range based on this. Finally, if the average distance of a certain point exceeds this threshold range, it is determined as an outlier and removed.

[0123] Step S61: For each point p in the point cloud i , calculate its neighborhood N i , and use K-nearest neighbor or radius search to determine the neighborhood range of each point. K-nearest neighbor selects the K points closest to the target point as the neighborhood, and radius search determines the neighborhood by setting a fixed radius;

[0124] Step S62: For each point p i , calculate its average distance d i to the neighborhood points N i :

[0125]

[0126] where p j belongs to the points in the neighborhood N i .

[0127] Step S63: Calculate the mean μ and standard deviation σ of the average distances of all points:

[0128]

[0129] where N is the total number of points in the point cloud, and i represents the index value from 1 to N.

[0130] Step S64: Set the threshold range [μ - ασ, μ + ασ], where α is for each point p i , if its average distance d i exceeds the threshold range, it is determined as an outlier, and outliers need to be removed.

[0131] Step S7: Calculate the bolt pose based on the RANSAC three-dimensional circle fitting algorithm. The circle parameters in three-dimensional space are defined as: center c, normal vector n, and radius r. The three-dimensional circle can be regarded as a two-dimensional circle on a plane, and the geometric constraint conditions are that all points are in the plane where the circle is located and the distance from the point to the center of the circle is less than the radius. RANSAC performs plane fitting through four steps: random sampling, model assumption, inlier verification, and iterative optimization;

[0132] Specifically, first, randomly sample 3 non - collinear points as the initial hypothesis. If they are collinear, resample. Calculate the plane where the circle lies and its normal vector based on the sampled points. Project the points onto the plane and then fit the two - dimensional circle parameters, and finally back - project them into three - dimensional space to obtain the initial circle. Subsequently, calculate the distances from all points to this three - dimensional circle (including the distance from the point to the plane and the distance from the point to the center of the circle within the plane). Determine the points with distances less than the threshold as inliers. The algorithm dynamically adjusts the maximum number of iterations according to the current inlier ratio, and through multiple iterations, retains the three - dimensional circle parameters with the most inliers and its inlier set.

[0133] Step S71: Three - dimensional circle fitting requires at least 3 non - collinear points. In each iteration, the algorithm randomly selects 3 points as the initial hypothesis. If the 3 points are collinear, that is, the vectors formed by the three points are linearly dependent, then a unique plane cannot be determined, and resampling is required;

[0134] Step S72: Based on the 3 sampled points P1(x1,x1,x1), P2(x2,x2,x2), P3(x3,x3,x3), calculate the three - dimensional circle parameters:

[0135] Determine the plane where the circle lies and calculate the plane normal vector n:

[0136] n = (P2 - P1) × (P3 - P1);

[0137] Define the points on the plane as P(x,y,z), and obtain the plane equation: n(P - P1) = 0. Project the three sampled points onto this plane coordinate system to obtain the two - dimensional coordinates Q1, Q2, Q3, and solve for the center (a,b) and radius r of the two - dimensional circle by algebraic method:

[0138] (Q ix -a) 2 +(Q iy -b) 2 =r 2 (i = 1,2,3);

[0139] Then back - project the two - dimensional center (a,b) into three - dimensional space to obtain the three - dimensional center c.

[0140] Step S73: For all points P i in the point cloud, calculate the distance from it to the three - dimensional center, which includes two parts. One is the distance from the point to the plane:

[0141] d p =|n(P i -c)|;

[0142] One is the distance from the point to the center within the plane:

[0143] d c =||P i -c||-r;

[0144] The distance from a point to a 3D circle is defined as If the distance is less than a preset threshold, it is determined as an inlier. The threshold is usually set according to the sensor noise characteristics;

[0145] Step S74: According to the current optimal inlier ratio Dynamically adjust the maximum number of iterations:

[0146]

[0147] where p is the confidence level, which is used to ensure that the probability of having all inliers in at least one sampling reaches p;

[0148] Step S75: By iterating steps S71 - S74, retain the 3D circle parameters with the largest number of inliers, and the set of inliers belonging to this 3D circle.

[0149] Step S8: For the 3D circle inliers obtained in step S7, use the Levenberg - Marquardt algorithm to optimize the circle fitting;

[0150] Specifically, first construct an error function using the inlier set and the initial circle parameters obtained in the RANSAC stage. The algorithm iteratively calculates the residuals of each point to the current circle and constructs a Jacobian matrix to characterize the sensitivity of the error to each parameter. In each iteration, the system calculates the parameter update amount by solving a system of linear equations and dynamically adjusts the update step size using a damping factor: increasing the step size to accelerate convergence when the error decreases, and decreasing the step size to improve stability when the error increases. Continuously monitor the error change and the parameter update amplitude during the iteration process. Terminate the optimization when the preset accuracy requirement is met, the parameters converge, or the maximum number of iterations is exceeded, and finally output the optimized 3D circle parameters. Display the center and the normal vector to the bolt point cloud as Figure 5 shown.

[0151] Step S81: Define the error function e i as the difference between the distance from the 3D circle inlier p i (x i , y i , z i ) obtained in step S7 to the circle and the circle radius r:

[0152] e i (c, r) = ||p i - c|| - r;

[0153] where ||p i - c|| is the Euclidean distance from the point p i to the center c(c x , c y , c z ):

[0154]

[0155] The goal is to minimize the sum of squared errors E for all points:

[0156]

[0157] Step S82: Initialize the center c0 and the radius r0 with the center and radius obtained in step S7. Define the number of iterations as k, then the center and radius for each iteration are c k and r k . Initialize the damping factor λ to 0.01.

[0158] Step S83: Calculate the residual vector e(c k , r k ), where each element is:

[0159] e i = ||p i - c k || - r k ;

[0160] Step S84: Calculate the Jacobian matrix J, where each row corresponds to the partial derivative of the error function of a point with respect to the parameters:

[0161]

[0162] Step S85: Calculate the update amount Δθ (where θ = [c x , c y , c z , r] T ):

[0163] (J T J + λI)Δθ = -J T e;

[0164] where I is the identity matrix.

[0165] Step S86: Update the parameters:

[0166] θ k+1 = θ k + Δθ;

[0167] Step S87: Calculate the new error E(θ k+1 ). If the error decreases, accept the update and decrease the damping factor; if the error increases, reject the update and increase the damping factor.

[0168] Step S88: Iteratively update steps S83 - S87 and stop the iteration when one of the following conditions is met: the error E(θ k)Less than a preset threshold; the parameter update amount ||Δθ|| is less than the preset threshold; the maximum number of iterations is reached.

[0169] Step S9: Identify the pose of the pair of nuts by recognizing multiple planes, including the upper plane at the end of the screw, the nut plane, and the bottom connector plane.

[0170] Specifically, first, use the RANSAC algorithm to detect all candidate planes in the point cloud, and select the three planes with the largest number of inliers as candidates. By calculating the average distance from the camera origin to each plane and combining the geometric constraint relationship of mechanical assembly (the nut plane is located between the screw end plane and the connector plane), determine the middle plane as the nut plane. Finally, apply the three-dimensional circle fitting optimization algorithm to the point cloud of this plane to accurately calculate the center position and normal vector direction of the nut, so as to obtain the complete nut pose information, as Figure 6 shown. After detecting the bolt and nut pose, return and display it in the overall point cloud, as Figure 7 shown.

[0171] Step S91: Calculate all planes in the overall point cloud of the nut by using the RANSAC plane fitting algorithm, and take the three planes with the top three number of inliers. Their inlier sets are respectively defined as N1, N2, and N3.

[0172] Step S92: Calculate the average distance d i from the origin (0, 0, 0) of the camera coordinate system to the inlier set N i of the plane:

[0173]

[0174] where N is the number of inliers in the plane, and p i is the inlier in the plane.

[0175] Step S93: According to the geometric relationship of the three planes, it can be known that the nut plane is located between the upper plane at the end of the screw and the connector plane. Sort d i from large to small, and take the second one and mark it as the nut plane.

[0176] Step S94: Perform the three-dimensional circle fitting of steps S7 - S8 on the nut plane to obtain the pose of the nut.

[0177] The present invention relates to a pose estimation method for bolt and nut fastening operations in the maintenance of catenary boom arms. Specifically, a complete bolt and nut pose detection process is proposed. Traditional manual maintenance methods have problems such as low efficiency, high risk of working at heights, and high work intensity. Therefore, the present invention adopts a method for estimating the pose of catenary bolts and nuts based on images and point clouds, aiming to assist catenary maintenance robots in achieving the maintenance task of fully automatic bolt fastening, thereby reducing the risk and work intensity of manual maintenance and constructing an intelligent and efficient catenary maintenance system.

[0178] The present invention uses the YOLOv8 deep learning object detection algorithm, which can accurately identify bolts and nuts for different types of catenary boom arms under different light conditions and shooting angles, and has high robustness. This algorithm has a fast recognition speed and a mature framework, which is convenient for subsequent upgrading and maintenance. After identifying the bolt and nut targets in the two-dimensional image, the corresponding relationship between the two-dimensional image obtained by the depth camera and the three-dimensional point cloud is combined to realize the segmentation of the bolt and nut point cloud. This method makes full use of the accuracy and rapidity of the YOLOv8 algorithm, avoids the difficulty of identifying targets in the point cloud, and effectively eliminates a large number of background invalid point clouds.

[0179] In terms of bolt and nut pose estimation, based on the structural characteristics of the bolt, the present invention innovatively adopts the RANSAC three-dimensional circle fitting algorithm to calculate the center and normal vector of the circular plane on the upper plane of the bolt, so as to accurately obtain the bolt pose. For the more complex nut pose estimation, based on the geometric relationship among the nut, screw rod and connecting piece, the distance from them to the origin is calculated by plane fitting, and then the nut plane is judged, and the RANSAC three-dimensional circle fitting algorithm is also used to calculate the nut pose.

[0180] The method proposed by the present invention combines the advantages of fast and accurate two-dimensional image object recognition and precise three-dimensional point cloud pose estimation, overcomes the defects that two-dimensional images lack depth information and cannot estimate the object pose, and three-dimensional point clouds lack color information and are difficult to identify targets. By innovatively using the three-dimensional circle fitting method to estimate the bolt and nut pose, it provides an accurate target orientation for the catenary maintenance robot, promotes the intelligent process of catenary maintenance, significantly reduces the work intensity and risk of workers, and provides strong technical support for the development of China's electrified railway.

[0181] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various equivalent changes, modifications, substitutions and variations can be made to these embodiments without departing from the principle and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalent scope.

Claims

1. A method for estimating the pose of bolts and nuts based on images and point clouds, characterized in that , The method includes: Step S1: Use the catenary maintenance robotic arm to carry a depth point cloud camera, and obtain bolt image data under different angles, lighting conditions, and backgrounds through the control variable method, and collect the catenary boom bolt and nut dataset; Step S2: Use the catenary boom bolt and nut dataset collected in Step S1, select appropriate model parameters according to the performance of the computing platform, train the YOLOv8 algorithm, and obtain the trained YOLOv8 model; Step S3: After the catenary maintenance robot equipped with a depth point cloud camera moves in front of the boom connection, take static pictures to obtain the two-dimensional grayscale image and three-dimensional point cloud image of the connection; Step S4: Locate the bolts and nuts based on the YOLOv8 deep learning object detection algorithm; Step S5: Segment the bolt and nut point cloud according to the camera internal parameter matrix; Step S6: The method of segmenting the point cloud by a rectangular range will include outlier points with a relatively large distance in the z-axis direction, which are removed by the method of point cloud statistical filtering; Step S7: Calculate the bolt pose based on the RANSAC three-dimensional circle fitting algorithm. The circle parameters in three-dimensional space are defined as: center c, normal vector n, and radius r. The three-dimensional circle is regarded as a two-dimensional circle on a plane, and the geometric constraint condition is that all points are located in the plane where the circle is located and the distance from the point to the center of the circle is less than the radius. RANSAC performs plane fitting through four steps: random sampling, model hypothesis, inlier verification, and iterative optimization; Step S8: For the inlier points within the three-dimensional circle obtained in Step S7, use the Levenberg-Marquardt algorithm to optimize the circle fitting; Step S9: Identify the poses of the nuts by recognizing multiple planes, including the upper plane of the screw end, the nut plane, and the bottom connection plane.

2. The method for estimating the poses of bolts and nuts based on images and point clouds according to claim 1, wherein The said Step S1 includes: Step S11: In the x-axis direction, collect 7 sample points at intervals of 5 cm. In the z-axis direction, collect 6 sample points at intervals of 4.5 cm. In the y-axis direction, based on the point cloud acquisition range of the depth point cloud camera, starting from 30 cm, collect 3 sample points at intervals of 10 cm; Step S12: Change the shooting angle, shooting object, and lighting conditions, and repeat Step S11 to obtain multiple catenary bolt and nut images; Step S13: Screen and sort the images collected in Step S12, remove the images with incomplete bolts, overexposed or underexposed, and construct a bolt image dataset; Step S14: Label the collected images, and use a rectangular box to select the bolts and nuts in the images.

3. The method for estimating the poses of bolts and nuts based on images and point clouds according to claim 1, characterized in that, The said Step S4 includes: Step S41: Load the trained YOLOv8 object detection model, perform object detection processing on the image, and obtain the confidence levels of each detection part; Step S42: Select the top five detection targets in terms of confidence, use the K-means clustering algorithm to divide the data into high-value and low-value, and remove the low-value data; Step S43: According to the screening in Step S42, obtain the image border coordinates of the bolts and nuts.

4. The method for estimating the poses of bolts and nuts based on images and point clouds according to claim 1, wherein, The said Step S5 includes: Step S51: The camera intrinsic parameter matrix K is used as the core parameter of the camera imaging model to define the projection transformation relationship of the three-dimensional space point from the camera coordinate system to the image pixel coordinate system. The expression is: Among them, f x , f y are the focal lengths of the x and y axes, with the unit of pixel, and c x , c y is the pixel coordinate of the optical center in the image; For a three-dimensional point P = (X, Y, Z) in the camera coordinate system, assuming that the depth direction of the camera is the z-axis, the pixel coordinates (u, v) projected to the image coordinate system are: Step S52: According to the correspondence between the three-dimensional points and the two-dimensional pixels obtained in step S51, the point cloud corresponding to the pixels in the rectangular frame of the bolt and nut in step S43 is calculated, segmented, and the invalid background point cloud information is eliminated to obtain the point cloud of the bolt and nut.

5. A method for estimating the pose of bolts and nuts based on images and point clouds according to claim 1, characterized in that, The step S6 comprises: Step S61: For each point p in the point cloud i , calculate its neighborhood N i , and use K-nearest neighbors or radius search to determine the neighborhood range of each point. K-nearest neighbors selects the K points closest to the target point as the neighborhood, and radius search determines the neighborhood by setting a fixed radius; Step S62: For each point p i , calculate its average distance d i to the neighborhood point N i : where p j is a point belonging to the neighborhood N i ; Step S63: Calculate the mean μ and standard deviation σ of the average distance of all points: Where N is the total number of points in the point cloud, and i represents the index value from 1 to N; Step S64: Set the threshold range [μ - ασ, μ + ασ], where α is for each point p i , if its average distance d i exceeds the threshold range, it is determined as an outlier and the outlier is removed.

6. The method for estimating the pose of bolts and nuts based on images and point clouds according to claim 1, characterized in that, The step S7 comprises: Step S71: 3D circle fitting requires no less than 3 non-collinear points. In each iteration, the algorithm randomly selects 3 points as initial assumptions. If the 3 points are collinear, that is, the vectors formed by the three points are linearly correlated, then the unique plane cannot be determined and resampling is required; Step S72: Calculate the three-dimensional circle parameters based on the three sampled points P1 (x1, y1, z1), P2 (x2, y2, z2), and P3 (x3, y3, z3): Determine the plane where the circle lies and calculate the plane normal vector n: n=(P2-P1)×(P3-P1); Define the point on the plane as P(x, y, z), and get the plane equation: n(P-P1)=0. Project the three sampled points to the plane coordinate system to get the two-dimensional coordinates Q1, Q2, Q3. Use algebraic methods to solve the center (a, b) and radius r of the two-dimensional circle: (Q ix -a) 2 +(Q iy -b) 2 =r 2 (i=1,2,3); Then back-project the two-dimensional center (a, b) into three-dimensional space to obtain the three-dimensional center c; Step S73: For all points P in the point cloud i , calculate the distance from it to the three-dimensional center of the circle, which consists of two parts. One is the distance from the point to the plane: d p = |n(P i - c)|; One is the distance from the center of the circle in the plane: d c = ||P i - c||- r; The distance from a point to a three-dimensional circle is defined as If the distance is less than a preset threshold, it is determined to be an inlier, and the threshold is set according to the sensor noise characteristics; Step S74: Dynamically adjust the maximum number of iterations according to the current optimal inlier ratio : k = 3 (minimum number of samples); Among them, p is the confidence level, which is used to ensure that the probability that at least one sample is all internal points reaches p; Step S75: By iterating steps S71-S74, the three-dimensional circle parameters with the largest number of inner points and the collection of inner points belonging to the three-dimensional circle are retained.

7. A method for estimating the pose of bolts and nuts based on images and point clouds according to claim 1, characterized in that, The step S8 comprises: Step S81: Define the error function e i For the three-dimensional in-circle point p i (x i , y i , z i ) obtained in step S7, the difference between the distance from the point to the circle and the circle radius r: e i (c,r) = ||p i - c|| - r; where ||p i - c|| is the Euclidean distance from point p i to the center c (c x , c y , c z ): The goal is to minimize the sum of squared errors E of all points: Step S82: Use the center and radius obtained in step S7 as the initial center c0 and initial radius r0, define the number of iterations as k, then the center and radius for each iteration are c k and r k , initialize the damping factor λ to 0.01; Step S83: Calculate the residual vector e(c k , r k ), where each element is: e i = ||p i - c k || - r k ; Step S84: Calculate the Jacobian matrix J, where each row corresponds to the partial derivative of the error function of a point with respect to the parameter: Step S85: Calculate the update amount Δθ (where θ = [c x , c y , c z , r] T ): (J T (J + λI)Δθ = -J T e; Where I is the identity matrix; Step S86: Update parameters: θ k+1 = θ k + Δθ; Step S87: Calculate the new error E(θ k+1 ). If the error decreases, accept the update and decrease the damping factor; if the error increases, reject the update and increase the damping factor; Step S88: Iteratively update steps S83 - S87, and stop the iteration when one of the following conditions is met: the error E(θ k ) is less than a preset threshold; the parameter update amount ||Δθ|| is less than a preset threshold; the maximum number of iterations is reached.

8. The method for estimating the pose of bolts and nuts based on images and point clouds according to claim 1, wherein, The step S9 comprises: Step S91: Calculate all planes in the overall point cloud of the nut by using the RANSAC plane fitting algorithm, and select the top three planes with the largest number of internal points, whose internal point sets are defined as N1, N2, and N3 respectively; Step S92: Calculate the average distance d from the origin (0, 0, 0) of the camera coordinate system to the set N of points in the plane i i :​ where N is the number of points in the plane, and p i is an interior point in the plane; Step S93: According to the geometric relationship of the three planes, the nut plane is located between the upper plane of the screw end and the connecting piece plane, and d i is sorted from large to small, and the second one is marked as the nut plane; Step S94: Perform three-dimensional circle fitting of steps S7-S8 on the nut plane to obtain the position and posture of the nut.

Images