Bullet train bolt loosening detection method based on robot adaptive pose adjustment

Abstract

The present application relates to a kind of motor train unit bolt loosening detection method based on robot adaptive pose adjustment, comprising: building data set and bolt target detection model, data set is used for model training after data enhancement;Bolt area is detected using model, and bolt point cloud is obtained;Whether the confidence of model is lower than expectation, if yes, pose adjustment is carried out, otherwise directly bolt loosening detection is carried out;Pose adjustment: according to bolt point cloud extraction base plane, and the displacement between the rotation matrix of camera and the position of bolt and origin is calculated, and the pose transformation relationship between camera and mechanical arm is combined, camera is controlled to reach target pose by mechanical arm, bolt area detection is carried out again, until the confidence meets preset expectation;Bolt loosening detection: bolt point cloud is segmented and plane fitting, bolt height is calculated, and whether bolt is loosened according to height is judged.Compared with prior art, the present application has the advantages of high detection precision, autonomous adaptation pose deviation and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of track detection, and in particular to a high-speed rail car underframe bolt loosening detection method based on robot adaptive pose adjustment. BACKGROUND

[0002] Currently, the daily maintenance of high-speed rail cars in the depot mainly adopts traditional manual detection methods, that is, naked eye recognition plus portable tool detection, and robot automatic detection is rarely used. In terms of naked eye recognition and judgment, human factors have a greater impact, and are easily affected by many factors such as the maintenance ditch environment, the working state of personnel, the technical level of personnel, and the like, and the situation of missed detection and repair may occur. In view of the obvious deficiencies in the efficiency, safety, reliability and the like of the existing manual detection and repair, the introduction of intelligent, information-based and three-dimensional repair operation methods can effectively improve the high-speed rail car preparation capacity and enhance the repair effect.

[0003] Many scholars in the early stage proposed methods for detecting bolt loosening using traditional machine vision, such as using the fixed structure of the fastener as a directional field feature as a feature descriptor, and through processing of the image sequence, the targeted acquisition of the fastener position was completed. The fastener defect recognition method based on hierarchical weighted fusion and Bayesian compressed sensing can automatically identify the defective fastener. However, this method requires an ideal background environment and does not have strong robustness. In recent years, with the continuous entry of deep learning into the public view, more and more scholars use deep convolutional neural networks for target detection. Researchers proposed using Faster R-CNN, TLMDDNet, and VGGNet to detect defective fasteners, but because of data imbalance, it is difficult to accurately identify defective fasteners, which is not conducive to improving the accuracy of fastener disease classification. In addition, some disease types such as fastener loosening require effective spatial information to correctly judge.

[0004] Since 2D images do not contain depth information and cannot measure the longitudinal displacement of the bolt loosening, and 3D point cloud feature calculation is complex, matching error is large, and target positioning is difficult, the single use of 2D images or 3D point cloud data has certain defects and cannot meet the requirements of fast and accurate bolt loosening detection. SUMMARY

[0005] The purpose of the present application is to solve the problems of depth information loss, low precision and being easily affected by image acquisition angle in traditional bolt loosening detection methods, and provide a high-speed train underframe bolt loosening detection method based on robot adaptive pose adjustment, use target detection based on deep learning to realize bolt area detection in 2D image, obtain target point cloud through pixel coordinate mapping method, analyze the characteristics of three-dimensional target point cloud, so as to realize fast and accurate bolt loosening detection, and propose a pose optimization decision algorithm based on structure information to solve the problems of occlusion, missing and angle error in data acquisition mode, which can automatically adjust its posture according to the collected data to adapt to position deviation and angle difference.

[0006] The purpose of the present application can be realized by the following technical solutions:

[0007] A high-speed train underframe bolt loosening detection method based on robot adaptive pose adjustment comprises the following steps:

[0008] Step 1) Collect data covering all bolt loosening points to be detected in multiple trips in the field where the high-speed train is located, and perform manual annotation to build a bolt target detection data set;

[0009] Step 2) Build a bolt target detection model, pre-train on a classic target detection public data set, and use data augmentation operation to expand the bolt target detection data set as the input of the pre-trained bolt target detection model, and perform transfer training on the model;

[0010] Step 3) Use the trained bolt target detection model to realize bolt area detection in 2D image, and use pixel coordinate mapping method to obtain bolt point cloud;

[0011] Step 4) Determine whether to adjust the pose according to the confidence of the bolt target detection model, if the confidence is lower than the preset expectation, perform steps 5)-7) to adjust the pose, otherwise directly perform step 8) to detect bolt loosening;

[0012] Step 5) Extract the base plane according to the bolt point cloud, calculate the base plane normal vector, calculate the camera rotation matrix that makes the camera face the bolt according to the angle between the bolt base plane normal vector and the camera coordinate system;

[0013] Step 6) Take the center point of the bolt point cloud as the position of the bolt in the camera coordinate system, and determine the displacement between the bolt position and the origin of the camera coordinate system;

[0014] Step 7) According to the camera rotation matrix, the displacement, and the pose transformation relationship between the camera and the mechanical arm, the mechanical arm is controlled to drive the camera to reach the target pose, so that the origin of the camera coordinate system coincides with the bolt position and the camera directly faces the bolt, and the bolt area detection is re-performed to generate a bolt point cloud until the confidence of the bolt target detection model meets the preset expectation.

[0015] Step 8) The bolt point cloud is segmented by region growing, the segmented point cloud is fitted by RANSAC plane, the bolt top plane and base plane are extracted by using the spatial geometric constraint of the bolt, and the distance between the two planes, i.e., the bolt height, is calculated.

[0016] Step 9) Based on the size relationship between the difference between the bolt height and the true value and the preset bolt loosening threshold, it is judged whether the bolt is loose.

[0017] The bolt target detection model adopts YOLOv6.

[0018] The data enhancement operation includes random cropping, flipping, changing saturation, hue, and contrast.

[0019] The step 5) includes the following steps:

[0020] Step 5-1) The bolt base plane is extracted according to the structural information constraint, and the plane equation is fitted by using the RANSAC plane fitting method.

[0021] Step 5-2) The normal vector of the base plane is calculated Wherein, α, β, γ are the included angles between the normal vector and the camera coordinate axes

[0022] Step 5-3) According to the included angle between the normal vector of the base plane and the camera coordinate system, the vector which is perpendicular to the camera coordinate system axis and the normal vector of the bolt base plane is taken as the rotation axis, so that the camera rotates around the rotation axis by an angle of γ to directly face the bolt, and the rotation vector is converted into a camera rotation matrix according to the Rodrigues formula.

[0023] The structural information constraint of the bolt is that the bolt head plane is parallel to the fastener plane, and there is no gap in the parallel direction, and the parallel direction is continuous, wherein the fastener plane is the base plane.

[0024] The rotation vector is converted into a camera rotation matrix according to the Rodrigues formula, specifically:

[0025]

[0026] Wherein, u x , u y ​u z They are the axes of rotation On the camera coordinate axis Components on each axis.

[0027] The pose transformation relationship between the camera and the robotic arm is obtained through hand-eye calibration.

[0028] The specific steps of controlling the robotic arm to move the camera to the target pose based on the camera rotation matrix, displacement, and pose transformation relationship between the camera and the robotic arm are as follows: Control the robotic arm according to the displacement vector between the bolt position and the origin of the camera coordinate system. Perform translational motion and pose transformation according to the camera target pose, wherein the camera target pose... This represents the pose transformation relationship between the camera and the robotic arm. This is the camera rotation matrix.

[0029] Step 8) includes the following steps:

[0030] Step 8-1) Perform preprocessing on the bolt point cloud, including downsampling and outlier removal;

[0031] Step 8-2) Segment the bolt point cloud using a region growing segmentation algorithm;

[0032] Step 8-3) Use the RANSAC algorithm to perform planar fitting on different point cloud clusters after point cloud segmentation;

[0033] Step 8-4) Based on the plane equation fitted in Step 8-3), extract the bolt top plane and base plane using the structural information constraints of the bolt;

[0034] Step 8-5) Determine the bolt height based on the distance between the bolt top plane and the base plane.

[0035] Step 8-3) includes the following steps:

[0036] Step 8-3-1) Randomly select three non-collinear points from the point cloud data and fit a plane equation:

[0037] z = ax + by + c

[0038] Step 8-3-2) Calculate the distance d from each point in the point cloud to the plane z = ax + by + c. i :

[0039]

[0040] Step 8-3-3) Calculate the distance d i Standard deviation:

[0041]

[0042] Step 8-3-4) Take the threshold t = 2σ, when d i When the value is greater than t, delete the corresponding point; otherwise, keep the corresponding point.

[0043] Step 8-3-5) Based on all retained points, return to Step 8-3-1) to refit the plane equation;

[0044] Step 8-3-6) Repeat steps 8-3-1) to 8-3-5) according to the preset number of iterations to obtain the final plane equation.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] (1) This invention integrates 2D and 3D information, uses deep learning-based target detection to detect bolt regions in 2D images, obtains target point clouds through pixel coordinate mapping, performs feature analysis on the three-dimensional target point clouds, thereby achieving fast and accurate bolt loosening detection, and solving the problems of missing depth information and low accuracy in traditional bolt loosening detection methods.

[0047] (2) This invention proposes a pose optimization decision algorithm based on structural information, which enables the robot to autonomously adjust its posture according to the collected data in order to adapt to problems such as position deviation and viewpoint difference.

[0048] (3) This invention proposes a pose adjustment triggering mechanism based on prior detection results. The data quality is judged according to the confidence of the model, thereby determining whether pose adjustment is needed. This enables the robot to autonomously judge the data acquisition quality, thereby improving detection accuracy while ensuring detection efficiency. Attached Figure Description

[0049] Figure 1 This is a flowchart of the method of the present invention;

[0050] Figure 2 A flowchart illustrating the method for making pose optimization decisions;

[0051] Figure 3 This is a schematic diagram of the method for detecting loose bolts. Detailed Implementation

[0052] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0053] This embodiment addresses the problems of traditional robot bolt loosening detection algorithms, such as lack of depth information, low accuracy, and susceptibility to data acquisition perspective. It provides a method for detecting loose bolts on the undercarriage of a high-speed train based on robot adaptive pose adjustment, including a bolt target detection algorithm, a pose optimization decision algorithm, and a bolt loosening detection algorithm. First, a bolt detection dataset is built, and a bolt target detection model is trained using the YOLOv6s deep learning algorithm. Regarding the bolt loosening detection algorithm, a bolt loosening detection algorithm based on 2D & 3D information fusion is proposed. 2D images do not contain depth information and cannot measure the longitudinal displacement of loose bolts, while 3D point cloud feature calculation is complex, has large matching errors, and makes target localization difficult. Using either 2D images or 3D point cloud data alone has certain shortcomings and cannot meet the requirements for fast and accurate bolt loosening detection. Therefore, 2D & 3D information fusion technology is introduced, using deep learning-based target detection to detect bolt regions in 2D images. The target point cloud is obtained through pixel coordinate mapping, and feature analysis is performed on the 3D target point cloud to achieve fast and accurate bolt loosening detection. Furthermore, considering that robots often encounter problems such as occlusion, missing data, and incorrect angles when collecting data on-site, this invention proposes a pose optimization decision algorithm based on structural information to address issues such as positional deviation and viewing angle differences during data acquisition. This algorithm enables the robot to autonomously adjust its posture based on the collected data to adapt to positional deviations and viewing angle differences. This invention achieves high detection accuracy while maintaining detection speed. The ability to detect loose bolts and adapt to environmental changes and robot position deviations through autonomous decision-making can improve the accuracy and efficiency of railway facility inspection, thereby enhancing the safety of railway transportation.

[0054] Specifically, such as Figure 1 As shown, it includes the following steps:

[0055] Step 1) Collect data from multiple trains at the site covering all loose bolt points to be inspected, and manually annotate the data samples using LabelImg data annotation software to generate bolt label information and build a bolt target detection dataset.

[0056] Step 2) Build a bolt target detection model YOLOv6, pre-train it on a classic public target detection dataset, expand the bolt target detection dataset with data augmentation operations and use it as input to the pre-trained bolt target detection model, and perform transfer training on the model.

[0057] To detect bolt regions in 2D images, a target detection model for bolt detection needs to be built and trained. However, there is currently no suitable publicly available bolt detection dataset. Therefore, it is necessary to utilize data collected at the EMU depot in step 1) covering all loose bolt points to be detected from multiple trips, and use this data to construct a dataset for training and testing. Simultaneously, data augmentation methods are used to expand the dataset and improve the model's robustness. Specifically, to avoid overfitting, data augmentation operations such as random cropping, flipping, and changes in saturation, hue, and contrast are performed on the images in the dataset during loading. Random flipping makes the target of interest appear in different positions in the image, reducing the model's sensitivity to the target's location and thus lessening its dependence on the target's position in the image. Randomly adjusting the saturation, hue, contrast, and brightness of the original image reduces the model's sensitivity to color. Random cropping enlarges the target in the image, helping the model learn to recognize large targets; while random image enlargement relatively shrinks the target in the image, helping the model learn to recognize small targets.

[0058] To reduce the workload of data collection, this embodiment first pre-trains the model on classic public object detection datasets such as VOC and MS-COCO, and then trains the pre-trained model on the collected dataset.

[0059] Step 3) Use the trained bolt target detection model to detect bolt regions in 2D images and obtain bolt point clouds using pixel coordinate mapping.

[0060] Step 4) Determine whether to adjust the posture based on the confidence level of the bolt target detection model. If the confidence level is lower than the preset expectation, that is, the target detection model is not very confident in identifying the target as a bolt, then the data collected by the robot is considered to be of poor quality. Then, proceed to steps 5)-7) to adjust the posture to obtain high-quality data. Otherwise, proceed directly to step 8) to detect bolt loosening.

[0061] The process of attitude optimization decision-making is as follows: Figure 2 As shown, this includes the following steps 5)-7):

[0062] Step 5) Extract the base plane from the bolt point cloud, calculate the base plane normal vector, and calculate the camera rotation matrix that makes the camera face the bolt based on the angle between the bolt base plane normal vector and the camera coordinate system.

[0063] Step 5-1) Extract the bolt base plane based on the bolt's structural information constraints, and fit the plane equation using the RANSAC plane fitting method. The bolt's structural information constraints are that the bolt head plane is parallel to the fastener plane (i.e., the base plane), and the two planes have no gap in the parallel direction and are adjacent and continuous in the parallel direction.

[0064] Step 5-2) Calculate the base plane normal vector Where α, β, and γ are the normal vector and the camera coordinate axis, respectively. The included angle of the three axes.

[0065] Step 5-3) To ensure the camera is directly facing the bolt for data acquisition, the camera needs to be rotated so that its z-axis aligns with the normal vector. Same direction, bolt base plane normal vector Due to the camera coordinate axis The cosines of the angles between the three axes form the basis of the relationship between the camera's z-axis and the normal vector. The included angle is γ, which is taken in the camera coordinate system. The normal vectors of the shaft and the bolt base plane are both perpendicular. As a rotation axis, let the camera rotate around the rotation axis By rotating by angle γ, the camera can be positioned directly opposite the bolt. Using the Roglic equation, the rotation vector can be transformed into a camera rotation matrix.

[0066]

[0067] Among them, u x u y u z They are the axes of rotation On the camera coordinate axis Components on each axis.

[0068] Step 6) Take the center point of the bolt point cloud as the position p of the bolt in the camera coordinate system. c =(x c ,y c ,z c And determine the displacement vector between the bolt position and the origin of the camera coordinate system.

[0069] Step 7) Based on the camera rotation matrix, displacement, and pose transformation relationship between the camera and the robotic arm, control the robotic arm to drive the camera to the target pose, so that the origin of the camera coordinate system coincides with the bolt position and the camera is facing the bolt, and return to step 3) to re-perform bolt region detection, generate bolt point cloud, until the confidence of the bolt target detection model meets the preset expectation.

[0070] First, control the robotic arm according to the displacement vector between the bolt position and the origin of the camera coordinate system. Perform translational motion. Secondly, the rotation matrix... and displacement vector Both are relative to the camera coordinate system. In order for the robotic arm to control the camera to move to the target position, the pose transformation needs to be converted to the robot coordinate system. The pose relationship between the camera and the robotic arm end effector is obtained through hand-eye calibration. In the robot coordinate system, the camera target pose is...

[0071] Steps 8)-9) are the bolt loosening detection algorithm, such as... Figure 3 As shown, specifically:

[0072] Step 8) Perform region growth segmentation on the bolt point cloud, perform RANSAC plane fitting on the segmented point cloud, extract the bolt top plane and base plane using the spatial geometric constraints of the bolt, and calculate the distance between the two planes, i.e. the bolt height.

[0073] Step 8-1) Perform preprocessing on the bolt point cloud, such as downsampling and removing outliers;

[0074] Step 8-2) Segment the bolt point cloud using a region growing segmentation algorithm;

[0075] Step 8-3) Use the RANSAC algorithm to perform planar fitting on different point cloud clusters after point cloud segmentation;

[0076] Step 8-3-1) Randomly select three non-collinear points from the point cloud data and fit a plane equation:

[0077] z = ax + by + c

[0078] Step 8-3-2) Calculate the distance d from each point in the point cloud to the plane z = ax + by + c. i :

[0079]

[0080] Step 8-3-3) Calculate the distance d i Standard deviation:

[0081]

[0082] Step 8-3-4) Take the threshold t = 2σ, when d i When the value is greater than t, delete the corresponding point; otherwise, keep the corresponding point.

[0083] Step 8-3-5) Based on all retained points, return to Step 8-3-1) to refit the plane equation;

[0084] Step 8-3-6) Repeat steps 8-3-1) to 8-3-5) n times to obtain the final plane equation.

[0085] Step 8-4) Based on the plane equation fitted in Step 8-3), extract the bolt top plane and base plane using the structural information constraints of the bolt;

[0086] Step 8-5) Determine the bolt height based on the distance between the bolt top plane and the base plane.

[0087] Step 9) If the difference between the bolt height and the pre-stored true value is greater than the preset bolt loosening threshold, then the bolt is determined to have a loosening fault; otherwise, no loosening fault has occurred.

[0088] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for detecting loose bolts on the undercarriage of a high-speed train based on robot adaptive pose adjustment, characterized in that, Includes the following steps: Step 1) Collect data from multiple trains covering all loose bolt points to be inspected at the train site, and manually label the data to build a bolt target detection dataset; Step 2) Build a bolt target detection model, pre-train it on a classic public target detection dataset, expand the bolt target detection dataset using data augmentation operations and use it as input to the pre-trained bolt target detection model, and perform transfer training on the model; Step 3) Use the trained bolt target detection model to detect bolt regions in 2D images and obtain bolt point clouds using pixel coordinate mapping. Step 4) Determine whether to adjust the attitude based on the confidence level of the bolt target detection model. If the confidence level is lower than the preset expectation, proceed to steps 5)-7) to adjust the attitude; otherwise, proceed directly to step 8) to detect bolt loosening. Step 5) Extract the base plane from the bolt point cloud, calculate the base plane normal vector, and calculate the camera rotation matrix that makes the camera face the bolt based on the angle between the bolt base plane normal vector and the camera coordinate system. Step 5) includes the following steps: Step 5-1) Extract the bolt base plane based on the bolt's structural information constraints, and fit the plane equation using the RANSAC plane fitting method; Step 5-2) Calculate the base plane normal vector ,in, , , Normal vector and camera coordinate axis respectively , , The included angle of the three axes; Step 5-3) Based on the angle between the base plane normal vector and the camera coordinate system, select the coordinate system... A vector whose normal vectors to both the axis and the bolt base plane are perpendicular. As a rotation axis, the camera rotates around the rotation axis Rotation The angle can be aligned with the bolt, and the rotation vector can be transformed into the camera rotation matrix according to the Rogrids formula; Step 6) Take the center point of the bolt point cloud as the position of the bolt in the camera coordinate system, and determine the displacement between the bolt position and the origin of the camera coordinate system; Step 7) Based on the camera rotation matrix, displacement, and pose transformation relationship between the camera and the robotic arm, control the robotic arm to drive the camera to the target pose, so that the origin of the camera coordinate system coincides with the bolt position and the camera is facing the bolt, and return to step 3) to re-perform bolt region detection, generate bolt point cloud, until the confidence of the bolt target detection model meets the preset expectation. Step 8) Perform region growth segmentation on the bolt point cloud, perform RANSAC plane fitting on the segmented point cloud, extract the bolt top plane and base plane using the spatial geometric constraints of the bolt, and calculate the distance between the two planes, i.e. the bolt height; Step 9) Determine whether the bolt is loose based on the relationship between the difference between the bolt height and the true value and the preset bolt loosening threshold.

2. The method for detecting loose bolts on the undercarriage of a high-speed train based on robot adaptive pose adjustment according to claim 1, characterized in that, The bolt target detection model uses YOLOv6.

3. The method for detecting loose bolts on the undercarriage of a high-speed train based on robot adaptive pose adjustment according to claim 1, characterized in that, The data augmentation operations include random cropping, flipping, and changing saturation, hue, and contrast.

4. The method for detecting loose bolts on the undercarriage of a high-speed train based on robot adaptive pose adjustment according to claim 1, characterized in that, The structural information constraint of the bolt is that the bolt head plane is parallel to the fastener plane, and the two planes are continuous in the parallel direction without any gap. The fastener plane is the base plane.

5. The method for detecting loose bolts on the undercarriage of a high-speed train based on robot adaptive pose adjustment according to claim 1, characterized in that, The specific steps for converting the rotation vector into a camera rotation matrix according to the Rogrids formula are as follows: in, They are the axes of rotation On the camera coordinate axis , , Components on each axis.

6. The method for detecting loose bolts on the undercarriage of a high-speed train based on robot adaptive pose adjustment according to claim 1, characterized in that, The pose transformation relationship between the camera and the robotic arm is obtained through hand-eye calibration.

7. The method for detecting loose bolts on the undercarriage of a high-speed train based on robot adaptive pose adjustment according to claim 1, characterized in that, The specific steps of controlling the robotic arm to move the camera to the target pose based on the camera rotation matrix, displacement, and pose transformation relationship between the camera and the robotic arm are as follows: Control the robotic arm according to the displacement vector between the bolt position and the origin of the camera coordinate system. Perform translational motion and pose transformation according to the camera target pose, wherein the camera target pose... , This represents the pose transformation relationship between the camera and the robotic arm. This is the camera rotation matrix.

8. The method for detecting loose bolts on the undercarriage of a high-speed train based on robot adaptive pose adjustment according to claim 1, characterized in that, Step 8) includes the following steps: Step 8-1) Preprocess the bolt point cloud by downsampling and removing outliers; Step 8-2) Segment the bolt point cloud using a region growing segmentation algorithm; Step 8-3) Use the RANSAC algorithm to perform planar fitting on different point cloud clusters after point cloud segmentation; Step 8-4) Based on the plane equation fitted in Step 8-3), extract the bolt top plane and base plane using the structural information constraints of the bolt; Step 8-5) Determine the bolt height based on the distance between the bolt top plane and the base plane.

9. The method for detecting loose bolts on the undercarriage of a high-speed train based on robot adaptive pose adjustment according to claim 8, characterized in that, Step 8-3) includes the following steps: Step 8-3-1) Randomly select three non-collinear points from the point cloud data and fit a plane equation: Step 8-3-2) Calculate the distance from each point in the point cloud to the plane. distance : Step 8-3-3) Calculate the distance Standard deviation: Step 8-3-4) Take the threshold ,when If the condition is met, delete the corresponding point; otherwise, keep the corresponding point. Step 8-3-5) Based on all retained points, return to Step 8-3-1) to refit the plane equation; Step 8-3-6) Repeat steps 8-3-1) to 8-3-5 according to the preset number of iterations to obtain the final plane equation.

Images