Rail bolt inspection system and method

Abstract

The application discloses a rail bolt maintenance system and method. The system comprises a vehicle frame, a line laser scanning device, a mechanical arm robot, a bolt locking device, a visual detection system and a machine control system. The system is based on a 3D visual system and a line laser scanning technology, and realizes intelligent inspection and maintenance of rail bolts through a laser scanning device, a mechanical arm robot, a bolt locking device and a control system carried on a body of a patrol device. The system can monitor the state of the bolts in real time, quickly identify problems such as loosening and wear, and accurately feedback after the method is used, thereby reducing manual intervention. Meanwhile, the automatic system improves the maintenance efficiency, reduces human errors, shortens the maintenance cycle, and greatly improves the safety and operation efficiency of the rail transit system.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of industrial automation and intelligent manufacturing, and particularly relates to a rail bolt maintenance system and method. BACKGROUND

[0002] The current railway maintenance in the field of rail transit is facing many challenges. With the rapid expansion of the railway network and the significant increase in train operation frequency, it is crucial to ensure the safety and reliability of railway infrastructure. Traditional maintenance methods mostly rely on manual operation, where workers use hand tools to regularly maintain the track and its components. However, this manual maintenance mode has problems such as heavy workload, low efficiency, inaccurate detection, and high labor intensity. Especially in the maintenance of key components such as bolts, human negligence or fatigue can easily lead to hidden dangers not being discovered in time. Bolts, as important fasteners in the rail transit system, may directly affect railway safety if they are loose or damaged, requiring high-frequency and high-precision maintenance.

[0003] Therefore, there is an urgent need for an automated maintenance technology that can monitor the state of bolts in real time through intelligent devices, quickly identify problems such as loosening and wear, and provide accurate feedback. SUMMARY

[0004] In view of the deficiencies of the prior art, one of the purposes of the present application is to provide a rail bolt maintenance system that improves maintenance efficiency, reduces human error, shortens the maintenance cycle, and significantly improves the safety and operational efficiency of the rail transit system, to meet the urgent needs of railway maintenance automation and intelligentization in the new era.

[0005] The second purpose of the present application is to provide a method for using the system.

[0006] The present application provides a rail bolt maintenance system, which includes a vehicle frame, a line laser scanning device, a mechanical arm robot, a bolt locking device, a visual detection system, and a machine control system.

[0007] The vehicle frame is equipped with a vehicle body power device and a vehicle body damping device, and two sets of line laser equipment mounting buckles are reserved at the front of the vehicle frame, and two mechanical arm robot mounting buckles are reserved at the rear of the vehicle frame, to correspond to the detection and maintenance of the two tracks.

[0008] The line laser scanning device is suspended at the front of the vehicle frame and is used to scan the state of the track and its components downward, with one line laser camera device arranged at each end to scan the bolts of the two tracks.

[0009] The mechanical arm robot is installed at the rear of the vehicle frame and is used for automatic positioning and moving to the position of the loose bolt, with one mechanical arm robot arranged at each end to correspond to the bolts of the two tracks.

[0010] The bolt locking device is located at the end of the mechanical arm and is responsible for providing a rotational torque to tighten or loosen the bolt; the mechanical arm transmission mechanism transmits the power of the driving mechanism to the end effector, which grasps the bolt and applies a rotational force; in order to prevent over-tightening or under-tightening, the bolt locking device has safety features such as torque limitation, angle limitation and fault alarm;

[0011] The visual detection system is used to receive data scanned by the line laser scanning device, and the 3D visual system processes and calculates the data after imaging to determine abnormal bolts and calculate their coordinate information, which is transmitted to the machine control system;

[0012] The machine control system is used to receive the abnormal bolt command from the visual detection system and instruct the mechanical arm robot to complete the precise bolt tightening or loosening task.

[0013] The machine control system is also designed with a self-diagnosis and self-repair mechanism; the self-diagnosis and self-repair mechanism includes a self-diagnosis module and a self-repair module;

[0014] The self-diagnosis module monitors the change of the mechanical arm torque in real time through the block torque anomaly detection, accurately identifies the abnormal state of the mechanical arm, and the abnormal state determination conditions are as follows:

[0015] Or

[0016] Wherein, is the theoretical torque based on the bolt diameter d, material yield strength , friction coefficient ; is the real-time detection of the mechanical arm torque; is the continuous overrun time;

[0017] The self-repair module automatically executes the tool withdrawal operation according to the detection result of the self-diagnosis module through the reverse tool withdrawal strategy, avoiding manual tool correction, and the angle of the reverse tool withdrawal strategy The calculation formula is as follows:

[0018]

[0019] Wherein, is the deviation between the actual torque and the theoretical threshold value; is the thread stiffness coefficient; G is the shear modulus of the bolt, L is the thread contact length, and d is the bolt diameter.

[0020] The application also provides a use method of the rail bolt maintenance system, which comprises the following steps:

[0021] S1. Scanning the rail bolt by the line laser scanning device to obtain line laser point cloud data;

[0022] S2. Constructing a bolt detection model based on a YOLOv11 network structure; inputting the scanning data into the bolt detection model to obtain a detection result;

[0023] S3. Judging the obtained detection result, if the bolt state is missing, sending an alarm information immediately; if the bolt state is loose, entering step S4; if the bolt state is perfect and fastened, completing the detection of the rail bolt;

[0024] S4. Based on the bolt array geometric constraint and the mechanical arm pose synchronous analysis, constructing a coordinate system dynamic matching mechanism to position the bolt and obtain the robot base bolt space coordinates;

[0025] S5. According to the obtained robot base bolt space coordinates, driving the mechanical arm robot to tighten the bolt through reinforcement learning dynamic scheduling and dynamic path planning, completing the detection and tightening of the rail bolt.

[0026] In step S2, the bolt detection model is based on the YOLOv11 network structure and has the following improvements:

[0027] The number of input layer channels is expanded from 3 to 7, the newly added channels 4 to 6 are used to input the normal vector of the line laser point cloud, and channel 7 is used to input the bolt surface curvature feature map;

[0028] The fixed rectangular sampling network is improved to use deformable convolution to guide the sampling point offset through the curvature feature map;

[0029] The curvature condition branch is introduced into the standard FReLU activation function;

[0030] A cross-scale attention module is added to the Neck layer to optimize the multi-scale feature fusion effect through channel-spatial dual domain weight distribution;

[0031] EloU Loss is used instead of CloU Loss as the loss function of the model, and the bolt bounding box regression accuracy is improved by introducing a width-height coverage penalty term;

[0032] The bolt detection result output by the bolt detection model includes bounding box coordinates, confidence and bolt state.

[0033] The improvement of the input layer makes the bolt detection model fully utilize the complementarity of RGB color information, laser geometric information and curvature shape information, realize the optimal feature combination under different environmental conditions, and greatly improve the accuracy and stability of bolt detection.

[0034] The deformable convolution guides the sampling point offset through the curvature feature map, increases the sampling density for high curvature areas (threads, edges) to capture geometric details, reduces the sampling density for low curvature areas (smooth surface) to improve the calculation efficiency, and adjusts the sampling mode adaptively for areas with sharp curvature changes to accurately capture geometric mutations.

[0035] The input feature tensor of the input layer is expressed using the following formula:

[0036]

[0037] wherein, is the input feature tensor; is the red component information in the RGB color information; is the green component information in the RGB color information; is the blue component information in the RGB color information; is the x-direction normal vector of the line laser point cloud; is the x-direction normal vector of the line laser point cloud; is the x-direction normal vector of the line laser point cloud; is the bolt surface curvature feature map data;

[0038] The bolt surface curvature feature map is obtained using the following steps:

[0039] The line laser point cloud data is preprocessed; the preprocessing includes outlier removal and voxel filtering;

[0040] The surface normal vector of each point is estimated based on the k-neighborhood using principal component analysis, and the principal component analysis is expressed using the following formula:

[0041]

[0042]

[0043] wherein, C is the covariance matrix, reflecting the distribution characteristics of the neighborhood points in the three-dimensional space; is the i-th point in the current processed 3D point cloud; is the mean vector of the neighborhood point coordinates; V is the eigenvector matrix of the covariance matrix C, and the column vector is the characteristic direction; is the diagonal matrix of the covariance matrix C;

[0044] For each point, a local coordinate system with the point as the origin and the normal vector as the z-axis is established, a local quadratic surface equation is fitted by using the least squares method, and two principal curvatures, average curvature and Gaussian curvature are obtained by solving the eigenvalues of the Hessian matrix;

[0045] Finally, the curvature information of the 3D point cloud is projected to the 2D image plane, and a curvature feature map containing the principal curvature features is generated by using a Gaussian kernel weighted distribution;

[0046] The local quadric surface equation is expressed by the following formula:

[0047]

[0048] wherein, is the local coordinate of the point cloud; is the first quadric surface coefficient; b is the second quadric surface coefficient; c is the third quadric surface coefficient; the quadric surface coefficients determine the bending degree of the surface;

[0049] The Hessian matrix is expressed by the following formula:

[0050]

[0051] The Gaussian kernel weighted distribution is expressed by the following formula:

[0052]

[0053] wherein, is a two-dimensional Gaussian kernel function used for smoothing the projection of the curvature value; is a Gaussian kernel standard deviation.

[0054] The feature map output by the deformable convolution is expressed by the following formula:

[0055]

[0056] wherein, is the feature map output by the deformable convolution; is a weight parameter; is the output feature map position; is a fixed offset at the kth position; is obtained by prediction through an offset network, and K=9 is the number of convolution kernels.

[0057] The extended FRelU activation function is expressed by the following formula:

[0058]

[0059] wherein, is an input feature map or tensor, is a standard spatial condition term, is a newly added curvature condition term; the two condition terms are fused through a weight to be learned, so that the activation function considers both the spatial position information and the geometric curvature information.

[0060] The operation of the cross-scale attention module is represented using the following equation:

[0061]

[0062] wherein, is the joint feature tensor after multi-scale feature concatenation, is the channel domain weight matrix, is the spatial domain weight matrix; is the output of the cross-scale attention module.

[0063] The EloU Loss loss function with the introduction of the width and height coverage penalty term is represented using the following equation:

[0064]

[0065] wherein, is the Euclidean distance; is the predicted box center coordinate; is the predicted box width; is the predicted box width and height; is the minimum bounding box width; is the minimum bounding box height; is the coverage of the predicted width and the ground truth box; is the coverage of the predicted height and the ground truth box; IoU is the intersection over union; is the true box width and height; is the true box width and height; is the true box center coordinate.

[0066] Step S4 includes the following steps:

[0067] Based on the bounding box coordinates output by the model, the pixel coordinates of the bolt center in the image coordinate system are calculated, which is represented using the following equation:

[0068]

[0069] wherein, represents the pixel position of the bolt center, is the top-left corner coordinate of the bounding box, is the bottom-right corner coordinate of the bounding box;

[0070] Subsequently, based on the collected line laser point cloud data, the bolt center is taken as the reference, and the depth value weighted fusion calculation is performed within a 5x5 pixel neighborhood, which is represented using the following equation:

[0071]

[0072] wherein, represents the depth value (unit: mm) at coordinate in the point cloud; is the depth value of the bolt center point in the camera coordinate system; i is the horizontal offset; j is the vertical offset; weighting coefficient Using standard deviation Gaussian function calculation:

[0073] ;

[0074] Using a pre-calibrated camera intrinsic parameter matrix The two-dimensional pixel coordinates are back-projected into the camera's three-dimensional space, and expressed using the following formula:

[0075]

[0076] in, The three-dimensional coordinates of the bolt in the camera coordinate system; camera intrinsic parameter matrix. , Represents the camera's focal length (in pixels). The coordinates of the camera's optical center;

[0077] The camera coordinate system coordinates are transformed to the robot arm base coordinate system using a rigid body transformation model, expressed by the following formula:

[0078]

[0079] in, For the transformation matrix, the internal rotation submatrix Describes the attitude relationship between coordinate systems; translation vector Characterizes spatial offset; Let be the spatial coordinates of the bolt in the robot's base coordinate system;

[0080] To address the coordinate drift problem caused by vibration in robotic arms, a vibration compensation mechanism is designed. Vibration intensity is analyzed by examining the jitter in bolt images across consecutive frames, and a vibration compensation vector is generated in real time. It can be expressed using the following formula:

[0081]

[0082] in, The vibration transmission coefficient is 0.15 (mm / pixel); N is the sampling window size. for Frame horizontal displacement; for Frame vertical displacement; The principal direction angle of vibration;

[0083] Aiming at the problem of coordinate drift caused by thermal deformation of the robot arm, a thermal deformation compensation mechanism is designed to utilize the geometric invariance of the bolt array, calculate the thermal expansion effect through the distance change between adjacent bolts, and generate a thermal deformation compensation vector , which is expressed by the following formula:

[0084]

[0085] wherein, is the expansion coefficient of steel; is the image center distance (pixel unit) of the same bolt pair in the current frame; is the image center distance of adjacent bolts recorded at the first detection; is the calibration reference temperature 25℃; is the temperature axial compensation factor;

[0086] Finally, the accurate spatial position coordinates of the bolt in the robot base coordinate system after dynamic compensation of the physical environment are obtained, which are expressed by the following formula:

[0087]

[0088] wherein, is the accurate spatial position coordinates of the bolt in the robot base coordinate system.

[0089] In step S5, the dynamic scheduling through reinforcement learning is specifically: designing an intelligent scheduling reward function to balance the work efficiency, energy consumption and safety, and driving the robot arm to select the optimal path, and the intelligent scheduling reward function R is expressed by the following formula:

[0090]

[0091] wherein, is the efficiency reward term; is the energy consumption penalty term; is the safety reward term; is the number of unfinished maintenance tasks; is the total number of maintenance tasks; is the energy consumption coefficient of the jth joint of the robot arm; is the angular acceleration of the jth joint of the robot arm; is the time window length; is the real-time distance between the end of the robot arm and the obstacle; is the observation distance; is the safety operation distance threshold; is the first coefficient of the intelligent scheduling reward function; is the second coefficient of the intelligent scheduling reward function; is the third coefficient of the intelligent scheduling reward function.

[0092] The dynamic path planning is expressed using the following formula:

[0093]

[0094] wherein, is a path cost function; represents a distance between the mechanical arm and the ith obstacle, represents the jth joint angle acceleration; is a first coefficient of the dynamic path planning; is a second coefficient of the dynamic path planning.

[0095] The application discloses a rail bolt maintenance system and method, which combines a 3D vision system, a line laser scanning technology and a deep learning theory, realizes high-precision detection and three-dimensional positioning of a rail bolt, and provides accurate data support for automatic inspection. BRIEF DESCRIPTION OF DRAWINGS

[0096] Figure 1 is a method flowchart of the application;

[0097] Figure 2 is a main body structure schematic diagram of the system in the embodiment;

[0098] Figure 3 is a mechanical arm robot structure schematic diagram of the application in the embodiment;

[0099] Figure 4 is a bolt locking device schematic diagram of the application in the embodiment;

[0100] In the figure: 1, line laser scanning device; 2, mechanical arm robot; 3, bolt locking device; 4, vehicle frame; 5, seat; 6, protective cover; 7, robot control cabinet; 8, rail; 9, target bolt. DETAILED DESCRIPTION

[0101] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. The application embodiments realize automatic detection and maintenance management of abnormal rail fastener bolts through a rail bolt automatic detection device based on a 3D vision system.

[0102] The application provides a rail bolt maintenance system, which comprises a vehicle frame, a line laser scanning device, a mechanical arm robot, a bolt locking device, a visual detection system and a machine control system.

[0103] The vehicle frame carries the vehicle body power device and the vehicle body shock absorption device, and the front part of the vehicle frame is provided with two sets of installation buckles for line laser equipment, and the rear part of the vehicle frame is provided with two installation buckles for mechanical arm robots, so as to detect and maintain two tracks;

[0104] The line laser scanning device is hung at the front part of the vehicle frame and is used for scanning the tracks and the states of parts of the tracks downward, and one line laser camera device is arranged at each of left and right ends of the line laser scanning device, so as to scan the bolts of the two tracks;

[0105] The mechanical arm robot is installed at the rear part of the vehicle frame and is used for automatically positioning and moving to a position where a loose bolt is located, and one mechanical arm robot is arranged at each of left and right ends of the mechanical arm robot, so as to correspond to the bolts of the two tracks;

[0106] The bolt locking device is located at the end of the mechanical arm and is responsible for providing a rotary torque to tighten or loosen the bolt; the mechanical arm transmission machine transmits the power of the driving machine to the end effector, the end effector grasps the bolt and applies a rotary force; in order to prevent over-tightening or under-tightening, the bolt locking device has safety features such as torque limitation, angle limitation and fault alarm;

[0107] The visual detection system is used for receiving data scanned by the line laser scanning device, and the 3D visual system processes and calculates the data after imaging, judges abnormal bolts and calculates coordinate information of the abnormal bolts, and transmits the abnormal bolt instructions to the machine control system;

[0108] The machine control system is used for receiving the abnormal bolt instructions transmitted by the visual detection system and commanding the mechanical arm robot to complete the accurate bolt tightening or loosening task.

[0109] The machine control system is also designed with a self-diagnosis and self-repair mechanism; the self-diagnosis and self-repair mechanism includes a self-diagnosis module and a self-repair module;

[0110] The self-diagnosis module monitors the torque change of the mechanical arm in real time through the block torque anomaly detection, accurately identifies the abnormal state of the mechanical arm, and the abnormal state determination condition is as follows:

[0111] Or

[0112] Among them, is the theoretical torque based on the bolt diameter d, the material yield strength and the friction coefficient ; is the real-time detection of the torque of the mechanical arm; is the continuous over-limit time;

[0113] The self-repair module automatically executes the tool withdrawal operation according to the detection result of the self-diagnosis module through the reverse tool withdrawal strategy, avoids manual tool correction, and the angle of the reverse tool withdrawal strategy is The calculation formula is as follows:

[0114]

[0115] wherein, is the deviation amount of the actual torque from the theoretical threshold value; is the thread stiffness coefficient; G is the shear modulus of the bolt, L is the thread contact length, and d is the bolt diameter.

[0116] The application also provides a use method of the rail bolt maintenance system, a flowchart of which is shown in the figure Figure 1 and includes the following steps:

[0117] S1. Scanning the rail bolt by means of a line laser scanning device to obtain line laser point cloud data;

[0118] S2. Constructing a bolt detection model based on a YOLOv11 network structure; inputting the scanning data into the bolt detection model to obtain a detection result;

[0119] In step S2, the bolt detection model is based on the YOLOv11 network structure and has the following improvements:

[0120] The number of channels of the input layer is expanded from 3 to 7, the newly added channels 4 to 6 are used to input the normal vector of the line laser point cloud, and channel 7 is used to input the bolt surface curvature feature map;

[0121] The fixed rectangular sampling network is improved to use deformable convolution to guide the sampling point offset through the curvature feature map;

[0122] The curvature condition branch is introduced into the standard FReLU activation function;

[0123] A cross-scale attention module is added to the Neck layer to optimize the multi-scale feature fusion effect through channel-spatial dual domain weight distribution;

[0124] EloU Loss is used to replace CloU Loss as the loss function of the model, and the bolt bounding box regression accuracy is improved by introducing a width-height coverage penalty term;

[0125] The bolt detection result output by the bolt detection model includes the bounding box coordinates, the confidence and the bolt state.

[0126] The improvement of the input layer makes the bolt detection model fully utilize the complementarity of RGB color information, laser geometric information and curvature shape information, realize the optimal feature combination under different environmental conditions, and greatly improve the accuracy and stability of bolt detection.

[0127] The deformable convolution guides the sampling point offset through the curvature feature map, increases the sampling density for high curvature areas (threads, edges) to capture geometric details, reduces the sampling density for low curvature areas (smooth surface) to improve the calculation efficiency, and adjusts the sampling mode adaptively for areas with sharp curvature changes to accurately capture geometric mutations.

[0128] The input feature tensor of the input layer is expressed using the following formula:

[0129]

[0130] wherein, is the input feature tensor; is the red component information in the RGB color information; is the green component information in the RGB color information; is the blue component information in the RGB color information; is the x-direction normal vector of the line laser point cloud; is the x-direction normal vector of the line laser point cloud; is the x-direction normal vector of the line laser point cloud; is the bolt surface curvature feature map data;

[0131] The bolt surface curvature feature map is obtained using the following steps:

[0132] The line laser point cloud data is preprocessed; the preprocessing includes outlier removal and voxel filtering;

[0133] The surface normal vector of each point is estimated based on the k-neighborhood using principal component analysis, and the principal component analysis is expressed using the following formula:

[0134]

[0135]

[0136] wherein, C is the covariance matrix, reflecting the distribution characteristics of the neighborhood points in the three-dimensional space; is the i-th point in the current processed 3D point cloud; is the mean vector of the neighborhood point coordinates; V is the eigenvector matrix of the covariance matrix C, and the column vector is the characteristic direction; is the diagonal matrix of the covariance matrix C;

[0137] For each point, a local coordinate system is established with the point as the origin and the normal vector as the z-axis, a local quadratic surface equation is fitted by using the least squares method, and two principal curvatures, average curvature and Gaussian curvature are obtained by solving the eigenvalues of the Hessian matrix;

[0138] Finally, the curvature information of the 3D point cloud is projected to the 2D image plane, and a curvature feature map containing the principal curvature features is generated by using a Gaussian kernel weighted distribution;

[0139] The local quadric surface equation is expressed by the following formula:

[0140]

[0141] wherein, is the point cloud local coordinate system coordinate; is the first quadratic surface coefficient; b is the second quadratic surface coefficient; c is the third quadratic surface coefficient; the quadratic surface coefficient determines the bending degree of the surface;

[0142] The Hessian matrix is expressed by the following formula:

[0143]

[0144] The Gaussian kernel weighted distribution is expressed by the following formula:

[0145]

[0146] wherein, is a two-dimensional Gaussian kernel function used for smoothing the projection of the curvature value; is the Gaussian kernel standard deviation.

[0147] The feature map output by the deformable convolution is expressed by the following formula:

[0148]

[0149] wherein, is the feature map output by the deformable convolution; is a weight parameter; is the output feature map position; is a fixed offset at the kth position; is obtained by prediction through the offset network, and K=9 is the number of convolution kernels.

[0150] The extended FRelU activation function is expressed by the following formula:

[0151]

[0152] wherein, is an input feature map or tensor, is a standard space condition term, is a newly added curvature condition term; the two condition terms are fused through a to-be-learned weight, so that the activation function considers both the spatial position information and the geometric curvature information.

[0153] The operation of the cross-scale attention module is represented using the following equation:

[0154]

[0155] wherein, is the joint feature tensor after multi-scale feature splicing, is the channel domain weight matrix, is the spatial domain weight matrix; is the output of the cross-scale attention module.

[0156] The EloU Loss loss function introducing the width and height coverage penalty term is represented using the following equation:

[0157]

[0158] wherein, is the Euclidean distance; is the predicted box center coordinate; is the predicted box width; is the predicted box width and height; is the minimum bounding box width; is the minimum bounding box height; is the coverage of the predicted width and the true value box; is the coverage of the predicted height and the true value box; IoU is the intersection over union; is the true box width and height; is the true box width and height; is the true box center coordinate.

[0159] S3. The obtained detection result is judged, if the bolt state is lost, an alarm information is immediately sent, if the bolt state is loose, step S4 is entered, if the bolt state is perfect and fastened, the detection of the rail bolt is completed;

[0160] S4. Based on the bolt array geometric constraint and the mechanical arm pose synchronous analysis, a coordinate system dynamic matching mechanism is constructed to position the bolt and obtain the robot base bolt space coordinates;

[0161] Step S4 includes the following steps:

[0162] Based on the boundary box coordinates output by the model, the pixel coordinates of the bolt center in the image coordinate system are calculated, which is represented using the following equation:

[0163]

[0164] wherein, represents the pixel position of the bolt center, is the top-left corner coordinate of the bounding box, is the right-bottom corner coordinate of the bounding box;

[0165] Subsequently, based on the collected line laser point cloud data, the depth value weighted fusion calculation is performed in the 5x5 pixel neighborhood with the bolt center as the reference, which is expressed by the following formula:

[0166]

[0167] wherein, represents the depth value (unit: mm) of the coordinates in the point cloud; is the depth value of the bolt center point in the camera coordinate system; i is the horizontal offset; j is the vertical offset; the weight coefficient is calculated by the Gaussian function of the standard deviation

[0168]

[0169] The two-dimensional pixel coordinates are back-projected to the camera three-dimensional space by using the pre-calibrated camera intrinsic matrix , which is expressed by the following formula:

[0170]

[0171] wherein, is the three-dimensional coordinates of the bolt in the camera coordinate system; the camera intrinsic matrix , represents the camera focal length (pixel unit), is the camera optical center coordinates;

[0172] The camera coordinate system coordinates are converted to the robot base coordinate system by the rigid body transformation model, which is expressed by the following formula:

[0173]

[0174] wherein, is the transformation matrix, the rotation sub-matrix in the interior describes the attitude relationship between the coordinate systems; the translation vector represents the spatial offset; is the spatial coordinates of the bolt in the robot base coordinate system;

[0175] A vibration compensation mechanism is designed to solve the problem of coordinate drift caused by the vibration of the robot arm; the vibration intensity is analyzed by the jitter of the bolt image position between consecutive frames, and the vibration compensation vector is generated in real time, which is expressed by the following formula:

[0176]

[0177] wherein, ​​​is the vibration transmission coefficient, taking 0.15 (mm / pix) as the value; N is the sampling window size; is frame horizontal displacement; is frame vertical displacement; is the vibration main direction angle;

[0178] For the problem of coordinate drift caused by thermal deformation of the mechanical arm, a thermal deformation compensation mechanism is designed to utilize the geometric invariance of the bolt array, calculate the thermal expansion effect through the distance change between adjacent bolts, and generate a thermal deformation compensation vector , which is expressed by the following formula:

[0179]

[0180] wherein, is the expansion coefficient of steel; is the image center distance (pixel unit) of the same bolt pair in the current frame; is the image center distance of adjacent bolts recorded at the first detection; is the calibration reference temperature 25℃; is the temperature axial compensation factor;

[0181] The final accurate spatial position coordinates of the bolt in the robot base coordinate system after dynamic compensation of the physical environment are obtained, which are expressed by the following formula:

[0182]

[0183] wherein, is the accurate spatial position coordinates of the bolt in the robot base coordinate system.

[0184] S5. According to the obtained robot base bolt spatial coordinates, the mechanical arm robot is driven to tighten the bolt through reinforcement learning dynamic scheduling and dynamic path planning to complete the detection and tightening of the rail bolt.

[0185] In step S5, the reinforcement learning dynamic scheduling is specifically: an intelligent scheduling reward function is designed to balance work efficiency, energy consumption and safety, and drive the mechanical arm to select the optimal path, and the intelligent scheduling reward function R is expressed by the following formula:

[0186]

[0187] wherein, is the efficiency reward term; is the energy consumption penalty term; is the safety reward term; is the number of unfinished maintenance tasks; is the total number of maintenance tasks; is the energy consumption coefficient of the jth joint of the mechanical arm; is the angular acceleration of the jth joint of the mechanical arm; is the length of the time window; is the real-time distance between the end of the mechanical arm and the obstacle; is the observation distance; is the safe operation distance threshold; is the first coefficient of the intelligent scheduling reward function; is the second coefficient of the intelligent scheduling reward function; is the third coefficient of the intelligent scheduling reward function.

[0188] The dynamic path planning is expressed using the following formula:

[0189]

[0190] wherein, is the path cost function; represents the distance between the mechanical arm and the ith obstacle, represents the angular acceleration of the jth joint; is the first coefficient of the dynamic path planning; is the second coefficient of the dynamic path planning.

[0191] The system of the present application is further described below in combination with one embodiment:

[0192] The present embodiment provides a technical solution: an automatic rail bolt detection system based on a 3D vision system, a structural schematic diagram of which is shown in Figure 2 , comprising a line laser scanning device 1, a mechanical arm robot 2, a bolt locking device 3, a vehicle frame 4, a seat 5, a protective cover 6, and a robot control cabinet 7;

[0193] The vehicle frame 4, as the main structure of the device, carries the line laser scanning device 1, the mechanical arm robot 2, the bolt locking device 3, the seat 5, the protective cover 6, the robot control cabinet 7, and the vehicle body power device and the vehicle body damping device arranged at the bottom of the vehicle; and two sets of mounting buckles for the line laser scanning device 1 are reserved in the front part of the vehicle frame, and two mounting buckles for the mechanical arm robot 2 are reserved in the rear part of the vehicle frame, so as to correspond to the detection and maintenance of the two tracks;

[0194] The line laser scanning device 1 is suspended in the front part of the vehicle frame 4 and is used for scanning the tracks and their component states downward; one line laser camera device is arranged at each of the left and right ends to scan the bolts of the two tracks; the protective cover 6 is installed outside the line laser scanning device 1 to protect the device and prevent irrelevant light interference;

[0195] The mechanical arm robot 2 has a structural schematic diagram as shown in Figure 3As shown, mounted at the rear of the vehicle frame 4, used for automatic positioning and moving to the position of loose bolts, a mechanical arm robot 2 is arranged at the left and right ends respectively to correspond to the bolts of the two tracks;

[0196] The bolt locking device 3 has a structure diagram as shown Figure 4 As shown, located at the end of the mechanical arm, used for locking and rotating the corresponding bolt after the mechanical arm moves to the position of loose bolt;

[0197] The robot control cabinet 7 is located behind the seat 5, and the built-in data processing system includes a visual detection system and a machine control system, which is used to identify abnormalities and control the mechanical arm robot 2 to operate;

[0198] Visual detection system: This system is used to receive the data scanned by the line laser scanning device 1, and the 3D visual system images and processes the data to judge the abnormal bolt and calculate its coordinate information, and then transmits it to the machine control system

[0199] Machine control system: receive the abnormal fastener instruction from the visual detection system, when the 3D visual system camera images the coordinates, calculate the positioning information of the fastener bolt relative to the mechanical arm robot 2 through algorithm, and the robot drives the bolt locking device 3 to tighten the fastener, and the machine control system can calculate the accurate position of the bolt according to the data provided by the 3D visual system, to ensure the accuracy of the robot during the locking operation.

[0200] In this embodiment, the device is based on 3D visual system and line laser scanning technology, through the line laser scanning device 1, mechanical arm robot 2, bolt locking device 3 and control system carried by the vehicle frame 4, to realize the intelligent inspection and maintenance of the track bolt. When the inspection starts, the line laser scanning device 1 installed at the front of the vehicle frame 4 scans vertically to the two sides of the track, and the three-dimensional data of the track and bolt is collected through the laser camera equipment at the left and right ends, and the data is transmitted to the visual detection system in the form of point cloud or structured matrix (such as [x, y, z, intensity]);

[0201] After the visual system images and processes the scanning data, it analyzes the state of the track and bolt, identifies the loose or abnormal bolt, and generates accurate three-dimensional coordinate information (such as [bolt_id, x, y, z, status]), and transmits these information to the machine control system.

[0202] Upon receiving the instructions from the vision system, the machine control system calculates the spatial position of the abnormal bolt relative to the robotic arm 2 and calculates the path according to the algorithm, instructing the robotic arm 2 installed at the rear of the vehicle frame to move to the designated position; after reaching the target position, the bolt locking device 3 at the end of the robotic arm 2 precisely aligns the bolt and performs a rotating tightening operation, ensuring the locking of the bolt and the safety of the track.

[0203] Throughout the entire inspection process, the robot control cabinet 7 is responsible for coordinating the visual detection, robotic arm movement, and locking device operation, and storing the detection data and operation records in JSON or CSV format (such as { "timestamp": "2024-10-12T10:30:00", "bolt_id": 101, "x": 35.2, "y": 120.4, "status": "tightened"}), to support subsequent data analysis and track maintenance management.

[0204] At the same time, the device is protected from external environmental light interference by the protective cover 6, and the built-in vibration and power device ensures the stability and efficiency of the device during the inspection process, thereby achieving intelligent, efficient, and accurate track inspection and automated maintenance.

Claims

1. A method for using a rail bolt inspection system, characterized in that, The rail bolt maintenance system comprises a vehicle frame, a line laser scanning device, a mechanical arm robot, a bolt locking device, a visual detection system and a machine control system. The vehicle frame is provided with a vehicle body power device and a vehicle body damping device, and two sets of line laser equipment mounting buckles are reserved on the front part of the vehicle frame, and two mechanical arm robot mounting buckles are reserved on the rear part of the vehicle frame, so as to correspond to the detection and maintenance of the two tracks. The line laser scanning device is hung on the front part of the vehicle frame and is used for scanning the tracks and their component states downward, and one line laser camera device is arranged at each of the left and right ends of the line laser scanning device to scan the bolts of the two tracks. The mechanical arm robot is installed on the rear part of the vehicle frame and is used for automatically positioning and moving to the position of the loose bolt, and one mechanical arm robot is arranged at each of the left and right ends of the mechanical arm robot to correspond to the bolts of the two tracks. The bolt locking device is located at the end of the mechanical arm and is responsible for providing a rotating torque to tighten or loosen the bolt; the mechanical arm transmission machine transmits the power of the driving machine to the end effector, the end effector grabs the bolt and applies a rotating force; in order to prevent over-tightening or under-tightening, the bolt locking device has safety features including torque limitation, angle limitation and fault alarm. The visual detection system is used for receiving the data scanned by the line laser scanning device, and the 3D visual system images and processes the data to judge the abnormal bolt and calculate the coordinate information of the abnormal bolt, and then transmits the abnormal bolt instruction to the machine control system. The machine control system is used for receiving the abnormal bolt instruction transmitted by the visual detection system and commanding the mechanical arm robot to complete the accurate bolt tightening or loosening task. The use method of the rail bolt maintenance system comprises the following steps: S1. Scanning the rail bolt by the line laser scanning device to obtain line laser point cloud data; S2. Constructing a bolt detection model based on a YOLOv11 network structure, inputting the scanning data into the bolt detection model, and obtaining a detection result; S3. Judging the obtained detection result, if the bolt state is missing, immediately sending an alarm information, if the bolt state is loose, entering step S4, if the bolt state is perfect and fastened, completing the detection of the rail bolt; S4. Based on the bolt array geometric constraint and the mechanical arm pose synchronous analysis, a coordinate system dynamic matching mechanism is constructed to position the bolt and obtain the robot base bolt space coordinates; S5. According to the obtained robot base bolt space coordinates, the mechanical arm robot is driven to tighten the bolt through reinforcement learning dynamic scheduling and dynamic path planning, and the detection and tightening of the rail bolt are completed. In step S2, the bolt detection model is based on the YOLOv11 network structure and has the following improvements: The number of input layer channels is expanded from 3 to 7, the newly added channels 4 to 6 are used to input the normal vector of the line laser point cloud, and channel 7 is used to input the bolt surface curvature feature map; The fixed rectangular sampling network is improved to use deformable convolution to guide the sampling point offset through the curvature feature map; The curvature condition branch is introduced into the standard FReLU activation function; A cross-scale attention module is added to the Neck layer to optimize the multi-scale feature fusion effect through channel-space dual domain weight distribution. Adopt EloU Loss instead of CloU Loss as the loss function of the model, and introduce a width-height coverage penalty term to improve the accuracy of bolt bounding box regression. The bolt detection result output by the bolt detection model includes a bounding box coordinate, a confidence, and a bolt state.

2. The method of use of claim 1, wherein, The machine control system is also designed with a self-diagnosis and self-repair mechanism; the self-diagnosis and self-repair mechanism includes a self-diagnosis module and a self-repair module. The self-diagnosis module monitors the torque change of the mechanical arm in real time through the card resistance torque anomaly detection, accurately identifies the abnormal state of the mechanical arm, and the abnormal state determination condition is as follows: or wherein, is a theoretical torque based on the bolt diameter d, material yield strength , friction coefficient ; is a real-time detected mechanical arm torque; is a continuous over-limit time; The self-repairing module automatically performs the tool retracting operation according to the detection result of the self-diagnosis module by the reverse tool retracting strategy, so that manual tool correction is avoided, and the angle of the reverse tool retracting strategy is The calculation formula is as follows: wherein, is the amount of deviation of the actual torque from the theoretical threshold value; is the thread stiffness coefficient; G is the shear modulus of the bolt, L is the thread contact length, and d is the bolt diameter.

3. The method of use of claim 1, wherein, The input feature tensor of the input layer is represented by the following formula: wherein, is an input feature tensor; is red component information in RGB color information; is green component information in RGB color information; is blue component information in RGB color information; is an x-direction normal vector of a linear laser point cloud; is an x-direction normal vector of a linear laser point cloud; is an x-direction normal vector of a linear laser point cloud; is a bolt surface curvature feature map data; The bolt surface curvature feature map is obtained using the following steps: The line laser point cloud data is preprocessed; the preprocessing includes outlier removal and voxel filtering; The surface normal vector of each point is estimated based on k-neighborhood using principal component analysis, and the principal component analysis is represented by the following formula: wherein C is a covariance matrix, reflecting the distribution characteristics of the neighboring points in the three-dimensional space; is the i-th point in the current processed 3D point cloud; is the mean vector of the neighboring point coordinates; V is an eigenvector matrix of the covariance matrix C, and the column vector is an eigen direction; is a diagonal matrix of the covariance matrix C; For each point, a local coordinate system is established with the point as the origin and the normal vector as the z-axis, a local quadratic surface equation is fitted using the least squares method, and two principal curvatures, average curvature and Gaussian curvature are obtained by solving the eigenvalues of the Hessian matrix; Finally, the curvature information of the 3D point cloud is projected onto the 2D image plane, and a curvature feature map containing the principal curvature feature is generated using Gaussian kernel weighting distribution; The local quadratic surface equation is represented by the following formula: wherein, is a point cloud local coordinate system coordinate; is a first quadratic surface coefficient; b is a second quadratic surface coefficient; c is a third quadratic surface coefficient; the quadratic surface coefficients determine the degree of curvature of the surface; The Hessian matrix is represented by the following formula: The Gaussian kernel weighting distribution is represented by the following formula: wherein, is a two-dimensional Gaussian kernel function used to smooth the projection of the curvature values; is the Gaussian kernel standard deviation.

4. The method of use of claim 1, wherein, The feature map output by the deformable convolution is represented by the following formula: wherein, is a feature map of deformable convolution output; is a weight parameter; is an output feature map position; is a fixed offset of the kth position; is obtained by offset network prediction, and K=9 is the number of convolution kernels; The curvature condition branch is introduced into the standard FReLU activation function, and is represented by the following formula: wherein, is an input feature map or tensor, is a standard spatial condition term, is a newly added curvature condition term; the two condition terms are fused through a to-be-learned weight, so that the activation function considers the spatial position information and the geometric curvature information simultaneously.

5. The method of use of claim 1, wherein, The operation of the cross-scale attention module is represented by the following formula: wherein, is the joint feature tensor after multi-scale feature concatenation, is the channel domain weight matrix, is the spatial domain weight matrix; is the output of the cross-scale attention module; The EloU Loss loss function with a width-height coverage penalty term is represented by the following formula: wherein, is the Euclidean distance; is the predicted bounding box center coordinate; is the predicted bounding box width; is the predicted bounding box width-height; is the minimum enclosing box width; is the minimum enclosing box height; is the width of the overlap between the prediction and ground truth bounding box; is the height of the overlap between the prediction and ground truth bounding box; IoU is the Intersection over Union; is the ground truth bounding box width-height; is the ground truth bounding box width-height; is the ground truth bounding box center coordinate.

6. The method of use of claim 1, wherein, Step S4 includes the following steps: Based on the bounding box coordinates output by the model, the pixel coordinates of the bolt center in the image coordinate system are calculated, and are represented by the following formula: wherein, represents a pixel position of the bolt center, is a left upper corner coordinate of the bounding box, is a right lower corner coordinate of the bounding box; Subsequently, based on the collected line laser point cloud data, the bolt center is taken as the reference, and depth value weighted fusion calculation is performed within a 5x5 pixel neighborhood, and is represented by the following formula: wherein, represents a depth value at a coordinate in the point cloud; is a depth value of the bolt center point in the camera coordinate system; i is a horizontal offset; j is a vertical offset; and the weight coefficient is calculated using a Gaussian function with a standard deviation of 0. ； Utilizing a pre-calibrated camera intrinsic matrix The two-dimensional pixel coordinates are back-projected into the camera three-dimensional space using the following equation: wherein, is the three-dimensional coordinate of the screw in the camera coordinate system; the camera intrinsic matrix , represents the camera focal length, is the camera optical center coordinate; The camera coordinate system coordinates are converted to the mechanical arm base coordinate system through a rigid transformation model, and are represented by the following formula: wherein, is a transformation matrix, the inner rotational sub-matrix describes the pose relationship between coordinate systems; the translation vector characterizes the spatial offset; is the spatial coordinate of the bolt in the robot base coordinate system; A vibration compensation mechanism is designed to solve the problem of coordinate drift caused by vibration of the robot arm. The vibration intensity is analyzed by analyzing the position jitter of the bolt image in consecutive frames, and a vibration compensation vector is generated in real time , which is expressed using the following equation: wherein, is the vibration transmission coefficient; N is the sampling window size; is frame horizontal displacement; is frame vertical displacement; is the vibration main direction angle; Aiming at the problem of coordinate drift caused by thermal deformation of the robot arm, a thermal deformation compensation mechanism is designed to utilize the geometric invariance of the bolt array, calculate the thermal expansion effect through the distance change of adjacent bolts, and generate a thermal deformation compensation vector , which is expressed by the following formula: wherein, is the steel expansion coefficient; is the image center distance of the same bolt pair in the current frame; is the image center distance of the adjacent bolt recorded at the first detection; is the calibration reference temperature 25℃; is the temperature axial compensation factor; Finally, the accurate spatial position coordinates of the bolt in the robot base coordinate system after dynamic compensation of the physical environment are obtained, and are represented by the following formula: wherein, is the precise spatial position coordinate of the screw in the robot base coordinate system.

7. The method of use of claim 1, wherein, In step S5, the dynamic scheduling through reinforcement learning is specifically: an intelligent scheduling reward function is designed to balance job efficiency, energy consumption and safety, drive the mechanical arm to select the optimal path, and the intelligent scheduling reward function R is represented by the following formula: wherein, is an efficiency reward term; is an energy consumption penalty term; is a safety reward term; is the number of uncompleted maintenance tasks; is the total number of maintenance tasks; is the energy consumption coefficient of the jth joint of the robot arm; is the angular acceleration of the jth joint of the robot arm; is the length of the time window; is the real-time distance between the end of the robot arm and the obstacle; is the observation distance; is the safety operation distance threshold value; is the first coefficient of the intelligent scheduling reward function; is the second coefficient of the intelligent scheduling reward function; is the third coefficient of the intelligent scheduling reward function.

8. The method of use of claim 1, wherein, In step S5, the dynamic path planning is represented by the following formula: wherein, is a path cost function; denotes the distance of the robot arm to the i-th obstacle, denotes the j-th joint angle acceleration; is a first coefficient of dynamic path planning; is a second coefficient of dynamic path planning.

Images