Railway fastener bolt tightness detection method and device

Abstract

The invention discloses a railway fastener bolt tightness detection method and device. The method comprises the following steps: acquiring an RGB image and a corresponding depth image of a railway fastener area; inputting the RGB image into the trained bolt area detection model, outputting a bolt area of the RGB image, and determining a two-dimensional coordinate of the bolt area; extracting depth data of a corresponding area from the depth image according to the two-dimensional coordinates of the bolt area, generating a depth sub-image of the bolt area, converting pixel coordinates of the depth sub-image into three-dimensional point cloud coordinates, and obtaining three-dimensional point cloud coordinate data of the bolt area; a PointNet point cloud classification technology is adopted to classify bolt states, and a bolt state classification result is obtained; according to the invention, the efficiency, accuracy and intelligent level of railway fastener defect detection can be improved.

Description

Technical Field

[0001] This invention relates to the field of railway fastener testing technology, and in particular to a method and apparatus for testing the tightness of railway fastener bolts. Background Technology

[0002] This section is intended to provide background or context for the embodiments of the invention set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.

[0003] Railway fasteners, as key connecting components of the track structure, directly affect track stability and train operation safety, making them a key area of ​​inspection for railway infrastructure. Currently, fastener bolt defect detection mainly relies on manual inspection or single-modal inspection technology: manual inspection is inefficient and highly subjective, making it difficult to meet the real-time monitoring needs of large-scale lines; traditional two-dimensional visual inspection, while able to identify surface defects such as rust and missing bolts, lacks effective perception capabilities for three-dimensional structural defects such as spatial displacement and posture changes caused by loosening; single three-dimensional point cloud processing technology, although able to obtain bolt spatial position information, is limited by the massive computational pressure of full-scene point cloud data, easily leading to low detection efficiency and high false positive rates in complex backgrounds. These shortcomings result in a persistently high rate of missed detection for bolt loosening defects and a lack of quantitative assessment capabilities, severely hindering the development of intelligent monitoring of railway fasteners. Summary of the Invention

[0004] This invention provides a method for detecting the tightness of railway fastener bolts, aiming to improve the efficiency, accuracy, and intelligence level of railway fastener defect detection. The method includes:

[0005] Acquire RGB images and corresponding depth images of the railway fastener area;

[0006] An RGB image is input into a trained bolt region detection model, which outputs the bolt region in the RGB image and determines the two-dimensional coordinates of the bolt region. The bolt region detection model is established based on an improved YOLOv8n network structure.

[0007] Based on the two-dimensional coordinates of the bolt area, the depth data of the corresponding area is extracted from the depth image. A depth sub-map of the bolt area is generated based on the depth data. The pixel coordinates of the depth sub-map are converted into three-dimensional point cloud coordinates to obtain the three-dimensional point cloud coordinate data of the bolt area.

[0008] PointNet point cloud classification technology is used to classify the bolt status based on the three-dimensional point cloud coordinate data of the bolt area, and the bolt status classification results are obtained; the bolt status classification results include loose state or tight state.

[0009] This invention also provides a railway fastener bolt tightness detection device to improve the efficiency, accuracy, and intelligence level of railway fastener defect detection. The device includes:

[0010] The image acquisition module is used to acquire RGB images and corresponding depth images of the railway fastener area;

[0011] The bolt 2D coordinate determination module is used to input an RGB image into a trained bolt region detection model, output the bolt region in the RGB image, and determine the 2D coordinates of the bolt region; the bolt region detection model is established based on an improved YOLOv8n network structure;

[0012] The bolt 3D point cloud coordinate determination module is used to extract the depth data of the corresponding area from the depth image based on the 2D coordinates of the bolt area, generate a depth sub-map of the bolt area based on the depth data, convert the pixel coordinates of the depth sub-map into 3D point cloud coordinates, and obtain the 3D point cloud coordinate data of the bolt area.

[0013] The bolt state classification module uses PointNet point cloud classification technology to classify the bolt state based on the three-dimensional point cloud coordinate data of the bolt area, and obtains the bolt state classification result; the bolt state classification result includes loose state or tight state.

[0014] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described method for detecting the tightness of railway fastener bolts.

[0015] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for detecting the tightness of railway fastener bolts.

[0016] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described method.

[0017] In this embodiment of the invention, RGB images and corresponding depth images of the railway fastener area are acquired; the RGB images are input into a trained bolt area detection model, which outputs the bolt area in the RGB image, thus determining the two-dimensional coordinates of the bolt area; the bolt area detection model is established based on an improved YOLOv8n network structure; based on the two-dimensional coordinates of the bolt area, depth data of the corresponding area is extracted from the depth image, a depth sub-image of the bolt area is generated based on the depth data, and the pixel coordinates of the depth sub-image are converted into three-dimensional point cloud coordinates to obtain the three-dimensional point cloud coordinate data of the bolt area; PointNet point cloud classification technology is used to classify the bolt state based on the three-dimensional point cloud coordinate data of the bolt area, resulting in bolt state classification results; the bolt state classification results include loose or tight states. In the above process, this embodiment of the invention accurately locates the bolt area and determines the two-dimensional coordinates of the bolt area based on the improved YOLOv8n model, significantly reducing the three-dimensional point cloud processing range and lowering computational complexity. Using PointNet technology, the bolt state is classified based on the three-dimensional point cloud coordinate data to achieve loose / tight state identification. This invention addresses the shortcomings of subjective inefficiency in manual inspection, inability of two-dimensional detection to perceive three-dimensional displacement, and low computational efficiency in three-dimensional full-scene processing by integrating RGB and depth images through a cross-modal collaborative mechanism. It achieves automated and high-precision identification of bolt loosening under high-speed operation conditions, significantly improving the efficiency, accuracy, and intelligence level of railway fastener defect detection. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0019] Figure 1 This is a flowchart of the railway fastener bolt tightness detection method in an embodiment of the present invention;

[0020] Figure 2 The image shown is an RGB image in an embodiment of the present invention.

[0021] Figure 3 This is a depth image in an embodiment of the present invention;

[0022] Figure 4 This is a diagram showing the bolt area positioning results in an embodiment of the present invention;

[0023] Figure 5 This is a 3D point cloud rendering of the bolt area after conversion in an embodiment of the present invention.

[0024] Figure 6 This is a flowchart for calculating the bolt loosening amount in an embodiment of the present invention;

[0025] Figure 7 This is a schematic diagram of a railway fastener bolt tightness detection device in an embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0027] Figure 1 This is a flowchart of a railway fastener bolt tightness detection method according to an embodiment of the present invention. The method includes:

[0028] Step 101: Acquire RGB images and corresponding depth images of the railway fastener area;

[0029] Step 102: Input the RGB image into the trained bolt region detection model, output the bolt region of the RGB image, and determine the two-dimensional coordinates of the bolt region; the bolt region detection model is established based on the improved YOLOv8n network structure;

[0030] Step 103: Based on the two-dimensional coordinates of the bolt area, extract the depth data of the corresponding area from the depth image, generate a depth sub-map of the bolt area based on the depth data, convert the pixel coordinates of the depth sub-map into three-dimensional point cloud coordinates, and obtain the three-dimensional point cloud coordinate data of the bolt area.

[0031] Step 104: Using PointNet point cloud classification technology, the bolt status is classified based on the three-dimensional point cloud coordinate data of the bolt area to obtain the bolt status classification result; the bolt status classification result includes loose state or tight state.

[0032] Each step is explained in detail below.

[0033] In step 101, RGB images and corresponding depth images of the railway fastener area are acquired.

[0034] In a specific embodiment, an integrated 2D / 3D imaging component for track equipment is installed on the inspection train or track inspection equipment. This imaging system supports continuous synchronous acquisition at inspection speeds up to 120 km / h, and can stably acquire multimodal data of the fastener area during high-speed train operation. Through integrated design and synchronous control technology, the spatiotemporal misalignment problem of traditional discrete sensors is effectively solved, significantly improving the quality of detection data in complex dynamic scenarios. The component ensures strict pixel-level alignment of the coordinate systems of both, synchronously acquiring two-dimensional RGB images and corresponding depth images of the railway fastener area. Figure 2 The RGB image in the embodiment of the present invention is as follows: Figure 2 As shown, the two-dimensional RGB image clearly presents the appearance features of components such as bolts and springs. Figure 3 For example, the depth image in the embodiment of the present invention, such as Figure 3 As shown, the depth map accurately records the three-dimensional distance information of each pixel, providing a high-precision, pixel-level aligned raw data foundation for subsequent fastener bolt inspection.

[0035] In step 102, the RGB image is input into the trained bolt region detection model, and the bolt region of the RGB image is output to determine the two-dimensional coordinates of the bolt region; the bolt region detection model is established based on the improved YOLOv8n network structure.

[0036] In a specific embodiment, the bolt region detection model established using an improved YOLOv8n network structure includes the following parts:

[0037] Data annotation: The bolt head region in the 2D RGB image was manually annotated using VOC format bounding boxes, with the category label "bolt". To address complex situations such as rust and occlusion of the bolt head, the smallest bounding rectangle of the fully visible portion was retained during annotation to ensure the diversity of the training data.

[0038] Model Construction: A lightweight YOLOv8n network architecture is adopted. Considering the small size of bolt targets (typically occupying 0.5%-5% of image pixels), a Focus module is introduced into the backbone network to enhance the feature extraction capability for small targets. The neck feature fusion path is optimized to enhance the ability of multi-scale features to express bolt details. A CIoU loss function combined with classification cross-entropy loss is used to improve localization accuracy and classification confidence.

[0039] Bolt area detection process: Input a two-dimensional RGB image, and output the two-dimensional coordinates and confidence level of the bolt area through YOLOv8 model inference.

[0040] In one embodiment, it further includes:

[0041] In the process of improving the YOLOv8n network structure, the Focus module is embedded into the backbone network of YOLOv8n, the original feature fusion path of YOLOv8n is replaced by the AFPN progressive feature pyramid network, and the original loss function IoU of YOLOv8n is optimized by the CIoU loss function.

[0042] In a specific embodiment, the main improvements to the YOLOv8n model include:

[0043] 1. Focus structure design

[0044] A Focus module is added before the initial convolutional layer of YOLOv8n. This module slices the input image (640×640×3) into a 2×2 grid, stitches them into four sub-images (320×320×12), and then performs a 3×3 convolution to compress the channels to 64, achieving downsampling without information loss. This structure preserves the edge details of small targets and improves the expression of shallow features, making it particularly suitable for small targets such as bolts.

[0045] 2. AFPN Progressive Feature Pyramid

[0046] The original PANet structure is replaced with the AFPN progressive feature pyramid, which fuses layer features through bidirectional cross-scale connections: Top-down path: High-level semantic information is upsampled to lower levels to supplement bolt contour details. Bottom-up path: Low-level spatial information is downsampled to higher levels to enhance small target localization capabilities.

[0047] 3. Loss Function Optimization

[0048] The CIoU loss function is used instead of the original IoU loss function. The CIoU loss function is more sensitive to the center offset of small targets, thus improving the bolt positioning accuracy. Figure 4 This is a diagram showing the bolt area positioning results in an embodiment of the present invention, such as... Figure 4 As shown, the area within the red box is the bolt area.

[0049] In step 103, based on the two-dimensional coordinates of the bolt area, the depth data of the corresponding area is extracted from the depth image, a depth sub-map of the bolt area is generated based on the depth data, and the pixel coordinates of the depth sub-map are converted into three-dimensional point cloud coordinates to obtain the three-dimensional point cloud coordinate data of the bolt area.

[0050] In one embodiment, converting the pixel coordinates of the depth submap into 3D point cloud coordinates includes:

[0051] Perform coordinate transformation according to the following formula:

[0052]

[0053] Z = D(u,v)

[0054] Where u0 and v0 are the optical center pixel coordinates of the camera, f x f is the focal length parameter along the camera's x-axis. y Let be the focal length parameter of the camera along the y-axis, D(u, v) be the depth value of the pixel (u, v) in the depth image, and X, Y, and Z be the 3D point cloud coordinates along the x-axis, y-axis, and z-axis, respectively.

[0055] Figure 5 This is a 3D point cloud image of the bolt region after conversion in an embodiment of the present invention. In a specific embodiment, after converting the pixel coordinates of the depth sub-image into 3D point cloud coordinates, the 3D point cloud coordinate set P of the bolt region is obtained:

[0056]

[0057] Where N is the number of 3D point cloud data, p i Let X represent the i-th 3D point cloud data. i ,Y i Z i Let x, y, z be the coordinates of the i-th 3D point cloud data in the x, y, z axes.

[0058] In step 104, PointNet point cloud classification technology is used to classify the bolt state based on the three-dimensional point cloud coordinate data of the bolt area, obtaining bolt state classification results; the bolt state classification results include loose state or tight state. By combining the small target detection capability of the improved YOLOv8n model with the PointNet point cloud classification algorithm, the identification and quantitative assessment of bolt loosening defects are realized.

[0059] In a specific embodiment, the PointNet model is used, which includes input transformation matrix (T-Net) for coordinate normalization of the original point cloud, point-by-point feature extraction (multilayer perceptron + ReLU activation), global feature pooling (max pooling), and a classification head (fully connected layer + Softmax). To address the spatial symmetry of the bolt point cloud, position encoding (XYZ coordinates as initial features) is introduced in the feature extraction layer to enhance sensitivity to the geometric structure of the point cloud.

[0060] Using 3D point cloud data of loose bolts (artificially simulated loosening state, bolt head displacement ≥2mm) and fastened bolts as training samples, the training and validation sets were divided in a 7:3 ratio. The Adam optimizer (learning rate 0.001) was employed with cross-entropy loss. Data augmentation (random point cloud rotation ±10°, scaling ±5%, translation ±1mm) was used during training to improve the model's generalization ability. The input set of 3D point cloud coordinates P for the bolt region was processed by PointNet, and the output classification result ("loose" or "fastened") was provided.

[0061] Figure 6 This is a flowchart for calculating the bolt loosening amount in an embodiment of the present invention. In one embodiment, after obtaining the bolt condition classification result, the method further includes:

[0062] Step 601: When the bolt status classification result is loose, obtain the three-dimensional point cloud coordinates of the bolt in the current loose state;

[0063] Step 602: Compare the three-dimensional point cloud coordinates of the bolt in the current loose state with the standard three-dimensional point cloud coordinates of the bolt in the known tight state;

[0064] Step 603: Based on the comparison results, determine the deviation between the bolt's three-dimensional point cloud coordinates in the current loosened state and the standard three-dimensional point cloud coordinates, and calculate the axial displacement and planar offset based on the deviation.

[0065] Step 604: Calculate the amount of bolt loosening based on the axial displacement and planar offset.

[0066] In one embodiment, the loosening amount of the bolt is calculated based on the axial displacement and planar offset, including:

[0067] Calculate the axial displacement using the following formula:

[0068] ΔZ=|Z avg -Z0|

[0069] Where ΔZ is the axial displacement, Z avg Z is the z-axis coordinate of the bolt in its current loose state, and Z0 is the standard z-axis coordinate of the bolt in its tightened state.

[0070] Calculate the plane offset using the following formula:

[0071]

[0072] Where ΔXY is the plane offset, X avg Let X be the x-axis coordinate of the bolt in its current loosened state, and Y be the y-axis coordinate. avg X0 represents the bolt's y-axis coordinate in the current loose state, X0 represents the bolt's standard x-axis coordinate in the tightened state, and Y0 represents the bolt's standard y-axis coordinate in the tightened state.

[0073] Calculate the amount of bolt loosening using the following formula:

[0074]

[0075] Where ΔS represents the amount of bolt loosening.

[0076] This invention also provides a railway fastener bolt tightness detection device, as described in the following embodiments. Since the principle by which this device solves the problem is similar to the railway fastener bolt tightness detection method, the implementation of this device can refer to the implementation of the railway fastener bolt tightness detection method; repeated details will not be elaborated further.

[0077] Figure 7 This is a schematic diagram of a railway fastener bolt tightness detection device according to an embodiment of the present invention. The device includes:

[0078] Image acquisition module 701 is used to acquire RGB images and corresponding depth images of the railway fastener area;

[0079] The bolt two-dimensional coordinate determination module 702 is used to input the RGB image into the trained bolt region detection model, output the bolt region of the RGB image, and determine the two-dimensional coordinates of the bolt region; the bolt region detection model is established based on the improved YOLOv8n network structure;

[0080] The bolt 3D point cloud coordinate determination module 703 is used to extract the depth data of the corresponding area from the depth image based on the 2D coordinates of the bolt area, generate a depth sub-map of the bolt area based on the depth data, convert the pixel coordinates of the depth sub-map into 3D point cloud coordinates, and obtain the 3D point cloud coordinate data of the bolt area.

[0081] The bolt state classification module 704 is used to classify the bolt state based on the three-dimensional point cloud coordinate data of the bolt area using PointNet point cloud classification technology, and obtain the bolt state classification result; the bolt state classification result includes loose state or tight state.

[0082] In one embodiment, the system further includes a bolt area detection model improvement module, specifically used for:

[0083] In the process of improving the YOLOv8n network structure, the Focus module is embedded into the backbone network of YOLOv8n, the original feature fusion path of YOLOv8n is replaced by the AFPN progressive feature pyramid network, and the original loss function IoU of YOLOv8n is optimized by the CIoU loss function.

[0084] In one embodiment, the bolt three-dimensional point cloud coordinate determination module 703 is specifically used for:

[0085] Convert the pixel coordinates of the depth submap to 3D point cloud coordinates using the following formula:

[0086]

[0087] Z = D(u,v)

[0088] Where u0 and v0 are the optical center pixel coordinates of the camera, f x f is the focal length parameter along the camera's x-axis. y Let represent the focal length parameter of the camera along the y-axis, D(u,v) be the depth value of the pixel (u,v) in the depth image, and X, Y, and Z be the 3D point cloud coordinates along the x-axis, y-axis, and z-axis, respectively.

[0089] In one embodiment, a bolt loosening amount calculation module is also included, specifically used for:

[0090] When the bolt status classification result is loose, obtain the three-dimensional point cloud coordinates of the bolt in the current loose state;

[0091] Compare the three-dimensional point cloud coordinates of the bolt in its current loose state with the standard three-dimensional point cloud coordinates of the bolt in its known tight state;

[0092] Based on the comparison results, the deviation between the bolt's three-dimensional point cloud coordinates in the current loosened state and the standard three-dimensional point cloud coordinates is determined, and the axial displacement and planar offset are calculated based on the deviation.

[0093] Calculate the amount of bolt loosening based on the axial displacement and planar offset.

[0094] In one embodiment, the bolt loosening amount calculation module is specifically used for:

[0095] Calculate the axial displacement using the following formula:

[0096] ΔZ=|Z avg -Z0|

[0097] Where ΔZ is the axial displacement, Z avg Z is the z-axis coordinate of the bolt in its current loose state, and Z0 is the standard z-axis coordinate of the bolt in its tightened state.

[0098] Calculate the plane offset using the following formula:

[0099]

[0100] Where ΔXY is the plane offset, X avg Let X be the x-axis coordinate of the bolt in its current loosened state, and Y be the y-axis coordinate. avg X0 represents the bolt's y-axis coordinate in the current loose state, X0 represents the bolt's standard x-axis coordinate in the tightened state, and Y0 represents the bolt's standard y-axis coordinate in the tightened state.

[0101] The amount of bolt loosening is calculated according to the following formula based on the axial displacement and planar offset:

[0102]

[0103] Where ΔS represents the amount of bolt loosening.

[0104] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described method for detecting the tightness of railway fastener bolts.

[0105] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for detecting the tightness of railway fastener bolts.

[0106] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described method for detecting the tightness of railway fastener bolts.

[0107] In this embodiment of the invention, RGB images and corresponding depth images of the railway fastener area are acquired; the RGB images are input into a trained bolt area detection model, which outputs the bolt area in the RGB image, thus determining the two-dimensional coordinates of the bolt area; the bolt area detection model is established based on an improved YOLOv8n network structure; based on the two-dimensional coordinates of the bolt area, depth data of the corresponding area is extracted from the depth image, a depth sub-image of the bolt area is generated based on the depth data, and the pixel coordinates of the depth sub-image are converted into three-dimensional point cloud coordinates to obtain the three-dimensional point cloud coordinate data of the bolt area; PointNet point cloud classification technology is used to classify the bolt state based on the three-dimensional point cloud coordinate data of the bolt area, resulting in bolt state classification results; the bolt state classification results include loose or tight states. In the above process, this embodiment of the invention accurately locates the bolt area and determines the two-dimensional coordinates of the bolt area based on the improved YOLOv8n model, significantly reducing the three-dimensional point cloud processing range and lowering computational complexity. Using PointNet technology, the bolt state is classified based on the three-dimensional point cloud coordinate data to achieve loose / tight state identification. This invention addresses the shortcomings of subjective inefficiency in manual inspection, inability of two-dimensional detection to perceive three-dimensional displacement, and low computational efficiency in three-dimensional full-scene processing by integrating RGB and depth images through a cross-modal collaborative mechanism. It achieves automated and high-precision identification of bolt loosening under high-speed operation conditions, significantly improving the efficiency, accuracy, and intelligence level of railway fastener defect detection.

[0108] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0109] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0110] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0111] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0112] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting the tightness of railway fastener bolts, characterized in that, include: Acquire RGB images and corresponding depth images of the railway fastener area; An RGB image is input into a trained bolt region detection model, which outputs the bolt region in the RGB image and determines the two-dimensional coordinates of the bolt region. The bolt region detection model is established based on an improved YOLOv8n network structure. Based on the two-dimensional coordinates of the bolt area, the depth data of the corresponding area is extracted from the depth image. A depth sub-map of the bolt area is generated based on the depth data. The pixel coordinates of the depth sub-map are converted into three-dimensional point cloud coordinates to obtain the three-dimensional point cloud coordinate data of the bolt area. PointNet point cloud classification technology is used to classify the bolt status based on the three-dimensional point cloud coordinate data of the bolt area, and the bolt status classification results are obtained; the bolt status classification results include loose state or tight state.

2. The method as described in claim 1, characterized in that, Also includes: In the process of improving the YOLOv8n network structure, the Focus module is embedded into the backbone network of YOLOv8n, the original feature fusion path of YOLOv8n is replaced by the AFPN progressive feature pyramid network, and the original loss function IoU of YOLOv8n is optimized by the CIoU loss function.

3. The method as described in claim 1, characterized in that, Converting the pixel coordinates of the depth submap to 3D point cloud coordinates includes: Perform coordinate transformation according to the following formula: Z = D(u,v) Where u0 and v0 are the optical center pixel coordinates of the camera, f x f is the focal length parameter along the camera's x-axis. y Let be the focal length parameter of the camera along the y-axis, D(u, v) be the depth value of the pixel (u, v) in the depth image, and X, Y, and Z be the 3D point cloud coordinates along the x-axis, y-axis, and z-axis, respectively.

4. The method as described in claim 1, characterized in that, After obtaining the bolt condition classification results, the following is also included: When the bolt status classification result is loose, obtain the three-dimensional point cloud coordinates of the bolt in the current loose state; Compare the three-dimensional point cloud coordinates of the bolt in its current loose state with the standard three-dimensional point cloud coordinates of the bolt in its known tight state; Based on the comparison results, the deviation between the bolt's three-dimensional point cloud coordinates in the current loosened state and the standard three-dimensional point cloud coordinates is determined, and the axial displacement and planar offset are calculated based on the deviation. Calculate the amount of bolt loosening based on the axial displacement and planar offset.

5. The method as described in claim 4, characterized in that, Calculate the amount of bolt loosening based on the axial displacement and planar offset, including: Calculate the axial displacement using the following formula: ΔZ=|Z avg -Z0| Where ΔZ is the axial displacement, Z avg Z is the z-axis coordinate of the bolt in its current loose state, and Z0 is the standard z-axis coordinate of the bolt in its tightened state. Calculate the plane offset using the following formula: Where ΔXY is the plane offset, X avg Let X be the x-axis coordinate of the bolt in its current loosened state, and Y be the y-axis coordinate. avg X0 represents the bolt's y-axis coordinate in the current loose state, X0 represents the bolt's standard x-axis coordinate in the tightened state, and Y0 represents the bolt's standard y-axis coordinate in the tightened state. Calculate the amount of bolt loosening using the following formula: Where ΔS represents the amount of bolt loosening.

6. A railway fastener bolt tightness detection device, characterized in that, include: The image acquisition module is used to acquire RGB images and corresponding depth images of the railway fastener area; The bolt 2D coordinate determination module is used to input an RGB image into a trained bolt region detection model, output the bolt region in the RGB image, and determine the 2D coordinates of the bolt region; the bolt region detection model is established based on an improved YOLOv8n network structure; The bolt 3D point cloud coordinate determination module is used to extract the depth data of the corresponding area from the depth image based on the 2D coordinates of the bolt area, generate a depth sub-map of the bolt area based on the depth data, convert the pixel coordinates of the depth sub-map into 3D point cloud coordinates, and obtain the 3D point cloud coordinate data of the bolt area. The bolt state classification module uses PointNet point cloud classification technology to classify the bolt state based on the three-dimensional point cloud coordinate data of the bolt area, and obtains the bolt state classification result; the bolt state classification result includes loose state or tight state.

7. The apparatus as claimed in claim 6, characterized in that, It also includes a bolt area detection model improvement module, specifically used for: In the process of improving the YOLOv8n network structure, the Focus module is embedded into the backbone network of YOLOv8n, the original feature fusion path of YOLOv8n is replaced by the AFPN progressive feature pyramid network, and the original loss function IoU of YOLOv8n is optimized by the CIoU loss function.

8. The apparatus as claimed in claim 6, characterized in that, The bolt 3D point cloud coordinate determination module is specifically used for: Convert the pixel coordinates of the depth submap to 3D point cloud coordinates using the following formula: Z = D(u,v) Where u0 and v0 are the optical center pixel coordinates of the camera, f x f is the focal length parameter along the camera's x-axis. y Let be the focal length parameter of the camera along the y-axis, D(u, v) be the depth value of the pixel (u, v) in the depth image, and X, Y, and Z be the 3D point cloud coordinates along the x-axis, y-axis, and z-axis, respectively.

9. The apparatus as claimed in claim 6, characterized in that, It also includes a bolt loosening amount calculation module, specifically used for: When the bolt status classification result is loose, obtain the three-dimensional point cloud coordinates of the bolt in the current loose state; Compare the three-dimensional point cloud coordinates of the bolt in its current loose state with the standard three-dimensional point cloud coordinates of the bolt in its known tight state; Based on the comparison results, the deviation between the bolt's three-dimensional point cloud coordinates in the current loosened state and the standard three-dimensional point cloud coordinates is determined, and the axial displacement and planar offset are calculated based on the deviation. Calculate the amount of bolt loosening based on the axial displacement and planar offset.

10. The apparatus as claimed in claim 9, characterized in that, The bolt loosening amount calculation module is specifically used for: Calculate the axial displacement using the following formula: ΔZ=|Z avg -Z0| Where ΔZ is the axial displacement, Z avg Z is the z-axis coordinate of the bolt in its current loose state, and Z0 is the standard z-axis coordinate of the bolt in its tightened state. Calculate the plane offset using the following formula: Where ΔXY is the plane offset, X avg Let X be the x-axis coordinate of the bolt in its current loosened state, and Y be the y-axis coordinate. avg X0 represents the bolt's y-axis coordinate in the current loose state, X0 represents the bolt's standard x-axis coordinate in the tightened state, and Y0 represents the bolt's standard y-axis coordinate in the tightened state. The amount of bolt loosening is calculated according to the following formula based on the axial displacement and planar offset: Where ΔS represents the amount of bolt loosening.

11. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 5.

13. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 5.

Images