Real-time track disease detection method and device

Abstract

The invention discloses a real-time track disease detection method and device. The method comprises the steps that a multi-modal image and positioning and attitude determination data of a track area are collected in real time through a GPU detection device; performing fusion and enhancement processing on the multi-modal image to obtain an enhanced image; inputting the enhanced image into a disease detection-based network, and identifying a disease target in real time; performing time sequence verification on disease targets in continuous multiple frames, and eliminating transient false detection based on a cross-frame Boolean voting result to confirm a stable disease target; risk assessment and grading are carried out based on the pixel coordinates and the widths of the disease targets; fusing the pixel coordinates of the stable disease target with positioning and attitude determination data, and mapping to a GNSS / RTK coordinate system; and uploading structured data containing the disease information and the positioning result to a cloud end in real time, and automatically triggering an alarm and maintenance order sending process based on the risk level.

Description

Technical Field

[0001] This invention relates to the fields of intelligent operation and maintenance of railway infrastructure and computer vision technology, specifically to a method and device for real-time detection of track defects. Background Technology

[0002] With the large-scale construction of high-speed railways in my country, the safety monitoring of track conditions is crucial to ensuring smooth train operation and extending the service life of tracks. As the operating mileage and operating time increase, defects in high-speed railway infrastructure are gradually becoming apparent. my country's high-speed railways traverse multiple geological and geomorphological units, have diverse track structure types, and are affected by complex climatic environments, making them prone to various typical damages such as surface cracks, gaps, and broken pieces, posing a threat to high-speed train operation.

[0003] There are some research achievements in China regarding defects or problems in ballastless tracks. Currently, mature applications mainly focus on detecting damage to ballastless track rails, fasteners, and track slab cracks. Compared with the actual needs of high-speed railway ballastless tracks, the following shortcomings still exist: 1) Existing integrated inspection vehicles and track slab crack and gap detectors cannot accurately detect defects such as gaps and voids; the detection scope is incomplete and does not meet the requirements of the "High-Speed ​​Railway Line Maintenance Rules" (Railway Engineering Regulations

[2023] No. 106). Currently, the railway industry mainly uses the following methods for track defect inspection: (1) Manual foot patrol: Maintenance personnel carry hand lamps or portable testing instruments to visually inspect or tap cracks, defects and gaps in the tracks. This method relies on subjective experience, has low work efficiency, high labor intensity and is easily limited by weather and visibility, making it difficult to meet the high-frequency inspection needs of high-density lines.

[0004] (2) Inspection trolley: fine inspection.

[0005] (3) Fixed or moving visual inspection systems: In recent years, these systems have begun to be equipped with area scan cameras and use traditional edge detection, template matching, or single convolutional neural networks (CNNs) for surface defect identification, but they still have significant shortcomings in the following aspects: (1) Weak ability to identify small targets: Crack width is often less than 0.5 mm. Currently, mobile detection devices are easily ignored after high-speed movement and multi-level downsampling. Existing single-scale or Anchor-based models have a high rate of missing small targets. (2) Poor resistance to environmental interference: Existing systems mostly rely on single visible light images, which are not robust enough to complex working conditions such as rain, snow, low light in tunnels, strong reflection, rust and stains; (3) Insufficient real-time processing: Some solutions use CPU or low-power SoC offline processing, which makes it difficult to complete frame-level inference at inspection speeds of 5-20 km / h, and thus cannot provide timely alarms; (4) Limited positioning accuracy: Most studies only output image pixel-level defect boxes, lacking deep integration with RTK-GNSS and inertial measurement unit (IMU), making it difficult to achieve centimeter-level defect landing points in road maintenance management systems; (5) Incomplete operation and maintenance loop: The detection results often remain at the "discovery" level and do not form a closed loop with cloud-based order dispatch, repair and review, resulting in information silos and making it difficult to support refined maintenance decisions.

[0006] In summary, existing technologies cannot simultaneously meet the comprehensive requirements of high-precision small target detection, robustness in complex environments, centimeter-level spatial positioning, and real-time alarm dispatching. The railway industry urgently needs a complete solution that can operate during the dynamic movement of the inspection trolley, integrates multimodal image enhancement and deep learning for small target detection, and combines GPU real-time inference with RTK / IMU high-precision positioning to achieve online accurate identification and closed-loop operation and maintenance management of various types of defects such as cracks, defects, gaps, and fastener defects. Summary of the Invention

[0007] To achieve the objective of this invention, this application provides a method for real-time detection of track defects, comprising: Step S1: Real-time acquisition of multimodal images and positioning / attitude data of the track area using a GPU detection device; Step S2: Perform fusion and enhancement processing on the multimodal images to obtain an enhanced image; Step S3: Input the enhanced image into a disease detection network based on Transformer and YOLOv8 to identify disease targets in real time; Step S4: Perform time-series verification on the disease targets in multiple consecutive frames, and eliminate transient false detections based on the cross-frame Boolean voting results to confirm stable disease targets; Step S5: Based on the pixel coordinates of the diseased target, perform risk assessment and level classification using the AHP-entropy weight method; Step S6: Fuse the pixel coordinates of the stable disease target with the positioning and attitude data, and map them to the GNSS coordinate system or RTK coordinate system to achieve centimeter-level positioning; Step S7: Upload the structured data containing disease information and location results to the cloud in real time, and automatically trigger alarms and maintenance dispatch processes based on risk level.

[0008] In some specific embodiments, step S1 includes: Step S11: Using a visible light camera and an infrared camera deployed on the GPU detection device, image sequences of the orbital region are acquired in real time under the synchronization of PPS pulse signals to obtain multimodal images of the orbital region; Step S12: Synchronously record the positioning and attitude data of the detection device when acquiring each frame of image through the GNSS / RTK module and the IMU module; wherein, the multimodal image includes visible light image and infrared image.

[0009] In some specific embodiments, step S2 includes: Step S21: Enhance the visible light image using the multi-scale Retinex algorithm; Step S22: By the following formula The enhanced visible light image is fused with the infrared image to generate a fused image, where R( I RGB This is the enhanced visible light image. α The fusion coefficient is... I IR Infrared image, I F To merge images; Step S23: Perform contrast-limited adaptive histogram equalization on the fused image to determine the final enhanced image.

[0010] In some specific embodiments, in step S3, a weighted loss function is introduced into the disease detection network based on Transformer and YOLOv8 to accurately identify small target diseases. The weighted loss function is determined according to the following formula: in, The weighted GIoU loss, The loss function is the standard GIoU loss function, where w and h are the width and height of the target bounding box, respectively. A preset size threshold is used to distinguish small targets, and γ is a hyperparameter that adjusts the weighting.

[0011] In some specific embodiments, step S4 includes: Step S41: For the same candidate target associated with the trajectory in multiple consecutive frames, calculate the average confidence of the candidate target as the result of cross-frame Boolean voting; Step S42: Compare the average confidence level with a preset threshold, and based on the cross-frame Boolean voting results, determine the candidate disease target as a stable disease target and output it, or remove it as a transient false detection.

[0012] In some specific embodiments, step S5 includes: Step S51: Construct multiple evaluation indicators based on the pixel coordinates of the diseased target; Step S52: Determine the combined weights of each evaluation index using the AHP-entropy weight method; Step S53: Determine the numerical range of the comprehensive risk index by combining the weights and the normalized quantitative values ​​of each indicator; Step S54: Based on the numerical range of the risk index, classify the disease into different risk levels.

[0013] In some specific embodiments, step S6 includes: Step S61: Using camera intrinsic parameters, transform the pixel coordinates of the stable disease target to the camera coordinate system through the pinhole model; Step S62: Combining the extrinsic parameters of the camera and the vehicle body, transform the camera coordinates to the GNSS / RTK coordinate system; Step S63: Using the bundle adjustment algorithm, the image coordinates of the stable disease target and its corresponding positioning and attitude data from multiple time frames are fused together, and combined with IMU attitude data, the final geographic coordinates and route coordinates of the stable disease target are optimized to ensure that the planar positioning error is no more than 3 cm.

[0014] To achieve the same inventive objective, this application also provides a real-time detection device for track defects, comprising: Multimodal synchronous acquisition unit: used for real-time acquisition of multimodal images and positioning and attitude data of the track area; Image enhancement preprocessing unit: used to perform fusion and enhancement processing on the multimodal image to obtain an enhanced image; Embedded GPU inference unit: used to input the enhanced image into a disease detection network based on Transformer and YOLOv8 to identify disease targets in real time; Timing consistency verification unit: used to perform timing verification on disease targets in multiple consecutive frames, and eliminate transient false detections based on cross-frame Boolean voting results to confirm stable disease targets; Disease risk assessment unit: used to perform risk assessment and level classification based on the pixel coordinates of the disease target using the AHP-entropy weight method; Centimeter-level coordinate mapping unit: used to fuse the pixel coordinates of the stable disease target with the positioning and attitude data, and map them to the GNSS coordinate system or RTK coordinate system to achieve centimeter-level positioning; Cloud-based operation and maintenance management unit: Used to upload structured data containing disease information and location results to the cloud in real time, and automatically trigger alarms and maintenance dispatch processes based on risk level.

[0015] In some specific embodiments, the embedded GPU inference unit introduces a weighted loss function into the disease detection network based on Transformer and YOLOv8 to accurately identify small-target diseases. The weighted loss function is determined according to the following formula: in, The weighted GIoU loss, The loss function is the standard GIoU loss function, where w and h are the width and height of the target bounding box, respectively. A preset size threshold is used to distinguish small targets, and γ is a hyperparameter that adjusts the weighting.

[0016] In some specific embodiments, the embedded GPU inference unit includes: Visible light image enhancement subunit: used to enhance visible light images using the multi-scale Retinex algorithm; Fusion image generation subunit: used by the following formula The enhanced visible light image is fused with the infrared image to generate a fused image, where R( I RGB This is the enhanced visible light image. α The fusion coefficient is... I IR Infrared image, I F To merge images; Enhanced Image Determination Subunit: Used to perform contrast-limited adaptive histogram equalization on the fused image to determine the final enhanced image.

[0017] The beneficial effects of the above technical solution are as follows: This invention significantly improves the accuracy and environmental robustness of small target defect identification through multimodal image fusion and Transformer-YOLOv8 network; it achieves centimeter-level positioning by integrating GNSS / RTK and IMU; and it realizes full-process intelligentization from accurate detection and precise positioning to intelligent decision-making and automated operation and maintenance by combining AHP-entropy weight risk assessment and cloud closed-loop dispatching, greatly improving the efficiency and safety of railway inspection. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a method for real-time detection of track defects, provided as an embodiment of the present invention; Figure 2 A schematic diagram of a real-time detection device for track defects provided in an embodiment of the present invention; Figure 3This is a schematic diagram of the structure of a real-time detection device for track defects, provided as an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0021] Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar symbols denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the invention, and should not be construed as limiting the invention.

[0022] Example 1 One embodiment of the present invention provides a method for real-time detection of track defects, referring to... Figure 1 As shown, it includes: Step S1: Real-time acquisition of multimodal images and positioning / attitude data of the track area using a GPU detection device; In a specific embodiment of the present invention, step S1 includes: Step S11: Using a visible light camera and an infrared camera deployed on the GPU detection device, image sequences of the orbital region are acquired in real time under the synchronization of PPS pulse signals to obtain multimodal images of the orbital region; Step S12: Synchronously record the positioning and attitude data of the detection device when acquiring each frame of image through the GNSS / RTK module and the IMU module; wherein, the multimodal image includes visible light image and infrared image.

[0023] Step S2: Perform fusion and enhancement processing on the multimodal images to obtain an enhanced image; In one specific embodiment of the present invention, step S2 includes: Step S21: Enhance the visible light image using the multi-scale Retinex algorithm; Step S22: By the following formula The enhanced visible light image is fused with the infrared image to generate a fused image, where R( I RGB This is the enhanced visible light image. α The fusion coefficient is... I IR Infrared image, I F To merge images; Step S23: Perform contrast-limited adaptive histogram equalization on the fused image to determine the final enhanced image.

[0024] Step S3: Input the enhanced image into a disease detection network based on Transformer and YOLOv8 to identify disease targets in real time; In a specific embodiment of the present invention, in step S3, a weighted loss function is introduced into the disease detection network based on Transformer and YOLOv8 to accurately identify small target diseases. The weighted loss function is determined according to the following formula: in, The weighted GIoU loss, The loss function is the standard GIoU loss function, where w and h are the width and height of the target bounding box, respectively. A preset size threshold is used to distinguish small targets, and γ is a hyperparameter that adjusts the weighting.

[0025] Step S4: Perform time-series verification on the disease targets in multiple consecutive frames, and eliminate transient false detections based on the cross-frame Boolean voting results to confirm stable disease targets; In a specific embodiment of the present invention, step S4 includes: Step S41: For the same candidate target associated with the trajectory in multiple consecutive frames, calculate the average confidence of the candidate target as the result of cross-frame Boolean voting; Step S42: Compare the average confidence level with a preset threshold, and based on the cross-frame Boolean voting results, determine the candidate disease target as a stable disease target and output it, or remove it as a transient false detection.

[0026] Step S5: Based on the pixel coordinates of the diseased target, perform risk assessment and level classification using the AHP-entropy weight method; In one specific embodiment of the present invention, step S5 includes: Step S51: Construct multiple evaluation indicators based on the pixel coordinates of the diseased target; Step S52: Determine the combined weights of each evaluation index using the AHP-entropy weight method; Step S53: Determine the numerical range of the comprehensive risk index by combining the weights and the normalized quantitative values ​​of each indicator; Step S54: Based on the numerical range of the risk index, classify the disease into different risk levels.

[0027] Step S6: Fuse the pixel coordinates of the stable disease target with the positioning and attitude data, and map them to the GNSS coordinate system or RTK coordinate system to achieve centimeter-level positioning; In a specific embodiment of the present invention, step S6 includes: Step S61: Using camera intrinsic parameters, transform the pixel coordinates of the stable disease target to the camera coordinate system through the pinhole model; Step S62: Combining the extrinsic parameters of the camera and the vehicle body, transform the camera coordinates to the GNSS / RTK coordinate system; Step S63: Using the bundle adjustment algorithm, the image coordinates of the stable disease target and its corresponding positioning and attitude data from multiple time frames are fused together, and combined with IMU attitude data, the final geographic coordinates and route coordinates of the stable disease target are optimized to ensure that the planar positioning error is no more than 3 cm.

[0028] Step S7: Upload the structured data containing disease information and location results to the cloud in real time, and automatically trigger alarms and maintenance dispatch processes based on risk level.

[0029] In one specific embodiment of the present invention, step S7 includes: Step S71: Encapsulate the data into a JSON format message containing defect ID, category, location coordinates, and risk level, and upload it in real time via 5G / MQTT protocol; Step S72: After receiving the data, the cloud platform visualizes it on the GIS map, automatically generates maintenance work orders for high-risk defects, and pushes alarm information to maintenance personnel.

[0030] This embodiment effectively overcomes interference from complex lighting and weather conditions through multimodal image fusion and enhancement technology. Combined with a YOLOv8 network incorporating a Transformer module, it significantly enhances the feature perception and real-time identification capabilities for defects such as small-scale cracks, greatly reducing the false negative and false positive rates while maintaining a high frame rate. Furthermore, by tightly integrating visual inspection results with GNSS / RTK and IMU data, and employing a pinhole model and bundle adjustment algorithm, this embodiment precisely maps defect locations from pixel coordinates to geographic coordinates, achieving accurate spatial positioning with a planar error of no more than 3 cm, providing irreplaceable spatial information support for subsequent maintenance. In addition, this embodiment uses the AHP-entropy weight method to quantify the risk assessment and classify defects, and automatically triggers cloud alarms and maintenance dispatch processes based on risk levels. This upgrades the traditional discrete inspection mode into an integrated intelligent operation and maintenance system encompassing "detection-assessment-decision-action," fundamentally improving the decision-making efficiency and automation level of railway maintenance.

[0031] Example 2 One embodiment of the present invention provides a method for real-time detection of track defects, comprising: Multimodal images and positioning / attitude data of the track area are acquired in real time using a GPU detection device; Specifically, visible light and infrared cameras deployed on the GPU detection device acquire image sequences of the track surface in real time under the synchronization of PPS pulse signals; the spatial coordinates and attitude data of the detection device when acquiring each frame of image are recorded synchronously through the GNSS / RTK module and IMU module.

[0032] The multimodal images are fused and enhanced to obtain an enhanced image; Specifically, the multi-scale Retinex algorithm is used to enhance visible light images; through the following formula... The enhanced visible light image is fused with the infrared image to generate a fused image, where R( I RGB This is the enhanced visible light image. α The fusion coefficient is... I IR Infrared image, I F To create a fused image, a contrast-limited adaptive histogram equalization process is performed on the fused image to determine the final enhanced image.

[0033] Specifically, during the high-speed inspection of the inspection vehicle, the system synchronously acquires RGB images in real time. With infrared images Spatial registration is achieved through camera calibration parameters. The two types of images are fused using the multi-scale Retinex (MSR) algorithm. Among them, parameters The entropy optimization criterion is used to determine the final enhanced image, which is then enhanced using CLAHE enhancement. .

[0034] The enhanced image is input into a disease detection network based on Transformer and YOLOv8 to identify disease targets in real time; Enhanced image The detection network is based on a Transformer encoder and a lightweight YOLOv8 backbone, specifically as follows: First, the YOLOv8 backbone network extracts feature maps at different scales (such as...). ); Secondly, a multi-scale feature pyramid (FPN) is constructed to fuse features and obtain rich spatial context; Then, a Transformer encoder layer is introduced to compute self-attention weights to enhance the global feature representation of the defective target: In the formula, For query, key, and value characteristics, The scaling factor is used. After enhancing features through the Transformer encoder, the model ultimately predicts the defect bounding box. Defect categories and confidence level .

[0035] A weighted loss function is introduced for small targets, and the weighted loss function is determined according to the following formula: in, The weighted GIoU loss, The loss function is the standard GIoU loss function, where w and h are the width and height of the target bounding box, respectively. A preset size threshold is used to distinguish small targets, and γ is a hyperparameter that adjusts the weighting. This significantly improves the detection sensitivity for defects such as microcracks.

[0036] (3) Quantitative performance verification and comparison Table 2 shows the performance of the model of this invention and the comparison model on the self-built Rail-Defect-2024 test set (7532 frames).

[0037] The results show that the algorithm of this invention improves mAP and small target recall by 18% and 24.7% respectively, reduces false negative rate by >50%, and maintains real-time performance of >50fps, verifying the outstanding technical effect.

[0038] This application effectively overcomes the interference of complex environments such as lighting changes, rain, snow, and oil pollution by using multimodal image fusion and enhancement technology. By combining a lightweight YOLOv8 network with a Transformer module and a targeted small target loss function, it significantly improves the ability to perceive small-scale defects such as cracks and gaps, achieving a synergistic optimization of high recall and high accuracy, and reducing the false negative rate by more than half.

[0039] Temporal verification of disease targets in multiple consecutive frames is performed, and transient false detections are eliminated based on cross-frame Boolean voting results to confirm stable disease targets; Specifically, for the same candidate target associated with the trajectory in multiple consecutive frames, the average confidence level of the candidate target is calculated; if the average confidence level is greater than or equal to a preset value, the target is determined to be a real disease and output; otherwise, it is discarded.

[0040] Specifically, the confidence level of the detected target in consecutive frames. Smoothing is performed, and cross-frame Boolean voting is used to determine the authenticity of the defect: Among them, threshold The ROC curve is optimized to ensure that the output is only confirmed for targets that appear stably across multiple frames, thus reducing false alarms.

[0041] The following is a specific case study (the numbers are taken from a typical excerpt of a real experimental log and have been anonymized) demonstrating the effect and result judgment process of "temporal consistency verification" in continuous frame crack detection: 1. Scenarios and Data Line section: A certain high-speed railway K247+800~K247+810 Inspection speed: 7 km·h -1 Sampling frame rate: 50 fps Detection window width: T = 5 frames Threshold selection: θ = 0.70 (determined by the optimal midpoint of the ROC curve) 2. Original detection sequence Note: IDs are generated based on trajectory association; if a frame is not detected, it is recorded as 0.

[0042] 3. Boolean voting calculation 3.1 Crack Candidate A (ID: C-001) 3.2 Noise Candidate (ID: C-002) 4. Effect Comparison The results show that the timing consistency check effectively eliminated the transient noise C-002 while maintaining the output of the real crack C-001, thereby significantly reducing false alarms and improving overall accuracy.

[0043] In summary, the system's average confidence level for the crack target C-001 within a 5-frame window was 0.736, higher than the threshold θ=0.70, therefore it was determined to be a genuine defect. The average confidence level for the reflective noise C-002, which only occasionally occurred in frames 1501 and 1503, was 0.168, lower than the threshold, and was therefore discarded. Based on 20,000 frames of experimental data, Boolean voting reduced the false alarm rate from 5.6% to 2.2%, improving precision by approximately 5.5%.

[0044] Step S5: Based on the pixel coordinates of the diseased target, perform risk assessment and level classification using the AHP-entropy weight method; Specifically, multiple evaluation indicators are constructed based on the pixel coordinates of the disease target; the combined weight of each evaluation indicator is determined using the AHP-entropy weight method; the numerical range of the comprehensive risk index is determined by the combined weight and the normalized quantified value of each indicator; and the disease is divided into different risk levels according to the numerical range of the risk index.

[0045] Specifically, a comprehensive risk score is calculated based on the characteristics of the target being detected: in, For the quantitative indicators of crack length and area, weights Determined by the AHP-entropy weight method. Based on the scoring results, defects are classified into general damage, Class I damage (minor), and Class II damage (serious) risk to guide subsequent maintenance decisions.

[0046] Below is a specific case study on disease risk classification (taken from confirmed crack defects in actual inspection records, with numerical values ​​anonymized), demonstrating how to apply the comprehensive scoring formula. Calculate the risk index, determine the risk level, and generate subsequent maintenance recommendations.

[0047] Scoring Indicators and Weighting Table 2. Detection Instance Data 3. Calculation of Comprehensive Risk Index 4. Risk Level Determination Set level threshold: Normal (general): R < 0.35 Grade I (Slight): 0.35 ≤ R < 0.60 Level II (Severe): R ≥ 0.60 The instance has an R value of 0.78, which is greater than or equal to 0.60. Therefore, it is classified as a Level II (Severe) risk.

[0048] 5. Maintenance Decision Example In summary, the system detected cracks in the high-speed rail track slab of a certain section, with quantitative indicators (q1,q2,q3,q4,q5) = (0.80, 0.60, 1.00, 0.90, 0.40). Based on the formula... R=0.78, which is higher than the Level II threshold of 0.60, indicating a serious risk and automatically generating a maintenance work order M2024-0615-02. On-site re-inspection confirmed the existence of the crack, verifying the accuracy and timeliness of the risk classification.

[0049] Step S6: Fuse the pixel coordinates of the stable disease target with the positioning and attitude data, and map them to the GNSS coordinate system or RTK coordinate system to achieve centimeter-level positioning; Specifically, the pixel coordinates of the stable disease target are transformed to the camera coordinate system using the camera intrinsic parameters through the pinhole model; the camera coordinates are transformed to the GNSS / RTK coordinate system by combining the extrinsic parameters of the camera and the device; and the final geographic coordinates and route coordinates of the stable disease target are optimized by fusing multi-frame observation data and IMU attitude data through the bundle adjustment algorithm to ensure that the planar positioning error is no more than 3 cm.

[0050] Detected defect image coordinates Mapping to the railway line spatial coordinate system using pinhole imaging and beam adjustment algorithms: It combines RTK-GNSS and IMU real-time positioning data to achieve centimeter-level positioning accuracy, and outputs the mileage, offset and elevation of the track location where the defect is located.

[0051] Specifically, the following is a real-world example of centimeter-level geographic coordinate mapping (taken from the K247 section of the Beijing-Shanghai High-Speed ​​Railway on May 12, 2024, with parameters anonymized), illustrating the formula implementation process and final error assessment.

[0052] 1. Camera calibration and external parameters The top surface of the orbit is approximated as flat.

[0053] 2. Pixel coordinates → Camera coordinates Detected defect box center Depth Solution Camera coordinates 3. Camera coordinates → Orbital world coordinates That is, on the track plane (Z≈0), the plane coordinates of the defect relative to the camera projection point are (X≈0.022m, Y≈-0.080m), which means it is about 22mm to the right of the track centerline and about 81mm forward.

[0054] 4. Integrating RTK-GNSS and odometer readings When the frame RTK coordinates are: Track inspection vehicle mileage markers: Will Translate to the GNSS reference point and project onto the line centerline to obtain: mileage Offset Elevation 5. Accuracy Verification The coordinates of the same point were measured using a trackside laser total station: The measured error is 2.8cm, which meets the centimeter-level positioning requirement of "≤3cm".

[0055] In summary, the defect pixel coordinates (1350, 820) are mapped to (X, Y) = (0.022m, -0.080m) using the formula. Combined with the RTK-GNSS output mileage of 247586.34m, offset of -0.022m, and elevation of 8.97m, and compared with the laser total station reference, the plane error is 2.8cm, verifying the accuracy and feasibility of the centimeter-level coordinate mapping module. This application integrates visual inspection results with GNSS / RTK and IMU data, and uses a pinhole model and bundle adjustment algorithm to accurately map the defect location from image pixel coordinates to the railway line coordinate system. The planar positioning accuracy reaches the centimeter level (≤3 cm), providing irreplaceable spatial information support for subsequent precise maintenance.

[0056] Step S7: Upload the structured data containing disease information and location results to the cloud in real time, and automatically trigger alarms and maintenance dispatch processes based on risk level.

[0057] Specifically, the data is encapsulated into a JSON format message containing defect ID, category, location coordinates, and risk level, and uploaded in real time via the 5G / MQTT protocol. After receiving the data, the cloud platform visualizes it on a GIS map, automatically generates maintenance work orders for high-risk defects, and pushes alarm information to maintenance personnel.

[0058] Confirmed test results are uploaded to the cloud platform in real time in a structured data packet format. The data format includes: defect number, category, location, size, confidence level, risk level, location coordinates, and image index information. The cloud platform automatically triggers SMS / APP alarms for high-risk defects, notifying relevant maintenance personnel to handle them in a timely manner, while also providing a GIS map visualization interface.

[0059] The following is a complete case study of "Result Feedback and Cloud Alarms", which includes three levels: (A) JSON message uploaded by the vehicle, (B) records stored in the cloud, and (C) automatic alarm content. A. Example of JSON message uploaded by the vehicle { "ID": "DEF-20240615-000512", Class: "Crack", "Position": { / / Orbital coordinates (centimeter level) "s": 247586.344, / / Mileage (m) "d_y": -0.022, / / Offset (m, positive on the right) "h": 8.972 / / Elevation (m) }, "Size": { / / Defect size "length_mm": 32.0, "width_mm": 1.2 }, "Confidence": 0.92, / / Detection confidence level P "RiskIndex": 0.78, / / Overall Risk R "RiskLevel": "II", / / I / II / II "ImgRef": "s3: / / rail-inspection / 2024 / 06 / 15 / 154302_rgb.jpg", "Timestamp": "2024-06-15T15:43:02.187Z" } ID: Consists of defect type abbreviation + date + serial number.

[0060] Class: Defect category (Crack, Spalling, Bolt_Missing, etc.).

[0061] Position: Uses mileage-offset-elevation format; derived from centimeter-level coordinate mapping.

[0062] Size: Original physical quantities such as length and width, which are used for backend re-evaluation.

[0063] Confidence / RiskIndex / RiskLevel: These correspond to the detection confidence level P, overall risk R, and dispersion level, respectively.

[0064] ImgRef: The storage path of the original or annotated image corresponding to the defect, which is convenient for manual review.

[0065] Timestamp: ISO-8601 format, unified time zone is UTC.

[0066] The messages are uploaded in real time via the MQTT topic rail / defect / json at QoS=1 level; if the 5G link fails, the messages are cached on the local SSD and resent in batches after the link is restored.

[0067] B. Cloud database import records (fragment from the defect_info table) After being stored in the database, the backend service automatically pushes the records to the GIS visualization layer, where defects are marked with a flashing red color on the route map; at the same time, they are written to the alert_queue for alarm services.

[0068] C. Automatic Alarm Content SMS / App push template [Intelligent Track Inspection Cloud] Level II Crack Alarm Route: A certain high-speed railway Mileage: K247+586.3 Offset: -22 mm Elevation: +8.97 m Dimensions: 32.0 mm × 1.2 mm Confidence level: 92% Please schedule a follow-up inspection within 24 hours. The work order has been automatically generated: #M20240615-02 Work order information (simplified form) Work order information (simplified form) Maintenance personnel can directly view the original defect image, location navigation, and historical comparison data by clicking on the work order in the App. After completing the repair, they can upload a review photo, and the closed-loop status will automatically update to "Reviewed".

[0069] In summary, crack defect DEF-20240615-000512 was detected by the testing device and uploaded to the cloud in the aforementioned JSON format. The system identified its risk level as Level II, automatically generated SMS and App alarms, and dispatched work order M20240615-02. Maintenance personnel completed the repair within 24 hours, and the residual error after verification was 2.1cm, verifying the real-time performance and closed-loop effectiveness of the result feedback and cloud alarm module.

[0070] From automatic defect identification and risk assessment to cloud-based alarms and maintenance dispatch, this invention streamlines the entire process of "detection-assessment-decision-action," upgrading the traditional discrete inspection model into an integrated intelligent operation and maintenance system. This completely changes the current situation of reliance on manual labor and information silos, significantly improving the automation and intelligence level of railway operation and maintenance. Furthermore, by integrating multimodal images, the Transformer-YOLOv8 deep learning architecture, and GNSS / IMU high-precision positioning, this invention achieves, for the first time on a mobile detection device platform, an integrated system for track defect detection, from "real-time accurate identification" to "centimeter-level spatial positioning" and then to "automated operation and maintenance closed loop." This fundamentally solves the core pain points of existing technologies, such as high missed detection rates for small targets, poor environmental robustness, insufficient positioning accuracy, and disconnect from the operation and maintenance process, significantly improving the intelligence level and maintenance efficiency of railway inspection.

[0071] Example 3 This embodiment provides a real-time detection device for track defects, referring to... Figure 2 As shown, it includes: Multimodal synchronous acquisition unit 10: used to acquire multimodal images and positioning and attitude data of the track area in real time through GPU detection device; Image enhancement preprocessing unit 20: used to perform fusion and enhancement processing on the multimodal image to obtain an enhanced image; Embedded GPU inference unit 30: used to input the enhanced image into a disease detection network based on Transformer and YOLOv8 to identify disease targets in real time; Timing consistency verification unit 40: used to perform timing verification on disease targets in multiple consecutive frames, and eliminate transient false detections based on cross-frame Boolean voting results to confirm stable disease targets; Disease risk assessment unit 50: used to perform risk assessment and level classification based on the pixel coordinates of the disease target using the AHP-entropy weight method; Centimeter-level coordinate mapping unit 60: used to fuse the pixel coordinates of the stable disease target with the positioning and attitude data, and map them to the GNSS coordinate system or RTK coordinate system to achieve centimeter-level positioning; Cloud-based operation and maintenance management unit 70: Used to upload structured data containing disease information and location results to the cloud in real time, and automatically trigger alarms and maintenance dispatch processes based on risk level.

[0072] In one specific embodiment of the present invention, in the embedded GPU inference unit 30, a weighted loss function is introduced into the disease detection network based on Transformer and YOLOv8 to accurately identify small target diseases. The weighted loss function is determined according to the following formula: in, The weighted GIoU loss, The loss function is the standard GIoU loss function, where w and h are the width and height of the target bounding box, respectively. A preset size threshold is used to distinguish small targets, and γ is a hyperparameter that adjusts the weighting.

[0073] In one specific embodiment of the present invention, the embedded GPU inference unit 30 includes: Visible light image enhancement subunit: used to enhance visible light images using the multi-scale Retinex algorithm; Fusion image generation subunit: used by the following formula The enhanced visible light image is fused with the infrared image to generate a fused image, where R( I RGB This is the enhanced visible light image. α The fusion coefficient is... I IR Infrared image, I F To merge images; Enhanced Image Determination Subunit: Used to perform contrast-limited adaptive histogram equalization on the fused image to determine the final enhanced image.

[0074] Example 4 This invention also provides an electronic device, with reference to... Figure 3 As shown, it includes a processor and a memory, the memory storing a computer program. When the computer program is executed by the processor, it implements the steps of a real-time detection method for track defects as described in any embodiment.

[0075] Example 5 This invention also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of a real-time detection method for track defects as described in any embodiment.

[0076] Example 6 This invention also provides a computer program that, when executed by a processor, implements the steps of a real-time detection method for track defects as described in any embodiment.

[0077] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0078] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. The embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the functions specified in one or more boxes. Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the invention. Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0079] The methods and apparatus provided by the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

[0080] In the description of this specification, references to terms such as "an embodiment," "some embodiments," "example," "specific example," or "a specific embodiment" or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for real-time detection of track defects, characterized in that, include: Step S1: Real-time acquisition of multimodal images and positioning / attitude data of the track area using a GPU detection device; Step S2: Perform fusion and enhancement processing on the multimodal images to obtain an enhanced image; Step S3: Input the enhanced image into a disease detection network based on Transformer and YOLOv8 to identify disease targets in real time; Step S4: Perform time-series verification on the disease targets in multiple consecutive frames, and eliminate transient false detections based on the cross-frame Boolean voting results to confirm stable disease targets; Step S5: Based on the pixel coordinates of the diseased target, perform risk assessment and level classification using the AHP-entropy weight method; Step S6: Fuse the pixel coordinates of the stable disease target with the positioning and attitude data, and map them to the GNSS coordinate system or RTK coordinate system; Step S7: Upload the structured data containing disease information and location results to the cloud in real time, and automatically trigger alarms and maintenance dispatch processes based on risk level.

2. The method for real-time detection of track defects according to claim 1, characterized in that, Step S1 includes: Step S11: Using a visible light camera and an infrared camera deployed on the GPU detection device, image sequences of the orbital region are acquired in real time under the synchronization of PPS pulse signals to obtain multimodal images of the orbital region; Step S12: Synchronously record the positioning and attitude data of the detection device when acquiring each frame of image through the GNSS / RTK module and the IMU module; wherein, the multimodal image includes visible light image and infrared image.

3. The method for real-time detection of track defects according to claim 1, characterized in that, Step S2 includes: Step S21: Enhance the visible light image using the multi-scale Retinex algorithm; Step S22: By the following formula The enhanced visible light image is fused with the infrared image to generate a fused image, where R( I RGB This is the enhanced visible light image. α The fusion coefficient is... I IR Infrared image, I F To merge images; Step S23: Perform contrast-limited adaptive histogram equalization on the fused image to determine the final enhanced image.

4. The method for real-time detection of track defects according to claim 1, characterized in that, In step S3, a weighted loss function is introduced into the disease detection network based on Transformer and YOLOv8 to accurately identify small-target diseases. The weighted loss function is determined according to the following formula: in, The weighted GIoU loss, The loss function is the standard GIoU loss function, where w and h are the width and height of the target bounding box, respectively. A preset size threshold is used to distinguish small targets, and γ is a hyperparameter that adjusts the weighting.

5. The method for real-time detection of track defects according to claim 1, characterized in that, Step S4 includes: Step S41: For the same candidate target associated with the trajectory in multiple consecutive frames, calculate the average confidence of the candidate target as the result of cross-frame Boolean voting; Step S42: Compare the average confidence level with a preset threshold, and based on the cross-frame Boolean voting results, determine the candidate disease target as a stable disease target and output it, or remove it as a transient false detection.

6. The method for real-time detection of track defects according to claim 1, characterized in that, Step S5 includes: Step S51: Construct multiple evaluation indicators based on the pixel coordinates of the diseased target; Step S52: Determine the combined weights of each evaluation index using the AHP-entropy weight method; Step S53: Determine the numerical range of the comprehensive risk index by combining the weights and the normalized quantitative values ​​of each indicator; Step S54: Based on the numerical range of the risk index, classify the disease into different risk levels.

7. The method for real-time detection of track defects according to claim 1, characterized in that, Step S6 includes: Step S61: Using camera intrinsic parameters, the pixel coordinates of the stable disease target are transformed to the camera coordinate system through the pinhole model; Step S62: Combining the extrinsic parameters of the camera and the vehicle body, transform the camera coordinates to the GNSS / RTK coordinate system; Step S63: Using the bundle adjustment algorithm, the image coordinates of the stable disease target and its corresponding positioning and attitude data from multiple time frames are fused together, and combined with IMU attitude data, the final geographic coordinates and route coordinates of the stable disease target are optimized to ensure that the planar positioning error is no more than 3 cm.

8. A real-time detection device for track defects, characterized in that, include: Multimodal synchronous acquisition unit: used to acquire multimodal images and positioning and attitude data of the track area in real time through GPU detection device; Image enhancement preprocessing unit: used to perform fusion and enhancement processing on the multimodal image to obtain an enhanced image; Embedded GPU inference unit: used to input the enhanced image into a disease detection network based on Transformer and YOLOv8 to identify disease targets in real time; Timing consistency verification unit: used to perform timing verification on disease targets in multiple consecutive frames, and eliminate transient false detections based on cross-frame Boolean voting results to confirm stable disease targets; Disease risk assessment unit: used to perform risk assessment and level classification based on the pixel coordinates of the disease target using the AHP-entropy weight method; Centimeter-level coordinate mapping unit: used to fuse the pixel coordinates of the stable disease target with the positioning and attitude data, and map them to the GNSS coordinate system or RTK coordinate system to achieve centimeter-level positioning; Cloud-based operation and maintenance management unit: Used to upload structured data containing disease information and location results to the cloud in real time, and automatically trigger alarms and maintenance dispatch processes based on risk level.

9. The real-time detection device for track defects according to claim 8, characterized in that, In the embedded GPU inference unit, a weighted loss function is introduced into the disease detection network based on Transformer and YOLOv8 to accurately identify small-target diseases. The weighted loss function is determined according to the following formula: in, The weighted GIoU loss, The loss function is the standard GIoU loss function, where w and h are the width and height of the target bounding box, respectively. A preset size threshold is used to distinguish small targets, and γ is a hyperparameter that adjusts the weighting.

10. The real-time detection device for track defects according to claim 8, characterized in that, The embedded GPU inference unit includes: Visible light image enhancement subunit: used to enhance visible light images using the multi-scale Retinex algorithm; Fusion image generation subunit: used by the following formula The enhanced visible light image is fused with the infrared image to generate a fused image, where R( I RGB This is the enhanced visible light image. α The fusion coefficient is... I IR Infrared image, I F To merge images; Enhanced Image Determination Subunit: Used to perform contrast-limited adaptive histogram equalization on the fused image to determine the final enhanced image.

Images