Subway tunnel unmanned aerial vehicle inspection system

Abstract

A subway tunnel unmanned aerial vehicle inspection system disclosed by the present invention comprises a flight platform, a depth camera, a binocular camera, a 3D laser radar, a TOF laser radar, a 4K holder, a crack analysis module and a track following flight module, the depth camera is used for infrared track identification, the binocular camera is used for providing a visual positioning function and collaborative path planning, and the 3D laser radar is used for 3D laser radar. The 3D laser radar is used for generating three-dimensional point cloud data of a tunnel environment and performing real-time mapping, the crack analysis module marks cracks based on a crack analysis algorithm when receiving environment real-time acquisition videos or pictures, and the track following flight module controls the course of the unmanned aerial vehicle based on rail features acquired by the depth camera. Through a multi-sensor fusion technology, a high-precision flight control algorithm and an intelligent path planning method, autonomous navigation, obstacle avoidance and inspection tasks of the unmanned aerial vehicle in a tunnel environment are realized, and the method is suitable for scenes of subway tunnel structure health monitoring, equipment state detection, potential safety hazard investigation and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent tunnel inspection, and in particular to a subway tunnel unmanned aerial vehicle inspection system. Background Art

[0002] As a vital component of urban rail transit, the structural health of subway tunnels is directly related to operational safety and efficiency. Traditional manual inspection methods are limited by the narrow space, low light conditions, and complex obstacle distribution within tunnels, resulting in low inspection efficiency, significant safety hazards, and high operational difficulty. In recent years, with the rapid development of drone technology, the use of drones for automated tunnel inspections has gradually become a trend. However, existing tunnel inspection drone systems still face the following technical bottlenecks in practical application: Insufficient flight stability: The environment inside subway tunnels is complex, with factors such as electromagnetic interference and airflow disturbances, which lead to poor flight stability of drones and difficulty in achieving precise control.

[0003] Limited positioning accuracy: GPS signals are denied in tunnels. Existing drones mostly rely on visual SLAM (Simultaneous Localization and Mapping) or inertial navigation systems for positioning. However, in long-distance, low-texture tunnel environments, the problem of accumulated positioning errors is prominent, making it difficult to meet high-precision inspection requirements.

[0004] Poor adaptability to low-light environments: The lighting conditions in tunnels are poor, and the performance of existing visual sensors and image processing algorithms degrades significantly in low-light environments, resulting in insufficient accuracy in tasks such as crack identification and equipment status detection.

[0005] Insufficient real-time data processing and transmission capabilities: The existing system has delays in real-time data collection, processing and return, making it difficult to meet the efficiency and real-time requirements of tunnel inspections.

[0006] To address the above problems, there is an urgent need to develop an intelligent inspection drone system that can achieve stable flight, high-precision positioning, real-time crack detection and data feedback in complex tunnel environments, so as to improve the automation level and operational efficiency of subway tunnel inspections. Summary of the Invention

[0007] The main purpose of the present invention is to provide a subway tunnel drone inspection system to solve the above technical problems.

[0008] To achieve the above objectives, the present invention provides a subway tunnel drone inspection system.

[0009] The subway tunnel drone inspection system includes a flight platform, a depth camera, a binocular camera, a 3D laser radar, a TOF laser radar, a 4K gimbal, a crack analysis module, and a track-following flight module. The depth camera is used to identify infrared tracks, the binocular camera is used to provide visual positioning functions and collaborative path planning, the 3D laser radar is used to generate three-dimensional point cloud data of the tunnel environment and perform real-time mapping. When the crack analysis module receives real-time video or pictures of the environment, it marks the cracks based on the crack analysis algorithm. The track-following flight module controls the heading of the drone based on the rail features collected by the depth camera.

[0010] In one embodiment, the crack analysis module converts the acquired 4K image from RGB to HSV color space based on the crack analysis algorithm, detects the image, locates the deformation area where the crack is located, and then marks the crack position and type on the original image.

[0011] In one embodiment, the track following flight module uses the YOLOv8 model to identify the features of two rails in the tunnel, extracts them, and calculates the offset between the drone and the center line of the track to adjust the heading of the drone.

[0012] In one embodiment, the roll angle of the drone is adjusted according to the following formula: .

[0013] In one embodiment, the yaw rate of the drone is adjusted according to the following formula: .

[0014] In one embodiment, the heading attitude of the UAV is updated based on the EKF; Define the state vector Used to represent the drone's position, velocity, and sensor bias: ; in: is the three-dimensional position of the drone; It’s speed; is a quaternion, representing attitude (heading, pitch, roll); are the accelerometer and gyroscope biases, respectively.

[0015] In one embodiment, the position and velocity of the drone are calculated based on the following formula: ; in: is the input (IMU readings, including acceleration and angular velocity); is the process noise (used to simulate the error); It is the state transfer equation that describes the UAV motion model.

[0016] The beneficial effects that can be achieved by the present invention: An embodiment of the present invention proposes a subway tunnel drone inspection system. This application uses multi-sensor fusion technology, high-precision flight control algorithm and intelligent path planning method to realize the autonomous navigation, obstacle avoidance and inspection tasks of the drone in the tunnel environment. It is suitable for scenarios such as subway tunnel structural health monitoring, equipment status detection and safety hazard investigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a schematic diagram of the structure of a subway tunnel drone inspection system according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the software system of the subway tunnel drone inspection system according to an embodiment of the present invention; Figure 3 It is a schematic diagram of system integration and data return according to an embodiment of the present invention.

[0018] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0019] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0020] like Figure 1 As shown, the subway tunnel drone inspection system includes a flight platform, a depth camera, a binocular camera, a 3D laser radar, a TOF laser radar, a 4K gimbal, a crack analysis module and a track following flight module mounted on the flight platform. The depth camera is used to identify infrared tracks, the binocular camera is used to provide visual positioning functions and collaborative path planning, the 3D laser radar is used to generate three-dimensional point cloud data of the tunnel environment and perform real-time mapping. When the crack analysis module receives real-time video or pictures of the environment, it marks the cracks based on the crack analysis algorithm. The track following flight module controls the heading of the drone based on the rail features collected by the depth camera.

[0021] In this application, the system is equipped with multimodal sensors, including a D435i depth camera, a T265 binocular camera, and a 3D lidar. High-precision positioning and path planning are achieved through visual data fusion using an EKF (Extended Kalman Filter). In this system, the binocular camera achieves precise local relative positioning, and the depth data provided by the depth camera is used to correct the positioning results in real time. The lidar collects the overall point cloud to construct a 3D tunnel model. The data from each sensor is layered and collaboratively fused to generate a drone visual information matrix that combines local positioning with global environmental perception. This provides solid data support for path planning and real-time obstacle avoidance, thereby achieving high-precision positioning and path planning.

[0022] 3D LiDAR is also used to construct a real-time 3D point cloud map of the tunnel environment. Time of Flight (TOF) LiDAR is combined with a high-precision odometer to provide centimeter-level positioning accuracy. This ensures the drone's flight stability in complex tunnel environments and supports real-time path updates in dynamic environments. Crack Detection and Data Feedback: Utilizing deep learning algorithms, the front-end system identifies structural issues such as cracks and deformations within the tunnel in real time, captures 4K video and fixed-distance photos, and transmits the data back to the back-end via 4G in real time for analysis.

[0023] Specifically, the system integrates a deep learning-based object detection algorithm, enabling real-time front-end identification of structural defects such as cracks and deformations within tunnels. First, the input 4K image is converted from RGB to HSV color space (using the Python colorsys module) to more intuitively distinguish the hue and saturation differences between the tunnel surface and defective areas. Using a YOLOv8 model trained on a large number of tunnel crack and deformation sample images (using a pre-set training dataset), the system performs preliminary image detection, quickly locating potential cracks and deformation areas. Cracked areas are then identified and labeled "cracks" within the image.

[0024] The data is then transmitted back to the backend platform in real time via the 4G / 5G network, enabling immediate processing and visualization of the inspection data. Furthermore, the system is equipped with a 4K high-definition camera, which supports fixed-distance shooting of high-resolution photos and videos. The data is then transmitted back to the backend platform in real time via the 4G / 5G network for comparative analysis and storage, enabling immediate processing and visualization of the inspection data.

[0025] In this application, the flight platform uses a six-rotor flight platform with a wheelbase of 850mm, which provides high stability and carrying capacity and is suitable for operations in narrow tunnel environments.

[0026] Sensor system: D435i Depth Camera: Used for infrared track recognition to achieve precise track following flight.

[0027] T265 binocular camera: Provides visual positioning capabilities and collaborates with the flight control system for path planning and stability control.

[0028] 3D LiDAR: Generates high-precision 3D point cloud data of tunnel environments, facilitating real-time mapping and obstacle detection.

[0029] TOF LiDAR: Provides high-precision odometer data to enhance positioning accuracy.

[0030] 4K PTZ: Used for high-definition shooting of structural conditions inside tunnels to ensure image quality.

[0031] High-brightness fill light: ensures clear images in low-light environments.

[0032] Please refer to Figure 2-3 The flight platform is equipped with PX4 firmware and MAVROS: PX4 provides the flight control basis, and MAVROS is used for data interaction between the drone and the ground control system.

[0033] ROS1 and Ubuntu 20.04: As a development platform, it provides sensor data drive, path planning and algorithm implementation. The track following flight module and crack analysis module are both installed on it.

[0034] A variant of the FastLIO algorithm combines data from lidar and visual sensors to achieve high-precision path planning and dynamic obstacle avoidance in tunnel environments. The system uses the visual data matrix to generate a high-density 3D point cloud image and combines it with IMU data to provide an initial pose estimate. During the preprocessing phase, the point cloud is denoised, and feature extraction and edge detection are performed on the image to ensure data quality. Next, a tightly coupled sensor fusion framework is used to match lidar data with visual features for more accurate real-time positioning.

[0035] Specifically, Fast-LIO provides high-precision position and orientation, the IMU provides high-frequency motion status, the depth camera provides additional environmental depth information, and the lidar acquires 3D point cloud information. This data is fused using an extended Kalman filter (EKF). Based on the sensor data, a gradient optimization and local obstacle avoidance strategy are used to fine-tune the path: Establish a local objective function: ; Indicates the deviation between the new trajectory and the original trajectory; Indicates the distance between the new trajectory and the obstacle; is a weight factor that can adjust the optimization priority; Optimization using gradient descent: Calculate the deviation from the current position to the target trajectory, and use gradient descent (continuously correct the trajectory points through numerical changes to make them as close as possible to the target path and far from obstacles) to optimize the position of the points. Combine the high-frequency state feedback of Fast-LIO to enable the UAV to adjust its flight attitude more smoothly, thereby completing obstacle avoidance and route fine-tuning.

[0036] Generate the planned path with Ego-Planner: Take the current position of the aircraft as the starting point and the target point as the ending point, construct a local search space, and use gradient descent to find an obstacle-free path. Obtain the current point cloud through Fast-LIO, and determine whether there are obstacles on the current path. If a collision is detected, trigger path replanning.

[0037] Pseudocode: (1) Obstacle detection def detect_obstacle(lidar_data, threshold=0.5): # Extract point cloud data point_cloud = process_lidar_data(lidar_data) # Determine if there are obstacles for point in point_cloud: if point.distance < threshold: return True # There are obstacles return False ((END)) Generate an obstacle avoidance path def replan_path(current_pos, goal_pos, map): # Calculate the path using A* / RRT* path = compute_path_Astar(current_pos, goal_pos, map) # Smooth with B-spline smoothed_path = bspline_smooth(path) return smoothed_path (3) Path execution while True: if detect_obstacle(lidar_data): new_path = replan_path(drone.position, target_position, map) execute_path(new_path) This invention utilizes an improved FastLIO (Fast Lightweight and Inertial Odometry) algorithm, combining inertial measurement unit (IMU) and lidar data. Fast-LIO provides high-precision position and attitude (position + attitude), the IMU provides high-frequency motion status, a depth camera provides environmental details, and the lidar acquires 3D point cloud information to identify obstacles. Combined with the ego-planner algorithm, this algorithm fine-tunes the flight path in real time, anticipates the dynamic changes of potential obstacles, and uses optimization strategies to rapidly generate new obstacle avoidance paths within the local search space. Ultimately, this achieves high-precision path planning and real-time obstacle avoidance. This algorithm ensures stable flight along a planned trajectory in complex tunnel environments while dynamically avoiding obstacles, significantly improving flight safety and reliability.

[0038] The crack detection algorithm, implemented in the crack analysis module, is optimized based on the YOLOv8 (You Only Look Once version 8) object detection framework. It achieves high-precision crack detection through the following technical solutions: YOLOv8 weights pre-trained on the COCO dataset are used as initial parameters to accelerate model convergence. The default CSPDarknet53 backbone network in YOLOv8 is used to extract multi-scale features. Feature Pyramid Network (FPN) and Path Aggregation Network (PAN) are used to fuse feature maps from different levels to enhance the detection of small cracks. The detection head outputs a predicted bounding box and its confidence score, with a screening threshold set to 0.5 (adjustable). Crack locations and types (e.g., horizontal and vertical cracks) are annotated on the original image.

[0039] The track-following flight algorithm, integrated into the track-following flight module, relies on real-time imagery captured by the drone's downward-looking camera. Using a trained YOLOv8 model, it accurately identifies the features of the two rails in the tunnel and achieves stable flight control. The drone's downward-looking camera continuously captures images of the tunnel track. After simple processing (such as denoising and contrast enhancement), the YOLOv8 model, fine-tuned with tunnel track samples, detects the two rails in real time, extracting their positional information and geometric features.

[0040] This model identifies the continuity and integrity of the track, ensuring no interruptions or anomalies during flight. Based on the detection results, the pixel offset between the drone and the track centerline is calculated, using the image centerline as a reference, and converted to actual distance (requiring a pixel-to-actual scale factor).

[0041] The yaw angle error is calculated based on the difference between the slope of the track centerline and the current heading angle of the drone.

[0042] Lateral control: Adjust the roll angle of the drone according to the lateral deviation. Formula: ; Heading control: Adjust the yaw rate (Yaw Rate) according to the heading error. The formula is: .

[0043] In the above embodiment, a state vector is defined in the data fusion stage , used to represent the drone's position, velocity, and sensor deviation, usually including:

[0044] in: is the three-dimensional position of the drone; It’s speed; is a quaternion, representing attitude (heading, pitch, roll); are the accelerometer and gyroscope biases, respectively.

[0045] In the prediction phase, the EKF uses IMU data to infer the drone’s position and velocity, and makes predictions based on a discrete state transition model:

[0046] in: is the input (IMU readings, including acceleration and angular velocity); is the process noise (used to simulate the error); It is the state transfer equation that describes the UAV motion model.

[0047] Position and velocity updates:

[0048]

[0049] in: It is composed of quaternions The calculated rotation matrix; is the linear acceleration provided by the IMU; is the bias of the accelerometer; is the acceleration noise.

[0050] Status Update:

[0051]

[0052] in: is the observation residual; pass Modifying state variables .

[0053] pseudocode: import numpy as npdef predict_state(x, u, dt): """ EKF prediction phase, state estimation based on IMU data""" p, v, q, ba, bg = x[:3], x[3:6], x[6:10], x[10:13],x[13:16] # Calculate new position and velocity R = quaternion_to_rotation_matrix(q) p_new = p +v * dt + 0.5 * R @ (u[:3] - ba) * dt**2 v_new = v + R @ (u[:3] - ba) * dt # Posture update q_new = quaternion_update(q, u[3:6] - bg, dt) return np.hstack((p_new, v_new, q_new, ba, bg))def update_state(x, P, z, H, R): """ EKF update phase, fusion of lidar or VIO data""" K = P @ HT @ np.linalg.inv(H @ P @ HT + R) x_new = x + K @ (z - H @ x) P_new = (np.eye(len(x)) - K @ H) @ P return x_new,P_new This application was tested in a dimly lit environment in an underground garage, simulating a subway tunnel. Multiple waypoints were set up in the test area to simulate subway stations. The drone took off autonomously from the pre-set takeoff point, achieving high-precision positioning and autonomous path planning based on the improved FastLIO algorithm. The specific implementation steps are as follows: Autonomous takeoff and positioning: After the drone is launched, it uses multi-sensor data fusion from the D435i depth camera, T265 binocular camera, and 3D lidar, combined with an improved FastLIO algorithm, to calculate its own position in real time and build an environmental map, achieving centimeter-level positioning accuracy.

[0054] Path planning and autonomous flight: The system generates the optimal flight path based on preset waypoints. The drone flies autonomously along the planned path, dynamically avoiding obstacles during the process to ensure flight stability and safety.

[0055] Environmental perception and data collection: The drone is equipped with a 4K high-definition gimbal camera to capture real-time images of the environment during flight, and transmit video streams and sensor data back to the terminal platform via 4G / 5G networks.

[0056] Crack detection and report generation: The terminal platform uses deep learning algorithms to perform real-time analysis of the returned image data, identify structural defects such as cracks and deformations on the tunnel wall, and automatically generate inspection reports including information such as defect location, size, and severity.

[0057] The test results show that the drone can stably complete autonomous flight and inspection tasks in a simulated tunnel environment, with a positioning accuracy of ±2cm and a crack detection accuracy of over 95%, fully verifying the reliability and practicality of the system.

[0058] Scenario 2: Autonomous drone inspection in a subway training base environment In this example, the system was tested in a training base that simulated a subway tunnel. This base contained typical subway tunnel structural features, such as tracks, tunnel walls, lighting equipment, and simulated cracks. The test aimed to verify the drone's inspection capabilities in a real tunnel environment. The specific implementation steps are as follows: Environment initialization and map construction: After the drone is started, it scans the training base environment using a 3D lidar and T265 binocular camera. Based on the improved FastLIO algorithm, it constructs a high-precision 3D point cloud map in real time and annotates key feature points (such as the track centerline and tunnel wall contour).

[0059] Autonomous Path Planning and Flight: The system generates a flight path based on the pre-set inspection mission. The drone autonomously flies along the centerline of the track, utilizing infrared recognition from the D435i depth camera to ensure precise alignment with the track. During flight, the drone dynamically adjusts its attitude to avoid obstacles such as lighting.

[0060] Real-time data collection and transmission: Drones equipped with 4K high-definition gimbal cameras capture tunnel walls at fixed distances, acquiring high-resolution image data. Simultaneously, 3D lidar updates environmental maps in real time, detecting changes in tunnel structures. All data is transmitted back to the terminal platform in real time via the 5G network.

[0061] Crack Detection and Structural Health Assessment: The terminal platform uses deep learning algorithms to analyze transmitted image data in real time, identifying defects such as cracks and spalling on the tunnel wall. It then combines this with 3D LiDAR data to assess the tunnel's structural health. The system automatically generates an inspection report, including defect location, size, and repair recommendations.

[0062] Endurance and performance test: In this test, the drone achieved a single flight time of 18 minutes, completing an inspection mission of approximately 500 meters of tunnel, with positioning accuracy maintained within ±3cm and a crack detection accuracy rate exceeding 93% (number of incorrectly identified samples / total number of samples ≤7%).

[0063] Test results show that the system can efficiently complete autonomous inspection tasks in the subway training base environment, has high-precision positioning, real-time data feedback and intelligent analysis capabilities, and is suitable for the inspection needs of actual subway tunnels.

[0064] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0065] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0066] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A subway tunnel drone inspection system, characterized by: The subway tunnel drone inspection system includes a flight platform, a depth camera, a binocular camera, a 3D laser radar, a TOF laser radar, a 4K gimbal, a crack analysis module, and a track-following flight module. The depth camera is used to identify infrared tracks, the binocular camera is used to provide visual positioning functions and collaborative path planning, the 3D laser radar is used to generate three-dimensional point cloud data of the tunnel environment and perform real-time mapping. When the crack analysis module receives real-time video or pictures of the environment, it marks the cracks based on the crack analysis algorithm. The track-following flight module controls the heading of the drone based on the rail features collected by the depth camera.

2. The subway tunnel drone inspection system according to claim 1, characterized in that: The crack analysis module converts the acquired 4K image from RGB to HSV color space based on the crack analysis algorithm, detects the image, locates the deformation area where the crack is located, and then marks the crack position and type on the original image.

3. The subway tunnel drone inspection system according to claim 2, characterized in that: The track-following flight module uses the YOLOv8 model to identify the features of the two rails in the tunnel, extract them, and calculate the offset between the drone and the center line of the track to adjust the drone's heading.

4. The subway tunnel drone inspection system according to claim 3 is characterized in that: Adjust the drone's roll angle according to the following formula: 。 5. The subway tunnel drone inspection system according to claim 3 is characterized in that: Adjust the drone's yaw rate according to the following formula: 。 6. The subway tunnel drone inspection system according to claim 2 is characterized in that: Update the heading attitude of the UAV based on EKF; Define the state vector Used to represent the drone's position, velocity, and sensor bias: ； in: is the three-dimensional position of the drone; It’s speed; is a quaternion, representing attitude (heading, pitch, roll); are the accelerometer and gyroscope biases, respectively.

7. The subway tunnel drone inspection system according to claim 2, characterized in that: The position and speed of the drone are calculated based on the following formula: ； in: is the input (IMU readings, including acceleration and angular velocity); is the process noise (used to simulate the error); It is the state transfer equation that describes the UAV motion model.

Images