A tunnel electromechanical equipment intelligent inspection method based on multi-vision algorithm fusion

Abstract

The application provides a tunnel electromechanical equipment intelligent inspection method based on multi-vision algorithm fusion, which comprises four steps of multi-modal data acquisition, AI algorithm processing, multi-modal data fusion positioning and intelligent terminal processing; the application synchronously collects continuous image data and three-dimensional point cloud data in the tunnel through a multi-sensor array carried on an intelligent inspection vehicle to form multi-modal data with time sequence information; multi-vision algorithm fusion is introduced, a light classification algorithm, a multi-target tracking algorithm and an improved target detection model are responsible for real-time operation by an edge computing module, fast equipment detection and multi-state recognition are completed, and the real-time performance and accuracy of detection are ensured; two-dimensional visual detection results are mapped to three-dimensional point cloud space through space-time registration and coordinate mapping, and the three-dimensional coordinates of abnormal equipment are accurately calculated. The application replaces manual inspection with AI recognition, ensures the real-time performance and reliability of the results, and realizes fine intelligent management and control of tunnel electromechanical equipment.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and intelligent transportation operation and maintenance technology, specifically involving an intelligent inspection method for tunnel electromechanical equipment based on multi-vision algorithm fusion. Background Technology

[0002] With the rapid development of my country's highway transportation network, tunnels, as an important component of transportation infrastructure, are seeing a continuous increase in both number and mileage. Tunnel electromechanical equipment (including lighting, fire-fighting facilities, and emergency indicator lights) is crucial for ensuring the safe operation of tunnels, and its operational status directly affects tunnel safety and emergency response capabilities. Traditional manual periodic inspections are not only inefficient but also suffer from significant drawbacks such as high personnel safety risks, difficulty in data management, and severe influence of inspector subjective judgment on inspection results. In recent years, intelligent inspection technology based on computer vision has been gradually applied. A search revealed Chinese patent publication number CN109658954A, which discloses a method for identifying and detecting electromechanical equipment. This method includes obtaining image data of tunnel electromechanical equipment using an inspection vehicle; performing Retinex-DCE image enhancement on the image data to obtain enhanced image data; using an improved YOLOv5s target detection model to detect the enhanced data; and outputting the equipment image along with its location and type.

[0003] The aforementioned patents have the following shortcomings: Most of the electromechanical equipment inside tunnels are small targets, and the lack of targeted optimization of image enhancement strategies and YOLOv5s network structures results in weak overall small target detection capabilities and a high rate of missed detections for long-distance equipment; the identification of lighting fixtures such as illumination lights and emergency indicator lights relies on time-series analysis capabilities, and the patents, which only rely on single-frame image recognition, cannot achieve dynamic fault identification of these fixtures; the confined space within tunnels obstructs signals, wireless communication signals attenuate, and GPS positioning accuracy is low, leading to significant errors in identifying the location of abnormal electromechanical equipment within the tunnel, making it difficult to meet the needs of refined operation and maintenance. Therefore, it is necessary to develop an intelligent inspection method for tunnel electromechanical equipment based on multi-vision algorithm fusion to solve the above problems. Summary of the Invention

[0004] To address the aforementioned issues, this invention discloses an intelligent inspection method for tunnel electromechanical equipment based on the fusion of multiple visual algorithms. By integrating various visual algorithms to replace human visual interpretation, the method ensures the objectivity and real-time nature of the results and significantly improves adaptability to small targets and complex environments.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows:

[0006] A method for intelligent inspection of tunnel electromechanical equipment based on multi-vision algorithm fusion includes:

[0007] Step 1. Multimodal data acquisition: The vehicle-mounted high-definition industrial camera and 3D laser scanner mounted on the intelligent inspection vehicle simultaneously acquire continuous images and 3D point cloud data in the tunnel to form a multimodal data unit with time-series information.

[0008] Step 2. AI algorithm processing: The improved target detection model is used to detect images in real time, and outputs the position coordinates, anomaly information and confidence scores of various abnormal electromechanical equipment; Based on the target detection and multi-target tracking algorithm, a unique tracking ID is assigned to each lighting lamp and emergency indicator light, and its operating status is judged through continuous time series analysis; A lightweight classification algorithm is used to classify the detected fire-fighting facilities in a secondary manner to accurately identify their abnormal status.

[0009] Step 3. Multimodal data fusion and localization: Spatiotemporal registration and fusion of 3D point cloud data and visual inspection results are performed to calculate the precise 3D spatial location of the abnormal device;

[0010] Step 4. Smart Terminal Processing: The smart terminal receives the algorithm processing results and displays them in real time. It also performs spatial registration using multimodal data to generate a detection report containing data such as anomaly type and 3D coordinates.

[0011] A further improvement to the method is that the three-dimensional laser scanner described in step 1 integrates an inertial measurement unit and an odometer positioning and attitude determination system to acquire position and attitude compensation information for precise stitching of the data collected by the three-dimensional laser scanner.

[0012] A further improvement to the method is that step 2 specifically comprises:

[0013] S21. Perform enhancement processing on the input image data to obtain enhanced image data;

[0014] S22. Process the enhanced image data using the improved target detection algorithm to obtain preliminary detection results;

[0015] S23. Combining the detection results with the multi-target tracking algorithm, assign a unique tracking ID to each lighting lamp and emergency indicator light, and determine the status of the lighting lamp and emergency indicator light, including normal, off, or flashing, by analyzing the facility status changes in consecutive frames of images.

[0016] S24. Combining the detection results with the post-processing classification algorithm, the detection results are further classified and filtered to accurately identify the abnormal status of fire protection facilities, including normal and damaged.

[0017] S25. The specific abnormal state and location of the output facility.

[0018] A further improvement to the method is that the specific method of the fusion processing in step 3 is as follows: based on the pre-calibrated camera-LiDAR extrinsic parameter matrix, the visual detection results output by the AI ​​algorithm are mapped to the three-dimensional point cloud space, and the real-time global pose of the inspection vehicle obtained by multi-sensor fusion is combined to calculate the precise three-dimensional coordinates of the abnormal equipment.

[0019] A further improvement to the method is that the intelligent terminal in step 4 is used to display the inspection status and detection results in real time, and automatically generates a comprehensive detection report containing the abnormality type and three-dimensional coordinates when an equipment abnormality is detected.

[0020] The beneficial effects of this invention are as follows:

[0021] This invention achieves intelligent, continuous, and automatic detection by synchronously acquiring images and 3D point cloud data within tunnels using a multi-sensor array mounted on a mobile platform. It introduces an end-edge collaborative architecture, where the data acquisition end consists of an industrial camera and a 3D laser scanner mounted on the inspection vehicle to collect tunnel scene data; the edge side is equipped with vehicle-mounted edge computing hardware to receive data synchronously acquired from the end side and run improved YOLO, classification, and tracking algorithms in real time, ensuring real-time detection and low latency. Through image preprocessing and improvements to the YOLOv11 network, this invention effectively solves core challenges such as dust interference and missed detection of small targets within tunnels. Multimodal data fusion positioning significantly improves the accuracy of locating abnormal equipment. This invention replaces human visual interpretation with an AI model, ensuring the objectivity and consistency of results, automating and intelligentizing the inspection process, and significantly improving the operational efficiency of tunnel electromechanical equipment. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating the operation of the present invention.

[0023] Figure 2 This is a diagram of the YOLOv11 network structure.

[0024] Figure 3 This is a diagram of the CAFM module structure. Detailed Implementation

[0025] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0026] like Figure 1 As shown, this invention discloses an intelligent inspection method for tunnel electromechanical equipment based on multi-vision algorithm fusion, achieving automated and precise inspection of tunnel electromechanical equipment. The specific technical steps are as follows:

[0027] S01: Multimodal data acquisition;

[0028] S02: AI algorithm processing;

[0029] S03: Multimodal data fusion positioning;

[0030] S04: Smart terminal processing.

[0031] The above technical steps are further explained below:

[0032] In step S01: Multimodal data acquisition involves simultaneously acquiring continuous images and 3D point cloud data within the tunnel using a vehicle-mounted high-definition industrial camera and a 3D laser scanner mounted on the intelligent inspection vehicle, forming a multimodal data unit with temporal information. The multi-sensor array includes two 20-megapixel industrial cameras, one facing forward and the other slightly upward and to the side of the tunnel, and a 3D laser scanner (scanning 2 million points per second). The inspection vehicle's travel speed is set to 15-20 kilometers per hour. The multi-sensor array is triggered by a built-in synchronization signal generator, performing synchronous acquisition with a 100-millisecond cycle. Specifically, the synchronization signal generator simultaneously sends a start command to all sensors, the industrial camera performs an exposure, and the laser scanner completes a line scan. Simultaneously, the positioning and attitude determination system integrated into the 3D laser scanner records the inspection vehicle's 3D coordinates (X, Y, Z) and attitude angles (roll, pitch, yaw) in real time, and all acquired data is stamped with a unified high-precision timestamp.

[0033] In step S02: the edge computing module processes the acquired image data, specifically by performing HSV luminance space adaptive enhancement processing on the input image data to obtain enhanced image data. Specifically, this invention employs an adaptive HSV-CLAHE image enhancement method based on luminance statistical features. First, the original RGB image is converted to the HSV color space, separating the three independent components: hue (H), saturation (S), and luminance (V). Then, the CLAHE cropping limit value is adaptively calculated, expressed as:

[0034]

[0035] Where L_global is the global average brightness of the V component, and L_max is the global average of the local maximum brightness map obtained after applying a 3×3 window maximum filter to the V component. 1.2 is an empirical value, which can be adjusted to 1.0-1.5 based on the actual dust concentration and lighting conditions in the tunnel. If L_global is below the threshold of 60, it is considered a low-light environment. In this case, clipLimit is constrained to 2.0-5.0 using a clipping limit expression; otherwise, the default value of 2.0 is used. Next, the CLAHE algorithm with a grid size of (8,8) is used to enhance the local contrast of the V component. Finally, the enhanced V component is merged back with the original H and S components and converted back to the RGB color space. This significantly improves the visibility of details in the dark areas of the tunnel while effectively suppressing excessive enhancement of highlight areas such as direct headlights. For example, at the tunnel entrance, when the image is locally overexposed and overall dark due to direct headlights, this adaptive method can enhance dark details (such as the outline of distant lights) to the visible range while suppressing excessive enhancement of highlight areas.

[0036] Further improvements are made by: extracting multi-scale features from the enhanced image through an improved YOLOv11 object detection network, and classifying and locating the electromechanical equipment area based on these features, distinguishing categories such as lighting, fire-fighting facilities, and emergency indicator lights. The improvements include: embedding a convolutional attention fusion module (CAFM) before the second C3K2 module in the backbone network to suppress feature noise caused by dust while extracting deeper features; replacing the standard convolutions in all CBS modules in the backbone and neck network with depthwise separable convolutions to reduce the overall number of parameters with minimal loss of accuracy, thus better adapting to edge computing devices; replacing the bounding box loss function with Inner-SIoU, utilizing the auxiliary bounding box concept to accelerate the convergence of small target bounding box regression and improve localization accuracy; and improving the YOLOv11 network structure diagram and CAFM module structure diagram as shown below. Figure 2 , Figure 3 .

[0037] Further improvements include: combining detection results with the multi-target tracking algorithm ByteTrack, assigning a unique tracking ID (e.g., ID_001, ID_002) to each lighting fixture and emergency indicator light; for each frame, extracting the pixel mean of the brightness V channel within the bounding box of each fixture as the brightness value of the current frame, and defining a rule for judging whether a fixture is in a normal state based on the brightness value, for example, a brightness value ≥ 200 is considered "normal" and a brightness value < 100 is considered "off". For each tracking ID, examining its brightness sequence within the most recent 5 seconds, if both "normal" and "off" frames exist in the sequence, the fixture is judged as "flickering"; if all frames in the sequence are "normal", it is judged as "normal"; if all frames in the sequence are "off", it is judged as "off". For example, the brightness values ​​of ID_125 are [210, 205, 208, ..., 212] for 5 consecutive seconds, all of which are "normal" and are therefore judged as "normal"; while the brightness sequence of ID_089 is [210, 45, 220, 40, 215, 50, ...], which includes both normal and off states, so it is judged as "flickering".

[0038] A further improvement lies in combining the detection results with the post-processing classification network ResNet50 to perform secondary classification and filtering of the detection results, accurately identifying the abnormal state of fire protection facilities. Specifically, after the improved YOLOv11 detects fire protection facilities, it crops the corresponding equipment area image from the original image based on its bounding box coordinates and inputs it into the pre-trained ResNet50 classification model. This correctly identifies the fire protection facilities (normal or damaged).

[0039] A further improvement is that the edge computing module packages all the above detection results (device category, abnormal status, bounding box coordinates, mileage) into a JSON data packet.

[0040] In step S03: Multimodal data fusion and localization. The 3D point cloud data and visual inspection results are spatiotemporally registered and fused to calculate the precise 3D spatial location of the abnormal device. This fusion process is based on edge computing units or cloud servers, and specifically includes:

[0041] First, the LOAM algorithm is run based on the point cloud data from the 3D laser scanner to extract line and surface features for inter-frame registration, estimating the local pose of the inspection vehicle in real time. Then, the LOAM pose is tightly coupled and fused with IMU, wheeled odometer, and GPS / BeiDou data from entrances and exits using extended Kalman filtering to obtain globally consistent real-time positioning results. When the AI ​​detects an anomaly, based on the global pose matrix of the inspection vehicle at that moment, combined with pre-calibrated camera intrinsic and camera-LiDAR extrinsic parameters, the coordinates of the center pixel of the detection frame are converted into a spatial ray in the geodetic coordinate system. This ray is then intersected with the corresponding laser point cloud data, and the centroid of a subset of the point cloud is taken as the 3D coordinates of the abnormal device. Finally, based on the tunnel's global point cloud map and design alignment data, these coordinates are converted into mileage markers, lateral offset, and height information for rapid location by maintenance personnel.

[0042] In step S04: Smart terminal processing. The smart terminal receives the algorithm processing results and displays them in real time. It also performs spatial registration using multimodal data, mapping the coordinates of abnormal targets detected in the two-dimensional image to a three-dimensional spatial coordinate system, generating a detection report containing data such as the abnormality type and three-dimensional coordinates. When a device abnormality is detected, the system automatically stores the coordinate information, status, and original image cropping of the abnormal device into the detection report and pushes it to the management terminal.

[0043] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A method for intelligent inspection of tunnel electromechanical equipment based on multi-vision algorithm fusion, characterized in that, include: Step 1. Multimodal data acquisition: The vehicle-mounted high-definition industrial camera and 3D laser scanner mounted on the intelligent inspection vehicle simultaneously acquire continuous images and 3D point cloud data in the tunnel to form a multimodal data unit with time-series information. Step 2. AI algorithm processing: The improved target detection model is used to detect images in real time, and outputs the position coordinates, anomaly information and confidence scores of various abnormal electromechanical equipment; Based on the target detection and multi-target tracking algorithm, a unique tracking ID is assigned to each lighting lamp and emergency indicator light, and its operating status is judged through continuous time series analysis; A lightweight classification algorithm is used to classify the detected fire-fighting facilities in a secondary manner to accurately identify their abnormal status. Step 3. Multimodal data fusion and localization: Spatiotemporal registration and fusion of 3D point cloud data and visual inspection results are performed to calculate the precise 3D spatial location of the abnormal device; Step 4. Smart Terminal Processing: The smart terminal receives the algorithm processing results and displays them in real time. It also performs spatial registration using multimodal data to generate a detection report containing anomaly types and three-dimensional coordinate data.

2. The intelligent inspection method for tunnel electromechanical equipment based on multi-vision algorithm fusion according to claim 1, characterized in that: The three-dimensional laser scanner described in step 1 includes a positioning and attitude determination system containing an inertial measurement unit and a wheeled odometer, which is used to acquire the position and attitude compensation information of the inspection vehicle and realize the accurate stitching of three-dimensional laser point cloud data.

3. The intelligent inspection method for tunnel electromechanical equipment based on multi-vision algorithm fusion according to claim 1, characterized in that: The AI ​​algorithm processing described in step 2 specifically includes: S21. Perform enhancement processing on the input image data to obtain enhanced image data; S22. Process the enhanced image data using the improved target detection algorithm to obtain preliminary detection results; S23. Combining the detection results with the multi-target tracking algorithm, assign a unique tracking ID to each lighting lamp and emergency indicator light, and determine the status of the lighting lamp and emergency indicator light, including normal, off, or flashing, by analyzing the facility status changes in consecutive frames of images. S24. Combining the detection results with the post-processing classification algorithm, the detection results are classified and filtered in a second way to accurately identify the abnormal status of fire protection facilities, including normal and damaged. S25. The specific abnormal state and location of the output facility.

4. The intelligent inspection method for tunnel electromechanical equipment based on multi-vision algorithm fusion according to claim 3, characterized in that: The improved target detection algorithm described in step S22 is improved by: embedding a convolutional attention fusion (CAFM) module before the second C3K2 module of the backbone network to suppress feature noise caused by dust and extract deeper features; replacing the standard convolutions of all CBS modules in the backbone network and neck network with depthwise separable convolutions to reduce the overall number of parameters with minimal loss of accuracy, thus making it more suitable for edge computing devices; and replacing the bounding box loss function with Inner-SIoU to accelerate the convergence of small target bounding box regression and improve localization accuracy using the auxiliary bounding box idea.

5. The intelligent inspection method for tunnel electromechanical equipment based on multi-vision algorithm fusion according to claim 1, characterized in that: The specific method of fusion processing is as follows: based on the pre-calibrated camera-LiDAR extrinsic parameter matrix, the visual detection results output by the AI ​​algorithm are mapped to the three-dimensional point cloud space, and the real-time global pose of the inspection vehicle obtained by multi-sensor fusion is combined to calculate the precise three-dimensional coordinates of the abnormal equipment.

6. The intelligent inspection method for tunnel electromechanical equipment based on multi-vision algorithm fusion according to claim 1, characterized in that: The intelligent terminal described in step 4 is used to display the inspection status and test results in real time. When an equipment abnormality is detected, it automatically generates a comprehensive test report containing the abnormality type and three-dimensional coordinates.

Images