Contact net spare part positioning and identifying device and method based on heterogeneous image fusion

Abstract

The present application relates to a contact net part positioning and identifying device based on heterogenous image fusion and a testing method, belonging to the field of image processing and intelligent operation and maintenance technology of railway contact net, comprising: a multi-source image acquisition module, an image registration module, an image fusion module, a part detection module, a positioning calculation module, a processor and a control module. Among them, visible light and infrared images are fused to enhance the recognition robustness of key components in complex environments; YOLOv12 detection model is applied to improve the detection accuracy of small targets and occluded targets; the overall structure is compact, suitable for deployment on site platforms such as inspection cars and trackside devices; modular design facilitates system maintenance and upgrading; the identification result can be used for automatic archiving, alarming and maintenance assistance.

Description

Technical Field

[0001] This invention belongs to the field of image processing and intelligent operation and maintenance technology of railway catenary, specifically involving a catenary component positioning and identification device and method based on heterogeneous image fusion. Background Technology

[0002] With the continuous improvement of railway electrification, the overhead contact system, as a key component of the power supply system, directly affects the safety of train operation. Traditional inspection methods rely on manual labor, which is inefficient and inaccurate, making it difficult to meet the requirements of modern high-speed rail intelligent maintenance. Existing single-vision inspection systems are easily affected by factors such as lighting and occlusion, making it difficult to reliably identify all critical components.

[0003] Therefore, at this stage, it is necessary to design a contact network component positioning and identification device and method based on heterogeneous image fusion to solve the above problems. Summary of the Invention

[0004] The purpose of this invention is to provide a contact wire component positioning and identification device and method based on heterogeneous image fusion, which is used to solve the technical problems existing in the prior art. The existing single vision detection system is easily affected by lighting, occlusion and other factors.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows: A contact wire component positioning and identification device based on heterogeneous image fusion includes: Multi-source image acquisition module: including RGB industrial camera and infrared camera, adopts hardware-level synchronous triggering mechanism, and controls the synchronous exposure of RGB industrial camera and infrared camera through unified trigger signal; Image registration module: The image registration module adopts a multi-stage registration strategy. First, feature points in RGB and infrared images are extracted based on corner detection algorithm; then, the RANSAC algorithm is used to estimate the transformation matrix and eliminate mismatched point pairs; finally, sub-pixel level template matching is used to optimize alignment accuracy and ensure accurate pixel correspondence in subsequent fusion processing. Image fusion module: A multi-scale deep neural network structure is used to extract the texture features of RGB images and the thermal distribution features of infrared images, respectively. After being concatenated through feature channels, the features are input into the fusion network for multi-layer semantic aggregation. Component detection module: Based on the YOLOv12 deep learning object detection model, the component detection module uses fused images as input. The positioning calculation module converts the center point of the two-dimensional detection box output by YOLOv12 into its actual position in the scene coordinate system by recording the camera's intrinsic and extrinsic parameters during image acquisition. Processor and control module: Integrated into the embedded platform, it completes model loading, image preprocessing, target detection and result output, and supports remote communication and platform linkage.

[0006] Furthermore, the multi-source image acquisition module includes a main control unit, an image anomaly recognition unit, a lens attachment detection unit, an automatic cleaning unit, an image verification unit, and an emergency maintenance warning unit. The main control unit is connected to the image anomaly recognition unit, the lens attachment detection unit, the automatic cleaning unit, the image verification unit, and the emergency maintenance warning unit, respectively. The image anomaly recognition unit is used to identify whether the RGB images captured by the RGB industrial camera are abnormal; The lens attachment detection unit is used to detect whether there is dirt attached to the lens of the RGB industrial camera; The automatic cleaning unit is used to automatically clean the lens of an RGB industrial camera; The image verification unit is used to verify the RGB images captured by the RGB industrial camera; The emergency maintenance warning unit is used to provide emergency warnings to maintenance personnel.

[0007] Furthermore, the main control unit controls the image anomaly recognition unit to be in a normally open working state, and controls the lens attachment detection unit, automatic cleaning unit, image verification unit, and emergency maintenance warning unit to be in a closed initial working state; When the image anomaly recognition unit detects an anomaly in the RGB image captured by the RGB industrial camera, the main control unit controls the lens attachment detection unit to be activated; When the lens attachment detection unit detects dirt on the lens of the RGB industrial camera, the main control unit controls the automatic cleaning unit to start. When the cumulative cleaning time of the automatic cleaning unit reaches the preset time, the main control unit controls the image verification unit to start. When the image verification unit verifies the RGB image captured by the RGB industrial camera and determines that the RGB image captured by the RGB industrial camera still has anomalies, the main control unit controls the emergency maintenance warning unit to be activated.

[0008] Furthermore, the multi-source image acquisition module is also equipped with a backup RGB industrial camera, which is normally closed. When the image verification unit verifies the RGB image captured by the RGB industrial camera and determines that the RGB image captured by the RGB industrial camera still has anomalies, the commonly used RGB industrial camera is turned off and the backup RGB industrial camera is turned on.

[0009] Furthermore, the multi-source image acquisition module also includes a cleaning action image detection unit; The cleaning action image detection unit is connected to the main control unit; The cleaning action image detection unit is used to detect whether the automatic cleaning unit automatically cleans the lens of the RGB industrial camera according to the preset cleaning action. The main control unit controls the initial state of the cleaning action image detection unit to be off; When the main control unit controls the automatic cleaning unit to start, it simultaneously controls the cleaning action image detection unit to start. If the cleaning action image detection unit detects that the automatic cleaning unit has not performed automatic cleaning of the lens of the RGB industrial camera according to the preset cleaning action, the emergency maintenance warning unit will issue a warning to the maintenance personnel that the cleaning action is abnormal.

[0010] Furthermore, the multi-source image acquisition module also includes a drone inspection and cleaning unit, which is used to remotely control a drone to clean the lens of an RGB industrial camera. When the emergency maintenance warning unit issues an alert to the maintenance personnel regarding abnormal cleaning operations, it triggers the activation of the drone patrol and cleaning unit.

[0011] Furthermore, the multi-source image acquisition module also includes a wireless communication unit, through which the main control unit interacts with the remote management center to achieve remote data transmission.

[0012] The contact wire component positioning and identification method based on heterogeneous image fusion uses the contact wire component positioning and identification device based on heterogeneous image fusion as described above to perform contact wire component positioning and identification based on heterogeneous image fusion.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: It integrates visible light and infrared images to enhance the robustness of key component identification in complex environments; it applies the YOLOv12 detection model to improve the detection accuracy of small targets and occluded targets; its overall structure is compact and suitable for deployment on field platforms such as train inspection vehicles and trackside devices; its modular design facilitates system maintenance and upgrades; and the identification results can be used for automatic archiving, alarms, and maintenance assistance.

[0014] Image acquisition, processing, and detection modules are deployed using an embedded hardware platform (such as Jetson Xavier). Visible light and infrared images are acquired through time synchronization and aligned by the image registration module. The fusion module employs a feature fusion strategy based on deep neural networks to enhance thermal feature representation while preserving texture information.

[0015] The recognition component employs a pre-trained YOLOv12 model to infer the location and category of key components such as supports, suspension cables, brackets, and locating pins from the fused images. This model supports high-precision detection in complex backgrounds and boasts excellent speed and lightweight design.

[0016] It can be deployed in patrol vehicles, trackside monitoring systems, or onboard intelligent terminals, supporting remote access and centralized control. The identification results can be transmitted back to the central system in real time for maintenance scheduling, fault early warning, and equipment file updates. Attached Figure Description

[0017] Figure 1 This is a schematic diagram illustrating the principle of Embodiment 1 in this solution implementation.

[0018] Figure 2 This is a schematic diagram of the operating logic of Embodiment 2 in this solution implementation. Detailed Implementation

[0019] Example 1: like Figure 1 As shown, a contact wire component positioning and recognition device based on heterogeneous image fusion is provided, which can effectively fuse multimodal information, thereby improving recognition accuracy and robustness. It includes: 1. Multi-source image acquisition module: This module includes an RGB industrial camera and an infrared camera. To ensure temporal and spatial consistency between the two image sources, the system employs a hardware-level synchronization triggering mechanism. A unified trigger signal controls the synchronous exposure of the RGB industrial camera and the infrared camera. During acquisition, the cameras are mounted on the same platform, and their intrinsic and extrinsic parameters are pre-calibrated (e.g., using the Zhang Zhengyou calibration method or a checkerboard joint calibration). The acquired images are saved as raw image files, along with timestamps and location information, providing fundamental data support for subsequent registration and fusion processing. The acquisition frequency can be dynamically adjusted according to train speed and monitoring accuracy requirements, typically set within the range of 5–15 Hz to balance resolution and processing efficiency.

[0020] 2. Image Registration Module: The image registration module employs a multi-stage registration strategy. First, feature points in the RGB and infrared images are extracted based on corner detection algorithms (such as Harris or Shi-Tomasi). Then, the RANSAC algorithm is used to estimate the transformation matrix and eliminate mismatched point pairs. Finally, sub-pixel-level template matching is used to further optimize the alignment accuracy, ensuring accurate pixel correspondence in subsequent fusion processing.

[0021] 3. Image Fusion Module: The image fusion module employs a multi-scale deep neural network structure to extract texture features from the RGB image and thermal distribution features from the infrared image, respectively. After feature channel concatenation, the images are input into a fusion network (such as a variant of U-Net) for multi-layer semantic aggregation. The final output fused image retains the visible light details and thermal response characteristics of key components, which helps improve the accuracy of subsequent detection modules.

[0022] 4. Component Detection Module: Based on the YOLOv12 deep learning object detection model, the component detection module uses fused images as input. The model employs an anchor-free mechanism to improve the detection accuracy of small targets (such as positioning pins) and introduces an attention mechanism to enhance attention to texture and thermal features. During training, a labeled multi-source fused image dataset is used, along with an IoU loss function to optimize the object bounding box regression accuracy, and multi-threaded asynchronous inference is supported to improve detection speed. YOLOv12 has stronger feature extraction capabilities and detection accuracy, making it suitable for complex real-world environments.

[0023] 5. Positioning Calculation Module: This module records the camera's intrinsic and extrinsic parameters (including position and pose) during image acquisition, converting the center point of the 2D detection box output by YOLOv12 into its actual position in the scene coordinate system. When combined with a LiDAR or binocular system, it can further achieve 3D positioning. This module also supports standardized positioning output based on the orbital coordinate system, facilitating integration with the overhead contact line maintenance system and assisting in manual inspection and maintenance planning.

[0024] 6. Processor and control module: Integrated into an embedded platform (such as Jetson Xavier or an industrial PC) to complete model loading, image preprocessing, target detection and result output, and support remote communication and platform linkage.

[0025] Among its features, it integrates visible light and infrared images to enhance the robustness of key component identification in complex environments; it applies the YOLOv12 detection model to improve the detection accuracy of small targets and occluded targets; its overall structure is compact and suitable for deployment on field platforms such as train inspection vehicles and trackside devices; its modular design facilitates system maintenance and upgrades; and the identification results can be used for automatic archiving, alarms, and maintenance assistance.

[0026] Image acquisition, processing, and detection modules are deployed using an embedded hardware platform (such as Jetson Xavier). Visible light and infrared images are acquired through time synchronization and aligned by the image registration module. The fusion module employs a feature fusion strategy based on deep neural networks to enhance thermal feature representation while preserving texture information.

[0027] The recognition component employs a pre-trained YOLOv12 model to infer the location and category of key components such as supports, suspension cables, brackets, and locating pins from the fused images. This model supports high-precision detection in complex backgrounds and boasts excellent speed and lightweight design.

[0028] This system can be deployed in patrol vehicles, trackside monitoring systems, or onboard intelligent terminals, supporting remote access and centralized control. Identification results can be transmitted back to the central system in real time for maintenance scheduling, fault warnings, and equipment file updates.

[0029] Example 2: Further, based on Example 1, such as Figure 2 As shown, the multi-source image acquisition module includes a main control unit, an image anomaly recognition unit, a lens attachment detection unit, an automatic cleaning unit, an image verification unit, and an emergency maintenance warning unit. The main control unit is connected to the image anomaly recognition unit, the lens attachment detection unit, the automatic cleaning unit, the image verification unit, and the emergency maintenance warning unit, respectively. The image anomaly recognition unit is used to identify whether the RGB images captured by the RGB industrial camera are abnormal; The lens attachment detection unit is used to detect whether there is dirt attached to the lens of the RGB industrial camera; The automatic cleaning unit is used to automatically clean the lens of an RGB industrial camera; The image verification unit is used to verify the RGB images captured by the RGB industrial camera; The emergency maintenance warning unit is used to provide emergency warnings to maintenance personnel.

[0030] Furthermore, the main control unit controls the image anomaly recognition unit to be in a normally open working state, and controls the lens attachment detection unit, automatic cleaning unit, image verification unit, and emergency maintenance warning unit to be in a closed initial working state; When the image anomaly recognition unit detects an anomaly in the RGB image captured by the RGB industrial camera, the main control unit controls the lens attachment detection unit to be activated; When the lens attachment detection unit detects dirt on the lens of the RGB industrial camera, the main control unit controls the automatic cleaning unit to start. When the cumulative cleaning time of the automatic cleaning unit reaches the preset time, the main control unit controls the image verification unit to start. When the image verification unit verifies the RGB image captured by the RGB industrial camera and determines that the RGB image captured by the RGB industrial camera still has anomalies, the main control unit controls the emergency maintenance warning unit to be activated.

[0031] The working logic is as follows: The specific advantages are as follows: High reliability: Through proactive maintenance, the probability of system failure due to lens contamination is greatly reduced, and it is adaptable to harsh environments such as dust, rain and snow along the contact network.

[0032] Intelligentization and automation: Reduced the workload of manual inspection and lens cleaning, and improved the system's automation level.

[0033] Energy-efficient and efficient: By adopting a condition-triggered mechanism, each functional module is started on demand, which reduces overall energy consumption and computing load.

[0034] Maintaining a closed loop: A complete closed loop is formed from "identifying the problem" to "attempting to solve it" and then to "verifying the results," with rigorous logic.

[0035] Furthermore, the multi-source image acquisition module is also equipped with a backup RGB industrial camera, which is normally closed. When the image verification unit verifies the RGB image captured by the RGB industrial camera and determines that the RGB image captured by the RGB industrial camera still has anomalies, the commonly used RGB industrial camera is turned off and the backup RGB industrial camera is turned on.

[0036] The specific advantages are as follows: Achieving "non-stop" operation and maintenance: This is the most crucial improvement. When the main camera experiences an unrepairable failure, the system can switch to the backup camera in a very short time, ensuring that the image acquisition and data fusion process is uninterrupted and guaranteeing the continuity of the overhead contact line inspection task.

[0037] Emergency maintenance is downgraded from "real-time response" to "planned maintenance": After switching to the backup camera, the maintenance of the main camera no longer requires emergency repairs and can be carried out at a leisurely pace during subsequent maintenance windows, which greatly reduces the pressure on operation and maintenance.

[0038] Improved system MTBF (Mean Time Between Failures): Through hardware redundancy and intelligent switching, the reliability of the entire image acquisition module is significantly improved.

[0039] Furthermore, the multi-source image acquisition module also includes a cleaning action image detection unit; The cleaning action image detection unit is connected to the main control unit; The cleaning action image detection unit is used to detect whether the automatic cleaning unit automatically cleans the lens of the RGB industrial camera according to the preset cleaning action. The main control unit controls the initial state of the cleaning action image detection unit to be off; When the main control unit controls the automatic cleaning unit to start, it simultaneously controls the cleaning action image detection unit to start. If the cleaning action image detection unit detects that the automatic cleaning unit has not performed automatic cleaning of the lens of the RGB industrial camera according to the preset cleaning action, the emergency maintenance warning unit will issue a warning to the maintenance personnel that the cleaning action is abnormal.

[0040] The specific advantage is: shifting from "what happened" to "why it happened".

[0041] Without a cleaning action image detection unit, the system can only know the result that "the image is still abnormal after cleaning," but cannot determine the cause. Possible causes include: The lens is too stubbornly dirty (cleaning is performed normally, but to no avail).

[0042] Malfunctions in the cleaning facility itself (such as stuck cleaning card, clogged nozzles, or drive failure) can cause cleaning actions to be not performed or to fail to meet standards.

[0043] The newly added detection unit directly targets and identifies the second type of cause, and its core value lies in: Precise fault isolation: Quickly pinpoint the root cause of "cleaning failure" to "cleaning unit failure" or "lens / camera failure", greatly shortening the diagnosis time for on-site maintenance personnel.

[0044] Avoid misjudgment and waste of resources: Prevent the system from repeatedly performing ineffective cleaning due to cleaning mechanism failure, or prematurely determining that the main camera hardware is damaged and triggering redundant switching.

[0045] Improve the reliability of maintenance actions: Ensure that automated actions are under control and meet the "deterministic" requirements of industrial systems.

[0046] Furthermore, the multi-source image acquisition module also includes a drone inspection and cleaning unit, which is used to remotely control a drone to clean the lens of an RGB industrial camera. When the emergency maintenance warning unit issues an alert to the maintenance personnel regarding abnormal cleaning operations, it triggers the activation of the drone patrol and cleaning unit.

[0047] The specific advantages are as follows: Solving the ultimate problem: When the automatic cleaning unit (assuming it is a fixed robotic arm or nozzle) suffers a physical malfunction that cannot be remotely recovered (such as mechanical jamming or pipeline rupture), drones can serve as the ultimate "surgical" emergency measure, avoiding the need to send personnel (possibly aerial work platforms) for on-site repairs, and greatly improving the system's survivability during unattended operation.

[0048] To deal with complex contamination: For large-scale, high-viscosity, or irregularly shaped deposits that are difficult to handle by conventional automatic cleaning units, drones may be equipped with more powerful cleaning tools (such as miniature high-pressure water guns and special solvent spraying) for targeted deep cleaning.

[0049] Suitable for special deployment environments: If the RGB industrial camera is installed in a location that is difficult for personnel to reach quickly and safely, such as a bridge, tunnel entrance, or overpass, the value of the drone will be even more prominent.

[0050] Furthermore, the multi-source image acquisition module also includes a wireless communication unit, through which the main control unit interacts with the remote management center to achieve remote data transmission.

[0051] The contact wire component positioning and identification method based on heterogeneous image fusion uses the contact wire component positioning and identification device based on heterogeneous image fusion as described above to perform contact wire component positioning and identification based on heterogeneous image fusion.

Claims

1. A contact wire component positioning and identification device based on heterogeneous image fusion, characterized in that, include: Multi-source image acquisition module: including RGB industrial camera and infrared camera, adopts hardware-level synchronous triggering mechanism, and controls the synchronous exposure of RGB industrial camera and infrared camera through unified trigger signal; Image registration module: The image registration module adopts a multi-stage registration strategy. First, feature points in RGB and infrared images are extracted based on corner detection algorithm; then, the RANSAC algorithm is used to estimate the transformation matrix and eliminate mismatched point pairs; finally, sub-pixel level template matching is used to optimize alignment accuracy and ensure accurate pixel correspondence in subsequent fusion processing. Image fusion module: A multi-scale deep neural network structure is used to extract the texture features of RGB images and the thermal distribution features of infrared images, respectively. After being concatenated through feature channels, the features are input into the fusion network for multi-layer semantic aggregation. Component detection module: Based on the YOLOv12 deep learning object detection model, the component detection module uses fused images as input. The positioning calculation module converts the center point of the two-dimensional detection box output by YOLOv12 into its actual position in the scene coordinate system by recording the camera's intrinsic and extrinsic parameters during image acquisition. Processor and control module: Integrated into the embedded platform, it completes model loading, image preprocessing, target detection and result output, and supports remote communication and platform linkage.

2. The contact wire component positioning and identification device based on heterogeneous image fusion according to claim 1, characterized in that, The multi-source image acquisition module includes a main control unit, an image anomaly recognition unit, a lens attachment detection unit, an automatic cleaning unit, an image verification unit, and an emergency maintenance warning unit. The main control unit is connected to the image anomaly recognition unit, the lens attachment detection unit, the automatic cleaning unit, the image verification unit, and the emergency maintenance warning unit, respectively. The image anomaly recognition unit is used to identify whether the RGB images captured by the RGB industrial camera are abnormal; The lens attachment detection unit is used to detect whether there is dirt attached to the lens of the RGB industrial camera; The automatic cleaning unit is used to automatically clean the lens of an RGB industrial camera; The image verification unit is used to verify the RGB images captured by the RGB industrial camera; The emergency maintenance warning unit is used to provide emergency warnings to maintenance personnel.

3. The contact wire component positioning and identification device based on heterogeneous image fusion according to claim 2, characterized in that, The main control unit controls the image anomaly recognition unit to be in a normally open working state, and controls the lens attachment detection unit, automatic cleaning unit, image verification unit, and emergency maintenance warning unit to be in a closed initial working state; When the image anomaly recognition unit detects an anomaly in the RGB image captured by the RGB industrial camera, the main control unit controls the lens attachment detection unit to be activated; When the lens attachment detection unit detects dirt on the lens of the RGB industrial camera, the main control unit controls the automatic cleaning unit to start. When the cumulative cleaning time of the automatic cleaning unit reaches the preset time, the main control unit controls the image verification unit to start. When the image verification unit verifies the RGB image captured by the RGB industrial camera and determines that the RGB image captured by the RGB industrial camera still has anomalies, the main control unit controls the emergency maintenance warning unit to be activated.

4. The contact wire component positioning and identification device based on heterogeneous image fusion according to claim 3, characterized in that, The multi-source image acquisition module is also equipped with a backup RGB industrial camera, which is normally closed. When the image verification unit verifies the RGB image captured by the RGB industrial camera and determines that the RGB image captured by the RGB industrial camera still has anomalies, the commonly used RGB industrial camera is turned off and the backup RGB industrial camera is turned on.

5. The contact wire component positioning and identification device based on heterogeneous image fusion according to claim 3, characterized in that, The multi-source image acquisition module also includes a cleaning action image detection unit; The cleaning action image detection unit is connected to the main control unit; The cleaning action image detection unit is used to detect whether the automatic cleaning unit automatically cleans the lens of the RGB industrial camera according to the preset cleaning action. The main control unit controls the initial state of the cleaning action image detection unit to be off; When the main control unit controls the automatic cleaning unit to start, it simultaneously controls the cleaning action image detection unit to start. If the cleaning action image detection unit detects that the automatic cleaning unit has not performed automatic cleaning of the lens of the RGB industrial camera according to the preset cleaning action, the emergency maintenance warning unit will issue a warning to the maintenance personnel that the cleaning action is abnormal.

6. The contact wire component positioning and identification device based on heterogeneous image fusion according to claim 5, characterized in that, The multi-source image acquisition module also includes a drone inspection and cleaning unit, which is used to clean the lens of the RGB industrial camera by remotely controlling the drone. When the emergency maintenance warning unit issues an alert to the maintenance personnel regarding abnormal cleaning operations, it triggers the activation of the drone patrol and cleaning unit.

7. The contact wire component positioning and identification device based on heterogeneous image fusion according to claim 2, characterized in that, The multi-source image acquisition module also includes a wireless communication unit, through which the main control unit interacts with the remote management center.

8. A method for locating and identifying overhead contact line components based on heterogeneous image fusion, characterized in that, The contact wire component positioning and identification device based on heterogeneous image fusion as described in any one of claims 1-7 is used to perform contact wire component positioning and identification based on heterogeneous image fusion.

Images