Multi-modal pantograph arcing detection method based on double resnet-50 and cross attention fusion
Patent Information
- Application Number
- CN202611086827.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-21
- Publication Date
- 2026-10-16
AI Technical Summary
[0005]本发明的目的在于提供一种基于双ResNet-50与交叉注意力融合的多模态受电弓燃弧检测方法，旨在解决现有技术中的单模态受电弓燃弧检测方案易受光照、环境温度干扰，误检、漏检问题突出，传统融合方式易造成模态信息污染，无法充分融合可见光与红外图像互补特征，燃弧识别精度低，无法适配轨道交通车载实时监测的可靠性要求的技术问题
[0011] This invention discloses a multimodal pantograph arc detection method based on dual ResNet-50 and cross-attention fusion, comprising the following steps: An industrial monochrome camera and a thermal imager are simultaneously mounted on a rigid bracket on a vehicle roof; imaging is synchronously triggered by a unified trigger, outputting a visible light monochrome image and a thermal image with perfectly aligned spatiotemporal fields of view; two structurally consistent and independent ResNet-50 residual feature extraction branches are constructed, and the obtained aligned visible light monochrome image and thermal image are respectively fed into the corresponding residual branches for feature extraction; the extracted features are subjected to cross-modal cross-attention operations to achieve cross-modal dynamic alignment of visible light texture details and infrared temperature contours, and fusion... The spatial global location association information of the two types of images is combined; the fusion feature with the richest global information is selected to perform ViT global encoding enhancement to obtain global enhanced features; the global enhanced features are sent to the YOLOv5 network, and multi-scale fusion is performed using FPN and PAN to obtain multi-scale aggregated features, where FPN is a feature pyramid network and PAN is a path aggregation network; the obtained multi-scale aggregated features are sent to a parameter-sharing MLP multilayer perceptron for classification and target detection; through the above steps, it is possible to effectively avoid interference from lighting and temperature environment, significantly reduce false detection and false detection rates, improve arc detection accuracy, and adapt to the reliability requirements of real-time monitoring on rail transit vehicles.
Smart Images
Figure CN122929339A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of pantograph-catenary fault detection technology in rail transit, and in particular to a multimodal pantograph arc detection method based on dual ResNet-50 and cross-attention fusion. Background Technology
[0002] The pantograph is a core component for onboard power collection in rail transit. When trains travel at high speeds, the overhead contact line pressure fluctuates, or abnormal track conditions occur, arcing can easily occur at the contact surface between the carbon sliding plate and the contact wire. This arcing exacerbates the erosion of the sliding plate and wire, reduces power quality, and introduces electromagnetic interference. In severe cases, it can lead to pantograph-catenary faults and train stoppages. Therefore, high-precision real-time detection of arcing is necessary. Currently, pure vision-based solutions for pantograph arcing detection are broadly divided into two categories: visible light black-and-white image detection and thermal imaging detection. Visible light detection is low-cost and fast-responding, but it is easily affected by strong light, tunnel lighting, and streetlight reflections, resulting in numerous false alarms. Thermal imaging relies on temperature rise to identify arcing, but it is affected by ambient temperature and equipment parameters; fluctuations in the reference temperature lead to frequent missed and false detections.
[0003] Traditional YOLOv5 single-modal models only take a single image as input and cannot utilize the modal complementarity of visible light high brightness and infrared high temperature: real arcing has obvious features in both types of images, while various interferences only show false features in single-modal mode. Single-modal solutions have difficulty distinguishing interference from real arcing, resulting in prominent problems of false detection and missed detection.
[0004] In summary, existing single-mode pantograph arc detection solutions are susceptible to interference from light and ambient temperature, resulting in significant false detections and missed detections. Traditional fusion methods are prone to modal information contamination, failing to fully integrate complementary features of visible light and infrared images, leading to low arc recognition accuracy and making them unsuitable for the reliability requirements of real-time monitoring on rail transit vehicles. Summary of the Invention
[0005] The purpose of this invention is to provide a multimodal pantograph arc detection method based on dual ResNet-50 and cross-attention fusion. This method aims to solve the technical problems of existing single-mode pantograph arc detection schemes, which are susceptible to interference from light and ambient temperature, resulting in prominent false detections and missed detections. Traditional fusion methods are prone to modal information contamination, cannot fully integrate complementary features of visible light and infrared images, have low arc recognition accuracy, and cannot meet the reliability requirements of real-time monitoring on rail transit vehicles.
[0006] To achieve the above objectives, this invention employs a multimodal pantograph arc detection method based on dual ResNet-50 and cross-attention fusion, comprising the following steps: An industrial monochrome camera and a thermal imager are installed synchronously on a rigid bracket on the roof. Imaging is triggered synchronously by a unified trigger, and the output of a visible light monochrome image and a thermal image with perfectly aligned spatiotemporal fields of view are produced. Two structurally consistent and independent ResNet-50 residual feature extraction branches are constructed, and the obtained aligned visible light black and white images and thermal images are respectively fed into the corresponding residual branches for feature extraction; The extracted features are subjected to cross-modal cross-attention operations to achieve cross-modal dynamic alignment of visible light texture details and infrared temperature contours, and to fuse the spatial global position association information of the two types of images. The fusion feature with the richest global information is selected to perform ViT global encoding enhancement, resulting in a globally enhanced feature. The global augmented features are fed into the YOLOv5 network, and multi-scale fusion is performed using FPN and PAN to obtain multi-scale aggregated features, where FPN is the feature pyramid network and PAN is the path aggregation network. The obtained multi-scale aggregated features are fed into a parameter-sharing MLP (Multilayer Perceptron) for classification and target detection.
[0007] In the steps of simultaneously installing an industrial monochrome camera and a thermal imager via a rigid bracket on the vehicle roof, synchronously triggering imaging using a unified trigger, and outputting a visible light monochrome image and a thermal image with perfectly aligned spatiotemporal fields of view: By using an industrial monochrome camera and a thermal imager to eliminate field-of-view shift, a trigger outputs a synchronization pulse signal to control the two devices to expose synchronously at the same time, outputting paired images with uniform size and perfectly matched timestamps.
[0008] In the step of constructing two structurally consistent and independent ResNet-50 residual feature extraction branches, and then feeding the obtained aligned visible light black-and-white image and thermal image into the corresponding residual branches for feature extraction: The two ResNet-50 residual branches operate independently without interfering with each other, and each branch outputs three layers of multi-scale features.
[0009] In the step of feeding the global augmented features into the YOLOv5 network, multi-scale fusion is performed using FPN and PAN to obtain multi-scale aggregated features, where FPN is a feature pyramid network and PAN is a path aggregation network: The role of FPN is to pass high-level semantics to high-resolution features, while the role of PAN is to transmit arc location information from the lower level back to the higher level. Through the combination of FPN and PAN, bidirectional information flow is achieved.
[0010] In the step of feeding the obtained multi-scale aggregated features into a parameter-sharing MLP (Multilayer Perceptron) for classification and object detection: The feature values are divided into 256 dimensions, and each dimension is processed through an MLP. The feature data of all dimensions share the MLP parameters.
[0011] This invention discloses a multimodal pantograph arc detection method based on dual ResNet-50 and cross-attention fusion, comprising the following steps: An industrial monochrome camera and a thermal imager are simultaneously mounted on a rigid bracket on a vehicle roof; imaging is synchronously triggered by a unified trigger, outputting a visible light monochrome image and a thermal image with perfectly aligned spatiotemporal fields of view; two structurally consistent and independent ResNet-50 residual feature extraction branches are constructed, and the obtained aligned visible light monochrome image and thermal image are respectively fed into the corresponding residual branches for feature extraction; the extracted features are subjected to cross-modal cross-attention operations to achieve cross-modal dynamic alignment of visible light texture details and infrared temperature contours, and fusion... The spatial global location association information of the two types of images is combined; the fusion feature with the richest global information is selected to perform ViT global encoding enhancement to obtain global enhanced features; the global enhanced features are sent to the YOLOv5 network, and multi-scale fusion is performed using FPN and PAN to obtain multi-scale aggregated features, where FPN is a feature pyramid network and PAN is a path aggregation network; the obtained multi-scale aggregated features are sent to a parameter-sharing MLP multilayer perceptron for classification and target detection; through the above steps, it is possible to effectively avoid interference from lighting and temperature environment, significantly reduce false detection and false detection rates, improve arc detection accuracy, and adapt to the reliability requirements of real-time monitoring on rail transit vehicles. Attached Figure Description
[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0013] Figure 1 This is a flowchart of the steps of the multimodal pantograph arc detection method based on dual ResNet-50 and cross-attention fusion of the present invention.
[0014] Figure 2 This is a schematic diagram showing the connection of the industrial monochrome camera, trigger, and thermal imager of the present invention.
[0015] Figure 3 This invention relates to a fusion scheme for specific diagnosis extraction and multi-scale attention fusion graph.
[0016] Figure 4 This is the ViT enhanced global semantic graph of the present invention.
[0017] Figure 5This is a multi-scale fusion diagram of the FPN and PAN of the YOLOv5 backbone network of the present invention.
[0018] Figure 6 This is a diagram illustrating the target recognition and classification of the detection head in this invention.
[0019] Figure 7 This is a performance accuracy comparison chart of YOLOv5 and the multimodal fusion scheme of this invention. Detailed Implementation
[0020] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.
[0021] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.
[0022] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."
[0023] Please see Figures 1-7 This invention provides a multimodal pantograph arc detection method based on dual ResNet-50 and cross-attention fusion, comprising the following steps: S100: An industrial monochrome camera and a thermal imager are synchronously mounted on a rigid bracket on the roof. Imaging is triggered synchronously by a unified trigger, and the output of a visible light monochrome image and a thermal image with perfectly aligned spatiotemporal fields of view is produced. S200: Construct two structurally consistent and independent ResNet-50 residual feature extraction branches, and feed the obtained aligned visible light black and white image and thermal image into the corresponding residual branches for feature extraction; S300: Perform cross-modal cross-attention operations on the extracted features to achieve cross-modal dynamic alignment of visible light texture details and infrared temperature contours, and fuse the spatial global position association information of the two types of images; S400: Select the fusion feature with the richest global information to perform ViT global coding enhancement to obtain the global enhanced feature; S500: The global augmented features are fed into the YOLOv5 network, and multi-scale fusion is performed using FPN and PAN to obtain multi-scale aggregated features, where FPN is the feature pyramid network and PAN is the path aggregation network; S600: The obtained multi-scale aggregated features are fed into a parameter-sharing MLP (Multilayer Perceptron) for classification and target detection.
[0024] Furthermore, in the step of simultaneously mounting an industrial monochrome camera and a thermal imager via a rigid bracket on the vehicle roof, synchronously triggering imaging using a unified trigger, and outputting a visible light monochrome image and a thermal image with perfectly aligned spatiotemporal fields of view: By using an industrial monochrome camera and a thermal imager to eliminate field-of-view shift, a trigger outputs a synchronization pulse signal to control the two devices to expose synchronously at the same time, outputting paired images with uniform size and perfectly matched timestamps.
[0025] Furthermore, in the step of constructing two structurally consistent and independent ResNet-50 residual feature extraction branches, and feeding the obtained aligned visible light black-and-white image and thermal image into the corresponding residual branches for feature extraction: The two ResNet-50 residual branches operate independently without interfering with each other, and each branch outputs three layers of multi-scale features.
[0026] Furthermore, in the step of feeding the global augmented features into the YOLOv5 network and performing multi-scale fusion using FPN and PAN to obtain multi-scale aggregated features, where FPN is a feature pyramid network and PAN is a path aggregation network: The role of FPN is to pass high-level semantics to high-resolution features, while the role of PAN is to transmit arc location information from the lower level back to the higher level. Through the combination of FPN and PAN, bidirectional information flow is achieved.
[0027] Furthermore, in the step of feeding the obtained multi-scale aggregated features into a parameter-sharing MLP (Multilayer Perceptron) for classification and object detection: The feature values are divided into 256 dimensions, and each dimension is processed through an MLP. The feature data of all dimensions share the MLP parameters. Figure 6 It uses a detection head for classification and target detection.
[0028] In this embodiment, an industrial monochrome camera and a thermal imager are first synchronously mounted on a rigid bracket on the vehicle roof. Imaging is triggered synchronously using a unified trigger, outputting a visible light monochrome image and a thermal image with perfectly aligned spatiotemporal fields of view. Then, two structurally consistent and independent ResNet-50 residual feature extraction branches are constructed. The aligned visible light monochrome image and thermal image are fed into their respective residual branches for feature extraction. The extracted features are then subjected to cross-modal cross-attention operations to achieve cross-modal dynamic alignment of visible light texture details and infrared temperature contours, fusing the spatial global positional association information of the two types of images. The fused feature with the richest global information is selected for ViT global encoding enhancement to obtain the global enhanced feature. Finally, the global enhanced feature is fed into the YOLOv5 network. Multi-scale fusion is performed using Feature Pyramid Network (FPN) and Path Aggregation Network (PAN) to obtain multi-scale aggregated features. FPN is a feature pyramid network, and PAN is a path aggregation network. FPN transmits high-level semantics to high-resolution features, while PAN transmits arcing location information from the lower level back to the higher level. Finally, the obtained multi-scale aggregated features are fed into a parameter-sharing Multilayer Perceptron (MLP) for classification and target detection. A detection head is used to divide the feature values into 256 dimensions, and each dimension is processed through the MLP. The feature data of all dimensions share the MLP parameters for classification and target detection. Through the above steps, it is possible to effectively avoid interference from lighting and temperature environments, significantly reduce the false detection and false negative rates, improve the arcing detection accuracy, and adapt to the reliability requirements of real-time monitoring on rail transit vehicles.
[0029] Furthermore, such as Figure 2 As shown, a fixed structure secures a monochrome industrial camera and a thermal imager to the top of the train, preventing any displacement between them during the inspection process. Adjustments to the structure ensure that the areas seen by both devices at the moment of taking a picture are roughly the same. The monochrome industrial camera and the thermal imager are synchronously triggered by a single trigger, ensuring that the industrial camera's imaging and the thermal imager's photographs show different imaging responses for the same area at the same time.
[0030] like Figure 3 As shown, Figure 4 The second part is based on a dual residual structure to extract multi-scale local features. This effectively utilizes both the high detail of 2D imaging images and the clear contours of thermal imagers; C3 (80×80) targets small arc regions, and C5 (20×20) targets large arc regions. The two modules are encoded independently to avoid information contamination caused by initial mixing.
[0031] Figure 4The third part is the multi-scale cross-attention fusion module, which is the core of the entire special diagnosis extraction. To accommodate the detection of different arc sizes, an attention mechanism needs to be applied to C3 / C4 / C5. The cross-attention mechanism follows the formula below: ; Here, Q, K, and V represent three parameters in the attention mechanism. The fusion of different scales here achieves cross-modal dynamic alignment. Essentially, it effectively combines contour information from infrared images to assist in determining the arcing region.
[0032] Network structure such as Figure 4 As shown, global semantic enhancement is performed on f5. Only f5 is used here, while f4 and f3 are not used, because f5 has stronger global information.
[0033] like Figure 5 As shown, multi-scale aggregation of FPN+PAN is an important component of the YOLOv5 network. Using only FPN and PAN, FPN (Feature Pyramid Network) is a classic network structure in object detection, transferring high-level semantics to high-resolution features. PAN (Path Aggregation Network) is a path aggregation network, responsible for transmitting arc localization information from the bottom layer (f3) back to the top layer (f5). The combination of FPN and PAN enables bidirectional information flow, allowing each layer (n3, n4, n5) to possess both high-level semantics and precise localization capabilities. From the bottom layer to the top layer, these layers correspond to arc regions of different sizes.
[0034] Furthermore, experiments demonstrate: such as Figure 7 As shown, FusionModel represents the target recognition scheme under fusion, and YOLO represents the model using the official large version of YOLOv5. Experimental results show that the fusion scheme outperforms YOLOv5 on multiple evaluation metrics. Specifically, mAP@0.5 improved from 0.78 to 0.86, indicating a significant improvement in detection accuracy. Precision-Recall curves and false negative / false positive rate analysis further validate that this method maintains higher precision at various recall levels. However, this performance improvement comes at the cost of increased computational complexity, specifically a decrease in inference speed.
[0035] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.
[0036] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.
Claims
1. A multimodal pantograph arc detection method based on dual ResNet-50 and cross-attention fusion, characterized in that, Includes the following steps: An industrial monochrome camera and a thermal imager are installed synchronously on a rigid bracket on the roof. Imaging is triggered synchronously by a unified trigger, and the output of a visible light monochrome image and a thermal image with perfectly aligned spatiotemporal fields of view are produced. Two structurally consistent and independent ResNet-50 residual feature extraction branches are constructed, and the obtained aligned visible light black and white images and thermal images are respectively fed into the corresponding residual branches for feature extraction; The extracted features are subjected to cross-modal cross-attention operations to achieve cross-modal dynamic alignment of visible light texture details and infrared temperature contours, and to fuse the spatial global position association information of the two types of images. The fusion feature with the richest global information is selected to perform ViT global encoding enhancement, resulting in a globally enhanced feature. The global augmented features are fed into the YOLOv5 network, and multi-scale fusion is performed using FPN and PAN to obtain multi-scale aggregated features, where FPN is the feature pyramid network and PAN is the path aggregation network. The obtained multi-scale aggregated features are fed into a parameter-sharing MLP (Multilayer Perceptron) for classification and object detection.
2. The multimodal pantograph arc detection method based on dual ResNet-50 and cross-attention fusion as described in claim 1, characterized in that, In the process of synchronously installing an industrial monochrome camera and a thermal imager using a rigid bracket on the vehicle roof, synchronously triggering imaging using a unified trigger, and outputting a visible light monochrome image and a thermal image with perfectly aligned spatiotemporal fields of view: By using an industrial monochrome camera and a thermal imager to eliminate field-of-view shift, a trigger outputs a synchronization pulse signal to control the two devices to expose synchronously at the same time, outputting paired images with uniform size and perfectly matched timestamps.
3. The multimodal pantograph arc detection method based on dual ResNet-50 and cross-attention fusion as described in claim 1, characterized in that, In the step of constructing two structurally consistent and independent ResNet-50 residual feature extraction branches, and feeding the obtained aligned visible light black and white image and thermal image into the corresponding residual branches for feature extraction: The two ResNet-50 residual branches operate independently without interfering with each other, and each branch outputs three layers of multi-scale features.
4. The multimodal pantograph arc detection method based on dual ResNet-50 and cross-attention fusion as described in claim 1, characterized in that, In the step of feeding the global augmented features into the YOLOv5 network and performing multi-scale fusion using FPN and PAN to obtain multi-scale aggregated features, where FPN is a feature pyramid network and PAN is a path aggregation network: The role of FPN is to transmit high-level semantics to high-resolution features, while the role of PAN is to transmit arc location information from the lower level back to the higher level. Through the combination of FPN and PAN, bidirectional information flow is achieved.
5. The multimodal pantograph arc detection method based on dual ResNet-50 and cross-attention fusion as described in claim 1, characterized in that, In the step of feeding the obtained multi-scale aggregated features into a parameter-shared MLP (Multilayer Perceptron) for classification and object detection: The feature values are divided into 256 dimensions, and each dimension is processed through an MLP. The feature data of all dimensions share the MLP parameters.