Power transmission line tree obstacle monitoring system and method based on multi-modal time sequence fusion and Transform-GNN hybrid network
By combining multimodal temporal fusion with a Transformer-GNN hybrid network, and integrating multi-source sensor data acquisition and edge computing, the problem of inaccurate tree obstacle detection in power transmission line inspection was solved, achieving efficient and real-time tree obstacle monitoring and dynamic obstacle avoidance, and reducing the missed detection rate and hardware costs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING CHENGZHI HEAVY IND CHART TECH CO LTD
- Filing Date
- 2025-11-24
- Publication Date
- 2026-04-24
AI Technical Summary
Existing technologies for tree obstacle detection in power transmission line inspections are inaccurate, lack comprehensive information, have weak predictive capabilities, and the models are not suitable for edge deployment, resulting in high false negative rates, response delays, and high hardware costs.
A hybrid network of multimodal temporal fusion and Transformer-GNN is adopted, and data is acquired by LiDAR, RGB-NIR multispectral cameras and thermal infrared cameras. Image features are extracted through SegFormer-L1 backbone network, Delaunay triangulation is used to construct tree graph structure, GraphSAGE is used to aggregate topological relationships and feature interaction is achieved through cross-modal attention mechanism, and Voxelmorph network is combined for temporal prediction. Edge computing is performed on Jetson TX2 platform and closed-loop control is achieved through ROS 2 nodes.
It achieves high-precision tree obstacle detection, reduces the missed detection rate, improves prediction accuracy, reduces invalid inspection mileage, lowers hardware costs, adapts to real-time operation of edge devices, and dynamically adjusts the route to avoid accidents.
Smart Images
Figure CN121921675A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent inspection technology for power transmission lines, and more specifically, to a system and method for monitoring tree obstructions on power transmission lines based on a hybrid network of multimodal temporal fusion and Transformer-GNN. Background Technology
[0002] With the continuous extension of power grid lines and the increasing complexity of terrain environments, traditional manual inspections or single image recognition methods have many shortcomings in monitoring tree obstacles along power transmission channels. Existing technologies often rely on single sensors (such as lidar or visible light cameras), making it difficult to comprehensively capture the morphological, spectral, and thermal radiation characteristics of vegetation, resulting in high false negative rates. Traditional methods lack continuous tracking of vegetation growth dynamics and cannot predict future canopy expansion trends, only passively addressing risks through periodic comprehensive inspections. Existing deep learning models (such as CNNs) have insufficient ability to fuse multimodal features from point clouds and images, especially in complex terrain where they are susceptible to noise interference. Furthermore, tree obstacle detection algorithms based on full point cloud traversal are computationally intensive, making real-time operation on edge devices difficult and requiring cloud processing, leading to response delays. Finally, tree obstacle detection results are not fed back to the flight control system in real time for flight path adjustments or obstacle avoidance.
[0003] Therefore, there is an urgent need for an intelligent inspection system that can integrate information from multiple sensor sources, has time-series analysis capabilities, and can be deployed on edge devices to achieve closed-loop control. Summary of the Invention
[0004] The purpose of this invention is to provide a transmission line tree obstacle monitoring system and method based on multimodal temporal fusion and Transformer-GNN hybrid network, so as to solve the problems of inaccurate tree obstacle detection, incomplete information, weak prediction ability and unsuitable model for edge deployment in the existing transmission line inspection mentioned in the background art.
[0005] To achieve the above objectives, the present invention aims to provide a power transmission line tree obstacle monitoring system based on a multimodal temporal fusion and Transformer-GNN hybrid network, comprising: The multi-source data acquisition module, equipped with a LiDAR sensor (point cloud density ≥300 points / m²), an RGB-NIR multispectral camera (resolution ≥0.05m / px), and a thermal infrared camera (resolution ≥0.2m / px) on the UAV, is used to simultaneously acquire three-dimensional structure, spectral reflectance, and thermal radiation data of the vegetation in the power transmission corridor. The preprocessing and registration module is used to remove ground and non-vegetation point clouds based on the RANSAC algorithm, and to achieve spatiotemporal alignment of LiDAR point clouds, RGB-NIR images and thermal infrared data using SIFT feature matching and ICP algorithm, with a registration error of <0.1m. The hybrid network analysis module is used to extract global semantic features of multi-scale images through the SegFormer-L1 backbone network. At the same time, it constructs a tree branch node graph structure based on Delaunay triangulation, aggregates spatial topological relationships using GraphSAGE, and projects image features to point cloud nodes through a cross-modal attention mechanism to achieve feature interaction. Finally, it outputs high-precision vegetation semantic segmentation results and tree point cloud spatial reconstruction data. The time-series prediction module is used to align multi-temporal point cloud data based on the Voxelmorph network and use LSTM to fuse temperature and precipitation meteorological factors to predict the tree canopy growth trajectory for the next 30 days. The edge computing module is used to deploy lightweight models (12M parameters) that have undergone knowledge distillation and pruning compression on the Jetson TX2 platform, achieving real-time inference of ≥5FPS through TensorRT's FP16+INT8 mixed precision acceleration. It also includes a closed-loop control module, which transmits the risk level to the UAV flight control system via the ROS 2 node, dynamically triggering obstacle avoidance or return-to-home commands in the line-flying mode.
[0006] As a further improvement to this technical solution, the hybrid network analysis module includes a Transformer unit, a GNN unit, and a cross-modal interaction unit; The Transformer unit is used to extract global semantic features of multi-scale images with SegFormer-L1 as the backbone network and an input resolution of 512×512. The GNN unit is used to construct a tree branch node graph structure based on Delaunay triangulation, and aggregate spatial topological relationships through GraphSAGE. The edge features include Euclidean distance and the angle between the normals. The cross-modal interaction unit is used to project image features onto point cloud nodes through an attention mechanism to achieve feature fusion.
[0007] This invention also provides a method for monitoring tree obstacles on transmission lines based on a hybrid network of multimodal temporal fusion and Transformer-GNN, comprising the following steps: S1. Multi-source data acquisition and calibration; S2. Preprocessing and registration; S3. Construct and train a Transformer-GNN hybrid network model; S4. Time-series change detection and growth trajectory prediction; S5, Risk Assessment and Alerts; S6, Edge computing and closed-loop control system deployment.
[0008] As a further improvement to this technical solution, the specific steps for multi-source data acquisition and calibration in step S1 are as follows: S11, the drone is equipped with a LiDAR, RGB-NIR multispectral camera and a thermal infrared camera to collect data simultaneously; S12. The intrinsic and extrinsic parameters of the RGB-NIR camera are calibrated using the Zhang Zhengyou checkerboard method, and the extrinsic parameters of the LiDAR and optical cameras are calibrated using a laser target plate, with a calibration accuracy of ±2cm. S13. Use a blackbody calibration source to calibrate the thermal infrared camera; temperature error < ±0.5℃. S14. Set LiDAR point cloud density > 300 points / m², RGB-NIR image resolution > 0.05m / px, and thermal infrared resolution > 0.2m / px.
[0009] As a further improvement to this technical solution, the specific steps for preprocessing and registration in step S2 are as follows: S21. Remove ground and non-vegetation point clouds based on voxel grid and RANSAC algorithm; S22. Combining ground control points (GCPs) and digital elevation models (DEMs), subpixel-level orthorectification is achieved using polynomial transformation; S23. Based on SIFT feature matching and ICP algorithm, the spatiotemporal alignment of RGB-NIR imagery, thermal infrared data and LiDAR point cloud is completed with a registration error of <0.1m.
[0010] As a further improvement to this technical solution, the specific steps for constructing and training the Transformer-GNN hybrid network model in step S3 are as follows: S31. Construct a Transformer-GNN hybrid network; S311. Using the SegFormer-L1 backbone network with an input resolution of 512×512, global features of multi-scale images are extracted. S312. Construct a graph structure based on Delaunay triangulation, where nodes represent key branch points and edge features include Euclidean distance and the angle between the normal and the edge. S313. Use GraphSAGE for message passing and insert a cross-modal attention module between Transformer and GNN to project image features onto point cloud nodes for interaction. S32, Multi-task Joint Training; S321. The semantic segmentation branch uses a hybrid loss of Dice and cross-entropy. S322. Point cloud branches are optimized using a weighted average of Chamfer distance and graph structure consistency loss. S323. Data Augmentation: The images are augmented with random rotation, color jitter, and Gaussian Blur, while the point cloud is augmented with random culling and scale perturbation.
[0011] As a further improvement to this technical solution, the specific steps for detecting temporal changes and predicting growth trajectories in step S4 are as follows: S41. Pair point clouds and images from multiple flights in the same area, and calculate temporal differences after alignment using a Voxelmorph network. S42. Using LSTM to process sequence difference features and integrating temperature and precipitation meteorological factors to predict future canopy expansion trajectory, the prediction error is <0.1m.
[0012] As a further improvement to this technical solution, the specific operational steps for risk assessment and alarm in step S5 are as follows: S51. Calculate the risk level based on the growth trajectory prediction results and the safe distance threshold (typical value 3m); S52 triggers ground station alarms or flight control system obstacle avoidance commands through decision nodes.
[0013] As a further improvement to this technical solution, the specific operational steps for deploying the edge computing and closed-loop control system in step S6 are as follows: S61. The number of parameters in the Transformer-GNN model was reduced from 50M to 12M through knowledge distillation and pruning. S62. Deploy the model on the Jetson TX2 platform and use TensorRT for FP16+INT8 mixed precision acceleration to achieve real-time processing of ≥5FPS. S63. Integrate ROS 2 nodes into the flight control system to complete closed-loop control of data synchronization, model inference, and dynamic route adjustment.
[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This transmission line tree obstacle monitoring system and method based on multimodal temporal fusion and Transformer-GNN hybrid network captures three-dimensional structure through LiDAR point cloud (≥300 points / m²), provides vegetation indices such as NDVI through multispectral imagery (0.05m / px), and detects canopy water stress through thermal infrared data (0.2m / px). It effectively overcomes the interference of different weather and complex backgrounds on single visual recognition and comprehensively improves the detection accuracy to over 98%.
[0015] 2. The transmission line tree obstacle monitoring system and method based on multimodal temporal fusion and Transformer-GNN hybrid network utilizes the Transformer-GNN architecture to achieve feature interaction through cross-modal attention modules. The semantic segmentation Dice coefficient reaches 0.92, and the point cloud reconstruction Chamfer distance is <0.05m, which is 20% higher than that of a single model (such as PointNet++).
[0016] 3. The transmission line tree obstacle monitoring system and method based on multimodal temporal fusion and Transformer-GNN hybrid network has a temporal difference prediction error of <0.1m based on LSTM, supports the prediction of tree canopy growth trajectory in the next 30 days, and shortens the risk warning response time to within 10 seconds.
[0017] 4. The transmission line tree obstacle monitoring system and method based on multimodal temporal fusion and Transformer-GNN hybrid network compresses the number of model parameters to 12M, achieves an inference speed of 5FPS on the Jetson TX2 platform, and reduces power consumption by 60%. It is compatible with UAVs and vehicle-mounted mobile platforms. At the same time, through risk level assessment and linkage with the ROS system, it can dynamically adjust the flight path or trigger a return to home after detecting a high-risk tree obstacle area.
[0018] 5. The transmission line tree obstacle monitoring system and method based on multimodal temporal fusion and Transformer-GNN hybrid network reduces the invalid inspection mileage by about 40% by dynamically adjusting the route, and the hardware cost is reduced by about 50% compared with the fixed monitoring system.
[0019] 6. The transmission line tree obstacle monitoring system and method based on multimodal temporal fusion and Transformer-GNN hybrid network is applicable to various typical transmission corridor environments such as mountainous areas, forest areas, and uninhabited areas, and has broad application prospects. Attached Figure Description
[0020] Figure 1 This is a schematic diagram of the transmission line tree obstacle monitoring method based on multimodal temporal fusion and Transformer-GNN hybrid network of the present invention. Detailed Implementation
[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0022] In one specific embodiment, the present invention provides a transmission line tree obstacle monitoring system based on multimodal time series fusion and Transformer-GNN hybrid network, including a multi-source data acquisition module, a preprocessing and registration module, a hybrid network analysis module, a time series prediction module, an edge computing module, and a closed-loop control module.
[0023] The multi-source data acquisition module, equipped with a LiDAR sensor (point cloud density ≥300 points / m²), an RGB-NIR multispectral camera (resolution ≥0.05m / px), and a thermal infrared camera (resolution ≥0.2m / px) on the UAV, is used to simultaneously acquire three-dimensional structure, spectral reflectance, and thermal radiation data of the vegetation along the power transmission corridor. Through multimodal data acquisition, the three-dimensional morphology, spectral characteristics, and thermal radiation information of the vegetation can be comprehensively captured, improving the richness and accuracy of the data and providing a solid foundation for subsequent analysis.
[0024] The preprocessing and registration module is used to remove ground and non-vegetation point clouds based on the RANSAC algorithm, and to achieve spatiotemporal alignment of LiDAR point clouds, RGB-NIR imagery, and thermal infrared data using SIFT feature matching and ICP algorithms, with a registration error of <0.1m. Irrelevant point cloud data is removed to improve data processing efficiency; high-precision registration ensures the consistency of multi-source data, providing a reliable guarantee for subsequent feature extraction and analysis.
[0025] The hybrid network analysis module extracts global semantic features from multi-scale images using a SegFormer-L1 backbone network. Simultaneously, it constructs a tree branch node graph structure based on Delaunay triangulation, aggregates spatial topological relationships using GraphSAGE, and projects image features onto point cloud nodes through a cross-modal attention mechanism to achieve feature interaction. The final output is high-precision vegetation semantic segmentation results and tree point cloud spatial reconstruction data. The hybrid network analysis module includes a Transformer unit, a GNN unit, and a cross-modal interaction unit. The Transformer unit uses a SegFormer-L1 backbone network with an input resolution of 512×512 to extract global semantic features from multi-scale images. The GNN unit constructs a tree branch node graph structure based on Delaunay triangulation and aggregates spatial topological relationships using GraphSAGE; edge features include the Euclidean distance and the angle between the edge and the normal. The cross-modal interaction unit projects image features onto point cloud nodes through an attention mechanism to achieve feature fusion. The Transformer-GNN architecture effectively fuses multi-modal features, improving the accuracy of semantic segmentation and point cloud reconstruction, and providing accurate foundational data for canopy growth prediction.
[0026] The time-series forecasting module aligns multi-temporal point cloud data using a Voxelmorph network and employs LSTM to fuse temperature and precipitation meteorological factors to predict tree canopy growth trajectories for the next 30 days. Through time-series forecasting, tree canopy growth trends can be understood in advance, providing a sufficient time window for risk warning and effectively reducing the risk of power grid accidents caused by tree obstructions.
[0027] The edge computing module is used to deploy a lightweight model (12M parameters) that has undergone knowledge distillation and pruning compression on the Jetson TX2 platform. Real-time inference of ≥5FPS is achieved through TensorRT's FP16+INT8 mixed precision acceleration. The combination of lightweight model and hardware acceleration technologies enables efficient real-time data processing capabilities, allowing the system to run stably on edge devices such as drones, thus improving inspection efficiency.
[0028] The closed-loop control module transmits the risk level to the UAV flight control system via the ROS 2 node, dynamically triggering obstacle avoidance or return-to-home commands in line-flying mode. It implements a closed-loop feedback mechanism from data acquisition, processing, and analysis to control, enabling dynamic adjustment of inspection strategies based on real-time risk levels, thus improving the targeting and effectiveness of inspections.
[0029] In one specific embodiment, such as Figure 1 As shown, the present invention also provides a method for monitoring tree obstacles on transmission lines based on a hybrid network of multimodal temporal fusion and Transformer-GNN, comprising the following steps: Step 1: Multi-source data acquisition and calibration. Specifically: The UAV is equipped with a LiDAR, an RGB-NIR multispectral camera, and a thermal infrared camera to acquire 3D point clouds, spectral images, and thermal radiation data, respectively. The Zhang Zhengyou checkerboard method was used to calibrate the intrinsic and extrinsic parameters of the RGB-NIR camera, and a laser target plate was used to calibrate the extrinsic parameters of the LiDAR and optical cameras, achieving a calibration accuracy of ±2cm. A blackbody calibration source was used to periodically calibrate the thermal infrared camera, ensuring that the temperature error of the thermal radiation data was <±0.5℃. The LiDAR point cloud density was set to ≥300 points / m², the RGB-NIR image resolution to ≥0.05m / px, and the thermal infrared resolution to ≥0.2m / px, meeting the requirements for high-precision canopy detail capture. This high-precision calibration ensured the accuracy of the multi-source data, providing a reliable data foundation for subsequent analysis.
[0030] Step 2: Preprocessing and registration. Specifically: Ground and non-vegetated points were removed using a voxel grid and the Random Sample Consensus Algorithm (RANSAC), preserving tree point clouds. Subpixel-level orthorectification was achieved using polynomial transformation, combining ground control points (GCPs) and a digital elevation model (DEM). Spatiotemporal alignment of RGB-NI, thermal infrared imagery, and LiDAR point clouds was completed using SIFT feature matching and the ICP (Iterative Closest Point) algorithm, with a registration error <0.1m. These preprocessing and registration steps improved data purity and consistency, creating favorable conditions for subsequent feature extraction and analysis.
[0031] The third step is to construct and train a Transformer-GNN hybrid network model. Specifically: 1. Construct a Transformer-GNN hybrid network.
[0032] The SegFormer-L1 backbone is adopted with an input resolution of 512×512. Global semantic features are extracted using a multi-scale encoder. A graph structure based on Delaunay triangulation is constructed, where nodes represent key branch points and edge features include Euclidean distance and the angle between the normal and edge features. GraphSAGE is used for message passing. A cross-modal attention module is inserted between the Transformer and GNN to project image features onto point cloud nodes and interact with them in combination with point cloud features.
[0033] 2. Multi-task joint training.
[0034] For the loss function, the semantic segmentation branch uses a hybrid loss of Dice and cross-entropy, while the point cloud branch uses a weighted joint training of Chamfer distance and graph structure consistency loss. For data augmentation, random rotation, color jittering, and Gaussian Blur are used for images, and random culling and scale perturbation are used for point clouds to improve the robustness of the model.
[0035] By leveraging the hybrid network model, the advantages of both Transformer and GNN are fully utilized, achieving effective fusion and interaction of multimodal features and improving the overall performance of the model.
[0036] Step 4: Detection of temporal changes and prediction of growth trajectory. Specifically: Point clouds and images from multiple flights in the same area were paired, and differences were calculated after alignment using a spatial deformation network (Voxelmorph) to extract temporal differences. LSTM was used to process the sequence difference features, and combined with meteorological and environmental factors, to predict future canopy expansion trajectories with a prediction error of <0.1m. Temporal change detection and growth trajectory prediction provide a scientific basis for risk warning and help to take measures in advance to avoid potential risks.
[0037] Step 5: Risk Assessment and Alerts. Specifically: The risk level is calculated based on the prediction results and a safe distance threshold (typically 3m), and the decision node triggers an early warning from the ground station or flight control system. This real-time risk assessment and alarm mechanism can promptly remind relevant personnel to take countermeasures, effectively reducing the risk of power grid accidents caused by tree obstructions.
[0038] Step 6: Deployment of edge computing and closed-loop control systems. Specifically: Through knowledge distillation and pruning, the number of parameters in the Transformer-GNN model was reduced from 50M to 12M, increasing inference speed by 2.5 times. The model was deployed on a Jetson TX2, using TensorRT for mixed-precision acceleration (FP16+INT8), achieving a single-board real-time processing rate of over 5 FPS. ROS2 nodes were integrated into the flight control system to achieve closed-loop control of data synchronization, model inference, and dynamic flight path adjustment. The deployment of edge computing and the closed-loop control system improved the system's real-time performance and intelligence, making inspection tasks more efficient and accurate.
[0039] The beneficial effects of the present invention will be illustrated below through specific embodiments: Implementation scenario: A 500kV transmission line corridor in a mountainous area, with dense vegetation and frequent historical tree obstructions.
[0040] Implementation steps: 1. Data Acquisition: The UAV is equipped with LiDAR, RGB-NIR cameras and thermal infrared sensors. The flight path is set to cover a serpentine area within 50 meters on both sides of the guide line. The flight altitude is about 50 meters. The UAV flies along the preset flight path to collect point cloud, multispectral images and thermal radiation data.
[0041] 2. Preprocessing and registration: RANSAC is used to remove ground points, and SIFT+ICP is used to align multi-source data (registration error < 0.1m) to generate a fused 3D model.
[0042] 3. Model Inference: The Jetson TX2 loads the SegFormer-GNN model and uses TensorRT inference to segment vegetation areas in real time and extract key branch nodes to predict the safe distance between tree canopy and guide wire.
[0043] 4. Time series analysis and early warning: Combining historical flight data and meteorological information (temperature, precipitation), the LSTM model predicts the future canopy expansion trajectory, triggers high-risk section (distance <3m) alarms, and assists in manual obstacle removal.
[0044] 5. Dynamic route adjustment: The ROS2 node controls the drone to switch to the line-laying flight mode, conducts detailed re-inspection of high-risk sections, and updates the inspection cycle of low-risk sections.
[0045] Implementation results: The detection miss rate decreased from 15% to 2%, the single inspection time was reduced by 30%, and three potential tree obstacle risks were successfully warned, avoiding line tripping accidents.
[0046] In summary, this invention solves the problems of inaccurate tree obstacle detection, incomplete information, weak predictive capabilities, and unsuitable models for edge deployment in existing transmission line inspections. Furthermore, this invention constructs an integrated intelligent tree obstacle detection system encompassing "perception-analysis-prediction-feedback" through multimodal data acquisition, high-precision preprocessing and registration, hybrid network feature extraction and analysis, time-series prediction, risk assessment and alarm, and edge computing and closed-loop control. This system is widely applicable to transmission corridor inspection tasks, improving the intelligence and automation level of power grid inspections.
[0047] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.
Claims
1. A power transmission line tree obstacle monitoring system based on multimodal temporal fusion and Transformer-GNN hybrid network, characterized in that, include: The multi-source data acquisition module, equipped with the UAV's LiDAR sensor, RGB-NIR multispectral camera, and thermal infrared camera, is used to simultaneously acquire three-dimensional structure, spectral reflectance, and thermal radiation data of the vegetation in the power transmission corridor. The preprocessing and registration module is used to remove ground and non-vegetation point clouds based on the RANSAC algorithm, and to achieve spatiotemporal alignment of LiDAR point clouds, RGB-NIR images and thermal infrared data using SIFT feature matching and ICP algorithm, with a registration error of <0.1m. The hybrid network analysis module is used to extract global semantic features of multi-scale images through the SegFormer-L1 backbone network. At the same time, it constructs a tree branch node graph structure based on Delaunay triangulation, aggregates spatial topological relationships using GraphSAGE, and projects image features to point cloud nodes through a cross-modal attention mechanism to achieve feature interaction. Finally, it outputs high-precision vegetation semantic segmentation results and tree point cloud spatial reconstruction data. The time-series prediction module is used to align multi-temporal point cloud data based on the Voxelmorph network and use LSTM to fuse temperature and precipitation meteorological factors to predict the tree canopy growth trajectory for the next 30 days. The edge computing module is used to deploy lightweight models (12M parameters) that have undergone knowledge distillation and pruning compression on the JetsonTX2 platform, and achieve ≥5FPS real-time inference through TensorRT's FP16+INT8 mixed precision acceleration. It also includes a closed-loop control module, which transmits the risk level to the UAV flight control system via the ROS 2 node, dynamically triggering obstacle avoidance or return-to-home commands in the line-flying mode.
2. The power transmission line tree obstacle monitoring system based on multimodal temporal fusion and Transformer-GNN hybrid network according to claim 1, characterized in that, The hybrid network analysis module includes a Transformer unit, a GNN unit, and a cross-modal interaction unit; The Transformer unit is used to extract global semantic features of multi-scale images with SegFormer-L1 as the backbone network and an input resolution of 512×512. The GNN unit is used to construct a tree branch node graph structure based on Delaunay triangulation, and aggregate spatial topological relationships through GraphSAGE. The edge features include Euclidean distance and the angle between the normals. The cross-modal interaction unit is used to project image features onto point cloud nodes through an attention mechanism to achieve feature fusion.
3. A method for monitoring tree obstacles on transmission lines based on multimodal temporal fusion and a Transformer-GNN hybrid network, characterized by the following steps: S1. Multi-source data acquisition and calibration; S2. Preprocessing and registration; S3. Construct and train a Transformer-GNN hybrid network model; S4. Time-series change detection and growth trajectory prediction; S5, Risk Assessment and Alerts; S6, Edge computing and closed-loop control system deployment.
4. The method for monitoring tree obstacles on transmission lines based on multimodal temporal fusion and a Transformer-GNN hybrid network according to claim 3, characterized in that, The specific steps for multi-source data acquisition and calibration in step S1 are as follows: S11, the drone is equipped with a LiDAR, RGB-NIR multispectral camera and a thermal infrared camera to collect data simultaneously; S12. The intrinsic and extrinsic parameters of the RGB-NIR camera are calibrated using the Zhang Zhengyou checkerboard method, and the extrinsic parameters of the LiDAR and optical cameras are calibrated using a laser target plate, with a calibration accuracy of ±2cm. S13. Use a blackbody calibration source to calibrate the thermal infrared camera; temperature error < ±0.5℃. S14. Set LiDAR point cloud density > 300 points / m², RGB-NIR image resolution > 0.05m / px, and thermal infrared resolution > 0.2m / px.
5. The method for monitoring tree obstacles on transmission lines based on multimodal temporal fusion and a Transformer-GNN hybrid network according to claim 3, characterized in that, The specific steps for preprocessing and registration in step S2 are as follows: S21. Remove ground and non-vegetation point clouds based on voxel grid and RANSAC algorithm; S22. Combining ground control points (GCPs) and digital elevation models (DEMs), subpixel-level orthorectification is achieved using polynomial transformation; S23. Based on SIFT feature matching and ICP algorithm, the spatiotemporal alignment of RGB-NIR imagery, thermal infrared data and LiDAR point cloud is completed with a registration error of <0.1m.
6. The method for monitoring tree obstacles on transmission lines based on multimodal temporal fusion and a Transformer-GNN hybrid network according to claim 3, characterized in that, The specific steps for constructing and training the Transformer-GNN hybrid network model in step S3 are as follows: S31. Construct a Transformer-GNN hybrid network; S311. Using the SegFormer-L1 backbone network with an input resolution of 512×512, global features of multi-scale images are extracted. S312. Construct a graph structure based on Delaunay triangulation, where nodes represent key branch points and edge features include Euclidean distance and the angle between the normal and the edge. S313. Use GraphSAGE for message passing and insert a cross-modal attention module between Transformer and GNN to project image features onto point cloud nodes for interaction. S32, Multi-task Joint Training; S321. The semantic segmentation branch uses a hybrid loss of Dice and cross-entropy. S322. Point cloud branches are optimized using a weighted average of Chamfer distance and graph structure consistency loss. S323. Data Augmentation: The images are augmented with random rotation, color jitter, and Gaussian Blur, while the point cloud is augmented with random culling and scale perturbation.
7. The method for monitoring tree obstacles on transmission lines based on multimodal temporal fusion and a Transformer-GNN hybrid network according to claim 3, characterized in that, The specific steps for detecting temporal changes and predicting growth trajectories in step S4 are as follows: S41. Pair point clouds and images from multiple flights in the same area, and calculate temporal differences after alignment using a Voxelmorph network. S42. Using LSTM to process sequence difference features and integrating temperature and precipitation meteorological factors to predict future canopy expansion trajectory, the prediction error is <0.1m.
8. The method for monitoring tree obstacles on transmission lines based on multimodal temporal fusion and a Transformer-GNN hybrid network according to claim 3, characterized in that, The specific steps for risk assessment and alarm in step S5 are as follows: S51. Calculate the risk level based on the growth trajectory prediction results and the safe distance threshold (typical value 3m); S52 triggers ground station alarms or flight control system obstacle avoidance commands through decision nodes.
9. The method for monitoring tree obstacles on transmission lines based on multimodal temporal fusion and a Transformer-GNN hybrid network according to claim 3, characterized in that, The specific steps for deploying the edge computing and closed-loop control system in step S6 are as follows: S61. The number of parameters in the Transformer-GNN model was reduced from 50M to 12M through knowledge distillation and pruning. S62. Deploy the model on the Jetson TX2 platform and use TensorRT for FP16+INT8 mixed precision acceleration to achieve real-time processing of ≥5FPS. S63. Integrate ROS 2 nodes into the flight control system to complete closed-loop control of data synchronization, model inference, and dynamic route adjustment.