Deep learning-based unmanned aerial vehicle inspection method for overhead transmission line

Abstract

The invention relates to the technical field of unmanned aerial vehicle inspection of overhead transmission lines, in particular to an unmanned aerial vehicle inspection method of an overhead transmission line based on deep learning, and the method comprises the steps: synchronously collecting a visible light image, infrared temperature data, point cloud data, a flight attitude and meteorological parameters through an unmanned aerial vehicle during inspection, and constructing a standardized inspection data set; visible light and infrared features are respectively extracted through a double-flow convolutional neural network, and a cross-modal attention mechanism is introduced into a shared feature layer to realize feature alignment and fusion. And further performing time correlation modeling and trend extrapolation on the historical inspection sequence of the equipment component through a time sequence prediction model to generate a health trend value and predictive alarm data. A reinforcement learning state vector is constructed in combination with a flight attitude, meteorological parameters, recognition confidence and predictive alarm data, an action space with track adjustment and secondary shooting as a core is established, and an unmanned aerial vehicle is driven to perform routing inspection path optimization and intelligent reinspection of a high-risk part through a reward function update strategy.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) inspection technology for overhead power transmission lines, and more particularly to a UAV inspection method for overhead power transmission lines based on deep learning. Background Technology

[0002] With the continuous expansion of the number and scale of overhead power transmission lines, traditional line inspection methods relying on manual labor or simple inspection equipment are no longer sufficient to meet the requirements of high reliability and high efficiency in operation and maintenance. Manual inspection is usually limited by factors such as terrain, weather, and human safety, resulting in problems such as long inspection cycles, low efficiency, and a high rate of missed hazard detection. Drones, on the other hand, are widely used in power transmission line inspection scenarios due to their flexible deployment, low cost, and high precision. However, existing drone inspection systems mainly rely on single visible light or infrared images for defect identification, which are greatly affected by environmental conditions such as lighting, temperature differences, and viewing angles, easily leading to unstable identification accuracy.

[0003] Meanwhile, traditional inspection systems typically employ static flight path planning strategies, which cannot dynamically adjust flight paths based on real-time identification results and risk predictions. This results in insufficient detection coverage of drones in key areas, making it difficult to achieve early warning and targeted re-inspection of potential hazards. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a deep learning-based method for unmanned aerial vehicle (UAV) inspection of overhead power transmission lines.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A deep learning-based method for unmanned aerial vehicle (UAV) inspection of overhead power transmission lines includes the following steps: S1. Based on the visible light images, infrared temperature data, point cloud data, flight attitude information and meteorological parameters collected synchronously by the UAV during the line inspection flight, and after performing time synchronization, spatial registration and noise filtering processing, a standardized inspection dataset is obtained. S2. Input the registered visible light image and infrared temperature image in the inspection dataset into the visual branch and infrared branch of the dual-stream convolutional neural network, respectively. Perform feature alignment and fusion in the shared feature layer based on the cross-modal attention mechanism, and output the defect identification result and confidence score of the equipment component. S3. Based on the defect identification results and confidence scores, input the historical inspection sequence data of the equipment components into the time series prediction network, perform time correlation modeling and trend extrapolation, obtain the health trend values ​​of the equipment components for several future periods, and generate predictive alarm data. S4. Based on the flight attitude data, meteorological parameters, confidence scores and predictive alarm data collected during the inspection process, construct the state vector of reinforcement learning, establish the action space with the trajectory adjustment of the secondary shooting as the candidate action, and execute the policy update with the reward function aimed at improving the defect confidence and minimizing energy consumption to obtain the trajectory optimization instruction. S5. Drive the UAV to perform a secondary inspection according to the trajectory optimization command.

[0006] Further, S1 includes the following steps: The drone simultaneously collects visible light images, infrared temperature data, point cloud data, flight attitude information, and meteorological parameters using its multimodal sensors, and performs synchronous calibration according to timestamps to obtain initial multi-source data. Distortion correction and brightness normalization are performed on the visible light image in the initial multi-source data, drift compensation and outlier pixel removal are performed on the infrared temperature data, outlier filtering and density resampling are performed on the point cloud data, and noise-suppressed multi-source corrected data are obtained. Based on the attitude parameters and extrinsic parameter matrices in the multi-source correction data, sensor spatial registration and projection alignment processing are performed to map the infrared image and point cloud data to the visible light image coordinate system, thereby obtaining a standardized inspection dataset.

[0007] Further, S2 includes the following steps: Based on the standardized inspection dataset, spatially registered visible light images and infrared temperature images are extracted and input into the visual branch and infrared branch of a dual-stream convolutional neural network, respectively, to perform multi-scale convolutional feature extraction and obtain visual feature maps and thermal feature maps. Based on the aforementioned visual feature map and thermal feature map, a cross-modal attention mechanism is used in the shared feature layer to calculate the correlation weights of the spatial dimension and the channel dimension, perform feature alignment and weighted fusion, and obtain a cross-modal fused feature vector. The cross-modal fusion feature vector is input into the fully connected layer and the batch normalization layer, and feature dimensionality reduction and distribution normalization are performed to obtain the fusion representation vector. Based on the fused representation vector, the probability distribution of defect categories for each equipment component is calculated through the classification output layer, and the defect identification results and corresponding confidence scores are output through the Softmax function.

[0008] Further, S3 includes the following steps: Based on the defect identification results and confidence scores, historical inspection sequence data of the corresponding equipment components in multiple inspection cycles are extracted, and the timestamps, sensor sampling frequencies and data dimensions are unified to obtain standardized time-series input samples. The standardized time-series input samples are input into a time-series prediction model based on a long short-term memory network. The dependency relationship between time steps is dynamically modeled through the input gate, forget gate, and output gate to extract the time-series feature information of health status in the inspection data. The time-series feature information is input into a fully connected computing layer, where feature aggregation and nonlinear mapping are performed, and the predicted health trend of the device components over several future inspection cycles is output. Predictive alarm data is generated based on the predicted health trend values ​​and the defect identification results of the current inspection cycle.

[0009] Furthermore, generating predictive alarm data based on the predicted health trend value and the defect identification results of the current inspection cycle includes the following steps: Based on the predicted health trend values, sliding window analysis and weighted average processing are performed on the health status data of each inspection cycle to obtain trend characteristic data reflecting the rate of change in equipment health. Based on the trend feature data, threshold detection based on the rate of health change is performed to obtain predictive alarm data.

[0010] Furthermore, the execution policy update includes the following steps: Based on the flight attitude data, meteorological parameters, confidence scores and predictive alarm data collected during the inspection process, the environmental state is feature extracted and normalized to construct a state vector, which is then input into the state layer of the reinforcement learning model. Based on the state vector, an action space is established with secondary shooting and trajectory adjustment as candidate actions. The Q-value of the state-action pair is estimated using a value evaluation network, and the immediate reward is calculated by combining the reward function to obtain the comprehensive benefit of the current action. Based on the comprehensive benefits, a policy gradient update algorithm is adopted to perform backpropagation and iterative optimization on the policy network parameters. The historical state, action and reward samples are retrained through the experience replay mechanism to output the trajectory optimization command.

[0011] Furthermore, the reward function is as follows: ; in, This represents the instantaneous reward value at time t; The increase in confidence level for defect identification before and after the action is performed at time t; Let be the normalized value of flight energy consumption at time t; The normalized value of the time taken for track adjustment or shooting action at time t; This is a constraint violation. , , and These are positive weighting coefficients.

[0012] Further, S5 includes the following steps: Based on the trajectory optimization instructions, the flight control parameters of the UAV are analyzed and mapped, and the trajectory adjustment parameters, shooting angle parameters and speed correction amount are converted into executable control signals. The control signal drives the UAV to perform a secondary inspection, and monitors the flight attitude and execution feedback in real time.

[0013] Furthermore, S5 also includes synchronizing the inspection results of the secondary inspection to the personnel terminal for alarm.

[0014] Furthermore, the alarms include graphical visual alarms and audio alarms.

[0015] The beneficial effects of this invention are as follows: First, it utilizes visible light images, infrared temperature data, point cloud data, flight attitude information, and meteorological parameters simultaneously collected by a UAV during its power line inspection flight. Through time synchronization, spatial registration, and noise filtering, a standardized inspection dataset is constructed, ensuring the consistency of different modal information in space and time from the data source level. A dual-stream convolutional neural network structure is used to extract features from the visible light and infrared images separately. A cross-modal attention mechanism is introduced into the shared feature layer to achieve deep alignment and fusion of multimodal information, thereby overcoming the accuracy fluctuation problem of single-modal recognition in complex environments and improving the stability and robustness of defect detection in power transmission line components. After defect identification, a time-series prediction model is further used to model the temporal correlation and trend extrapolation of historical inspection sequence data of equipment components, generating future health trend values ​​and predictive alarm data to achieve early warning of potential risks. Finally, this solution combines flight attitude, meteorological parameters, identification confidence, and predictive alarm data to construct a reinforcement learning state vector, establish an action space centered on flight path adjustment and secondary shooting, and implement policy updates through a reward function aimed at improving defect confidence and minimizing energy consumption, thereby achieving adaptive optimization of UAV inspection paths and intelligent re-inspection of high-risk locations. Attached Figure Description

[0016] Figure 1 This is a flowchart of the steps of a deep learning-based unmanned aerial vehicle (UAV) inspection method for overhead power transmission lines in this invention.

[0017] Figure 2 This is a flowchart of the steps for performing policy updates in this invention. Detailed Implementation

[0018] Please see Figures 1-2 As shown, this invention relates to a method for unmanned aerial vehicle (UAV) inspection of overhead power transmission lines based on deep learning, comprising the following steps: S1. Based on the visible light images, infrared temperature data, point cloud data, flight attitude information and meteorological parameters collected synchronously by the UAV during the line inspection flight, and after performing time synchronization, spatial registration and noise filtering processing, a standardized inspection dataset is obtained. S2. Input the registered visible light image and infrared temperature image in the inspection dataset into the visual branch and infrared branch of the dual-stream convolutional neural network, respectively. Perform feature alignment and fusion in the shared feature layer based on the cross-modal attention mechanism, and output the defect identification result and confidence score of the equipment component. S3. Based on the defect identification results and confidence scores, input the historical inspection sequence data of the equipment components into the time series prediction network, perform time correlation modeling and trend extrapolation, obtain the health trend values ​​of the equipment components for several future periods, and generate predictive alarm data. S4. Based on the flight attitude data, meteorological parameters, confidence scores and predictive alarm data collected during the inspection process, construct the state vector of reinforcement learning, establish the action space with the trajectory adjustment of the secondary shooting as the candidate action, and execute the policy update with the reward function aimed at improving the defect confidence and minimizing energy consumption to obtain the trajectory optimization instruction. S5. Drive the UAV to perform a secondary inspection according to the trajectory optimization command.

[0019] In some embodiments, 110 kV–220 kV overhead transmission lines in mountainous-plain mixed terrain are selected as pilot projects. The UAV platform is equipped with a visible light camera, an uncooled infrared imaging module, a 16-line mechanical lidar, an IMU, and a meteorological micro-station (wind speed, temperature, humidity, irradiance). Microsecond-level timestamp synchronization is achieved through a unified PTP clock and hardware triggering. During the maiden voyage of the line inspection, the system operates along the conductor direction at a cruising speed of 25–28 m / s and an altitude of 50–60 m, synchronously acquiring visible light, infrared, attitude, and meteorological data at a trigger frequency of 2–3 Hz, while simultaneously acquiring point cloud scans at 10 Hz. To reduce errors caused by aircraft shaking and sunlight flicker, pipeline preprocessing employs temporal Kalman filtering and line-of-sight stabilization for image stabilization, combined with multi-sensor spatial registration based on extrinsic matrices: using the visible light camera as the primary coordinate system, infrared images are radiometrically corrected and then registered at the sub-pixel level using the thermal-visible light mapping relationship obtained from a calibration board; point clouds are first filtered out for statistical outliers (the proportion threshold is typically 5%) and resampled using voxels (voxel side length 0.05–0.1 m), and then projected onto the visible light plane using pose calculation results, thus constructing a standardized inspection dataset of "image—thermal image—point cloud—attitude / weather". Unlike existing schemes based solely on single-modal images, this standardized dataset maintains cross-modal spatiotemporal consistency within the same coordinate system, significantly reducing registration errors introduced by wind sway, viewing angle changes, and temperature fluctuations. In the defect identification stage, a dual-stream convolutional neural network architecture is adopted: the visual branch uses an improved pyramid feature network to extract multi-scale texture and edge information of insulators, spacers, fittings, and conductor surfaces, while the infrared branch uses shallow-deep hybrid convolution to capture hot spots, cold spots, and temperature gradients along the line. Both branches introduce a cross-modal attention mechanism in the shared feature layer, with attention key-value pairs derived from thermal and visible light features respectively, to address the bidirectional imbalance problem of "visually significant but thermally inconspicuous" and "significant thermal anomalies but lack of visual detail." To avoid information redundancy caused by traditional splicing or simple weighting, the implementation uses channel and spatial dual-domain attention gating to suppress redundant channels and strengthen cross-domain consistent defect candidate regions. The model is trained on a self-built sample library, covering all seasons, dawn / dusk, and backlighting conditions, with a positive-to-negative sample ratio of approximately 1:2. Robustness is enhanced through multi-scale random cropping, color temperature perturbation, and infrared radiation calibration. During online inference, each visible-infrared pair is fused and outputs a defect category (such as insulator cracks, flashover signs, overheating of fittings, conductor ablation, etc.) and a confidence score. Unlike the existing static process of "identification equals alarm", this embodiment uses the identification result and its confidence score as input triggers for time series prediction. At the ground station dispatch terminal, the historical inspection sequence of components (usually covering the last 3-6 inspection cycles, including comparisons within the same season) is uniformly resampled and missing data is compensated before being fed into the time series prediction model based on a Long Short-Term Memory (LSTM) network.The model uses input gates, forget gates, and output gates to separate and model the slow degradation and short-term fluctuations across cycles, outputting health trend values ​​for the next N inspection cycles (N=3–8, configurable), and generating predictive alarm data by combining the trend slope and fluctuation persistence. This embodiment no longer fixes the alarm threshold to a single temperature rise or single-frame appearance anomaly, but introduces a "trend-intensity-persistence" judgment, allowing minor but persistent anomalies to be identified earlier than the critical point, significantly reducing the sensitivity of traditional schemes to misjudgments of one-off extreme conditions. In the trajectory decision-making stage, the embodiment encodes flight attitude (roll / pitch / yaw rate of change), meteorological parameters (gusts, visibility, irradiance), component-level confidence gain expectations, and predictive alarm data into state vectors, and establishes a parameterizable action space containing only two categories: "secondary shooting" and "track fine-tuning" (e.g., adjusting lateral offset by 0.5–2 m, changing pitch angle by 2–6°, closing altitude by 5–15 m, and adding 3–5 camera poses). The policy-value joint reinforcement learning framework evaluates the benefits of each action online at airborne and ground-based collaborative inference nodes. Benefits consider both the improvement in defect confirmation and costs such as energy consumption, time consumption, and crosswind risk. Lightweight real-time optimization is achieved through policy gradient updates. Unlike existing workflows of "preset route + manual go-around," this embodiment forms a closed-loop control of "identification—prediction—decision—re-collection": when the policy network determines that a component has visual obstruction or thermal radiation interference under the current lighting or wind direction, the system immediately issues a trajectory optimization command, enabling the UAV to complete directional re-inspection within the same sortie; if the confidence level significantly improves after the second collection and the trend alarm remains effective, the component is marked as a high-priority object and written into the maiden flight plan of subsequent batches.

[0020] Further, S1 includes the following steps: The drone simultaneously collects visible light images, infrared temperature data, point cloud data, flight attitude information, and meteorological parameters using its multimodal sensors, and performs synchronous calibration according to timestamps to obtain initial multi-source data. Distortion correction and brightness normalization are performed on the visible light image in the initial multi-source data, drift compensation and outlier pixel removal are performed on the infrared temperature data, outlier filtering and density resampling are performed on the point cloud data, and noise-suppressed multi-source corrected data are obtained. Based on the attitude parameters and extrinsic parameter matrices in the multi-source correction data, sensor spatial registration and projection alignment processing are performed to map the infrared image and point cloud data to the visible light image coordinate system, thereby obtaining a standardized inspection dataset.

[0021] In some embodiments, a sampling clock offset model for each sensor is first established. Millisecond-level synchronization is achieved through hardware trigger signals and PTP (Precision Time Protocol). Using the IMU attitude calculation results as the time axis, asynchronous frames are resampled using linear interpolation to ensure the consistency of multimodal data in the time domain. This process achieves parameter calibration by minimizing the timestamp offset residual function, thereby solving the time drift problem caused by differences in sensor sampling frequencies. In the image and point cloud data preprocessing stage, for uneven illumination and lens distortion in visible light images, geometric correction is performed using camera calibration parameters based on a multinomial model, and brightness normalization is achieved by combining histogram equalization and Gamma adaptive adjustment. For infrared temperature data, a pixel response curve is established through a multi-frame sliding window, and linear regression compensation is performed on the drift component. At the same time, a pixel anomaly detection algorithm based on statistical distribution is used to remove noise points. After point cloud data is filtered by Euclidean distance to remove outliers, a resampling method based on density kernel estimation is used to achieve sparse-dense balance while maintaining the geometric structure, making the point cloud distribution more suitable for the subsequent projection mapping process. In the spatial registration stage, the rotation matrix and translation vector obtained from the UAV attitude calculation are used as initial extrinsic parameters to construct a rigid body transformation model from radar to camera. First, feature point matching is used to obtain the correspondence between the visible light image and the point cloud in 2D-3D space. An improved RANSAC algorithm is used to eliminate mismatched samples, and a high-precision extrinsic parameter matrix is ​​obtained through least-squares optimization. Then, a perspective projection model is used to map the 3D coordinates of the point cloud onto the visible light image plane, achieving the fusion of depth and texture information. For infrared images, a mutual information-based image registration algorithm is used to calculate the transformation matrix between the infrared and visible light channels, ensuring that the thermal radiation intensity field and the visible texture field overlap in the same coordinate system. Throughout the data standardization process, the algorithm introduces a spatiotemporal constraint consistency criterion, simultaneously minimizing time synchronization error and spatial projection error, to obtain the globally optimal registration parameters through joint optimization. Finally, a standardized inspection dataset with temporal alignment, spatial uniformity, and noise suppression is generated.

[0022] Further, S2 includes the following steps: Based on the standardized inspection dataset, spatially registered visible light images and infrared temperature images are extracted and input into the visual branch and infrared branch of a dual-stream convolutional neural network, respectively, to perform multi-scale convolutional feature extraction and obtain visual feature maps and thermal feature maps. Based on the aforementioned visual feature map and thermal feature map, a cross-modal attention mechanism is used in the shared feature layer to calculate the correlation weights of the spatial dimension and the channel dimension, perform feature alignment and weighted fusion, and obtain a cross-modal fused feature vector. The cross-modal fusion feature vector is input into the fully connected layer and the batch normalization layer, and feature dimensionality reduction and distribution normalization are performed to obtain the fusion representation vector. Based on the fused representation vector, the probability distribution of defect categories for each equipment component is calculated through the classification output layer, and the defect identification results and corresponding confidence scores are output through the Softmax function.

[0023] It should be noted that the dual-stream convolutional neural network takes the standardized inspection dataset generated in step S1 as input, and first extracts spatially registered visible light images and infrared temperature images from it. To fully capture the feature level differences between different modalities, the visual branch employs an improved residual convolution structure. In the lower-level convolution stages, edge and texture features are extracted using small-scale kernels (3×3), and in the higher-level convolution stages, dilated convolutions (dilation=2,4) are used to expand the receptive field, enhancing the perception of large-scale structural defects (such as insulator damage and conductor sagging). The infrared branch, considering the high continuity of thermal field feature distribution, uses lightweight deep separable convolution to extract temperature gradient and thermal anomaly distribution features, and inserts channel attention modules in the middle layers to enhance local thermal anomaly responses, resulting in a thermal feature map.

[0024] In the shared feature layer, a cross-modal attention mechanism is introduced to achieve dynamic alignment of visual and infrared features. This mechanism constructs key and value matrices from the infrared feature channels and a query matrix from the visible light feature channels, respectively, and uses dot-product attention to calculate the response correlation of different modalities in spatial and channel dimensions. The algorithm achieves adaptive enhancement of salient regions by aligning the weight matrices; that is, when there is a temperature anomaly in the infrared image but weak visual texture, the model automatically increases the attention weight of thermal features; conversely, when there are obvious geometric defects in the visible light image but no significant temperature deviation, the feature contribution of the visual channel is increased. To avoid feature drift caused by modal conflicts, the model adopts a two-layer normalization mechanism in the fusion stage: channel normalization suppresses feature amplitude differences, and spatial normalization maintains local consistency. Finally, a cross-modal fused feature vector is formed by weighted summation to achieve information synergy at the feature level. The fused feature vector is mapped to a high-dimensional feature space through a fully connected layer, and the feature distribution is standardized by batch normalization, thereby accelerating convergence and suppressing gradient oscillation problems in mini-batch training. The normalized fusion representation vector is input to the classification output layer. The output dimension corresponds to the categories of key components of the transmission line (such as insulators, fittings, conductors, spacers, etc.) and their possible defect types (such as cracks, loosening, overheating, corrosion, flashover signs, etc.). The classification layer uses the Softmax function to transform the output into a probability distribution, calculates the confidence score for each type of defect, and selects the category with the highest probability as the final identification result. During the training phase, the model uses a cross-entropy loss function combined with confidence regularization constraints to minimize the KL divergence between the predicted probability distribution and the true label, thereby improving the model's classification certainty. Simultaneously, a multi-scale input strategy is used, i.e., the same target region is repeatedly sampled at different resolutions for feature enhancement, reducing the sensitivity of the feature layer to scale changes. Compared with traditional single-modal convolutional networks or simple feature concatenation methods, this method achieves explicit correlation modeling at the feature layer through a cross-modal attention mechanism, enabling the model to achieve robust defect identification under complex lighting, background interference, and partial occlusion conditions.

[0025] Further, S3 includes the following steps: Based on the defect identification results and confidence scores, historical inspection sequence data of the corresponding equipment components in multiple inspection cycles are extracted, and the timestamps, sensor sampling frequencies and data dimensions are unified to obtain standardized time-series input samples. The standardized time-series input samples are input into a time-series prediction model based on a long short-term memory network. The dependency relationship between time steps is dynamically modeled through the input gate, forget gate, and output gate to extract the time-series feature information of health status in the inspection data. The time-series feature information is input into a fully connected computing layer, where feature aggregation and nonlinear mapping are performed, and the predicted health trend of the device components over several future inspection cycles is output. Predictive alarm data is generated based on the predicted health trend values ​​and the defect identification results of the current inspection cycle.

[0026] In some embodiments, historical inspection sequence data of specific equipment components are first extracted from the defect identification results and confidence scores obtained in the preceding steps. This sequence covers information such as feature timestamps, image feature statistics, infrared temperature difference values, and structural status labels across multiple inspection cycles. To eliminate the time mismatch problem caused by inconsistent sampling frequencies and data dimensions of different sensors in multiple inspections, the embodiment employs a time normalization algorithm to resample the data at equal intervals and uses Z-score standardization to balance the numerical feature distribution, thereby generating standardized time-series input samples in a unified format. In terms of model structure, the LSTM network serves as the core time-series prediction module, capable of maintaining state transfer and dynamically learning long-range dependencies between time steps. The input gate controls the degree of influence of the current inspection data on the hidden state, the forget gate adjusts the retention ratio of old states, and the output gate is responsible for outputting the phased health trend vector. Through this gated recurrent mechanism, the model can capture the slow evolution pattern from minor anomalies to obvious defects in long-term inspection sequences. For example, when the insulator temperature rises linearly over multiple cycles while the image texture contrast gradually decreases, the model can identify potential degradation trends from the cumulative dependence of time steps. Even if the confidence change in a single inspection is not significant, it can still output a health trend decay signal. The temporal features output by the LSTM are aggregated and nonlinearly mapped through a fully connected layer. The ReLU activation function is used to enhance the nonlinear expressive power and improve the model's fusion effect on multimodal inputs, ultimately generating health trend prediction values ​​for several future inspection cycles. To improve the stability of the prediction, the system introduces a sliding window mechanism and a multi-step feedforward prediction strategy during the training phase, enabling the model to learn both short-term fluctuations and long-term trends simultaneously, thereby effectively addressing local biases caused by environmental disturbances or measurement anomalies. When the health trend prediction results are combined with the defect identification results of the current cycle, the algorithm generates predictive alarm data through trend difference analysis. If the rate of decline of the trend prediction value exceeds a set threshold or the slope of the trend curve remains negative, the system automatically marks it as a "warning state" and outputs the alarm type and intensity level to prompt maintenance personnel to intervene and inspect in advance. Unlike traditional alarm triggering methods based on static thresholds, this embodiment uses time-related modeling and a recursive state update mechanism to make alarms evolve and be forward-looking, enabling early detection before defects reach a critical level.

[0027] Furthermore, generating predictive alarm data based on the predicted health trend value and the defect identification results of the current inspection cycle includes the following steps: Based on the predicted health trend values, sliding window analysis and weighted average processing are performed on the health status data of each inspection cycle to obtain trend characteristic data reflecting the rate of change in equipment health. Based on the trend feature data, threshold detection based on the rate of health change is performed to obtain predictive alarm data.

[0028] In some embodiments, firstly, a sliding time window is constructed for the predicted health trends over multiple consecutive inspection cycles. The window length is adaptively adjusted based on the equipment type and inspection frequency (e.g., 3–5 cycles for insulators and 5–8 cycles for conductors or fittings). Within the window, a weighted average is applied to the health score sequence at each time point, with newer times assigned higher weights to highlight the sensitivity to short-term anomalies. An exponential decay model is used for the weighting function to ensure the algorithm can respond quickly when detecting sudden changes while maintaining a smooth suppression effect on normal fluctuations. After sliding window processing, trend feature data characterizing the rate of change in health status is obtained. This feature can quantify the equipment degradation rate and stability to identify potential signs of accelerated failure. Subsequently, threshold detection based on the rate of change is performed on the trend feature data. Unlike traditional single threshold judgment, this embodiment adopts a two-layer adaptive threshold strategy: the first-layer threshold is set by statistically analyzing the standard deviation of health scores in the historical distribution to identify the initial boundary of abnormal fluctuations; the second-layer threshold is dynamically adjusted in conjunction with the range of defect confidence fluctuations within the inspection cycle, constituting a refined constraint on the judgment condition. When the rate of change of the trend exceeds the dynamic threshold and the increase in confidence is insufficient to offset the health degradation, the model automatically determines that the component has entered a potential abnormal state and generates predictive alarm data, including alarm level, trend direction and affected component type information.

[0029] Furthermore, the execution policy update includes the following steps: Based on the flight attitude data, meteorological parameters, confidence scores and predictive alarm data collected during the inspection process, the environmental state is feature extracted and normalized to construct a state vector, which is then input into the state layer of the reinforcement learning model. Based on the state vector, an action space is established with secondary shooting and trajectory adjustment as candidate actions. The Q-value of the state-action pair is estimated using a value evaluation network, and the immediate reward is calculated by combining the reward function to obtain the comprehensive benefit of the current action. Based on the comprehensive benefits, a policy gradient update algorithm is adopted to perform backpropagation and iterative optimization on the policy network parameters. The historical state, action and reward samples are retrained through the experience replay mechanism to output the trajectory optimization command.

[0030] In some embodiments, an architecture based on collaborative operation between airborne and ground-based edge nodes is adopted. In each decision cycle, flight attitude data (roll / pitch / yaw and their rates of change, airspeed and altitude), meteorological parameters (instantaneous and gust wind speed, wind direction, visibility, irradiance), defect identification confidence and its relative improvement, and predictive alarm data (component-level risk level, trend slope and duration) are time-aligned and input into the feature engineering module to complete normalization, outlier suppression and dimensionality compression, and construct a state vector containing environmental feasibility, imaging quality expectations and risk priorities. At the same time, energy consumption and remaining range estimates from the flight control side are injected as cost features to form a state representation that can be directly fed into the policy network and value network. The action space is parameterized into a combination of two major categories of atomic actions: "secondary shooting" and "track adjustment". The former includes continuous parameters such as shooting distance, fuselage pitch angle, camera gimbal tilt angle and exposure level, while the latter includes continuous variables such as lateral shift, altitude fine-tuning and heading angle correction. To ensure executability and safety, actions are filtered through flight control constraints (battery threshold, no-fly zone boundary, maximum tilt / overload limit) and meteorological feasible domains before generation. If necessary, the system backtracks to the nearest feasible solution in the neighborhood to avoid outputting "high-gain but unflyable" commands. The value evaluation network estimates the Q-value of state-action pairs, assessing the combined benefit of different actions in the same state. The immediate reward consists of the gain from recognition confidence, the normalization cost of energy and time consumption, and the penalty for constraint violation. During training, temporal difference objective-stabilized value learning is used, while during inference, a hybrid exploration combining ε-greedy and entropy regularization is employed to maintain the ability to explore uncertain scenarios. The policy network (Actor) outputs the parameter distribution of continuous actions, and the value network (Critic) provides advantage estimation to reduce variance. In the online phase, a mini-batch replay buffer carries the most recent N decision segments for intermittent incremental updates, and a target network or delayed update mechanism suppresses divergence caused by non-stationary distributions. To adapt to the dual constraints of energy efficiency and schedule in power line inspections, a cost-sensitive sampling reweighting method is introduced during the training process: when the remaining flight range is low or the gusts are strong, the weight of the corresponding sample in the replay is increased, enabling the strategy to converge to a robust solution more quickly in the "low remaining energy / severe weather" domain. Taking "crosswind 7–9 m / s, strong afternoon radiation, and slight thermal anomaly in the insulator string" as an example, the defect confidence level obtained from the previous flight path is 0.62, and the trend model gives a medium-level warning for this component. The current state vector shows an increased probability of lateral jitter and an increased risk of image flare. The value network gives a lower reward for "direct passage" and a higher reward for the composite action of "lateral shift 0.8–1.2 m + decrease in flight altitude 8–12 m + gimbal negative bias 3–5° + moderate exposure reduction". The strategy network samples the composite action accordingly, and after filtering through the feasible domain, it sends it to the flight control. The confidence level of the secondary data acquisition and transmission is increased to 0.81, the energy consumption increment is controlled within 7% of the flight mileage, the immediate reward is significantly positive, and the experience fragment is written into the playback buffer for the next update.Unlike the existing process of "static route + manual interpretation + go-around", this embodiment completes "identification - evaluation - re-inspection - relearning" in a closed loop within a single flight, and achieves an interpretable trade-off between energy consumption, time consumption and reliability through cost-sensitive strategy updates.

[0031] Furthermore, the reward function is as follows: ; in, This represents the instantaneous reward value at time t; The increase in confidence level for defect identification before and after the action is performed at time t; Let be the normalized value of flight energy consumption at time t; The normalized value of the time taken for track adjustment or shooting action at time t; This is a constraint violation. , , and These are positive weighting coefficients.

[0032] It should be noted that the reward function primarily uses the increase in defect identification confidence as the positive return. When the UAV improves the identification result and increases confidence by adjusting its flight path or performing secondary imaging, the model receives a higher positive reward. Simultaneously, normalized cost terms are set for flight energy consumption and action execution time to penalize high-energy-consuming or low-efficiency actions, ensuring the UAV's energy utilization and mission continuity in complex environments. Furthermore, to ensure flight safety and mission stability, if an action violates constraints, such as insufficient battery power, excessive attitude tilt, or weather conditions exceeding permissible limits, a constraint penalty term is introduced, providing a negative reward for that action. Finally, four weighting coefficients control the relative influence of confidence improvement, energy consumption, time, and safety constraints on the reward, enabling the reinforcement learning model to adaptively adjust its decision preferences at different task stages. For example, in high-risk defect areas, the system automatically increases the weight of identification accuracy, while in long-range, low-risk areas, it tends to favor the energy-optimal strategy. Through this reward design, the UAV can achieve a dynamic balance between identification accuracy, energy consumption, and operational efficiency during inspection, significantly improving the overall intelligence level and environmental adaptability of the inspection process.

[0033] Further, S5 includes the following steps: Based on the trajectory optimization instructions, the flight control parameters of the UAV are analyzed and mapped, and the trajectory adjustment parameters, shooting angle parameters and speed correction amount are converted into executable control signals. The control signal drives the UAV to perform a secondary inspection, and monitors the flight attitude and execution feedback in real time.

[0034] In some embodiments, the system first receives a trajectory optimization command generated by a reinforcement learning module. This command includes multiple continuous control parameters, including a trajectory adjustment vector, a shooting attitude correction angle, and a flight speed correction. To ensure that the command can be directly executed by the UAV flight control system, the algorithm decomposes and semantically maps the command vector through a control parameter parser, converting the high-level policy output into a low-level flight control parameter set. For example, when the reinforcement learning model outputs a combined command of "reducing altitude by 10 meters, shifting to the left by 0.8 meters, adjusting the gimbal pitch angle by 5°, and decelerating by 15%", the system automatically converts it into the desired altitude, displacement compensation, and angle increment inputs under a PID control framework, and encapsulates them into executable control signals via the flight control bus. During the execution phase, the UAV performs a secondary inspection task based on the generated control signals. The flight control system monitors attitude changes, speed response, and energy consumption fluctuations in real time at a frequency of 20–50 Hz, while simultaneously performing closed-loop corrections for trajectory deviations and gimbal attitude. To ensure flight stability in complex weather conditions, the system introduces an attitude fusion algorithm based on Extended Kalman Filter (EKF), which dynamically corrects flight state estimates by combining IMU data and GNSS positioning information. This ensures that the UAV maintains safe attitude constraints and smooth transitions when performing trajectory adjustments. During the secondary inspection, the image acquisition module simultaneously triggers multimodal sensors to achieve refined imaging and angle compensation sampling for high-confidence defect areas. After the inspection is completed, the system transmits real-time feedback on flight attitude, energy consumption, and image quality. The decision layer evaluates the effectiveness of the action, such as the improvement rate of recognition confidence, the coverage of the re-inspected area, and changes in energy consumption. If the feedback results deviate significantly from the model's expectations, empirical samples are written into the reinforcement learning replay buffer to optimize subsequent policy updates.

[0035] Furthermore, S5 also includes synchronizing the inspection results of the secondary inspection to the personnel terminal for alarm.

[0036] Furthermore, the alarms include graphical visual alarms and audio alarms.

[0037] Specifically, after the UAV completes a secondary inspection of high-risk areas based on flight path optimization commands, the defect category, confidence score, health trend change rate, and geographic coordinate information output by the identification model are encapsulated into an inspection result data packet and uploaded to the ground workstation in real time via a 5G or self-organizing network communication link. The system performs data parsing and format standardization on the server side and synchronizes the results to the personnel monitoring interface.

[0038] On the personnel side, alarm information is presented and alerted in two ways: firstly, through graphical visualization alarms, the system marks the location of abnormal components on the digital map of the transmission line using heat overlay, and simultaneously displays a fused view of the identified image in visible light and infrared light, allowing maintenance personnel to intuitively judge the type and severity of defects; secondly, through an audio alarm mechanism, when the predictive alarm level reaches a preset threshold (such as high risk or a continuous trend of abnormality), the system triggers audio prompts and flashing warnings to provide immediate risk alerts. In addition, the personnel interface can directly retrieve multimodal raw images from secondary inspections, along with relevant attitude and meteorological parameters, and supports manual review and correction of the identification results.

[0039] The above embodiments are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A deep learning-based unmanned aerial vehicle inspection method for overhead power transmission lines, characterized in that, The method comprises the following steps: S1, based on the visible light image, infrared temperature data, point cloud data, flight attitude information and meteorological parameters synchronously collected by the unmanned aerial vehicle during the line patrol flight, time synchronization, space registration and noise filtering processing are performed to obtain a standardized inspection data set; S2, the registered visible light image and infrared temperature image in the inspection data set are respectively input into the visual branch and infrared branch of the double-flow convolutional neural network, feature alignment and fusion are performed based on the cross-modal attention mechanism in the shared feature layer, and the defect recognition result and confidence score of the equipment component are output; S3, according to the defect recognition result and confidence score, the historical inspection sequence data of the equipment component is input into the time prediction network, time correlation modeling and trend extrapolation are performed, the health trend value of the equipment component in the future several periods is obtained, and predictive alarm data is generated; S4, based on the flight attitude data, meteorological parameters, confidence score and predictive alarm data collected during the inspection, a state vector of reinforcement learning is constructed, an action space taking the second shooting track adjustment as the candidate action is established, and a reward function taking the defect confidence score improvement and the minimum energy consumption as the goal is used to perform strategy update to obtain a track optimization instruction; S5, according to the track optimization instruction, the unmanned aerial vehicle is driven to perform secondary inspection. 2.The overhead transmission line unmanned aerial vehicle inspection method based on deep learning according to claim 1, characterized in that, The S1 comprises the following steps: The visible light image, infrared temperature data, point cloud data, flight attitude information and meteorological parameters are synchronously collected by the multi-modal sensor carried by the unmanned aerial vehicle, and are synchronously calibrated according to the time stamp to obtain initial multi-source data; The visible light image in the initial multi-source data is subjected to distortion correction and brightness normalization processing, the infrared temperature data is subjected to drift compensation and abnormal pixel elimination, and the point cloud data is subjected to outlier filtering and density resampling to obtain multi-source corrected data subjected to noise suppression; Based on the attitude parameters and extrinsic parameter matrix in the multi-source corrected data, sensor space registration and projection alignment processing are performed, the infrared image and point cloud data are mapped to the visible light image coordinate system, and a standardized inspection data set is obtained. 3.The overhead transmission line unmanned aerial vehicle inspection method based on deep learning of claim 1, wherein, The S2 comprises the following steps: Based on the standardized inspection data set, the spatially registered visible light image and infrared temperature image are extracted and input into the visual branch and infrared branch of the double-flow convolutional neural network, multi-scale convolution feature extraction is performed, and visual feature maps and thermal feature maps are obtained; Based on the visual feature maps and thermal feature maps, the correlation weights of the spatial dimension and the channel dimension are calculated in the shared feature layer through the cross-modal attention mechanism, feature alignment and weighted fusion are performed, and a cross-modal fusion feature vector is obtained; The cross-modal fusion feature vector is input into the full connection layer and batch normalization layer, feature dimension reduction and distribution standardization processing are performed, and a fusion representation vector is obtained; Based on the fusion representation vector, the probability distribution of each equipment component defect category is calculated through the classification output layer, and the defect recognition result and the corresponding confidence score are output through the Softmax function. 4.The overhead transmission line unmanned aerial vehicle inspection method based on deep learning according to claim 1, characterized in that, The S3 comprises the following steps: Based on the defect identification result and the confidence score, historical inspection sequence data of the corresponding equipment component in multiple inspection cycles is extracted, and timestamps, sensor sampling frequencies and data dimensions are unified to obtain standardized time series input samples; The standardized time series input samples are input into a time series prediction model based on a long short-term memory network, the dependence relationship between time steps is dynamically modeled through input gates, forget gates and output gates, and the time series feature information of the health state in the inspection data is extracted; The time series feature information is input into a fully connected calculation layer to perform feature aggregation and non-linear mapping processing, and output the health trend prediction value of the equipment component in the future several inspection cycles; Predictive alarm data is generated according to the health trend prediction value and the defect identification result of the current inspection cycle.

5. The deep learning-based unmanned aerial vehicle inspection method for overhead transmission lines according to claim 4, characterized in that, Generating predictive alarm data according to the health trend prediction value and the defect identification result of the current inspection cycle includes the following steps: Based on the health trend prediction value, sliding window analysis and weighted average processing are performed on the health state data of each inspection cycle to obtain trend feature data reflecting the health change rate of the equipment; According to the trend feature data, threshold detection based on the health change rate is performed to obtain predictive alarm data. 6.The overhead transmission line unmanned aerial vehicle inspection method based on deep learning according to claim 1, characterized in that, The execution strategy update includes the following steps: Based on the flight attitude data, weather parameters, confidence scores and predictive alarm data collected during the inspection process, feature extraction and normalization processing are performed on the environmental state, a state vector is constructed, and the state layer of the reinforcement learning model is input; According to the state vector, an action space is established with the second shooting and the flight path adjustment as candidate actions, the Q value of the state-action pair is estimated using a value evaluation network, and the instantaneous reward is calculated combined with a reward function to obtain the comprehensive income of the current action; Based on the comprehensive income, a policy gradient update algorithm is used to perform back propagation and iterative optimization on the policy network parameters, and the historical state, action and reward samples are retrained through an experience replay mechanism to output flight path optimization instructions.

7. The unmanned aerial vehicle inspection method for overhead transmission lines based on deep learning according to claim 6, characterized in that, The reward function is as follows: ； wherein, is an instant reward value at time t; is an increase in defect recognition confidence before and after action execution at time t; is a flight energy consumption normalization value at time t; is a flight path adjustment or shooting action time consumption normalization value at time t; is a constraint violation term; , , and is a positive weight coefficient. 8.The overhead transmission line unmanned aerial vehicle inspection method based on deep learning according to claim 1, characterized in that, The S5 includes the following steps: Based on the flight path optimization instructions, the flight control parameters of the unmanned aerial vehicle are analyzed and mapped to instructions, and the flight path adjustment parameters, shooting angle parameters and speed correction amount are converted into executable control signals; According to the control signals, the unmanned aerial vehicle performs secondary inspection, and real-time monitoring of flight attitude and execution feedback is performed. 9.The overhead transmission line unmanned aerial vehicle inspection method based on deep learning according to claim 8, characterized in that, The S5 also includes synchronizing the inspection results of the secondary inspection to the personnel terminal for alarm. 10.The overhead transmission line unmanned aerial vehicle inspection method based on deep learning according to claim 9, characterized in that, The alarm includes graphical visual alarm and audio alarm.

Images