Unmanned aerial vehicle infrared temperature measurement system for power transmission line inspection

Abstract

The invention discloses an unmanned aerial vehicle infrared temperature measurement system for power transmission line inspection, and aims to improve the efficiency and safety of power transmission line inspection. The system is composed of an unmanned aerial vehicle platform provided with a real-time dynamic positioning assembly, an infrared thermal imaging camera, a visible light camera and a multi-mode sensor, and accurate flight positioning and effective obstacle avoidance are ensured. The infrared thermal imaging camera is responsible for collecting infrared images to measure temperature, and the visible light camera is used for collecting visible light images to assist in recognizing structure and environment information. The intelligent flight control unit in the system carries out flight path planning and obstacle identification according to sensor data, and the communication module supports real-time transmission of the data. And the data processing and analyzing unit further analyzes the acquired image by using a deep learning model, identifies a key part and measures the temperature of the key part. The operation control software platform provides a setting interface for a user, permits setting of inspection parameters, and generates an inspection report containing key information after inspection is completed.

Description

Technical Field

[0001] The present invention relates to the technical field of power transmission line inspection, and in particular to an unmanned aerial vehicle (UAV) infrared temperature measurement system for power transmission line inspection. Background Art

[0002] In existing technologies, transmission line inspections primarily rely on visual inspections performed by ground inspectors or traditional drone technology. While these methods have improved inspection efficiency to a certain extent, they still have several significant limitations. First, ground inspectors have difficulty reaching transmission lines at high altitudes or in complex terrain, and inspections are time-consuming and labor-intensive. While traditional drone technology can overcome terrain limitations and provide high-altitude inspection services, most rely solely on visible light cameras, making comprehensive condition assessments difficult. This is particularly true for identifying early-stage damage and abnormal temperature rise in critical areas such as transmission line insulators.

[0003] Furthermore, traditional methods for measuring transmission line temperature suffer from insufficient accuracy. Temperature is a key indicator of transmission line health, but traditional infrared temperature measurement equipment often requires operation at specific distances and angles, making it challenging for mobile platforms like drones. Furthermore, existing drones often lack effective obstacle detection and avoidance systems, making flight safety difficult to ensure. Furthermore, due to a lack of efficient data processing and analysis capabilities, the vast amount of collected data cannot be analyzed and processed immediately, delaying the provision of decision support.

[0004] Therefore, it is very necessary to develop a new type of drone infrared temperature measurement system for transmission line inspection. Summary of the Invention

[0005] This application provides a drone infrared temperature measurement system for power transmission line inspection to improve the efficiency and safety of power transmission line inspection.

[0006] This application provides a drone infrared temperature measurement system for power transmission line inspection, comprising:

[0007] The UAV platform includes a real-time dynamic positioning component, an infrared thermal imaging camera, a visible light camera, and a multimodal sensor module. The real-time dynamic positioning component is used to collect position information to ensure the UAV's accurate flight positioning. The infrared thermal imaging camera is used to collect infrared images of transmission lines to measure their temperature. The visible light camera collects visible light images to identify the structure and environmental information of the transmission lines. The multimodal sensor module includes a laser radar and an ultrasonic sensor to collect sensor data to achieve obstacle detection and avoidance, ensuring the UAV's safe flight.

[0008] The intelligent flight control unit is used to identify and avoid obstacles based on the sensor data transmitted by the drone platform; and to plan the flight path based on the inspection parameters sent by the communication module and the location information transmitted by the drone platform;

[0009] A communication module, supporting 4G / 5G and satellite communications, is used to send image data and location information collected by the drone platform to the data processing and analysis unit; receive inspection parameters sent by the control software platform, and send the inspection parameters to the intelligent flight control unit;

[0010] A data processing and analysis unit is configured to receive image data and location information sent by the communication module; analyze the image data using a trained deep learning model to identify key locations of the transmission line; measure and analyze the temperature of the key locations to obtain temperature measurement data and analysis results; and transmit the image data, location information, temperature measurement data, and analysis results to an operation control software platform; wherein the key locations include conductors, towers, and insulators;

[0011] The operation control software platform is used to provide users with a setting interface, which allows users to set inspection parameters, including inspection areas, specific points of interest, and inspection frequencies; send the inspection parameters set by the user to the communication module; during the inspection process, display the drone's location, flight status, image data collected by the drone, and temperature measurement data of the transmission line to the user; after the inspection is completed, generate an inspection report to provide decision support for the maintenance team, wherein the inspection report includes the location of temperature anomalies and analysis results.

[0012] Furthermore, the real-time dynamic positioning component includes a global positioning system receiver and a real-time dynamic positioning module to improve the accuracy and stability of drone positioning.

[0013] Furthermore, the multimodal sensor module also includes a wind speed sensor for real-time monitoring and adjusting the flight speed of the drone to ensure safe flight under different wind speed conditions.

[0014] Furthermore, the intelligent flight control unit includes an obstacle recognition and avoidance mechanism based on a deep reinforcement learning model. The deep reinforcement learning model uses a two-layer network architecture. The first layer is a convolutional neural network, which is used to process and analyze image data from the multimodal sensor module to achieve rapid recognition of the surrounding environment and obstacle location. The second layer is a long short-term memory network, which is responsible for analyzing the change of obstacle position information provided by the convolutional neural network over time, predicting the movement trend of obstacles and the optimal flight strategy of the drone relative to obstacles.

[0015] The deep reinforcement learning model implements obstacle recognition and avoidance through the following steps:

[0016] Using environmental data collected in real time by a multimodal sensor module as input to a deep reinforcement learning model; wherein the environmental data includes infrared images, visible light images, and lidar point cloud data;

[0017] The convolutional neural network layer processes input data, extracts features of obstacles, and locates the precise position of obstacles; the convolutional neural network layer is composed of a convolutional layer, a pooling layer, and a fully connected layer;

[0018] The long short-term memory network layer receives the obstacle location information output by the convolutional neural network layer and combines it with past flight data to predict the movement trend of the obstacle and the optimal flight strategy of the drone relative to the obstacle, where the optimal flight strategy includes the estimated obstacle avoidance direction;

[0019] The reward function provided by the following formula 1 guides the optimization of the drone's flight strategy:

[0020]

[0021] Where R(t) represents the reward value at time point t, which aims to maximize the safety and flight efficiency of the drone when it circumvents obstacles; d(t) is the distance between the drone and the nearest obstacle; d0 is the normalization factor of the distance, which is used to adjust the influence of the distance; v(t) is the current flight speed; v opt is the optimal flight speed; θ(t) is the angle between the UAV’s flight direction and the expected obstacle avoidance direction; α, β, and γ are weight parameters used to balance the influence of each part in the reward function.

[0022] Furthermore, the intelligent flight control unit performs path planning by executing the following steps:

[0023] Based on the drone's current location and received inspection area parameters, a graph model is constructed. The graph model uses the drone's current location as the starting point and each specific point of interest within the inspection area as the target node. Each node in the graph model represents a potential location point, and each edge represents a possible path from one node to another. The cost of an edge consists of two parts: the actual distance from the current node to the next node, and the estimated distance from the current node to the final target node.

[0024] The A* algorithm is used to search the constructed graph model. The A* algorithm determines the search direction by calculating the total cost f(n) of each node in the graph model, where f(n) = g(n) + h(n), g(n) represents the actual cost from the starting point to the current node n, and h(n) represents the estimated cost from the current node n to the target node. The path with the lowest total cost is preferred.

[0025] Monitor the environmental data provided by the multimodal sensor module. When new obstacles are detected, adjust the nodes and edges in the graph model accordingly to avoid the flight path being too close to the obstacles.

[0026] Furthermore, the deep learning model used by the data processing and analysis unit includes a data preprocessing layer, a feature extraction layer, a reinforcement learning layer, and an output decoding layer;

[0027] The data preprocessing layer is used to preprocess the infrared image and visible light image provided by the UAV platform to obtain preprocessed infrared image and visible light image; wherein the preprocessing includes using image normalization technology to scale the image pixel values ​​to the range of [0, 1] to unify the brightness and contrast levels of different images; applying a Gaussian filter to remove noise and smooth the image to reduce image noise; and enhancing contrast by adjusting the image histogram;

[0028] The feature extraction layer uses a depthwise separable convolutional network structure to process the preprocessed infrared image and visible light image to obtain feature representations of key parts of the transmission line. The depthwise separable convolutional network structure includes depthwise convolution and pointwise convolution. The depthwise convolution applies a separate filter to the input image to extract local features. The pointwise convolution combines the outputs of the depthwise convolution in the depth direction to form a higher-level feature representation. The feature representation of the key parts of the transmission line includes shape, size, and texture information.

[0029] The reinforcement learning layer combines the graph convolutional network and the reinforcement learning algorithm to process the feature representation output by the feature extraction layer to obtain the predicted position of the key parts and the dynamically adjusted recognition strategy;

[0030] The output decoding layer classifies the existence probability of the key parts according to the predicted positions of the key parts provided by the reinforcement learning layer using the softmax function, and applies non-maximum suppression to process overlapping prediction boxes to eliminate repeated detections, thereby obtaining the classification probability of the key parts and the final determined position coordinates.

[0031] Furthermore, the reinforcement learning layer employs a framework that combines a graph convolutional network and a deep reinforcement learning algorithm. The graph convolutional network uses the feature representations output by the feature extraction layer to construct a graph structure, wherein the nodes in the graph structure represent the key parts provided by the feature extraction layer, and the edges reflect the spatial relationships between the key parts. The graph convolutional network captures and integrates the relationships between the key parts by performing convolution operations on the graph structure, and outputs an updated feature representation for each node.

[0032] The deep reinforcement learning algorithm uses the updated feature representation provided by the graph convolutional network as state input; using the set reward mechanism, it gives the model positive or negative feedback based on the accuracy of the prediction and the efficiency of the recognition strategy; the agent in the deep reinforcement learning algorithm calculates an action strategy through the policy network based on the current state and reward, aiming to dynamically adjust the recognition strategy to optimize the performance of the deep reinforcement learning model.

[0033] Furthermore, the data processing and analysis unit is specifically used to perform temperature measurement and analysis on the identified key parts; wherein, temperature measurement is achieved by converting the thermal radiation intensity in the infrared image into a temperature value; the analysis process includes comparing the measured temperature value with a preset temperature threshold to determine whether each key part is overheated or has a temperature abnormality.

[0034] This application has the following beneficial technical effects:

[0035] (1) By integrating real-time dynamic positioning components with multimodal sensor modules, the system ensures that the drone can accurately perform flight positioning and effectively detect and avoid obstacles, significantly improving the safety of the inspection process.

[0036] (2) The combined use of infrared thermal imaging cameras and visible light cameras not only allows for temperature measurement of transmission lines, but also provides structural and environmental information, making inspections more comprehensive and accurate. This is particularly important in the early detection of potential faults and abnormal temperature rises, helping to prevent larger equipment failures and power supply interruptions.

[0037] (3) The introduction of the intelligent flight control unit enables the drone to automatically plan the optimal flight path based on preset inspection parameters and real-time environmental information, further improving inspection efficiency and coverage.

[0038] (4) The use of data processing and analysis units, which utilize advanced deep learning technology to intelligently analyze the collected image data, can accurately identify key parts of the transmission line and perform temperature measurements. This process greatly reduces the need for manual intervention and improves the speed and accuracy of data processing.

[0039] (5) The operation control software platform provides users with an intuitive interface, making the setting of inspection tasks simple and easy. At the same time, it can display the inspection progress and results in real time and generate inspection reports containing detailed analysis and maintenance recommendations in a timely manner, greatly enhancing the decision-making support capabilities of the power grid operation and maintenance team. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a schematic diagram of a drone infrared temperature measurement system for power transmission line inspection provided in the first embodiment of the present application. DETAILED DESCRIPTION

[0041] The following description sets forth many specific details to facilitate a thorough understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of the present application. Therefore, the present application is not limited to the specific implementations disclosed below.

[0042] The first embodiment of the present application provides a drone infrared temperature measurement system for power transmission line inspection. Figure 1 , which is a schematic diagram of the first embodiment of this application. Figure 1 The first embodiment of the present application provides a detailed description of a drone infrared temperature measurement system for power transmission line inspection.

[0043] The UAV infrared temperature measurement system for power transmission line inspection includes a UAV platform 101, an intelligent flight control unit 102, a communication module 103, a data processing and analysis unit 104 and an operation control software platform 105.

[0044] The UAV platform 101 includes a real-time dynamic positioning component, an infrared thermal imaging camera, a visible light camera, and a multimodal sensor module. The real-time dynamic positioning component is used to collect position information to ensure the accurate flight positioning of the UAV. The infrared thermal imaging camera is used to collect infrared images of the transmission line to measure the temperature of the transmission line. The visible light camera collects visible light images to identify the structure and environmental information of the transmission line. The multimodal sensor module includes a lidar and an ultrasonic sensor, which are used to collect sensor data to achieve obstacle detection and avoidance, ensuring the safe flight of the UAV.

[0045] In this embodiment, UAV platform 101 is designed as a highly integrated device specifically for performing power line inspections and infrared temperature measurement tasks. UAV platform 101 includes multiple components designed to enable efficient, accurate, and safe inspections of power lines.

[0046] First, the UAV Platform 101 is equipped with a real-time dynamic positioning component, which utilizes high-precision positioning technologies such as RTK (Real-Time Kinematic Positioning) to provide the UAV with precise location information. This technology enables the UAV to maintain stable flight in complex environments, ensuring precise movement along predetermined inspection routes, thereby significantly improving inspection accuracy and efficiency.

[0047] Secondly, the infrared thermal imaging camera is a core component of the UAV Platform 101. It captures infrared images of power transmission lines and their surroundings. By analyzing these images, it can accurately measure the temperature of various parts of the transmission lines. This technology is crucial for early detection of potential faults and abnormal hot spots on transmission lines, enabling timely maintenance measures and avoiding potential safety hazards.

[0048] UAV Platform 101 also includes a visible light camera that captures visible light images to identify the structure and environmental information of the transmission line. This information is valuable for analyzing the physical condition of the transmission line, its surrounding environment, and factors that may affect the line.

[0049] Another key technology in the UAV Platform 101 is its multimodal sensor module, which includes lidar and ultrasonic sensors. These sensors collect real-time environmental data, enabling the detection and identification of obstacles ahead. Based on this data, the drone can automatically adjust its flight path to avoid collisions, ensuring safe flight in complex environments.

[0050] Furthermore, the real-time dynamic positioning component includes a global positioning system receiver and a real-time dynamic positioning module to improve the accuracy and stability of drone positioning.

[0051] The real-time dynamic positioning component consists of two main parts: a Global Positioning System (GPS) receiver and a Real-Time Kinematic (RTK) module. The GPS receiver receives signals from the Global Positioning System to determine the drone's approximate position on the Earth's surface. While traditional GPS positioning provides high positioning accuracy, in certain situations, such as in complex terrain or in environments with dense buildings, GPS signals may be interfered with, affecting positioning accuracy.

[0052] To address this issue, the component further integrates an RTK module. RTK is an enhanced GPS-based positioning technology that provides centimeter-level positioning accuracy. It receives GPS signals from fixed reference stations and performs differential processing on them compared to the GPS signals received by the drone, significantly improving positioning accuracy and stability. This high-precision positioning capability is crucial for ensuring the drone's precise flight along its planned path and accurately locating key areas of power transmission lines for temperature measurement.

[0053] To implement this real-time dynamic positioning component, knowledge of GPS and RTK technologies is required, including an understanding of GPS signal processing, RTK differential technology, and practical experience integrating these technologies into UAV platforms. Furthermore, an understanding of how to process the data generated by the RTK module is required to ensure that the UAV's flight control system can utilize this highly accurate position information for effective flight path planning and obstacle avoidance.

[0054] The introduction of this component significantly improves the positioning accuracy and stability of drones when performing transmission line inspection tasks, thereby ensuring the efficiency and safety of the inspection tasks.

[0055] Furthermore, the multimodal sensor module also includes a wind speed sensor for real-time monitoring and adjusting the flight speed of the drone to ensure safe flight under different wind speed conditions.

[0056] The multimodal sensor module includes components such as lidar and ultrasonic sensors for obstacle detection and avoidance. By adding a wind speed sensor to this module, the drone system can monitor wind speed changes in the surrounding environment in real time. The wind speed sensor works by measuring the change in air velocity relative to the drone, thereby obtaining current wind speed data.

[0057] This data is crucial for drone systems because it directly impacts the drone's flight stability and safety. When encountering high wind conditions, the system receives wind speed data through the intelligent flight control unit 102 and adjusts the drone's flight speed or attitude accordingly to offset the effects of wind speed and ensure stable flight. For example, if it detects an increase in headwind, the system may increase the drone's propulsion to maintain the intended flight path and speed.

[0058] To achieve this function, the wind speed sensor needs to be tightly integrated with the drone's flight control system. This requires the ability to integrate sensors, collect and process data, and adjust flight parameters based on real-time data.

[0059] Furthermore, the selection and placement of wind speed sensors are crucial for ensuring their functionality. The sensors need to be sensitive enough to detect subtle wind speed changes while also being stable enough to avoid misreading in complex wind conditions. Their installation location must also consider their impact on the drone's aerodynamic characteristics and potential interference with other drone functions, such as camera vision.

[0060] Based on the above, it should be possible to implement a technical solution that incorporates wind speed sensors to improve the stability and safety of drone flights in varying wind speed conditions. This capability significantly improves the reliability of drones performing transmission line inspections, ensuring they can safely and effectively complete inspections in a changing natural environment.

[0061] The design of the drone platform 101 in this embodiment takes into account the actual needs of modern power system operation and maintenance, and realizes the automation and intelligence of transmission line inspection through highly integrated technical components.

[0062] The intelligent flight control unit 102 is used to identify and avoid obstacles based on the sensor data transmitted by the drone platform; and to plan the flight path based on the inspection parameters sent by the communication module and the location information transmitted by the drone platform.

[0063] In the drone infrared temperature measurement system provided in this embodiment, the intelligent flight control unit 102 plays a crucial role. It serves as the brain of the system, coordinating data processing and command execution among the various sensors and execution modules on the drone platform 101. This unit is specifically designed to improve the efficiency, safety, and accuracy of power transmission line inspections.

[0064] The intelligent flight control unit 102 is based on an advanced microprocessor and specialized flight control algorithms. It receives and processes data from the drone platform 101's real-time dynamic positioning component, multimodal sensor modules (including lidar and ultrasonic sensors), and image information from infrared thermal imaging cameras and visible light cameras. This data is crucial for the drone to identify and avoid obstacles, ensuring safe flight in complex environments.

[0065] When planning a flight path, the intelligent flight control unit 102 uses inspection parameters from the communication module 103, combined with the drone's current location information, to calculate an optimal flight path. This calculation takes into account the geographic characteristics of the inspection area, specific points of interest, and the obstacle avoidance paths required for flight safety. Advanced algorithms built into the intelligent flight control unit 102 enable real-time updates to the flight path to respond to unexpected situations, such as unexpected obstacles or weather changes.

[0066] The intelligent flight control unit 102 also works closely with the system's communication module 103 to ensure that image data and location information collected by the drone platform 101 are transmitted in real time to the data processing and analysis unit 104. It is also responsible for receiving inspection parameters from the operation control software platform 105 and adjusting the flight plan accordingly.

[0067] Furthermore, the intelligent flight control unit includes an obstacle recognition and avoidance mechanism based on a deep reinforcement learning model. The deep reinforcement learning model uses a two-layer network architecture. The first layer is a convolutional neural network, which is used to process and analyze image data from the multimodal sensor module to achieve rapid recognition of the surrounding environment and obstacle location. The second layer is a long short-term memory network, which is responsible for analyzing the change of obstacle position information provided by the convolutional neural network over time, predicting the movement trend of obstacles and the optimal flight strategy of the drone relative to obstacles.

[0068] The deep reinforcement learning model implements obstacle recognition and avoidance through the following steps:

[0069] Using environmental data collected in real time by a multimodal sensor module as input to a deep reinforcement learning model; wherein the environmental data includes infrared images, visible light images, and lidar point cloud data;

[0070] The convolutional neural network layer processes input data, extracts features of obstacles, and locates the precise position of obstacles; the convolutional neural network layer is composed of a convolutional layer, a pooling layer, and a fully connected layer;

[0071] The long short-term memory network layer receives the obstacle location information output by the convolutional neural network layer and combines it with past flight data to predict the movement trend of the obstacle and the optimal flight strategy of the drone relative to the obstacle.

[0072] The reward function provided by the following formula 1 guides the optimization of the drone's flight strategy:

[0073]

[0074] Where R(t) represents the reward value at time point t, which aims to maximize the safety and flight efficiency of the drone when it bypasses obstacles; d(t) is the distance between the drone and the nearest obstacle, d0 is the normalization factor of the distance, which is used to adjust the influence of the distance; v(t) is the current flight speed, v optis the optimal flight speed; θ(t) is the angle between the UAV's flight direction and the expected obstacle avoidance direction; α, β, and γ are weight parameters used to balance the influence of each part in the reward function. 4. The UAV infrared temperature measurement system according to claim 1 is characterized in that the intelligent flight control unit includes an obstacle recognition and avoidance mechanism, and the obstacle recognition and avoidance mechanism is implemented based on a deep reinforcement learning model; the deep reinforcement learning model adopts a two-layer network architecture, the first layer is a convolutional neural network, which is used to process and analyze image data from the multimodal sensor module, so as to achieve rapid recognition of the surrounding environment and obstacle positioning; the second layer is a long short-term memory network, which is responsible for analyzing the change of obstacle position information provided by the convolutional neural network over time, predicting the movement trend of obstacles and the optimal flight strategy of the UAV relative to obstacles;

[0075] The deep reinforcement learning model implements obstacle recognition and avoidance through the following steps:

[0076] Using environmental data collected in real time by a multimodal sensor module as input to a deep reinforcement learning model; wherein the environmental data includes infrared images, visible light images, and lidar point cloud data;

[0077] The convolutional neural network layer processes input data, extracts features of obstacles, and locates the precise position of obstacles; the convolutional neural network layer is composed of a convolutional layer, a pooling layer, and a fully connected layer;

[0078] The long short-term memory network layer receives the obstacle location information output by the convolutional neural network layer and, combined with past flight data, predicts the movement trend of the obstacle and the optimal flight strategy of the UAV relative to the obstacle, wherein the optimal flight strategy includes the estimated obstacle avoidance direction;

[0079] The reward function provided by the following formula 1 guides the optimization of the drone's flight strategy:

[0080]

[0081] Where R(t) represents the reward value at time point t, which aims to maximize the safety and flight efficiency of the drone when it bypasses obstacles; d(t) is the distance between the drone and the nearest obstacle, d0 is the normalization factor of the distance, which is used to adjust the influence of the distance; v(v) is the current flight speed, v opt is the optimal flight speed; θ(t) is the angle between the UAV’s flight direction and the expected obstacle avoidance direction; α, β, and γ are weight parameters used to balance the influence of each part in the reward function.

[0082] The following are detailed steps and code writing guidelines for implementing this mechanism, which are intended to ensure that those skilled in the art can implement it according to this description.

[0083] First, environmental data is collected by multiple components on the drone. These include an infrared camera for capturing infrared images, a visible light camera for acquiring visible light images, and a lidar for mapping the three-dimensional shapes of surrounding objects. This data is transmitted in real time to the intelligent flight control unit and serves as input to the deep reinforcement learning (DRL) model.

[0084] Next, the first layer of the DRL model is a convolutional neural network (CNN), which is responsible for processing the input image data. CNN extracts features from the image and locates obstacles through a series of convolutional layers, pooling layers, and fully connected layers.

[0085] The following code can be used as a reference during implementation:

[0086] import tensorflow as tf

[0087] from tensorflow.keras import layers,models

[0088] def create_cnn_model(input_shape):

[0089] model = models.Sequential()

[0090] model.add(layers.Conv2D(32,(3,3),activation='relu',input_shape=input_shape))

[0091] model.add(layers.MaxPooling2D((2,2)))

[0092] model.add(layers.Conv2D(64,(3,3),activation='relu'))

[0093] model.add(layers.MaxPooling2D((2,2)))

[0094] model.add(layers.Conv2D(128,(3,3),activation='relu'))

[0095] model.add(layers.Flatten())

[0096] model.add(layers.Dense(512,activation='relu'))

[0097] model.add(layers.Dense(2,activation='linear'))#output the location of the obstacle

[0098] return model

[0099] The Long Short-Term Memory (LSTM) layer then receives the obstacle location information output by the CNN layer and combines it with past flight data to predict the movement trend of obstacles and the optimal flight path for the drone. The implementation of the LSTM layer can be based on the following reference code:

[0100] def create_lstm_model(input_shape):

[0101] model = models.Sequential()

[0102] model.add(layers.LSTM(128,return_sequences=True,input_shape=input_shape))

[0103] model.add(layers.LSTM(128))

[0104] model.add(layers.Dense(512,activation='relu'))

[0105] model.add(layers.Dense(2, activation='linear'))#Output optimal flight path and strategy

[0106] return model

[0107] Finally, the reward function optimizes the drone's flight strategy based on the relative distance between the drone and the obstacle, the flight speed, and the flight direction. The reward function can be used to guide the learning of the drone's flight strategy during the model training process:

[0108] def calculate_reward(d,v,theta,alpha,beta,gamma,d_0,v_opt):

[0109] reward=-alpha*tf.exp(-d / d_0)+beta*(v_opt-tf.abs(v-v_opt))+gamma*tf.exp(theta)

[0110] return reward

[0111] By combining the aforementioned CNN and LSTM implementations and utilizing a defined reward function, a complete deep reinforcement learning model can be constructed to implement the obstacle recognition and avoidance mechanism in the intelligent flight control unit. This system enables drones to safely and efficiently perform power transmission line inspections in complex environments.

[0112] First, the Deep Reinforcement Learning (DRL) model is based on a convolutional neural network (CNN) and a long short-term memory network (LSTM) to implement the obstacle recognition and avoidance mechanism. This model works through the following steps:

[0113] 1. Environmental data input: Environmental data provided by the UAV platform, including infrared images, visible light images, and lidar point cloud data, is used as input to the model. This data contains important information about the surrounding environment and potential obstacles.

[0114] 2. Convolutional Neural Network Layer: In this layer, the input environmental data is used to extract the features of obstacles and locate their precise positions. Through a combination of convolutional layers, pooling layers, and fully connected layers, CNN is able to extract useful features from the original image and identify obstacles.

[0115] 3. Long Short-Term Memory Network Layer: Next, the LSTM layer uses the obstacle location information output by the CNN layer and combines it with past flight data to predict obstacle movement trends and the drone's optimal flight path relative to the obstacles. By memorizing past state information, the LSTM layer can make accurate predictions in dynamically changing environments.

[0116] 4. Application of Reward Function: The model optimizes the drone's flight strategy using a designed reward function. This reward function comprehensively considers the distance between the drone and the obstacle, the difference between the current flight speed and the optimal flight speed, and the adjustment of flight direction. It aims to maximize the safety and efficiency of the drone's flight as it navigates around obstacles.

[0117] In the reinforcement learning framework, the reward function is a criterion for evaluating the quality of a given action, which directly affects the decision-making during the learning process. The following is a more detailed explanation of the use of this reward function in the context of obstacle recognition and avoidance for drones:

[0118] Environment and Agent: In this scenario, the drone is considered the "agent" in reinforcement learning, and its surrounding environment includes obstacles, flight area, etc. The agent (drone) receives the state of the environment through sensors (such as infrared images, visible light images, and lidar point cloud data).

[0119] Action: The agent (drone) can perform multiple actions at each time point t, which affect the drone's flight direction, speed, etc. The choice of which action depends in part on the guidance of the reward function.

[0120] Reward function: The reward function R(t) calculates a numerical reward based on the current state of the environment and the actions taken by the agent. The goal of this function is to evaluate the effectiveness of the drone's obstacle avoidance strategy, including both safety and efficiency.

[0121] Safety is determined by the distance d(t) between the drone and the obstacle. The longer the distance, the higher the safety, and therefore the higher the reward.

[0122] The efficiency is mainly determined by the flight speed v(t) of the UAV and the optimal flight speed v opt The difference is used to evaluate whether the flight direction of the UAV is close to the expected obstacle avoidance direction.

[0123] Policy Update: Based on the reward value returned by the reward function, the reinforcement learning algorithm (such as deep Q learning, policy gradient, etc.) updates its policy, learning which action to choose in a given state to maximize the total future reward. In this process, the drone "learns" how to adjust its flight strategy based on the current environment to optimize safety and efficiency.

[0124] 5. Iterative Learning: By continuously interacting with the environment, receiving feedback (rewards), and updating its strategy, the drone gradually learns and improves its obstacle recognition and avoidance capabilities. This process typically involves numerous iterations, with each iteration bringing the drone's behavior closer to the optimal strategy.

[0125] Combined with reinforcement learning, the design and application of reward functions are key to enabling drones to autonomously learn how to efficiently and safely navigate obstacles during flight. In this way, drones not only remain safe in complex environments but can also adjust their flight strategies based on specific mission requirements, such as reaching their destination quickly.

[0126] Overall, this deep reinforcement learning model combines the power of CNNs and LSTMs with a carefully designed reward function to achieve efficient and accurate obstacle recognition and avoidance strategies. Training and applying this model requires large amounts of data and complex computations, but ultimately significantly improves the drone's ability to fly autonomously in complex environments.

[0127] The reward function provided by the following formula 1 guides the optimization of the drone's flight strategy:

[0128]

[0129] Where R(t) represents the reward value at time point t, which aims to maximize the safety and flight efficiency of the drone when it avoids obstacles;

[0130] d(t) is the distance between the UAV and the nearest obstacle, which is measured in real time by a multimodal sensor module, such as a lidar;

[0131] d0 is the normalization factor of the distance, which is used to adjust the influence of the distance. It is a fixed value obtained through experiments or simulations and aims to optimize the model performance.

[0132] v(t) is the current flight speed, which is provided in real time by the UAV’s flight control system;

[0133] v opt is the optimal flight speed, which is pre-set based on flight conditions and safety standards;

[0134] θ(t) is the angle between the UAV’s flight direction and the expected obstacle avoidance direction;

[0135] α, β, and γ are weight parameters used to balance the influence of various parts in the reward function, which are obtained through experiments or directly obtained from expert knowledge.

[0136] The following provides a more systematic implementation description, covering the construction of the deep reinforcement learning model architecture, data preprocessing, training strategy, reward function implementation, environment simulation and testing, and the specific steps and methods for performance evaluation.

[0137] Model Architecture:

[0138] Convolutional Neural Network (CNN): Design a network consisting of multiple convolutional layers, pooling layers, and at least one fully connected layer. Each convolutional layer uses ReLU as the activation function to increase nonlinear processing capabilities, while the final fully connected layer aims to convert the learned features into specific location information of obstacles.

[0139] Long Short-Term Memory (LSTM) network: Design an LSTM network that receives the output of the CNN and historical flight data as input. This network is responsible for learning the movement trends of obstacles and the ability to dynamically adjust flight strategies based on the current environment.

[0140] Data preprocessing:

[0141] Format the collected infrared images, visible light images, and lidar point cloud data into a format that the network can process. This may include steps such as image resizing, normalization, and conversion to tensors. To accommodate the CNN input requirements, ensure that all image data has the same size and color channels.

[0142] Training strategy:

[0143] Use labeled datasets to train CNNs to learn to identify and localize obstacles. These datasets should contain examples of various obstacle types and different environmental conditions.

[0144] The LSTM is trained using historical flight data and obstacle dynamics data generated by simulation to enable the model to predict the future positions of obstacles and develop obstacle avoidance strategies.

[0145] Specific implementation of the reward function:

[0146] When implementing the reward function, the current state of the drone (including distance to obstacles, flight speed, and direction) is taken as input and a reward value is calculated. This reward value is used to adjust the model's weights, biasing the model towards producing a more optimal flight strategy.

[0147] Environmental simulation and testing:

[0148] Develop or use an existing virtual environment simulator that can simulate various flight conditions and obstacle scenarios. Train and test the DRL model in this simulated environment to verify its obstacle avoidance capabilities.

[0149] Performance evaluation:

[0150] Define performance metrics, such as obstacle avoidance success rate, path optimization, and flight stability. Collect these performance data by testing the model in different environments and conditions, and analyze the overall effectiveness of the model.

[0151] Following these steps, developers will be able to write code to implement this deep reinforcement learning model using a specific programming language (such as Python) and a deep learning framework (such as TensorFlow or PyTorch). The specific implementation details of each step, such as network architecture configuration, data preprocessing methods, and hyperparameter settings during training, will directly impact the model's performance and efficiency. Therefore, developers need to carefully adjust and optimize these parameters based on the specific needs of the project and environmental conditions.

[0152] Furthermore, the intelligent flight control unit performs path planning by executing the following steps:

[0153] Based on the drone's current location and received inspection area parameters, a graph model is constructed. The graph model uses the drone's current location as the starting point and each specific point of interest within the inspection area as the target node. Each node in the graph model represents a potential location point, and each edge represents a possible path from one node to another. The cost of an edge consists of two parts: the actual distance from the current node to the next node, and the estimated distance from the current node to the final target node.

[0154] The A* algorithm is used to search the constructed graph model. The A* algorithm determines the search direction by calculating the total cost f(n) of each node in the graph model, where f(n) = g(n) + h(n), g(n) represents the actual cost from the starting point to the current node n, and h(n) represents the estimated cost from the current node n to the target node. The path with the lowest total cost is preferred.

[0155] Monitor the environmental data provided by the multimodal sensor module. When new obstacles are detected, adjust the nodes and edges in the graph model accordingly to avoid the flight path being too close to the obstacles.

[0156] First, the intelligent flight control unit builds a graph model based on the current location information of the drone and the received inspection area parameters. This graph model is the basis of the entire path planning process. It takes the current location of the drone as the starting point and defines each point of interest specified in the inspection mission as a target node. Each node included in the graph represents a spatial location point that the drone may fly over, and each edge represents a potential flight path between two nodes. In order to quantify the quality of these paths, a cost is assigned to each edge. The cost consists of two main parts: one is the actual flight distance from the current node to the next node, and the other is the estimated distance from the current node to the target node. The latter is used to predict the additional effort required to reach the target.

[0157] Next, the intelligent flight control unit uses the A* algorithm to conduct an in-depth search of this graph model to find the optimal flight path. A* is an efficient search algorithm that guides the search by calculating the total cost of each node in the graph: the actual cost g(n) from the starting point to the current node plus the estimated cost h(n) from the current node to the target node. By prioritizing the path with the lowest total cost, the A* algorithm efficiently determines the optimal route for the drone to fly from the starting point to each point of interest.

[0158] Furthermore, the intelligent flight control unit monitors environmental data provided by the multimodal sensor module in real time to detect new obstacles. When new obstacles are detected, the system instantly adjusts the nodes and edges in the graph model to prevent the planned flight path from coming too close to these obstacles. This dynamic adjustment mechanism ensures the safety of the drone during inspection missions and allows it to flexibly respond to changing environmental conditions.

[0159] In the intelligent flight control unit of a drone's infrared temperature measurement system, path planning is key to ensuring the drone completes its inspection mission safely and efficiently. This process not only involves planning the path from the current location to the inspection area but also dynamically adjusting the flight path based on real-time environmental data to address new obstacles. The following details how, when a new obstacle is detected, the intelligent flight control unit adjusts the nodes and edges in the graph model to avoid the flight path from being too close to the obstacle.

[0160] First, the multimodal sensor module, including lidar and ultrasonic sensors, continuously scans the drone's surroundings, detecting and identifying obstacles in real time. Once a new obstacle is identified, its location is immediately transmitted to the intelligent flight control unit.

[0161] After receiving the location information of the new obstacle, the intelligent flight control unit will perform the following steps to adjust the image model:

[0162] 1. Obstacle Zone Definition: Based on the size and shape of the obstacle, the intelligent flight control unit determines a safety buffer zone around the obstacle. This zone takes into account the drone's safe flight distance, ensuring that the drone can fly around the obstacle without getting too close.

[0163] 2. Node and Edge Adjustment: The intelligent flight control unit then examines the nodes and edges in the graph model, particularly those that intersect or are too close to the obstacle buffer zone. Nodes located within or on the edge of the obstacle buffer zone are marked as impassable or removed from the graph model. Edges connecting these nodes are also removed or re-adjusted to prevent the path planning algorithm from choosing these blocked paths.

[0164] 3. Path Replanning: Once the graph model is adjusted, the intelligent flight control unit re-executes the A* algorithm to search for a new path using the updated graph model. Because the graph model has eliminated paths to new obstacles, the re-planned flight path will safely bypass these obstacles, ensuring the drone's flight safety.

[0165] 4. Real-time Updates and Feedback: The intelligent flight control unit continuously receives real-time environmental data from the multimodal sensor module and dynamically adjusts the graph model. This process occurs continuously throughout the inspection flight, ensuring that the drone can flexibly respond to sudden obstacles in the environment.

[0166] Through these steps, the intelligent flight control unit can effectively respond to newly emerging obstacles in the environment, ensuring a safe and efficient drone flight path. Key to this process lies in real-time environmental monitoring and data analysis, as well as rapid and accurate graph model adjustments and path replanning. This method not only avoids known obstacles but also flexibly handles the challenges posed by newly detected obstacles, providing an effective obstacle avoidance strategy for drones.

[0167] Through these steps, the intelligent flight control unit provides an efficient and safe flight path planning solution for drones. This not only improves the efficiency and effectiveness of power line inspections, but also significantly reduces the potential risks of flight accidents.

[0168] The following is the reference implementation code for path planning:

[0169]

[0170]

[0171]

[0172] In this example, the Node class represents a node in the graph. The a_star_search function implements the A* search algorithm, which accepts a starting position, a goal position, and a list of obstacles as input and returns a path from the starting position to the goal position (if one exists). The adjust_path_for_obstacles function provides a simple framework that shows how to adjust a path when new obstacles are detected.

[0173] Please note that this is just an example. In actual applications, more factors need to be considered, such as the size and shape of the obstacle, and the safe distance between the drone and the obstacle.

[0174] The communication module 103 supports 4G / 5G and satellite communications. The communication module is used to send the image data and location information collected by the drone platform to the data processing and analysis unit; receive the inspection parameters sent by the control software platform, and send the inspection parameters to the intelligent flight control unit.

[0175] The communication module 103 plays a vital role in the UAV infrared temperature measurement system for power transmission line inspection provided in this embodiment. It is responsible for realizing data transmission and communication between various components in the system.

[0176] Communication module 103 supports multiple communication standards, including but not limited to 4G, 5G, and satellite communications, enabling UAV platform 101 to maintain real-time connectivity with data processing and analysis unit 104 and operation control software platform 105 in virtually any environment while performing power line inspections. This multi-standard support ensures flexible and reliable data transmission, particularly in remote areas or areas with patchy network coverage.

[0177] The communication module 103's primary functions include receiving and transmitting data. It receives inspection parameters from the operational control software platform 105. These parameters guide the drone's flight path planning, specific areas of interest, and inspection frequency. Once these parameters are received, the communication module 103 forwards them to the intelligent flight control unit 102, which uses them to plan the drone's flight path and execute inspection missions.

[0178] Meanwhile, the communication module 103 is responsible for transmitting key data collected by the drone platform 101 to the data processing and analysis unit 104. This data includes, but is not limited to, image data captured by the infrared thermal imaging camera and the visible light camera, as well as precise location information provided by the real-time dynamic positioning component. This data is the basis for identifying key parts of the transmission line and performing temperature measurement and analysis.

[0179] The internal design of the communication module 103 takes into account the needs of high efficiency and low energy consumption, ensuring continuous and stable communication capabilities for the system without adding excessive burden. In addition, the module uses advanced encryption technology to ensure the security of data transmission and prevent any unauthorized access or data leakage.

[0180] To implement the communication module 103, knowledge of wireless communications and network programming is required, including but not limited to an understanding of 4G, 5G, and satellite communication technologies, as well as the ability to implement a communication protocol stack on a microprocessor or corresponding hardware. In addition, knowledge of data encryption and secure transmission is also necessary.

[0181] In summary, communication module 103 provides a robust and flexible communication foundation for the transmission line inspection drone infrared temperature measurement system. This allows the drone to maintain real-time communication with the operational control software platform while performing its mission, while ensuring data security and integrity. The design and implementation of this module is crucial to the successful operation of the entire system.

[0182] The data processing and analysis unit 104 is configured to receive image data and location information sent by the communication module; analyze the image data using a trained deep learning model based on the image data to identify key locations of the transmission line; measure and analyze the temperature of the key locations to obtain temperature measurement data and analysis results; and transmit the image data, location information, temperature measurement data, and analysis results to the operation control software platform; wherein the key locations include conductors, towers, and insulators.

[0183] Data processing and analysis unit 104 plays a crucial role in the transmission line inspection drone infrared temperature measurement system provided in this embodiment. As the core of the system's intelligent analysis, it is responsible for processing and analyzing the image data and location information collected by drone platform 101, thereby enabling accurate identification, temperature measurement, and further analysis of key transmission line locations. The following describes data processing and analysis unit 104 in detail.

[0184] The data processing and analysis unit 104 is designed as an integrated software system deployed on a high-performance computing platform, which can be a server located in a ground control center or a cloud computing environment. It first receives image data and location information transmitted from the communication module 103. This data includes infrared images captured by an infrared thermal imaging camera for temperature measurement, and visible light images captured by a visible light camera for identifying the structure and environmental information of the transmission line. Furthermore, the precise location information provided by the real-time dynamic positioning component is crucial for locating key locations.

[0185] Next, the data processing and analysis unit 104 analyzes the received image data using pre-trained deep learning models. These deep learning models are specifically designed to identify key transmission line components, such as conductors, towers, and insulators. The model training process relies on a large amount of image data covering transmission line images under a variety of conditions, ensuring the accuracy and robustness of the model in practical applications.

[0186] After identifying key locations on the transmission line, the data processing and analysis unit 104 further measures and analyzes the temperatures of these locations to generate temperature data. This process not only relies on infrared images provided by the infrared thermal imaging camera but also incorporates visible light images and location information to improve the accuracy and reliability of temperature measurements.

[0187] Finally, the data processing and analysis unit 104 transmits the processed and analyzed results, including image data, location information, temperature measurement data, and analysis results, to the operation control software platform 105. These results are used to display real-time information about the inspection process to the user and generate a final inspection report, which includes the location and analysis results of temperature anomalies.

[0188] To implement the functions of the data processing and analysis unit 104, relevant knowledge and skills in machine learning, image processing, and data analysis are required. The design and training of deep learning models requires an understanding of modern artificial intelligence technologies, including convolutional neural networks (CNNs). Furthermore, achieving efficient data transmission and processing requires a deep understanding of data encoding, network communications, and high-performance computing.

[0189] Furthermore, the deep learning model used by the data processing and analysis unit includes a data preprocessing layer, a feature extraction layer, a reinforcement learning layer, and an output decoding layer;

[0190] The data preprocessing layer is used to preprocess the infrared image and visible light image provided by the UAV platform to obtain preprocessed infrared image and visible light image; wherein the preprocessing includes using image normalization technology to scale the image pixel values ​​to the range of [0, 1] to unify the brightness and contrast levels of different images; applying a Gaussian filter to remove noise and smooth the image to reduce image noise; and enhancing contrast by adjusting the image histogram;

[0191] The feature extraction layer uses a depthwise separable convolutional network structure to process the preprocessed infrared image and visible light image to obtain feature representations of key parts of the transmission line. The depthwise separable convolutional network structure includes depthwise convolution and pointwise convolution. The depthwise convolution applies a separate filter to the input image to extract local features. The pointwise convolution combines the outputs of the depthwise convolution in the depth direction to form a higher-level feature representation. The feature representation of the key parts of the transmission line includes shape, size, and texture information.

[0192] The reinforcement learning layer combines the graph convolutional network and the reinforcement learning algorithm to process the feature representation output by the feature extraction layer to obtain the predicted position of the key parts and the dynamically adjusted recognition strategy;

[0193] The output decoding layer classifies the existence probability of the key parts according to the predicted positions of the key parts provided by the reinforcement learning layer using the softmax function, and applies non-maximum suppression to process overlapping prediction boxes to eliminate repeated detections, thereby obtaining the classification probability of the key parts and the final determined position coordinates.

[0194] The reinforcement learning layer uses a framework that combines a graph convolutional network and a deep reinforcement learning algorithm. The graph convolutional network uses the feature representation output by the feature extraction layer to construct a graph structure, where the nodes in the graph structure represent the key parts identified by the feature extraction layer, and the edges reflect the spatial relationships between the key parts. The graph convolutional network captures and integrates the relationships between the key parts by performing convolution operations on the graph structure, and outputs an updated feature representation for each node.

[0195] The deep reinforcement learning algorithm uses the updated feature representation provided by the graph convolutional network as state input; using the set reward mechanism, it gives the model positive or negative feedback based on the accuracy of the prediction and the efficiency of the recognition strategy; the agent in the deep reinforcement learning algorithm calculates an action strategy through the policy network based on the current state and reward, aiming to dynamically adjust the recognition strategy to optimize the performance of the deep reinforcement learning model.

[0196] The data preprocessing layer plays a crucial role in preparing image data for subsequent image analysis and feature extraction stages.

[0197] First, the data preprocessing layer receives two types of image data from the drone platform: infrared and visible light images. Infrared images are used to capture the thermal distribution of power lines and their surroundings, while visible light images provide visual information about the environment. Both types of image data are extremely important, but in their raw form, they may not be suitable for direct use in deep learning model training and analysis, so preprocessing is required.

[0198] The preprocessing steps are divided into the following stages:

[0199] 1. Image normalization: This process involves scaling the image pixel values ​​to a uniform range, usually [0, 1]. This step is achieved by dividing each pixel value by 255 (the maximum possible pixel value) to unify the brightness and contrast levels of different images, making model training more stable.

[0200] 2. Noise Removal: Applying a Gaussian filter is a common noise removal technique that effectively smooths images and reduces noise introduced by factors such as sensor errors and environmental interference. By performing a low-pass filtering operation on the image, the Gaussian filter retains most important image details while removing small noise points.

[0201] 3. Contrast Enhancement: Adjusting the image's histogram is a method for enhancing image contrast, improving the visual quality and making key areas of the transmission line stand out more clearly. Contrast enhancement often involves histogram equalization or other advanced techniques to enhance low-contrast areas in the image, optimizing the overall image dynamic range.

[0202] Through these steps, the data preprocessing layer outputs images of higher quality and consistency, providing a solid foundation for feature extraction and subsequent image analysis. This processed image data significantly improves the accuracy and efficiency of the model's identification of key areas, ensuring that the drone-based infrared temperature measurement system can effectively perform transmission line inspections.

[0203] The following is the reference implementation code of the data preprocessing layer:

[0204]

[0205]

[0206] The above code first reads the infrared image and the visible light image, and then preprocesses them as follows:

[0207] Image normalization: Scale image pixel values ​​to the range [0, 1] to make model training more stable.

[0208] Noise Removal: Use a Gaussian filter to smooth the image and reduce noise.

[0209] Contrast enhancement: Increase image contrast through histogram equalization to make key areas more prominent.

[0210] After preprocessing, the image can be used for deep learning model training and analysis. Note that the cv2.equalizeHist function only works with single-channel images. For color images, you may need to convert them to grayscale or process each channel separately. In addition, in practical applications, you may also need to consider other preprocessing steps, such as image resizing and data augmentation, to suit specific models and tasks.

[0211] The feature extraction layer is responsible for extracting key information from the preprocessed image for subsequent analysis and identification of key parts of the transmission line.

[0212] The feature extraction layer uses a depthwise separable convolutional network structure to process the infrared and visible light images optimized by the data preprocessing layer. This network structure is unique in its efficient feature extraction capabilities, which maintains network performance while significantly reducing computational complexity and the number of model parameters.

[0213] The implementation of the depthwise separable convolutional network includes:

[0214] 1. Depthwise Convolution: This stage applies filters independently to each channel of the image to extract local features. Deep convolution effectively identifies essential image features such as texture, edges, and shape, which are crucial for subsequently identifying key parts of the transmission line. Each filter focuses on a specific image channel, reducing computational effort while preserving detail in feature extraction.

[0215] 2. Pointwise Convolution: Next, a pointwise convolution operation combines the outputs of the depthwise convolutions to form a more complex feature representation. This step, implemented by applying a 1×1 convolution kernel, aims to integrate the local features extracted in the previous step to generate a feature representation that represents higher-level abstract concepts. These high-level features are crucial for distinguishing between key components of a transmission line, such as conductors, towers, and insulators.

[0216] The output of the feature extraction layer is a set of high-dimensional feature representations that capture detailed information about the shape, size, and texture of key transmission line components. These feature representations provide rich input for the subsequent reinforcement learning layers, enabling the model to more accurately identify and locate key components.

[0217] The following is the reference implementation code of the feature extraction layer:

[0218]

[0219]

[0220] The above code defines a depthwise separable convolutional network (DSN) for extracting key information from a preprocessed image. The network first processes each channel of the image independently through depthwise convolutional layers to extract local features. Next, pointwise convolutional layers combine these local features depthwise to form a more complex feature representation. The network then uses a global average pooling layer to convert the feature map into a feature vector, which can be directly used for subsequent analysis or recognition tasks.

[0221] The reinforcement learning layer plays a crucial role. It not only identifies key parts of the transmission line from extracted features but also dynamically adjusts the recognition strategy to cope with complex and changing environments. This layer's implementation involves two core technologies: graph convolutional networks (GCNs) and deep reinforcement learning (DRL) algorithms.

[0222] After the feature extraction layer, high-dimensional feature representations of the key parts of the transmission line are obtained. These features include information such as shape, size, and texture. The reinforcement learning layer first processes these feature representations using a graph convolutional network (GCN). The GCN's design enables it to process graph-structured data. This is because the key parts of the transmission line and their relationships can be naturally represented as a graph, where nodes represent key parts and edges represent the spatial relationships between these parts. By performing convolution operations on this graph structure, the GCN can effectively integrate the relationship information between nodes and generate an updated feature representation for each key part. These feature representations not only contain information about the key part itself, but also incorporate information about the surrounding environment.

[0223] Next, the reinforcement learning layer introduces a deep reinforcement learning (DRL) algorithm to further process the GCN output. At this stage, the DRL algorithm represents the features processed by the GCN as environmental states and determines the optimal action strategy based on these states. Through exploration and exploitation mechanisms, the agent in the DRL algorithm learns how to take actions in a given environmental state to maximize a reward function, which is designed based on the accuracy of the prediction and the efficiency of the recognition strategy. Through continuous learning and adjustment, the DRL algorithm is able to optimize the recognition strategy of key parts to cope with various complex environmental changes.

[0224] The final output of the reinforcement learning layer is the predicted location and category of each key part, as well as a dynamic recognition strategy optimized for the current environment and model state. These outputs are passed to the output decoding layer, which uses the softmax function and non-maximum suppression (NMS) technique to determine the final classification and location coordinates of the key parts.

[0225] Through this method of combining graph convolutional networks and deep reinforcement learning algorithms, the enhanced learning layer not only improves the accuracy of recognition, but also enhances the model's adaptability to environmental changes, ensuring that in complex and changing environments, the drone infrared temperature measurement system can efficiently and accurately complete the inspection task of the transmission line.

[0226] The following is the reference implementation code of the reinforcement learning layer:

[0227]

[0228]

[0229] The above code provides a basic framework for reinforcement learning layers, including simplified implementations of graph convolutional networks (GCNs) and deep reinforcement learning (DRL) algorithms. In practice, the model architecture, parameters, and training process may require detailed adjustments based on the specific application scenario.

[0230] The output decoding layer follows the reinforcement learning layer and is responsible for converting the predicted position and recognition strategy provided by the reinforcement learning layer into specific recognition results, including the classification probability and exact location coordinates of the key parts of the transmission line.

[0231] The output decoding layer's workflow begins by receiving the predicted locations of key components from the reinforcement learning layer. This information includes not only the predicted coordinates of each key component, but also the relationships and characteristics between them, resulting from the joint optimization of a graph convolutional network and a deep reinforcement learning algorithm.

[0232] First, to convert this complex data into specific classification results, the output decoding layer uses a softmax function. This function processes the predictions from the reinforcement learning layer and generates a probability distribution for each key part, thereby determining the most likely category for each part. The application of the softmax function ensures the reliability and accuracy of the model output, even in complex backgrounds or when the relationships between key parts are ambiguous.

[0233] Next, to address the issue of overlapping prediction boxes that may occur during the prediction process, the output decoding layer uses non-maximum suppression (NMS). This technique effectively handles overlapping prediction boxes, retaining the most likely prediction box and removing other boxes with significant overlap, eliminating duplicate detections. This step is crucial for improving the accuracy and reliability of the system in real-world applications.

[0234] Ultimately, the output decoding layer generates detailed classification results and precise location coordinates for key transmission line components. These results not only include the component type (such as conductors, towers, and insulators), but also precisely indicate each component's location within the image, providing accurate guidance for subsequent inspection tasks.

[0235] The reference implementation code of the output decoding layer is as follows:

[0236]

[0237]

[0238] The above code is a simplified demonstration. In actual implementation, it needs to be adjusted according to the specific prediction result format and the structure of the deep learning model.

[0239] The entire output decoding layer is designed to convert the analysis results of the deep learning model into specific and feasible inspection guidance information, ensuring that the drone infrared temperature measurement system can efficiently and accurately complete the automated inspection of transmission lines.

[0240] Furthermore, the data processing and analysis unit is specifically used to perform temperature measurement and analysis on the identified key parts; wherein, temperature measurement is achieved by converting the thermal radiation intensity in the infrared image into a temperature value; the analysis process includes comparing the measured temperature value with a preset temperature threshold to determine whether each key part is overheated or has a temperature abnormality.

[0241] First, the drone's thermal imaging camera continuously captures infrared images of the transmission line during flight. These images provide a detailed record of the thermal radiation from the transmission line and its surroundings, with the brightness level of each pixel corresponding to a specific intensity of thermal radiation. This infrared imaging technology enables contactless temperature measurement of target objects, making it particularly suitable for inspecting inaccessible transmission lines.

[0242] Next, the data processing and analysis unit takes over the infrared image data. In the first step, the system converts the thermal radiation intensity in the image into specific temperature values. This conversion process, based on the infrared imaging camera's calibration parameters and a specific algorithm, ensures a precise mapping of thermal radiation intensity to temperature. The temperature values ​​at each pixel are combined to form a detailed temperature distribution map of the transmission line and its components.

[0243] The temperature analysis process then begins. The system has built-in temperature thresholds, which are set based on safety standards and historical data for transmission line operations. By comparing measured temperature values ​​against these thresholds, the system automatically identifies critical areas with abnormal temperatures. If the temperature in a particular area exceeds the safety threshold, it indicates a potential overheating issue and requires immediate attention. This analysis goes beyond simple temperature comparisons and may involve more complex pattern recognition to identify potential causes and patterns of abnormal temperatures.

[0244] Ultimately, all measurement and analysis results are transmitted to the operation and control software platform, where they are collated and summarized into detailed inspection reports. These reports not only identify the specific locations of temperature anomalies but also provide corresponding analysis results to support decision-making by the maintenance team. Through this process, the drone-based infrared temperature measurement system effectively monitors the health of transmission lines, prevents failures, and ensures stable grid operation.

[0245] In the drone infrared temperature measurement system provided in this embodiment, a deep learning model plays a central role, analyzing collected image data and identifying key locations on power transmission lines. Training a deep learning model is an iterative process, aimed at improving the model's understanding and predictive capabilities through continuous learning and adjustment. The following is a detailed description of the training steps.

[0246] 1. Data preparation: First, collect and prepare training data. In order to train deep learning models, data usually needs to be labeled, such as labeling objects in images in image recognition tasks.

[0247] 2. Data preprocessing: Before starting training, the data needs to be preprocessed. This may include data cleaning, normalization, standardization or other forms of transformation, with the aim of improving the efficiency and effectiveness of model training.

[0248] 3. Divide the dataset: Divide the dataset into training, validation, and test sets. The training set is used for model training, the validation set is used to adjust model parameters and prevent overfitting, and the test set is used to evaluate model performance after training.

[0249] 4. Model Selection: Choose an appropriate deep learning model based on the nature of the problem. This choice depends on the task type (e.g., classification, regression, clustering, etc.) and the data type (e.g., image, text, etc.). In this example, we use the deep learning model introduced earlier.

[0250] 5. Model training: The model is trained using the training set. This step involves feeding the model data and letting it try to predict the correct output by adjusting its internal parameters. Through backpropagation and optimization algorithms such as gradient descent, the model's weights and biases are continuously updated to minimize prediction errors.

[0251] 6. Model Evaluation and Adjustment: Use the validation set to evaluate model performance. Based on the model's performance on the validation set, adjust the model's parameters or structure to improve its accuracy and generalization. This step may require multiple iterations to find the optimal model configuration.

[0252] 7. Model Testing: After training and tuning, the model is evaluated using an independent test set. This step is crucial for verifying the model’s performance on unseen data and provides a true indicator of model performance.

[0253] 8. Model deployment: Once the model passes the testing phase, it can be deployed for actual prediction tasks. During the deployment process, the model may need to be further optimized to meet the performance and resource consumption requirements of the specific application.

[0254] The above steps provide an overview of the complete training process from data preparation to model deployment, which is suitable for deep learning application scenarios and ensures the feasibility and implementability of the technology.

[0255] The operation control software platform 105 is used to provide a setting interface for the user, which allows the user to set inspection parameters, including inspection areas, specific points of interest, and inspection frequencies; send the inspection parameters set by the user to the communication module; during the inspection process, display the drone's location, flight status, image data collected by the drone, and temperature measurement data of the transmission line to the user; after the inspection is completed, generate an inspection report to provide decision support for the maintenance team, wherein the inspection report includes the location of temperature anomalies and analysis results.

[0256] The operational control software platform 105 is a key component of the UAV infrared temperature measurement system for power line inspections. It provides users with an intuitive, easy-to-use interface for setting, monitoring, and analyzing power line inspection tasks. Designed to be highly user-friendly, the platform allows operators to easily set inspection parameters, including but not limited to inspection areas, specific points of interest, and inspection frequency. These parameters are then transmitted to the intelligent flight control unit 102 via the communication module 103 to guide the UAV's flight and inspection operations.

[0257] The operation control software platform 105 not only transmits inspection parameters but also receives and displays the drone's location, flight status, and collected image data in real time. This functionality is crucial for ensuring the drone follows its planned route and promptly identifies and focuses on specific areas of the transmission line. Furthermore, the platform can display infrared and visible light image data analyzed and processed by the data processing and analysis unit 104, including transmission line temperature information. This design allows users to monitor the inspection process in real time and obtain critical information promptly.

[0258] After the inspection is complete, another important function of the operation control software platform 105 is to generate an inspection report. This report not only summarizes the data collected during the inspection, but also includes the location and analysis results of temperature anomalies analyzed by the data processing and analysis unit 104 based on the deep learning model. This detailed inspection report enables the maintenance team to make quick and accurate decisions, improving the operational safety and reliability of the power system.

[0259] Although the present application is disclosed as above with the preferred embodiments, it is not intended to limit the present application. Any person skilled in the art may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

Claims

1. A drone infrared temperature measurement system for power transmission line inspection, characterized in that: include: The UAV platform includes a real-time dynamic positioning component, an infrared thermal imaging camera, a visible light camera, and a multimodal sensor module. The real-time dynamic positioning component is used to collect position information to ensure the UAV's accurate flight positioning. The infrared thermal imaging camera is used to collect infrared images of transmission lines to measure their temperature. The visible light camera collects visible light images to identify the structure and environmental information of the transmission lines. The multimodal sensor module includes a laser radar and an ultrasonic sensor to collect sensor data to achieve obstacle detection and avoidance, ensuring the UAV's safe flight. The intelligent flight control unit is used to identify and avoid obstacles based on the sensor data transmitted by the drone platform; and to plan the flight path based on the inspection parameters sent by the communication module and the location information transmitted by the drone platform; A communication module, supporting 4G / 5G and satellite communications, is used to send image data and location information collected by the drone platform to the data processing and analysis unit; receive inspection parameters sent by the control software platform, and send the inspection parameters to the intelligent flight control unit; A data processing and analysis unit is configured to receive image data and location information sent by the communication module; analyze the image data using a trained deep learning model to identify key locations of the transmission line; measure and analyze the temperature of the key locations to obtain temperature measurement data and analysis results; and transmit the image data, location information, temperature measurement data, and analysis results to an operation control software platform; wherein the key locations include conductors, towers, and insulators; The operation control software platform is used to provide users with a setting interface, which allows users to set inspection parameters, including inspection areas, specific points of interest, and inspection frequencies; send the inspection parameters set by the user to the communication module; during the inspection process, display the drone's location, flight status, image data collected by the drone, and temperature measurement data of the transmission line to the user; after the inspection is completed, generate an inspection report to provide decision support for the maintenance team, wherein the inspection report includes the location of temperature anomalies and analysis results.

2. The UAV infrared temperature measurement system according to claim 1, characterized in that: The real-time dynamic positioning component includes a global positioning system receiver and a real-time dynamic positioning module to improve the accuracy and stability of drone positioning.

3. The UAV infrared temperature measurement system according to claim 1, characterized in that: The multimodal sensor module also includes a wind speed sensor for real-time monitoring and adjusting the flight speed of the drone to ensure safe flight under different wind speed conditions.

4. The UAV infrared temperature measurement system according to claim 1, characterized in that: The deep learning model used by the data processing and analysis unit includes a data preprocessing layer, a feature extraction layer, an enhanced learning layer and an output decoding layer; The data preprocessing layer is used to preprocess the infrared image and visible light image provided by the UAV platform to obtain preprocessed infrared image and visible light image; wherein the preprocessing includes using image normalization technology to scale the image pixel values ​​to the range of [0, 1] to unify the brightness and contrast levels of different images; applying a Gaussian filter to remove noise and smooth the image to reduce image noise; and enhancing contrast by adjusting the image histogram; The feature extraction layer uses a depthwise separable convolutional network structure to process the preprocessed infrared image and visible light image to obtain feature representations of key parts of the transmission line. The depthwise separable convolutional network structure includes depthwise convolution and pointwise convolution. The depthwise convolution applies a separate filter to the input image to extract local features. The pointwise convolution combines the outputs of the depthwise convolution in the depth direction to form a higher-level feature representation. The feature representation of the key parts of the transmission line includes shape, size, and texture information. The reinforcement learning layer combines the graph convolutional network and the reinforcement learning algorithm to process the feature representation output by the feature extraction layer to obtain the predicted position of the key parts and the dynamically adjusted recognition strategy; The output decoding layer classifies the existence probability of the key parts according to the predicted positions of the key parts provided by the reinforcement learning layer using the softmax function, and applies non-maximum suppression to process overlapping prediction boxes to eliminate repeated detections, thereby obtaining the classification probability of the key parts and the final determined position coordinates.

5. The UAV infrared temperature measurement system according to claim 4, characterized in that: The reinforcement learning layer uses a framework that combines a graph convolutional network and a deep reinforcement learning algorithm. The graph convolutional network uses the feature representations output by the feature extraction layer to construct a graph structure, where the nodes in the graph structure represent the key parts provided by the feature extraction layer, and the edges reflect the spatial relationships between the key parts. The graph convolutional network captures and integrates the relationships between the key parts by performing convolution operations on the graph structure, and outputs an updated feature representation for each node. The deep reinforcement learning algorithm uses the updated feature representation provided by the graph convolutional network as state input; using the set reward mechanism, it gives the model positive or negative feedback based on the accuracy of the prediction and the efficiency of the recognition strategy; the agent in the deep reinforcement learning algorithm calculates an action strategy through the policy network based on the current state and reward, aiming to dynamically adjust the recognition strategy to optimize the performance of the deep reinforcement learning model.

6. The UAV infrared temperature measurement system according to claim 1, characterized in that: The data processing and analysis unit is specifically used to perform temperature measurement and analysis on the identified key parts; wherein, temperature measurement is achieved by converting the thermal radiation intensity in the infrared image into a temperature value; the analysis process includes comparing the measured temperature value with a preset temperature threshold to determine whether each key part is overheated or has a temperature abnormality.

7. The UAV infrared temperature measurement system according to claim 1, characterized in that: The intelligent flight control unit includes an obstacle recognition and avoidance mechanism based on a deep reinforcement learning model. The deep reinforcement learning model uses a two-layer network architecture. The first layer is a convolutional neural network that processes and interprets image data from the multimodal sensor module to enable rapid recognition of the surrounding environment and obstacle location. The second layer is the long short-term memory network, which is responsible for analyzing the changes in obstacle position information provided by the convolutional neural network over time, predicting the movement trend of obstacles and the optimal flight strategy of the drone relative to the obstacles; The deep reinforcement learning model implements obstacle recognition and avoidance through the following steps: Using environmental data provided by the drone platform as input to the deep reinforcement learning model; wherein the environmental data includes infrared images, visible light images, and lidar point cloud data; The convolutional neural network layer processes input data, extracts features of obstacles, and locates the precise position of obstacles; the convolutional neural network layer is composed of a convolutional layer, a pooling layer, and a fully connected layer; The long short-term memory network layer receives the obstacle location information output by the convolutional neural network layer and combines it with past flight data to predict the movement trend of the obstacle and the optimal flight strategy of the drone relative to the obstacle, where the optimal flight strategy includes the estimated obstacle avoidance direction; The reward function provided by the following formula 1 guides the optimization of the drone's flight strategy: Where R(t) represents the reward value at time point t, which aims to maximize the safety and flight efficiency of the drone when it circumvents obstacles; d(t) is the distance between the drone and the nearest obstacle; d0 is the normalization factor of the distance, which is used to adjust the influence of the distance; v(t) is the current flight speed; v opt is the optimal flight speed; θ(t) is the angle between the UAV’s flight direction and the expected obstacle avoidance direction; α, β, and γ are weight parameters used to balance the influence of each part in the reward function.

8. The UAV infrared temperature measurement system according to claim 1, characterized in that: The intelligent flight control unit performs path planning by executing the following steps: Based on the drone's current location and received inspection area parameters, a graph model is constructed. The graph model uses the drone's current location as the starting point and each specific point of interest within the inspection area as the target node. Each node in the graph model represents a potential location point, and each edge represents a possible path from one node to another. The cost of an edge consists of two parts: the actual distance from the current node to the next node, and the estimated distance from the current node to the final target node. The A* algorithm is used to search the constructed graph model. The A* algorithm determines the search direction by calculating the total cost f(n) of each node in the graph model, where f(n) = g(n) + h(n), g(n) represents the actual cost from the starting point to the current node n, and h(n) represents the estimated cost from the current node n to the target node. The path with the lowest total cost is preferred. Monitor the environmental data provided by the multimodal sensor module. When new obstacles are detected, adjust the nodes and edges in the graph model accordingly to avoid the flight path being too close to the obstacles.

Images