A method and system for collaborative optimization of nest deployment and path planning of a power inspection unmanned aerial vehicle based on hierarchical deep reinforcement learning

Abstract

The application relates to the technical field of autonomous operation of intelligent inspection unmanned aerial vehicles for power systems, and discloses a method and system for collaborative optimization of nest deployment and path planning of power inspection unmanned aerial vehicles based on hierarchical deep reinforcement learning. The unmanned aerial vehicle nest site selection and the unmanned aerial vehicle path planning are uniformly modeled as a hierarchical Markov decision process, and an upper and lower two-layer hierarchical near-end policy optimization network is constructed; the upper network is responsible for searching, adjusting and optimizing the nest deployment scheme, and the lower network is responsible for generating the unmanned aerial vehicle inspection path under the current nest deployment scheme and meeting the power constraint; through mutual state feedback, reward transmission and course joint training, the collaborative optimization of global space resource configuration and local flight behavior planning is realized. The application has the beneficial effects of being capable of simultaneously optimizing the nest deployment and the inspection path, improving the system-level collaborative efficiency, significantly reducing the large-scale path planning calculation time, improving the effective coverage capacity of the nest deployment on the inspection points, and the like.

Description

Technical Field

[0001] This invention relates to the fields of intelligent power system inspection, autonomous operation of unmanned aerial vehicles (UAVs), UAV nest deployment, energy-constrained path planning, multi-agent collaborative optimization, and deep reinforcement learning, and particularly to a method and system for collaborative optimization of energy-constrained UAV nest location and inspection path for large-scale power networks. Background Technology

[0002] In recent years, drones have been widely used in logistics, agriculture, infrastructure inspection, and power line inspection due to their advantages such as maneuverability, low deployment cost, and strong adaptability to complex terrain. Among these, power line inspection is one of the most representative application scenarios for drone technology. Traditional manual inspection methods suffer from problems such as high labor intensity, high risk, low efficiency, and limited coverage. Drone inspection can effectively improve inspection efficiency and safety.

[0003] However, the large-scale automated application of drones in power grid inspection has long been limited by battery life. The flight time and radius of a single drone are limited, making it difficult for a drone to complete the entire inspection task in a single flight when inspecting long-distance transmission lines or large-area distribution networks. The emergence of drone nests offers a new solution to this problem. Drone nests can serve as automated take-off, landing, recovery, and charging platforms for drones, enabling them to perform long-term, continuous, and automated inspection tasks under unattended conditions. Existing technologies also include schematic models demonstrating the deployment of drone charging stations in power distribution network systems, allowing drones to complete continuous inspection tasks by charging in nests.

[0004] Based on the search and analysis, existing technologies for drone nest deployment and inspection path planning can be broadly categorized as follows.

[0005] The first category is UAV nest location methods based on facility location models. UAV nest location is essentially related to classic location problems such as the P-median problem, the P-center problem, the maximum coverage location problem, and the capacity-constrained facility location problem. These methods typically base their solutions on the distance, coverage area, construction cost, or service capacity between candidate nest locations and inspection demand points, using mathematical programming, branch-and-split pricing, integer programming, graph convolutional networks combined with clustering algorithms, and deep reinforcement learning location models. Their advantage is the ability to establish clear optimization objectives for spatial resource allocation problems. However, they often treat nest location as a relatively static spatial coverage problem, failing to fully consider the flight paths, return-to-base charging, dynamic changes in battery power, and task execution order during actual UAV inspections.

[0006] The second category comprises UAV path planning methods based on traditional optimization or heuristic algorithms. Existing multi-UAV path planning methods typically include genetic algorithms, ant colony optimization, gray wolf optimization, large neighborhood search, Theta* algorithm, and artificial potential field methods. These methods are effective for small- to medium-scale path planning problems, generating inspection paths through manually designed search rules. However, in large-scale power grids, the number of inspection points can reach thousands, and UAVs need to dynamically switch between flight, inspection, return, and charging activities, leading to a dramatic increase in problem scale. Traditional heuristic methods often rely on fixed rules and manually generated parameters, lacking the ability to accumulate experience from historical tasks. They typically require re-solving whenever the task instance changes, making them unsuitable for dynamic and complex inspection scenarios.

[0007] The third category is path planning or joint optimization of location paths based on exact solvers. For example, mathematical programming solvers such as Gurobi and OR-Tools can obtain optimal or near-optimal solutions for small-scale problems or problems with clear constraints. However, as the number of inspection points, candidate nests, and drones increases, the computation time of exact solvers rises sharply, making it difficult to meet the rapid decision-making requirements in large-scale power line inspections. On small-scale and large-scale datasets, although Gurobi and OR-Tools can obtain optimal solutions, their computation times reach tens of thousands or even hundreds of thousands of seconds, respectively. The proposed method, however, significantly reduces computation time while maintaining a small difference in optimality.

[0008] The fourth category is UAV path planning methods based on deep reinforcement learning. Deep reinforcement learning can learn policies through interaction with the environment, no longer relying entirely on manually designed rules, and has certain advantages in dynamic path planning, action selection, and multi-agent collaboration. For example, methods such as DDQN, Dueling DQN, and PPO have been used for UAV path planning, vehicle path planning, and inspection task optimization. However, when nest selection and UAV path planning are modeled as a unified end-to-end reinforcement learning problem, the state space and action space expand dramatically, the decision chain becomes longer, and reward signals become sparse and delayed, leading to difficulties in policy exploration, credit allocation, and convergence instability. Traditional end-to-end reinforcement learning or single-layer reinforcement learning methods result in exponential growth of the state and action spaces, lengthened decision time domains, and sparse and delayed rewards, thus severely hindering value function and policy learning.

[0009] The fifth category is hierarchical reinforcement learning and methods for solving composite NP problems. The coupled problem of nesting and path planning can be viewed as a composite NP problem composed of multiple NP-hard subproblems combined or nested. This type of problem not only involves the hierarchical relationships and interdependencies between multiple subproblems but is also influenced by factors such as time, energy, spatial topology, and dynamic task states. Hierarchical deep reinforcement learning can naturally express multi-layered decision structures, decomposing complex tasks into high-level and low-level policies, thus providing a new paradigm for solving this type of composite optimization problem. Hierarchical deep reinforcement learning inherits the structural decomposition idea of ​​traditional optimization methods and automatically learns hierarchical representations and decision patterns through data-driven approaches, providing scalable and adaptive solutions for composite NP problems with temporal and spatial dependencies and stochasticity.

[0010] In summary, while existing technologies have made some progress in facility site selection, UAV path planning, accurate solution, heuristic search, reinforcement learning, and hierarchical decision-making, they still have the following shortcomings in large-scale power grid inspection scenarios.

[0011] First, existing methods often handle drone nest location selection and drone path planning separately, lacking global collaborative optimization capabilities. Nest location determines the drone's reachability, return distance, and charging convenience, while path planning, in turn, affects the actual utilization value of the nest. If the nest location is determined independently first, and then the inspection path is planned separately, it is easy to encounter problems such as the nest coverage area appearing reasonable, but the drone frequently detouring, having excessively long return distances, or insufficient coverage in some areas during actual missions.

[0012] Second, existing static path design methods are ill-suited for large-scale dynamic scenarios. Traditional methods such as genetic algorithms, ant colony algorithms, and large neighborhood search typically rely on fixed heuristic rules and offline static instance solutions, lacking the ability to accumulate experience through continuous environmental interaction and adaptively adjust strategies. When the inspection area, number of poles, candidate drone nest locations, number of drones, or battery parameters change, a new solution is often required, making it difficult to meet real-time or near-real-time inspection and scheduling requirements.

[0013] Third, the computational cost of exact solution methods is too high. For large-scale power networks containing thousands of towers, multiple lines, and numerous candidate generator houses, while exact mathematical programming methods can provide relatively optimal solutions, the computation time increases rapidly with the problem size. Experimental tests show that Gurobi takes 25,596 seconds and 67,240 seconds for small-scale and large-scale path planning tasks, respectively, while OR-Tools takes 40,680 seconds and 106,880 seconds, respectively, making it difficult to directly apply to practical inspection tasks that require rapid planning.

[0014] Fourth, single-layer reinforcement learning methods face the problem of state space and action space expansion. If nest deployment, checkpoint selection, flight actions, charging actions, and mission termination actions are all placed in a unified action space, the policy network needs to make long-term decisions in an extremely high-dimensional state, which can easily lead to problems such as low exploration efficiency, sparse rewards, unstable training, and difficulty in credit allocation.

[0015] Fifth, existing reward functions often fail to fully reflect the expert knowledge required for power line inspection tasks. Drone inspection tasks not only require short paths but also need to ensure pole coverage, maintain the drone's remaining battery power, avoid redundant visits, guarantee a return to the drone's charging station, and balance construction costs, inspection time, and task completion quality. If the reward function only uses a single indicator such as path length or coverage, it is easy for the strategy to be biased towards local optima, making it difficult to obtain high-quality solutions that meet engineering constraints.

[0016] Sixth, the engineering feasibility of existing path planning methods is insufficient. Some methods can obtain good paths in simulations, but they do not fully consider the charging capacity of real UAV nests, UAV battery capacity, line topology distribution, tower coordinates, and comparison with manual flight control strategies. Therefore, it is difficult to prove their practical value in real power line inspection scenarios.

[0017] In summary, current technologies cannot simultaneously meet the comprehensive requirements of large-scale power grid inspection, including the rationality of drone nest deployment, path planning efficiency, energy-constrained safety, training stability, computational speed, and practical engineering feasibility. Currently, there is still a lack of a method to simultaneously consider multiple factors such as drone nest deployment, drone battery capacity, inspection point coverage, return charging, task completion time, and path costs in large-scale power pole inspection scenarios, forming a self-learning, adaptive, and scalable collaborative optimization method for drone nest location and drone inspection paths. Summary of the Invention

[0018] The technical problem this invention aims to solve is how to simultaneously consider multiple factors such as drone nest deployment, drone battery capacity, inspection point coverage, return charging, task completion time, and path cost in large-scale power pole inspection scenarios, and form a self-learning, adaptive, and scalable method for collaborative optimization of drone nest location and drone inspection path.

[0019] To address the aforementioned technical issues, this invention provides a collaborative optimization method for power line inspection UAV nest deployment and path planning based on hierarchical deep reinforcement learning. It unifies UAV nest selection and UAV path planning into a hierarchical Markov decision process and constructs a two-layer hierarchical near-end policy optimization network. The upper-layer network is responsible for searching, adjusting, and optimizing nest deployment schemes, while the lower-layer network is responsible for generating UAV inspection paths that satisfy power constraints under the current nest deployment scheme. Through state feedback, reward transfer, and curriculum-based joint training between the upper and lower layers, collaborative optimization of global spatial resource allocation and local flight behavior planning is achieved.

[0020] The specific steps include the following: Step 1: Obtain power network inspection task data and build the task environment; Step 2: Establish a hierarchical decision-making model for nest location and path planning; Step 3: Construct a hierarchical near-end strategy optimization network to optimize nesting deployment; Step 4: Construct a lower-level hierarchical near-end policy optimization network to generate energy-constrained inspection paths; Step 5: Construct a hierarchical reward function guided by expert knowledge; Step Six: Construct an action mask and a comparison state feedback mechanism; Step 7: Use a tiered group relative course optimization strategy for training; Step 8: Output the drone nest deployment plan, drone inspection route, and charging schedule plan.

[0021] Preferably, when executing step one, the basic data of the power network to be inspected is first obtained, including the set of pole or tower coordinates, line topology, set of candidate UAV nest locations, nest construction cost, number of available nests, number of UAVs, UAV battery capacity, flight speed, energy consumption per unit distance, inspection and data collection time, charging time, task start and end points, and inspection coverage requirements; let the set of power inspection points be: , in, Indicates the first Coordinates of the poles or power equipment to be inspected. Indicates the number of inspection points; let the set of candidate nest locations be: , in, Indicates the first Candidate nest locations Indicate the number of candidate drone nests; let the set of drones be: , in, This indicates the number of drones; each drone has a maximum battery capacity. Current battery level Flight speed Energy consumption coefficient per unit distance Inspection and collection of energy consumption coefficients and charging rate ; Based on the above data, construct a drone inspection simulation environment; In the specific execution of step two, the joint problem of nest location selection and path planning is modeled as a hierarchical Markov decision process; the overall state is represented as: , in, This represents static global information determined by the problem instance, including tower coordinates, line topology, candidate drone nest locations, nest construction costs, drone parameters, and mission scale. This represents dynamic local information that is constantly updated with environmental interactions, including the current nest deployment scheme, the set of inspected towers, the current location of the drone, the remaining battery power of the drone, historical action selections, historical reward feedback, and inactive action mask information; a comparison state space is further introduced; the comparison state space is used to compare the dominance relationships between new and old nest deployment schemes, new and old path planning schemes, or new and old multi-objective solutions, and the comparison results are fed back to the policy network to suppress invalid actions and guide policy search; historical actions and historical rewards are aggregated into historical feedback features: , , in, Indicates the past number The selected action or optimization operator, This indicates the corresponding reward. Indicates the forgetting factor, Indicates the length of the history window.

[0022] Preferably, in step three, the upper-layer hierarchical near-end policy optimization network is used to optimize the UAV nest deployment scheme; let the nest deployment scheme be represented as a binary vector: , in, Indicates the candidate position Deploy drone nests, Indicates no deployment; The upper-layer policy network outputs the selection probabilities of various nest deployment optimization operators based on the current state, and selects the most suitable problem instance-guided optimization operator based on the probability. Specifically, these include: a swap operator, used to swap the state or order of two or three candidate positions in the encoding to adjust the nest deployment spatial structure; a flip operator, used to add nests to currently undeployed positions or remove nests from currently deployed positions; and a destruction operator, used to remove the most unfavorable nest position or local structure in the current scheme based on objective dimensions such as task time, construction cost, or path cost. The upper-layer policy network adopts an encoder-decoder structure. The encoder receives the current nest deployment scheme, candidate nest features, and inspection requirement point information, and maps them to a shared latent space. A multi-head attention mechanism is used to model the relationship between nest deployment decisions, backup nest positions, and inspection requirement coordinates. The decoder further integrates historical feedback features and action mask information to output the feasible probability distribution of candidate optimization operators.

[0023] Preferably, in step four, the lower-level hierarchical near-end strategy optimization network is used to generate inspection paths that satisfy the UAV's energy constraints under the given nest deployment scheme in the upper layer; the lower-level state includes the UAV's current position, remaining battery power, set of inspected towers, set of uninspected towers, nest position, reachability judgment information, and local topology information; the lower-level action space includes the following basic actions: flight actions, controlling the UAV to fly in a straight line towards the target tower, target waypoint, or target nest; The charging action controls the drone to return to its nest and perform charging; the inspection and data collection action controls the drone to collect images, infrared data, point cloud data, or other inspection data at the target tower or equipment location; the mission termination action is used to end the current inspection mission when the mission completion conditions are met or the termination state is reached. The lower-level strategy adopts an option mechanism, abstracting the drone behavior into three types of options: inspecting a certain pole, returning to the nest to charge, and terminating the mission. For the inspection option, the drone flies from its current position to the target pole, collects data, and terminates the option after the data collection is completed. For the charging option, the drone flies to the nest and charges to the set capacity before terminating the option. For the mission termination option, the drone returns to the destination or completes the marking action before terminating the mission. The lower-level policy network adopts an actor-critic structure based on LSTM; the value network uses global observation information to evaluate the current state value, and the policy network uses local observation information to generate the current action; the fully connected layer first extracts observation features, and LSTM is used to capture the temporal dependencies in UAV path planning. Finally, the value head and action head output the state value and action probability, respectively.

[0024] Preferably, in step five, to ensure that the strategy learning results conform to the engineering constraints of the power inspection task, a hierarchical reward function guided by expert knowledge is designed. This reward function includes three parts: advantage difference-driven reward, dominance relationship evaluation reward, and lower-level node reward. The total reward can be expressed as: , in, Rewards driven by competitive advantage To evaluate rewards based on the dominance relationship, As a reward for lower-level nodes, These are the weighting coefficients; Advantage-driven rewards are used to measure the improvement of a new solution relative to a baseline solution after performing the current action; they can be expressed as: , in, Indicates the current state. Indicates the current action. This represents the multi-objective normalized improvement value after the action is performed. This indicates the baseline performance before the action is executed; the reward can guide the strategy to prioritize actions that can improve path distance, task time, construction cost, or coverage. The dominance evaluation reward is used to adjust the reward based on the Pareto dominance relationship between the new and old solutions; if the new solution dominates the old solution, a higher positive reward is given; if the new and old solutions do not dominate each other, an exploratory reward is given; if the old solution dominates the new solution or there is no significant difference between the two, the reward is reduced or a penalty is given; this mechanism can promote the strategy to search for non-dominated and dominant solutions and avoid the strategy from repeating ineffective actions in the long term. Lower-level node rewards are used to constrain drone path behavior; these rewards include remaining battery power safety rewards or penalties, repeated visit penalties, and path length rewards or penalties; if the drone has insufficient remaining battery power to safely return to the nest, a penalty is imposed; if the drone repeatedly visits a pole that has already been inspected, a penalty is imposed; if the drone completes more effective inspection points with a shorter path, a positive reward is imposed.

[0025] Preferably, in step six, to improve the efficiency of strategy search during the upper-level nest deployment optimization, an action masking mechanism is constructed to temporarily block inactive or recently invalid actions from the candidate action set. After each nest deployment optimization operator or UAV action is executed, the system compares the dominance relationship between the new and old solutions. If the current operator does not generate a non-dominant solution or causes the constraint to be infeasible, a mask is applied to the operator in the next decision step to shift the strategy to other candidate actions. The masking information can be maintained through a hash table, Boolean vector, or action feasibility matrix.

[0026] Preferably, in step seven, to improve training stability and exploration efficiency, a hierarchical group relative course optimization strategy is introduced; the upper-layer nest deployment scheme and the lower-layer inspection path scheme are organized into several candidate output groups, and each output group is scored using a reward model; then, the relative advantage of each candidate output is calculated based on the average performance within the group or the performance of the reference strategy, and the upper and lower layer policy networks are updated accordingly; the training process adopts the concept of course learning, gradually increasing according to the task difficulty. The first phase is a localized, simple inspection phase. This phase primarily trains the drone to complete local pole coverage and safely return to its nest within a small area. The reward function emphasizes basic coverage rate, return success rate, and avoiding crashes. The second stage is the medium-scale energy-constrained inspection stage; this stage expands the inspection range and the number of towers, and introduces charging frequency, dynamic power balance, repeated access penalty and crash penalty, so that the model learns to reasonably arrange flight, inspection and charging behavior under power constraints. The third stage is the global multi-objective collaborative optimization stage. This stage simultaneously considers the cost of rack construction, inspection coverage efficiency, path length, task completion time, and route topology to achieve comprehensive optimization of rack deployment and path planning; In the distributed training approach, multiple worker processes interact with the simulation environment in parallel on multiple GPUs, generating experience data and storing it in a shared replay buffer. When the buffer reaches a set capacity, each GPU samples the training data and calculates the gradients of the policy network and the value network respectively. The master node averages the gradients of each GPU and then updates the global network parameters.

[0027] Preferably, after training is completed in step eight, the power network instance to be optimized is input into the system; the upper-layer policy network outputs the drone nest deployment plan, including candidate locations for drone nest deployment, the number of drone nests, and the service area; the lower-layer policy network outputs the inspection path for each drone, including the order of inspected towers, flight path, return to the nest for charging time, the path for continuing inspection after charging, and the mission termination location; the final output includes: a set of drone nest deployment locations; the sequence of inspected towers for each drone; the flight path and mileage of each drone; the power change curve of each drone; the time and location of each return to the nest for charging; the total mission completion time; the number of effective inspection points; the total path distance and energy consumption; and prompts for uncovered towers or abnormal constraints.

[0028] Corresponding to the above method, the present invention also provides a collaborative optimization system for the deployment and path planning of power inspection drone nests based on hierarchical deep reinforcement learning, including: a power network data input module, used to obtain tower coordinates, line topology, candidate nest locations, drone parameters and inspection task requirements; The inspection environment simulation module is used to simulate the flight, inspection, return, charging, and mission termination behavior of drones. The upper-layer nest deployment optimization module is used to optimize the network selection nest deployment optimization operator and update the nest deployment scheme through a hierarchical near-end strategy. The lower-level path planning module is used to generate energy-constrained inspection paths for UAVs through an LSTM-based actor-critic network. The expert reward calculation module is used to calculate advantage difference-driven rewards, dominance relationship assessment rewards, and node-level path rewards. The action mask and comparison status module is used to determine the feasibility of actions, the dominance relationship between new and old schemes, and the masking of invalid actions; The hierarchical group relative course training module is used to train the upper and lower layer policy networks step by step according to the task difficulty; The results output and visualization module is used to output the nesting deployment plan, inspection path, charging scheduling plan, task time, path length and coverage effect.

[0029] Compared with existing nesting methods, traditional path planning methods, exact solution methods, and single-layer reinforcement learning methods, the present invention has the following advantages when using the technical solution of the present invention: First, it can simultaneously optimize nest deployment and inspection paths, improving system-level collaborative efficiency. Existing methods typically handle nest location selection and path planning separately, while this invention, through feedback between the upper-level nest deployment strategy and the lower-level path planning strategy, enables the UAV path planning results to influence nest deployment optimization in reverse, thereby avoiding the problem of "statically reasonable nest location coverage but inefficient actual path".

[0030] Second, it can significantly reduce the computation time for large-scale path planning. Experiments show that on small datasets, Gurobi achieves the optimal distance of 14162.80m but takes 25596s to compute, while the method of this invention achieves a distance of 14472.66m, with an optimality difference of only 2.19%, and a computation time of 152s. On large datasets, Gurobi achieves a distance of 29480.38m and takes 67240s to compute, while the method of this invention achieves a distance of 29725.31m, with an optimality difference of only 0.83%, and a computation time of 312s. These results demonstrate that this invention reduces computation time from tens of thousands of seconds to hundreds of seconds while maintaining near-optimal solution quality.

[0031] Third, compared to ordinary reinforcement learning methods, this invention achieves both higher solution quality and faster solution speed. Experiments show that on large-scale datasets, the PPO method has a distance of 33230.28m, a gap of 12.72%, and a computation time of 622s; the DoubleDQN method has a distance of 29969.75m, a gap of 1.66%, and a computation time of 1015s; while the method of this invention has a distance of 29725.31m, a gap of 0.83%, and a computation time of 312s. This demonstrates that the hierarchical policy structure achieves a better balance between path quality and inference efficiency.

[0032] Fourth, it improves the effective coverage of inspection points by the nested deployment. Site selection experiments show that the present invention achieves better mean and lower standard deviation in more than 81.25% of test instances across multiple evaluation metrics such as Hypervolume, Spacing, Ω-dominance, and IGD, indicating that it outperforms the comparative methods in terms of solution set quality and stability.

[0033] Fifth, this invention increases the number of effective inspection points in large-scale inspection tasks. The number of effective inspection coordinates and average solution time were compared under conditions of 6, 8, and 10 tower nests. The results show that this invention typically achieves a higher number of effective inspection points in both small-scale and large-scale tower inspection scenarios. Particularly under the condition of 10 tower nests in a large-scale dataset, the effective inspection coordinates of this invention reach 5159±62, higher than PMP-GCN's 4914±66 and RL-LS's 4792±58.

[0034] Sixth, improve inspection efficiency in real-world scenarios. Experiments compared the inspection routes generated by the method of this invention with those planned by professional UAV flight controllers. The results showed that in four UAV nest scenarios, the present invention reduced the total task time by 19.54% to 25.46% compared to manual flight control strategies. This demonstrates that the present invention is not only effective in simulation environments but also has practical engineering application value in real-world power distribution line inspection scenarios.

[0035] Seventh, improve training stability and exploration efficiency. This invention optimizes the curriculum by hierarchical grouping, gradually transitioning complex tasks from simple local inspections to global multi-objective collaborative optimization. This reduces the problems of reward sparsity, difficulty in collecting positive samples, and policy oscillation in the early training stages of reinforcement learning, enabling the model to learn nesting and path planning strategies more stably.

[0036] Eighth, it enhances the interpretability of the project. The output of this invention includes not only the final path, but also the drone nest deployment scheme, the optimized operator selection process, the drone charging and return behavior, power changes, and task coverage. Therefore, it is easier for dispatchers to understand the basis for path generation, and it is also more suitable for deployment, verification, and secondary adjustments by power inspection systems.

[0037] In summary, this invention provides a self-learning, adaptive, and scalable method and system for collaborative optimization of UAV nest location and UAV inspection path. Attached Figure Description

[0038] Figure 1 This is the upper-layer network architecture diagram in the overall network architecture diagram of the collaborative optimization method and system for power inspection UAV nest deployment and path planning based on hierarchical deep reinforcement learning in this invention.

[0039] Figure 2 This is the lower-level network architecture diagram in the overall network architecture diagram of the collaborative optimization method and system for power inspection UAV nest deployment and path planning based on hierarchical deep reinforcement learning in this invention. Detailed Implementation

[0040] The following is in conjunction with the appendix Figure 1 To the attached Figure 2 The present invention will be described in further detail below.

[0041] This invention provides a method and system for collaborative optimization of UAV nest location and inspection path for energy-constrained power grids. The method unifies UAV nest location selection and UAV path planning into a hierarchical Markov decision process and constructs a two-layer hierarchical near-end policy optimization network. The upper-layer network is responsible for searching, adjusting, and optimizing nest deployment schemes, while the lower-layer network is responsible for generating UAV inspection paths that meet power constraints under the current nest deployment scheme. Through state feedback, reward transfer, and curriculum-based joint training between the upper and lower layers, collaborative optimization of global spatial resource allocation and local flight behavior planning is achieved. The specific implementation can be carried out according to the following steps: Step 1: Acquire power network inspection task data and build the task environment First, obtain the basic data of the power network to be inspected, including the set of pole or tower coordinates, line topology, set of candidate drone nest locations, nest construction cost, number of available nests, number of drones, drone battery capacity, flight speed, energy consumption per unit distance, inspection and data collection time, charging time, mission start and end points, and inspection coverage requirements.

[0042] Let the set of power inspection points be:

[0043] in, Indicates the first Coordinates of the poles or power equipment to be inspected. This represents the number of inspection points. Let the set of candidate nest locations be: , in, Indicates the first Candidate nest locations This represents the number of candidate drone nests. Let the drone set be: , in, This indicates the number of drones. Each drone has a maximum battery capacity. Current battery level Flight speed Energy consumption coefficient per unit distance Inspection and collection of energy consumption coefficients and charging rate .

[0044] Based on the above data, a drone inspection simulation environment is constructed. This environment can simulate the drone's flight, inspection, return, charging, and mission termination behaviors according to the drone's current location, remaining battery power, status of the pole to be inspected, location of the drone nest, and line topology, and output corresponding state transitions and reward feedback.

[0045] Step 2: Establish a hierarchical decision-making model for nest location and path planning. This invention models the joint problem of nest location and path planning as a hierarchical Markov decision process. The overall state is represented as: , in, This represents static global information determined by the problem instance, including tower coordinates, line topology, candidate drone nest locations, nest construction costs, drone parameters, and mission scale. This represents dynamic local information that is constantly updated with environmental interactions, including the current drone nest deployment plan, the set of inspected poles, the current location of the drone, the remaining battery power of the drone, historical action selections, historical reward feedback, and inactive action mask information.

[0046] Unlike traditional reinforcement learning state spaces, this invention further introduces a comparison state space. This comparison state space is used to compare the dominance relationships between new and old nest deployment schemes, new and old path planning schemes, or new and old multi-objective solutions, and feeds the comparison results back to the policy network to suppress ineffective actions and guide policy search.

[0047] In one implementation, historical actions and historical rewards are aggregated into historical feedback features:

[0048] , in, Indicates the past number The selected action or optimization operator, This indicates the corresponding reward. Indicates the forgetting factor, This indicates the length of the historical window. This design enables policy networks to utilize recent action-reward patterns while reducing the interference of premature historical information on current decisions.

[0049] Step 3: Construct a hierarchical near-end strategy optimization network to optimize nesting deployment. The upper-layer hierarchical near-end policy optimization network is used to optimize the drone nest deployment scheme. Let the nest deployment scheme be represented as a binary vector:

[0050] in, Indicates the candidate position Deploy drone nests, This indicates that deployment will not be carried out.

[0051] The upper-level policy network outputs the selection probability of various nest deployment optimization operators based on the current state, and selects the most suitable problem instance-oriented optimization operator based on the probability.

[0052] Specifically, this includes: swap operators, used to swap the state or order of two or three candidate positions in the encoding to adjust the nest deployment space structure; The flip operator is used to add a nest in a currently undeployed location or remove a nest in a currently deployed location. The destruction operator is used to remove the most unfavorable nest location or local structure in the current scheme based on objective dimensions such as task time, construction cost, or path cost.

[0053] The upper-layer policy network adopts an encoder-decoder structure. The encoder receives the current nest deployment plan, candidate nest features, and inspection requirement point information, and maps them to a shared latent space; a multi-head attention mechanism is used to model the relationship between nest deployment decisions, backup nest locations, and inspection requirement coordinates; the decoder further integrates historical feedback features and action mask information to output the feasible probability distribution of candidate optimization operators.

[0054] With this design, the upper-level strategy no longer directly enumerates all possible nest combinations, but instead learns which type of deployment adjustment action should be prioritized under different problem instances, thereby improving the search efficiency for large-scale nest location problems.

[0055] Step 4: Construct a lower-level hierarchical near-end policy optimization network to generate energy-constrained inspection paths. The lower-layer hierarchical near-end policy optimization network is used to generate inspection paths that satisfy the UAV's energy constraints under the given nest deployment scheme in the upper layer. The lower-layer state includes the UAV's current position, remaining battery power, set of inspected towers, set of uninspected towers, nest position, reachability judgment information, and local topology information.

[0056] The lower motion space includes the following basic motions: Flight maneuvers: Control the drone to fly in a straight line toward the target tower, target waypoint, or target drone nest; The charging action controls the drone to return to its nest and perform charging. The inspection and data collection action controls the drone to perform image, infrared, point cloud or other inspection data collection at the target tower or equipment location; The task termination action is used to end the current inspection task when the task completion conditions are met or the termination state is reached.

[0057] In one implementation, the lower-level strategy further employs an option mechanism, abstracting the drone's behavior into three categories: inspecting a specific pole, returning to the nest to recharge, and terminating the mission. For the inspection option, the drone flies from its current location to the target pole, collects data, and terminates the option after data collection is complete. For the recharging option, the drone flies to the nest and charges to the set capacity before terminating the option. For the mission termination option, the drone returns to the destination or completes a marking action before terminating the mission.

[0058] The lower-level policy network employs an LSTM-based actor-critic structure. The value network uses global observation information to evaluate the current state value, while the policy network uses local observation information to generate the current action. The fully connected layers first extract observation features, and the LSTM is used to capture temporal dependencies in UAV path planning. Finally, the value head and action head output the state value and action probability, respectively. The lower-level network addresses the partial observability and temporal dependency issues in multi-UAV path planning precisely through this LSTM-type actor-critic structure.

[0059] Step 5: Construct a hierarchical reward function guided by expert knowledge To ensure that the strategy learning results conform to the engineering constraints of power line inspection tasks, this invention designs a hierarchical reward function guided by expert knowledge. This reward function includes three parts: advantage difference-driven reward, dominance relationship evaluation reward, and lower-level node reward.

[0060] The total reward can be expressed as:

[0061] in, Rewards driven by competitive advantage To evaluate rewards based on the dominance relationship, As a reward for lower-level nodes, These are the weighting coefficients.

[0062] Advantage-driven rewards measure the improvement of a new solution relative to a baseline solution after performing the current action. It can be expressed as:

[0063] in, Indicates the current state. Indicates the current action. This represents the multi-objective normalized improvement value after the action is performed. This indicates the baseline performance before the action is executed. This reward guides the strategy to prioritize actions that improve path distance, task time, construction costs, or coverage.

[0064] The dominance evaluation reward is used to adjust rewards based on the Pareto dominance relationship between the old and new solutions. A higher positive reward is given if the new solution dominates the old solution; an exploratory reward is given if the old and new solutions do not dominate each other; and a lower reward or penalty is given if the old solution dominates the new solution or if there is no significant difference between the two. This mechanism encourages the strategy to search towards non-dominated and dominant solutions, avoiding the strategy from repeatedly taking ineffective actions in the long run.

[0065] Lower-level node rewards are used to constrain the drone's path behavior. These rewards include a remaining battery power safety reward or penalty, a revisit penalty, and a path length reward or penalty. A penalty is imposed if the drone's remaining battery power is insufficient to safely return to the nest; a penalty is imposed if the drone revisits a pole that has already been inspected; and a positive reward is given if the drone completes more effective inspection points with a shorter path. The reward function is designed with three parts: ADD reward, DE reward, and lower-level node reward, which work together to guide the upper and lower level strategies to generate high-quality, energy-efficient inspection trajectories.

[0066] Step Six: Construct Action Masks and Compare State Feedback Mechanisms In optimizing upper-level drone nest deployment, certain actions may lead to exceeding the nest quantity limit, insufficient coverage, or exceeding construction cost constraints. In lower-level path planning, certain actions may result in insufficient drone battery power, inability to return to the nest, or repeated visits to completed poles. To improve strategy search efficiency, this invention constructs an action masking mechanism to temporarily block inactive or recently invalid actions from the candidate action set.

[0067] Specifically, after each execution of the nest deployment optimization operator or UAV action, the system compares the dominance relationship between the new and old solutions. If the current operator does not produce a non-dominated solution or results in constraint infeasibility, a mask is applied to the operator in the next decision step, causing the policy to shift to other candidate actions. The mask information can be maintained through a hash table, Boolean vector, or action feasibility matrix.

[0068] This mechanism can significantly reduce invalid searches, enabling strategies to focus more quickly on feasible and high-quality action areas in large-scale problems.

[0069] Step 7: Training using a tiered group relative course optimization strategy. To improve training stability and exploration efficiency, this invention introduces a hierarchical group relative course optimization strategy. This strategy organizes the upper-layer nesting deployment scheme and the lower-layer inspection path scheme into several candidate output groups, and scores each group's output using a reward model. Subsequently, using the average performance within the group or the performance of a reference strategy as a benchmark, the relative advantage of each candidate output is calculated, and the upper and lower layer policy networks are updated accordingly.

[0070] The training process adopts a course-based learning approach, with the difficulty of the tasks gradually increasing.

[0071] The first phase is a localized, simple inspection phase. This phase primarily trains the drone to complete localized pole and tower coverage within a small area and safely return to its nest. The reward function emphasizes basic coverage, return success rate, and avoiding crashes.

[0072] The second stage is the medium-scale energy-constrained inspection stage. This stage expands the inspection range and the number of towers, and introduces charging frequency, dynamic power balance, repeated access penalty, and crash penalty, enabling the model to learn to rationally arrange flight, inspection, and charging behaviors under power constraints.

[0073] The third stage is the global multi-objective collaborative optimization stage. This stage simultaneously considers the cost of nest construction, inspection coverage efficiency, path length, task completion time, and line topology to achieve comprehensive optimization of nest deployment and path planning.

[0074] In distributed training, multiple worker processes interact with the simulation environment in parallel across multiple GPUs, generating experience data and storing it in a shared replay buffer. Once the buffer reaches a set capacity, each GPU samples the training data and calculates the gradients of the policy network and value network respectively. The master node averages the gradients from each GPU and then updates the global network parameters. The HGRCO strategy improves learning stability and exploration efficiency by gradually adjusting the training difficulty and reward sparsity through course learning.

[0075] Step 8: Output the drone nest deployment plan, drone inspection routes, and charging scheduling plan. After training, the power network instance to be optimized is input into the system. The upper-layer policy network outputs the drone nest deployment plan, including candidate locations for nest deployment, the number of nests, and the service area; the lower-layer policy network outputs the inspection path for each drone, including the order of inspected towers, flight path, return to nest for charging time, continued inspection path after charging, and mission termination location.

[0076] The final output includes: a set of drone nest deployment locations; the sequence of inspection towers for each drone; the flight path and mileage of each drone; the battery change curve of each drone; the time and location of each return to the nest for charging; the total task completion time; the number of effective inspection points; the total path distance and energy consumption; and prompts for uncovered towers or abnormal constraints.

[0077] In terms of system architecture: Corresponding to the above method, this invention also provides a collaborative optimization system for power grid inspection UAV nest deployment and path planning based on hierarchical deep reinforcement learning, including: a power network data input module for acquiring tower coordinates, line topology, candidate nest locations, UAV parameters, and inspection task requirements; an inspection environment simulation module for simulating UAV flight, inspection, return, charging, and task termination behaviors; an upper-layer nest deployment optimization module for selecting nest deployment optimization operators and updating nest deployment schemes through a hierarchical near-end policy optimization network; a lower-layer path planning module for generating UAV energy-constrained inspection paths through an LSTM-based actor-critic network; an expert reward calculation module for calculating advantage difference-driven rewards, dominance relationship evaluation rewards, and node-level path rewards; an action mask and comparison state module for judging action feasibility, dominance relationship between new and old schemes, and masking invalid actions; a hierarchical group relative course training module for progressively training the upper and lower layer policy networks according to task difficulty; and a result output and visualization module for outputting nest deployment schemes, inspection paths, charging scheduling plans, task time, path length, and coverage effect.

[0078] This invention has been experimentally verified using real power network tower coordinate data and actual distribution line inspection scenarios. Two real datasets were used, obtained by preprocessing the latitude and longitude coordinates of transmission towers in a power grid in a certain region of China: one was a small-scale dataset including three power lines approximately 5km long, totaling 1000 towers, with an inspection area of ​​approximately 6km. 2 The other is a large-scale dataset, including 10 power lines of approximately 5km each, totaling 5000 poles and towers, with an inspection area of ​​approximately 20km. 2 The experimental platform was configured with an Intel 4310 CPU 2.10 GHz, four Nvidia A800 80G GPUs, and an Ubuntu 20.04.6 system. The drone parameters were based on the standard configuration of the DJI Matrice M301 drone and DJI drone nest.

[0079] Example 1: Deployment of small-scale power line equipment racks and UAV inspection path planning In this embodiment, a patrol area of ​​approximately 6km is selected, comprising 3 power lines, approximately 1000 poles, and an inspection area of ​​approximately 6km. 2The small-scale power grid is taken as the object. First, the parameters such as the latitude and longitude coordinates of the poles, the line topology, the candidate nest locations, the number of deployable nests, the drone battery capacity, flight speed, charging time, and inspection and data collection time are input.

[0080] The system first generates an initial nest deployment plan using an upper-layer nest deployment strategy network, and continuously adjusts nest positions through optimization operators such as swapping, flipping, and destruction. After each adjustment, a lower-layer path planning network generates a drone inspection path under the current nest deployment plan, determining whether the drone can complete the pole visit, return to the nest for charging, and continue the mission within battery capacity constraints. The system calculates rewards based on path length, coverage effect, energy security, repeated visits, and dominance relationships, and feeds these rewards back to the upper and lower layer strategy networks.

[0081] In path planning experiments, the method of this invention achieved a path distance of 14472.66m on a small dataset, with an optimality difference of 2.19% and a computation time of 152s. In comparison, while Gurobi achieved an optimal result of 14162.80m, its computation time reached 25596s; DoubleDQN achieved a path distance of 14441.07m with a computation time of 386s; and PPO achieved a path distance of 16673.86m with a computation time of 262s. This invention significantly reduces computation time while maintaining near-optimal path quality.

[0082] Example 2: Cooperative Optimization of Cell Location and Inspection Path in Large-Scale Power Networks In this embodiment, an inspection area of ​​approximately 20km is selected, comprising 10 power lines, approximately 5,000 poles, and an inspection zone of approximately 20km. 2 The system takes a large-scale power grid as its target. It follows the same data input, environment construction, upper-layer nesting deployment, and lower-layer path planning process as in Example 1, but with a larger task scale and a more complex candidate solution space.

[0083] In large-scale datasets, the method of this invention yields a path distance of 29725.31m, with an optimality gap of 0.83% and a computation time of 312s. In comparison, Gurobi achieves an optimal path distance of 29480.38m, but with a computation time of 67240s; OR-Tools takes 106880s; DoubleDQN yields a path distance of 29969.75m with a computation time of 1015s; and PPO yields a path distance of 33230.28m with a computation time of 622s. Experimental results demonstrate that in large-scale scenarios, this invention can obtain a near-optimal inspection scheme within hundreds of seconds, significantly outperforming the combined performance of traditional exact solutions and ordinary reinforcement learning methods.

[0084] Example 3: Verification of Effective Inspection Coverage under Different Numbers of Cell Towers To verify the drone nest deployment capability of this invention in scenarios with limited drone resources, 6, 8, and 10 drone nests were set up respectively, and the number of effective inspection coordinates that could be covered within the reachable range by different methods was counted.

[0085] In small-scale datasets, with 6 tower racks, the effective inspection coordinates of this invention are 1786±56, higher than PMP-GCN's 1079±73 and RL-LS's 1574±28. With 10 tower racks, the effective inspection coordinates reach 2033±12, higher than PMP-GCN's 1891±27 and RL-LS's 1987±45. In large-scale datasets, with 10 tower racks, the effective inspection coordinates reach 5159±62, higher than PMP-GCN's 4914±66 and RL-LS's 4792±58. These results demonstrate that this invention can more effectively cover large-area tower inspection tasks even with a limited number of tower racks.

[0086] Example 4: Verification in a Real-World Power Distribution Line Inspection Scenario To further verify the feasibility of this invention in real-world engineering scenarios, the inspection routes generated by this invention were compared with those designed by professional flight control personnel. Experimental results show that, with four nest settings, the task times for the professional flight control personnel's approach were 4852.68s, 7387.86s, 8706.29s, and 10951.62s, respectively; while the corresponding task times for the method of this invention were 4059.41s, 6132.11s, 6939.29s, and 9005.61s, representing reductions in task time of 19.54%, 20.48%, 25.46%, and 21.64%, respectively.

[0087] This embodiment illustrates that the present invention can flexibly plan the access sequence and return charging behavior according to the power constraints of the UAV in real power distribution line inspection tasks, thereby improving the rationality of battery utilization, shortening the task completion time, and enhancing the actual operation efficiency of the UAV power inspection system.

[0088] The key improvements and points of protection of this invention mainly include the following aspects.

[0089] First, it protects and improves a technical process that unifies and coordinates the selection of drone nests with the planning of drone inspection paths. This process does not select sites separately and then plan paths separately, but achieves collaborative decision-making between nest deployment and inspection paths through status feedback and reward transmission between upper and lower level strategies.

[0090] Second, we protect and improve a two-layer hierarchical near-end policy optimization framework for power line inspection scenarios. The upper-layer policy is used for drone nest deployment and spatial resource allocation, while the lower-layer policy is used for UAV energy-constrained path planning, inspection data collection, and return-to-charge behavior decisions. This hierarchical structure can reduce the state space and action space complexity of end-to-end reinforcement learning.

[0091] Third, an improved optimization operator strategy selection mechanism for nest location is protected. This mechanism represents nest deployment adjustments as problem instances such as swapping, flipping, and destruction, which guide the optimization operator. The policy network selects the most suitable operator based on the current deployment status, candidate nest characteristics, tower demand coordinates, historical feedback, and mask information.

[0092] Fourth, an improved policy search mechanism based on comparing state spaces and action masks is protected. This mechanism improves search efficiency in large-scale combinatorial optimization problems by comparing the dominance relationships between new and old solutions and masking infeasible or invalid actions.

[0093] Fifth, we protect an improved lower-level LSTM actor-commentator decision structure for energy-constrained inspection path planning of UAVs. This structure utilizes global observations for value assessment, local observations for action generation, and a temporal memory mechanism to characterize the long-term dependencies between UAV flight, inspection, return to base, and charging.

[0094] Sixth, an improved hierarchical reward function guided by expert knowledge is protected. This reward function combines advantage difference-driven rewards, dominance relationship evaluation rewards, and lower-level node rewards, while constraining path length, coverage effect, remaining battery power, repeated visits, charging behavior, and multi-objective dominance relationships.

[0095] Seventh, an improved hierarchical group relative curriculum optimization training strategy is protected. This strategy gradually reduces reward sparsity and improves training stability and exploration efficiency in large-scale power line inspection tasks by grouping candidate solutions, calculating group relative advantages, learning the curriculum, and conducting distributed PPO training.

[0096] Eighth, protect the application of the above methods in drone inspections of transmission lines, distribution lines, low-voltage distribution networks, substations, new energy power plants and other infrastructure.

[0097] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. These computer software products can be stored in computer-readable storage media, such as ROM / RAM, magnetic disks, optical disks, etc., and include several instructions to cause a computer device, such as a personal computer, server, or network device, to execute the methods disclosed in the embodiments.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A collaborative optimization method for the deployment and path planning of power line inspection drone nests based on hierarchical deep reinforcement learning, characterized in that: The selection of UAV nests and UAV path planning are modeled as a hierarchical Markov decision process, and a two-layer hierarchical near-end policy optimization network is constructed. The upper-layer network is responsible for searching, adjusting, and optimizing the nest deployment scheme, while the lower-layer network is responsible for generating UAV inspection paths that meet power constraints under the current nest deployment scheme. Through status feedback, reward transmission, and course-based joint training between the upper and lower layers, the coordinated optimization of global spatial resource allocation and local flight behavior planning is achieved.

2. The method for collaborative optimization of power line inspection UAV nest deployment and path planning based on hierarchical deep reinforcement learning according to claim 1, characterized in that: The specific steps include the following: Step 1: Obtain power network inspection task data and build the task environment; Step 2: Establish a hierarchical decision-making model for nest location and path planning; Step 3: Construct a hierarchical near-end strategy optimization network to optimize nesting deployment; Step 4: Construct a lower-level hierarchical near-end policy optimization network to generate energy-constrained inspection paths; Step 5: Construct a hierarchical reward function guided by expert knowledge; Step Six: Construct an action mask and a comparison state feedback mechanism; Step 7: Use a tiered group relative course optimization strategy for training; Step 8: Output the drone nest deployment plan, drone inspection route, and charging schedule plan.

3. The method for collaborative optimization of power line inspection UAV nest deployment and path planning based on hierarchical deep reinforcement learning according to claim 2, characterized in that: Step one first acquires basic data of the power network to be inspected, including the set of pole or tower coordinates, line topology, set of candidate drone nest locations, nest construction cost, number of available nests, number of drones, drone battery capacity, flight speed, energy consumption per unit distance, inspection and data collection time, charging time, mission start and end points, and inspection coverage requirements; Let the set of power inspection points be: ， in, Indicates the first Coordinates of the poles or power equipment to be inspected. Indicates the number of inspection points; let the set of candidate nest locations be: ， in, Indicates the first Candidate nest locations Indicate the number of candidate drone nests; let the set of drones be: ， in, This indicates the number of drones; each drone has a maximum battery capacity. Current battery level Flight speed Energy consumption coefficient per unit distance Inspection and collection of energy consumption coefficients and charging rate ; Based on the above data, construct a drone inspection simulation environment; In the specific execution of step two, the joint problem of nest location selection and path planning is modeled as a hierarchical Markov decision process; the overall state is represented as: ， in, This represents static global information determined by the problem instance, including tower coordinates, line topology, candidate drone nest locations, nest construction costs, drone parameters, and mission scale. This represents dynamic local information that is constantly updated with environmental interactions, including the current nest deployment scheme, the set of inspected towers, the current location of the drone, the remaining battery power of the drone, historical action selections, historical reward feedback, and inactive action mask information; a comparison state space is further introduced; the comparison state space is used to compare the dominance relationships between new and old nest deployment schemes, new and old path planning schemes, or new and old multi-objective solutions, and the comparison results are fed back to the policy network to suppress invalid actions and guide policy search; historical actions and historical rewards are aggregated into historical feedback features: ， ， in, Indicates the past number The selected action or optimization operator, This indicates the corresponding reward. Indicates the forgetting factor, Indicates the length of the history window.

4. The method for collaborative optimization of power line inspection UAV nest deployment and path planning based on hierarchical deep reinforcement learning according to claim 3, characterized in that: Step 3 involves using the upper-layer hierarchical near-end policy optimization network to optimize the UAV nest deployment scheme; let the nest deployment scheme be represented as a binary vector: ， in, Indicates the candidate position Deploy drone nests, Indicates no deployment; The upper-layer policy network outputs the selection probabilities of various nest deployment optimization operators based on the current state, and selects the most suitable problem instance-guided optimization operator based on the probability. Specifically, these include: a swap operator, used to swap the state or order of two or three candidate positions in the encoding to adjust the nest deployment spatial structure; a flip operator, used to add nests to currently undeployed positions or remove nests from currently deployed positions; and a destruction operator, used to remove the most unfavorable nest position or local structure in the current scheme based on objective dimensions such as task time, construction cost, or path cost. The upper-layer policy network adopts an encoder-decoder structure. The encoder receives the current nest deployment scheme, candidate nest features, and inspection requirement point information, and maps them to a shared latent space. A multi-head attention mechanism is used to model the relationship between nest deployment decisions, backup nest positions, and inspection requirement coordinates. The decoder further integrates historical feedback features and action mask information to output the feasible probability distribution of candidate optimization operators.

5. The method for collaborative optimization of power line inspection UAV nest deployment and path planning based on hierarchical deep reinforcement learning according to claim 4, characterized in that: In step four, the lower-level hierarchical near-end policy optimization network is used to generate inspection paths that meet the UAV's energy constraints under the given nest deployment scheme in the upper layer. The lower-level status includes the drone's current location, remaining battery power, set of inspected poles, set of uninspected poles, location of the drone nest, reachability assessment information, and local topology information; the lower-level action space includes the following basic actions: flight actions, controlling the drone to fly in a straight line toward the target pole, target waypoint, or target drone nest; The charging action controls the drone to return to its nest and perform charging; the inspection and data collection action controls the drone to collect images, infrared data, point cloud data, or other inspection data at the target tower or equipment location. The task termination action is used to end the current inspection task when the task completion conditions are met or the termination state is reached. The lower-level strategy adopts an option mechanism, abstracting the drone behavior into three types of options: inspecting a certain tower, returning to the nest to charge, and terminating the mission. For the inspection option, the drone will fly from its current location to the target tower, collect data, and terminate the option after the data collection is completed; for the charging option, the drone will fly to the nest and charge to the set capacity before terminating the option; for the mission termination option, the drone will return to the destination or complete the marking action before terminating the option. The lower-level policy network adopts an actor-critic structure based on LSTM; the value network uses global observation information to evaluate the current state value, and the policy network uses local observation information to generate the current action; the fully connected layer first extracts observation features, and LSTM is used to capture the temporal dependencies in UAV path planning. Finally, the value head and action head output the state value and action probability, respectively.

6. The method for collaborative optimization of power line inspection UAV nest deployment and path planning based on hierarchical deep reinforcement learning according to claim 5, characterized in that: Step 5: In order to ensure that the strategy learning results conform to the engineering constraints of the power inspection task, a hierarchical reward function guided by expert knowledge is designed during the execution process. This reward function includes three parts: advantage difference-driven reward, dominance relationship evaluation reward, and lower-level node reward. The total reward is expressed as follows: ， in, Rewards driven by competitive advantage To evaluate rewards based on the dominance relationship, As a reward for lower-level nodes, These are the weighting coefficients; Advantage-driven rewards are used to measure the improvement of a new solution relative to a baseline solution after performing the current action; they are expressed as: ， in, Indicates the current state. Indicates the current action. This represents the multi-objective normalized improvement value after the action is performed. This indicates the baseline performance before the action is executed; the reward can guide the strategy to prioritize actions that can improve path distance, task time, construction cost, or coverage. The dominance evaluation reward is used to adjust the reward based on the Pareto dominance relationship between the new and old solutions; if the new solution dominates the old solution, a higher positive reward is given; if the new and old solutions do not dominate each other, an exploratory reward is given; if the old solution dominates the new solution or there is no significant difference between the two, the reward is reduced or a penalty is given; this mechanism can promote the strategy to search for non-dominated and dominant solutions and avoid the strategy from repeating ineffective actions in the long term. Lower-level node rewards are used to constrain drone path behavior; these rewards include remaining battery power safety rewards or penalties, repeated visit penalties, and path length rewards or penalties; if the drone has insufficient remaining battery power to safely return to the nest, a penalty is imposed; if the drone repeatedly visits a pole that has already been inspected, a penalty is imposed; if the drone completes more effective inspection points with a shorter path, a positive reward is imposed.

7. The method for collaborative optimization of power line inspection UAV nest deployment and path planning based on hierarchical deep reinforcement learning according to claim 2, characterized in that: Step 6: In the optimization of upper-layer nest deployment, in order to improve the efficiency of policy search, an action masking mechanism is constructed to temporarily block inactive or recently invalid actions from the candidate action set. After each execution of the nest deployment optimization operator or UAV action, the system compares the dominance relationship between the new and old solutions. If the current operator does not produce a non-dominated solution or causes the constraint to be infeasible, a mask is applied to the operator in the next decision, causing the strategy to shift to other candidate actions. The mask information is maintained through a hash table, Boolean vector, or action feasibility matrix.

8. The method for collaborative optimization of power line inspection UAV nest deployment and path planning based on hierarchical deep reinforcement learning according to claim 2, characterized in that: In step seven, to improve training stability and exploration efficiency, a hierarchical group relative course optimization strategy is introduced. The upper-layer nest deployment scheme and the lower-layer inspection path scheme are organized into several candidate output groups, and each output group is scored using a reward model. Then, the relative advantage of each candidate output is calculated based on the average performance within the group or the performance of the reference strategy, and the upper and lower layer policy networks are updated accordingly. The training process adopts the concept of course learning, gradually increasing according to the task difficulty. The first phase is a localized simple inspection phase, in which the drone is trained to complete local pole coverage and safely return to the nest in a small area. The reward function emphasizes basic coverage, return success rate and avoidance of crashes. The second stage is the medium-scale energy-constrained inspection stage; this stage expands the inspection range and the number of towers, and introduces charging frequency, dynamic power balance, repeated access penalty and crash penalty, so that the model learns to arrange flight, inspection and charging behavior under power constraints. The third stage is the global multi-objective collaborative optimization stage, which simultaneously considers the cost of nest construction, inspection coverage efficiency, path length, task completion time and line topology relationship to achieve comprehensive optimization of nest deployment and path planning. In the distributed training approach, multiple worker processes interact with the simulation environment in parallel on multiple GPUs, generating experience data and storing it in a shared replay buffer. When the buffer reaches a set capacity, each GPU samples the training data and calculates the gradients of the policy network and the value network respectively. The master node averages the gradients of each GPU and then updates the global network parameters.

9. The method for collaborative optimization of power line inspection UAV nest deployment and path planning based on hierarchical deep reinforcement learning according to claim 2, characterized in that: After training is completed in step eight, the power network instance to be optimized is input into the system. The upper-layer policy network outputs the drone nest deployment plan, including candidate locations for drone nest deployment, the number of drone nests, and the service area. The lower-layer policy network outputs the inspection path for each drone, including the order of inspected poles, flight path, return to the nest for charging time, the path for continuing inspection after charging, and the mission termination location. The final output includes: a set of drone nest deployment locations; the sequence of inspected poles for each drone; the flight path and mileage for each drone; the power change curve for each drone; the time and location of each return to the nest for charging; the total mission completion time; the number of effective inspection points; the total path distance and energy consumption; and prompts for uncovered poles or abnormal constraints.

10. A collaborative optimization system for the deployment and path planning of power line inspection drone nests based on hierarchical deep reinforcement learning, characterized in that: include: The power network data input module is used to obtain tower coordinates, line topology, candidate drone nest locations, drone parameters, and inspection task requirements. The inspection environment simulation module is used to simulate the flight, inspection, return, charging, and mission termination behavior of drones. The upper-layer nest deployment optimization module is used to optimize the network selection nest deployment optimization operator and update the nest deployment scheme through a hierarchical near-end strategy. The lower-level path planning module is used to generate energy-constrained inspection paths for UAVs through an LSTM-based actor-critic network. The expert reward calculation module is used to calculate advantage difference-driven rewards, dominance relationship assessment rewards, and node-level path rewards. The action mask and comparison status module is used to determine the feasibility of actions, the dominance relationship between new and old schemes, and the masking of invalid actions; The hierarchical group relative course training module is used to train the upper and lower layer policy networks step by step according to the task difficulty; The results output and visualization module is used to output the nesting deployment plan, inspection path, charging scheduling plan, task time, path length and coverage effect.

Images