Unmanned aerial vehicle path planning method and device for hanging falling protector

Abstract

The invention discloses an unmanned aerial vehicle path planning method and device for hanging a falling protector, and the method comprises the steps: collecting the environment data of an electric power tower, employing a hidden Markov model to observe the environment data of the electric power tower, and obtaining an initialized current state and a state transition probability; according to the power tower environment data and the starting point and the ending point of flight of the unmanned aerial vehicle, an ant colony algorithm is adopted for initial path planning, and an initial path is obtained; inputting the initialized current state, the state transition probability and the initialization path into a pre-established deep reinforcement learning model for calculation, and outputting an optimized path; iteratively executing the following operations until a first stop condition is met: updating a path through an ant colony algorithm, updating a current state and a state transition probability, inputting the current state and the state transition probability into the deep reinforcement learning model for learning, and outputting an updated optimized path; and outputting a final planned path when the first stop condition is met. According to the method, the planned path can be accurately provided for the unmanned aerial vehicle hanging the falling protector.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle (UAV) path planning, and in particular to a method and device for UAV path planning for suspension of a fall arrester. Background Art

[0002] There are great safety risks in working at heights, especially on towers. When workers work on towers, they need to face problems such as wind force, temperature changes, and complex working environments, all of which increase the risk of falling. At present, fall prevention technology mainly relies on personal protective equipment (such as safety belts, safety ropes), collective protective equipment (such as protective nets), and aerial work platforms. In addition, safety training and standardized operating procedures are also one of the important means of fall prevention. Drone assistance is the key development direction of high-altitude fall prevention in the future. Drone assistance is the use of drone technology for high-altitude monitoring and work assistance to reduce the time and risk of workers working directly at high altitudes. Drones can conduct real-time monitoring at high-altitude work sites, provide a comprehensive perspective of the environment and workers' conditions, and promptly detect and feedback potential dangerous situations. Drones can transport small tools and materials, reduce the number of times workers frequently go up and down the tower, and thus reduce the risk of falling. In certain specific tasks, drones can directly participate in operations, such as inspecting tower structures, installing or disassembling equipment, etc., thereby replacing some high-altitude operations. Improve safety: Drones reduce the time workers stay at high altitudes, fundamentally reducing the risk of falling. Drones can quickly reach the work site and operate flexibly in complex and dangerous environments, improving overall work efficiency.

[0003] In addition, when workers are working on the iron tower, they need to hang a fall arrester. If a drone is used to hang the fall arrester, the number of times workers frequently go up and down the iron tower can be reduced, thereby reducing the probability of falling. For example, patent text CN219764342U provides a fall arrester fixing device; it includes a drone mounting module, a hanging ring, a fixing module, a safety buckle, a safety rope, a differential fall arrester and a throwing module. The hanging ring is connected to the drone mounting module, the fixing module is fixedly connected to the hanging ring, the safety buckle is detachably connected to the fixing module, the throwing module is fixedly connected to the fixing module and is located above the safety buckle, the safety rope passes through the safety buckle, and the safety rope runs through the differential fall arrester. By connecting the drone to the drone mounting module and connecting the hanging ring to the drone mounting module, the drone is remotely controlled, and the drone is mounted and lowered to the fixed angle steel at the highest point of the operation, and the safety rope is fixed to the fixed angle steel, the installation is completed, and the differential fall arrester is easy to install and the work efficiency is improved. However, the path planning of a drone hanging a fall arrester is different from the inspection path planning of a drone. In the path planning of a drone hanging a fall arrester, there is a possibility of encountering obstacles that may change, such as moving personnel on the power pole tower or installed equipment, which brings difficulties to the drone path planning. Summary of the invention

[0004] The present invention provides a method and device for planning a path of a UAV with a fall arrester suspended, which can accurately provide a planned path for the UAV with the fall arrester suspended.

[0005] A method for planning a path for a drone with a fall arrester suspended, comprising:

[0006] Determine the start and end points of the drone flight;

[0007] Collect power tower environmental data, use hidden Markov model to observe the power tower environmental data, and obtain the initialized current state and state transition probability;

[0008] According to the power tower environmental data, the starting point and the end point of the UAV flight, an ant colony algorithm is used to perform initial path planning to obtain an initialization path;

[0009] Input the initialized current state, state transition probability, and initialized path into the pre-established deep reinforcement learning model for calculation, and output the optimized path;

[0010] Iteratively perform the following operations until the first stop condition is met: according to the optimized path, update the pheromone in the ant colony algorithm, output the updated path, and update the current state and state transition probability at the same time, input the updated path, current state and state transition probability into the deep reinforcement learning model for learning, and output the updated optimized path;

[0011] When the first stopping condition is met, the final planned path is output through the deep reinforcement learning model.

[0012] Furthermore, the power tower environment data includes the flight space of the drone and the initial obstacle position;

[0013] After collecting the power tower environmental data, it also includes:

[0014] The flight space is divided into grid units.

[0015] Furthermore, the hidden Markov model is used to observe the power tower environmental data to obtain the initialized current state and state transition probability, including:

[0016] According to the initial obstacle position, the initial state probability vector is established. The obstacle position is used as the hidden state and the UAV's lidar data is used as the observation value to establish the state transfer matrix and observation probability matrix respectively.

[0017] According to the initial state probability vector, the state transfer matrix and the observation probability matrix, the initialized current state and state transfer probability are obtained.

[0018] Furthermore, according to the power tower environmental data, the starting point and the end point of the UAV flight, an ant colony algorithm is used to perform initial path planning to obtain an initialization path, including:

[0019] Set the number of ants, initialize pheromones, and evenly distribute the initial pheromones on all paths;

[0020] The ants are arranged at the grid unit of the starting point, and each ant searches for a path according to the initial pheromone. Each ant moves to a feasible grid unit until reaching the grid unit of the end point, thereby obtaining an initialized path.

[0021] Furthermore, the deep reinforcement learning model includes an online network and a target network;

[0022] The initialized current state, state transition probability, and initialization path are input into the pre-established deep reinforcement learning model for calculation, and the optimized path is output, including:

[0023] Initialize the online network, target network, and experience replay pool;

[0024] The online network randomly selects a path from the initialization path and calculates the environmental feedback of each action performed in the selected path;

[0025] Calculate and obtain the next state according to the current state and the state transition probability;

[0026] The action, environmental feedback, current state and next state are combined into experience and stored in an experience replay pool;

[0027] The online network and the target network are trained according to the experience to update the parameters of the deep reinforcement learning model;

[0028] Repeat the following steps until the second stop condition is met: the online network again randomly selects a path from the initialization path to calculate experience and stores it in the experience replay pool, the online network and the target network are trained according to the experience, and the parameters of the deep reinforcement learning model are updated;

[0029] The optimized path is output when the second stop condition is met.

[0030] Furthermore, the online network and the target network are trained according to the experience to update the parameters of the deep reinforcement learning model, including:

[0031] The online network randomly extracts experience from the experience replay pool, calculates the Q value in the current state according to the experience, the target network calculates the maximum Q value in the next state according to the experience, and calculates the target Q value according to the Q value in the current state and the maximum Q value in the next state;

[0032] Calculate the loss value according to the target Q value, and update the parameters of the deep reinforcement learning model based on the gradient descent method;

[0033] When the second stopping condition is met, the updated deep reinforcement learning model is used to select all actions with the highest target Q value to construct the optimization path.

[0034] Furthermore, according to the optimized path, the pheromone in the ant colony algorithm is updated, and the updated path is output, including:

[0035] Performing quality assessment on the optimization path to obtain quality assessment parameters;

[0036] calculating updated pheromones according to the quality assessment parameters;

[0037] Distribute the updated pheromones evenly on all paths;

[0038] The ants are arranged at the grid unit at the starting point, and each ant searches for a path according to the updated pheromone. Each ant moves to a feasible grid unit until reaching the grid unit at the end point, thereby obtaining an updated path.

[0039] Furthermore, the quality evaluation of the optimized path includes an evaluation of the length of the optimized path and a reward for successfully avoiding obstacles; the quality evaluation parameter is a sum of the quantified length of the optimized path and the quantified reward for successfully avoiding obstacles;

[0040] The ratio of the length of the quantified optimized path to the quantity evaluation parameter is taken as the updated pheromone.

[0041] A UAV path planning device based on deep reinforcement learning applied to the above method comprises:

[0042] A determination module is used to determine the starting point and end point of the UAV flight;

[0043] The observation module is used to collect the environmental data of the power tower, observe the environmental data of the power tower using the hidden Markov model, and obtain the initialized current state and state transition probability;

[0044] An initial planning module is used to perform initial path planning using an ant colony algorithm according to the power tower environmental data and the starting point and end point of the UAV flight to obtain an initialization path;

[0045] An optimization module is used to input the initialized current state, state transition probability and initialization path into a pre-established deep reinforcement learning model for calculation and output an optimized path;

[0046] An iteration module is used to iteratively perform the following operations until a first stop condition is met: according to the optimization path, update the pheromone in the ant colony algorithm, output the updated path, and update the current state and state transition probability at the same time, input the updated path, current state and state transition probability into the deep reinforcement learning model for learning, and output the updated optimization path;

[0047] An output module is used to output a final planned path through the deep reinforcement learning model when a first stopping condition is met.

[0048] Furthermore, the power tower environment data includes the flight space of the drone and the initial obstacle position;

[0049] After the observation module collects the power tower environmental data, it also includes:

[0050] The flight space is divided into grid units.

[0051] Furthermore, the observation module uses a hidden Markov model to observe the power tower environmental data to obtain the initialized current state and state transition probability, including:

[0052] According to the initial obstacle position, the initial state probability vector is established. The obstacle position is used as the hidden state and the UAV's lidar data is used as the observation value to establish the state transfer matrix and observation probability matrix respectively.

[0053] According to the initial state probability vector, the state transfer matrix and the observation probability matrix, the initialized current state and state transfer probability are obtained.

[0054] Furthermore, the initial planning module uses an ant colony algorithm to perform initial path planning according to the power tower environmental data and the starting point and end point of the UAV flight to obtain an initialization path, including:

[0055] Set the number of ants, initialize pheromones, and evenly distribute the initial pheromones on all paths;

[0056] The ants are arranged at the grid unit of the starting point, and each ant searches for a path according to the initial pheromone. Each ant moves to a feasible grid unit until reaching the grid unit of the end point, thereby obtaining an initialized path.

[0057] Furthermore, the deep reinforcement learning model includes an online network and a target network;

[0058] The optimization module inputs the initialized current state, state transition probability and initialization path into a pre-established deep reinforcement learning model for calculation, and outputs an optimization path, including:

[0059] Initialize the online network, target network, and experience replay pool;

[0060] The online network randomly selects a path from the initialization path and calculates the environmental feedback of each action performed in the selected path;

[0061] Calculate and obtain the next state according to the current state and the state transition probability;

[0062] The action, environmental feedback, current state and next state are combined into experience and stored in an experience replay pool;

[0063] The online network and the target network are trained according to the experience to update the parameters of the deep reinforcement learning model;

[0064] Repeat the following steps until the second stop condition is met: the online network again randomly selects a path from the initialization path to calculate experience and stores it in the experience replay pool, the online network and the target network are trained according to the experience, and the parameters of the deep reinforcement learning model are updated;

[0065] The optimized path is output when the second stop condition is met.

[0066] Furthermore, the online network and the optimization module of the target network are trained according to the experience to update the parameters of the deep reinforcement learning model, including:

[0067] The online network randomly extracts experience from the experience replay pool, calculates the Q value in the current state according to the experience, the target network calculates the maximum Q value in the next state according to the experience, and calculates the target Q value according to the Q value in the current state and the maximum Q value in the next state;

[0068] Calculate the loss value according to the target Q value, and update the parameters of the deep reinforcement learning model based on the gradient descent method;

[0069] When the second stopping condition is met, the updated deep reinforcement learning model is used to select all actions with the highest target Q value to construct the optimization path.

[0070] Furthermore, the iteration module updates the pheromone in the ant colony algorithm according to the optimization path and outputs the updated path, including:

[0071] Performing quality assessment on the optimization path to obtain quality assessment parameters;

[0072] calculating updated pheromones according to the quality assessment parameters;

[0073] Distribute the updated pheromones evenly on all paths;

[0074] The ants are arranged at the grid unit at the starting point, and each ant searches for a path according to the updated pheromone. Each ant moves to a feasible grid unit until reaching the grid unit at the end point, thereby obtaining an updated path.

[0075] Furthermore, the quality evaluation of the optimized path includes an evaluation of the length of the optimized path and a reward for successfully avoiding obstacles; the quality evaluation parameter is a sum of the quantified length of the optimized path and the quantified reward for successfully avoiding obstacles;

[0076] The ratio of the length of the quantified optimized path to the quantity evaluation parameter is taken as the updated pheromone.

[0077] An electronic device comprises a processor and a storage device, wherein the storage device stores a plurality of instructions, and the processor is used to read the instructions and execute the above method.

[0078] The method and device for drone path planning for fall arrester suspension provided by the present invention have at least the following beneficial effects:

[0079] Taking into account the obstacles such as moving personnel and equipment on the power poles and towers that may be encountered in the path planning of the UAV with a fall arrester, the hidden Markov model is used to observe the environmental dynamics, providing a basis for the path planning of the UAV; the ant colony algorithm and the deep reinforcement learning algorithm are combined for path planning, and the advantages of the ant colony algorithm are used to realize environmental observation in the path planning of the UAV. The deep reinforcement learning algorithm is used to ensure that the UAV interacts with the environment in a dynamically changing environment, and plans an effective path to ensure the accuracy of the path planning of the UAV with a fall arrester. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 A flow chart of an embodiment of a method for planning a path for a drone with fall arrester suspension provided by the present invention.

[0081] Figure 2 A flow chart of an embodiment of observing power tower environmental data in a method for UAV path planning for fall arrester suspension provided by the present invention.

[0082] Figure 3 A flow chart of an embodiment of initial path planning in a method for path planning of a drone with fall arrester suspension provided by the present invention.

[0083] Figure 4 A flowchart of an embodiment of reinforcement learning in a method for drone path planning for fall arrester suspension provided by the present invention.

[0084] Figure 5 A flowchart of an embodiment of updating a path in a path planning method for a drone with fall arrester suspension provided by the present invention.

[0085] Figure 6 A schematic structural diagram of an embodiment of a UAV path planning device for suspension of a fall arrester provided by the present invention. DETAILED DESCRIPTION

[0086] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0087] refer to Figure 1 In some embodiments, a method for drone path planning for fall arrester suspension is provided, comprising:

[0088] S1. Determine the starting point and end point of the drone flight;

[0089] S2, collecting power tower environmental data, using hidden Markov model to observe the power tower environmental data, and obtaining the initialized current state and state transition probability;

[0090] S3, according to the power tower environmental data, the starting point and the end point of the UAV flight, using the ant colony algorithm to perform initial path planning to obtain an initialization path;

[0091] S4, inputting the initialized current state, state transition probability and initialized path into a pre-established deep reinforcement learning model for calculation, and outputting an optimized path;

[0092] S5, iteratively performing the following operations until the first stop condition is met: according to the optimized path, updating the pheromone in the ant colony algorithm, outputting the updated path, and updating the current state and state transition probability at the same time, inputting the updated path, current state and state transition probability into the deep reinforcement learning model for learning, and outputting the updated optimized path;

[0093] S6. When the first stopping condition is met, the final planned path is output through the deep reinforcement learning model.

[0094] Specifically, in step S1, the starting point of the drone may be a take-off point on the ground, and the end point may be a position on the power tower where a fall arrester needs to be hung.

[0095] Further, in step S2, the power tower environment data includes the flight space of the drone and the initial obstacle position;

[0096] After collecting the power tower environmental data, it also includes:

[0097] The flight space is divided into grid units.

[0098] Specifically, power tower environmental data can be collected through laser radar, image acquisition devices, etc.

[0099] Further, in step S2, reference Figure 2 , the hidden Markov model is used to observe the power tower environmental data to obtain the initialized current state and state transition probability, including:

[0100] S21. According to the initial obstacle position, an initial state probability vector is established, and the obstacle position is used as a hidden state and the UAV's lidar data is used as an observation value to respectively establish a state transfer matrix and an observation probability matrix;

[0101] S22. Obtain an initialized current state and state transition probability according to the initial state probability vector, the state transition matrix and the observation probability matrix.

[0102] Specifically, in step S21, the initial obstacle position is feature quantized, an initial state probability vector is established, all possible states and all possible observations of the obstacle position are obtained, the corresponding state is generated according to the distribution of the initial state probability vector, the observation probability matrix is ​​generated according to the observation probability of the state, and the state transfer matrix is ​​generated according to the transition probability between the states. The obstacle position is taken as the hidden state, and the laser radar data of the drone is taken as the observation value, and the state transfer matrix and the observation probability matrix are established respectively. In the hidden Markov model, the hidden state refers to those states that cannot be directly observed in practice, but the state whose existence can be inferred by the observed data. There is a certain probability relationship between the hidden state and the observed state. The hidden state represents some states that cannot be directly observed but affect the generation of the observed data, while the observed state is the data that can be directly observed. Each hidden state corresponds to a set of observation probabilities, which represents the probability of generating a certain observation value under this state. Therefore, in this embodiment, the dynamic obstacle position is taken as the hidden state, the laser radar data of the drone is taken as the observation value, the obstacle position and the laser radar data are feature quantized, such as vector or unique hot encoding, and then the state transfer matrix and the observation probability matrix are established.

[0103] Further, in step S2, the initialized current state and state transition probability are obtained according to the initial state probability vector, the state transfer matrix and the observation probability matrix. Specifically, the initial current state is determined according to the initial state probability vector. The initial state probability vector contains the probability distribution of each state at the initial moment. Through this vector, the probability of each state at the initial moment can be determined. The state transfer matrix describes the probability of the system transferring from one hidden state to another hidden state. It is a two-dimensional matrix, and each element in the matrix represents the probability of transferring from one hidden state i to another hidden state j. The state transition probability is obtained according to the state transfer matrix.

[0104] Further, refer to Figure 3In step S3, according to the power tower environment data, the starting point and the end point of the UAV flight, an ant colony algorithm is used to perform initial path planning to obtain an initialization path, including:

[0105] S31, set the number of ants, initialize pheromones, and evenly distribute the initial pheromones on all paths;

[0106] S32, setting the ants at the grid unit at the starting point, each ant searches for a path according to the initial pheromone, and each ant moves to a feasible grid unit until reaching the grid unit at the end point, thereby obtaining an initialized path.

[0107] Specifically, in step S31, each "ant" represents an independent intelligent agent responsible for searching for paths in the environment. Each ant moves according to the current state and available paths until it reaches the target. Pheromones are chemical substances left by ants on the path, which are used to indicate the quality of the path. The higher the pheromone concentration, the better the path, and other ants are more inclined to choose paths with high pheromone concentrations when choosing paths. In this embodiment, pheromones include the length of the path and whether obstacles are avoided.

[0108] Furthermore, in step S2, the ants are set at the grid unit of the starting point, and each ant searches for a path according to the initial pheromone, and each ant moves to a feasible grid unit until reaching the grid unit of the end point, thereby obtaining an initialized path.

[0109] Further, in step S4, reference Figure 4 , input the initialized current state, state transition probability and initialization path into the pre-established deep reinforcement learning model for calculation, and output the optimized path, including:

[0110] S41, initializing the online network, target network and experience replay pool;

[0111] S42, the online network randomly selects a path from the initialization path, and calculates the environmental feedback of each action executed in the selected path;

[0112] S43, calculating and obtaining the next state according to the current state and the state transition probability;

[0113] S44, the action, environmental feedback, current state and next state are combined into experience and stored in an experience playback pool;

[0114] S45, the online network and the target network are trained according to the experience, and the parameters of the deep reinforcement learning model are updated;

[0115] S46, repeat the following steps until the second stop condition is met: the online network again randomly selects a path from the initialization path to calculate experience and stores it in the experience replay pool, the online network and the target network are trained according to the experience, and the parameters of the deep reinforcement learning model are updated;

[0116] S47: Output the optimized path when the second stop condition is met.

[0117] Specifically, the deep reinforcement learning model (DQN, Deep Q-Network) is an algorithm based on deep learning and reinforcement learning. In this embodiment, the current state, state transition probability output by the hidden Markov model, and the initialization path obtained by the ant colony algorithm are used as inputs of the deep reinforcement learning model to train it to output an optimized path.

[0118] Furthermore, in step S41, the deep reinforcement learning model includes two networks, namely an online network and a target network. The online network is responsible for collecting experience and gradient descent, and the target network is responsible for calculating the Q value of the next state. The two have the same structure. The experience replay pool is used to store the experience gained during the calculation process and initialize the parameters of the online network and the target network.

[0119] Further, in step S42, the online network randomly selects a path from the initialization path, and calculates the environmental feedback of each action performed in the selected path. In this embodiment, the environmental feedback can be a reward for performing an action, including positive rewards, negative rewards, and immediate rewards. If the drone successfully reaches the target location or the selected path avoids obstacles, a positive reward can be given to encourage the drone to continue to choose similar paths. If the drone hits an obstacle, flies out of the allowed area, or fails to reach the target within the specified time, a negative reward can be given. This is intended to punish undesirable choices and encourage drones to learn to avoid these situations. The immediate reward can be set as immediate feedback for reaching a certain location, or dynamically adjusted according to the efficiency of the path (such as flight distance, time, etc.).

[0120] Furthermore, in step S43 and step S44, the next state is calculated based on the current state and the state transition probability, and the action, environmental feedback, current state and next state constitute a quadruple of experience, which is used to optimize and train the deep reinforcement learning model.

[0121] Furthermore, in step S45, the online network and the target network are trained according to the experience to update the parameters of the deep reinforcement learning model, including:

[0122] The online network randomly extracts experience from the experience replay pool, calculates the Q value in the current state according to the experience, the target network calculates the maximum Q value in the next state according to the experience, and calculates the target Q value according to the Q value in the current state and the maximum Q value in the next state;

[0123] The loss value is calculated according to the target Q value, and the parameters of the deep reinforcement learning model are updated based on the gradient descent method.

[0124] Specifically, the experience randomly drawn from the experience replay pool can be used as a sample. For each drawn experience, the Q value in the current state can be calculated according to the Q function. The target network can calculate the target Q value using the following formula:

[0125] Q(s,a)=r+γmax a′ Q(s′, a′); (1)

[0126] Among them, Q(s,a) represents the target Q value, r represents the reward, γ represents the discount factor, and max a′ Q(s′, a′) represents the maximum Q value of the next state, s represents the state, a represents the action, s′ represents the next state, and a′ represents the optimal action for the next state.

[0127] Furthermore, according to the target Q value, the loss value is calculated based on the loss function, and the loss value is minimized based on gradient descent and the parameters of the deep reinforcement learning model are updated. Specifically, the parameters of the target network can be updated using the parameters of the online network after a certain number of steps.

[0128] Furthermore, in step S47, when the second stop condition is met, the updated deep reinforcement learning model is used to select all actions with the highest target Q value to construct an optimization path.

[0129] In some embodiments, the second stop condition may include: the number of training times reaches a preset number, the target Q value converges, the number of experiences in the experience replay pool reaches a preset number, the training time reaches a preset duration, etc.

[0130] Further, refer to Figure 5 In step S5, the pheromone in the ant colony algorithm is updated according to the optimized path, and the updated path is output, including:

[0131] S51, performing quality assessment on the optimization path to obtain quality assessment parameters;

[0132] S52, calculating updated pheromones according to the quality assessment parameters;

[0133] S53, evenly distributing the updated pheromone on all paths;

[0134] S54, setting the ants at the grid unit at the starting point, each ant searches for a path according to the updated pheromone, and each ant moves to a feasible grid unit until reaching the grid unit at the end point, thereby obtaining an updated path.

[0135] Furthermore, in step S51, the quality assessment of the optimized path includes an assessment of the length of the optimized path and a reward for successfully avoiding obstacles; the quality assessment parameter is the sum of the quantified length of the optimized path and the quantified reward for successfully avoiding obstacles.

[0136] Specifically, the length of the optimization path can be quantified as L k If the obstacle is successfully avoided, a positive reward is given, otherwise a negative reward is given, and the reward is quantified as R k , then the quality assessment parameter Q k =L k +R k .

[0137] Furthermore, in step S52, the ratio of the length of the quantified optimized path to the quantity evaluation parameter is used as the updated pheromone.

[0138] In some embodiments, an evaporation rate may also be set, and the pheromone may evaporate to a certain extent according to the evaporation rate.

[0139] In step S53 and step S54, the updated pheromones are evenly distributed on all paths, and each ant searches for a path based on the updated pheromones. Each ant moves to a feasible grid unit until it reaches the end grid unit, thereby obtaining an updated path.

[0140] Furthermore, in step S5, the hidden Markov model updates the current state and the state transition probability, and can be updated using a forward algorithm according to the state transition matrix and the observation probability matrix.

[0141] Furthermore, in step S5, after obtaining the updated path output by the ant colony algorithm and the updated current state and state transition probability output by the hidden Markov algorithm, the updated path, current state and state transition probability are used as inputs of the deep reinforcement learning model, and the deep reinforcement learning model is used again for learning and calculation, that is, steps S42 to S47 are executed again: the online network randomly selects a path from the updated path, and calculates the environmental feedback of performing each action in the selected path; according to the updated current state and state transition probability, the next state is calculated; the action, environmental feedback, current state and next state are combined into experience and stored in the experience replay pool; the online network and the target network are trained according to the experience, and the parameters of the deep reinforcement learning model are updated; the following steps are repeated until the second stop condition is met: the online network again randomly selects a path from the initialization path to calculate the experience and stores it in the experience replay pool, the online network and the target network are trained according to the experience, and the parameters of the deep reinforcement learning model are updated; the optimized path is output when the second stop condition is met. After that, the pheromone is updated again according to the output optimized path, the path is updated by the ant colony algorithm, the current state and state transition probability are updated by the hidden Markov model, and the deep reinforcement learning model is trained again as the input of the deep reinforcement learning model. This process is repeated until the first stopping condition is met, and the final planned path is output by the deep reinforcement learning model.

[0142] In some embodiments, the first stop condition may include: reaching a preset number of cycles, verifying that the performance of the deep reinforcement learning model has met the requirements, the learning time has reached a preset duration, etc.

[0143] Further, refer to Figure 6 In some embodiments, a UAV path planning device based on deep reinforcement learning applied to the above method is provided, comprising:

[0144] The determination module 201 is used to determine the start and end point of the UAV flight;

[0145] An observation module 202 is used to collect power tower environmental data, observe the power tower environmental data using a hidden Markov model, and obtain an initialized current state and state transition probability;

[0146] The initial planning module 203 is used to perform initial path planning using an ant colony algorithm according to the power tower environmental data and the starting point and end point of the UAV flight to obtain an initialization path;

[0147] The optimization module 204 is used to input the initialized current state, state transition probability and initialization path into a pre-established deep reinforcement learning model for calculation, and output an optimized path;

[0148] The iteration module 205 is used to iteratively perform the following operations until the first stop condition is met: according to the optimized path, the pheromone in the ant colony algorithm is updated, the updated path is output, and the current state and the state transition probability are updated at the same time, the updated path, the current state and the state transition probability are input into the deep reinforcement learning model for learning, and the updated optimized path is output;

[0149] The output module 206 is used to output the final planned path through the deep reinforcement learning model when the first stop condition is met.

[0150] Furthermore, the power tower environment data includes the flight space of the drone and the initial obstacle position;

[0151] After the observation module 202 collects the power tower environment data, it also includes:

[0152] The flight space is divided into grid units.

[0153] Furthermore, the observation module 202 uses a hidden Markov model to observe the power tower environmental data to obtain the initialized current state and state transition probability, including:

[0154] According to the initial obstacle position, the initial state probability vector is established. The obstacle position is used as the hidden state and the UAV's lidar data is used as the observation value to establish the state transfer matrix and observation probability matrix respectively.

[0155] According to the initial state probability vector, the state transfer matrix and the observation probability matrix, the initialized current state and state transfer probability are obtained.

[0156] Furthermore, the initial planning module 203 uses an ant colony algorithm to perform initial path planning according to the power tower environment data and the starting point and end point of the UAV flight to obtain an initialization path, including:

[0157] Set the number of ants, initialize pheromones, and evenly distribute the initial pheromones on all paths;

[0158] The ants are arranged at the grid unit of the starting point, and each ant searches for a path according to the initial pheromone. Each ant moves to a feasible grid unit until reaching the grid unit of the end point, thereby obtaining an initialized path.

[0159] Furthermore, the deep reinforcement learning model includes an online network and a target network;

[0160] The optimization module 204 inputs the initialized current state, state transition probability and initialization path into the pre-established deep reinforcement learning model for calculation, and outputs the optimization path, including:

[0161] Initialize the online network, target network, and experience replay pool;

[0162] The online network randomly selects a path from the initialization path and calculates the environmental feedback of each action performed in the selected path;

[0163] Calculate and obtain the next state according to the current state and the state transition probability;

[0164] The action, environmental feedback, current state and next state are combined into experience and stored in an experience replay pool;

[0165] The online network and the target network are trained according to the experience to update the parameters of the deep reinforcement learning model;

[0166] Repeat the following steps until the second stop condition is met: the online network again randomly selects a path from the initialization path to calculate experience and stores it in the experience replay pool, the online network and the target network are trained according to the experience, and the parameters of the deep reinforcement learning model are updated;

[0167] The optimized path is output when the second stop condition is met.

[0168] Furthermore, the optimization module 204 performs training based on the experience and updates the parameters of the deep reinforcement learning model, including:

[0169] The online network randomly extracts experience from the experience replay pool, calculates the Q value in the current state according to the experience, the target network calculates the maximum Q value in the next state according to the experience, and calculates the target Q value according to the Q value in the current state and the maximum Q value in the next state;

[0170] Calculate the loss value according to the target Q value, and update the parameters of the deep reinforcement learning model based on the gradient descent method;

[0171] When the second stopping condition is met, the updated deep reinforcement learning model is used to select all actions with the highest target Q value to construct the optimization path.

[0172] Furthermore, the iteration module 205 updates the pheromone in the ant colony algorithm according to the optimized path and outputs the updated path, including:

[0173] Performing quality assessment on the optimization path to obtain quality assessment parameters;

[0174] calculating updated pheromones according to the quality assessment parameters;

[0175] Distribute the updated pheromones evenly on all paths;

[0176] The ants are arranged at the grid unit at the starting point, and each ant searches for a path according to the updated pheromone. Each ant moves to a feasible grid unit until reaching the grid unit at the end point, thereby obtaining an updated path.

[0177] Furthermore, the quality evaluation of the optimized path includes an evaluation of the length of the optimized path and a reward for successfully avoiding obstacles; the quality evaluation parameter is a sum of the quantified length of the optimized path and the quantified reward for successfully avoiding obstacles;

[0178] The ratio of the length of the quantified optimized path to the quantity evaluation parameter is taken as the updated pheromone.

[0179] In some embodiments, an electronic device is also provided, including a processor and a storage device, wherein the storage device stores a plurality of instructions, and the processor is used to read the instructions and execute the above method.

[0180] The above-mentioned embodiments provide a method and device for drone path planning for fall arrester suspension, which at least have the following beneficial effects:

[0181] Taking into account the obstacles such as moving personnel and equipment on the power poles and towers that may be encountered in the path planning of the UAV with a fall arrester, the hidden Markov model is used to observe the environmental dynamics, providing a basis for the path planning of the UAV; the ant colony algorithm and the deep reinforcement learning algorithm are combined for path planning, and the advantages of the ant colony algorithm are used to realize environmental observation in the path planning of the UAV. The deep reinforcement learning algorithm is used to ensure that the UAV interacts with the environment in a dynamically changing environment, and plans an effective path to ensure the accuracy of the path planning of the UAV with a fall arrester.

[0182] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for planning a path for an unmanned aerial vehicle with a fall arrester suspended therefrom, characterized in that: include: Determine the start and end points of the drone flight; Collect power tower environmental data, use hidden Markov model to observe the power tower environmental data, and obtain the initialized current state and state transition probability; According to the power tower environmental data, the starting point and the end point of the UAV flight, an ant colony algorithm is used to perform initial path planning to obtain an initialization path; Input the initialized current state, state transition probability, and initialized path into the pre-established deep reinforcement learning model for calculation, and output the optimized path; Iteratively perform the following operations until the first stop condition is met: according to the optimized path, update the pheromone in the ant colony algorithm, output the updated path, and update the current state and state transition probability at the same time, input the updated path, current state and state transition probability into the deep reinforcement learning model for learning, and output the updated optimized path; When the first stopping condition is met, the final planned path is output through the deep reinforcement learning model.

2. The method according to claim 1, characterized in that: The power tower environment data includes the flight space of the UAV and the initial obstacle position; After collecting the power tower environmental data, it also includes: The flight space is divided into grid units.

3. The method according to claim 2, characterized in that The hidden Markov model is used to observe the environmental data of the power tower to obtain the initialized current state and state transition probability, including: According to the initial obstacle position, the initial state probability vector is established. The obstacle position is used as the hidden state and the UAV's lidar data is used as the observation value to establish the state transfer matrix and observation probability matrix respectively. According to the initial state probability vector, the state transfer matrix and the observation probability matrix, the initialized current state and state transfer probability are obtained.

4. The method according to claim 2, characterized in that: According to the power tower environmental data, the starting point and the end point of the UAV flight, the ant colony algorithm is used to perform initial path planning to obtain the initialization path, including: Set the number of ants, initialize pheromones, and evenly distribute the initial pheromones on all paths; The ants are arranged at the grid unit of the starting point, and each ant searches for a path according to the initial pheromone. Each ant moves to a feasible grid unit until reaching the grid unit of the end point, thereby obtaining an initialized path.

5. The method according to claim 1, characterized in that The deep reinforcement learning model includes an online network and a target network; The initialized current state, state transition probability, and initialization path are input into the pre-established deep reinforcement learning model for calculation, and the optimized path is output, including: Initialize the online network, target network, and experience replay pool; The online network randomly selects a path from the initialization path and calculates the environmental feedback of each action performed in the selected path; Calculate the next state according to the current state and the state transition probability; The action, environmental feedback, current state and next state are combined into experience and stored in an experience replay pool; The online network and the target network are trained according to the experience to update the parameters of the deep reinforcement learning model; Repeat the following steps until the second stop condition is met: the online network randomly selects a path from the initialization path again to calculate experience and stores it in the experience replay pool, the online network and the target network are trained according to the experience, and the parameters of the deep reinforcement learning model are updated; The optimized path is output when the second stop condition is met.

6. The method according to claim 5, characterized in that The online network and the target network are trained according to the experience to update the parameters of the deep reinforcement learning model, including: The online network randomly extracts experience from the experience replay pool, calculates the Q value in the current state according to the experience, the target network calculates the maximum Q value in the next state according to the experience, and calculates the target Q value according to the Q value in the current state and the maximum Q value in the next state; Calculate the loss value according to the target Q value, and update the parameters of the deep reinforcement learning model based on the gradient descent method; When the second stopping condition is met, the updated deep reinforcement learning model is used to select all actions with the highest target Q value to construct the optimization path.

7. The method according to claim 4, characterized in that According to the optimized path, the pheromone in the ant colony algorithm is updated and the updated path is output, including: Performing quality assessment on the optimization path to obtain quality assessment parameters; calculating updated pheromones according to the quality assessment parameters; Distribute the updated pheromones evenly on all paths; The ants are arranged at the grid unit at the starting point, and each ant searches for a path according to the updated pheromone. Each ant moves to a feasible grid unit until reaching the grid unit at the end point, thereby obtaining an updated path.

8. The method according to claim 7, characterized in that The quality evaluation of the optimized path includes an evaluation of the length of the optimized path and a reward for successfully avoiding obstacles; the quality evaluation parameter is a sum of the quantified length of the optimized path and the quantified reward for successfully avoiding obstacles; The ratio of the length of the quantified optimized path to the quantity evaluation parameter is taken as the updated pheromone.

9. A UAV path planning device based on deep reinforcement learning applied to the method as claimed in any one of claims 1 to 8, characterized in that: include: A determination module is used to determine the starting point and end point of the UAV flight; The observation module is used to collect the environmental data of the power tower, observe the environmental data of the power tower using the hidden Markov model, and obtain the initialized current state and state transition probability; An initial planning module is used to perform initial path planning using an ant colony algorithm according to the power tower environmental data and the starting point and end point of the UAV flight to obtain an initialization path; An optimization module is used to input the initialized current state, state transition probability and initialization path into a pre-established deep reinforcement learning model for calculation and output an optimized path; An iteration module is used to iteratively perform the following operations until a first stop condition is met: according to the optimization path, update the pheromone in the ant colony algorithm, output the updated path, and update the current state and state transition probability at the same time, input the updated path, current state and state transition probability into the deep reinforcement learning model for learning, and output the updated optimization path; An output module is used to output a final planned path through the deep reinforcement learning model when a first stopping condition is met.

10. An electronic device, characterized in that: The method comprises a processor and a storage device, wherein the storage device stores a plurality of instructions, and the processor is used to read the instructions and execute the method according to any one of claims 1 to 8.

Images