UAV tower inspection and charging scheduling method based on deep reinforcement learning

Abstract

The present invention relates to a method for unmanned aerial vehicle tower inspection and charging scheduling based on deep reinforcement learning, comprising obtaining inspection information, obtaining the inspection information, wherein the inspection information includes the coordinates of an iron tower to be inspected and the remaining power of the iron tower, and planning an inspection route set according to the inspection information; when the unmanned aerial vehicle performs an inspection task, calculating the average energy consumption rate and the remaining survival time of the unmanned aerial vehicle, and dividing the running time of the inspection route set into a plurality of charging cycles; calculating a charging scheduling scheme of an iron tower node according to the remaining power of the iron tower, and modeling and solving a charging sequence planning problem in the charging scheduling scheme according to a DQN charging sequence optimization algorithm, and obtaining a charging path sequence of the unmanned aerial vehicle; sequentially visiting the iron tower nodes to be charged according to the charging sequence path sequence and performing the inspection task; and comprehensively considering multiple factors such as the energy situation, location information and failure factors of the iron tower nodes, so as to achieve more comprehensive and intelligent inspection and charging scheduling planning.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle inspection and charging scheduling, and in particular to a method for unmanned aerial vehicle tower inspection and charging scheduling based on deep reinforcement learning. Background Art

[0002] A power tower is a metal structure for erecting high-voltage transmission lines. It is usually made of steel or concrete. Its main function is to support high-voltage transmission lines, protect the lines from the influence of the external environment, and facilitate the maintenance and repair of transmission lines. In order to ensure its safe use, it needs to be regularly maintained and inspected. The drone-based power tower maintenance and inspection method is an efficient, accurate and safe inspection method. Compared with the traditional manual inspection method, this method has significant advantages such as simple operation, high efficiency and low risk, which can greatly improve the management efficiency of power towers. At the same time, combined with artificial intelligence technology, automation can be achieved in data processing and analysis, reducing the workload and improving work efficiency.

[0003] However, the flight distance and operation time of the UAV are limited by power. When the power is low, the operation needs to be interrupted and returned for charging. The round trip not only wastes time, but is also not conducive to the normal operation, reducing the operation efficiency. The method to solve this problem generally adopts to set a charging device on the tower node. However, since the charging devices in all tower nodes are not connected to the mains, the battery energy storage capacity in the tower nodes using batteries is different. It is difficult for UAVs to complete long-term inspection tasks between multiple towers in harsh outdoor environments. When the inspection task execution time is too long, the UAV is difficult to reach the next tower node due to insufficient power; when the charging task execution time is too long, the completion efficiency of the UAV inspection task is low and the node resources are wasted. The present invention provides a UAV tower inspection and charging scheduling method based on deep reinforcement learning. Summary of the invention

[0004] In view of the technical problems existing in the prior art, the present invention provides a UAV tower inspection and charging scheduling method based on deep reinforcement learning.

[0005] The technical solution of the present invention to solve the above technical problems is as follows: a method for UAV tower inspection and charging scheduling based on deep reinforcement learning, comprising the following steps:

[0006] S1: Obtain inspection information, which includes the coordinates of the tower to be inspected and the remaining power of the tower, and plan an inspection route set according to the inspection information;

[0007] S2: When the drone performs an inspection task, the average energy consumption rate and the remaining survival time of the drone are calculated, and the running time of the inspection route set is divided into multiple charging cycles;

[0008] S3: Calculate the charging scheduling plan of the tower node according to the remaining power of the tower, and model and solve the charging sequence planning problem in the charging scheduling plan according to the DQN charging sequence optimization algorithm to find the charging path sequence of the drone;

[0009] S4: Visit the tower nodes to be charged in sequence according to the charging sequence path sequence and perform inspection tasks.

[0010] Furthermore, the UAV tower inspection and charging scheduling method based on deep reinforcement learning, when the UAV performs the inspection task, calculates the average energy consumption rate and the remaining survival time of the UAV, and divides the running time of the inspection route set into multiple charging cycle steps, including:

[0011] The average energy consumption rate is calculated by using the weighted average method, using the charging cycle identifier as the weight for calculating the energy consumption rate. The calculation method is:

[0012] ;

[0013] In the formula, Represents the base station's estimate of the energy consumption rate of tower node i after receiving m pieces of remaining energy value information; The time when tower node i records the mth piece of remaining energy information.

[0014] Furthermore, in the UAV tower inspection and charging scheduling method based on deep reinforcement learning, the calculation method of the remaining survival time is:

[0015] ;

[0016] is the remaining energy of the drone at tower node i; is the average energy consumption rate.

[0017] Furthermore, in the UAV tower inspection and charging scheduling method based on deep reinforcement learning, the DQN charging sequence optimization algorithm includes:

[0018] Since the charging sequence planning problem has a discrete action space, the DQN algorithm is used to convert the charging sequence optimization of the drone into a reward maximization problem in the DQN algorithm; the action space of the drone is defined as ; Indicates that the drone replenishes energy at tower node i; during the charging scheduling process, each time an action is executed, the environment will feed back a reward Q to the drone and update it to a new environmental state; when the drone reaches the final state, the total reward is calculated, and the Q function is trained to make the drone obtain as much reward as possible, thereby constructing the optimal charging sequence for the drone;

[0019] Initialize charging path sequence CS;

[0020] Selecting actions with a greedy strategy ;

[0021] Executing actions in the simulator And get corresponding rewards ;

[0022] Add to To charging sequence ;

[0023] Update the dataset .

[0024] Furthermore, the UAV tower inspection and charging scheduling method based on deep reinforcement learning includes:

[0025] Consider the tower node energy factor , Tower node location factors and tower node failure factors :

[0026]

[0027] is the average energy consumption rate of tower nodes; is the remaining energy of the drone at tower node i; is the distance between the current node to be charged and the next node to be charged, α is the penalty coefficient, when the drone is in state Execute an action After the charging sequence When a node fails, a negative reward is fed back to the DQN algorithm through the reward function.

[0028] Furthermore, in the UAV tower inspection and charging scheduling method based on deep reinforcement learning, the reward function includes:

[0029] ;

[0030] Where K is the number of charging rounds.

[0031] The beneficial effects of the present invention are:

[0032] The present invention optimizes the charging sequence and node charging duration in UAV tower inspection and charging scheduling from the two dimensions of space and time, further reduces the charging cost while ensuring the node energy supply, and realizes UAV tower inspection tasks and charging scheduling tasks with optimized efficiency.

[0033] Through deep reinforcement learning, the system can intelligently select the optimal inspection and charging path under different conditions. Using Q functions for real-time optimization, decisions can be dynamically adjusted according to environmental changes, avoiding the limitations of traditional static planning; drones can maximize inspection efficiency, reduce task execution time, and increase task completion rate; the optimized path can effectively avoid task interruptions due to insufficient power.

[0034] Comprehensive consideration of the energy status, location information and failure factors of the tower nodes can optimize the charging scheduling plan, improve battery utilization, and ensure that the drone always has sufficient power during inspection missions. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of the optimization problem of the tower inspection task and the charging task of the UAV of the present invention;

[0036] Figure 2 This is a schematic diagram of the optimized UAV tower inspection task of the present invention;

[0037] Figure 3 It is a schematic diagram of the charging scheduling method in the optimized UAV tower inspection mission of the present invention.

[0038] Figure 4 This is a flowchart of the drone tower inspection and charging scheduling method based on deep reinforcement learning. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0040] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0041] In the description of the present application, the term "for example" is used to mean "used as an example, illustration or description". Any embodiment described as "for example" in the present application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid unnecessary details to obscure the description of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.

[0042] like Figure 1 As shown in the figure, the tower node is equipped with a charging device for the drone. Assume that the drone system receives inspection information requests from tower nodes A, B, E, F, and G. If the path optimization method is used, the shortest path for the drone is: A→B→F→G→E. However, the above path planning does not take into account the situation where the drone runs out of battery energy during the tower inspection process. In the method before optimization, the drone runs out of battery power at nodes F and G, and needs to be charged for a long time; the optimized method takes into account the power consumption of the drone when it reaches the tower node, so it can perform the inspection task of the next node after a short charging time. That is, the existing scheduling optimization algorithm only considers the planning of the spatial dimension, but does not consider the planning of the time dimension.

[0043] In one embodiment, Figure 4 As shown in the figure, the specific implementation steps for optimizing the above problems through the UAV tower inspection and charging scheduling method based on deep reinforcement learning include the following:

[0044] S1: Obtain inspection information, which includes the coordinates of the tower to be inspected and the remaining power of the tower, and plan an inspection route set according to the inspection information;

[0045] S2: When the drone performs an inspection task, the average energy consumption rate and the remaining survival time of the drone are calculated, and the running time of the inspection route set is divided into multiple charging cycles;

[0046] S3: Calculate the charging scheduling plan of the tower node according to the remaining power of the tower, and model and solve the charging sequence planning problem in the charging scheduling plan according to the DQN charging sequence optimization algorithm to find the charging path sequence of the drone;

[0047] S4: Visit the tower nodes to be charged in sequence according to the charging sequence path sequence and perform inspection tasks.

[0048] In this embodiment, if Figure 1 In the scenario shown, a charging device for a drone is installed on the tower node. Assuming that the drone system receives inspection information requests from tower nodes A, B, E, F, and G, if the path optimization method is used, the shortest path for the drone is: A→B→F→G→E (the same is true in reverse). The scenario setting here is that each tower node can provide sufficient power for the drone. However, in actual situations, since the charging device in the tower is an energy supply device that stores limited power, such as a battery, after the drone performs different inspection tasks, the power consumption on each tower node is different. In order to increase the utilization rate of the battery and ensure the normal progress of the drone inspection task, the drone inspection task and the charging scheduling method are intelligently planned according to the different power storage conditions of the tower nodes and the drone inspection tasks.

[0049] Specifically: In one embodiment, the operation time of the drone inspection route set is divided into a plurality of charging cycles. During the charging cycle, the step of sending a charging request to the tower node according to the average energy consumption rate of the drone and its own remaining survival energy includes:

[0050] When the drone performs inspection tasks, it establishes a real-time communication connection with the tower node. The drone can obtain the remaining stored power of the tower node and send a charging request to the charging device of the tower node. The drone sends the remaining survival energy of the drone to the tower node at a fixed time period. The average energy consumption rate is used to calculate the remaining energy of the drone. Calculate the remaining survival time , send a charging request when the drone needs to be charged { , , , },in It is the identification of the tower node; is the location coordinate of the tower; and They are the energy demand and remaining survival time when the drone sends a charging request to the tower; after the drone sends the charging request, the signal record is stored in the charging service pool;

[0051] The average energy consumption rate is calculated by using the weighted average method, using the charging cycle identifier as the weight for calculating the energy consumption rate. The calculation method is:

[0052] ;

[0053] In the formula, Represents the base station's estimate of the energy consumption rate of tower node i after receiving m pieces of remaining energy value information; The time for recording the mth piece of remaining energy information for tower node i; the average energy consumption rate is used here because in the process of performing inspection tasks, the inspection requirements for different tower nodes are different, and the time and power consumption around different towers are different. According to the average energy consumption rate, the drone power consumed in a certain period of time in the inspection task can be balanced.

[0054] The calculation method of the remaining survival time is:

[0055] ;

[0056] is the remaining energy of the drone at tower node i; is the average energy consumption rate.

[0057] by Figure 1 For example, let the drone patrol along the shortest path, A→B→F→G→E. If the drone's power can support the drone to complete the patrol of points A and B and reach point F at the same time, it needs to be fully charged at F. After the drone is fully charged at point F and continues to perform the patrol tasks of G and E, the remaining power cannot return to the starting point of the base station. In this case, it needs to be charged at point E, and the charged power can meet the drone's return from point E to the starting point of the base station. In this example, its charging cycle 1 is in ABF, and charging cycle 2 is in GE. If it is difficult to meet the charging requirements during the charging cycle, the path cannot achieve the drone's cruising mission, such as Figure 1 As shown in the figure, if the remaining power at points B and F is insufficient, there will not be enough power at point F to perform the inspection task to point G. Therefore, the drone inspection path needs to be replanned according to the drone charging sequence.

[0058] The DQN charging sequence optimization algorithm includes:

[0059] Since the charging sequence planning problem has a discrete action space, the DQN algorithm is used to convert the charging sequence optimization of the drone into a reward maximization problem in the DQN algorithm; the action space of the drone is defined as ; Indicates that the drone replenishes energy at tower node i; during the charging scheduling process, each time an action is executed, the environment will feed back a reward Q to the drone and update it to a new environmental state; when the drone reaches the final state, the total reward is calculated, and the Q function is trained to make the drone obtain as much reward as possible, thereby constructing the optimal charging sequence for the drone;

[0060] Initialize charging path sequence CS;

[0061] Selecting actions with a greedy strategy ;

[0062] Executing actions in the simulator And get corresponding rewards ;

[0063] Add to To charging sequence ;

[0064] Update the dataset .

[0065] Specifically, define the state space of the drone , which can include information such as the drone’s current energy state, its location at the tower node, and its action space , that is, the action of replenishing energy at each tower node, uses the DQN algorithm to train a Q function to estimate the expected reward value of performing each action in each state. The training process of this Q function will be continuously updated and optimized according to the rewards obtained by the drone when performing actions in the actual environment; execute the greedy strategy to select actions in the simulator. At each time step, according to the current state, use the trained Q function to select an action. According to the selected action execution, simulate the execution of the action in the simulation environment, and get the corresponding reward according to the actual effect; and according to the negative rewards generated during the Q function training process, remove the inspection path according to the negative rewards;

[0066] The action space includes:

[0067] Consider the tower node energy factor , Tower node location factors and tower node failure factors :

[0068] ;

[0069] is the average energy consumption rate of tower nodes; is the remaining energy of the drone at tower node i; is the distance between the current node to be charged and the next node to be charged, x is the number of charging tower nodes, α is the penalty coefficient, when the drone is in state Execute an action After the charging sequence When a node fails, a negative reward is fed back to the DQN algorithm through the reward function.

[0070] The reward function includes:

[0071] ;

[0072] Where K is the number of charging rounds.

[0073] like Figure 3The battery storage energy of each tower node in the E→G→F→B→A path shown in the figure. During the inspection process of the drone, the drone cannot be charged at points F and B, which will cause it to be unable to perform the inspection task. According to the DQN algorithm, a negative reward will appear, which means that the inspection action of the drone will fail at points F and B, and it is judged that the path cannot meet the requirements of the inspection task.

[0074] like Figure 2 and Figure 3 As shown in the figure, the inspection route improved by the above method is: A→B→E→F→G. In this inspection route, the charging nodes are on A, E, and G. After each charging, the power demand of subsequent inspections can be met, avoiding the situation where the drone is short of power during the inspection mission.

[0075] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0076] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A UAV tower inspection and charging scheduling method based on deep reinforcement learning, characterized in that: The following steps are involved: S1: Obtain inspection information, which includes the coordinates of the tower to be inspected and the remaining power of the tower, and plan an inspection route set according to the inspection information; S2: When the drone performs an inspection task, the average energy consumption rate and the remaining survival time of the drone are calculated, and the running time of the inspection route set is divided into multiple charging cycles; S3: Calculate the charging scheduling plan of the tower node according to the remaining power of the tower, and model and solve the charging sequence planning problem in the charging scheduling plan according to the DQN charging sequence optimization algorithm to find the charging path sequence of the drone; S4: visit the tower nodes to be charged in sequence according to the charging sequence path sequence and perform inspection tasks; When the drone performs an inspection task, the average energy consumption rate and the remaining survival time of the drone are calculated, and the running time of the inspection route set is divided into multiple charging cycle steps, including: The average energy consumption rate is calculated by using the weighted average method, using the charging cycle identifier as the weight for calculating the energy consumption rate. The calculation method is: ; In the formula, Represents the base station's estimate of the energy consumption rate of tower node i after receiving m pieces of remaining energy value information; The time for recording the mth piece of remaining energy information for tower node i; The calculation method of the remaining survival time is: ； is the remaining energy of the drone at tower node i; is the average energy consumption rate; The DQN charging sequence optimization algorithm includes: Since the charging sequence planning problem has a discrete action space, the DQN algorithm is used to convert the charging sequence optimization of the drone into a reward maximization problem in the DQN algorithm; the action space of the drone is defined as ; Indicates that the drone replenishes energy at tower node i; during the charging scheduling process, each time an action is executed, the environment will feed back a reward Q to the drone and update it to a new environmental state; when the drone reaches the final state, the total reward is calculated, and the Q function is trained to make the drone obtain as much reward as possible, thereby constructing the optimal charging sequence for the drone; Initialize charging path sequence CS; Selecting actions with a greedy strategy ; Executing actions in the simulator And get corresponding rewards ; Add to To charging sequence ; Update the dataset ; The action space includes: considering the tower node energy factor , Tower node location factors and tower node failure factors : ； is the average energy consumption rate of tower nodes; is the remaining energy of the drone at tower node i; is the distance between the current node to be charged and the next node to be charged, x is the number of charging tower nodes, α is the penalty coefficient, when the drone is in state Execute an action After the charging sequence When a node fails, a negative reward is fed back to the DQN algorithm through the reward function; The reward function includes: ； Where K is the number of charging rounds.

Images