Power inspection task allocation method based on reinforcement learning

Abstract

The invention relates to an electric power inspection task allocation method based on reinforcement learning, and belongs to the field of electric power inspection task allocation. According to the method, a deep Q network (DQN) is used, the state is mapped to an action space through a deep learning technology, high-dimensional input can be processed, and the method is excellent in performance in complex tasks. In electric power inspection, reinforcement learning can optimize task allocation of the unmanned aerial vehicle by acquiring inspection data (such as the electric quantity, the position and the weather condition of the unmanned aerial vehicle) in real time, so that efficient inspection is realized. In addition, the self-adaptive characteristic of reinforcement learning enables the method to rapidly adjust the strategy in case of emergency, and guarantees the smooth operation of the inspection work.

Description

Technical Field

[0001] The present invention belongs to the field of power inspection task allocation, and in particular relates to a power inspection task allocation method based on reinforcement learning. Background Art

[0002] In modern power systems, the stability and safety of transmission lines are the basis for ensuring power supply. With the development of drone technology, multi-drone systems have gradually become an important tool for power inspection. Its flexibility and efficiency enable drones to cover a wide range of transmission lines in a short period of time, improving the real-time and accuracy of inspections. Combining drones with effective inspection task allocation algorithms can effectively improve the efficiency of inspections.

[0003] In a multi-UAV system, task allocation is a key factor in ensuring efficient operation. Effective task allocation can optimize resource utilization, improve work efficiency, and reduce execution costs. In addition, a good task allocation strategy can also reduce conflicts between UAVs and improve inspection quality, thereby reducing safety hazards and economic losses caused by failure to detect faults in a timely manner. At present, research on task allocation mainly focuses on several heuristic algorithms such as genetic algorithms, ant colony algorithms, and particle swarm algorithms. Although heuristic algorithms have many advantages, they generally have limitations such as insufficient flexibility and high computational and communication overhead, which restricts their use scenarios. Artificial intelligence algorithms represented by reinforcement learning have performed well in the field of task allocation and scheduling. Summary of the invention

[0004] The purpose of the present invention is to solve the problem of task allocation in power inspection scenarios, so as to improve the real-time performance and accuracy of drones in the inspection process, and provide a power inspection task allocation method based on reinforcement learning. By applying the reinforcement learning technology in advanced machine learning, the present invention is committed to achieving reasonable allocation of drone tasks in power inspection scenarios; the present invention not only optimizes resource utilization, improves work efficiency, and reduces execution costs, but also reduces conflicts between drones and improves inspection quality, thereby reducing safety hazards and economic losses caused by failure to discover faults in a timely manner.

[0005] To achieve the above object, the technical solution of the present invention is: a method for allocating power inspection tasks based on reinforcement learning, comprising:

[0006] Step 1: Initialize the power inspection scene, including: the number of inspection drones is 3, the x and y coordinates of the power tower position are random numbers between 0-10, the x and y coordinates of the initial position of the drone are random numbers between 0-10, the drone power is initialized to 100%, etc. According to the initialized parameters, the power inspection task is allocated in an environment of size 1000*1000;

[0007] Step 2: Calculate the distance between the drone and the power tower. Use the Euclidean distance to measure the straight-line distance between the drone and the power tower. Through distance calculation, determine the distance between the drone and each power tower in real time and optimize the flight path of the drone.

[0008] Step 3. Initialize experience playback: Create a fixed-size memory bank to store task assignment information including drone state information, i.e., the experience of the agent interacting with the environment (state, action, reward, next state). Initialize the parameters of the Q network and copy these parameters to the target network. The Q network is used to estimate the Q value of the current state, and the target network is used to calculate the target Q value;

[0009] Step 4: Generate training data. The agent performs actions in the environment, collects experience and stores it in the experience replay memory bank. Sample training data from the memory bank, randomly extract a batch of experience (state, action, reward, next state) from the experience replay memory bank for training. Input the current state of the extracted experience into the Q network, and calculate the Q value of all possible actions.

[0010] Step 5: Calculate the loss and update the Q network. Use the current Q value and the target Q value to calculate the loss function. Back-propagate the loss through the gradient descent method, update the weights of the Q network, update the target network, and copy the parameters of the Q network to the target network at regular intervals.

[0011] Step 6: Repeat the training process, and keep repeating steps 4 to 5 until the Q network converges and the agent's strategy is optimized;

[0012] Step 7: Based on the trained single-UAV allocation model and multi-UAV allocation model, a deep reinforcement learning model for power inspection task allocation is combined, and the final allocation plan is output using the policy network.

[0013] In one embodiment of the present invention, the experience replay and Q network are initialized, and a fixed-size experience replay library is initialized to store the task allocation experience of the UAV in the power environment. Each experience contains four elements: the current state, the action taken, the reward obtained and the next state, that is, (state, action, reward, next_state). A neural network is constructed as the current Q network to estimate the action value in the current state, that is, the Q value.

[0014] In one embodiment of the present invention, for a single drone allocation model, the following environmental components are defined:

[0015] State space: record the drone number, the Euclidean distance between the drone and the power line, and the remaining power of the drone;

[0016] Action space: Select a numbered drone from the three drones as an independent action, that is, the action space size of the single drone allocation model is 3;

[0017] Reward function: The cost is obtained by calculating the Euclidean distance between the drone and the power tower and adding the line length. In addition, it is checked whether the drone has enough power for inspection. If the power is insufficient, the cost is -1000 points. Then the minimum cost is calculated, and the absolute value of the difference between the cost and the minimum cost is used as the reward value.

[0018] In one embodiment of the present invention, for a multi-UAV allocation model where only two UAVs perform tasks, the following environmental components are defined:

[0019] State space: records the number of the drone performing the mission, the Euclidean distance between the drone and the power transmission line, and the remaining power of the drone;

[0020] Action space: Select two drones out of three drones as an independent action, that is, the action space size of the multi-drone allocation model is 6;

[0021] Reward function: First calculate the Euclidean distance between the first drone and the power tower and add the line length, then add the Euclidean distance between the second drone and the power tower to get the cost value, then calculate the minimum cost value, and use the absolute value of the difference between the cost value and the minimum cost value as the reward value.

[0022] In one embodiment of the present invention, the power inspection task allocation model defines the following environmental components:

[0023] State space: records the number of the drone performing the mission, the Euclidean distance between the drone and the power transmission line, and the remaining power of the drone;

[0024] Action space: Selecting a single UAV or multiple UAVs to perform a task as an independent action, that is, the action space size of the power inspection task allocation model is 2;

[0025] Reward function: Select single UAV or multiple UAVs to perform the task, and use the reward function of the single UAV allocation model and the multi-UAV allocation model to calculate the reward.

[0026] In one embodiment of the present invention, generating training data includes:

[0027] Action selection strategy: Use the ε-greedy strategy to balance exploration and exploitation. Select a random action for exploration with a probability of ε, and select the best action estimated by the current Q network, that is, the action with the largest Q value, with a probability of 1-ε. During the training process, ε will gradually decrease, so that the agent gradually shifts from exploration to exploitation.

[0028] Interact with the environment: The agent selects actions based on the current strategy and interacts with the environment to obtain the next state and immediate reward;

[0029] Store experience: Add task assignment experience to the experience playback library, replacing older experience;

[0030] Sample training data from the memory bank, including:

[0031] Random sampling experience: randomly extract a small batch of experience from the experience replay memory bank for training; calculate the current Q value: input the current state into the Q network, obtain the Q value distribution of each possible action, and then extract the Q value of the action actually performed by the agent in the current state;

[0032] Calculate the target Q value: input the next state into the target network, obtain the Q values ​​of all possible actions in the next state, select the maximum Q value, i.e. the optimal Q value, and update it to the target Q value of the current state through the Bellman equation: y = r + γmaxQ'(s', a'), where r is the immediate reward, γ is the discount factor, Q' is the Q value of the target network, s' is the state at the next moment, and a' is the action at the next moment.

[0033] In one embodiment of the present invention, calculating the loss and updating the Q network includes: using the mean square error MSE as a loss function to calculate the error between the current Q value and the target Q value: Where N is the batch size, y i is the target Q value of the current state for batch i, s i is the state at batch i, a i is the action at batch i; through back propagation and gradient descent methods, the loss function is minimized, the weights of the Q network are updated, and the learning rate is adjusted to control the update step;

[0034] Update the target network, including: synchronize the Q network and the target network, copy the parameters of the current Q network to the target network at a fixed number of steps called the target network update frequency; soft update of parameters, using a soft update strategy, that is, by setting a smaller step coefficient τ, the target network parameters are gradually approached to the Q network parameters: θ'=τθ+(1-τ)θ', where θ is the Q network parameter, θ' is the target network parameter, and τ is the step coefficient.

[0035] In one embodiment of the present invention, the repeated training process includes: continuously repeating steps 4 to 5, the agent explores and learns in the environment, and the Q network gradually approaches the optimal value. The training continues until one of the following conditions is met: the specified maximum number of training rounds is reached; the performance of the agent in the environment reaches the expected level, the reward value or the target value converges; the change of the Q value decreases, and the loss function stabilizes at a lower value; after the training is completed, the parameters of the Q network are saved.

[0036] In one embodiment of the present invention, the cost calculation of the power inspection task allocation model is as follows: i,j Defined as:

[0037] C i,j =d i,j +L j

[0038] Among them, d i,j is the distance between the drone and the power tower, L j It is the length of the inspection route of the power tower, that is, the fixed cost required to complete the inspection task.

[0039] In one embodiment of the present invention, the constraints of the power inspection task allocation model include:

[0040] 1) Power Constraints

[0041]

[0042] This constraint ensures that the power of each drone is B i Sufficient support to complete the designated inspection tasks;

[0043] 2) Inspection completion constraints

[0044]

[0045] This constraint ensures that the inspection task of each power tower can be completed by at least one drone;

[0046] 3) Action constraints

[0047]

[0048] This constraint restricts each drone to choose an action a i , ensuring that drones can only choose single or double inspection options;

[0049] 4) Time constraints

[0050] total_time≤T

[0051] This constraint limits the time total_time of all inspection tasks to be completed within the given total time T.

[0052] Compared with the prior art, the present invention has the following beneficial effects: The present invention proposes a systematic modeling method, which clarifies the key elements in task allocation, including state space, action space and reward function design. This modeling framework provides a clear theoretical basis for the subsequent application of reinforcement learning algorithms, so that complex task allocation problems can be effectively solved. In addition, the present invention solves the problem of unreasonable task allocation in the face of emergencies in power inspections. Through the adaptive characteristics of reinforcement learning, it can quickly adjust strategies in the face of emergencies to ensure the smooth progress of inspection work. This technology can not only optimize the task allocation of drones by acquiring inspection data (such as drone power, location, etc.) in real time, but also provides a solution specifically for task allocation problems. It is suitable for unmanned driving, environmental monitoring, agricultural inspections and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0054] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0055] Embodiment 1:

[0056] The present invention is achieved through the following technical solutions: Figure 1 As shown, a reinforcement learning method for UAV inspection tasks for power transmission lines includes the following steps:

[0057] Step 1: First, we set the relevant parameters, including: the number of inspection drones is 3, the x and y coordinates of the power tower position are random numbers between 0 and 10, the x and y coordinates of the initial position of the drone are random numbers between 0 and 10, and the drone power is initialized to 100%. Based on the initialized parameters, the power inspection task is allocated in an environment of size 1000*1000.

[0058] Step 2: Calculate the distance between the drone and the power tower. In power inspection tasks, especially when using drones to inspect transmission lines, calculating the distance between the drone and the power tower is an important step. This distance calculation can help drones plan the optimal path, optimize task scheduling, and ensure the efficiency and safety of the inspection process.

[0059] To achieve this goal, the present invention uses the Euclidean distance to measure the straight-line distance between the drone and the power tower. The Euclidean distance is a common geometric distance metric that can accurately calculate the "straight-line" distance between two points. In two-dimensional space, given the coordinates of two points (x1, y1) and (x2, y2), the Euclidean distance between them can be calculated by the following formula:

[0060]

[0061] In the power inspection task, in order to more accurately calculate the distance between each inspection point and the power tower, we use this distance calculation, the system can determine the distance between the drone and each power tower in real time, and then optimize the flight path of the drone, reduce energy consumption, and avoid unnecessary flight distance. At the same time, based on this distance information, the inspection sequence of the drone can be better arranged, the efficiency and accuracy of the inspection can be improved, and the safe operation of the power equipment can be ensured.

[0062] Step 3, state acquisition and action selection. Use the get_observation method to get the current state. According to different action codes, calculate the cost of single or multiple drones inspecting the power tower and use it as part of the state. In the set_action method of the environment, select the action using a single drone or multiple drones DQN according to the action code. The single drone DQN selects a single drone for inspection according to the current state, and the multiple drone DQN selects multiple drones for inspection.

[0063] The training process of the power grid inspection task allocation DQN is as follows: First, initialize the experience replay (ExperienceReplay) to create a fixed-size memory bank to store the experience (state, action, reward, next state) of the agent's interaction with the environment. Among them, the buffer size size = 1000000, the batch size batch_size = 32, the number of reward steps n_step = 3, and the discount factor gamma = 0.99. Initialize the parameters of the Q network and copy these parameters to the target network. Among them, the batch size batch_size = 1024, the learning rate lr = 0.0001, the discount factor gamma = 0.99, the exploration rate episilo = 0.9, and the Q network update frequency iteration = 100.

[0064] Then generate training data for power inspection, the agent performs actions in the inspection environment, collects experience and stores it in the experience playback memory. Sample training data from the memory, randomly extract a batch of experience (state, action, reward, next state) from the memory for training to break the correlation between adjacent training samples and enhance the independence of training samples. Input the current state into the Q network and calculate the Q value of all possible actions. Input the next state into the target network, calculate the Q value of all possible actions, and select the maximum Q value as the target Q value.

[0065] The next step is to calculate the loss and update the Q network. The loss function is calculated using the current Q value and the target Q value. In this step, a perceptron neural network consisting of three linear layers is constructed. Layer1 has an output size of 512, Layer2 has an input size of 512 and an output size of 512, and Layer3 has an input size of 512 and an output size of 512. The loss is back-propagated through the gradient descent method to update the weights of the Q network. Every 100 steps, the parameters of the Q network are copied to the target network to maintain the stability of the target value.

[0066] Finally, repeat the training process, and keep repeating steps 2 to 5 until the Q network converges and the agent's strategy is optimized.

[0067] Step 4, action execution and reward calculation. In the step method of the environment, execute the selected action and calculate the inspection cost. If the drone cannot reach the power tower, the cost is infinite. Then calculate the cost of the optimal solution and calculate the reward based on the cost difference between the current solution and the optimal solution. If the current solution is close to the optimal solution, the reward is higher, otherwise the reward is lower.

[0068] Among them, in the specific method, the inspection cost of a single drone is calculated, including the distance between the drone and the power tower and the length of the inspection line. If the drone’s battery is insufficient to complete the inspection, a large negative reward is returned. In the multi-drone task allocation method, the inspection cost of multiple drones is calculated, including the distance between multiple drones and the power tower and the length of the inspection line.

[0069] Step 5: Training and evaluation. To train the single-drone allocation model and the multi-drone allocation model, firstly, the single-drone allocation scheme interacts with the environment, obtains experience and puts it into the experience pool, then extracts batch experience from the experience pool, trains the single-machine policy network, and finally regularly updates the target policy network and repeats this process.

[0070] Step 6: Based on the trained single-drone allocation model and multi-drone allocation model, a deep reinforcement learning model for power inspection drone task allocation is combined, and the final allocation plan is output using the policy network.

[0071] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.

Claims

1. A method for allocating power inspection tasks based on reinforcement learning, characterized in that: include: Step 1: Initialize the power inspection scene, including: the number of inspection drones, the x and y coordinates of the power tower location, the initial position of the drone, and the drone power; Step 2: Calculate the distance between the drone and the power tower. Use the Euclidean distance to measure the straight-line distance between the drone and the power tower. Through distance calculation, determine the distance between the drone and each power tower in real time and optimize the flight path of the drone. Step 3: Initialize experience replay: Create a fixed-size memory bank to store task assignment information including drone status information; initialize the parameters of the Q network and copy these parameters to the target network. The Q network is used to estimate the Q value of the current state, and the target network is used to calculate the target Q value. Step 4: Generate training data. The agent performs actions in the environment, collects experience and stores it in the experience replay memory bank. Sample training data from the memory bank, randomly extract a batch of experience from the experience replay memory bank for training. Input the current state of the extracted experience into the Q network, and calculate the Q value of all possible actions. Step 5: Calculate the loss and update the Q network. Use the current Q value and the target Q value to calculate the loss function. Back-propagate the loss through the gradient descent method, update the weights of the Q network, update the target network, and copy the parameters of the Q network to the target network at regular intervals. Step 6: Repeat the training process, and keep repeating steps 4 to 5 until the Q network converges and the agent's strategy is optimized; Step 7: Based on the trained single-UAV allocation model and multi-UAV allocation model, a deep reinforcement learning model for power inspection task allocation is combined, and the final allocation plan is output using the policy network.

2. According to the method for allocating power inspection tasks based on reinforcement learning in claim 1, it is characterized in that: Initialize the experience replay and Q network. Initialize a fixed-size experience replay library to store the task allocation experience of drones in power environments. Each experience contains four elements: current state, action taken, reward obtained, and next state, i.e. (state, action, reward, next_state). Build a neural network as the current Q network to estimate the action value in the current state, i.e., Q value.

3. The method for allocating power inspection tasks based on reinforcement learning according to claim 1 is characterized in that: For the single-drone allocation model, the following environmental components are defined: State space: record the drone number, the Euclidean distance between the drone and the power line, and the remaining power of the drone; Action space: Select a numbered drone from the three drones as an independent action, that is, the action space size of the single drone allocation model is 3; Reward function: The cost is obtained by calculating the Euclidean distance between the drone and the power tower and adding the line length. In addition, it is checked whether the drone has enough power for inspection. If the power is insufficient, the cost is -1000 points. Then the minimum cost is calculated, and the absolute value of the difference between the cost and the minimum cost is used as the reward value.

4. The method for allocating power inspection tasks based on reinforcement learning according to claim 1, characterized in that: For the multi-UAV allocation model with only two UAVs performing tasks, the following environment components are defined: State space: records the number of the drone performing the mission, the Euclidean distance between the drone and the power transmission line, and the remaining power of the drone; Action space: Select two drones out of three drones as an independent action, that is, the action space size of the multi-drone allocation model is 6; Reward function: First calculate the Euclidean distance between the first drone and the power tower and add the line length, then add the Euclidean distance between the second drone and the power tower to get the cost value, then calculate the minimum cost value, and use the absolute value of the difference between the cost value and the minimum cost value as the reward value.

5. The method for allocating power inspection tasks based on reinforcement learning according to claim 1, characterized in that: The power inspection task allocation model defines the following environmental components: State space: records the number of the drone performing the mission, the Euclidean distance between the drone and the power transmission line, and the remaining power of the drone; Action space: Selecting a single UAV or multiple UAVs to perform a task as an independent action, that is, the action space size of the power inspection task allocation model is 2; Reward function: Select single UAV or multiple UAVs to perform the task, and use the reward function of the single UAV allocation model and the multi-UAV allocation model to calculate the reward.

6. The method for allocating power inspection tasks based on reinforcement learning according to claim 1, characterized in that: Generate training data, including: Action selection strategy: Use the ε-greedy strategy to balance exploration and exploitation. Select a random action for exploration with a probability of ε, and select the best action estimated by the current Q network, that is, the action with the largest Q value, with a probability of 1-ε. During the training process, ε will gradually decrease, so that the agent gradually shifts from exploration to exploitation. Interact with the environment: The agent selects actions based on the current strategy and interacts with the environment to obtain the next state and immediate reward; Store experience: Add task assignment experience to the experience playback library, replacing older experience; Sample training data from the memory bank, including: Random sampling experience: randomly extract a small batch of experience from the experience replay memory bank for training; calculate the current Q value: input the current state into the Q network, obtain the Q value distribution of each possible action, and then extract the Q value of the action actually performed by the agent in the current state; Calculate the target Q value: input the next state into the target network, obtain the Q values ​​of all possible actions in the next state, select the maximum Q value, i.e. the optimal Q value, and update it to the target Q value of the current state through the Bellman equation: y = r + γmaxQ'(s', a'), where r is the immediate reward, γ is the discount factor, Q' is the Q value of the target network, s' is the state at the next moment, and a' is the action at the next moment.

7. The method for allocating power inspection tasks based on reinforcement learning according to claim 1, characterized in that: Calculate the loss and update the Q network, including: using the mean square error MSE as the loss function, calculate the error between the current Q value and the target Q value: Where N is the batch size, y i is the target Q value of the current state for batch i, s i is the state at batch i, a i is the action at batch i; through back propagation and gradient descent methods, the loss function is minimized, the weights of the Q network are updated, and the learning rate is adjusted to control the update step; The target network is updated, including: synchronizing the Q network and the target network, copying the parameters of the current Q network to the target network at a fixed number of steps called the target network update frequency; soft updating of parameters, using a soft update strategy, that is, by setting a smaller step coefficient τ, the target network parameters are gradually approached to the Q network parameters: θ'=τθ+(1-τ)θ', where θ is the Q network parameter, θ' is the target network parameter, and τ is the step coefficient.

8. The method for allocating power inspection tasks based on reinforcement learning according to claim 1, characterized in that: Repeat the training process, including: repeating steps 4 to 5, the agent explores and learns in the environment, and the Q network gradually approaches the optimal value. Training continues until one of the following conditions is met: the specified maximum number of training rounds is reached; the agent's performance in the environment reaches the expected level, the reward value or the target value converges; the change in the Q value decreases, and the loss function stabilizes at a lower value; after training is completed, the parameters of the Q network are saved.

9. The method for allocating power inspection tasks based on reinforcement learning according to claim 1, characterized in that: Cost calculation of power inspection task allocation model, single drone inspection cost C i,j Defined as: C i,j =d i,j +L j Among them, d i,j is the distance between the drone and the power tower, L j It is the length of the inspection route of the power tower, that is, the fixed cost required to complete the inspection task.

10. The method for allocating power inspection tasks based on reinforcement learning according to claim 9, characterized in that: The constraints of the power inspection task allocation model include: 1) Power Constraints This constraint ensures that the power of each drone is B i Sufficient support to complete the designated inspection tasks; 2) Inspection completion constraints This constraint ensures that the inspection task of each power tower can be completed by at least one drone; 3) Action constraints This constraint restricts each drone to choose an action a i , ensuring that drones can only choose single or double inspection options; 4) Time constraints total_time≤T This constraint limits the time total_time of all inspection tasks to be completed within the given total time T.

Images