Overhead line planning method based on deep reinforcement learning

Abstract

The present invention provides an overhead line planning method based on deep reinforcement learning, which belongs to the field of smart grid. The method comprises: step 1, modeling the path planning process of an intelligent agent in a continuous state space as a Markov decision process, and designing a dual-objective path planning model; step 2, designing a reward function for the dual-objective path planning model; step 3, using an experience quadruple to detect whether the route of the intelligent agent is in violation of the rules; step 4, optimizing the dual-objective path planning model based on deep reinforcement learning. The method can quickly and accurately plan multiple line selection results, greatly improving the work efficiency of planners.

Description

Technical Field

[0001] The present invention belongs to the field of smart grids, and specifically relates to an overhead line planning method based on deep reinforcement learning. Background Art

[0002] Intelligent planning and design plays a vital role in fields such as power infrastructure construction in cities and suburbs. With the acceleration of urbanization, more and more places are facing the challenge of overhead line planning in complex geographical environments. These challenges are caused by the uneven distribution of various spatial objects and the control of safety, cost and impact on public life. This ultimately leads to the efficiency and effectiveness of existing planning and design algorithms being difficult to match current needs. Overhead line planning is an important research scenario for intelligent line selection systems. Its purpose is to recommend multiple lines that connect the starting point and the midpoint under the transition of the intermediate point, while meeting the line specifications and minimizing the distance cost as much as possible.

[0003] At present, the automatic planning methods for overhead lines are mainly divided into two categories: the first category is the method of rasterizing maps. This method divides the map of the continuous state space into discrete units, and quantifies and rates the environmental factors that affect the line planning, and finally uses a greedy algorithm or a heuristic algorithm to achieve the optimal path solution. Although this type of method simplifies the problem, the state transition is limited to a few limited directions (the most common is the basic eight directions), which makes the planning results very sensitive to the resolution of the rasterization. The size of the resolution corresponds to a fixed step unit and cannot cope with dynamic step scenarios. In addition, each independent grid on the map has only two situations: "walkable" and "unwalkable", and does not consider the situation of crossing obstacles. The second type of method is the algorithm of continuous state space sampling (such as the RRT series algorithm). This type of method does not rasterize the map, but regards the plannable range as the search space, and generates candidate paths by continuously randomly sampling the search space until the end point is reached. Although RRT is planned in continuous space, the sampling process is completely random, which limits the final line effect. In addition, deep neural networks are also used to optimize the selection of sampling points. This type of method still ignores the condition that the global map is known information. This results in the planning process being split into multiple greedy processes, limiting the overall optimization effect of the final route.

[0004] With the development of artificial intelligence in recent years, many convolutional neural network frameworks have provided effective learning solutions for global environmental information, and a large number of studies have applied reinforcement learning methods to planning problems. Especially in complex, unstructured or dynamically changing environments, by utilizing trial-and-error learning and delayed reward mechanisms in reinforcement learning frameworks, researchers have successfully solved many problems that are difficult to handle with traditional methods, such as finding the optimal path or strategy in complex environments. In addition, with the development of deep reinforcement learning, algorithms such as Soft Actor-Critic (SAC) have also performed well in planning tasks in continuous action spaces, and can effectively balance the relationship between exploration and utilization to find a better solution. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides an overhead line planning method based on deep reinforcement learning, which can complete efficient and relatively reasonable planning tasks and provide assistance to relevant engineering personnel.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] An overhead line planning method based on deep reinforcement learning comprises the following steps:

[0008] Step 1: Model the path planning process of the agent in the continuous state space as a Markov decision process and design a dual-objective path planning model;

[0009] Step 2: Design a reward function for the dual-objective path planning model;

[0010] Step 3: Use the experience quadruple to detect whether the agent's route is in violation of the rules;

[0011] Step 4: Optimize the dual-objective path planning model based on deep reinforcement learning.

[0012] The beneficial effects of the present invention are:

[0013] The planning process of overhead lines involves 360 degrees without blind spots, is highly exploratory, and can efficiently provide multiple lines; the length of each step in the planning process is dynamic and leapfrog, which is more in line with the actual process of overhead line planning; each step is planned after efficiently understanding the global environment through neural networks, reducing meaningless sampling and improving the efficiency of each step; greatly saving labor and time costs, and providing auxiliary decision-making assistance for engineers. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a principle block diagram of an overhead line planning method based on deep reinforcement learning in the present invention. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0016] like Figure 1 As shown, the present invention provides an overhead line planning method based on deep reinforcement learning, which first proposes the input and output of the overhead line planning problem.

[0017] Input: A preconfigured map with the coordinates of the starting and destination points on the map.

[0018] Output: Plan out the arrive The path τ follows the following objectives:

[0019] Main goal: minimize the planned path distance;

[0020] Sub-goal: minimize the number of intermediate points;

[0021] Formally, the overall goal is to make a linear combination of the above two objectives Maximize as follows:

[0022] ,

[0023] in, is a hyperparameter that weights the sub-goals, represents the distance of the planned path, Indicates the number of intermediate points in the planned path.

[0024] The overhead line planning problem is a natural sequential decision problem that matches the reinforcement learning framework. The path planning process in the continuous state space is modeled as a Markov decision process (MDP), and all path planning solutions are transformed into finding the optimal decision strategy in the MDP. process. In the overhead path planning problem, the agent can be regarded as the current planning point. The MDP process can usually be represented by a five-tuple (S, A, P, R, γ). Among them, S represents the state set of the agent in the environment, the state s∈S represents the state of the agent in the environment, that is, the position in the current environment after moving from the starting point; A is the space containing all the actions that the agent can take, action a∈A; p∈P is the agent in a given state s, after taking action a to go to the next state probability; ∈R indicates that the agent performs the current action according to the strategy π The immediate reward after , the detailed design of the reward function will be described below; is a discount factor used to measure the importance of rewards generated in future steps. The experience pool stores four tuples Used to record the learning process, t is the current step.

[0025] In the overhead path planning problem based on reinforcement learning, the goal is to optimize the cumulative reward G, using Represents the cumulative optimization reward from the current step t to the end step T. It accumulates the immediate rewards obtained by each sequential action in the path, as shown in the following formula:

[0026] ,

[0027] In the formula, T is the terminal step. In addition, the goal of reinforcement learning is to find an optimal strategy To maximize long-term rewards , for this purpose we use the Q network To measure the quality of the strategy of taking action a in state s, as shown in the following formula:

[0028] Q π s , a = E π G t s t = s , a t = a ] ,

[0029] in, E π [·] Represents calculation strategy The corresponding expected value.

[0030] like Figure 1 The figure shows a principle block diagram of an overhead line planning method based on deep reinforcement learning of the present invention. The solid arrow lines represent the data flow in the training and reasoning stages, and the dotted lines are only used to represent the training stage. In the training DRL part, the double deep Q network structure DDQN [Deep reinforcement learning with double Q-learning] is used to decouple the action selection and evaluation through two networks (Q-network and target network). There are three main components responsible for interacting with the environment (i.e., state representation, reward calculation, and arrival detection), which are carefully designed interactions between the agent and the environment, and are described in detail as follows:

[0031] Status indicates:

[0032] The state in the MDP is determined in this component to keep track of the spatial state of the agent in the environment.

[0033] Reward calculation:

[0034] This component calculates the immediate reward through the reward function. Its design consists of two parts, namely, the reward guides the agent towards the destination and the reward optimizes the step count sub-goal. The reward calculation includes distance reward calculation, quantity reward calculation, and arrival reward calculation.

[0035] Buffer:

[0036] For the current generated experience quad Stored in the experience pool for training deep neural networks.

[0037] The functions of each component are introduced in detail below.

[0038] (1) Status indication

[0039] State s represents the current state of the agent in the environment, including: point information and map information. Although the map information remains constant during the task, it tells the agent the distribution of obstacles in the overall environment. Point information includes starting point information, destination point information, and current point information. The position of the starting point tells the agent where it comes from, the position of the destination point tells the agent which direction to go, and the position of the current point tells the agent where it is. There are two commonly used ways to represent positions in the framework of reinforcement learning, namely one-hot encoding and longitude and latitude encoding. One-hot encoding loses geographical features, and longitude and latitude encoding is difficult to directly integrate with map information.

[0040] Based on the above analysis, it is proposed to represent point information on a two-dimensional plane, which not only retains the geographical features of the points, but also is easy to combine with map information for subsequent intelligent agent learning. However, compared with the information provided by the map image, the information provided by the starting point and the end point on the two-dimensional plane is very sparse (only one pixel has a value). In order to reduce the impact of the data sparsity problem of the corresponding channel of the point data on the subsequent network learning, inspired by the key point processing of the posture estimation task, it is proposed to generate a heat map in the form of Gaussian distribution to represent point features, which is convenient for subsequent network calculations. The calculation of constructing a Gaussian distribution heat map for the coordinate point (x, y) is as follows:

[0041] ,

[0042] in, is the standard deviation used to control the size of the thermal range, Represents the horizontal coordinates of all pixels on the map, Represents the ordinates of all pixels on the map, A heatmap representing a Gaussian distribution at coordinates (x, y).

[0043] In general, the state of the environment is represented by a multi-channel two-dimensional vector, which includes a map information channel and a point channel (starting point, current point, destination point). Its purpose is to transform the one-dimensional point information into two-dimensional information and combine it with the two-dimensional map information as the current state to enter the neural network learning.

[0044] (2) Reward calculation

[0045] In order to ensure efficient learning of the dual-objective path planning model, this paper designs a reward function as follows:

[0046] ,

[0047] in, represents the main goal reward related to the distance of the planned path, represents the sub-goal reward, is a preset hyperparameter used to weigh the main goal and sub-goals. The main goal is the reward for minimizing the path distance (that is, Figure 1 The distance reward calculation shown in Figure 2) is the reward for minimizing the number of intermediate points (i.e. Figure 1 In order to make the model converge faster, we introduce a and Much larger termination bonus (That is Figure 1 Arrival reward calculation shown in Figure 2), when the agent reaches the destination, The previous parameters is 1, otherwise it is 0.

[0048] In order to make the agent more inclined to get closer to the destination each time, the following function calculation is designed :

[0049] ,

[0050] in, represents the previous position of the agent, represents the current position of the agent, Represents the location of the destination. With this heuristic design, the agent’s new location is usually closer to the destination than the previous location.

[0051] (3) Detection

[0052] The experience quadruple collected from the previous step It will be stored in the experience pool of the buffer. The current state will be checked as follows: whether it has crossed or entered (No obstacles such as poles or towers can be placed on the road), do you want to enter? (can cross obstacles where poles cannot be placed), whether to go out of the map and whether to reach the end. If any of these occurs, the path planning ends and is recorded as an experience quadruple, and a new round of experience collection starts from the starting point. These experiences are collected for Q network training. The Q network structure will output a new action and use the probability Explore a random action. This prevents the agent from getting stuck in trying similar actions.

[0053] (4) Model training

[0054] In reinforcement learning, training consists of multiple segments. The model training code is shown in the algorithm below. A segment refers to the agent at a preset end step. From a random starting point to a random destination, if the agent moves within a given maximum number of steps If the agent cannot reach the destination, it will also terminate at this step and start the next segment. Lines 5-9 in the algorithm below indicate that at each step of each segment, the state s is used as input, and a random number is used to determine whether the current action is more inclined to exploration action or more inclined to the optimal action under the current experience value. For exploration, the agent will randomly select an action in the action space; for the current experience value, the agent will take the action generated by the neural network.

[0055]

[0056] At the current time t, the agent takes the current action After that, get immediate rewards from the environment , the state of the environment changes from the current state Transition to next state . Target network for agent training It has the same structure as the Q network, and the intermediate parameters are and The target network is used to estimate The Q value generated in , the calculation of the target network is as follows:

[0057] ,

[0058] The process of calculating the loss is to compare the Q network structure under batch processing The resulting Q value and target network Estimated Q value The process is shown in the following formula:

[0059] L= ∑ batchsize [ V target - V Q )] 2 batchsize ,

[0060] represents the batch size when the neural network is learning, Indicates the specific loss size used for gradient backpropagation.

[0061] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for overhead line planning based on deep reinforcement learning, characterized in that: The steps include: Step 1: Modeling the path planning process of the agent in the continuous state space as a Markov decision process, and designing a dual-objective path planning model, wherein in the Markov decision process, the state representation of the agent includes representing the position of the agent using a Gaussian distribution heat map; Step 2: Design a reward function for the dual-objective path planning model; Step 3: Use the experience quadruple to detect whether the agent's route is in violation of the rules; Step 4: Optimize the dual-objective path planning model based on deep reinforcement learning; The step 1 comprises: Set the input and output of the overhead line planning problem, where the input is a pre-configured map and the coordinates of the starting point and the destination point on the map are given, and the output is the intelligent agent planned in the map from the starting point To destination The path τ is designed to design a dual-objective path planning model and make a linear combination of the dual objectives Maximization, the dual objectives include a main objective and a sub-objective, the main objective is to minimize the planned path distance, and the sub-objective is to minimize the number of intermediate points in the planned path; wherein, , is a hyperparameter that weights the sub-goals, represents the distance of the planned path, Indicates the number of intermediate points in the planned path; The step 2 includes designing a reward function as follows: ， in, represents the main goal reward related to the distance of the planned path, represents the sub-goal reward, is the preset hyperparameter used to weigh the main goal and sub-goal, Represents the termination reward. When the agent reaches the destination, the intermediate parameter is 1, otherwise it is 0; The main goal reward related to the distance of the planned path is calculated based on the following formula , so that the new position of the agent is closer to the destination than the previous position: ， in, represents the previous position of the agent, represents the current position of the agent, Indicates the location of the destination point; The step 3 comprises: Set the experience quad Used to record the learning process, t is the current step, Respectively represent the current state of the agent, the current action of the agent, and the agent performs the current action of the agent according to the strategy π The immediate reward after, the next state of the agent, the current state of the agent The agent's route is checked as follows: whether it crosses or enters , whether to walk in , whether to go out of the pre-configured map and whether to reach the destination. If one of them occurs, the path planning ends. It means obstacles such as towers cannot be crossed or installed. It indicates that obstacles where towers cannot be placed can be crossed. Each step of the detection process is recorded as a set of experience quadruple, and a new round of detection and collection of detection results are started from the starting point. The recorded experience quadruple is collected for Q network structure The Q network structure The strategy used to measure taking action a in state s The quality of the exploration space is Randomly select an action, or use a new action generated by the neural network learning result, where: ， in, Representation strategy The corresponding expected value; ， In the formula, , [0,1] is a discount factor that measures the importance of rewards generated in future steps.

2. According to claim 1, a method for overhead line planning based on deep reinforcement learning is characterized in that: The step 4 comprises: Use the target network Estimating Q-network structure The Q value generated in , target network and Q network structure Has the same structure: ， In the formula, and Represents the Q network structure respectively and target network The intermediate parameters of The process of calculating the loss is to compare the Q network structure under batch processing The resulting Q value and target network Estimated Q value The process is shown as follows: ， represents the batch size when the neural network is learning, Indicates the loss size used for gradient backpropagation.

Images