Line path planning scheme generation method and system based on deep reinforcement learning

Abstract

The invention discloses a line path planning scheme generation method and system based on deep reinforcement learning, and the method comprises the steps: firstly constructing a grid map and an evaluation index system of a planning region, determining the index weight based on a Critic method to construct a grid comprehensive cost evaluation model, and then constructing a path planning model with the minimum total cost as a target, the method comprises the following steps of: firstly, extracting an optimal power transmission line planning path, describing the optimal power transmission line planning path as a Markov decision process, then introducing mixed noise and an epsilon-greedy strategy into a depth deterministic strategy gradient algorithm, improving a priority experience playback mechanism to obtain an improved depth deterministic strategy gradient algorithm, and finally training an intelligent agent based on the algorithm and outputting an optimal power transmission line planning path. According to the method, mixed noise is added, the exploring ability of the intelligent agent is improved, a greedy strategy is introduced to accelerate the learning efficiency of the initial stage of the algorithm and reduce the exploring frequency so as to reduce unnecessary trial and error, and finally, an instant reward value is fused into a priority experience playback mechanism to help the intelligent agent to accurately distinguish the importance of different experiences and accelerate the learning efficiency of the intelligent agent.

Description

Technical Field

[0001] The present invention relates to a path plan acquisition method, belongs to the field of transmission line path planning, and in particular to a line path planning plan generation method and system based on deep reinforcement learning. Background Art

[0002] With the continuous growth of energy consumption and the continuous increase in electricity load demand, the design and planning of transmission lines, as an important carrier of power transmission, are particularly important. Reasonable transmission line planning can effectively meet the growing electricity demand and is an important guarantee for achieving sustainable energy development.

[0003] Traditional transmission line planning methods usually rely on manual field surveys and geographic map analysis, and determine the optimal path through the experience and judgment of designers. However, when facing complex geographical environments, the efficiency and adaptability of relying solely on manual work will be greatly limited. Therefore, it is necessary to combine intelligent technology to meet the needs of formulating optimal transmission line paths in different complex environments. In the existing intelligent path planning technical solutions, there are many technical means such as Dijkstra algorithm, DQN algorithm, taboo search algorithm, etc. However, in the above technical means, not only the importance of different geographic information is not considered, but also the learning efficiency and convergence of the algorithm are low, resulting in a high cost for generating the final transmission line path planning solution. Therefore, there is an urgent need for a more efficient means to achieve low-cost formulation of the optimal path while shortening the path length, and finally complete the transmission line path planning in a complex environment. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects and problems of low efficiency in the prior art and to provide a more efficient method and system for generating a route path planning solution based on deep reinforcement learning.

[0005] To achieve the above objectives, the technical solution of the present invention is: a method for generating a route planning solution based on deep reinforcement learning, comprising:

[0006] S1. Construct a grid map of the transmission line planning area;

[0007] S2, construct an indicator system to evaluate the comprehensive cost of each grid, and determine the indicator weights based on the Critic method to construct a grid comprehensive cost evaluation model;

[0008] S3, constructing a transmission line path planning model with the goal of minimizing the total cost of transmission line path planning; the total cost includes the total comprehensive cost across the grid and the cost of the connection line between each grid;

[0009] S4. Describe the transmission line path planning model as a Markov decision process;

[0010] S5. Introduce the mixed noise containing Gaussian noise and OU noise and the ε-greedy strategy into the deep deterministic policy gradient algorithm, and improve the priority experience replay mechanism based on the dual standards of temporal difference error and reward value to obtain an improved deep deterministic policy gradient algorithm;

[0011] S6. Train the intelligent agent based on the improved deep deterministic policy gradient algorithm and output the optimal transmission line planning path.

[0012] The step S2 specifically includes:

[0013] S21, constructing an evaluation matrix X; the evaluation matrix includes m grids and n evaluation indicators; the expression of the evaluation matrix is ​​as follows:

[0014]

[0015] Where: x ij is the jth evaluation index value of the i-th grid in the evaluation matrix;

[0016] S22, standardize the evaluation matrix to obtain the standardized matrix Y ij ; The standardized expression is as follows:

[0017] Y ij =(y ij ) m×n ;

[0018]

[0019] Where: y ij is the jth evaluation index value of the i-th grid in the standardized matrix;

[0020] S23. Calculate the mean of the scores of each evaluation index in the standardized matrix Standard Deviation j and the Gini coefficient ε j ; The expressions of the mean, standard deviation and Gini coefficient are as follows:

[0021]

[0022]

[0023] Where: x kj is the evaluation value of the jth indicator of the kth grid;

[0024] S24, based on mean With standard deviation s j Calculate the coefficient of variation V of each evaluation index in the standardized matrix j; The expression of the coefficient of variation is as follows:

[0025]

[0026] S25. Calculate the correlation coefficient ρ between different evaluation indicators kl ; The expressions of the correlation coefficients are as follows:

[0027]

[0028] Where: y ik Y ij The evaluation value of the kth indicator of the i-th grid in , Y ij The mean of the kth indicator in y il Y ij The evaluation value of the lth indicator of the ith grid in , Y ij The mean value of the lth indicator in ;

[0029] S26, based on the coefficient of variation V j , Gini coefficient ε j And the correlation coefficient ρ kl , calculate the comprehensive information volume η of each evaluation index j ; The expression of the comprehensive information amount is as follows:

[0030]

[0031] S27, based on the comprehensive information amount η j Calculate the comprehensive weight of each evaluation index ω j ; The expression of the comprehensive weight is as follows:

[0032]

[0033] S28, based on comprehensive weight ω j With the normalized matrix Y ij , obtain the comprehensive score vector M of the grid; the expression of the comprehensive score vector is as follows:

[0034] M=Y ij ×ω;

[0035] Where: ω is the comprehensive weight ω j The comprehensive weight vector.

[0036] In step S3, the expression of the transmission line path planning model is as follows:

[0037]

[0038] Where: min C is the objective function, N is the total number of grids crossed, λ is the cost coefficient per unit length of the line, x i ,y i is the coordinate position of the i-th grid.

[0039] In step S4, the decision elements of the agent in the Markov decision process include the state space S t , action space A and comprehensive reward function r t ;

[0040] The state space expression is as follows:

[0041] S t =(x t-1 ,y t-1 , x t ,y t );

[0042] Where: x t ,y t is the coordinate of the agent at time t; x t-1 ,y t-1 is the coordinate of the agent at time t-1;

[0043] The expression of the action space is as follows:

[0044] A={a1, a2, a3, a4, a5, a6, a7, a8}; a t ∈A;

[0045] Among them: a t is the movement direction of the agent in the action space at time t;

[0046] The comprehensive reward function r t Includes target attraction rewards Inflection Point Rewards Total Cost Award C t ; The expression of the comprehensive reward function is as follows:

[0047]

[0048] Among them: α1, α2, α3 represent the coefficient factors of target attraction reward, inflection point reward, and total cost reward in the comprehensive reward function respectively;

[0049] The goal attracts rewards The expression is as follows:

[0050]

[0051] Where: x f ,y f is the grid coordinate of the target point, dorigin is the distance between the initial point and the target point, d t is the distance between the agent and the target point at time t;

[0052] The inflection point reward The expression is as follows:

[0053]

[0054] In the above formula, D is the relative direction factor, which is expressed as follows:

[0055]

[0056] in: is the direction vector at time t-1, is the direction vector at time t;

[0057] The total cost reward C t The expression is as follows:

[0058]

[0059] Where: M t is the comprehensive score of the grid where the agent is located at time t.

[0060] The step S5 specifically includes:

[0061] S51. In the output action of the deep deterministic policy gradient algorithm, a mixed noise including Gaussian noise and OU noise is added; the expression of the OU noise is as follows:

[0062]

[0063] Among them: a t is the action state of the agent at time t, is the mean of the action sampling data, β is the learning rate, σ1 is the fluctuation degree of the noise, W t For Brownian motion;

[0064] The Gaussian noise is ε~(N, σ 2 ) is superimposed on the exploration strategy of the output action, then the action a after mixed noise optimization is t It is expressed as follows:

[0065] a t ~N Gaussian (μ t +N OU (da t ), σ 2 );

[0066] Where: 2is the Gaussian variance;

[0067] S52, combining the ε-greedy strategy with the deep deterministic policy gradient algorithm; the expression of the ε-greedy strategy is as follows:

[0068]

[0069] Where: c is a constant, i is the total number of trainings, y is the current number of trainings, and α is a constant;

[0070] Then the output action expression of the deep deterministic policy gradient algorithm is as follows:

[0071]

[0072] Where: μ θ (s t ) is the output action when there is no noise, δ is the mixed noise, θ is the algorithm parameter, X is a value uniformly distributed on [0, 1], and A is the action space;

[0073] S53. Define the sampling probability of sample data in the experience pool of the deep deterministic policy gradient algorithm, as follows:

[0074]

[0075] Where: P(f) is the probability of extracting experience f, p f 、p n are the priorities of experience f and experience n respectively, and N is the number of experience samples;

[0076] S54, calculate the timing difference error value δ f and the immediate reward value R f The timing difference error value δ f and the immediate reward value R f The expression is as follows:

[0077] δ f =r t +γQ ω′ (s t+1 , a t+1 )-Q ω (s t , a t );R f =r t ;

[0078] Where: r t is the comprehensive reward function; γ is the discount factor, Q ω′ (s t+1 , a t+1 ) is the output value of parameter ω′, Q ω (st , a t ) is the output value of parameter ω;

[0079] S55, based on the timing difference error value δ f and the immediate reward value R f Calculate the priority of the experience f respectively, and its expressions are as follows:

[0080]

[0081] Where: p f_TD is the priority of the experience f calculated based on the time series difference error value, rank_TD(f) is the ranking of the experience f according to the time series difference error value, p f_R is the priority of experience f calculated based on the reward value, and rank_R(f) is the ranking of experience f according to the immediate reward value;

[0082] S56, combined with p f_TD With p f_R , construct the priority of experience f, which is expressed as follows:

[0083]

[0084] Where: p f is the priority of experience f, ω1 and ω2 are weights, and ω1+ω2=1;

[0085] S57. Based on the priority and sampling probability, experience samples in the experience pool are extracted to update the parameters of the deep deterministic policy gradient algorithm to obtain an improved deep deterministic policy gradient algorithm.

[0086] The step S6 specifically includes:

[0087] S61, initializing the parameters of the grid comprehensive cost evaluation model and setting the training parameters;

[0088] S62, initializing the exploration noise, obtaining the position state S of the planned transmission line path at time t t ;

[0089] S63, based on the current strategy, select the direction of the transmission line path planning at the next moment, and in action O t And add mixed noise, then perform action O t To obtain the comprehensive reward function r of the planned path of this transmission line t After the feedback, enter the position state S at the next moment t+1 ;

[0090] S64, select the current transmission line planning path to obtain (s t , a t, r t , S t+1 ) is stored in the experience pool, and its priority is set according to the temporal difference error value and immediate reward value of the sample, and then the experience sample is learned and the sample priority is updated;

[0091] Where: s t is the state space, a t is the action state of the agent at time t, r t is the comprehensive reward function;

[0092] S65. Update the parameters of the actor network and the critic network in the deep deterministic policy gradient algorithm;

[0093] S66, repeat steps S63-S65 until the training of all time periods is completed, and then proceed to step S67;

[0094] S67. Repeat steps S62-S66 until the maximum number of training times is reached, and output the optimal planning path of the transmission line.

[0095] A route planning scheme generation system based on deep reinforcement learning, the system comprising:

[0096] A grid map construction module is used to construct a grid map of the transmission line planning area;

[0097] The cost evaluation model construction module is used to construct an indicator system for evaluating the comprehensive cost of each grid, and to determine the indicator weights based on the Critic method to construct a grid comprehensive cost evaluation model;

[0098] A path planning model building module is used to build a transmission line path planning model with the goal of minimizing the total cost of transmission line path planning; the total cost includes the total comprehensive cost across the grid and the cost of the connection line between each grid;

[0099] A Markov decision process description module is used to describe the transmission line path planning model as a Markov decision process;

[0100] The gradient algorithm improvement module is used to introduce mixed noise including Gaussian noise and OU noise and ε-greedy strategy into the deep deterministic policy gradient algorithm, and improve the priority experience replay mechanism based on the dual standards of temporal difference error and reward value to obtain an improved deep deterministic policy gradient algorithm;

[0101] The optimal planning path output module is used to train the intelligent agent based on the improved deep deterministic policy gradient algorithm and output the optimal transmission line planning path.

[0102] Preferably, for the specific steps of each module in the above system, please refer to the description of Example 2.

[0103] Compared with the prior art, the present invention has the following beneficial effects:

[0104] The invention discloses a method and system for generating a line path planning scheme based on deep reinforcement learning. The method first constructs a grid map and an evaluation index system of a planning area, and determines the index weight based on the Critic method to construct a grid comprehensive cost evaluation model. Then, a path planning model is constructed with the minimum total cost as the goal, and it is described as a Markov decision process. Then, mixed noise and ε-greedy strategy are introduced into a deep deterministic policy gradient algorithm, and a priority experience replay mechanism is improved to obtain an improved deep deterministic policy gradient algorithm. Finally, an intelligent agent is trained based on the algorithm to output an optimal transmission line planning path. In the application, the design increases the exploration ability of the intelligent agent by adding mixed noise, and introduces an ε-greedy strategy to accelerate the learning efficiency of the initial stage of the algorithm, while reducing the exploration frequency of the intelligent agent, so as to more efficiently utilize existing experience and reduce unnecessary trial and error actions. Finally, an instant reward value is integrated into the priority experience replay mechanism as a new priority judgment standard, which can help the intelligent agent accurately distinguish the importance of different experiences and accelerate the learning efficiency of the intelligent agent. Therefore, the efficiency of the invention is relatively high. BRIEF DESCRIPTION OF THE DRAWINGS

[0105] Figure 1 It is a flow chart of the steps of the method of the present invention.

[0106] Figure 2 It is a structural diagram of the improved deep deterministic policy gradient algorithm of the present invention.

[0107] Figure 3 It is a schematic diagram of the comparison results of the reward changes during the training process of this scheme in Example 1 of the present invention and other schemes.

[0108] Figure 4 It is a schematic diagram of the comparison results of the planning effects of this scheme and other schemes in Example 1 of the present invention.

[0109] Figure 5 It is a schematic diagram of the system structure of the present invention.

[0110] Figure 6 It is a schematic diagram of the device structure of the present invention.

[0111] In the figure: grid map construction module 1, cost evaluation model construction module 2, path planning model construction module 3, Markov decision process description module 4, gradient algorithm improvement module 5, optimal planning path output module 6, processor 7, memory 8, computer program code 81. DETAILED DESCRIPTION

[0112] The present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0113] Embodiment 1:

[0114] See also Figure 1 , a method for generating a route path planning solution based on deep reinforcement learning, comprising:

[0115] S1. Construct a grid map of the transmission line planning area;

[0116] The raster map is obtained as follows: first, high-definition satellite images are captured using GIS, and each pixel in the image is identified and classified, and different areas are marked; then, the pixel values ​​in the continuous space are converted into discrete grid units. Considering that there are impassable areas in the planning area, an obstacle layer and a passable area layer are constructed, and the two layers are superimposed to form the final raster map.

[0117] S2, construct an indicator system to evaluate the comprehensive cost of each grid, and determine the indicator weights based on the Critic method to construct a grid comprehensive cost evaluation model;

[0118] The evaluation index system takes environmental geography, infrastructure, and social economy as the optimal path. Among these objects, the influencing factors are further refined and used as the final evaluation indicators.

[0119] The comprehensive cost evaluation model is a tool used to evaluate the cost-effectiveness of different planning schemes in transmission line planning; it provides a quantitative cost evaluation framework for the intelligent agent by considering multiple evaluation indicators such as environmental geography, infrastructure, and socio-economics. These evaluation indicators constitute the core of the model, and they reflect the key factors in transmission line planning, such as terrain obstacles, land use costs, and social impacts.

[0120] Furthermore, the step S2 specifically includes:

[0121] S21, constructing an evaluation matrix X; the evaluation matrix includes m grids and n evaluation indicators; for the evaluation matrix X with m grids and n evaluation indicators, its expression is as follows:

[0122]

[0123] Where: x ij is the jth evaluation index value of the i-th grid in the evaluation matrix;

[0124] S22, standardize the evaluation matrix to obtain the standardized matrix Y ij ; The standardized expression is as follows:

[0125] Y ij=(y ij ) m×n ;

[0126]

[0127] Where: y ij is the jth evaluation index value of the i-th grid in the standardized matrix;

[0128] S23. Calculate the mean of the scores of each evaluation index in the standardized matrix Standard Deviation j and the Gini coefficient ε j ; The expressions of the mean, standard deviation and Gini coefficient are as follows:

[0129]

[0130]

[0131] Where: x kj is the evaluation value of the jth indicator of the kth grid;

[0132] S24, based on mean With standard deviation s j Calculate the coefficient of variation of each evaluation index in the standardized matrix V j; the expression of the coefficient of variation is as follows:

[0133]

[0134] S25. Calculate the correlation coefficient ρ between different evaluation indicators kl ; The expressions of the correlation coefficients are as follows:

[0135]

[0136] Where: y ik Y ij The evaluation value of the kth indicator of the i-th grid in , Y ij The mean of the kth indicator in y il Y ij The evaluation value of the lth indicator of the ith grid in , Y ij The mean value of the lth indicator in ;

[0137] S26, based on the coefficient of variation V j , Gini coefficient ε j And the correlation coefficient ρ kl , calculate the comprehensive information volume η of each evaluation index j ; The expression of the comprehensive information amount is as follows:

[0138]

[0139] S27, based on the comprehensive information amount η j Calculate the comprehensive weight of each evaluation index ω j ; The expression of the comprehensive weight is as follows:

[0140]

[0141] S28, based on comprehensive weight ω j With the normalized matrix Y ij , obtain the comprehensive score vector M of the grid; the expression of the comprehensive score vector is as follows:

[0142] M=Y ij ×ω;

[0143] Where: ω is the comprehensive weight ω j The comprehensive weight vector formed;

[0144] For grids that do not allow passage, set them to a larger cost value, which is set to 10000 in this solution.

[0145] S3, constructing a transmission line path planning model with the goal of minimizing the total cost of transmission line path planning; the total cost includes the total comprehensive cost across the grid and the cost of the connection line between each grid;

[0146] Furthermore, the expression of the transmission line path planning model is as follows:

[0147]

[0148] Where: min C is the objective function, N is the total number of grids crossed, λ is the cost coefficient per unit length of the line (the cost needs to be determined according to the line material and actual situation), x i ,y i is the coordinate position of the i-th grid.

[0149] S4. Describe the transmission line path planning model as a Markov decision process;

[0150] In this scheme, the transmission line path planning model is described as a Markov decision process (MDP), and the MDP problem is solved by an improved deep deterministic policy gradient (DDPG) algorithm. The DDPG algorithm can quickly converge to a better strategy by simultaneously learning the policy function and the value function. In transmission line path planning, this means that a better line path solution can be found more quickly, reducing the consumption of computing resources and time.

[0151] Decision elements refer to the actions that the agent can take in each state in the Markov decision process, as well as the immediate rewards brought about by these actions and the probability of future state transitions; in the context of transmission line planning, decision elements may include choosing different paths, considering different transmission tower locations, etc.

[0152] The cost information provided by the comprehensive cost evaluation model directly affects the decision-making process of the agent. At each decision moment, the agent will choose the best action based on the cost calculated by the evaluation model. In other words, the comprehensive cost evaluation model provides the agent with a standard for evaluating the consequences of different decisions, and the agent uses these standards to guide its path planning in complex environments.

[0153] In this way, it is ensured that the intelligent agent can take all relevant cost factors into account when planning transmission lines and make the most economical and reasonable decisions accordingly; the combination of this model and decision-making elements improves the efficiency and quality of transmission line planning and ensures the economy and feasibility of the planning scheme.

[0154] Furthermore, the decision elements of the agent in the Markov decision process include the state space S t , action space A and comprehensive reward function r t ;

[0155] The state space uses the current plane coordinates of the agent and the coordinate position of the previous moment as the state of the agent, and its expression is as follows:

[0156] S t =(x t-1 ,y t-1 , x t ,y t );

[0157] Where: x t ,y t is the coordinate of the agent at time t; x t-1 ,y t-1 is the coordinate of the agent at time t-1;

[0158] In the grid map, the action of the agent can be any of the eight movement directions in the state space A, namely east, south, west, north, northeast, northwest, southeast, and southwest, which are expressed as follows:

[0159] A={a1, a2, a3, a4, α5, a6, a7, a8}; a t ∈A;

[0160] Among them: a tis the movement direction of the agent in the action space at time t; a1, a2, a3, a4, a5, a6, a7, a8 represent the eight movement directions;

[0161] A suitable reward function can effectively promote the convergence of the algorithm. In traditional reinforcement learning algorithms, the agent will only receive a reward after completing a task, and there is no reward for the previous series of behaviors. However, this sparse reward mechanism will prevent the agent from obtaining effective rewards in a timely manner when facing a complex environment, making it difficult for the algorithm to converge. Therefore, this scheme designs a comprehensive reward function to guide the agent to quickly explore the optimal path.

[0162] The comprehensive reward function r t Includes target attraction rewards Inflection Point Rewards Total Cost Award C t ; The expression of the comprehensive reward function is as follows:

[0163]

[0164] Among them: α1, α2, α3 represent the coefficient factors of target attraction reward, inflection point reward, and total cost reward in the comprehensive reward function respectively;

[0165] The goal attracts rewards The expression is as follows:

[0166]

[0167]

[0168] Where: x f ,y f is the grid coordinate of the target point, d origin is the distance between the initial point and the target point, d t is the distance between the agent and the target point at time t;

[0169] Inflection Point Rewards It can guide the agent to reduce the occurrence of inflection points as much as possible in path selection, avoid building too many towers and thus increase construction costs; the inflection point reward The expression is as follows:

[0170]

[0171] In the above formula, D is the relative direction factor, which is used to determine whether the inflection point appears, and is expressed as follows:

[0172]

[0173] in: is the direction vector at time t-1, is the direction vector at time t;

[0174] The total cost reward C t The expression is as follows:

[0175]

[0176] Where: M t is the comprehensive score of the grid where the agent is located at time t.

[0177] S5. Introduce the mixed noise containing Gaussian noise and OU noise and the ε-greedy strategy into the deep deterministic policy gradient algorithm, and improve the priority experience replay mechanism based on the dual standards of temporal difference error and reward value to obtain an improved deep deterministic policy gradient algorithm;

[0178] Furthermore, the step S5 specifically includes:

[0179] S51. Adding mixed noise to the output action of the deep deterministic policy gradient algorithm; the mixed noise includes OU noise and Gaussian noise;

[0180] In this scheme, a mixed noise consisting of Gaussian noise and Ornstein-Uhlenbeck (OU) noise is added to the output of the policy network, introducing a certain degree of randomness into its original deterministic process to improve the agent's exploration ability in complex environments; OU noise has memory, that is, there is a certain correlation between the values ​​at adjacent moments, rather than completely independent random variables; this property allows the use of OU noise to maintain a certain smoothness in the exploration process, which helps the agent to effectively explore and learn in the continuous action space.

[0181] The expression of the OU noise is as follows:

[0182]

[0183] Among them: a t is the action state of the agent at time t, is the mean of the action sampling data, β is the learning rate, σ1 is the fluctuation degree of the noise, W t For Brownian motion;

[0184] Gaussian noise is a random process without memory. Each of its random changes is independent of each other and is not affected by any previous moment. Therefore, in this scheme, Gaussian noise is ε~(N,σ 2 ) is superimposed on the exploration strategy of the output action, then the action a after mixed noise optimization is t It is expressed as follows:

[0185] a t ~N Gaussian (μ t +N OU (da t ), σ 2 );

[0186] Where: 2 is the Gaussian variance;

[0187] The addition of mixed noise increases the agent's exploration ability to a certain extent, but the deterministic gradient still plays a dominant role in the exploration process, making the agent still lacking in early exploration ability. To this end, this scheme combines the improved deep deterministic policy gradient (DDPG) with the ε-greedy strategy to further increase the early exploration ability, while updating the ε value based on exponential decay, thereby stabilizing its behavior in the later stage and reducing excessive random exploration.

[0188] S52, combining the ε-greedy strategy with the deep deterministic policy gradient algorithm; the expression of the ε-greedy strategy is as follows:

[0189]

[0190] Where: ε is the ε value, c is a constant greater than 1, i is the total number of trainings, t is the current number of trainings, α is a constant, α∈(0,0.5);

[0191] After combining the DDPG algorithm with the ε-greedy strategy, the output action expression of the deep deterministic policy gradient algorithm is as follows:

[0192]

[0193] Where: μ θ (s t ) is the output action of the policy network without noise, δ is the added mixed noise, θ is the parameter of the policy network algorithm, X is a value uniformly distributed on [0, 1], and A is the action space.

[0194] In this scheme, the learning efficiency of the agent is accelerated based on an improved priority experience replay mechanism. The traditional priority experience replay mechanism uses the temporal difference error (TD-error) as the basis for evaluating the priority of experience. The importance of each experience is determined by calculating the difference between the current estimated value and the target value. Experience with a higher TD-error value has a higher probability of being extracted. This scheme not only considers experiences with higher TD-error values, but also incorporates experiences that receive more direct rewards into the evaluation system. It sets a priority experience replay mechanism through dual standards to more comprehensively evaluate the importance of experience.

[0195] S53. Define the sampling probability of sample data in the experience pool of the deep deterministic policy gradient algorithm, as follows:

[0196]

[0197] Where: P(f) is the probability of extracting experience f, p f 、p n are the priorities of experience f and experience n respectively, and N is the number of experience samples;

[0198] In this scheme, the sampling probability is defined to calculate the probability of sample data in the experience pool being extracted to update the model parameters. The larger the sampling probability, the greater the possibility that the sample will be extracted. After calculating the priority level p of the sample, the sampling probability of the sample can be obtained according to the formula.

[0199] S54, calculate the timing difference error value δ f and the immediate reward value R f ; The expressions of the temporal difference error value and the immediate reward value are as follows:

[0200] δ f =r t +γQ ω′ (s t+1 , a t+1 )-Q ω (s t , a t );R f =r t ;

[0201] Where: r t is the comprehensive reward function; γ is the discount factor, Q ω′ (s t+1 , a t+1 ) is the output value of the target review network with parameter ω′, Q ω (s t , a t ) is the output value of the main comment network with parameter ω;

[0202] See also Figure 2 ,Furthermore, the DDPG algorithm consists of an action network and a comment network, each of which has two modules: the main network and the target network. The main comment network is based on the current state S t and the action a selected by the main action network t To output the Q value, ω is the parameter of the neural network in the comment network. The target comment network is based on the state S at the next moment. t+1 and the action a selected by the target action network t+1The target Q value is output to guide the main review network to update its parameters, and ω′ is the parameter of the neural network in the target review network. This content belongs to the conventional concept in reinforcement learning, so it will not be repeated in this solution.

[0203] S55, based on the timing difference error value δ f and the immediate reward value R f Calculate the priority of the experience f respectively, and its expressions are as follows:

[0204]

[0205] Where: p f_TD is the priority of the experience f calculated based on the timing difference error value, rank_TD(f) is the ranking of the experience f according to the timing difference error value (preferably from high to low), p f_R is the priority of experience f calculated based on the reward value, and rank_R(f) is the ranking of experience f according to the immediate reward value (preferably from high to low);

[0206] S56, combined with p f_TD With p f_R , construct the priority of experience f, which is expressed as follows:

[0207]

[0208] Where: p f is the priority of experience f, ω1 and ω2 are weights, and ω1+ω2=1;

[0209] S57. Based on the priority and sampling probability, experience samples in the experience pool are extracted to update the parameters of the deep deterministic policy gradient algorithm and complete the optimization of the deep deterministic policy gradient algorithm.

[0210] S6. Train the intelligent agent based on the improved deep deterministic policy gradient algorithm and output the optimal transmission line planning path.

[0211] Furthermore, the step S6 specifically includes:

[0212] S61, initializing the parameters of the grid comprehensive cost evaluation model and setting the training parameters;

[0213] S62, initializing the exploration noise, obtaining the position state S of the planned transmission line path at time t t ;

[0214] S63, based on the current strategy, select the direction of the transmission line path planning at the next moment, and in action O t And add mixed noise, then perform action O t To obtain the comprehensive reward function r of the planned path of this transmission linet After the feedback, enter the position state s at the next moment t+1 ;

[0215] S64, select the current transmission line planning path to obtain (s t , a t , r t ,s t+1 ) is stored in the experience pool, and its priority is set according to the temporal difference error value and immediate reward value of the sample, and then the experience sample is learned and the sample priority is updated;

[0216] Where: s t is the state space, a t is the action state of the agent at time t, r t is the comprehensive reward function;

[0217] S65. Update the parameters of the actor network and the critic network in the deep deterministic policy gradient algorithm;

[0218] S66, repeat steps S63-S65 until the training of all time periods is completed, and then proceed to step S67;

[0219] S67. Repeat steps S62-S66 until the maximum number of training times is reached, and output the optimal planning path of the transmission line.

[0220] In this example, the proposed improved deep deterministic policy gradient algorithm (hereinafter referred to as DDPG algorithm) is compared with other DRL methods in terms of reward changes during training. Figure 3 .Depend on Figure 3It can be seen that the reward value obtained by the algorithm proposed in the present invention tends to be stable after about 600 training rounds, which is significantly improved compared to the original DDPG algorithm, which tends to converge after about 800 training rounds. In addition, the reward obtained by the improved DDPG algorithm when it converges is higher than that of the DDPG algorithm; this is mainly because the method proposed in this scheme introduces the improved priority experience replay mechanism into the DDPG algorithm, which accelerates the learning efficiency of the agent, and through a more reasonable priority judgment standard, enables the agent to effectively use training resources and obtain higher reward returns. At the same time, the original DDPG algorithm has a significant drop in reward value during the early exploration process, indicating that the reward is sparse and it is difficult for the agent to select valuable actions. The comprehensive reward function of the method proposed in this scheme can guide the agent to obtain higher rewards and find the optimal action faster. In addition, when training for 100 to 150 rounds, the original DDPG algorithm has been converging to a smaller value, falling into the dilemma of local optimality, which affects the convergence speed of the algorithm. The method proposed in this scheme does not fall into the local optimum during the training process. This is because the introduced mixed noise and ε-greedy strategy increase the randomness of the agent's early action selection. Although this causes certain fluctuations in the early learning, through continuous learning and training, the later agents reduce the exploration space under the guidance of the decaying ε value and make full use of the learned experience, so that the later rewards gradually converge.

[0221] In this embodiment, the planning effects of this scheme and other methods are further compared, see Figure 4 .Depend on Figure 4 It can be seen that the planning effects of the methods proposed in this scheme are better than those of the DDPG algorithm. The path length planned by the original DDPG algorithm is 36.7km, including 26 inflection points. In contrast, the improved DDPG algorithm shortens the path length to 28.6km and only includes 15 inflection points. This is because the method proposed in this scheme adds inflection point rewards to the reward function, so that the algorithm considers the number of inflection points in the path while selecting the optimal path, generating a smoother route without sacrificing the path length.

[0222] Embodiment 2:

[0223] See also Figure 5 , a route planning scheme generation system based on deep reinforcement learning, comprising:

[0224] A grid map construction module 1 is used to construct a grid map of the transmission line planning area;

[0225] Cost evaluation model construction module 2 is used to construct an indicator system for evaluating the comprehensive cost of each grid, and to determine the indicator weights based on the Critic method to construct a grid comprehensive cost evaluation model;

[0226] Furthermore, the cost evaluation model construction module 2 is used to construct a grid comprehensive cost evaluation model according to the following steps:

[0227] S21, constructing an evaluation matrix X; the evaluation matrix includes m grids and n evaluation indicators; the expression of the evaluation matrix is ​​as follows:

[0228]

[0229] Where: x ij is the jth evaluation index value of the i-th grid in the evaluation matrix;

[0230] S22, standardize the evaluation matrix to obtain the standardized matrix Y ij ; The standardized expression is as follows:

[0231] Y ij =(y ij )m×n;

[0232]

[0233] Where: y ij is the jth evaluation index value of the i-th grid in the standardized matrix;

[0234] S23. Calculate the mean of the scores of each evaluation index in the standardized matrix Standard Deviation j and the Gini coefficient ε j ; The expressions of the mean, standard deviation and Gini coefficient are as follows:

[0235]

[0236]

[0237] Where: x kj is the evaluation value of the jth indicator of the kth grid;

[0238] S24, based on mean With standard deviation s j Calculate the coefficient of variation V of each evaluation index in the standardized matrix j ; The expression of the coefficient of variation is as follows:

[0239]

[0240] S25. Calculate the correlation coefficient ρ between different evaluation indicators kl ; The expressions of the correlation coefficients are as follows:

[0241]

[0242] Where: y ik Y ij The evaluation value of the kth indicator of the i-th grid in , Y ij The mean of the kth indicator in y il Y ij The evaluation value of the lth indicator of the ith grid in , Y ij The mean value of the lth indicator in ;

[0243] S26, based on the coefficient of variation V j , Gini coefficient ε j And the correlation coefficient ρ kl , calculate the comprehensive information volume η of each evaluation index j ; The expression of the comprehensive information amount is as follows:

[0244]

[0245] S27, based on the comprehensive information amount η j Calculate the comprehensive weight of each evaluation index ω j ; The expression of the comprehensive weight is as follows:

[0246]

[0247] S28, based on comprehensive weight ω j With the normalized matrix Y ij , obtain the comprehensive score vector M of the grid; the expression of the comprehensive score vector is as follows:

[0248] M=Y ij ×ω;

[0249] Where: ω is the comprehensive weight ω j The comprehensive weight vector.

[0250] A path planning model building module 3 is used to build a transmission line path planning model with the goal of minimizing the total cost of transmission line path planning; the total cost includes the total comprehensive cost of crossing the grid and the cost of the connection line between each grid;

[0251] Furthermore, the transmission line path planning model constructed by the path planning model construction module 3 is as follows:

[0252]

[0253] Where: min C is the objective function, N is the total number of grids crossed, λ is the cost coefficient per unit length of the line, x i ,y i is the coordinate position of the i-th grid;

[0254] A Markov decision process description module 4 is used to describe the transmission line path planning model as a Markov decision process;

[0255] Furthermore, the decision elements of the agent in the Markov decision process described by the Markov decision process description module 4 include the state space S t , action space A and comprehensive reward function r t ;

[0256] The state space expression is as follows:

[0257] s t =(x t-1 ,y t-1 , x t ,y t );

[0258] Where: x t ,y t is the coordinate of the agent at time t; x t-1 ,y t-1 is the coordinate of the agent at time t-1;

[0259] The expression of the action space is as follows:

[0260] A={a1, a2, a3, a4, a5, a6, a7, a8}; a t ∈A;

[0261] Among them: a t is the movement direction of the agent in the action space at time t;

[0262] The comprehensive reward function r t Includes target attraction rewards Inflection Point Rewards Total Cost Award C t ; The expression of the comprehensive reward function is as follows:

[0263]

[0264] Among them: α1, α2, α3 represent the coefficient factors of target attraction reward, inflection point reward, and total cost reward in the comprehensive reward function respectively;

[0265] The goal attracts rewards The expression is as follows:

[0266]

[0267] Where: x f ,y f is the grid coordinate of the target point, dorigin is the distance between the initial point and the target point, d t is the distance between the agent and the target point at time t;

[0268] The inflection point reward The expression is as follows:

[0269]

[0270] In the above formula, D is the relative direction factor, which is expressed as follows:

[0271]

[0272] in: is the direction vector at time t-1, is the direction vector at time t;

[0273] The total cost reward C t The expression is as follows:

[0274]

[0275] Where: M t is the comprehensive score of the grid where the agent is located at time t.

[0276] Gradient algorithm improvement module 5 is used to introduce mixed noise including Gaussian noise and OU noise and ε-greedy strategy into the deep deterministic policy gradient algorithm, and improve the priority experience replay mechanism based on the dual standards of temporal difference error and reward value to obtain an improved deep deterministic policy gradient algorithm;

[0277] Further, the gradient algorithm improvement module 5 improves the deep deterministic policy gradient algorithm according to the following steps:

[0278] S51. In the output action of the deep deterministic policy gradient algorithm, a mixed noise including Gaussian noise and OU noise is added; the expression of the OU noise is as follows:

[0279]

[0280] Among them: a t is the action state of the agent at time t, is the mean of the action sampling data, β is the learning rate, σ1 is the fluctuation degree of the noise, W t For Brownian motion;

[0281] The Gaussian noise is ε~(N, σ 2 ) is superimposed on the exploration strategy of the output action, then the action a after mixed noise optimization is t It is expressed as follows:

[0282] a t ~N Gaussian (μ t +N OU (da t ), σ 2 );

[0283] Where: 2 is the Gaussian variance;

[0284] S52, combining the ε-greedy strategy with the deep deterministic policy gradient algorithm; the expression of the ε-greedy strategy is as follows:

[0285]

[0286] Where: c is a constant, i is the total number of trainings, t is the current number of trainings, and α is a constant;

[0287] Then the output action expression of the deep deterministic policy gradient algorithm is as follows:

[0288]

[0289] Where: μ θ (s t ) is the output action when there is no noise, δ is the mixed noise, θ is the algorithm parameter, X is a value uniformly distributed on [0, 1], and A is the action space;

[0290] S53. Define the sampling probability of sample data in the experience pool of the deep deterministic policy gradient algorithm, as follows:

[0291]

[0292] Where: P(f) is the probability of extracting experience f, p f 、p n are the priorities of experience f and experience n respectively, and N is the number of experience samples;

[0293] S54, calculate the timing difference error value δ f and the immediate reward value R f The timing difference error value δ f and the immediate reward value R f The expression is as follows:

[0294] δ f =r t +γQ ω′ (s t+1 , a t+1 )-Q ω (s t , a t );

[0295] R f =r t ;

[0296] Where: r t is the comprehensive reward function; γ is the discount factor, Q ω′ (s t+1 , a t+1 ) is the output value of parameter ω′, Q ω (s t , a t ) is the output value of parameter ω;

[0297] S55, based on the timing difference error value δ f and the immediate reward value R f Calculate the priority of the experience f respectively, and its expressions are as follows:

[0298]

[0299] Where: p f_TD is the priority of the experience f calculated based on the time series difference error value, rank_TD(f) is the ranking of the experience f according to the time series difference error value, p f_R is the priority of experience f calculated based on the reward value, and rank_R(f) is the ranking of experience f according to the immediate reward value;

[0300] S56, combined with p f_TD With p f_R , construct the priority of experience f, which is expressed as follows:

[0301]

[0302] Where: p f is the priority of experience f, ω1 and ω2 are weights, and ω1+ω2=1;

[0303] S57. Based on the priority and sampling probability, experience samples in the experience pool are extracted to update the parameters of the deep deterministic policy gradient algorithm to obtain an improved deep deterministic policy gradient algorithm.

[0304] The optimal planning path output module 6 is used to train the intelligent agent based on the improved deep deterministic policy gradient algorithm and output the optimal transmission line planning path;

[0305] Furthermore, the optimal planning path output module 6 outputs the optimal transmission line planning path according to the following steps:

[0306] S61, initializing the parameters of the grid comprehensive cost evaluation model and setting the training parameters;

[0307] S62, initializing the exploration noise, obtaining the position state S of the planned transmission line path at time t t ;

[0308] S63, based on the current strategy, select the direction of the transmission line path planning at the next moment, and in action O t And add mixed noise, then perform action O t To obtain the comprehensive reward function r of the planned path of this transmission line t After the feedback, enter the position state S at the next moment t+1 ;

[0309] S64, select the current transmission line planning path to obtain (s t , a t , r t ,s t+1 ) is stored in the experience pool, and its priority is set according to the temporal difference error value and immediate reward value of the sample, and then the experience sample is learned and the sample priority is updated;

[0310] Where: S t is the state space, a t is the action state of the agent at time t, r t is the comprehensive reward function;

[0311] S65. Update the parameters of the actor network and the critic network in the deep deterministic policy gradient algorithm;

[0312] S66, repeat steps S63-S65 until the training of all time periods is completed, and then proceed to step S67;

[0313] S67. Repeat steps S62-S66 until the maximum number of training times is reached, and output the optimal planning path of the transmission line.

[0314] Embodiment 3:

[0315] See also Figure 6 In this embodiment, a route planning scheme generating device based on deep reinforcement learning is also included, and the device includes a processor 7 and a memory 8;

[0316] The memory 8 is used to store the computer program code 81 and transmit the computer program code 81 to the processor 7;

[0317] The processor 7 is used to execute the line path planning solution generation method based on deep reinforcement learning described in Example 1 according to the instructions in the computer program code 81.

[0318] In this embodiment, a computer-readable storage medium is also included, in which computer-executable instructions are stored. When the computer-executable instructions are executed on a computer, the method for generating a line path planning solution based on deep reinforcement learning described in Example 1 is implemented.

[0319] Generally speaking, the computer instructions for implementing the method of the present invention may be carried in any combination of one or more computer-readable storage media. Non-transitory computer-readable storage media may include any computer-readable media, except for the signal itself that is temporarily propagating.

[0320] Computer program code for performing the operation of the present invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, SMalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages, in particular, Python suitable for neural network computing and platform frameworks based on TensorFlow, PyTorch, etc. can be used. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer or to an external computer (for example, using an Internet service provider to connect via the Internet) through any type of network, including a local area network (LAN) or a wide area network (WAN).

[0321] The above-mentioned device and non-temporary computer-readable storage medium can be found in the specific description of a method for generating a route path planning solution based on deep reinforcement learning and its beneficial effects, which will not be repeated here.

[0322] Although the embodiments of the present invention have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for generating a route planning scheme based on deep reinforcement learning, characterized in that: include: S1. Construct a grid map of the transmission line planning area; S2, construct an indicator system to evaluate the comprehensive cost of each grid, and determine the indicator weights based on the Critic method to construct a grid comprehensive cost evaluation model; S3, constructing a transmission line path planning model with the goal of minimizing the total cost of transmission line path planning; the total cost includes the total comprehensive cost across the grid and the cost of the connection line between each grid; S4. Describe the transmission line path planning model as a Markov decision process; S5. Introduce the mixed noise containing Gaussian noise and OU noise and the ε-greedy strategy into the deep deterministic policy gradient algorithm, and improve the priority experience replay mechanism based on the dual standards of temporal difference error and reward value to obtain an improved deep deterministic policy gradient algorithm; S6. Train the intelligent agent based on the improved deep deterministic policy gradient algorithm and output the optimal transmission line planning path.

2. The method for generating a route path planning solution based on deep reinforcement learning according to claim 1, characterized in that: The step S2 specifically includes: S21, constructing an evaluation matrix X; the evaluation matrix includes m grids and n evaluation indicators; the expression of the evaluation matrix is ​​as follows: Where: x ij is the jth evaluation index value of the i-th grid in the evaluation matrix; S22, standardize the evaluation matrix to obtain the standardized matrix Y ij ; The standardized expression is as follows: AND ij (and ij ) m×n ; Where: y ij is the jth evaluation index value of the i-th grid in the standardized matrix; S23. Calculate the mean of the scores of each evaluation index in the standardized matrix Standard Deviation j and the Gini coefficient ε j ; The expressions of the mean, standard deviation and Gini coefficient are as follows: Where: x kj is the evaluation value of the jth indicator of the kth grid; S24, based on mean With standard deviation S j Calculate the coefficient of variation V of each evaluation index in the standardized matrix j ; The expression of the coefficient of variation is as follows: S25. Calculate the correlation coefficient ρ between different evaluation indicators kl ; The expressions of the correlation coefficients are as follows: Where: y ik Y ij The evaluation value of the kth indicator of the i-th grid in , Y ij The mean of the kth indicator in y il Y ij The evaluation value of the lth indicator of the uth grid in, Y ij The mean value of the lth indicator in ; S26, based on the coefficient of variation V j , Gini coefficient ε j And the correlation coefficient ρ kl , calculate the comprehensive information volume η of each evaluation index j ; The expression of the comprehensive information amount is as follows: S27, based on the comprehensive information amount η j Calculate the comprehensive weight of each evaluation index ω j ; The expression of the comprehensive weight is as follows: S28, based on comprehensive weight ω j With the normalized matrix Y ij , obtain the comprehensive score vector M of the grid; the expression of the comprehensive score vector is as follows: M=Y ij ×ω; Where: ω is the comprehensive weight ω j The comprehensive weight vector.

3. The method for generating a route path planning solution based on deep reinforcement learning according to claim 2, characterized in that: In step S3, the expression of the transmission line path planning model is as follows: Where: min C is the objective function, N is the total number of grids crossed, λ is the cost coefficient per unit length of the line, x i ,y i is the coordinate position of the i-th grid.

4. The method for generating a route planning scheme based on deep reinforcement learning according to claim 1, characterized in that: In step S4, the decision elements of the agent in the Markov decision process include the state space S t , action space A and comprehensive reward function r t ; The state space expression is as follows: s t (x) t-1 ,and t-1 ,x t ,and t ); Where: x t ,y t is the coordinate of the agent at time t; x t-1 ,y t-1 is the coordinate of the agent at time t-1; The expression of the action space is as follows: A={a1, a2, a3, a4, a5, a6, a7, a8}; a t ∈A; Among them: a t is the movement direction of the agent in the action space at time t; The comprehensive reward function r t Includes target attraction rewards Inflection Point Rewards Total Cost Award C t ; The expression of the comprehensive reward function is as follows: Among them: α1, α2, α3 represent the coefficient factors of target attraction reward, inflection point reward, and total cost reward in the comprehensive reward function respectively; The goal attracts rewards The expression is as follows: Where: x f ,y f is the grid coordinate of the target point, d origin is the distance between the initial point and the target point, d t is the distance between the agent and the target point at time t; The inflection point reward The expression is as follows: In the above formula, D is the relative direction factor, which is expressed as follows: in: is the direction vector at time -1, is the direction vector at time t; The total cost reward C t The expression is as follows: Where: M t is the comprehensive score of the grid where the agent is located at time t.

5. The method for generating a route path planning solution based on deep reinforcement learning according to claim 1, characterized in that: The step S5 specifically includes: S51. In the output action of the deep deterministic policy gradient algorithm, a mixed noise including Gaussian noise and 0U noise is added; the expression of the 0U noise is as follows: Among them: a t is the action state of the agent at time t, is the mean of the action sampling data, β is the learning rate, σ1 is the fluctuation degree of the noise, W t For Brownian motion; The Gaussian noise is ε~(N, σ 2 ) is superimposed on the exploration strategy of the output action, then the action a after mixed noise optimization is t It is expressed as follows: a t ~N Gaussian (m t +N OU (yes) t ),s 2 ); Where: 2 is the Gaussian variance; S52, combining the ε-greedy strategy with the deep deterministic policy gradient algorithm; the expression of the ε-greedy strategy is as follows: α∈(0，0.5)； Where: c is a constant, i is the total number of trainings, t is the current number of trainings, and α is a constant; Then the output action expression of the deep deterministic policy gradient algorithm is as follows: Where: μ θ (s t ) is the output action when there is no noise, δ is the mixed noise, θ is the algorithm parameter, X is a value uniformly distributed on [0, 1], and A is the action space; S53. Define the sampling probability of sample data in the experience pool of the deep deterministic policy gradient algorithm, as follows: Where: P(f) is the probability of extracting experience f, p f 、p n are the priorities of experience f and experience n respectively, and N is the number of experience samples; S54, calculate the timing difference error value δ f and the immediate reward value R f The timing difference error value δ f and the immediate reward value R f The expression is as follows: δ f =r t +γQ ω′ (s t+1 ,a t+1 )-Q ω (s t ,a t ); R f ＝r t ； Where: r t is the comprehensive reward function; γ is the discount factor, Q ω′ (s t+1 , a t+1 ) is the output value of parameter ω′, Q ω (s t , a t ) is the output value of parameter ω; S55, based on the timing difference error value δ f and the immediate reward value R f Calculate the priority of the experience f respectively, and its expressions are as follows: Where: p f_TD is the priority of the experience f calculated based on the time series difference error value, rank_TD(f) is the ranking of the experience f according to the time series difference error value, p f_R is the priority of experience f calculated based on the reward value, and rank_R(f) is the ranking of experience f according to the immediate reward value; S56, combined with p f_TD With p f_R , construct the priority of experience f, which is expressed as follows: Where: p f is the priority of experience f, ω1 and ω2 are weights, and ω1+ω2=1; S57. Based on the priority and sampling probability, experience samples in the experience pool are extracted to update the parameters of the deep deterministic policy gradient algorithm to obtain an improved deep deterministic policy gradient algorithm.

6. The method for generating a route path planning solution based on deep reinforcement learning according to claim 1, characterized in that: The step S6 specifically includes: S61, initializing the parameters of the grid comprehensive cost evaluation model and setting the training parameters; S62, initializing the exploration noise, obtaining the position state S of the planned transmission line path at time t t ; S63, based on the current strategy, select the direction of the transmission line path planning at the next moment, and in action O t And add mixed noise, then perform action O t To obtain the comprehensive reward function r of the planned path of this transmission line t After the feedback, enter the position state s at the next moment t+1 ; S64, select the current transmission line planning path to obtain (s t , a t , r t ,s t+1 ) is stored in the experience pool, and its priority is set according to the temporal difference error value and immediate reward value of the sample, and then the experience sample is learned and the sample priority is updated; Where: s t is the state space, a t is the action state of the agent at time t, r t is the comprehensive reward function; S65. Update the parameters of the actor network and the critic network in the deep deterministic policy gradient algorithm; S66, repeat steps S63-S65 until the training of all time periods is completed, and then proceed to step S67; S67. Repeat steps S62-S66 until the maximum number of training times is reached, and output the optimal planning path of the transmission line.

7. A route planning scheme generation system based on deep reinforcement learning, characterized in that: The system comprises: A grid map construction module (1), used to construct a grid map of the transmission line planning area; The cost evaluation model construction module (2) is used to construct an indicator system for evaluating the comprehensive cost of each grid, and to determine the indicator weights based on the Critic method to construct a grid comprehensive cost evaluation model; A path planning model building module (3) is used to build a transmission line path planning model with the goal of minimizing the total cost of transmission line path planning; the total cost includes the total comprehensive cost of crossing grids and the cost of connecting lines between each grid; A Markov decision process description module (4), used for describing the transmission line path planning model as a Markov decision process; A gradient algorithm improvement module (5) is used to introduce mixed noise including Gaussian noise and 0U noise and ε-greedy strategy into the deep deterministic policy gradient algorithm, and improve the priority experience replay mechanism based on the dual standards of temporal difference error and reward value to obtain an improved deep deterministic policy gradient algorithm; The optimal planning path output module (6) is used to train the intelligent agent based on the improved deep deterministic policy gradient algorithm and output the optimal transmission line planning path.

8. The line path planning scheme generation system based on deep reinforcement learning according to claim 7 is characterized in that: The cost evaluation model construction module (2) is used to construct a grid comprehensive cost evaluation model according to the following steps: S21, constructing an evaluation matrix X; the evaluation matrix includes m grids and n evaluation indicators; the expression of the evaluation matrix is ​​as follows: Where: x ij is the jth evaluation index value of the i-th grid in the evaluation matrix; S22, standardize the evaluation matrix to obtain the standardized matrix Y ij ; The standardized expression is as follows: AND ij (and ij ) m×n ; Where: y ij is the jth evaluation index value of the i-th grid in the standardized matrix; S23. Calculate the mean of the scores of each evaluation index in the standardized matrix Standard Deviation j and the Gini coefficient ε j ; The expressions of the mean, standard deviation and Gini coefficient are as follows: Where: x kj is the evaluation value of the jth indicator of the kth grid; S24, based on mean With standard deviation S j Calculate the coefficient of variation V of each evaluation index in the standardized matrix j The change The expression of the different coefficients is as follows: S25, calculating the correlation coefficient ρkl between different evaluation indicators; the expressions of the correlation coefficients are as follows: Where: y ik Y ij The evaluation value of the kth indicator of the i-th grid in , Y ij The mean of the kth indicator in y il Y ij The evaluation value of the ,th indicator of the ith grid in , Y ij The mean value of the lth indicator in ; S26, based on the coefficient of variation V j , Gini coefficient ε j And the correlation coefficient ρ kl , calculate the comprehensive information volume η of each evaluation index j ; The expression of the comprehensive information amount is as follows: S27, calculating the comprehensive weight ωj of each evaluation index based on the comprehensive information amount ηj; the expression of the comprehensive weight is as follows: S28, based on comprehensive weight ω j With the normalized matrix Y ij , obtain the comprehensive score vector M of the grid; the expression of the comprehensive score vector is as follows: M=Y ij ×ω; Where: ω is the comprehensive weight ω j The comprehensive weight vector.

9. The line path planning scheme generation system based on deep reinforcement learning according to claim 7, characterized in that: The transmission line path planning model constructed by the path planning model construction module (3) is as follows: Where: min C is the objective function, N is the total number of grids crossed, λ is the cost coefficient per unit length of the line, x i ,y i is the coordinate position of the i-th grid; The decision elements of the intelligent agent in the Markov decision process described by the Markov decision process description module (4) include the state space S t , action space A and comprehensive reward function r t ; The state space expression is as follows: s t (x) t-1 ,and t-1 ,x t ,and t ); Where: x t ,y t is the coordinate of the agent at time t; x t-1 ,y t-1 is the coordinate of the agent at time t-1; The expression of the action space is as follows: A={a1, a2, a3, a4, a5, a6, a7, a8}; a t ∈A; Among them: a t is the movement direction of the agent in the action space at time t; The comprehensive reward function r t Includes target attraction rewards Inflection Point Rewards Total Cost Award C t ; The expression of the comprehensive reward function is as follows: Among them: α1, α2, α3 represent the coefficient factors of target attraction reward, inflection point reward, and total cost reward in the comprehensive reward function respectively; The goal attracts rewards The expression is as follows: Where: x f ,y f is the grid coordinate of the target point, d origin is the distance between the initial point and the target point, d t is the distance between the agent and the target point at time t; The inflection point reward The expression is as follows: In the above formula, D is the relative direction factor, which is expressed as follows: in: is the direction vector at time t-1, is the direction vector at time t; The total cost reward C t The expression is as follows: Where: M t is the comprehensive score of the grid where the agent is located at time t.

10. The line path planning scheme generation system based on deep reinforcement learning according to claim 7, characterized in that: The gradient algorithm improvement module (5) improves the deep deterministic policy gradient algorithm according to the following steps: S51. In the output action of the deep deterministic policy gradient algorithm, a mixed noise including Gaussian noise and OU noise is added; the expression of the OU noise is as follows: Among them: a t is the action state of the agent at time t, is the mean of the action sampling data, β is the learning rate, σ1 is the fluctuation degree of the noise, W t For Brownian motion; The Gaussian noise is ε~(N, σ 2 ) is superimposed on the exploration strategy of the output action, then the action a after mixed noise optimization is t It is expressed as follows: a t ~N Gaussian (m t +N OU (yes) t ),s 2 ); Where: τ 2 is the Gaussian variance; S52, combining the ε-greedy strategy with the deep deterministic policy gradient algorithm; the expression of the ε-greedy strategy is as follows: α∈(0，0.5)； Where: c is a constant, i is the total number of trainings, t is the current number of trainings, and α is a constant; Then the output action expression of the deep deterministic policy gradient algorithm is as follows: Where: μ θ (s t ) is the output action when there is no noise, δ is the mixed noise, θ is the algorithm parameter, X is a value uniformly distributed on [0, 1], and A is the action space; S53. Define the sampling probability of sample data in the experience pool of the deep deterministic policy gradient algorithm, as follows: Where: P(f) is the probability of extracting experience f, p f 、p n are the priorities of experience f and experience n respectively, and N is the number of experience samples; S54, calculate the timing difference error value δ f and the immediate reward value R f The timing difference error value δ f and the immediate reward value R f The expression is as follows: δ f =r t +γQ ω′ (s t+1 ,a t+1 )-Q ω (s t ,a t ); R f ＝r t ； Where: r t is the comprehensive reward function; γ is the discount factor, Q ω′ (s t+1 , a t+1 ) is the output value of parameter ω′, Q ω (s t , a t ) is the output value of parameter ω; S55, based on the timing difference error value δ f and the immediate reward value R f Calculate the priority of the experience f respectively, and its expressions are as follows: Where: p f_TD is the priority of the experience f calculated based on the time series difference error value, rank_TD(f) is the ranking of the experience f according to the time series difference error value, p f_R is the priority of experience f calculated based on the reward value, and rank_R(f) is the ranking of experience f according to the immediate reward value; S56, combined with p f_TD With p f_R , construct the priority of experience f, which is expressed as follows: Where: p f is the priority of experience f, ω1 and ω2 are weights, and ω1+ω2=1; S57. Based on the priority and sampling probability, experience samples in the experience pool are extracted to update the parameters of the deep deterministic policy gradient algorithm to obtain an improved deep deterministic policy gradient algorithm.

Images