Distribution network protection control method and system based on deep reinforcement learning

Abstract

The present invention discloses a distribution network protection control method and system based on deep reinforcement learning, including an intelligent agent acquiring local measurement data, using the acquired perception information as environmental state information for deep reinforcement learning; the intelligent agent acquiring action types and parameters as action space information for deep reinforcement learning; designing a reward function in the process of interaction between the intelligent agent and the environment; constructing a deep reinforcement learning neural network model; training the deep learning neural network; making autonomous decisions on the acquired perception information based on the trained deep reinforcement learning neural network model to obtain instructions for controlling the circuit breaker action. The present invention changes the idea of ​​traditional current protection that uses current setting value as the basis for action judgment, and allows the protection device to autonomously perceive the operating state of the distribution network, and through continuous trial and error learning, adaptively adjust the protection action strategy to meet the selectivity and speed of protection actions in a highly uncertain distribution network environment.

Description

Technical Field

[0001] The present invention belongs to the field of relay protection of power distribution networks, and specifically relates to a protection control method and system for a distributed power distribution network based on deep reinforcement learning. Background Art

[0002] Traditional distribution networks widely use three-stage current protection, advocate the continuous high-proportion development of non-fossil energy, and connect a large number of distributed power sources to the distribution network, which makes the distribution network present different fault characteristics. The setting calculation of traditional current protection faces huge challenges. The main challenges include: (1) The flexible and complex operation mode of the distribution network leads to drastic changes in the fault current level. On the one hand, it is difficult to determine the maximum and minimum operation modes in the setting calculation. On the other hand, the calculated constant value has poor performance in most operation modes, and there are problems such as small protection range and long fault removal delay; (2) The large-scale access of distributed power sources in the distribution network has transformed the distribution network from a single-ended radial network to a multi-source network. The direction of the fault current is uncertain, which makes the upper and lower relationships between protections change dynamically, increasing the difficulty of setting coordination; (3) More and more renewable energy sources are connected to the grid using power electronic inverters. Inverter-type distributed power sources still have flexible control capabilities under fault or abnormal operating conditions, and output strong nonlinear fault characteristics, resulting in the fault characteristics of the distribution network being very different from those of the traditional distribution network.

[0003] In order to improve the performance of distribution network relay protection, domestic and foreign scholars have conducted a lot of research from different perspectives. However, most of the methods proposed so far focus on the optimization of traditional current protection setting parameters: with the help of the communication system, the distribution network protection master station is used to obtain the grid topology, line parameters, load parameters, distributed power grid connection status, etc., and the protection optimization setting values ​​under different operating modes are calculated offline or online, and then the current protection setting values ​​are updated through the communication system, so that the protection setting values ​​can be adaptively adjusted and updated as the system operation mode changes. The problems of these studies are as follows: (1) It is necessary to obtain regional information through the master station with the help of the communication system, which has high requirements on the processing capacity, communication bandwidth and real-time performance of the master station. The control response speed is slow, and it is highly dependent on the communication system and the master station. When the master station or communication fails, the entire protection scheme will not work; (2) It is still limited to the current threshold as the criterion. Various optimization setting algorithms are complex, and it is difficult to achieve online setting. It is easily affected by the error of the grid operation parameters, resulting in inaccurate judgment criteria. It is only applicable to small distribution networks and low distributed power penetration. Summary of the invention

[0004] The purpose of the present invention is to solve the above-mentioned deficiencies in the prior art, and to provide a protection and control method and system for a distribution network containing distributed power sources based on deep reinforcement learning. The distribution network protection and control problem is described as a multi-agent reinforcement learning problem, and the idea of ​​using the current setting value as the basis for action judgment of traditional current protection is changed. The protection device is allowed to autonomously perceive the operating status of the distribution network, and through continuous trial and error learning, the protection action strategy is adaptively adjusted to meet the selectivity and speed of protection actions in a highly uncertain distribution network environment.

[0005] The technical solution adopted by the present invention to achieve the above-mentioned purpose is:

[0006] A distribution network protection control system based on deep reinforcement learning, comprising: configuring an intelligent agent unit at each circuit breaker node in a distribution network containing distributed power sources, wherein each intelligent agent senses the operating state of the distribution network through local measurement data, outputs instructions to control the circuit breaker action and obtains rewards from the next state of the distribution network operation, and continuously iterates to converge on the optimal action strategy; the intelligent agent unit is mounted on a relay protection device, and comprises a modeling optimization module and a real-time action determination module;

[0007] The modeling optimization module is used to take the detection signal of the relay protection device as input, train the deep learning neural network so that it can output a corresponding action signal according to the detection signal, and obtain an optimized deep learning neural network model;

[0008] The real-time action determination module is used to collect detection signals from the distribution network site in real time and input them into the optimized deep learning neural network model, so as to obtain the action signal output by the model and output it to the circuit breaker to execute the action instruction.

[0009] The modeling optimization module includes a signal detection module, an action space module, a reward function module, a deep learning neural network building module, and an algorithm training module;

[0010] The signal detection module is used to input instructions to control the relay protection device to collect environmental status information of the current node; the environmental status information includes current signals, voltage signals, phase angle measurement values, frequency measurement values ​​and the switch status of the corresponding circuit breaker;

[0011] The action space module is used to output the action type and parameters corresponding to the environmental state information according to the ε greedy rule, and the action type is the on / off action and action delay of the control circuit breaker;

[0012] The reward function module is used to determine the reward rules during the interaction between the agent and the environment;

[0013] The deep learning neural network building module is used to use a deep neural network to approximate the Q function and build a deep learning neural network model DQN;

[0014] The algorithm training module is used to use the environmental state information obtained by the signal detection module as the input of the deep learning neural network, and the action space information obtained by the action space module as the output of the deep learning neural network, and train the deep learning neural network according to the reward rules set by the reward function module to obtain an optimized deep learning neural network model.

[0015] The reward rules include:

[0016]

[0017] The main protection zone refers to the control range of the current circuit breaker node, and the backup protection zone refers to the control range of the adjacent nodes of the current circuit breaker.

[0018] The algorithm training module includes simulation system training and actual system training;

[0019] The simulation system training is used to first pre-learn the agent in the simulation system based on actual working conditions to accumulate corresponding experience knowledge;

[0020] The actual system training is used to put the intelligent agent into the actual system for training and learning based on pre-learning, so as to improve the efficiency and performance of reinforcement learning.

[0021] The real-time action determination module includes an optimal action determination module and an action output module;

[0022] The optimal action determination module is used to make an autonomous decision on the state obtained by the signal detection module according to the optimized deep reinforcement learning neural network model to obtain an action to control the circuit breaker;

[0023] The action output module is used to output an action execution instruction to the circuit breaker according to the action of controlling the circuit breaker.

[0024] A distribution network protection and control method based on deep reinforcement learning is proposed. An intelligent agent unit is configured at each circuit breaker node in a distribution network containing distributed power sources. The intelligent agent unit senses the operation status of the distribution network through local measurement data, outputs instructions to control the circuit breaker action, and obtains rewards from the next state of the distribution network operation. The intelligent agent unit is configured at each circuit breaker node in the ... is configured at each circuit breaker node in the distribution network containing distributed power sources. The intelligent agent unit is configured at each circuit breaker node in the distribution network containing distributed power sources. The intelligent agent unit is configured at each circuit breaker node in the distribution network containing distributed power sources. The intelligent agent unit is configured at each circuit breaker node in the distribution network containing distributed power sources. The intelligent agent unit is configured at

[0025] Step 1: Taking the detection signal of the relay protection device as input, training the deep learning neural network so that it can output a corresponding action signal according to the detection signal, and obtaining an optimized deep learning neural network model;

[0026] Step 2: Collect the detection signals of the distribution network site in real time and input them into the optimized deep learning neural network model, so as to obtain the action signal output by the model and output it to the circuit breaker to execute the action instruction.

[0027] The step 1 uses the acquired environmental state information as the input of the deep learning neural network, uses the acquired action space information as the output of the deep learning neural network, trains the deep learning neural network according to the reward rule, and obtains an optimized deep learning neural network model; specifically includes:

[0028] Step 1-1, initialize the deep learning neural network model parameters ω;

[0029] Step 1-2, observe the environment to obtain state S, and select action A according to the ε greedy rule; the environmental state information includes current signal, voltage signal, phase angle measurement value, frequency measurement value and the switch state of the corresponding circuit breaker; the action type is to control the opening and closing actions and action delay of the circuit breaker;

[0030] Step 1-3: Take action A, obtain the reward R given by the environment, and observe the new state S';

[0031] Step 1-4: store the obtained (S, A, R, S') into the experience pool D;

[0032] Step 1-5: Randomly select B (S, A, R, S') from D to form a set, and use the gradient descent method to update the parameters ω of the network model;

[0033] Steps 1-6: Through continuous iterative updates, the parameter ω eventually becomes stable, and the optimized deep learning neural network model DQN is obtained.

[0034] The reward rules include:

[0035]

[0036] The main protection zone refers to the control range of the current circuit breaker node, and the backup protection zone refers to the control range of the adjacent nodes of the current circuit breaker.

[0037] The training steps include simulation system training and actual system training;

[0038] The simulation system training first pre-learns the agent based on actual working conditions in the simulation system to accumulate corresponding experience knowledge;

[0039] The actual system training, based on pre-learning, puts the intelligent agent into the actual system for training and learning, so as to improve the efficiency and performance of reinforcement learning.

[0040] The step 2 comprises:

[0041] An autonomous decision is made on the acquired state according to the optimized deep reinforcement learning neural network model to obtain an action for controlling the circuit breaker; and an action execution instruction is output to the circuit breaker according to the action for controlling the circuit breaker.

[0042] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0043] The method proposed in the present invention does not need to calculate the set current and the step-by-step coordination time, and the protection control system can independently determine whether to act according to the current operating status of the distribution network; after the protection control system is connected to the network and operated, it does not require manual intervention and can adapt to changes in the operating status of the distribution network, including changes in the capacity and location of distributed power sources and different fault types and locations; it can be well applied to distribution network protection control scenarios containing distributed power sources. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a structural block diagram of a protection control system for a distribution network containing distributed power sources based on deep reinforcement learning in a specific implementation manner of the present invention;

[0045] Figure 2 It is a principle diagram of a dual reinforcement learning method based on simulation learning and reality learning in a specific implementation manner of the present invention;

[0046] Figure 3 This is a flow chart of a protection and control method for a distribution network containing distributed power sources based on deep reinforcement learning in a specific implementation manner of the present invention. DETAILED DESCRIPTION

[0047] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation method of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the invention, so the present invention is not limited by the specific implementation disclosed below.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0049] Distribution network protection control is a typical problem of solving the optimal control problem of model-free dynamic systems. Each relay protection device is regarded as an intelligent agent, which perceives the operating status of the distribution network through local measurement data (current, voltage, phase angle, frequency, switch status), and obtains rewards from the next state of the distribution network after performing an action (open or close). After continuous iterations, it converges to the optimal action strategy.

[0050] In practical applications, each agent only knows local measurement information, and assuming that there is no information exchange between agents, then each agent i does not know the exact location of the actions and faults of the downstream adjacent agents. Therefore, each agent needs a local control strategy π i , which transforms the local observation s i Mapped to control action a i , that is, a i =π i (s i ). The optimal relay protection problem in the network is described as a reinforcement learning problem. The goal is to maximize the cumulative rewards obtained by all agents. Let the control strategy of each agent be π i , which takes the local observation s i,t Mapped to local control action a i,t ,but

[0051]

[0052] Among them, γ is the discount factor, R is the reward value, and t is the current time of the system.

[0053] Figure 1 The structure block diagram of the protection control system of the distribution network containing distributed power sources based on deep reinforcement learning in the specific implementation mode of the present invention is as follows: Figure 1 As shown, the system comprises:

[0054] An intelligent agent unit is configured at each circuit breaker node in a distribution network containing distributed power sources. Each intelligent agent senses the operation status of the distribution network through local measurement data, outputs instructions to control the circuit breaker action and obtains rewards from the next state of the distribution network operation, and continuously iterates to converge on the optimal action strategy; the intelligent agent unit is mounted on a relay protection device, and includes a modeling optimization module and a real-time action determination module;

[0055] The modeling optimization module is used to take the detection signal of the relay protection device as input, train the deep learning neural network so that it can output a corresponding action signal according to the detection signal, and obtain an optimized deep learning neural network model;

[0056] The real-time action determination module is used to collect detection signals from the distribution network site in real time and input them into the optimized deep learning neural network model, so as to obtain the action signal output by the model and output it to the circuit breaker to execute the action instruction.

[0057] Furthermore, the modeling optimization module includes a signal detection module, an action space module, a reward function module, a deep learning neural network building module, and an algorithm training module;

[0058] The signal detection module is used to input instructions to control the relay protection device to collect environmental status information of the current node; the environmental status information includes current signals, voltage signals, phase angle measurement values, frequency measurement values ​​and the switch status of the corresponding circuit breaker;

[0059] The action space module is used to output the action type and parameters corresponding to the environmental state information according to the ε greedy rule, and the action type is the on / off action and action delay of the control circuit breaker;

[0060] Furthermore, the upper and lower limits of the action delay are set to [0.05s, 1.1s], with a discretization interval of 0.002s;

[0061] The reward function module is used to determine the reward rules in the process of interaction between the agent and the environment, specifically:

[0062]

[0063] The main protection zone refers to the control range of the current circuit breaker node, and the backup protection zone refers to the control range of the adjacent nodes of the current circuit breaker.

[0064] The deep learning neural network building module is used to use a deep neural network to approximate the Q function and build a deep learning neural network model DQN;

[0065] When the environmental state is discrete and finite, learning can be performed by training the Q value table. When the environment is continuous and the state space is large, such as using current, voltage and other measurement values ​​to perceive the operating state of the distribution network, this is a continuous analog quantity and it is impossible to store the state through a table. The state dimension must be compressed. The solution is to use a deep neural network to approximate the Q function, which is the DQN (Deep Q-Network) algorithm.

[0066] Q(s,a)≈Q ω (s,a)

[0067] Where ω is the parameter of the neural network.

[0068] In the DQN algorithm, the target Q value calculation formula in the non-terminal state is:

[0069]

[0070] Among them, R is the instant reward of the current state, γ is the discount factor, is the predicted value of the next state.

[0071] The algorithm training module is used to use the environmental state information obtained by the signal detection module as the input of the deep learning neural network, and the action space information obtained by the action space module as the output of the deep learning neural network, and train the deep learning neural network according to the reward rules set by the reward function module to obtain an optimized deep learning neural network model. The specific training process is as follows:

[0072] Step 1-1, initialize the deep learning neural network model parameters ω;

[0073] Step 1-2, observe the environment to obtain state S, and select action A according to the ε greedy rule; the environmental state information includes current signal, voltage signal, phase angle measurement value, frequency measurement value and the switch state of the corresponding circuit breaker; the action type is to control the opening and closing actions and action delay of the circuit breaker;

[0074] Step 1-3: Take action A, obtain the reward R given by the environment, and observe the new state S';

[0075] Step 1-4: store the obtained (S, A, R, S') into the experience pool D;

[0076] Step 1-5: Randomly select B (S, A, R, S') from D to form a set, and use the gradient descent method to update the parameters ω of the network model;

[0077] Steps 1-6: Through continuous iterative updates, the parameter ω eventually becomes stable, and the optimized deep learning neural network model DQN is obtained.

[0078] Furthermore, the algorithm training module includes simulation system training and actual system training;

[0079] Reinforcement learning is a spontaneous learning based on rewards. In the early stage of reinforcement learning, due to the lack of corresponding knowledge, random actions can easily lead to system instability, slow convergence and low training efficiency. In order to avoid this situation, based on the framework of traditional reinforcement learning, the efficiency and performance of reinforcement learning are improved by combining pre-learning in the simulation system with real-time learning in the actual system. Figure 2 As shown in the figure, the intelligent agent first performs pre-learning in the simulation system based on the actual working conditions for a sufficient number of iterations, seeks the optimal strategy through continuous interactive learning with the simulation system, accumulates corresponding experience and knowledge, and then is put into the actual system for training and learning.

[0080] The real-time action determination module includes an optimal action determination module and an action output module;

[0081] The optimal action determination module is used to make an autonomous decision on the state obtained by the signal detection module according to the optimized deep reinforcement learning neural network model to obtain an action to control the circuit breaker;

[0082] The action output module is used to output an action execution instruction to the circuit breaker according to the action of controlling the circuit breaker.

[0083] Figure 3 The flowchart of the protection control method of the distribution network containing distributed power sources based on deep reinforcement learning in the specific implementation mode of the present invention is as follows: Figure 3 As shown, an intelligent unit is configured at each circuit breaker node in a distribution network containing distributed power sources, which senses the operation status of the distribution network through local measurement data, outputs instructions to control the circuit breaker action and obtains rewards from the next state of the distribution network operation, and continuously iterates to converge on the optimal action strategy; the following steps are included:

[0084] Step 1: Taking the detection signal of the relay protection device as input, training the deep learning neural network so that it can output a corresponding action signal according to the detection signal, and obtaining an optimized deep learning neural network model;

[0085] Step 2: Collect the detection signals of the distribution network site in real time and input them into the optimized deep learning neural network model, so as to obtain the action signal output by the model and output it to the circuit breaker to execute the action instruction.

[0086] Furthermore, the step 1 uses the acquired environmental state information as the input of the deep learning neural network, uses the acquired action space information as the output of the deep learning neural network, trains the deep learning neural network according to the reward rule, and obtains an optimized deep learning neural network model; specifically includes:

[0087] Step 1-1, initialize the deep learning neural network model parameters ω;

[0088] Step 1-2, observe the environment to obtain state S, and select action A according to the ε greedy rule; the environmental state information includes current signal, voltage signal, phase angle measurement value, frequency measurement value and the switch state of the corresponding circuit breaker; the action type is to control the opening and closing actions and action delay of the circuit breaker;

[0089] Step 1-3: Take action A, obtain the reward R given by the environment, and observe the new state S';

[0090] Step 1-4: store the obtained (S, A, R, S') into the experience pool D;

[0091] Step 1-5: Randomly select B (S, A, R, S') from D to form a set, and use the gradient descent method to update the parameters ω of the network model;

[0092] Steps 1-6: Through continuous iterative updates, the parameter ω eventually becomes stable, and the optimized deep learning neural network model DQN is obtained.

[0093] The reward rules include:

[0094]

[0095] The main protection zone refers to the control range of the current circuit breaker node, and the backup protection zone refers to the control range of the adjacent nodes of the current circuit breaker.

[0096] The training steps include simulation system training and actual system training;

[0097] The simulation system training first pre-learns the agent based on actual working conditions in the simulation system to accumulate corresponding experience knowledge;

[0098] The actual system training, based on pre-learning, puts the intelligent agent into the actual system for training and learning, so as to improve the efficiency and performance of reinforcement learning.

[0099] Furthermore, step 2 includes:

[0100] An autonomous decision is made on the acquired state according to the optimized deep reinforcement learning neural network model to obtain an action for controlling the circuit breaker; and an action execution instruction is output to the circuit breaker according to the action for controlling the circuit breaker.

[0101] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any simple modification, change and equivalent structural change made to the above embodiment based on the technical essence of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A distribution network protection control system based on deep reinforcement learning, characterized in that: include: An intelligent agent unit is configured at each circuit breaker node in a distribution network containing distributed power sources. Each intelligent agent senses the operation status of the distribution network through local measurement data, outputs instructions to control the circuit breaker action and obtains rewards from the next state of the distribution network operation, and continuously iterates to converge on the optimal action strategy; the intelligent agent unit is mounted on a relay protection device, and includes a modeling optimization module and a real-time action determination module; The modeling optimization module is used to take the detection signal of the relay protection device as input, train the deep learning neural network so that it can output a corresponding action signal according to the detection signal, and obtain an optimized deep learning neural network model; The modeling and optimization module includes: a. Signal detection module, used to input instructions to control the relay protection device to collect the environmental status information of the current node; The environmental status information includes current signals, voltage signals, phase angle measurements, frequency measurements and the switch status of the corresponding circuit breaker; b. Action space module, used to output action types and parameters corresponding to environmental state information according to the ε greedy rule, the action types are the on and off actions and action delays of the control circuit breaker; c. Reward function module, which is used to determine the reward rules during the interaction between the agent and the environment; d. The deep learning neural network building module is used to use a deep neural network to approximate the Q function and build a deep learning neural network model DQN; e. an algorithm training module, which is used to use the environmental state information obtained by the signal detection module as the input of the deep learning neural network, and the action space information obtained by the action space module as the output of the deep learning neural network, and train the deep learning neural network according to the reward rule set by the reward function module to obtain an optimized deep learning neural network model; The real-time action determination module is used to collect detection signals from the distribution network site in real time and input them into the optimized deep learning neural network model, so as to obtain the action signal output by the model and output it to the circuit breaker to execute the action instruction.

2. The distribution network protection control system based on deep reinforcement learning according to claim 1 is characterized in that: The reward rules include: The main protection zone refers to the control range of the current circuit breaker node, and the backup protection zone refers to the control range of the adjacent nodes of the current circuit breaker.

3. The distribution network protection control system based on deep reinforcement learning according to claim 1 is characterized in that: The algorithm training module includes simulation system training and actual system training; The simulation system training is used to first pre-learn the agent in the simulation system based on actual working conditions to accumulate corresponding experience knowledge; The actual system training is used to put the intelligent agent into the actual system for training and learning based on pre-learning, so as to improve the efficiency and performance of reinforcement learning.

4. The distribution network protection control system based on deep reinforcement learning according to claim 1 is characterized in that: The real-time action determination module includes an optimal action determination module and an action output module; The optimal action determination module is used to make an autonomous decision on the state obtained by the signal detection module according to the optimized deep learning neural network model to obtain an action to control the circuit breaker; The action output module is used to output an action execution instruction to the circuit breaker according to the action of controlling the circuit breaker.

5. A distribution network protection control method based on deep reinforcement learning, characterized in that: An intelligent unit is configured at each circuit breaker node in a distribution network containing distributed power sources. The intelligent unit senses the operation status of the distribution network through local measurement data, outputs instructions to control the circuit breaker action, and obtains rewards from the next state of the distribution network operation. The intelligent unit continuously iterates and converges to the optimal action strategy. The intelligent unit includes the following steps: Step 1: Take the detection signal of the relay protection device as input, train the deep learning neural network so that it can output the corresponding action signal according to the detection signal, and obtain the optimized deep learning neural network model; the modeling optimization process includes: inputting instructions to control the relay protection device to collect the environmental state information of the current node; the environmental state information includes current signal, voltage signal, phase angle measurement value, frequency measurement value and the switch state of the corresponding circuit breaker; output the action type and parameters corresponding to the environmental state information according to the ε greedy rule, and the action type is to control the opening and closing actions and action delay of the circuit breaker; determine the reward rule in the interaction process between the intelligent agent and the environment; use the deep neural network to approximate the Q function and construct the deep learning neural network model DQN; use the environmental state information obtained by the signal detection module as the input of the deep learning neural network, and the action space information obtained by the action space module as the output of the deep learning neural network, and train the deep learning neural network according to the reward rule set by the reward function module to obtain the optimized deep learning neural network model; Step 2: Collect the detection signals of the distribution network site in real time and input them into the optimized deep learning neural network model, so as to obtain the action signals output by the model and output them to the circuit breaker to execute the action instructions.

6. The distribution network protection control method based on deep reinforcement learning according to claim 5 is characterized in that: The step 1 uses the acquired environmental state information as the input of the deep learning neural network, uses the acquired action space information as the output of the deep learning neural network, trains the deep learning neural network according to the reward rule, and obtains an optimized deep learning neural network model; specifically includes: Step 1-1, initialize the deep learning neural network model parameters ω; Step 1-2, observe the environment to obtain state S, and select action A according to the ε greedy rule; the environmental state information includes current signal, voltage signal, phase angle measurement value, frequency measurement value and the switch state of the corresponding circuit breaker; the action type is to control the opening and closing actions and action delay of the circuit breaker; Step 1-3: Take action A, obtain the reward R given by the environment, and observe the new state S'; Step 1-4: store the obtained (S, A, R, S') into the experience pool D; Step 1-5: Randomly select B (S, A, R, S') from D to form a set, and use the gradient descent method to update the parameters ω of the network model; Steps 1-6: Through continuous iterative updates, the parameter ω eventually becomes stable, and the optimized deep learning neural network model DQN is obtained.

7. The distribution network protection control method based on deep reinforcement learning according to claim 6 is characterized in that: The reward rules include: The main protection zone refers to the control range of the current circuit breaker node, and the backup protection zone refers to the control range of the adjacent nodes of the current circuit breaker.

8. The distribution network protection control method based on deep reinforcement learning according to claim 5 is characterized in that: The training steps include simulation system training and actual system training; The simulation system training first pre-learns the agent based on actual working conditions in the simulation system to accumulate corresponding experience knowledge; The actual system training, based on pre-learning, puts the intelligent agent into the actual system for training and learning, so as to improve the efficiency and performance of reinforcement learning.

9. The distribution network protection control method based on deep reinforcement learning according to claim 5 is characterized in that: The step 2 comprises: An autonomous decision is made on the acquired state according to the optimized deep learning neural network model to obtain an action for controlling the circuit breaker; and an action execution instruction is output to the circuit breaker according to the action for controlling the circuit breaker.

Images