Method and system for automatically adjusting section power based on deep reinforcement learning

Abstract

The invention provides an automatic section power adjustment method and system based on deep reinforcement learning, and the method comprises the steps: obtaining the current state information of a power grid, the current state information comprises the number of a to-be-adjusted section, the current power of the to-be-adjusted section, the target power of the to-be-adjusted section, the current active output of each generator, the maximum active output of each generator and the minimum active output of each generator; the current state information is input to a preset intelligent agent, so that the intelligent agent makes a decision according to the current state information and outputs a corresponding action instruction group, and the intelligent agent is obtained through training based on a deep reinforcement learning algorithm; and adjusting the power of each generator according to the action instruction group. The power grid section power is automatically adjusted in combination with the deep reinforcement learning technology, and the accuracy of automatic power adjustment is improved.

Description

Technical Field

[0001] The present application relates to the technical field of artificial intelligence and power grid power control, and in particular to a method and system for automatic section power adjustment based on deep reinforcement learning. Background Art

[0002] The existing power grid section power adjustment technology plays an important role in the calculation of power grid operation mode and safety and stability analysis. With the development of power systems dominated by new energy, inter-regional energy transmission has become more frequent, which has brought greater pressure to dispatchers. Therefore, the research demand for automatic section power adjustment is becoming increasingly urgent.

[0003] In the current power grid power flow calculation, the methods for controlling section power mainly include the traditional power flow section constraint method, sensitivity analysis method and optimization control method. The traditional constraint method adjusts by adding section power constraints, but its calculation speed is slow, so it requires multiple iterations. The sensitivity analysis law focuses on the selection and calculation of sensitivity indicators, but in actual work, it does not pay enough attention to the determination of the adjustment amount. In addition, some literature uses nonlinear programming to solve the optimization control problem, but in large-scale interconnected power grids, convergence cannot be guaranteed and the calculation takes a long time. With the gradual increase in the proportion of new energy in the power system, cross-regional energy transmission will become more frequent, and traditional control methods may be difficult to meet the demand. In actual operation, simulation software is often used to assist in section power adjustment. Although it can ensure the balance between power generation and load, the process is complicated, the workload is large, and it relies on expert experience. When the section power is significantly different from the set value, the power flow calculation may not converge. Summary of the invention

[0004] In response to the above technical problems, the present application provides a method and system for automatic section power adjustment based on deep reinforcement learning, which combines deep reinforcement learning technology to realize automatic adjustment of power section power of the power grid and improve the accuracy of automatic power adjustment.

[0005] In a first aspect, an embodiment of the present application provides a method for automatic section power adjustment based on deep reinforcement learning, comprising:

[0006] Acquire current state information of the power grid, the current state information including the number of the section to be adjusted, the current power of the section to be adjusted, the target power of the section to be adjusted, the current active output of each generator, the maximum active output of each generator, and the minimum active output of each generator;

[0007] Inputting the current state information into a preset intelligent agent, so that the intelligent agent makes a decision according to the current state information, and then outputs a corresponding action instruction group, wherein the intelligent agent is obtained by training based on a deep reinforcement learning algorithm;

[0008] The power of each generator is adjusted according to the action instruction group.

[0009] In a possible implementation, the agent makes a decision based on the current state information and then outputs a corresponding action instruction group, including:

[0010] Calculating a first sensitivity index, a second sensitivity index, and a third sensitivity index of each generator according to the current state information and a preset sensitivity calculation method;

[0011] According to the current state information, the first sensitivity index, the second sensitivity index and the third sensitivity index of each generator, power adjustment screening and power compensation screening are performed on each generator to obtain a forward power adjustment generator set, a reverse power adjustment generator set and a power compensation generator set;

[0012] Generate a power adjustment instruction group and a power compensation instruction group according to the current state information, the forward power adjustment generator set, the reverse power adjustment generator set and the power compensation generator set;

[0013] The power adjustment instruction group and the power compensation instruction group are combined to output the action instruction group.

[0014] Further, the first sensitivity index, the second sensitivity index and the third sensitivity index of each generator are calculated according to the current state information and a preset sensitivity calculation method, including:

[0015] Calculate a first sensitivity index and a second sensitivity index of each generator according to the current power of the section to be adjusted, the maximum active output of each generator, and the minimum active output of each generator;

[0016] A third sensitivity index of each generator is calculated according to the sum of the first sensitivity index and the second sensitivity index.

[0017] Further, the power adjustment screening and power compensation screening are performed on each generator according to the current state information, the first sensitivity index, the second sensitivity index and the third sensitivity index of each generator to obtain a forward power adjustment generator set, a reverse power adjustment generator set and a power compensation generator set, including:

[0018] Perform forward power adjustment screening on each generator according to the current state information and each first sensitivity index to obtain a corresponding forward power adjustment generator set;

[0019] Perform reverse power adjustment screening on each generator according to the current state information and each second sensitivity index to obtain a corresponding reverse power adjustment generator set;

[0020] Calculating and obtaining a current power fluctuation boundary of a power grid according to the forward power adjustment generator set and the reverse power adjustment generator set;

[0021] Power compensation screening is performed on each generator according to the preset constraint conditions, the third sensitivity index and the current power fluctuation boundary to obtain a corresponding power compensation generator set.

[0022] Further, generating a power adjustment instruction group and a power compensation instruction group according to the current state information, the forward power adjustment generator set, the reverse power adjustment generator set and the power compensation generator set includes:

[0023] According to the relative size of the current power of the section to be adjusted and the target power of the section to be adjusted, determining the current power adjustment mode as a forward power adjustment mode or a reverse power adjustment mode;

[0024] If the current power adjustment mode is the forward power adjustment mode, the action value range of the agent is divided into n sub-intervals according to the upper and lower boundaries of the action space of the agent, and each sub-interval is matched one by one with each generator in the forward power adjustment generator set, where n is the number of generators in the forward power adjustment generator set;

[0025] If the current power adjustment mode is the reverse power adjustment mode, the action value range of the agent is divided into n sub-intervals according to the upper and lower boundaries of the action space of the agent, and each sub-interval is matched one by one with each generator in the reverse power adjustment generator set, where n is the number of generators in the reverse power adjustment generator set;

[0026] Inputting the current state information into a preset deep reinforcement learning network so that the deep reinforcement learning network outputs a number of action values;

[0027] Mapping each action value to each corresponding generator through the corresponding sub-interval, thereby generating the power adjustment instruction group;

[0028] The power compensation action of each generator in the power compensation generator set is determined according to the respective action values, and then the power compensation instruction group is generated.

[0029] In a possible implementation, the step of obtaining the intelligent agent based on deep reinforcement learning algorithm training includes:

[0030] According to the Markov decision process and the deep reinforcement learning algorithm, an initial intelligent agent is constructed, wherein the initial intelligent agent includes an initial neural network, a state space, an action space, a reward function and a discount factor, and the initial neural network includes an Actor network and a Critic network;

[0031] Inputting a plurality of training data into the initial agent in sequence, and performing a plurality of rounds of training, wherein in each round, the initial agent continuously makes decisions according to the current training data, and combines the state, action and reward feedback corresponding to each decision into experience data, and stores each experience data in a preset experience pool, until the initial agent meets the preset conditions and completes the current round of training;

[0032] During the training of the initial agent, if the experience pool reaches a preset capacity, the training is suspended, the experience data in the experience pool is taken out, and the network parameters of the initial neural network are updated according to the experience data;

[0033] During the training process of the initial intelligent agent, if the number of decision steps of the initial intelligent agent reaches a maximum preset number of steps, the training is terminated to obtain the intelligent agent.

[0034] The embodiment of the present application provides a method for automatic section power adjustment based on deep reinforcement learning, which uses deep reinforcement learning algorithm training to obtain an intelligent agent, which makes decisions based on the current state information of the power grid and outputs corresponding action instructions, and then adjusts the power of each generator according to the action instruction group to adjust the power of the target section to the target power. The method provided by the embodiment of the present application has higher accuracy than the existing power flow section constraint method, sensitivity analysis method and optimization control method due to the combination of artificial intelligence technology, and the intelligent agent used is obtained by pre-training through a deep reinforcement learning algorithm, does not rely on expert experience in the actual decision-making process, and can realize automatic and efficient power grid section power adjustment.

[0035] In a second aspect, an embodiment of the present application provides a section power automatic adjustment system based on deep reinforcement learning, including an acquisition module, a decision module and a power adjustment module;

[0036] The acquisition module is used to acquire the current state information of the power grid, and the current state information includes the number of the section to be adjusted, the current power of the section to be adjusted, the target power of the section to be adjusted, the current active output of each generator, the maximum active output of each generator, and the minimum active output of each generator;

[0037] The decision module is used to input the current state information into a preset intelligent agent, so that the intelligent agent makes a decision according to the current state information, and then outputs a corresponding action instruction group, wherein the intelligent agent is obtained based on deep reinforcement learning algorithm training;

[0038] The power adjustment module is used to adjust the power of each generator according to the action instruction group.

[0039] In one possible implementation, the decision module includes a sensitivity calculation unit, a generator screening unit, an instruction generation unit, and an output unit;

[0040] The sensitivity calculation unit is used to calculate the first sensitivity index, the second sensitivity index and the third sensitivity index of each generator according to the current state information and a preset sensitivity calculation method;

[0041] The generator screening unit is used to perform power adjustment screening and power compensation screening on each generator according to the current state information, the first sensitivity index, the second sensitivity index and the third sensitivity index of each generator, to obtain a forward power adjustment generator set, a reverse power adjustment generator set and a power compensation generator set;

[0042] The instruction generation unit is used to generate a power adjustment instruction group and a power compensation instruction group according to the current state information, the forward power adjustment generator set, the reverse power adjustment generator set and the power compensation generator set;

[0043] The output unit is used to combine the power adjustment instruction group and the power compensation instruction group to output the action instruction group.

[0044] Further, the instruction generation unit generates a power adjustment instruction group and a power compensation instruction group according to the current state information, the forward power adjustment generator set, the reverse power adjustment generator set and the power compensation generator set, including:

[0045] According to the relative size of the current power of the section to be adjusted and the target power of the section to be adjusted, determining the current power adjustment mode as a forward power adjustment mode or a reverse power adjustment mode;

[0046] If the current power adjustment mode is the forward power adjustment mode, the action value range of the agent is divided into n sub-intervals according to the upper and lower boundaries of the action space of the agent, and each sub-interval is matched one by one with each generator in the forward power adjustment generator set, where n is the number of generators in the forward power adjustment generator set;

[0047] If the current power adjustment mode is the reverse power adjustment mode, the action value range of the agent is divided into n sub-intervals according to the upper and lower boundaries of the action space of the agent, and each sub-interval is matched one by one with each generator in the reverse power adjustment generator set, where n is the number of generators in the reverse power adjustment generator set;

[0048] Inputting the current state information into a preset deep reinforcement learning network so that the deep reinforcement learning network outputs a number of action values;

[0049] Mapping each action value to each corresponding generator through the corresponding sub-interval, thereby generating the power adjustment instruction group;

[0050] The power compensation action of each generator in the power compensation generator set is determined according to the respective action values, and then the power compensation instruction group is generated.

[0051] In a possible implementation, the step of obtaining the intelligent agent based on deep reinforcement learning algorithm training includes:

[0052] According to the Markov decision process and the deep reinforcement learning algorithm, an initial intelligent agent is constructed, wherein the initial intelligent agent includes an initial neural network, a state space, an action space, a reward function and a discount factor, and the initial neural network includes an Actor network and a Critic network;

[0053] Inputting a plurality of training data into the initial agent in sequence, and performing a plurality of rounds of training, wherein in each round, the initial agent continuously makes decisions according to the current training data, and combines the state, action and reward feedback corresponding to each decision into experience data, and stores each experience data in a preset experience pool, until the initial agent meets the preset conditions and completes the current round of training;

[0054] During the training of the initial agent, if the experience pool reaches a preset capacity, the training is suspended, the experience data in the experience pool is taken out, and the network parameters of the initial neural network are updated according to the experience data;

[0055] During the training process of the initial intelligent agent, if the number of decision steps of the initial intelligent agent reaches a maximum preset number of steps, the training is terminated to obtain the intelligent agent. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 : A flow chart of a method for automatic section power adjustment based on deep reinforcement learning provided in an embodiment of the present application.

[0057] Figure 2: A flow chart of intelligent output action instructions in a method for automatic section power adjustment based on deep reinforcement learning provided in an embodiment of the present application.

[0058] Figure 3 : Schematic diagram of the deep neural network structure of the intelligent agent in the automatic section power adjustment method based on deep reinforcement learning provided in an embodiment of the present application.

[0059] Figure 4 : Schematic diagram of the training process of the initial intelligent agent in a method for automatic section power adjustment based on deep reinforcement learning provided in an embodiment of the present application.

[0060] Figure 5 : A structural diagram of multi-threaded parallel training of the initial intelligent agent in a method for automatic section power adjustment based on deep reinforcement learning provided in an embodiment of the present application.

[0061] Figure 6 : A schematic diagram of the system structure for conducting a simulation experiment in a method for automatic section power adjustment based on deep reinforcement learning provided in an embodiment of the present application.

[0062] Figure 7 : A structural diagram of a section power automatic adjustment system based on deep reinforcement learning provided in an embodiment of the present application.

[0063] Figure 8 : A structural diagram of a decision-making module in a section power automatic adjustment system based on deep reinforcement learning provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0065] It should be noted that the step numbers in the text are only for the convenience of explanation of the specific embodiments and do not serve to limit the order in which the steps are executed. In the description of this application, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0066] Embodiment 1:

[0067] like Figure 1As shown, embodiment 1 provides a method for automatic section power adjustment based on deep reinforcement learning, including steps S1-S3:

[0068] Step S1, obtaining current state information of the power grid, wherein the current state information includes the number of the section to be adjusted, the current power of the section to be adjusted, the target power of the section to be adjusted, the current active output of each generator, the maximum active output of each generator, and the minimum active output of each generator;

[0069] Step S2, inputting the current state information into a preset intelligent agent, so that the intelligent agent makes a decision according to the current state information, and then outputs a corresponding action instruction group, wherein the intelligent agent is obtained by training based on a deep reinforcement learning algorithm;

[0070] Step S3: adjusting the power of each generator according to the action instruction group.

[0071] The embodiment of the present application provides a method for automatic section power adjustment based on deep reinforcement learning, which uses deep reinforcement learning algorithm training to obtain an intelligent agent, which makes decisions based on the current state information of the power grid and outputs corresponding action instructions, and then adjusts the power of each generator according to the action instruction group to adjust the power of the target section to the target power. The method provided by the embodiment of the present application has higher accuracy than the existing power flow section constraint method, sensitivity analysis method and optimization control method due to the combination of artificial intelligence technology, and the intelligent agent used is obtained by pre-training through a deep reinforcement learning algorithm, does not rely on expert experience in the actual decision-making process, and can realize automatic and efficient power grid section power adjustment.

[0072] In a possible implementation, in step S2, the agent makes a decision based on the current state information, and then outputs a corresponding action instruction group, such as Figure 2 As shown, steps S201-S204 are included:

[0073] Step S201, calculating a first sensitivity index, a second sensitivity index, and a third sensitivity index of each generator according to the current state information and a preset sensitivity calculation method;

[0074] Step S202: Perform power adjustment screening and power compensation screening on each generator according to the current state information, the first sensitivity index, the second sensitivity index, and the third sensitivity index of each generator, to obtain a forward power adjustment generator set, a reverse power adjustment generator set, and a power compensation generator set;

[0075] Step S203, generating a power adjustment instruction group and a power compensation instruction group according to the current state information, the forward power adjustment generator set, the reverse power adjustment generator set and the power compensation generator set;

[0076] Step S204: Combine the power adjustment instruction group and the power compensation instruction group to output the action instruction group.

[0077] The embodiment of the present application provides a method for an intelligent agent to output an action instruction group according to current state information. First, the first sensitivity index, the second sensitivity index and the third sensitivity index of each generator are calculated respectively, and then each generator is screened according to the sensitivity index to determine the forward power adjustment generator set, the reverse power adjustment generator set and the power compensation generator set. In the process of power adjustment by the intelligent agent, the generator with high sensitivity is usually adjusted first. At the same time, in order to maintain the balance between power generation and load, the active output of the generator with low sensitivity is also adjusted accordingly. Based on this operating experience, the embodiment of the present application introduces a generator screening and power compensation mechanism when constructing the intelligent agent. Specifically, for the section to be adjusted, the sensitivity index of each generator is first calculated and sorted, and the unit with high sensitivity is used as the adjustment object, and the unit with low sensitivity is used as the power compensation unit to balance the imbalance caused by adjusting the active power of the unit with high sensitivity. This strategy can significantly reduce the action space of the intelligent agent, thereby reducing the possibility of non-convergence of the flow calculation or over-limit of the balancing machine during the training process, and improving the efficiency and accuracy of automatic adjustment of the section power.

[0078] Further, in step S201, the first sensitivity index, the second sensitivity index and the third sensitivity index of each generator are calculated according to the current state information and a preset sensitivity calculation method, including:

[0079] Calculate a first sensitivity index and a second sensitivity index of each generator according to the current power of the section to be adjusted, the maximum active output of each generator, and the minimum active output of each generator;

[0080] A third sensitivity index of each generator is calculated according to the sum of the first sensitivity index and the second sensitivity index.

[0081] In a preferred embodiment, the calculation formulas of the first sensitivity index, the second sensitivity index and the third sensitivity index are:

[0082]

[0083] in, and The sensitivity indexes to be referred to by the generator when the transmission power of a given section needs to be adjusted in the forward and reverse directions are selected, namely, the first sensitivity index and the second sensitivity index, and the generator set with the larger sensitivity value is selected first; This is the sensitivity index that should be referred to when selecting a power compensation unit, that is, the third sensitivity index. Units with a small sensitivity value are preferably selected. and are the transmission power of a given section when the active output of generator k is adjusted to the maximum and minimum values, respectively, init is the current power of the section to be adjusted, and Ω is the set of adjustable generators remaining in the power grid except the balancing machine. Different from the definition of traditional sensitivity, the sensitivity used in the embodiment of the present application is not calculated by the change in section power when the generator output changes by 1 unit, but is determined by the change in section power when the generator output changes from the current value to the maximum or minimum value. When the active power of generator k is adjusted to calculate the sensitivity, the active power imbalance generated is not borne by the balancing machine, but a specific generator is selected for power compensation, and the power imbalance caused by the adjustment of generator k is balanced by automatically adjusting its active power.

[0084] Further, in step S202, the power adjustment screening and power compensation screening are performed on each generator according to the current state information, the first sensitivity index, the second sensitivity index and the third sensitivity index of each generator, to obtain a forward power adjustment generator set, a reverse power adjustment generator set and a power compensation generator set, including:

[0085] Perform forward power adjustment screening on each generator according to the current state information and each first sensitivity index to obtain a corresponding forward power adjustment generator set;

[0086] Perform reverse power adjustment screening on each generator according to the current state information and each second sensitivity index to obtain a corresponding reverse power adjustment generator set;

[0087] Calculating and obtaining a current power fluctuation boundary of a power grid according to the forward power adjustment generator set and the reverse power adjustment generator set;

[0088] Power compensation screening is performed on each generator according to the preset constraint conditions, the third sensitivity index and the current power fluctuation boundary to obtain a corresponding power compensation generator set.

[0089] In an embodiment of the present application, based on current state information, a first sensitivity index is used to determine a forward power adjustment generator set, a second sensitivity index is used to determine a reverse power adjustment generator set, and the current power fluctuation boundary of the power grid is calculated based on the forward power adjustment generator set and the reverse power adjustment generator set, and then a power compensation generator set is screened out in combination with the third sensitivity and preset constraints. By introducing generator power adjustment screening and power compensation screening, the embodiment of the present application can greatly reduce the action space of the intelligent agent, reduce the probability of the balance machine output exceeding the limit and the power flow not converging, ensure that the intelligent agent can quickly accumulate effective samples in the process of exploring the environment, thereby ensuring the convergence of the model, improving the model training speed and the decision-making speed in the model application process, and improving the efficiency and accuracy of the automatic adjustment of the section power.

[0090] In a preferred embodiment, the specific implementation process of step S202 is:

[0091] After calculating the three sensitivity indicators of each generator to a given section, the generator will be and Sort from large to small to form a generator set Ψ for positive and negative adjustment of section power pos and neg At the same time, according to Sort from small to large to get the generator set Ψ ban , used to perform power compensation. Then, according to the target power P of the given section tar , select the minimum number of generators required to adjust the section power to the target value through the following formula, where P tar ≥P init When the positive power adjustment generator set ψ pos , which is used to positively adjust the section power to the target value; otherwise, a reverse power adjustment generator set ψ is formed neg :

[0092] n=minN

[0093]

[0094] Where: ρ is the margin coefficient when screening the generators to be adjusted, ρ≥1; N is the number of generators that meet the constraints of the above formula, and n is the number of generators to be adjusted.

[0095] Then, according to the forward power adjustment generator set and the reverse power adjustment generator set, the current power fluctuation boundary of the power grid is calculated:

[0096]

[0097]

[0098] Among them, n pos and n neg The forward power adjustment generator set ψ pos and reverse power adjustment generator set ψ neg The number of generators in ΔP max and ΔP min are the upper and lower boundaries of the current power fluctuation of the power grid, and are the maximum and minimum active output of generator k, P G，k is the current active output of generator k.

[0099] Calculate the upper limit of power fluctuation ΔP max , lower boundary ΔP min Then, from the set Ψ ban Select the power compensation generator set ψ ban To compensate power, the power compensation generator set ψ ban Divided into 3 subsets ψ a , b , c ψ a The generator in can be used for both positive compensation power and reverse compensation power; b The generator in is only used for forward compensation power; c The generator in is only used for reverse compensation power. While satisfying the constraints, the optimization objective function can be obtained from the set Ψ ban The power compensation generator set ψ for power compensation is obtained from ban .

[0100] The objective function is:

[0101]

[0102] The constraints are:

[0103]

[0104] Among them, λ is the screening threshold of the power compensation unit, and the purpose is to select the generator with a larger adjustable margin as the compensation unit.

[0105] Further, in step S203, generating a power adjustment instruction group and a power compensation instruction group according to the current state information, the forward power adjustment generator set, the reverse power adjustment generator set and the power compensation generator set includes:

[0106] According to the relative size of the current power of the section to be adjusted and the target power of the section to be adjusted, determining the current power adjustment mode as a forward power adjustment mode or a reverse power adjustment mode;

[0107] If the current power adjustment mode is the forward power adjustment mode, the action value range of the agent is divided into n sub-intervals according to the upper and lower boundaries of the action space of the agent, and each sub-interval is matched one by one with each generator in the forward power adjustment generator set, where n is the number of generators in the forward power adjustment generator set;

[0108] If the current power adjustment mode is the reverse power adjustment mode, the action value range of the agent is divided into n sub-intervals according to the upper and lower boundaries of the action space of the agent, and each sub-interval is matched one by one with each generator in the reverse power adjustment generator set, where n is the number of generators in the reverse power adjustment generator set;

[0109] Inputting the current state information into a preset deep reinforcement learning network so that the deep reinforcement learning network outputs a number of action values;

[0110] Mapping each action value to each corresponding generator through the corresponding sub-interval, thereby generating the power adjustment instruction group;

[0111] The power compensation action of each generator in the power compensation generator set is determined according to the respective action values, and then the power compensation instruction group is generated.

[0112] In an embodiment of the present application, the action value range of the agent is divided into n sub-intervals, and each sub-interval is matched one-to-one with each generator in the power adjustment generator set, so as to construct a mapping relationship between the action space of the agent and the adjustable active output space of the generator. Each time the deep reinforcement learning network generates an action value, the action value will fall into the corresponding sub-interval, and then correspond to a certain generator, so as to realize the mapping from the action value to the power adjustment value of the generator, and finally obtain the power adjustment instruction of each generator. At the same time, when the output of the generator changes, in order to avoid excessive power fluctuations in the system causing the balancing machine to exceed the limit, a power compensation operation must be performed, and the power compensation action of each generator in the power compensation generator set is determined according to each action value, and a power compensation instruction group is generated to reduce the power fluctuation in the power system and improve the system stability of automatic section power adjustment.

[0113] In a preferred embodiment, the specific implementation process of step S203 is:

[0114] After determining the current power adjustment mode, the value range of the action value a is divided into n sub-intervals [a i , ai+1 ), i = 1, 2, ..., n, the selected generator is determined according to the subinterval into which the action value a falls, where a i and a i+1 are the lower and upper boundaries of the ith subinterval, respectively.

[0115] When P tar ≥P init When , that is, when the current power adjustment mode is the forward power adjustment mode, the sub-interval division rule is:

[0116]

[0117] When P tar <P init When the current power adjustment mode is the reverse power adjustment mode, the sub-interval division rule is:

[0118]

[0119] Among them, a max and a min are constants, which are the upper and lower boundaries of the agent's action space, i = 1, 2, ..., n; a1 = a min .

[0120] At time t, when P tar ≥P init When a certain action value a of the agent is t Mapping to the set ψ pos Active output of generator k:

[0121]

[0122] On the contrary, when P tar <P init When a certain action value a of the agent is t Mapping to the set ψ neg Active output of generator k:

[0123]

[0124] Among them, the value of k depends on the action value a t The interval [a i , a i+1 ) OK, P G，k，t is the active output of generator k at time t, P G，k is the current active output of generator k, and are the maximum and minimum active output of generator k respectively.

[0125] According to the above method, each action value generated by the intelligent agent is mapped to the corresponding generator through each corresponding sub-interval, thereby generating the power adjustment instruction group. At the same time, the power compensation action of each generator in the power compensation generator set is determined according to the each action value, thereby generating the power compensation instruction group.

[0126] In a possible implementation, the step of obtaining the intelligent agent based on deep reinforcement learning algorithm training includes:

[0127] According to the Markov decision process and the deep reinforcement learning algorithm, an initial intelligent agent is constructed, wherein the initial intelligent agent includes an initial neural network, a state space, an action space, a reward function and a discount factor, and the initial neural network includes an Actor network and a Critic network;

[0128] Inputting a plurality of training data into the initial agent in sequence, and performing a plurality of rounds of training, wherein in each round, the initial agent continuously makes decisions according to the current training data, and combines the state, action and reward feedback corresponding to each decision into experience data, and stores each experience data in a preset experience pool, until the initial agent meets the preset conditions and completes the current round of training;

[0129] During the training of the initial agent, if the experience pool reaches a preset capacity, the training is suspended, the experience data in the experience pool is taken out, and the network parameters of the initial neural network are updated according to the experience data;

[0130] During the training process of the initial intelligent agent, if the number of decision steps of the initial intelligent agent reaches a maximum preset number of steps, the training is terminated to obtain the intelligent agent.

[0131] The embodiment of the present application provides a method for training an intelligent agent, constructing an initial intelligent agent according to a Markov decision process and a deep reinforcement learning algorithm, and sequentially inputting a number of training data into the initial intelligent agent for several rounds of training. During the training process, the embodiment of the present application uses the information generated by the interaction between the intelligent agent and the environment to form experience data and store it in an experience pool, and then updates the network parameters of the initial neural network according to the experience data, ensuring that the decision-making ability of the intelligent agent can be stably improved, and improving the data utilization rate and training efficiency during the training process.

[0132] In a preferred embodiment, the embodiment of the present application performs environmental modeling of the power system according to the requirements of the Markov decision process quintuple (S, A, P, R, γ), and constructs an initial intelligent agent in combination with the DDPG algorithm. The environmental modeling of the power system generally focuses on the setting of state (S), action (A), reward (R), and discount factor (γ).

[0133] The state quantity is an abstract representation of the current environmental state, which is used to guide the decision-making of the intelligent agent. The environmental states must be different and can guide the intelligent agent to make appropriate actions. It needs to contain information about the task to be completed by the intelligent agent. The formula of the state quantity is as follows, which includes the number of the section to be adjusted, the target power, and the active power output of each generator set.

[0134]

[0135] In the formula, m represents the number of the section to be adjusted: represents the transmission power target setting value of the power section m to be adjusted; Represents the active power output value of generator set k; K represents the total number of generator sets in the system excluding balancing machines.

[0136] The action value corresponds to the decision of the agent in the current state, that is, the active output adjustment space of each generator set. The DDPG algorithm outputs continuous action values ​​and limits the action space to a reasonable range through specific mapping.

[0137] a t =[a min , a max ]

[0138] In the formula, a min , a max are the lower and upper limits of the action value space respectively. During the execution of the action, the value is mapped to the adjustable space of the generator active output.

[0139] From the definition of state quantity, we know that the next environment state is only determined by the current environment state and the action executed, which satisfies the definition of Markov decision process.

[0140]

[0141] In the above formula, is a state-to-action pair (s t , a t ) to the next state s t+1 A mapping of the probability distribution of .

[0142] The difficulty of reinforcement learning problems increases with the complexity of the environment. When the power system becomes more complex, the design of a reward function that enables the agent to learn the optimal strategy becomes more difficult and necessary. The reward function is used to evaluate whether the actions taken by the agent are effective. The embodiment of the present application designs a continuous reward function, which rewards actions that make the section power closer to the target value based on the adjustment result of the section power.

[0143]

[0144] In the formula, p m Represents the transmission power value of section m after the power adjustment in step t. This reward function ensures that the agent can obtain reward feedback at every step of interaction with the environment, which can avoid the situation where the reward is too sparse and the training is difficult to converge. After power adjustment, if the section power is closer to the target value, the greater the reward value, and the farther away, the smaller the reward value.

[0145] In addition, during the training of the agent, it is also necessary to set the round termination conditions. In the power system, a round means that the agent starts from the initial state and continuously adjusts the active power of the generator set in a certain section until the round termination conditions are met. These conditions include that the section power reaches the predetermined target, the power flow calculation converges, and the active power of the balancing machine does not exceed the limit.

[0146] In the embodiment of the present application, the deep neural network in the initial agent generally adopts a fully connected neural network, and the network structure is as follows: Figure 3 As shown in the figure. The number of nodes in the input layer is determined by the number of generators in the power grid, and the input data is a two-dimensional vector. The number of hidden layer networks and nodes shared by the Actor network and the Critic network are affected by the size of the power grid structure, and the hidden layer network structures used by power grids of different sizes are set differently. The branch parts of the Actor network and the Critic network are also affected by the size of the power grid structure, and the hidden layer activation functions are both hyperbolic tangent functions (Tanh); the output layers of the Actor network and the Critic network are both single nodes. The loss function optimization method uses the Adam optimization algorithm.

[0147] In a preferred embodiment, the training process of the initial agent based on the DDPG algorithm is as follows: Figure 4 As shown. First, select appropriate training parameters, initialize the neural network and experience pool, set the maximum capacity of the experience pool to C and the maximum number of steps in the entire training process to T. Next, the agent observes the state of the environment, performs actions, and obtains reward feedback, while storing these experience data in the experience pool. When the training reaches a certain termination condition, that is, the error between the section power and the target value meets the accuracy requirements, the power flow calculation converges, and the active output of the balancing machine does not exceed the limit, the round ends.

[0148] Furthermore, the DDPG algorithm can use multi-threaded parallel methods to improve the model training speed. The specific implementation method is as follows Figure 5As shown in the figure, the agents in multiple threads explore their own copies of the environment without interfering with each other and accumulate sample data. All sub-agents do not use their own sample data to update their own strategies. The data is collected centrally and extracted in batches to update the global network parameters. The sub-agent obtains new network parameters from the global network and updates its own strategy (no environmental exploration is performed during the update process). After the update, the sub-thread continues to collect sample data using the latest strategy. During the training process, it is necessary to continuously check whether the current number of steps has reached the update cycle C of the neural network. If it has reached it, all experiences are taken from the experience pool, and the neural network parameters are updated using the small batch stochastic gradient descent method or the Adam algorithm; otherwise, continue to explore the environment and accumulate experience data. Finally, if the current number of steps reaches the maximum number of training steps T, the entire training process is terminated, otherwise the experience accumulation and network update steps are repeated.

[0149] Furthermore, in order to verify the effectiveness of the method provided in the embodiment of the present application, the embodiment of the present application conducted a simulation experiment based on the IEEE-39 node power grid example. The settings of the algorithm and model training parameters in the simulation experiment are shown in Table 1:

[0150] Table 1 Algorithm and model training parameter settings

[0151]

[0152] like Figure 6 As shown, there are 10 generators in the IEEE39-node system, of which generator No. 10 is a balancing machine, and the rated power of all generators is set to 1100MW. Figure 6 The two sections marked by green and orange dashed lines are used to train and test the model. Section 1 covers three tie lines (21, 16), (24, 16) and (19, 16), while Section 2 has only one tie line (3, 4). The direction of the dashed line represents the direction of power transmission. The initial power value of Section 1 is 828MW, and its adjustable range is between 200 and 1400MW. The initial power value of Section 2 is 37MW, and its adjustable range is -200 to 400MW. When the adjusted section power is within the range of ±10MW of the target value, it is considered qualified.

[0153] The parameter settings of the fully connected neural network are as follows: the Actor network and the Critic network share a two-layer hidden layer neural network, with 700 nodes in the first layer and 500 nodes in the second layer; the independent parts of the Actor network and the Critic network also use a two-layer hidden layer neural network, with 500 nodes in the first layer and 200 nodes in the second layer; the input layer has a total of 11 nodes; and the output layer has 1 node.

[0154] During the model training process, the target section and target power value are randomly generated for each round of interaction between the agent and the environment. A target section is randomly selected, and the target power value is randomly generated within the corresponding power adjustable range according to the target section. By regularly checking the success rate of the model's adjustment strategy at a fixed number of target power values, the convergence status of the model training can be checked. Experiments have shown that the model with generator screening and power compensation mechanisms can gradually converge with the increase in the number of parameter updates. When the number of parameter updates reaches 800, the test accuracy reaches 100%. On the contrary, the model without generator screening and power compensation mechanisms did not converge under the same number of parameter updates, and the test accuracy did not improve significantly. The experimental results show that adding generator screening and power compensation mechanisms can effectively reduce the difficulty of model training and shorten the training time.

[0155] After the DDPG algorithm model training converged, the test results on sections 1 and 2 were obtained. After analysis, it was found that within the range of section power adjustment, the model can give the correct adjustment strategy for any given target power value, and the adjustment results meet the requirements of error accuracy. At the same time, under different sections and different target power values, the active output of the balancing machine in the system can be maintained within the rated range, and the overall power flow of the system has not changed significantly. It can be seen that the proposed method can automatically and continuously adjust the section power by training the model only given the target section and target power value.

[0156] The training scheme of the power section power automatic adjustment model was planned as a whole. The model parameters were set, and finally the effectiveness of the proposed method was verified through simulation experiments. The experimental results show that adding generator screening and power compensation mechanisms can effectively reduce the difficulty of model training and shorten the training time. When only the target section and target power values ​​are given, the model can automatically and continuously adjust the section power. While ensuring the adjustment accuracy, it greatly reduces the dependence on the professional experience of the operation and dispatching personnel and avoids a large number of manual calculation processes.

[0157] Embodiment 2:

[0158] like Figure 7 As shown, embodiment 2 provides a section power automatic adjustment system based on deep reinforcement learning, including an acquisition module 10, a decision module 20 and a power adjustment module 30;

[0159] The acquisition module 10 is used to acquire the current state information of the power grid, and the current state information includes the number of the section to be adjusted, the current power of the section to be adjusted, the target power of the section to be adjusted, the current active output of each generator, the maximum active output of each generator, and the minimum active output of each generator;

[0160] The decision module 20 is used to input the current state information into a preset intelligent agent, so that the intelligent agent makes a decision according to the current state information, and then outputs a corresponding action instruction group, wherein the intelligent agent is obtained based on deep reinforcement learning algorithm training;

[0161] The power adjustment module 30 is used to adjust the power of each generator according to the action instruction group.

[0162] In one possible implementation, Figure 8 As shown, the decision module 20 includes a sensitivity calculation unit 201, a generator screening unit 202, an instruction generation unit 203 and an output unit 204;

[0163] The sensitivity calculation unit 201 is used to calculate the first sensitivity index, the second sensitivity index and the third sensitivity index of each generator according to the current state information and a preset sensitivity calculation method;

[0164] The generator screening unit 202 is used to perform power adjustment screening and power compensation screening on each generator according to the current state information, the first sensitivity index, the second sensitivity index and the third sensitivity index of each generator, to obtain a forward power adjustment generator set, a reverse power adjustment generator set and a power compensation generator set;

[0165] The instruction generation unit 203 is used to generate a power adjustment instruction group and a power compensation instruction group according to the current state information, the forward power adjustment generator set, the reverse power adjustment generator set and the power compensation generator set;

[0166] The output unit 204 is used to combine the power adjustment instruction group and the power compensation instruction group to output the action instruction group.

[0167] Further, the instruction generation unit 203 generates a power adjustment instruction group and a power compensation instruction group according to the current state information, the forward power adjustment generator set, the reverse power adjustment generator set and the power compensation generator set, including:

[0168] According to the relative size of the current power of the section to be adjusted and the target power of the section to be adjusted, determining the current power adjustment mode as a forward power adjustment mode or a reverse power adjustment mode;

[0169] If the current power adjustment mode is the forward power adjustment mode, the action value range of the agent is divided into n sub-intervals according to the upper and lower boundaries of the action space of the agent, and each sub-interval is matched one by one with each generator in the forward power adjustment generator set, where n is the number of generators in the forward power adjustment generator set;

[0170] If the current power adjustment mode is the reverse power adjustment mode, the action value range of the agent is divided into n sub-intervals according to the upper and lower boundaries of the action space of the agent, and each sub-interval is matched one by one with each generator in the reverse power adjustment generator set, where n is the number of generators in the reverse power adjustment generator set;

[0171] Inputting the current state information into a preset deep reinforcement learning network so that the deep reinforcement learning network outputs a number of action values;

[0172] Mapping each action value to each corresponding generator through the corresponding sub-interval, thereby generating the power adjustment instruction group;

[0173] The power compensation action of each generator in the power compensation generator set is determined according to the respective action values, and then the power compensation instruction group is generated.

[0174] In a possible implementation, the step of obtaining the intelligent agent based on deep reinforcement learning algorithm training includes:

[0175] According to the Markov decision process and the deep reinforcement learning algorithm, an initial intelligent agent is constructed, wherein the initial intelligent agent includes an initial neural network, a state space, an action space, a reward function and a discount factor, and the initial neural network includes an Actor network and a Critic network;

[0176] Inputting a plurality of training data into the initial agent in sequence, and performing a plurality of rounds of training, wherein in each round, the initial agent continuously makes decisions according to the current training data, and combines the state, action and reward feedback corresponding to each decision into experience data, and stores each experience data in a preset experience pool, until the initial agent meets the preset conditions and completes the current round of training;

[0177] During the training of the initial agent, if the experience pool reaches a preset capacity, the training is suspended, the experience data in the experience pool is taken out, and the network parameters of the initial neural network are updated according to the experience data;

[0178] During the training process of the initial intelligent agent, if the number of decision steps of the initial intelligent agent reaches a maximum preset number of steps, the training is terminated to obtain the intelligent agent.

[0179] The embodiment of the present application provides a section power automatic adjustment system based on deep reinforcement learning, which uses deep reinforcement learning algorithm training to obtain an intelligent agent, which makes decisions based on the current state information of the power grid and outputs corresponding action instructions, and then adjusts the power of each generator according to the action instruction group to adjust the power of the target section to the target power. The method provided by the embodiment of the present application has higher accuracy than the existing power flow section constraint method, sensitivity analysis method and optimization control method due to the combination of artificial intelligence technology, and the intelligent agent used is obtained by pre-training through a deep reinforcement learning algorithm, does not rely on expert experience in the actual decision-making process, and can realize automatic and efficient power grid section power adjustment.

[0180] The more detailed working principle and step flow of this embodiment can refer to, but are not limited to, the relevant records of Embodiment 1.

[0181] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of protection of the present application.

Claims

1. A method for automatic section power adjustment based on deep reinforcement learning, characterized in that: include: Acquire current state information of the power grid, the current state information including the number of the section to be adjusted, the current power of the section to be adjusted, the target power of the section to be adjusted, the current active output of each generator, the maximum active output of each generator, and the minimum active output of each generator; Inputting the current state information into a preset intelligent agent, so that the intelligent agent makes a decision according to the current state information, and then outputs a corresponding action instruction group, wherein the intelligent agent is obtained by training based on a deep reinforcement learning algorithm; The power of each generator is adjusted according to the action instruction group.

2. The method for automatic section power adjustment based on deep reinforcement learning as claimed in claim 1, characterized in that: The agent makes a decision based on the current state information, and then outputs a corresponding action instruction group, including: Calculating a first sensitivity index, a second sensitivity index, and a third sensitivity index of each generator according to the current state information and a preset sensitivity calculation method; According to the current state information, the first sensitivity index, the second sensitivity index and the third sensitivity index of each generator, power adjustment screening and power compensation screening are performed on each generator to obtain a forward power adjustment generator set, a reverse power adjustment generator set and a power compensation generator set; Generate a power adjustment instruction group and a power compensation instruction group according to the current state information, the forward power adjustment generator set, the reverse power adjustment generator set and the power compensation generator set; The power adjustment instruction group and the power compensation instruction group are combined to output the action instruction group.

3. The method for automatic section power adjustment based on deep reinforcement learning as claimed in claim 2, characterized in that: The calculating the first sensitivity index, the second sensitivity index and the third sensitivity index of each generator according to the current state information and a preset sensitivity calculation method includes: Calculate a first sensitivity index and a second sensitivity index of each generator according to the current power of the section to be adjusted, the maximum active output of each generator, and the minimum active output of each generator; A third sensitivity index of each generator is calculated according to the sum of the first sensitivity index and the second sensitivity index.

4. The method for automatic section power adjustment based on deep reinforcement learning as claimed in claim 2, characterized in that: The method of performing power adjustment screening and power compensation screening on each generator according to the current state information, the first sensitivity index, the second sensitivity index and the third sensitivity index of each generator to obtain a forward power adjustment generator set, a reverse power adjustment generator set and a power compensation generator set includes: Perform forward power adjustment screening on each generator according to the current state information and each first sensitivity index to obtain a corresponding forward power adjustment generator set; Perform reverse power adjustment screening on each generator according to the current state information and each second sensitivity index to obtain a corresponding reverse power adjustment generator set; Calculating and obtaining a current power fluctuation boundary of a power grid according to the forward power adjustment generator set and the reverse power adjustment generator set; Power compensation screening is performed on each generator according to the preset constraint conditions, the third sensitivity index and the current power fluctuation boundary to obtain a corresponding power compensation generator set.

5. The method for automatic section power adjustment based on deep reinforcement learning as claimed in claim 2, characterized in that: The generating of a power adjustment instruction group and a power compensation instruction group according to the current state information, the forward power adjustment generator set, the reverse power adjustment generator set and the power compensation generator set comprises: According to the relative size of the current power of the section to be adjusted and the target power of the section to be adjusted, determining the current power adjustment mode as a forward power adjustment mode or a reverse power adjustment mode; If the current power adjustment mode is the forward power adjustment mode, the action value range of the agent is divided into n sub-intervals according to the upper and lower boundaries of the action space of the agent, and each sub-interval is matched one by one with each generator in the forward power adjustment generator set, where n is the number of generators in the forward power adjustment generator set; If the current power adjustment mode is the reverse power adjustment mode, the action value range of the agent is divided into n sub-intervals according to the upper and lower boundaries of the action space of the agent, and each sub-interval is matched one by one with each generator in the reverse power adjustment generator set, where n is the number of generators in the reverse power adjustment generator set; Inputting the current state information into a preset deep reinforcement learning network so that the deep reinforcement learning network outputs a number of action values; Mapping each action value to each corresponding generator through the corresponding sub-interval, thereby generating the power adjustment instruction group; The power compensation action of each generator in the power compensation generator set is determined according to the respective action values, and then the power compensation instruction group is generated.

6. A method for automatic section power adjustment based on deep reinforcement learning as described in any one of claims 1 to 5, characterized in that: The method of obtaining the intelligent agent based on deep reinforcement learning algorithm training includes: According to the Markov decision process and the deep reinforcement learning algorithm, an initial intelligent agent is constructed, wherein the initial intelligent agent includes an initial neural network, a state space, an action space, a reward function and a discount factor, and the initial neural network includes an Actor network and a Critic network; Inputting a plurality of training data into the initial agent in sequence, and performing a plurality of rounds of training, wherein in each round, the initial agent continuously makes decisions according to the current training data, and combines the state, action and reward feedback corresponding to each decision into experience data, and stores each experience data in a preset experience pool, until the initial agent meets the preset conditions and completes the current round of training; During the training of the initial agent, if the experience pool reaches a preset capacity, the training is suspended, the experience data in the experience pool is taken out, and the network parameters of the initial neural network are updated according to the experience data; During the training process of the initial intelligent agent, if the number of decision steps of the initial intelligent agent reaches a maximum preset number of steps, the training is terminated to obtain the intelligent agent.

7. A section power automatic adjustment system based on deep reinforcement learning, characterized in that: It includes an acquisition module, a decision module and a power adjustment module; The acquisition module is used to acquire the current state information of the power grid, and the current state information includes the number of the section to be adjusted, the current power of the section to be adjusted, the target power of the section to be adjusted, the current active output of each generator, the maximum active output of each generator, and the minimum active output of each generator; The decision module is used to input the current state information into a preset intelligent agent, so that the intelligent agent makes a decision according to the current state information, and then outputs a corresponding action instruction group, wherein the intelligent agent is obtained based on deep reinforcement learning algorithm training; The power adjustment module is used to adjust the power of each generator according to the action instruction group.

8. The section power automatic adjustment system based on deep reinforcement learning as claimed in claim 7, characterized in that: The decision module includes a sensitivity calculation unit, a generator screening unit, an instruction generation unit and an output unit; The sensitivity calculation unit is used to calculate the first sensitivity index, the second sensitivity index and the third sensitivity index of each generator according to the current state information and a preset sensitivity calculation method; The generator screening unit is used to perform power adjustment screening and power compensation screening on each generator according to the current state information, the first sensitivity index, the second sensitivity index and the third sensitivity index of each generator, to obtain a forward power adjustment generator set, a reverse power adjustment generator set and a power compensation generator set; The instruction generation unit is used to generate a power adjustment instruction group and a power compensation instruction group according to the current state information, the forward power adjustment generator set, the reverse power adjustment generator set and the power compensation generator set; The output unit is used to combine the power adjustment instruction group and the power compensation instruction group to output the action instruction group.

9. The automatic section power adjustment system based on deep reinforcement learning as claimed in claim 8, characterized in that: The instruction generation unit generates a power adjustment instruction group and a power compensation instruction group according to the current state information, the forward power adjustment generator set, the reverse power adjustment generator set and the power compensation generator set, including: According to the relative size of the current power of the section to be adjusted and the target power of the section to be adjusted, determining the current power adjustment mode as a forward power adjustment mode or a reverse power adjustment mode; If the current power adjustment mode is the forward power adjustment mode, the action value range of the agent is divided into n sub-intervals according to the upper and lower boundaries of the action space of the agent, and each sub-interval is matched one by one with each generator in the forward power adjustment generator set, where n is the number of generators in the forward power adjustment generator set; If the current power adjustment mode is the reverse power adjustment mode, the action value range of the agent is divided into n sub-intervals according to the upper and lower boundaries of the action space of the agent, and each sub-interval is matched one by one with each generator in the reverse power adjustment generator set, where n is the number of generators in the reverse power adjustment generator set; Inputting the current state information into a preset deep reinforcement learning network so that the deep reinforcement learning network outputs a number of action values; Mapping each action value to each corresponding generator through the corresponding sub-interval, thereby generating the power adjustment instruction group; The power compensation action of each generator in the power compensation generator set is determined according to the respective action values, and then the power compensation instruction group is generated.

10. A section power automatic adjustment system based on deep reinforcement learning as described in any one of claims 7 to 9, characterized in that: The method of obtaining the intelligent agent based on deep reinforcement learning algorithm training includes: According to the Markov decision process and the deep reinforcement learning algorithm, an initial intelligent agent is constructed, wherein the initial intelligent agent includes an initial neural network, a state space, an action space, a reward function and a discount factor, and the initial neural network includes an Actor network and a Critic network; Inputting a plurality of training data into the initial agent in sequence, and performing a plurality of rounds of training, wherein in each round, the initial agent continuously makes decisions according to the current training data, and combines the state, action and reward feedback corresponding to each decision into experience data, and stores each experience data in a preset experience pool, until the initial agent meets the preset conditions and completes the current round of training; During the training of the initial agent, if the experience pool reaches a preset capacity, the training is suspended, the experience data in the experience pool is taken out, and the network parameters of the initial neural network are updated according to the experience data; During the training process of the initial intelligent agent, if the number of decision steps of the initial intelligent agent reaches a maximum preset number of steps, the training is terminated to obtain the intelligent agent.

Images