A power intelligent dispatching system based on reinforcement learning

Abstract

The present invention relates to a power system dispatching system based on reinforcement learning, aiming to improve the efficiency and stability of the power system. The method links the environmental state of the power system with the dispatching action by establishing the interaction between the reinforcement learning agent and the environmental model, and converts the environmental state and the dispatching action into Q value through the neural network model. At the same time, the method takes into account various constraints in the power system and adopts the constraint optimization method to deal with these constraints. Experimental results show that the method has better adaptability, intelligence and real-time performance, can better adapt to the dynamic changes and uncertainties of the power system, and can effectively improve the efficiency and stability of the power system.

Description

Technical Field

[0001] The present disclosure relates to, but is not limited to, the technical field of electric power dispatching, and in particular to an electric power intelligent dispatching system based on reinforcement learning. Background Art

[0002] With the rapid development of the global economy, the safe and stable operation of power systems has become increasingly important. As one of the infrastructures of modern society, the stability of the power system is directly related to the development of social economy and the quality of people's lives. However, due to the complexity and uncertainty of the power system, the dispatching and operation management of the power system face many challenges. Traditional power system dispatching methods rely on manual experience and rules, and often cannot cope with complex power markets and power system operations, resulting in low efficiency and poor stability of power system dispatching, and even causing power system failures and accidents.

[0003] In this context, reinforcement learning, as an emerging artificial intelligence technology, provides a new solution for power system dispatching and operation management. Reinforcement learning algorithms allow agents to autonomously learn and optimize strategies through interaction with the environment, thereby realizing intelligent dispatching and operation management of power systems. The application of reinforcement learning algorithms in the field of power system dispatching is gaining more and more attention, and some important results have been achieved.

[0004] Compared with traditional power system dispatching methods, power system dispatching methods based on reinforcement learning have better adaptability, intelligence and real-time performance, can better adapt to the dynamic changes and uncertainties of the power system, and can effectively improve the efficiency and stability of the power system. However, the existing power system dispatching methods based on reinforcement learning still face some challenges and problems. Specifically, they are mainly manifested in the following aspects:

[0005] 1. Model training takes a long time. Since reinforcement learning algorithms require continuous trial and error and adjustment, it takes a lot of time and computing resources to train the model, especially for large-scale power systems.

[0006] 2. Model instability. Since the power system itself has uncertainty and dynamic changes, the learning results of the reinforcement learning model will also be affected by these factors, resulting in model instability. This means that in practical applications, the model needs to be continuously adjusted and optimized to ensure that it can better adapt to the dynamic changes and uncertainties of the power system.

[0007] 3. Poor model interpretability. Reinforcement learning algorithm is a black box algorithm, and it is difficult to explain and understand its internal decision-making process. This means that for some scenarios that require high model interpretability, reinforcement learning algorithm may not be applicable.

[0008] 4. The scope of application of the model is limited. Reinforcement learning algorithms are mainly applicable to solving problems in continuous state space and action space. For some discrete state space and action space problems, the application of reinforcement learning algorithms is relatively limited.

[0009] Existing technologies have some limitations in solving these problems. Traditional power system dispatching methods are usually rule-based or experience-based, requiring a lot of manpower and time to manually write and adjust rules, and are difficult to adapt to complex and changing power markets and power system changes. Traditional optimization methods also have problems such as low computational efficiency and low solution accuracy, making it difficult to meet real-time and accuracy requirements. In addition, existing reinforcement learning methods usually require a long training time and a large number of data samples when dealing with complex power system dispatching problems, and there are also certain challenges in the interpretability and stability of the model. Summary of the invention

[0010] The present disclosure provides an intelligent power dispatching system based on reinforcement learning, which can perform intelligent dispatching for various changes and uncertainties in the power system, realize the optimized operation and stability guarantee of the power system, thereby improving the efficiency, safety and reliability of the power system, and has broad application prospects and social and economic benefits; in addition, the system has better adaptability, intelligence and real-time performance, can better adapt to the dynamic changes and uncertainties of the power system, and can effectively improve the efficiency and stability of the power system.

[0011] In order to solve the above problems, the technical solution of the present invention is achieved as follows:

[0012] A power intelligent dispatching system based on reinforcement learning, the system comprising: at least one power system, and the power system comprises at least one power production unit, at least one power transmission unit and at least one power consumption unit; at least one reinforcement learning agent, wherein each reinforcement learning agent is connected to at least one power production unit, at least one power transmission unit and at least one power consumption unit, for generating dispatching actions and acting on the power production unit, the power transmission unit and the power consumption unit to complete the dispatching of the power system; an environmental model, the environmental model updates the environmental state according to the environmental state information of the power production unit, the power transmission unit and the power consumption unit, and transmits the environmental state to each reinforcement learning agent.

[0013] Furthermore, the environmental status information is the output parameters of the power production unit, the power transmission unit and the power consumption unit; the output parameter of the power production unit is the power production amount, the output parameter of the power transmission unit is the power transmission amount; the output parameter of the power consumption unit is the power consumption amount.

[0014] Furthermore, before updating the environmental state according to the environmental state information of the power production unit, the power transmission unit and the power consumption unit, the environmental model first calculates environmental feedback according to the environmental state information of the power production unit, the power transmission unit and the power consumption unit; the environmental feedback includes: electricity price environmental feedback, which is used to represent the current price in the electricity market; power supply capacity environmental feedback, which is used to represent the power supply capacity of the current power system; load demand environmental feedback, which is used to represent the load demand of the current power system.

[0015] Furthermore, the environmental model is expressed using the following formula:

[0016] S t+1 =f(S t ,A t ,O t+1 )=g(S t ,A t )+h(O t+1 );

[0017] Among them, S t represents the environmental state at time t, A t represents the scheduling action at time t, O t+1 represents the environmental feedback at time t+1; function g represents the environmental state transfer function; function h represents the observation function; the dispatching action is defined as the operation performed when the power system is dispatched.

[0018] Furthermore, the system also includes a reinforcement learning control center, which uses a preset control model to link the scheduling action of each reinforcement learning agent with the environmental state and convert it into a reward signal for the reinforcement learning agent; the control model is established based on an improved feedback rule reinforcement learning algorithm, and its update rule is expressed using the following formula:

[0019] Q(S t ,A t )←(1-α)Q(S t ,A t )+α[R t +γmax A′ Q(S t+1 ,A′)-Q(S t ,A t )];

[0020] Among them, Q(S t ,A t ) indicates that in the environmental state S t Take scheduling action A t The Q value represents the expected return that can be obtained by taking a certain scheduling action under given environmental state information; α is the learning rate, R t In the environment state S t Take scheduling action A t The reward signal obtained after, γ is the discount factor, S t+1 Indicates that scheduling action A is taken t The next environmental state after

[0021] A′=arg max a Q(S t+1 ,a);

[0022] Among them, a represents the scheduling action taken in the next state, Q(S t+1 ,a) represents the Q value of taking scheduling action a in the next environmental state; when updating the control model, it is necessary to first estimate the Q value of taking the optimal scheduling action in the next environmental state, so it is necessary to select the scheduling action with the largest Q value in the next environmental state.

[0023] Furthermore, the control model is expressed using the following formula:

[0024]

[0025] Among them, S t and A t Represent the environment state and scheduling action respectively, θ is the parameter of the neural network, T is the maximum time step, γ is the discount factor; π is a strategy, which means that in the environment state S t Take scheduling action A t The control model takes the environment state and the scheduling action as input and generates the corresponding Q value; max π It means taking the maximum value of all strategies π, that is, the maximum expected cumulative reward for selecting the scheduling action under the current environment state; Represents the expected value, which means that in the current environment state S t Take scheduling action A t In the case of , the expected value of the reward is obtained according to the strategy π; therefore, It means that among all possible strategies, the strategy with the largest expected cumulative reward is selected, that is, the optimal strategy; k is the number of time steps for cumulative rewards, which means the discounted cumulative sum of the immediate rewards of each step from the current time step t to the end of the maximum time step T; the value range of k is 0≤k≤Tt.

[0026] Furthermore, the control model adopts a deep Q network algorithm, and its training objective is expressed as:

[0027] Y t =R t +γmax A′ Q(S t+1 ,A′;θ - );

[0028] Among them, Y t represents the training target of the tth sample, R t In the environment state S t Take scheduling action A t The reward signal after, γ is the discount factor, S t+1 Indicates that scheduling action A is taken t The next environment state after - represents the fixed parameters used to calculate the target value, that is, the parameters of the deep Q network are periodically updated.

[0029] Furthermore, the reinforcement learning agent is trained using a reinforcement learning algorithm based on historical experience, and its reward signal is expressed using the following formula:

[0030] R t =β1w1P t +β2w2exp(-β3(D t -C t ) 2 )+β4w3r t ;

[0031] Among them, w1, w2 and w3 are constants, P t represents the electricity market price at time t, D t and C t They represent the power consumption and power transmission at time t, r t represents the stability of the power system at time t; β1, β2, β3, and β4 are constants used to control the scale and shape of the reward signal.

[0032] Furthermore, the reinforcement learning algorithm learns based on historical experience and adjusts the dispatching action according to the real-time information of the electricity market, specifically including: the reinforcement learning agent is trained using the experience replay method and selects the dispatching action according to the current environmental state; the dispatching action is selected using the ∈-greedy strategy, that is, the dispatching action with the largest current Q value is selected with a probability of 1-∈, and a dispatching action is randomly selected with a probability of ∈.

[0033] Furthermore, before generating a dispatching action, the reinforcement learning agent will include restrictions as constraints to ensure the stable operation of the power system; the restrictions include: capacity restrictions of power production units and power consumption units, line capacity restrictions of power transmission units, and stability restrictions of the power system; the reinforcement learning agent uses a constrained optimization method to process these restrictions and incorporates them into the optimization objective of reinforcement learning to obtain the optimal solution that meets the constraints.

[0034] The electric power intelligent dispatching system based on reinforcement learning of the present invention has the following features:

[0035] Beneficial effects:

[0036] Intelligent dispatching of the power system is realized. Traditional power system dispatching methods often rely on experience and rules, and have poor response capabilities to various changes and uncertainties in the power system, making it difficult to achieve optimal dispatching and resource allocation. The present invention uses deep reinforcement learning technology to automatically learn and optimize the dispatching strategy of the power system, thereby improving the efficiency and economy of the power system. Through intelligent dispatching, it is possible to maximize the use of power resources, reduce power production and transmission costs, and improve the quality and reliability of power supply while ensuring the stable operation of the power system.

[0037] The optimal dispatching of the power system is realized. The present invention can realize the optimal solution that meets the conditions by incorporating various restrictions and constraints of the power system into the optimization target of reinforcement learning. At the same time, the present invention also adopts a deep reinforcement learning algorithm based on Q-learning, which can continuously optimize the dispatching strategy during the learning process to realize efficient and optimal dispatching of the system. The reliability and flexibility of the power system is one of the important indicators of power system dispatching. Traditional dispatching methods are often difficult to adapt to the changes and uncertainties of the power system, which limits the reliability and flexibility of the power system. The present invention adopts deep reinforcement learning technology, which can automatically learn the dispatching strategy of the power system, make real-time adjustments according to various changes and uncertainties in the power system, and improve the reliability and flexibility of the power system. Through intelligent dispatching, various changes and uncertainties in the power system, such as electricity, can be better coped with.

[0038] Improved the stability and safety of the power system. The stability and safety of the power system is one of the important goals of power system dispatching. Traditional dispatching methods are often unable to meet the complexity and uncertainty of the power system, and easily lead to instability and accidents in the power system. The present invention adopts deep reinforcement learning technology, which can adaptively respond to various changes and uncertainties in the power system and take corresponding measures to ensure the safe and stable operation of the power system. Through intelligent dispatching, the supply and demand relationship, power load and transmission of the power system can be finely regulated, the operating risk and failure rate of the power system can be reduced, and the safety and stability of the power system can be improved.

[0039] Wide applicability. The power intelligent dispatching system and method provided by the present invention are applicable to power systems of various sizes and types, can be adjusted and optimized according to specific circumstances, and have strong applicability and flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of the system structure of a power intelligent dispatching system based on reinforcement learning provided by an embodiment of the present invention; DETAILED DESCRIPTION

[0041] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present disclosure clearer and more understandable, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0042] Example 1

[0043] refer to Figure 1 , an intelligent power dispatching system based on reinforcement learning, the system comprising: at least one power system, and the power system comprises at least one power production unit, at least one power transmission unit and at least one power consumption unit; at least one reinforcement learning agent, wherein each reinforcement learning agent is connected to at least one power production unit, at least one power transmission unit and at least one power consumption unit, for generating dispatching actions and acting on the power production unit, the power transmission unit and the power consumption unit to complete the dispatching of the power system; an environmental model, the environmental model updates the environmental state according to the environmental state information of the power production unit, the power transmission unit and the power consumption unit, and transmits the environmental state to each reinforcement learning agent.

[0044] Each reinforcement learning agent chooses a scheduling action a at each time step t t Then the environmental model calculates the environmental feedback O according to the status information of all units in the power system. t+1, and passes the state information to each reinforcement learning agent. Then, each agent uses the neural network model to estimate the maximum expected cumulative reward of taking each scheduling action in the current state, and selects the optimal scheduling action based on these estimates. At the same time, each reinforcement learning agent also updates the parameters of the neural network model through the reinforcement learning algorithm based on the current state and scheduling action.

[0045] The environmental model updates the environmental state based on the state information of all units in the power system and transmits the state information to each reinforcement learning agent. Environmental feedback includes electricity price status, power supply capacity status and load demand status, which are used to calculate the reward signal of the reinforcement learning agent in the current state. At the same time, the state transfer function and observation function of the environmental model are also calculated based on the state information of all units in the power system and environmental feedback.

[0046] When the power system is running, each power production unit, power transmission unit, and power consumption unit will send out its own operating status information. This information includes the power unit's electricity, electricity price, power supply capacity, etc. The environmental model will calculate the current environmental state based on this status information, and pass the environmental state and status information to each reinforcement learning agent. The reinforcement learning agent uses the neural network model to calculate the maximum expected cumulative reward for each scheduling action based on the current environmental state and status information, and selects the optimal scheduling action.

[0047] After selecting a dispatch action, each reinforcement learning agent sends its dispatch action information to the environment model. The environment model calculates the environmental feedback for the next time step based on the operating status information of all power units and the dispatch action information of the agent, and transmits the status information and environmental feedback to each reinforcement learning agent. The reinforcement learning agent updates the parameters of the neural network model through the reinforcement learning algorithm based on the current state and dispatch action. This process is iterated continuously until the system reaches a stable state.

[0048] After the system reaches a stable state, each reinforcement learning agent will select the optimal dispatching action under different power system states according to its own neural network model, thereby achieving the goal of intelligent power dispatching, such as minimizing total cost and maximizing total benefit. Through this interactive method, the reinforcement learning agent and the environment model achieve efficient information exchange and learning with each other, enabling the system to achieve more intelligent and efficient power dispatching.

[0049] Through this interactive approach, the reinforcement learning agent can gradually learn how to choose the optimal dispatching action under different environmental conditions of the power system, thereby achieving the goal of intelligent power dispatching.

[0050] Example 2

[0051] Based on the previous embodiment, the environmental status information is the output parameters of the power production unit, the power transmission unit and the power consumption unit; the output parameter of the power production unit is the power production amount, the output parameter of the power transmission unit is the power transmission amount; the output parameter of the power consumption unit is the power consumption amount.

[0052] Specifically, the environment state information is passed to the reinforcement learning agent. In the reinforcement learning power intelligent dispatching system, each agent selects a dispatch action based on its own strategy and current state value, and sends the selected dispatch action to the environment model. At the end of each time step, the environment model sends the value of the current state as environmental feedback to each agent, and the agent updates its strategy based on this feedback information in order to obtain higher rewards in future time steps.

[0053] Example 3

[0054] Based on the previous embodiment, before the environmental model updates the environmental status according to the environmental status information of the power production unit, the power transmission unit and the power consumption unit, it first calculates the environmental feedback according to the environmental status information of the power production unit, the power transmission unit and the power consumption unit; the environmental feedback includes: electricity price environmental feedback, which is used to represent the current price in the electricity market; power supply capacity environmental feedback, which is used to represent the power supply capacity of the current power system; load demand environmental feedback, which is used to represent the load demand of the current power system.

[0055] Specifically, environmental feedback can include the following information:

[0056] Electricity price environment feedback: Indicates the current price of electricity market. According to the market supply and demand relationship, the electricity price will change with the change of electricity supply and demand relationship.

[0057] Power supply capacity environmental feedback: Indicates the power supply capacity of the current power system. The power supply capacity is closely related to the state of the power system, and is affected by factors such as the output power of the power production unit, the line capacity of the power transmission unit, and the demand of the power consumption unit.

[0058] Load demand environment feedback: Indicates the current load demand of the power system. Load demand usually refers to the power demand of power consumption units, which changes with time and season.

[0059] These environmental feedback information will be used to calculate the rewards obtained by the agent in the current state, so as to evaluate whether the agent's behavior is in line with expectations. In the framework of reinforcement learning, the agent continuously learns and improves its behavior strategy through interaction with the environment in order to obtain higher rewards in future time steps.

[0060] Example 4

[0061] Based on the previous embodiment, the environment model is expressed using the following formula:

[0062] S t+1 =f(S t ,A t ,O t+1 )=g(S t ,A t )+h(O t+1 );

[0063] Among them, S t represents the environmental state at time t, A t represents the scheduling action at time t, O t+1 represents the environmental feedback at time t+1; function g represents the environmental state transfer function; function h represents the observation function; the dispatching action is defined as the operation performed when the power system is dispatched.

[0064] Specifically, the role of the environmental model is to calculate the current environmental state based on the environmental state information of the power production unit, power transmission unit, and power consumption unit, and pass the environmental state and environmental state information to each reinforcement learning agent. The environmental model is actually a function that takes environmental state information, scheduling actions, and environmental feedback as input, and outputs the environmental state information and environmental feedback for the next time step.

[0065] Specifically, the working process of the environmental model is as follows:

[0066] Receive environmental status information of power production units, power transmission units and power consumption units, including the power quantity, power price, power supply capacity, etc. of the power units. This information will constitute the current environmental status S t .

[0067] Receive the scheduling action information A of the reinforcement learning agent t ,The dispatch action information usually includes the switch status of the power unit, the ,power generation, load dispatch, etc.

[0068] According to the current environmental status S t and scheduling action information A t , calculate the environmental feedback O for the next time step t+1 Environmental feedback includes electricity price status, power supply capacity status and load demand status, which are used to represent the current electricity market price, power supply capacity and load demand of the power system.

[0069] Feedback the calculated environment to O t+1 and the environmental state S tPassed to each reinforcement learning agent so that they can make the best decision in the next time step.

[0070] According to the dispatch action information A passed back t , update the environmental status information of the power production unit, power transmission unit and power consumption unit, and obtain the environmental status S of the next time step t+1 .

[0071] The above process is iterated continuously until the system reaches a stable state.

[0072] Example 5

[0073] On the basis of the previous embodiment, the system further includes a reinforcement learning control center, which uses a preset control model to link the scheduling action of each reinforcement learning agent with the environmental state and convert it into a reward signal for the reinforcement learning agent; the control model is established based on an improved feedback rule reinforcement learning algorithm, and its update rule is expressed using the following formula:

[0074] Q(S t ,A t )←(1-α)Q(S t ,A t )+α[R t +γmax A′ Q(S t+1 ,A′)-Q(S t ,A t )];

[0075] Among them, Q(S t ,A t ) indicates that in the environmental state S t Take scheduling action A t The Q value represents the expected return that can be obtained by taking a certain scheduling action under given environmental state information; α is the learning rate, R t In the environment state S t Take scheduling action A t The reward signal obtained after, γ is the discount factor, S t+1 Indicates that scheduling action A is taken t The next environmental state after

[0076] A′=argmax a Q(S t+1 ,a);

[0077] Among them, a represents the scheduling action taken in the next state, Q(S t+1,a) represents the Q value of taking scheduling action a in the next environmental state; when updating the control model, it is necessary to first estimate the Q value of taking the optimal scheduling action in the next environmental state, so it is necessary to select the scheduling action with the largest Q value in the next environmental state.

[0078] Specifically, the learning process of reinforcement learning algorithms is usually divided into two stages: exploration and exploitation. In the exploration stage, the reinforcement learning agent will randomly select scheduling actions to explore more states and scheduling actions; in the exploitation stage, the reinforcement learning agent will select the optimal scheduling action based on the knowledge it has learned.

[0079] The key to the learning process is how the reinforcement learning agent learns the optimal strategy. During the learning process, the reinforcement learning algorithm will continuously update the estimated value function to obtain the optimal scheduling action selection strategy. Common reinforcement learning algorithms include Q-learning, SARSA, Actor-Critic, etc.

[0080] In reinforcement learning, each reinforcement learning agent has its own state space, scheduling action space, and reward function. The reinforcement learning agent will select a scheduling action from the scheduling action space based on the current state and execute the scheduling action. After executing the scheduling action, the environment will return a reward signal based on the current state and scheduling action. The reward signal is used to evaluate the quality of the scheduling action.

[0081] Example 6

[0082] Based on the previous embodiment, the control model is expressed using the following formula:

[0083]

[0084] Among them, S t and A t Represent the environment state and scheduling action respectively, θ is the parameter of the neural network, T is the maximum time step, γ is the discount factor; π is a strategy, which means that in the environment state S t Take scheduling action A t The control model takes the environment state and the scheduling action as input and generates the corresponding Q value; max π It means taking the maximum value of all strategies π, that is, the maximum expected cumulative reward for selecting the scheduling action under the current environment state; Represents the expected value, which means that in the current environment state S t Take scheduling action A t In the case of , the expected value of the reward is obtained according to the strategy π; therefore, It means that among all possible strategies, the strategy with the largest expected cumulative reward is selected, that is, the optimal strategy; k is the number of time steps for cumulative rewards, which means the discounted cumulative sum of the immediate rewards of each step from the current time step t to the end of the maximum time step T; the value range of k is 0≤k≤Tt.

[0085] Specifically, Q(S t ,A t ; θ) represents the state S t Take scheduling action A t The Q value at the time θ is the parameter of the neural network. This Q value can be understood as the long-term reward of taking this scheduling action under the current environment state.

[0086] Q * (S t ,A t ) represents the optimal Q-value function, which is the maximum long-term return that can be obtained among all strategies. It is represented by max π , which means taking the maximum value among all possible strategies π.

[0087] Indicates that according to the current strategy π in state S t Take scheduling action A t The expected discounted cumulative reward starting from time t after γ is calculated. Where γ is the discount factor used to discount the value of future rewards. R t+k+1 Represents the immediate reward at time t+k+1.

[0088] The control model is a deep reinforcement learning algorithm that links the scheduling actions of each reinforcement learning agent to the state of the environment and converts it into a reward signal for the agent. Specifically, the model receives and environment state S t and dispatch action A t As input, and output the corresponding Q value Q(S t ,A t ; θ). The model parameters θ are learned through the training process of the neural network.

[0089] The neural network model is a deep reinforcement learning algorithm that is used to link the actions of each reinforcement learning agent with the state of the environment and convert it into a reward signal for the agent. Specifically, the model receives the state S t and action A t As input, and output the corresponding Q value Q(S t ,A t ; θ). The model parameters θ are learned through the training process of the neural network.

[0090] During the training process, the model is based on the current state St and optional action A t , select the action A′ corresponding to the largest Q value through the greedy strategy, that is, Then execute A′ in the environment and get the next state S t+1 and instant reward R t+1 Then, the model converts S t+1 and A′ as the next state and action, repeat the above steps until the maximum time step T or the terminal state is reached. In this process, the model accumulates the immediate reward R t+1 , in order to maximize the cumulative reward value

[0091] During training, the model uses a Q-learning based update rule to update its parameters θ to maximize the cumulative reward value. Specifically, the model takes action A at time step t. t After that, according to the next state S t+1 and instant reward R t+1 , calculate the target Q value Q * (S t ,A t )=R t+1 +γmax A′ Q(S t+1 ,A′;θ), where γ is the discount factor. Then, the model uses the mean square error between the target Q value and the current estimated Q value to update the parameter θ so that the current estimated Q value gradually approaches the true target Q value. Specifically, the model uses the following update formula:

[0092]

[0093] Among them, α is the learning rate, It is the gradient operator for the parameter θ. The meaning of the update formula is that according to the mean square error between the target Q value and the current estimated Q value, the gradient is calculated and the parameter θ is updated so that the current estimated Q value gradually approaches the true target Q value.

[0094] Example 7

[0095] Based on the previous embodiment, the control model adopts a deep Q network algorithm, and its training objective is expressed as:

[0096] Y t =R t +γmax A′ Q(S t+1 ,A′;θ - );

[0097] Among them, Y t represents the training target of the tth sample, R t In the environment state St Take scheduling action A t The reward signal after, γ is the discount factor, S t+1 Indicates that scheduling action A is taken t The next environment state after - represents the fixed parameters used to calculate the target value, that is, the parameters of the deep Q network are periodically updated.

[0098] Example 8

[0099] Based on the previous embodiment, the reinforcement learning agent is trained using a reinforcement learning algorithm based on historical experience, and its reward signal is expressed using the following formula:

[0100] R t =β1w1P t +β2w2exp(-β3(D t -C t ) 2 )+β4w3r t ;

[0101] Among them, w1, w2 and w3 are constants, P t represents the electricity market price at time t, D t and C t They represent the power consumption and power transmission at time t, r t represents the stability of the power system at time t; β1, β2, β3, and β4 are constants used to control the scale and shape of the reward signal.

[0102] Example 9

[0103] On the basis of the previous embodiment, the reinforcement learning algorithm learns based on historical experience and adjusts the dispatching action according to the real-time information of the electricity market, specifically including: the reinforcement learning agent is trained using the experience replay method and selects the dispatching action according to the current environmental state; the dispatching action is selected using the ∈-greedy strategy, that is, the dispatching action with the largest current Q value is selected with a probability of 1-∈, and a dispatching action is randomly selected with a probability of ∈.

[0104] The calculation formula of this reward signal includes three weights, namely w1, w2 and w3. Among them, w1 represents the weight of the electricity market price, w2 represents the weight of the error between electricity consumption and electricity transmission, and w3 represents the weight of the stability of the power system.

[0105] In the electricity market price part, a linear method is used to calculate the reward signal, namely w1P t , where P trepresents the electricity market price at time t. This means that when the electricity market price is higher, the reward signal obtained by the agent will also be higher, which encourages the agent to produce or supply electricity during the peak price period.

[0106] The error between power consumption and power transmission uses Gaussian distribution to calculate the reward signal, that is, β2w2exp(-β3(D t -C t ) 2 ), where β2 and β3 are constants that control the scale and shape of the reward signal. This means that when the error between power consumption and power transmission is smaller, the reward signal obtained by the agent will be higher, encouraging the agent to dispatch power more accurately.

[0107] In the power system stability part, the reward signal is calculated by using some indicators, namely w3R t , where R t represents the stability of the power system at time t. This means that when the stability of the power system is better, the reward signal obtained by the agent will be higher, which encourages the agent to pay more attention to the stability of the power system when performing power dispatch.

[0108] In summary, the calculation formula of the reward signal includes three aspects: electricity market price, the error between electricity consumption and electricity transmission, and the stability of the power system. It encourages the intelligent agent to perform power dispatch more accurately while paying attention to the stability of the power system to achieve the optimal power dispatch effect.

[0109] Example 10

[0110] On the basis of the previous embodiment, the reinforcement learning agent will include restriction conditions as constraints before generating scheduling actions to ensure the stable operation of the power system; the restriction conditions include: capacity limitations of power production units and power consumption units, line capacity limitations of power transmission units, and stability limitations of the power system; the reinforcement learning agent adopts a constrained optimization method to process these restriction conditions and incorporates them into the optimization goal of reinforcement learning to obtain the optimal solution that meets the constraint conditions.

[0111] Specifically, constrained optimization can be achieved by introducing the Lagrange multiplier method. For an optimization problem with m constraints, m Lagrange multipliers λ1,λ2,…,λ m , and transform the constraints into equality constraints to obtain the following Lagrangian function:

[0112]

[0113] Among them, x is the decision variable to be optimized, f(x) is the objective function, and g i (x) is the i-th constraint, λ i is the corresponding Lagrange multiplier. The minimum value of the Lagrange function is equal to the minimum value of the optimization problem. Therefore, the optimal solution of the optimization problem can be solved by solving the gradient of the Lagrange function. Specifically, the values ​​of the decision variables and Lagrange multipliers can be determined by solving the following set of equations:

[0114]

[0115] When introducing constraints into the optimization objective of reinforcement learning, the constraints can be transformed into equality constraints, and the equality constraints and the objective function can be combined into a Lagrangian function, and then the objective function and the constraints can be considered simultaneously in the solution process. Specifically, at each time step t, the objective function can be expressed as the reward signal r t , the constraints are expressed as equality constraints g(S t ,A t )=0, and then merge them into a Lagrangian function, and update the parameter θ of the neural network model by solving the gradient of the Lagrangian function. In this way, the optimal solution that meets the constraints can be obtained, thereby achieving safe and stable operation of the power system.

[0116] Specifically, the power system also includes an electricity market, and the reinforcement learning agent obtains the electricity market price from the electricity market; the electricity market price is calculated using the following formula:

[0117] P = ∫ D m(D)∫ S j(S)u(D,S)dSdD;

[0118] Among them, P represents the price of electricity, D represents electricity consumption, and S represents electricity production; function m represents the probability distribution of electricity consumption; function j represents the probability distribution of electricity production; function u represents the probability distribution of supply and demand relationship.

[0119] It should be noted that the above-mentioned device (equipment) embodiment and readable storage medium embodiment belong to the same concept as the method embodiment, and the specific implementation process is detailed in the method embodiment. The technical features in the method embodiment are applicable to the device embodiment, and will not be repeated here.

[0120] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0121] In hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0122] The preferred embodiments of the present disclosure are described above with reference to the accompanying drawings, but the scope of the present disclosure is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the present disclosure shall be within the scope of the present disclosure.

Claims

1. A power intelligent dispatching system based on reinforcement learning, characterized in that: The system comprises: at least one power system, and the power system comprises at least one power production unit, at least one power transmission unit and at least one power consumption unit; at least one reinforcement learning agent, wherein each reinforcement learning agent is connected to at least one power production unit, at least one power transmission unit and at least one power consumption unit, so as to generate a scheduling action and act on the power production unit, the power transmission unit and the power consumption unit to complete the scheduling of the power system; an environmental model, wherein the environmental model updates the environmental state according to the environmental state information of the power production unit, the power transmission unit and the power consumption unit, and transmits the environmental state to each reinforcement learning agent; the environmental state information is the output parameters of the power production unit, the power transmission unit and the power consumption unit; the output parameter of the power production unit is the power production amount, the output parameter of the power transmission unit is the power transmission amount; the output parameter of the power consumption unit is the power consumption amount; the environmental model is represented by the following formula: S t+1 =f(S t ,A t ,O t+1 )=g(S t ,A t )+h(O t+1 ); Among them, S t represents the state of the environment at time step t, A t represents the scheduling action at time step t, O t+1 represents the environmental feedback at time step t+1, which includes the electricity price state, power supply capacity state and load demand state, and is used to represent the current power market price, power supply capacity and load demand of the power system; function g represents the environmental state transfer function; function h represents the observation function; the dispatching action is defined as the operation performed when dispatching the power system; The system also includes a reinforcement learning control center, which uses a preset control model to link the scheduling action of each reinforcement learning agent with the environmental state and convert it into a reward signal for the reinforcement learning agent; the control model is established based on an improved feedback rule reinforcement learning algorithm, and its update rule is expressed using the following formula: Q(S t ,A t )←(1-α)Q(S t ,A t )+α[R t +γmax A′ Q(S t+1 ,A′)-Q(S t ,A t )]; Among them, Q(S t , A t ) indicates that in the environmental state S t Take scheduling action A t The Q value represents the expected return that can be obtained by taking a certain scheduling action under given environmental state information; α is the learning rate, R t Indicates that in the environmental state S t Take scheduling action A t The reward signal obtained after, γ is the discount factor, S t+1 Indicates that scheduling action A is taken t The next environmental state after A′=arg max a Q(S t+1 ,a); Among them, a represents the scheduling action taken in the next state, Q(S t+1 , a) represents the Q value of taking scheduling action a in the next environmental state; when updating the control model, it is necessary to first estimate the Q value of taking the optimal scheduling action in the next environmental state, so it is necessary to select the scheduling action with the largest Q value in the next environmental state; A′ is the action corresponding to the largest Q value; The control model is expressed using the following formula: Among them, S t and A t Represent the environment state and scheduling action respectively, θ is the parameter of the neural network, T is the maximum time step, γ is the discount factor; π is a strategy, which means that in the environment state S t Take scheduling action A t The control model takes the environment state and the scheduling action as input and generates the corresponding Q value; max π It means taking the maximum value of all strategies π, that is, the maximum expected cumulative reward for selecting the scheduling action under the current environment state; Represents the expected value, which means that in the current environment state S t Take scheduling action A t In the case of , the expected value of the reward is obtained according to the strategy π; therefore, It means that among all possible strategies, the strategy with the largest expected cumulative reward is selected, that is, the optimal strategy; k is the time step of cumulative reward, which means the time step of the discounted cumulative sum of the instantaneous reward of each step from the current time step t to the end of the maximum time step T; the value range of k is 0≤k≤Tt; R t+k+1 represents the immediate reward at time t+k+1; During the training process, the control model uses a Q-learning-based update rule to update its parameters θ to maximize the cumulative reward value, including: the control model takes action A at time step t t After that, according to the next state S t+1 and instant reward R t+1 , calculate the target Q value, the formula is as follows: Q * (S t ,A t )=R t+1 +γmax A′ Q(S t+1 ,A′;θ); Where γ is the discount factor; the control model then uses the mean squared error between the target Q value and the current estimated Q value to update the parameter θ as follows: Among them, α is the learning rate, is the gradient operator of the parameter θ; the reinforcement learning agent is trained using a reinforcement learning algorithm based on historical experience, and its reward signal is expressed using the following formula: R t =β1w1P t +β2w2exp(-β3(D t -C t ) 2 )+β4w3r t ; Among them, w1, w2 and w3 are constants, P t represents the electricity market price at time step t, D t and C t represent the power consumption and power transmission at time step t, r t represents the stability of the power system at time step t; β1, β2, β3, and β4 are constants used to control the scale and shape of the reward signal.

2. The system according to claim 1, characterized in that The environmental model first calculates environmental feedback according to the environmental status information of the power production unit, the power transmission unit and the power consumption unit before updating the environmental status according to the environmental status information of the power production unit, the power transmission unit and the power consumption unit.

3. The system according to claim 2, characterized in that The control model adopts a deep Q network algorithm, and its training objective is expressed as: Y t =R t +γmax A′ Q(S t+1 ,A′;θ - ); Among them, Y t represents the training target of the tth sample, R t Indicates that in the environmental state S t Take scheduling action A t The reward signal after, γ is the discount factor, S t+1 Indicates that scheduling action A is taken t The next environment state after - represents the fixed parameters used to calculate the target value, that is, the parameters of the deep Q network are periodically updated.

4. The system according to claim 3, characterized in that The reinforcement learning algorithm learns based on historical experience and adjusts the dispatching action according to the real-time information of the power market, specifically including: the reinforcement learning agent is trained using the experience replay method and selects the dispatching action according to the current environmental state; the dispatching action is selected using the ∈-greedy strategy, that is, the dispatching action with the largest current Q value is selected with a probability of 1-∈, and a dispatching action is randomly selected with a probability of ∈.

5. The system according to claim 4, characterized in that Before generating a scheduling action, the reinforcement learning agent will include constraints as constraints; the constraints include: capacity constraints of power production units and power consumption units, line capacity constraints of power transmission units, and stability constraints of the power system; the reinforcement learning agent uses a constrained optimization method to process these constraints and incorporates them into the optimization goal of reinforcement learning to obtain an optimal solution that meets the constraints.

Images