A multi-energy system economic dispatch method based on knowledge-assisted deep reinforcement learning

Abstract

The application discloses a kind of multi-energy system economic dispatching methods based on knowledge auxiliary deep reinforcement learning, and the specific steps of the method are as follows: first, deep deterministic policy gradient algorithm constructs an intelligent agent for receiving state information transmitted by environment and exploring corresponding optimal unit output, the algorithm can obtain optimal scheduling strategy and reduce dimension disaster.Secondly, knowledge auxiliary reinforcement learning algorithm is used as protector in the middle layer of intelligent agent and environment, and the protector can accelerate the convergence speed of deep deterministic policy gradient algorithm, while ensuring that the intelligent agent will not endanger the safety of system equipment in the process of exploration and trial-and-error.Finally, in the case where it is difficult to establish an accurate system model, the multi-energy system economic dispatching method based on knowledge auxiliary deep reinforcement learning can effectively obtain the optimal economy and convergence of system scheduling.The application can improve the safety of the system and enhance the ability of the system to operate economically and efficiently.

Description

Technical Field

[0001] This invention belongs to the field of integrated energy system optimization scheduling, specifically a multi-energy system economic scheduling method based on knowledge-assisted deep reinforcement learning. Background Technology

[0002] To address the current global challenges of fossil fuel scarcity and environmental pollution, power systems have evolved into multi-energy systems with the power system as the core and interconnected with various other energy sources. However, at present, establishing accurate models for multi-energy systems with renewable energy integration remains challenging because both renewable energy output and the demand from various energy sources are uncertain. The accuracy of multi-energy system models is affected by the accuracy of renewable energy output prediction and user demand prediction. Current methods for handling uncertainty include robust optimization and stochastic optimization. However, robust optimization, which considers the worst-case scenario, tends to yield overly conservative results. Stochastic optimization addresses uncertainty through scenario generation, but excessive scenario generation increases computational burden. Furthermore, while stochastic optimization aims to optimize the expected return across all scenarios, the actual return of a single scenario may not be maximized.

[0003] Deep deterministic policy gradient algorithms, belonging to deep reinforcement learning algorithms, are methods that obtain optimal scheduling strategies by having an agent explore the environment and continuously update its own policy. This method is data-driven and can enable multi-energy systems to achieve optimal economic benefits without requiring the manual creation of precise models of uncertainty in new energy sources. However, when the agent's own policy has not yet reached its optimal state, it may engage in behaviors that harm the safety of equipment within the system or compromise the reliability of the system's power supply. Summary of the Invention

[0004] To address the challenge of establishing accurate models for optimal scheduling of multi-energy systems and to improve the economic efficiency and security of multi-energy system operation, this invention provides an economic scheduling method for multi-energy systems based on knowledge-assisted deep reinforcement learning. By leveraging the synergistic effect of data-driven approaches and expert knowledge, the optimal economic scheduling strategy for multi-energy systems can be obtained, thereby improving system security and enhancing the system's economic and efficient operation capabilities.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] The present invention provides an economic scheduling method for multi-energy systems based on knowledge-assisted deep reinforcement learning, comprising the following steps:

[0007] (1) Obtain the initial state variables of the multi-energy system and pass them to the agent;

[0008] (2) After receiving the state quantity, the agent obtains the corresponding action quantity through its own policy and adds a Gaussian noise to the action quantity for exploration;

[0009] (3) The protector acquires the environmental state and the action quantity generated by the agent. Based on expert knowledge, it judges whether the action quantity exceeds the safe operating range of the multi-functional system device. If the action quantity exceeds the safe operating range of the system, it corrects it and transmits the corrected action to the environment for execution, while feeding back a penalty cost greater than zero to the agent. If the action quantity is within the safe operating range of the device, it directly transmits the action quantity to the environment for execution and feeds back a penalty cost equal to zero to the agent.

[0010] (4) After receiving the action quantity from the protector, the environment executes the action to obtain the system's operating cost and acquires the information of the next moment's state, and then transmits the operating cost and the next moment's state quantity to the agent.

[0011] (5) Repeat steps (2), (3) and (4) to store the system state quantity, agent action quantity, penalty cost, running cost and system state quantity at the next moment obtained each time in the agent's experience replay pool. The agent randomly extracts data from the experience replay pool and updates its own policy according to the deep deterministic policy gradient algorithm.

[0012] The intelligent agent comprises four neural networks: a behavior network, a critique network, a behavior-target network, and a critique-target network. The behavior network and the behavior-target network have the same architecture, as do the critique network and the critique-target network. All four networks are generated using randomly initialized weights. The behavior network uses the system state variable s. t The network takes a Gaussian noise as input and adds it to the network output to obtain a continuous action. The behavior network model is as follows:

[0013]

[0014] In the formula a t (s t () represents the new motion quantity after adding Gaussian noise to the original motion quantity; N represents the raw action quantity output by the behavior network; t (0,λ k σ 2 ) represents Gaussian noise; λ k σ 2 λ is the variance of the Gaussian noise; λ∈(0,1) is the decay factor; k is the current iteration number. The Gaussian noise added to the action will gradually decrease during the agent's iterative learning process.

[0015] The new action quantities generated by the behavioral network are specifically represented as follows:

[0016]

[0017] In the formula The percentage of the input power of the electric boiler to its rated power at time t; and For the variable {0,1}, This indicates that the battery is charging. This indicates that the battery is discharging. This indicates that the heat storage tank is being filled with heat. This indicates that the heat storage tank is releasing heat; Let t be the percentage of battery capacity that has been charged or discharged at time t, and let the state of charge / discharge change from t to t. Decide; Let t be the percentage of heat charge / discharge in the thermal storage tank relative to its capacity, and let the heat charge / discharge state change from... The agent's initial policy is a random policy, and the agent's policy will be updated through iterative learning.

[0018] The state variables of the multi-energy system in step (1) are:

[0019]

[0020] In the formula s t This represents the system state vector at time t; t is the time the system is at. The percentage of energy contained in the system battery at time t relative to the battery's rated capacity; The percentage of energy contained in the system's thermal storage tank at time t relative to the tank's rated capacity; Let t be the system electrical load; Let t be the system heat load; The unit electricity price for the system's transaction with the power grid at time t is such that the system can purchase electricity from the power grid to make up for its own power shortage, or it can sell electricity to the power grid to generate revenue. The unit price of gas purchased from the natural gas network by the system at time t; For the photovoltaic output of the system at time t; among the eight state variables, and The internal state variables of the system are determined by the actions given by the agent, t, P t PV This is an external state variable of the system, read from smart meters in the environment.

[0021] The calibration includes calibration of battery charging and discharging operation, calibration of heat storage tank charging and discharging operation, and calibration of electric boiler input power. The calibration of battery charging and discharging operation is performed according to the following formula:

[0022]

[0023] In the formula This represents the percentage of the battery's charge / discharge capacity after correction. Improve battery charging efficiency; This refers to the battery discharge efficiency.

[0024] The correction of the heat storage tank's charging and discharging operation is performed according to the following formula:

[0025]

[0026] In the formula This represents the percentage of heat released from the accrued heat storage tank relative to its capacity. Improve the heat storage tank's charging efficiency; The heat release efficiency of the heat storage tank.

[0027] The input power correction of the electric boiler is performed according to the following formula:

[0028]

[0029] In the formula This represents the percentage of the input power of the electric boiler to its rated capacity after correction. This refers to the rated capacity of the thermal storage tank. Rated capacity of the electric boiler; η eb This refers to the energy conversion efficiency of an electric boiler.

[0030] The penalty cost is:

[0031]

[0032] In the formula The penalty cost is used to correct for the resulting penalty; M is the penalty coefficient.

[0033] The operating cost is calculated using the following formula:

[0034]

[0035] In the formula The system operating cost at time t; Electricity purchase and sale for the time-t system. A value greater than zero indicates that the system purchases electricity from the grid, while a value less than zero indicates that the system sells electricity to the grid. The system purchases gas from the natural gas network at time t.

[0036] The system's electricity purchase and sale volume is:

[0037]

[0038] In the formula This refers to the rated capacity of the battery.

[0039] The system's gas purchase volume is:

[0040]

[0041] In the formula η gb The conversion efficiency of the gas-fired boiler;

[0042] Intrinsic state in the system's state variables at the next time step for:

[0043]

[0044] Intrinsic state in the system's state variables at the next time step for:

[0045]

[0046] The next-moment state quantity passed from the system to the agent is represented as:

[0047]

[0048] In the next moment, the external factor state variable in the state quantity is read by the smart meter from the environment.

[0049] The format of the data stored in the experience playback pool is:

[0050]

[0051] In the formula The reward received by the intelligent agent;

[0052] The agent obtains the action 'a' corresponding to the current state through a behavioral network. t (s t ), obtain the value Q(s) of state-behavior pairs through critical networks. t ,a t (s t The action 'a' corresponding to the next state is obtained through the behavioral target network. t+1 (s t+1 The value Q(s) of the next-time state-action pair is obtained by critiquing the target network. t+1 ,a t+1 (s t+1 The agent randomly draws data from the experience replay pool to train the neural network, where the loss function of the critique network is:

[0053]

[0054] The critique network updates by minimizing the loss function, while the behavior network updates via policy gradients. The policy gradient of the behavior network is expressed as:

[0055]

[0056] In the formula, θ aThe network parameters are defined for the behavioral neural network; the critical target network and the behavioral target network are implemented using a soft update algorithm.

[0057] θ Q′ ←τθ Q +(1-τ)θ Q′

[0058] θ a′ ←τθ a +(1-τ)θ a′

[0059] In the formula, θ Q To critique network parameters, θ Q′ To critique the target network parameters, θ a Behavioral network parameters, θ a′ Here, τ represents the target network parameters, and τ represents the soft update magnitude.

[0060] The beneficial effects obtained by this invention are:

[0061] This invention provides a multi-energy system economic scheduling method based on knowledge-assisted deep reinforcement learning. In cases where it is difficult to obtain an accurate model of the multi-energy system, the optimal economic scheduling strategy of the multi-energy system can be obtained in a data-driven manner. The method also protects the system in a safe operating range with the assistance of expert knowledge, thereby improving the system's security and enhancing its economic and efficient operation capabilities.

[0062] Utilizing expert knowledge to assist learning can prevent unsafe behaviors. Expert knowledge does not require an accurate system model; it only needs to obtain the rated parameters of the devices within the system. At the same time, expert knowledge can accelerate the learning speed in unsafe action areas by providing a penalty cost to the agent through closed-loop feedback, and promptly update its own strategy to the safe working area, so that the system can achieve optimal economic benefits and safety. Attached Figure Description

[0063] Figure 1 The present invention provides an economic scheduling framework for multi-energy systems based on knowledge-assisted deep reinforcement learning.

[0064] Figure 2 This is a schematic diagram of the multi-energy system structure of the method of the present invention;

[0065] Figure 3 The method of this invention is an economic scheduling process for multi-energy systems based on knowledge-assisted deep reinforcement learning. Detailed Implementation

[0066] The present invention will now be described in conjunction with the accompanying drawings. The present invention proposes an economic scheduling method for multi-energy systems based on knowledge-assisted deep reinforcement learning, which will be explained in detail below with reference to the accompanying drawings:

[0067] Figure 1 This invention provides an economic scheduling framework for multi-energy systems based on knowledge-assisted deep reinforcement learning. The framework includes:

[0068] 1) A deep deterministic strategy gradient algorithm is used to construct an agent to receive state information from the environment and explore the corresponding optimal unit output;

[0069] 2) Knowledge-assisted reinforcement learning algorithms act as protectors in the middle layer between the agent and the environment. They correct the actions of the agent to ensure that the agent will not endanger the safety of the system equipment during the exploration and trial process. At the same time, they accelerate the learning speed of the agent by feeding back the penalty cost to the agent.

[0070] 3) The environment executes the safety actions generated by the protector and calculates the system's operating cost and the system's state information at the next moment, and transmits the operating cost and the system's state information at the next moment to the agent for learning;

[0071] 4) After the agent completes its learning, it obtains an economic scheduling strategy for a multi-energy system based on knowledge-assisted deep reinforcement learning.

[0072] Figure 2 This is a schematic diagram of the multi-energy system structure of the method of the present invention. The multi-energy system involves three energy forms: electricity, heat, and natural gas. Electricity and heat are coupled by an electric boiler, and natural gas and heat are coupled by a gas boiler. The system utilizes distributed rooftop photovoltaics as a renewable energy source. It also includes batteries and thermal storage tanks for storing electricity and heat. The system and the power grid can engage in bidirectional energy trading. When photovoltaic power generation is insufficient, the system can supplement the energy deficit by discharging from the batteries or purchasing electricity from the grid. When photovoltaic power generation is excessive, the system can choose to charge the batteries or sell electricity to the grid to utilize the energy. The system can also purchase natural gas from the natural gas grid to generate heat to meet user heat demands.

[0073] Figure 3 This invention describes an economic scheduling process for multi-energy systems based on knowledge-assisted deep reinforcement learning. The detailed steps are as follows:

[0074] (1) Obtain the initial state variables of the multi-energy system and pass them to the agent.

[0075] (2) After receiving the state quantity, the agent obtains the corresponding action quantity through its own policy and adds a Gaussian noise to the action quantity for exploration.

[0076] (3) The protector acquires the environmental state quantity and the action quantity generated by the agent. Based on expert knowledge, it judges whether the action quantity exceeds the safe working range of the multi-functional system device. If the action quantity exceeds the safe working range of the device, it corrects it and transmits the corrected action to the environment for execution. At the same time, it feeds back a penalty cost greater than zero to the agent. If the action quantity is within the safe working range of the device, it directly transmits the action quantity to the environment for execution and feeds back a penalty cost equal to zero to the agent.

[0077] (4) After receiving the action quantity from the protector, the environment executes the action to obtain the system's operating cost and acquires the information of the state at the next moment, and then transmits the operating cost and the state quantity at the next moment to the intelligent agent.

[0078] (5) Repeat steps (2), (3) and (4) to store the system state quantity, agent action quantity, penalty cost, running cost and system state quantity at the next moment obtained each time in the agent's experience replay pool. The agent randomly extracts data from the experience replay pool and updates its own policy according to the deep deterministic policy gradient algorithm. After the agent completes learning the specified learning data, the obtained policy is the multi-energy system economic scheduling policy based on knowledge-assisted deep reinforcement learning.

[0079] Specifically, the state variables of the multi-energy system in step (1) are:

[0080]

[0081] In the formula s t This represents the system state vector at time t; t is the time the system is at. The percentage of energy contained in the system battery at time t relative to the battery's rated capacity; The percentage of energy contained in the system's thermal storage tank at time t relative to the tank's rated capacity; Let t be the system electrical load; Let t be the system heat load; The unit electricity price for the system's transaction with the power grid at time t is such that the system can purchase electricity from the power grid to make up for its own power shortage, or it can sell electricity to the power grid to generate revenue. The unit price of natural gas purchased from the network by the system at time t; P t PV For the photovoltaic output of the system at time t; among the eight state variables, and The internal state variables of the system are determined by the actions given by the agent, t, P t PV This is an external state variable of the system, read from smart meters in the environment.

[0082] Specifically, in step (2), the agent contains four neural networks: a behavior network, a critique network, a behavior target network, and a critique target network. The behavior network and the behavior target network have the same architecture, and the critique network and the critique target network have the same architecture. All four networks are generated by randomly initializing weights. The behavior network takes the system state variables as input and adds Gaussian noise to the network output to obtain a continuous action.

[0083]

[0084] In the formula a t (s t () represents the new motion quantity after adding Gaussian noise to the original motion quantity. N represents the raw action quantity output by the behavior network; t (0,λ k σ 2 ) represents Gaussian noise, λ k σ 2 Let be the variance of the Gaussian noise, λ∈(0,1) be the attenuation factor, and k be the current iteration number. The Gaussian noise added to the action will gradually decrease during the agent's iterative learning process; a t (s t Specifically, it is expressed as:

[0085]

[0086] In the formula The percentage of the input power of the electric boiler to its rated power at time t; and For the variable {0,1}, This indicates that the battery is charging. This indicates that the battery is discharging. This indicates that the heat storage tank is being filled with heat. This indicates that the heat storage tank is releasing heat; Let t be the percentage of battery capacity that has been charged or discharged at time t, and let the state of charge / discharge change from t to t. Decide; Let t be the percentage of heat charge / discharge in the thermal storage tank relative to its capacity, and let the heat charge / discharge state change from... The agent's initial policy is a random policy, and the agent's policy will be updated through iterative learning.

[0087] Specifically, in step (3), the protector will correct and penalize actions that exceed the system's safe operating range. The method for correcting battery charging and discharging actions is as follows:

[0088]

[0089] In the formula This represents the percentage of the battery's charge / discharge capacity after correction. Improve battery charging efficiency; For battery discharge efficiency;

[0090] The method for calibrating the charging and discharging action of the heat storage tank is as follows:

[0091]

[0092] In the formula This represents the percentage of heat released from the accrued heat storage tank relative to its capacity. Improve the heat storage tank's charging efficiency; The heat release efficiency of the heat storage tank;

[0093] The method for correcting the input power of an electric boiler is as follows:

[0094]

[0095] In the formula This represents the percentage of the input power of the electric boiler to its rated capacity after correction. This refers to the rated capacity of the thermal storage tank. Rated capacity of the electric boiler; η eb The energy conversion efficiency of the electric boiler; the penalty cost due to correction is:

[0096]

[0097] In the formula The penalty cost is used to correct for the resulting penalty; M is the penalty coefficient.

[0098] Specifically, the system operating cost in step (4) is:

[0099]

[0100] In the formula The system operating cost at time t; Electricity purchase and sale for the time-t system. A value greater than zero indicates that the system purchases electricity from the grid, while a value less than zero indicates that the system sells electricity to the grid. The system purchases gas from the natural gas network at time t.

[0101] The system's electricity purchase and sale volume is:

[0102]

[0103] In the formula This refers to the rated capacity of the battery.

[0104] The system's gas purchase volume is:

[0105]

[0106] In the formula η gb The conversion efficiency of the gas-fired boiler;

[0107] Intrinsic state in the system's state variables at the next time step for:

[0108]

[0109] Intrinsic state in the system's state variables at the next time step for:

[0110]

[0111] The next-moment state quantity passed from the system to the agent is represented as:

[0112]

[0113] In the next moment, the external factor state variable in the state quantity is read by the smart meter from the environment.

[0114] Specifically, the format of the data stored in the experience replay pool in step (5) is as follows:

[0115]

[0116] In the formula The reward received by the agent; the agent obtains the action 'a' corresponding to its current state through the behavioral network. t (s t ), obtain the value Q(s) of state-behavior pairs through critical networks. t ,a t (s t The action 'a' corresponding to the next state is obtained through the behavioral target network. t+1 (s t+1 The value Q(s) of the next-time state-action pair is obtained by critiquing the target network. t+1 ,a t+1 (s t+1 )).

[0117] The agent randomly draws data from the experience replay pool to train the neural network, where the loss function of the critique network is:

[0118]

[0119] The critique network updates by minimizing the loss function, while the behavior network updates by using the policy gradient, which is expressed as:

[0120]

[0121] In the formula, θa The network parameters are defined for the behavioral neural network; the critical target network and the behavioral target network are implemented using a soft update algorithm.

[0122] θ Q′ ←τθ Q +(1-τ)θ Q′

[0123] θ a′ ←τθ a +(1-τ)θ a′

[0124] In the formula, θ Q To critique network parameters, θ Q′ To critique the target network parameters, θ a Behavioral network parameters, θ a′ Here, τ represents the target network parameters, and τ represents the soft update magnitude.

[0125] In summary, this invention constructs an agent using a deep deterministic policy gradient algorithm to receive state information from the environment and explore the corresponding optimal unit output. A knowledge-assisted reinforcement learning algorithm is used as a protector, placed in the middle layer between the agent and the environment. This protector corrects the agent's actions to ensure that the agent does not endanger the safety of the system equipment during exploration and trial. Simultaneously, by feeding back penalty costs to the agent, the learning speed is accelerated. The environment executes the safety actions generated by the protector and calculates the system's operating cost and the system's state information at the next moment. This operating cost and the system state information at the next moment are then transmitted to the agent for learning. After the agent completes its learning, a multi-energy system economic scheduling strategy based on knowledge-assisted deep reinforcement learning is obtained. This invention, through its knowledge-assisted deep reinforcement learning-based multi-energy system economic scheduling method, can obtain the optimal economic scheduling strategy for multi-energy systems in a data-driven manner when it is difficult to obtain an accurate model of the multi-energy system. It also protects the system within a safe operating range using expert knowledge assistance, improving system safety and enhancing the system's economic and efficient operation capabilities.

Claims

1. An economic scheduling method for multi-energy systems based on knowledge-assisted deep reinforcement learning, characterized in that, Includes the following steps: (1) Obtain the initial state variables of the multi-energy system and pass them to the agent; (2) After receiving the state quantity, the agent obtains the corresponding action quantity through its own policy and adds a Gaussian noise to the action quantity for exploration; (3) The protector acquires the environmental state quantity and the action quantity generated by the agent. Based on expert knowledge, it judges whether the action quantity exceeds the safe working range of the multi-functional system device. If the action quantity exceeds the safe working range of the system, it corrects it and transmits the corrected action quantity to the environment for execution, while feeding back a penalty cost greater than zero to the agent. If the action quantity is within the safe working range of the device, it directly transmits the action quantity to the environment for execution and feeds back a penalty cost equal to zero to the agent. (4) After receiving the action quantity from the protector, the environment executes the action to obtain the system's operating cost and acquires the state quantity at the next moment, and then transmits the operating cost and the state quantity at the next moment to the agent. (5) Repeat steps (2), (3) and (4) to store the system state quantity, agent action quantity, penalty cost, running cost and system state quantity at the next moment obtained each time in the agent's experience replay pool. The agent randomly extracts data from the experience replay pool and updates its own policy according to the deep deterministic policy gradient algorithm. The intelligent agent comprises four neural networks: a behavior network, a critique network, a behavior-target network, and a critique-target network. The behavior network and the behavior-target network have the same architecture, as do the critique network and the critique-target network. All four networks are generated using a randomized weight initialization method. The behavior network uses system state variables... The network takes a Gaussian noise as input and adds it to the network output to obtain a continuous action. The behavior network model is as follows: ， In the formula The new motion quantity generated by superimposing Gaussian noise on the original motion quantity; The raw motion quantities output by the behavioral network; It is Gaussian noise; The variance of the Gaussian noise; It is the attenuation factor; This represents the current iteration number; The new action quantities generated by the behavioral network are specifically represented as follows: ， In the formula For a moment The percentage of input power to rated power of the electric boiler; and for variable, This indicates that the battery is charging. This indicates that the battery is discharging. This indicates that the heat storage tank is being filled with heat. This indicates that the heat storage tank is releasing heat; For a moment The percentage of battery charge / discharge capacity relative to the battery's capacity; the state of charge / discharge from... Decide; For a moment The percentage of heat charge / discharge in the thermal storage tank relative to its capacity; the state of charge / discharge changes from... The agent's initial policy is a random policy, and the agent's policy will be updated through iterative learning. The state variables of the multi-energy system in step (1) are: ， In the formula Indicates time System state vector; The current state of the system; For a moment The percentage of energy contained in the system battery relative to the battery's rated capacity; For a moment The percentage of energy contained in the system's heat storage tank relative to its rated capacity; For a moment System electrical load; For a moment System heat load; For a moment The system trades electricity with the power grid at a unit price. The system can purchase electricity from the power grid to make up for its own power shortage, or it can sell electricity to the power grid to generate revenue. For a moment The system purchases gas from the natural gas network at the unit price. For a moment System photovoltaic output; Of the eight state variables, and These are internal state variables of the system, determined by the actions given by the agent. , , , , , These are external state variables of the system, read from smart meters in the environment; The correction in step (3) includes correction of battery charging and discharging operation, correction of heat storage tank charging and discharging operation, and correction of electric boiler input power. The correction of battery charging and discharging operation is performed according to the following formula: ， In the formula This represents the percentage of the battery's charge / discharge capacity after correction. Improve battery charging efficiency; For battery discharge efficiency; The correction of the heat storage tank's charging and discharging operation is performed according to the following formula: ， In the formula This represents the percentage of heat released from the accrued heat storage tank relative to its capacity. Improve the heat storage tank's charging efficiency; The heat release efficiency of the heat storage tank; The input power correction of the electric boiler is performed according to the following formula: ， In the formula This represents the percentage of the input power of the electric boiler to its rated capacity after correction. This refers to the rated capacity of the thermal storage tank. This refers to the rated capacity of the electric boiler. This refers to the energy conversion efficiency of an electric boiler.

2. The multi-energy system economic scheduling method based on knowledge-assisted deep reinforcement learning according to claim 1, characterized in that, The penalty cost is: ， In the formula To correct the resulting penalty costs; This is the penalty coefficient.

3. The multi-energy system economic scheduling method based on knowledge-assisted deep reinforcement learning according to claim 1, characterized in that, The operating cost is calculated using the following formula: ， In the formula For a moment System operating costs; For a moment Electricity purchased and sold by the system A value greater than zero indicates that the system purchases electricity from the grid, while a value less than zero indicates that the system sells electricity to the grid. For a moment The system purchases gas from the natural gas network; The system's electricity purchase and sale volume is: ， In the formula This refers to the rated capacity of the battery. The system's gas purchase volume is: ， In the formula The conversion efficiency of the gas-fired boiler; Intrinsic state in the system's state variables at the next time step for: ， Intrinsic state in the system's state variables at the next time step for: ， The next-moment state quantity passed from the system to the agent is represented as: ， The external state variables in the next state variable are read from the environment by the smart meter.

4. The multi-energy system economic scheduling method based on knowledge-assisted deep reinforcement learning according to claim 1, characterized in that, The format of the data stored in the experience playback pool is: ， In the formula The reward received by the intelligent agent; The agent obtains the action corresponding to the current state through a behavioral network. To obtain the value of state-behavior pairs through critical networks The action corresponding to the next time step is obtained through the behavior target network. The value of the next-moment state-action pair is obtained by critiquing the target network. The agent randomly draws data from the experience replay pool to train the neural network, where the loss function of the critique network is: ， The critique network updates by minimizing the loss function, while the behavior network updates via policy gradients. The policy gradient of the behavior network is: ， In the formula, The network parameters are defined for the behavioral neural network; the critical target network and the behavioral target network are implemented using a soft update algorithm. ， ， In the formula, To critique network parameters, To critique the target network parameters, Behavioral network parameters For the target network parameters of the behavior, This refers to the soft update range.

Images