Multi-region integrated energy system optimization scheduling method, system, equipment and medium

Abstract

The invention discloses a multi-region integrated energy system optimization scheduling method, system and device and a medium, and the method comprises the steps: dividing a power distribution network into a plurality of regions, setting an intelligent agent for an integrated energy system of each region, enabling each intelligent agent to update a target network through a reinforcement learning algorithm, obtaining a target updated network; quantifying importance weights of parameters in the target update networks, selecting a plurality of elite agents from the agents, and performing global aggregation on the parameters in the target update networks of the elite agents according to the importance weights to obtain aggregation parameters; and sharing the aggregation parameters among all intelligent agents through a soft update method, so that the intelligent agents after sharing the parameters solve an objective function of the power distribution network, an optimal scheduling scheme for the multi-region integrated energy system is obtained, the power distribution network is controlled to execute, and the accuracy of an integrated energy optimal scheduling result is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of energy optimization and scheduling, and in particular to a multi-regional integrated energy system optimization and scheduling method, system, equipment and medium. Background Art

[0002] With the widespread application of agent learning and multi-agent systems in complex environments, traditional single optimization methods have become difficult to meet the increasingly complex and changeable task requirements in energy optimization scheduling scenarios. Relying solely on elite selection mechanisms to identify excellent agents, they are unable to fully utilize the potential of other agents in the system, resulting in inaccurate optimization scheduling results for comprehensive energy.

[0003] It can be seen that how to improve the accuracy of comprehensive energy optimization scheduling results has become a technical problem that technical personnel in this field need to solve urgently. Summary of the invention

[0004] The present invention provides a multi-region integrated energy system optimization scheduling method, system, equipment and medium to solve the problem of how to improve the accuracy of the integrated energy optimization scheduling results.

[0005] In order to solve the above technical problems, the first aspect of the present invention provides a multi-region integrated energy system optimization scheduling method, comprising: The distribution network is divided into a plurality of regions, and an intelligent agent is set for the integrated energy system of each region, so that each intelligent agent updates its own target network through a reinforcement learning algorithm to obtain a target updated network; Quantifying the importance weights of the parameters in each of the target update networks, and selecting a number of elite agents from each of the agents, and globally aggregating the parameters in the target update networks of each of the elite agents according to the importance weights to obtain aggregated parameters; The aggregation parameters are shared among all intelligent agents through a soft update method, so that each intelligent agent after sharing the parameters can solve the objective function of the distribution network, obtain an optimized scheduling plan for the multi-regional integrated energy system and control the execution of the distribution network.

[0006] As one preferred solution, the method of enabling each of the intelligent agents to update its own target network through a reinforcement learning algorithm to obtain a target update network includes: Initializing the environment and algorithm parameters of each of the agents, and constructing a playback buffer for each of the agents; Constructing a state set and an action set based on the integrated energy system of each of the regions, and constructing a reward function according to the economic cost of the distribution network, so that each of the intelligent agents interacts with the initialized environment through a deep deterministic policy gradient algorithm based on the initialized algorithm parameters, obtains a state value, an action value and a reward value and stores them in the playback buffer; The target network of each of the agents is trained and updated using the data in the playback buffer to obtain a target update network of each of the agents.

[0007] As one of the preferred solutions, the quantification of the importance weights of the parameters in each target update network includes: Performing forward propagation on each of the target update networks to quantify the loss function under the current strategy, and quantifying the gradient of each parameter in each of the target update networks according to the loss function under the current strategy; The absolute values ​​of the gradients are averaged over multiple time steps to obtain the importance weight of each parameter in each target update network.

[0008] As one preferred solution, the step of selecting a number of elite agents from the agents includes: Standardizing the reward value corresponding to each of the intelligent agents to obtain a standardized reward value; Initialize the weight and learning rate of each of the intelligent agents, and quantify the selection probability of each of the intelligent agents according to the initialization result; Selecting an elite intelligent agent according to each of the selection probabilities, and updating the weight of the selected elite intelligent agent according to the standardized reward value and the operation support of the distribution network; Based on the updated self-weight, each of the agents is controlled to iteratively execute the quantization step of the selection probability and the selection step of the elite agent until a preset iteration round is reached, and a number of selected elite agents are output.

[0009] As one of the preferred solutions, the parameters in the target update network of each elite agent are globally aggregated according to each importance weight to obtain the aggregated parameters, including: Each of the importance weights is used as a parameter aggregation weight of each of the elite agents to globally aggregate the parameters in the target update network of each of the elite agents to obtain the aggregated parameters.

[0010] As one of the preferred solutions, the sharing of the aggregation parameters is achieved by the following formula: In the formula, , Update the network parameters for the kth agent before and after sharing the aggregated parameters. is the soft update ratio; is the aggregation parameter of the jth parameter; is the jth parameter; D select For elite intelligent agents; is element-wise multiplication.

[0011] As one of the preferred solutions, the intelligent agents after sharing parameters solve the objective function of the distribution network, obtain the optimal dispatching plan for the multi-regional integrated energy system and control the distribution network to execute, including: Controlling the target update network of each of the intelligent agents to share the aggregation parameters to obtain a final target network, and performing performance evaluation on each of the final target networks; When the evaluation result does not reach the preset target, repeatedly performing the network updating step and the parameter sharing step based on each of the final target networks until the evaluation result of each updated final target network reaches the preset target; Under the operating constraints of the integrated energy system in each of the regions, the objective function is constructed with the total cost of the distribution network as the target, and the objective function is solved by each of the intelligent agents when the evaluation result reaches the preset target, so as to obtain an optimized scheduling plan for the multi-region integrated energy system to control the execution of the distribution network.

[0012] A second aspect of the present invention provides a multi-region integrated energy system optimization and dispatching system, comprising: A network update module is used to divide the distribution network into a number of areas, and set an intelligent agent for the integrated energy system of each area, so that each intelligent agent updates its own target network through a reinforcement learning algorithm to obtain a target updated network; A parameter aggregation module, used to quantify the importance weights of the parameters in each target update network, and select a number of elite agents from each of the agents, and globally aggregate the parameters in the target update network of each of the elite agents according to the importance weights to obtain aggregated parameters; The optimization scheduling module is used to share the aggregation parameters among all intelligent agents through a soft update method, so that each intelligent agent after sharing the parameters can solve the objective function of the distribution network, obtain an optimized scheduling plan for the multi-regional integrated energy system and control the execution of the distribution network.

[0013] The third aspect of the present invention provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the multi-regional integrated energy system optimization scheduling method as described above is implemented.

[0014] The fourth aspect of the present invention provides a computer-readable storage medium, which includes a stored computer program, wherein when the device where the computer-readable storage medium is located executes the computer program, the multi-regional integrated energy system optimization scheduling method as described above is implemented.

[0015] Compared with the prior art, the embodiments of the present invention have the following advantages: (1) Each intelligent agent is responsible for the integrated energy system in its area. Through reinforcement learning, it continuously optimizes its own dispatching strategy, which can reduce the burden of central control and improve the overall dispatching efficiency. The elite selection mechanism enables the distribution network system to concentrate learning resources, strengthen and optimize the best performing strategy at the fastest speed, and introduce an elite intelligent agent parameter sharing mechanism, so that the parameters of the intelligent agent with excellent performance are shared with other intelligent agents, promoting the improvement of the learning efficiency and convergence speed of the whole system. (2) In view of the fact that the impact of different parameters on model performance may be significantly different during the sharing process, the importance weights of the parameters in the target update network of each intelligent agent are introduced to dynamically evaluate the contribution of each parameter to performance optimization, and the aggregated parameters are shared among all intelligent agents, so that each intelligent agent can learn the globally optimal dispatching strategy, which helps to improve the intelligence level and dispatching efficiency of the entire distribution network. In addition, the soft update strategy is combined with parameter sharing, which can not only alleviate the instability caused by direct parameter transmission, but also improve the overall adaptability and robustness of the system in a complex dynamic environment. (3) By combining reinforcement learning algorithms, elite selection mechanisms and gradient importance weights, the collaborative efficiency and optimization performance of multi-agent systems in complex and changing environments can be significantly improved, providing more economical and efficient solutions for scenarios such as multi-regional integrated energy scheduling, thereby improving the accuracy of integrated energy optimization scheduling results in distribution networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the implementation mode will be briefly introduced below. Obviously, the drawings described below are only some implementation modes of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flow chart of a multi-region integrated energy system optimization scheduling method provided by a certain embodiment of the present invention; Figure 2 It is a structural diagram of a comprehensive energy system model in a single region provided by a certain embodiment of the present invention; Figure 3is a flow chart of a multi-region integrated energy system optimization scheduling method provided by another embodiment of the present invention; Figure 4 It is a structural diagram of a multi-region integrated energy system optimization and dispatching system provided by a certain embodiment of the present invention; Figure 5 It is a structural diagram of an electronic device provided by a certain embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings and embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. The purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] In the description of this application, the terms "first", "second", "third", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", "third", etc. may explicitly or implicitly include one or more of the feature. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0020] In the description of the present application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be a connection between the two components. The terms "vertical", "horizontal", "left", "right", "upper", "lower" and similar expressions used herein are for illustrative purposes only, and do not indicate or imply that the system or component referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0021] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those commonly understood by those skilled in the art. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood by specific circumstances.

[0022] In one embodiment, if Figure 1 As shown, the first aspect of the present invention provides a multi-region integrated energy system optimization scheduling method, comprising: S1. Divide the distribution network into several areas, and set an intelligent agent for the integrated energy system of each area, so that each intelligent agent updates its own target network through a reinforcement learning algorithm to obtain a target updated network; Specifically, the present invention collects historical load data, energy production data (such as solar energy, wind energy), line parameters, etc. of the distribution network, and uses clustering algorithms (such as K-means) or graph theory methods (such as segmentation based on network topology) to divide the distribution network into several areas, so as to ensure that the energy supply and demand in each area is relatively balanced and easy to manage; wherein, the structure of the comprehensive energy system model in a single area is as follows Figure 2 As shown, it includes comprehensive energy such as wind power, photovoltaics, electric energy storage, generators, natural gas, as well as cold, heat, and electricity loads, and sets an independent intelligent agent for each area so that the intelligent agent can independently train its local model in a distributed environment.

[0023] In one embodiment, the step of enabling each of the intelligent agents to update its own target network through a reinforcement learning algorithm to obtain a target update network includes: Initializing the environment and algorithm parameters of each of the agents, and constructing a playback buffer for each of the agents; Constructing a state set and an action set based on the integrated energy system of each of the regions, and constructing a reward function according to the economic cost of the distribution network, so that each of the intelligent agents interacts with the initialized environment through a deep deterministic policy gradient algorithm based on the initialized algorithm parameters, obtains a state value, an action value and a reward value and stores them in the playback buffer; The target network of each of the agents is trained and updated using the data in the playback buffer to obtain a target update network of each of the agents.

[0024] Specifically, the present invention uses the deep deterministic policy gradient algorithm (DDPG) to optimize the model of the agent in each area, and interacts with the model parameters through the edge server, specifically including: First, the environment of each agent is initialized, that is, the learning rate of the Actor network and the Critic network, the target network update rate and the exploration noise are set, and the algorithm parameters of each agent are initialized, including a Critic network and an Actor network, and a playback buffer is established for each agent to store experience data; and in order to match the operating characteristics and goals of the integrated energy system, the state set, action set and reward function are constructed based on the integrated energy system of each region; Among them, The state set includes the electric load demand, heat load demand and cooling load demand, photovoltaic power generation, the state of the energy storage unit, the current dispatch interval, and the electricity price at the future time in the integrated energy system of a certain area, so as to fully reflect the current status of the distribution network area. Then the state of the integrated energy system of a certain area can be expressed as: In the formula, s is the state; k is the agent, which can also represent the region; t is the time; is the electricity price at time t; is the capacity of the energy storage device in region k at time t; The power load of the user; is the operating power of the generator in region k at time t; is the wind power generation power in region k at time t; is the photovoltaic power generation power in region k at time t; is the output power of the generator.

[0025] The action set should cover all possible actions that the agent can take, such as adjusting energy distribution, controlling equipment power, etc. However, since the action dimension can be reduced through equality constraints in economic scheduling, the action set in the integrated energy system of a certain area can be designed as: In the formula, For action; , and They are the climbing power of the energy storage equipment, the combined cooling, heating and power unit and the generator respectively.

[0026] The reward function in the integrated energy system of each region consists of two parts: one part is used to optimize the economic cost of the integrated energy system dispatch in the region, and the other part is used to punish the consequences of the agent violating constraints or taking inappropriate actions. Therefore, the reward function expression is as follows: In the formula, is the reward function; It is the negative value of the total economic cost of the integrated energy system in a single region (it can also be understood as the kth region); is the penalty for violating the constraint; is the power generation cost of generators in a single region; The equipment loss cost of the unit in a single area; Maintaining the optimal cost for energy storage equipment in a single region; The cost of purchasing electricity from the main power grid in a single region; is the natural gas purchase cost to meet the demand of trigeneration units in a single region; θ1, θ2, θ3, θ4, and θ5 are the numbers of corresponding weights respectively; is the penalty weight; J is the constraint set. If the constraint is violated, the size of the constraint exceeded is calculated; , , They are the output power of the integrated energy system in a single area and its minimum and maximum values.

[0027] Then, the local training process begins, where each agent independently performs decision-making and learning operations through the DDPG algorithm; each agent collects experience data by interacting with the environment. , and the current state value in the formula , Current action value , Current reward value and the next state value Stored in the experience replay buffer, it provides necessary training data support for the algorithm's network update.

[0028] During the training process, the update of the Actor network is to maximize the expected reward by adjusting the policy function. Therefore, in each update, the parameters of the Actor network are updated by calculating the gradient of the Q network to the action generated by the current policy, which is specifically expressed by the following formula: In the formula, Parameters of the Actor network; is the Actor network learning rate; is the gradient operator; is the expected total return under the current state; are the parameters of the Q network; is the expected result of the samples in the experience replay buffer; is the policy function.

[0029] The target value generated by the target network is used to update the parameters of the Critic network, and the Critic network is updated by minimizing the loss function. The update process is expressed by the following formula: In the formula, is the target value; is the attenuation coefficient; is the expected total reward in the next state; is the loss function; The value of the action taken in the next state; Actions generated for the Actor network; is Gaussian noise; is the standard deviation of the noise; c is the clipping threshold.

[0030] Finally, the Critic network and the Actor network are updated by soft updating to reduce instability in the learning process. The update method is as follows: In the formula, is the target network update rate.

[0031] In addition, the target update network in the present invention does not refer to a specific network, but the final network obtained after each agent is trained by the reinforcement learning algorithm. The present invention can improve the energy management efficiency of the distribution network, reduce operating costs, and enhance the adaptability of the distribution network system by setting up agents for the integrated energy system in each area of ​​the distribution network and applying the DDPG algorithm for training and updating.

[0032] S2, quantifying the importance weights of the parameters in each of the target update networks, and selecting a number of elite agents from each of the agents, and globally aggregating the parameters in the target update networks of each of the elite agents according to the importance weights to obtain aggregated parameters; In one embodiment, the quantifying the importance weights of the parameters in each target update network includes: Performing forward propagation on each of the target update networks to quantify the loss function under the current strategy, and quantifying the gradient of each parameter in each of the target update networks according to the loss function under the current strategy; The absolute values ​​of the gradients are averaged over multiple time steps to obtain the importance weight of each parameter in each target update network.

[0033] Specifically, the present invention forward-propagates the Critic network of each agent by replaying the data in the buffer, obtains the current prediction result, and calculates the loss function (such as mean square error loss, cross entropy loss, etc.) under the current strategy to quantify the difference between the prediction result and the true label. Then, through the back-propagation algorithm, the gradient of the loss function is back-propagated from the output layer to the input layer, and the contribution of each parameter to the loss function, that is, the gradient of the parameter, is calculated during the back-propagation process. Finally, the absolute value of each gradient is averaged over multiple time steps. The average value obtained reflects the average influence of the parameter in the optimization process, that is, the importance weight of the parameter. In this way, the importance weight of each parameter in the Critic network can be obtained; wherein, the importance weight is expressed by the following formula: In the formula, is the importance weight of the i-th parameter in the Critic network; T is the total duration of iteration before each parameter exchange; is the loss function of the Critic network of agent k at time t; is the i-th parameter in the Critic network.

[0034] The present invention can identify the parameters that have a greater impact on the loss function by quantifying the importance weight of each parameter, so that in the subsequent parameter update process, more attention can be paid to these important parameters, thereby improving the training efficiency; the calculation of the importance weight takes into account the gradient information within multiple time steps, which helps to capture the long-term impact of the parameters and helps the model learn more generalized features during the training process, thereby improving the generalization ability of the model; according to the importance weight of the parameter, the step size or learning rate of the parameter update can be adjusted. For parameters with higher importance, a smaller step size can be used for fine adjustment; for parameters with lower importance, a larger step size can be used for rapid update, which helps to balance the training speed and stability of the model; at the same time, it also provides additional explanatory information for the model. By analyzing the importance weight of the parameters, it can be understood which features or information the model mainly relies on when making predictions, which is convenient for researchers to better understand the decision-making process of the model and make targeted improvements.

[0035] In one embodiment, the step of selecting a plurality of elite agents from each of the agents comprises: Standardizing the reward value corresponding to each of the intelligent agents to obtain a standardized reward value; Initialize the weight and learning rate of each of the intelligent agents, and quantify the selection probability of each of the intelligent agents according to the initialization result; Selecting an elite intelligent agent according to each of the selection probabilities, and updating the weight of the selected elite intelligent agent according to the standardized reward value and the operation support of the distribution network; Based on the updated self-weight, each of the agents is controlled to iteratively execute the quantization step of the selection probability and the selection step of the elite agent until a preset iteration round is reached, and a number of selected elite agents are output.

[0036] Specifically, the reward value of each agent is standardized by the following formula: In the formula, is the standardized reward value; is the reward value of the agent; It is the reward value corresponding to when the actions of the agent at all times are 0; the present invention establishes a reward value threshold. When the standardized reward value is greater than the reward value threshold, it means that the strategy trained by the agent is positive, which will increase the reward of the agent; when the standardized reward value is less than the reward value threshold, it means that the strategy trained by the agent is negative, which will reduce the reward of the agent.

[0037] Then initialize the learning rate of each agent and the corresponding self-weight so that the self-weight, where the self-weight reflects the relative importance or influence of the agent in the current iteration, and the learning rate is used to control the step size or speed of the agent when updating its own weight, and calculate the selection probability corresponding to each agent through its own weight; then select the elite agent according to the calculated selection probability, and update the self-weight of the selected elite agent through the standardized reward value and the operation support of the distribution network, and iterate the quantization step of the selection probability and the selection step of the elite agent using the updated self-weight of each agent until the preset iteration round is reached, and output the selected elite agents; where the selection probability is calculated by the following formula: In the formula, is the selection probability of the agent in the nth round of iteration; is the update ratio; is the weight corresponding to agent k; N is the total number of agents; The preset iteration round.

[0038] The updating process of its own weight is expressed by the following formula: Where D select is the selected agent; The operation support of the distribution network; is the selection probability of the selected agent in the nth round of iteration.

[0039] The present invention ensures that agents are compared and evaluated on a unified scale by standardizing reward values ​​and iteratively updating weights, while the selection process of elite agents promotes the retention and further development of excellent agents, which helps to improve the performance of the entire agent group. The operation support of the distribution network is introduced as an additional evaluation indicator, so that when updating the weight, the agent not only considers the performance of the current task, but also considers its contribution or adaptability to the distribution network, which helps the agent to better adapt to the specific environment and needs of the distribution network. The convergence speed of the agent performance is accelerated by iteratively executing the quantization of the selection probability, the selection of elite agents and the updating of their own weights, and the preset iteration rounds ensure that the algorithm reaches a stable state within a reasonable time and outputs the optimal agent group.

[0040] In one embodiment, the globally aggregating the parameters in the target update network of each elite agent according to each importance weight to obtain the aggregated parameters includes: Each of the importance weights is used as a parameter aggregation weight of each of the elite agents to globally aggregate the parameters in the target update network of each of the elite agents to obtain the aggregated parameters.

[0041] Specifically, in the reinforcement learning algorithm, parameter sharing is a key step, that is, the global aggregation of the parameters of the elite agents is performed in an edge server, and the aggregated parameters are sent to all agents. At the same time, the parameter importance weights are used to enhance the update of key parameters and soft updates are adopted to reduce the potential adverse factors caused by parameter transfer.

[0042] S3, sharing the aggregation parameters among all intelligent agents through a soft update method, so that each intelligent agent after sharing the parameters solves the objective function of the distribution network, obtains an optimal scheduling plan for the multi-regional integrated energy system and controls the distribution network to execute; In one embodiment, the sharing of the aggregation parameters, that is, the sharing mechanism of the critic network parameters of each agent can be expressed as: In the formula, , Update the network parameters for the kth agent before and after sharing the aggregated parameters. is the soft update ratio; is the aggregation parameter of the jth parameter; is the jth parameter; D select For elite intelligent agents; is element-wise multiplication.

[0043] In addition, the present invention can also control each elite agent to upload its local parameter sample to a central node (such as an enhanced agent or a collaborative platform), so that the central node can aggregate these parameter samples, calculate the center point of all parameter samples, and the Mahalanobis distance of each parameter sample to the center point; then remove the parameter samples whose distance from the center point exceeds a preset distance threshold, that is, eliminate abnormal and outlier parameters, and perform weighted averaging on the remaining parameter samples, and the aggregated parameters can also be obtained; then the central node returns the aggregated parameter results to each agent, and after receiving the aggregated parameters, each agent does not directly replace the local parameters, but adds the aggregated parameters to its own local parameters with its corresponding importance weight as a coefficient, thereby achieving a smooth transition of the parameters.

[0044] In one embodiment, the process of solving the objective function of the distribution network by each of the intelligent agents after sharing parameters, obtaining an optimal dispatching scheme for the multi-regional integrated energy system and controlling the distribution network to execute the scheme includes: Controlling the target update network of each of the intelligent agents to share the aggregation parameters to obtain a final target network, and performing performance evaluation on each of the final target networks; When the evaluation result does not reach the preset target, repeatedly performing the network updating step and the parameter sharing step based on each of the final target networks until the evaluation result of each updated final target network reaches the preset target; Under the operating constraints of the integrated energy system in each of the regions, the objective function is constructed with the total cost of the distribution network as the target, and the objective function is solved by each of the intelligent agents when the evaluation result reaches the preset target, so as to obtain an optimized scheduling plan for the multi-region integrated energy system to control the execution of the distribution network.

[0045] Specifically, after each round of parameter sharing, each intelligent agent will perform a performance evaluation on its final target network in order to further optimize the strategy. If the evaluation result does not reach the preset target (such as the accuracy rate does not reach 90%, or the loss function value does not drop below 0.1), the network update step and the parameter sharing step are repeated based on these final target networks; multiple iterations are performed until the evaluation results of each updated final target network reach the preset target; and in the integrated energy system of each region, considering its operating constraints, the objective function is constructed with the total cost of the distribution network as the target; wherein the objective function is the total cost of the region divided by the distribution network , specifically including the cost of generator power generation, unit equipment loss, the cost of maintaining the optimal point of energy storage equipment, the cost of purchasing electricity from the main power grid and the integrated energy system, and the cost of purchasing natural gas from the natural gas grid. The various costs contained in a single region are expressed by the following formula: In the formula, a kj , b kj and c kj are the cost coefficients of the generators in region k respectively; is the output power of the jth generator in region k at time t; is the optimal output power of the jth generator in region k; is the charging and discharging power of the i-th energy storage device in region k at time t; is the maximum service life of the i-th energy storage device; is the optimal capacity of the i-th energy storage device in region k at time t; is the electricity price at time t; is the power traded between the corresponding region in region k and the main grid at time t. A value greater than 0 indicates selling power, and a value less than 0 indicates purchasing power; For natural gas transaction costs; is the input power of the CCHP unit in area k at time t.

[0046] Its operating constraints include: The operation constraints of the CCHP unit are expressed by the following formula: In the formula, , and are the electricity, heat and cooling power outputs of the CCHP unit at time t respectively; , and They are the power generation efficiency, heat-to-electricity ratio and cooling-to-electricity ratio of the trigeneration unit respectively.

[0047] The upper and lower limit operation constraints and ramp constraints of the CCHP unit are as follows: In the formula, and They are the minimum and maximum input power limits of the CCHP unit respectively; and They are the sliding rate and climbing rate of the CCHP unit respectively.

[0048] Electric boiler operation constraints , expressed by the following formula: In the formula, is the efficiency of the electric boiler; is the output power of the electric boiler at time t.

[0049] The thermal power balance constraint of the electric boiler is expressed by the following formula: In the formula, is the heat load at time t; The operating efficiency of the electric boiler; It is the thermal power output of the combined cooling, heating and power unit.

[0050] Electric Refrigerator Operation Constraints , expressed by the following formula: In the formula, is the efficiency of the electric refrigerator; is the output power of the electric refrigerator at time t.

[0051] The cooling power balance constraint of the electric refrigerator is expressed by the following formula: In the formula, is the cooling load at time t; is the operating efficiency of the refrigerant; is the cooling power output of the combined heating, cooling and power unit at time t.

[0052] The upper and lower limit operation constraints and climbing constraints of the generator are expressed by the following formula: In the formula, and are the minimum and maximum output power limits of the generator respectively; and are the landslide rate and climbing rate of the generator respectively.

[0053] The state of charge constraint of the energy storage system is expressed as follows: In the formula, is the state of charge of the energy storage system at time t; and are the lower and upper limits of SoC respectively.

[0054] The SoC at the next moment is expressed as: In the formula, is the charge / discharge power at time t; is the capacity of the energy storage system; is the time interval.

[0055] The charging and discharging power limits and ramp constraints are as follows: In the formula, and are the minimum and maximum output power limits of the energy storage device, respectively; and are the landslide rate and climbing rate of the energy storage equipment respectively.

[0056] Constraints on internal electricity transactions in integrated energy systems It should be expressed by the following formula: The upper and lower limits of the power transmitted to the integrated energy system of the main power grid are as follows.

[0057] In the formula, The upper limit of power transmitted to the main grid; is the lower limit of power transmitted to the main grid.

[0058] When the final target network evaluation results of each agent reach the preset target, these agents are used to solve the objective function to obtain the optimal dispatching scheme of the multi-regional integrated energy system to control the execution of the distribution network; in this process, each agent can learn and make decisions autonomously based on local data and experience, and upload the model to the edge server, and exchange information through the Critic network parameter sharing mechanism of each agent. This sharing strategy not only realizes the local exchange of information, but also enhances the collaborative optimization effect between regions. Each agent updates the model based on its own independent learning results. The driving role of the elite agent significantly accelerates the learning process of other agents and improves the operating efficiency and economy of the entire system.

[0059] Through parameter aggregation and sharing, the present invention enables each intelligent agent to learn from the experience and knowledge of other intelligent agents, thereby improving their collaborative capabilities, which is conducive to achieving more efficient energy scheduling and optimization in multi-regional integrated energy systems; through iterative updates and performance evaluations, it can ensure that the final target network of each intelligent agent reaches a high performance level, which is conducive to improving the stability and accuracy of the entire multi-agent system; constructing an objective function with the total cost of the distribution network as the target, and obtaining an optimized scheduling plan through intelligent agent solution, which is conducive to achieving the economy and sustainable development of the multi-regional integrated energy system; through parameter sharing and intelligent agent collaboration, the system can adapt to different operating requirements and constraints more flexibly, which is conducive to improving the flexibility and robustness of the entire system.

[0060] In one embodiment, the process of another multi-regional integrated energy system optimization scheduling method is as follows Figure 3 As shown, in order to further optimize the energy management efficiency of the integrated energy system while ensuring data privacy and improve the adaptability of the integrated energy system in complex environments, first, each area is managed by an independent agent, and the agent takes six steps to operate: 1. Upload data set, 2. Local model training, 3. Elite agent selection, 4. Parameter upload, 5. Parameter aggregation, 6. Global model update. Different from the traditional federated reinforcement learning method, the reinforcement learning method adopted in this application does not need to aggregate the model parameters of all agents, but only selects the parameters of elite agents for aggregation. This method of only selecting elite agent aggregation parameters reduces the communication overhead to a certain extent, thereby enabling poorly performing agents to benefit from the strategies of elite agents and reduce the time required for independent exploration; then, by sharing high-quality parameters, the strategy convergence of inefficient agents is accelerated, and the overall learning efficiency is improved; and parameter sharing among multiple agents helps to cope with individual environmental differences and reduce the risk of a single agent falling into a local optimum. In addition, parameter importance weighting enhances the optimization of key parameters, thereby effectively avoiding excessive updates of irrelevant parameters and improving the efficiency and stability of the sharing process; it is used to solve the scheduling problem of multi-regional integrated energy systems and effectively solves the problem of poorly performing agents wandering in suboptimal strategies. Through the reinforcement learning algorithm, while maintaining exploration, the best performing agent is focused on, and the parameters of the elite agent with the highest reward value are shared with the poorly performing agent. The gradient importance weight mechanism is used to measure the contribution of the elite agent's parameters to model optimization, effectively improving the optimization efficiency of important parameters and reducing the impact of irrelevant parameters, thereby improving the stability and overall performance of the sharing process.

[0061] In the embodiment of the present application, based on the problem of how to improve the accuracy of the comprehensive energy optimization scheduling results, a multi-regional comprehensive energy system optimization scheduling method is designed, which realizes dividing the distribution network into several regions, setting an intelligent agent for the comprehensive energy system of each region, so that each of the intelligent agents updates its own target network through a reinforcement learning algorithm to obtain a target update network; quantifying the importance weights of the parameters in each of the target update networks, and selecting several elite intelligent agents from each of the intelligent agents, and globally aggregating the parameters in the target update network of each of the elite intelligent agents according to the importance weights to obtain aggregated parameters; sharing the aggregated parameters among all intelligent agents through a soft update method, so that each of the intelligent agents after sharing the parameters solves the objective function of the distribution network, and obtains an optimized scheduling scheme for the multi-regional comprehensive energy system and controls the execution of the distribution network. The technical solution; not only reduces the communication overhead, but also further improves the learning speed of each intelligent agent while ensuring data privacy, so that the poorly performing intelligent agent can get rid of the local optimum to a certain extent and avoid wandering in the suboptimal strategy.

[0062] It should be noted that although the steps in the above flowchart are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders.

[0063] In another embodiment, if Figure 4 As shown, the second aspect of the present invention provides a multi-region integrated energy system optimization scheduling system, comprising: The network update module 10 is used to divide the distribution network into a plurality of regions, and to set an intelligent agent for the integrated energy system of each region, so that each intelligent agent updates its own target network through a reinforcement learning algorithm to obtain a target updated network; A parameter aggregation module 20 is used to quantify the importance weights of the parameters in each target update network, and select a number of elite agents from each of the agents, and globally aggregate the parameters in the target update network of each of the elite agents according to the importance weights to obtain aggregated parameters; The optimization scheduling module 30 is used to share the aggregation parameters among all intelligent agents through a soft update method, so that each intelligent agent after sharing the parameters can solve the objective function of the distribution network, obtain an optimized scheduling plan for the multi-regional integrated energy system and control the execution of the distribution network.

[0064] It should be noted that each module in the above-mentioned multi-regional integrated energy system optimization and scheduling system can be fully or partially implemented by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. For the specific definition of a multi-regional integrated energy system optimization and scheduling system, please refer to the definition of a multi-regional integrated energy system optimization and scheduling method above. The two have the same functions and effects, which will not be repeated here.

[0065] A third aspect of the present invention provides an electronic device, the electronic device comprising: processor, memory, and bus; The bus is used to connect the processor and the memory; The memory is used to store operation instructions; The processor is used to call the operation instruction, and the executable instruction enables the processor to perform operations corresponding to the multi-regional integrated energy system optimization scheduling method shown in the first aspect of the present application.

[0066] In an alternative embodiment, an electronic device is provided, such as Figure 5 As shown, Figure 5 The electronic device 5000 shown includes: a processor 5001 and a memory 5003. The processor 5001 and the memory 5003 are connected, such as through a bus 5002. Optionally, the electronic device 5000 may also include a transceiver 5004. It should be noted that in actual applications, the transceiver 5004 is not limited to one, and the structure of the electronic device 5000 does not constitute a limitation on the embodiments of the present application.

[0067] Processor 5001 may be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 5001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0068] The bus 5002 may include a path to transmit information between the above components. The bus 5002 may be a PCI bus or an EISA bus, etc. The bus 5002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0069] The memory 5003 may be a ROM or other type of static storage device that can store static information and instructions, a RAM or other type of dynamic storage device that can store information and instructions, or an EEPROM, a CD-ROM or other optical disk storage, an optical disk storage (including a compressed optical disk, a laser disk, an optical disk, a digital versatile disk, a Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0070] The memory 5003 is used to store application code for executing the solution of the present application, and the execution is controlled by the processor 5001. The processor 5001 is used to execute the application code stored in the memory 5003 to implement the content shown in any of the above method embodiments.

[0071] Among them, electronic devices include but are not limited to: mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc.

[0072] The fourth aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the method for optimizing and scheduling a multi-regional integrated energy system shown in the first aspect of the present application is implemented.

[0073] Another embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding content in the aforementioned method embodiment.

[0074] In addition, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the above method are implemented.

[0075] In summary, the present invention relates to the technical field of energy optimization and scheduling, and discloses a method, system, device and medium for optimizing and scheduling a multi-region integrated energy system, including dividing a distribution network into several regions, setting an intelligent agent for the integrated energy system of each of the regions, so that each of the intelligent agents updates its own target network through a reinforcement learning algorithm to obtain a target update network; quantifying the importance weights of the parameters in each of the target update networks, and selecting several elite intelligent agents from each of the intelligent agents, and globally aggregating the parameters in the target update network of each of the elite intelligent agents according to the importance weights to obtain aggregated parameters; sharing the aggregated parameters among all intelligent agents through a soft update method, so that each of the intelligent agents after sharing the parameters solves the objective function of the distribution network, obtains an optimized scheduling plan for the multi-region integrated energy system and controls the execution of the distribution network, so as to improve the accuracy of the integrated energy optimization scheduling results.

[0076] Each embodiment in this specification is described in a progressive manner, and the same or similar parts of each embodiment can be directly referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. It should be noted that the technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0077] The above-mentioned embodiments only express several preferred implementation modes of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in the technical field, several improvements and substitutions can be made without departing from the technical principles of the present invention, and these improvements and substitutions should also be regarded as the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be based on the protection scope of the claims.

Claims

1. A multi-regional integrated energy system optimization scheduling method, characterized in that: include: The distribution network is divided into a plurality of regions, and an intelligent agent is set for the integrated energy system of each region, so that each intelligent agent updates its own target network through a reinforcement learning algorithm to obtain a target updated network; Quantifying the importance weights of the parameters in each of the target update networks, and selecting a number of elite agents from each of the agents, and globally aggregating the parameters in the target update networks of each of the elite agents according to the importance weights to obtain aggregated parameters; The aggregation parameters are shared among all intelligent agents through a soft update method, so that each intelligent agent after sharing the parameters can solve the objective function of the distribution network, obtain an optimized scheduling plan for the multi-regional integrated energy system and control the execution of the distribution network.

2. A multi-regional integrated energy system optimization scheduling method according to claim 1, characterized in that: The method of enabling each of the intelligent agents to update its own target network through a reinforcement learning algorithm to obtain a target update network includes: Initializing the environment and algorithm parameters of each of the agents, and constructing a playback buffer for each of the agents; Constructing a state set and an action set based on the integrated energy system of each of the regions, and constructing a reward function according to the economic cost of the distribution network, so that each of the intelligent agents interacts with the initialized environment through a deep deterministic policy gradient algorithm based on the initialized algorithm parameters, obtains a state value, an action value and a reward value and stores them in the playback buffer; The target network of each of the agents is trained and updated using the data in the playback buffer to obtain a target update network of each of the agents.

3. The multi-regional integrated energy system optimization scheduling method according to claim 1 is characterized in that: The quantifying the importance weights of the parameters in each target updating network includes: Performing forward propagation on each of the target update networks to quantify the loss function under the current strategy, and quantifying the gradient of each parameter in each of the target update networks according to the loss function under the current strategy; The absolute values ​​of the gradients are averaged over multiple time steps to obtain the importance weight of each parameter in each target update network.

4. A multi-regional integrated energy system optimization scheduling method according to claim 2, characterized in that: The step of selecting a plurality of elite intelligent agents from each of the intelligent agents comprises: Standardizing the reward value corresponding to each of the intelligent agents to obtain a standardized reward value; Initialize the weight and learning rate of each of the intelligent agents, and quantify the selection probability of each of the intelligent agents according to the initialization result; Selecting an elite intelligent agent according to each of the selection probabilities, and updating the weight of the selected elite intelligent agent according to the standardized reward value and the operation support of the distribution network; Based on the updated self-weight, each of the agents is controlled to iteratively execute the quantization step of the selection probability and the selection step of the elite agent until a preset iteration round is reached, and a number of selected elite agents are output.

5. The multi-regional integrated energy system optimization scheduling method according to claim 1 is characterized in that: The globally aggregating the parameters in the target update network of each elite agent according to each importance weight to obtain the aggregated parameters includes: Each of the importance weights is used as a parameter aggregation weight of each of the elite agents to globally aggregate the parameters in the target update network of each of the elite agents to obtain the aggregated parameters.

6. A multi-regional integrated energy system optimization scheduling method according to claim 1, characterized in that: The sharing of the aggregation parameters is achieved by the following formula: In the formula, , Update the network parameters for the kth agent before and after sharing the aggregated parameters. is the soft update ratio; is the aggregation parameter of the jth parameter; is the jth parameter; D select For elite intelligent agents; is element-wise multiplication.

7. The multi-regional integrated energy system optimization scheduling method according to claim 1 is characterized in that: The method of making each of the intelligent agents after sharing parameters solve the objective function of the distribution network, obtain an optimized dispatching plan for the multi-regional integrated energy system and control the distribution network to execute, includes: Controlling the target update network of each of the intelligent agents to share the aggregation parameters to obtain a final target network, and performing performance evaluation on each of the final target networks; When the evaluation result does not reach the preset target, repeatedly performing the network updating step and the parameter sharing step based on each of the final target networks until the evaluation result of each updated final target network reaches the preset target; Under the operating constraints of the integrated energy system in each of the regions, the objective function is constructed with the total cost of the distribution network as the target, and the objective function is solved by each of the intelligent agents when the evaluation result reaches the preset target, so as to obtain an optimized scheduling plan for the multi-region integrated energy system to control the execution of the distribution network.

8. A multi-region integrated energy system optimization and dispatching system, characterized in that: include: A network update module is used to divide the distribution network into a number of areas, and set an intelligent agent for the integrated energy system of each area, so that each intelligent agent updates its own target network through a reinforcement learning algorithm to obtain a target updated network; A parameter aggregation module, used to quantify the importance weights of the parameters in each target update network, and select a number of elite agents from each of the agents, and globally aggregate the parameters in the target update network of each of the elite agents according to the importance weights to obtain aggregated parameters; The optimization scheduling module is used to share the aggregation parameters among all intelligent agents through a soft update method, so that each intelligent agent after sharing the parameters can solve the objective function of the distribution network, obtain an optimized scheduling plan for the multi-regional integrated energy system and control the execution of the distribution network.

9. An electronic device, characterized in that: It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, it implements the multi-regional integrated energy system optimization scheduling method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the device where the computer-readable storage medium is located executes the computer program, the multi-regional integrated energy system optimization scheduling method as described in any one of claims 1 to 7 is implemented.

Images