Multi-region integrated energy system optimization scheduling method, system, equipment and medium

Abstract

The present invention discloses a multi-region integrated energy system optimization scheduling method, system, equipment and medium, including dividing a distribution network into several regions, setting an intelligent agent for the integrated energy system of each region, so that each intelligent agent updates its own target network through a reinforcement learning algorithm to obtain a target update network; quantifying the importance weights of parameters in each target update network, and selecting several elite intelligent agents from each intelligent agent, and globally aggregating the parameters in the target update network of each elite intelligent agent according to each importance weight to obtain aggregated parameters; sharing the aggregated parameters among all intelligent agents through a soft update method, so that each intelligent agent after sharing the parameters solves the objective function of the distribution network, obtains an optimized scheduling plan for the multi-region integrated energy system and controls the execution of the distribution network, so as to improve the accuracy of the integrated energy optimization scheduling results.

Description

Technical Field

[0001] The present invention relates to the technical field of energy optimization and scheduling, and in particular to a multi-region integrated energy system optimization and scheduling method, system, equipment and medium. Background Art

[0002] With the widespread application of intelligent learning and multi-agent systems in complex environments, traditional single optimization methods have become difficult to meet the increasingly complex and changeable task requirements in energy optimization scheduling scenarios. Moreover, relying solely on elite selection mechanisms to identify excellent agents cannot fully utilize the potential of other agents in the system, resulting in inaccurate optimization scheduling results for comprehensive energy.

[0003] It can be seen that how to improve the accuracy of comprehensive energy optimization scheduling results has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the Invention

[0004] The present invention provides a multi-region integrated energy system optimization scheduling method, system, equipment and medium to solve the problem of how to improve the accuracy of integrated energy optimization scheduling results.

[0005] To solve the above technical problems, the first aspect of the present invention provides a multi-region integrated energy system optimization scheduling method, comprising:

[0006] The distribution network is divided into a plurality of regions, and an intelligent agent is set for the integrated energy system of each region, so that each intelligent agent updates its own target network through a reinforcement learning algorithm to obtain a target updated network;

[0007] quantifying the importance weights of the parameters in each target update network, selecting a number of elite agents from each of the agents, and globally aggregating the parameters in the target update network of each of the elite agents according to the importance weights to obtain aggregated parameters;

[0008] The aggregation parameters are shared among all intelligent agents through a soft update method, so that each intelligent agent after sharing the parameters solves the objective function of the distribution network, obtains an optimized scheduling plan for the multi-region integrated energy system and controls the execution of the distribution network.

[0009] As one preferred solution, the method of enabling each of the intelligent agents to update its own target network through a reinforcement learning algorithm to obtain a target updated network includes:

[0010] Initializing the environment and algorithm parameters of each agent, and constructing a playback buffer for each agent;

[0011] Constructing a state set and an action set based on the integrated energy system of each of the regions, and constructing a reward function based on the economic cost of the distribution network, so that each of the intelligent agents interacts with the initialized environment through a deep deterministic policy gradient algorithm based on the initialized algorithm parameters, obtains a state value, an action value, and a reward value, and stores them in the playback buffer;

[0012] The target network of each agent is trained and updated using the data in the playback buffer to obtain a target updated network of each agent.

[0013] As one preferred solution, the quantifying the importance weights of the parameters in each target update network includes:

[0014] Performing forward propagation on each of the target update networks to quantify the loss function under the current strategy, and quantifying the gradient of each parameter in each of the target update networks according to the loss function under the current strategy;

[0015] The absolute values of the gradients are averaged over multiple time steps to obtain the importance weight of each parameter in each target update network.

[0016] As one preferred solution, the step of selecting a number of elite agents from the agents includes:

[0017] Normalizing the reward values corresponding to the agents to obtain standardized reward values;

[0018] Initializing the weight and learning rate of each of the agents, and quantifying the selection probability of each of the agents based on the initialization results;

[0019] Selecting an elite agent according to each of the selection probabilities, and updating the weight of the selected elite agent according to the standardized reward value and the operation support of the distribution network;

[0020] Based on the updated weight of itself, each agent is controlled to iteratively execute the quantization step of the selection probability and the selection step of the elite agent until a preset iteration round is reached, and a number of selected elite agents are output.

[0021] As one preferred solution, the parameters in the target update network of each elite agent are globally aggregated according to each importance weight to obtain the aggregated parameters, including:

[0022] Each of the importance weights is used as a parameter aggregation weight of each of the elite agents to globally aggregate the parameters in the target update network of each of the elite agents to obtain the aggregated parameters.

[0023] As one of the preferred solutions, the sharing of the aggregation parameters is achieved by the following formula:

[0024]

[0025] Where, 、 Update the network parameters for the kth agent before and after sharing the aggregated parameters; is the soft update ratio; is the aggregation parameter of the jth parameter; is the jth parameter; D select For elite intelligent agents; is element-wise multiplication.

[0026] As one preferred solution, the process of enabling each of the intelligent agents after sharing parameters to solve the objective function of the distribution network, obtaining an optimized scheduling plan for the multi-region integrated energy system and controlling the execution of the distribution network includes:

[0027] Controlling the target update network of each of the intelligent agents to share the aggregation parameters to obtain a final target network, and performing performance evaluation on each of the final target networks;

[0028] When the evaluation result does not reach the preset target, repeatedly performing the network updating step and the parameter sharing step based on each of the final target networks until the evaluation result of each updated final target network reaches the preset target;

[0029] Under the operating constraints of the integrated energy system in each of the regions, the objective function is constructed with the total cost of the distribution network as the target, and the objective function is solved by each of the intelligent agents when the evaluation result reaches the preset target, so as to obtain an optimized scheduling plan for the multi-region integrated energy system to control the execution of the distribution network.

[0030] A second aspect of the present invention provides a multi-region integrated energy system optimization and scheduling system, comprising:

[0031] A network update module is used to divide the distribution network into several areas and set an intelligent agent for the integrated energy system of each area, so that each intelligent agent updates its own target network through a reinforcement learning algorithm to obtain a target updated network;

[0032] a parameter aggregation module, configured to quantify the importance weights of the parameters in each target update network, select a number of elite agents from each of the agents, and globally aggregate the parameters in the target update network of each of the elite agents according to the importance weights to obtain aggregated parameters;

[0033] The optimization scheduling module is used to share the aggregation parameters among all intelligent agents through the soft update method, so that each intelligent agent after sharing the parameters solves the objective function of the distribution network, obtains the optimized scheduling plan for the multi-region integrated energy system and controls the execution of the distribution network.

[0034] The third aspect of the present invention provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements the multi-region integrated energy system optimization scheduling method as described above.

[0035] The fourth aspect of the present invention provides a computer-readable storage medium, which includes a stored computer program, wherein when the device where the computer-readable storage medium is located executes the computer program, it implements the multi-region integrated energy system optimization scheduling method as described above.

[0036] Compared with the prior art, the embodiments of the present invention have the following advantages:

[0037] (1) Each intelligent agent is responsible for the integrated energy system in its area. By continuously optimizing its own dispatching strategy through reinforcement learning, it can reduce the burden of central control and improve the overall dispatching efficiency. The elite selection mechanism enables the distribution network system to concentrate learning resources, strengthen and optimize the best-performing strategy at the fastest speed, and introduce an elite intelligent agent parameter sharing mechanism, so that the parameters of the intelligent agent with excellent performance are shared with other intelligent agents, promoting the improvement of the learning efficiency and convergence speed of the entire system.

[0038] (2) In order to address the problem that the impact of different parameters on model performance may be significantly different during the sharing process, the importance weights of the parameters in the target update network of each intelligent agent are introduced to dynamically evaluate the contribution of each parameter to performance optimization, and the aggregated parameters are shared among all intelligent agents, so that each intelligent agent can learn the globally optimal scheduling strategy, which helps to improve the intelligence level and scheduling efficiency of the entire distribution network; and combining the soft update strategy with parameter sharing can not only alleviate the instability factors caused by direct parameter transmission, but also improve the overall adaptability and robustness of the system in complex dynamic environments;

[0039] (3) By combining reinforcement learning algorithms, elite selection mechanisms and gradient importance weights, the collaborative efficiency and optimization performance of multi-agent systems in complex and changing environments can be significantly improved, providing more economical and efficient solutions for scenarios such as multi-regional integrated energy scheduling, thereby improving the accuracy of integrated energy optimization scheduling results in distribution networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings required for use in the implementation. Obviously, the drawings described below are only some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 This is a flow chart of a multi-region integrated energy system optimization scheduling method provided by one embodiment of the present invention;

[0042] Figure 2 This is a structural diagram of a comprehensive energy system model within a single region provided by an embodiment of the present invention;

[0043] Figure 3 is a flow chart of a multi-region integrated energy system optimization scheduling method provided by another embodiment of the present invention;

[0044] Figure 4 This is a structural diagram of a multi-region integrated energy system optimization and scheduling system provided by an embodiment of the present invention;

[0045] Figure 5 This is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings and embodiments. Obviously, the embodiments described are only some embodiments of the present invention, rather than all embodiments. The purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0047] In the description of this application, the terms "first," "second," "third," etc. are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first," "second," "third," etc. may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0048] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be a communication between the two components. The terms "vertical", "horizontal", "left", "right", "up", "down" and similar expressions used herein are for illustrative purposes only, and do not indicate or imply that the system or component referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0049] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in this application have the same meanings as those commonly understood by those skilled in the art. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood by those skilled in the art in specific circumstances.

[0050] In one embodiment, if Figure 1 As shown, the first aspect of the present invention provides a multi-region integrated energy system optimization scheduling method, comprising:

[0051] S1. Divide the distribution network into several regions, and set up an intelligent agent for the integrated energy system of each region, so that each intelligent agent updates its own target network through a reinforcement learning algorithm to obtain a target updated network;

[0052] Specifically, the present invention collects historical load data, energy production data (such as solar energy, wind energy), line parameters, etc. of the distribution network, and uses clustering algorithms (such as K-means) or graph theory methods (such as segmentation based on network topology) to divide the distribution network into several areas, ensuring that the energy supply and demand in each area is relatively balanced and easy to manage; wherein, the structure of the integrated energy system model in a single area is as follows Figure 2 As shown, it includes comprehensive energy such as wind power, photovoltaics, electric energy storage, generators, natural gas, as well as cold, heat, and electricity loads, and sets up an independent intelligent agent for each area so that the intelligent agent can independently train its local model in a distributed environment.

[0053] In one embodiment, the step of enabling each of the intelligent agents to update its own target network through a reinforcement learning algorithm to obtain a target updated network includes:

[0054] Initializing the environment and algorithm parameters of each agent, and constructing a playback buffer for each agent;

[0055] Constructing a state set and an action set based on the integrated energy system of each of the regions, and constructing a reward function based on the economic cost of the distribution network, so that each of the intelligent agents interacts with the initialized environment through a deep deterministic policy gradient algorithm based on the initialized algorithm parameters, obtains a state value, an action value, and a reward value, and stores them in the playback buffer;

[0056] The target network of each agent is trained and updated using the data in the playback buffer to obtain a target updated network of each agent.

[0057] Specifically, the present invention uses the Deep Deterministic Policy Gradient (DDPG) algorithm to optimize the models of the agents in each region and interacts with the model parameters through the edge server, specifically including:

[0058] First, the environment of each agent is initialized, that is, the learning rate of the Actor network and the Critic network, the target network update rate and the exploration noise are set, and the algorithm parameters of each agent are initialized, including a Critic network and an Actor network, and a playback buffer is established for each agent to store experience data; and in order to highly match the operating characteristics and goals of the integrated energy system, the state set, action set and reward function are constructed based on the integrated energy system of each region; Among them,

[0059] The state set includes the electric load demand, heating load demand, cooling load demand, photovoltaic power generation, energy storage unit status, current scheduling interval, and future electricity prices in the integrated energy system of a certain area, so as to fully reflect the current status of the distribution network area. The state of the integrated energy system of a certain area can be expressed as:

[0060]

[0061] Where s is the state; k is the agent, which can also represent the region; t is the time; is the electricity price at time t; is the capacity of the energy storage device in region k at time t; The user's power load; is the generator operating power in region k at time t; is the wind power generation power in region k at time t; is the photovoltaic power generation power in region k at time t; is the output power of the generator.

[0062] The action set should cover all possible actions that the agent can take, such as adjusting energy distribution, controlling equipment power, etc. However, since the action dimension can be reduced through equality constraints in economic scheduling, the action set in the integrated energy system of a certain area can be designed as follows:

[0063]

[0064] Where, For action; 、 and They are the climbing power of energy storage equipment, combined heating, cooling and power units and generators respectively.

[0065] The reward function in each region's integrated energy system consists of two parts: one part is used to optimize the economic cost of the regional integrated energy system scheduling, and the other part is used to punish the consequences of the agent violating constraints or taking inappropriate actions. Therefore, the reward function expression is as follows:

[0066]

[0067] Where, is the reward function; is the negative value of the total economic cost of the integrated energy system within a single region (which can also be understood as the kth region); is the penalty the agent receives when violating the constraint; is the power generation cost of generators in a single region; The equipment loss cost of the unit in a single area; Maintaining optimal cost for energy storage equipment within a single region; The cost of purchasing electricity from the main grid in a single region; is the natural gas purchase cost to meet the demand of CCHP units in a single region; θ1, θ2, θ3, θ4, and θ5 are the numbers of corresponding weights respectively; is the penalty weight; J is the constraint set. If the constraint is violated, the size of the constraint exceeded is calculated; 、 、 They are the output power of the integrated energy system in a single area and its minimum and maximum values respectively.

[0068] Then, the local training process begins, where each agent independently performs decision-making and learning operations using the DDPG algorithm. Each agent collects experience data by interacting with the environment. , and the current state value in the formula , current action value , current reward value and the next state value Stored in the experience replay buffer, it provides necessary training data support for the algorithm's network update.

[0069] During the training process, the update of the Actor network is to maximize the expected reward by adjusting the policy function. Therefore, in each update, the parameters of the Actor network are updated by calculating the gradient of the Q network for the action generated by the current policy. It is specifically expressed by the following formula:

[0070]

[0071] Where, Parameters of the Actor network; is the Actor network learning rate; is the gradient operator; is the expected total return under the current state; are the parameters of the Q network; is the expected result of the samples in the experience replay buffer; is the policy function.

[0072] The parameters of the Critic network are updated by the target value generated by the target network, and the Critic network is updated by minimizing the loss function. The update process is expressed by the following formula:

[0073]

[0074]

[0075] Where, is the target value; is the attenuation coefficient; is the expected total reward in the next state; is the loss function; The value of the action to be taken in the next state; Actions generated for the Actor network; is Gaussian noise; is the standard deviation of the noise; c is the clipping threshold.

[0076] Finally, a soft update method is used to update the Critic network and the Actor network to reduce instability during the learning process. The update method is as follows:

[0077]

[0078] Where, is the target network update rate.

[0079] Furthermore, the target update network in this invention doesn't refer to a specific network, but rather the final network obtained after each agent is trained using a reinforcement learning algorithm. By assigning agents to the integrated energy system in each region of the distribution network and applying the DDPG algorithm for training and updating, this invention can improve the energy management efficiency of the distribution network, reduce operating costs, and enhance the adaptability of the distribution network system.

[0080] S2. quantifying the importance weights of the parameters in each target update network, selecting a number of elite agents from each of the agents, and performing global aggregation on the parameters in the target update network of each of the elite agents according to the importance weights to obtain aggregated parameters;

[0081] In one embodiment, quantifying the importance weights of the parameters in each target update network includes:

[0082] Performing forward propagation on each of the target update networks to quantify the loss function under the current strategy, and quantifying the gradient of each parameter in each of the target update networks according to the loss function under the current strategy;

[0083] The absolute values of the gradients are averaged over multiple time steps to obtain the importance weight of each parameter in each target update network.

[0084] Specifically, the present invention forward propagates the data in the playback buffer to the Critic network of each agent to obtain the current prediction result, and calculates the loss function (such as mean square error loss, cross entropy loss, etc.) under the current strategy to quantify the difference between the prediction result and the true label. Then, through the backpropagation algorithm, the gradient of the loss function is backpropagated from the output layer to the input layer, and the contribution of each parameter to the loss function, that is, the gradient of the parameter, is calculated during the backpropagation process. Finally, the absolute value of each gradient is averaged over multiple time steps. The average value obtained reflects the average influence of the parameter in the optimization process, that is, the importance weight of the parameter. In this way, the importance weight of each parameter in the Critic network can be obtained; wherein, the importance weight is expressed by the following formula:

[0085]

[0086] Where, is the importance weight of the i-th parameter in the Critic network; T is the total duration of iteration before each parameter exchange; is the loss function of the Critic network of agent k at time t; is the i-th parameter in the Critic network.

[0087] By quantifying the importance weight of each parameter, the present invention can identify the parameters that have a greater impact on the loss function, so that in the subsequent parameter update process, more attention can be paid to these important parameters, thereby improving training efficiency; the calculation of the importance weight takes into account the gradient information within multiple time steps, which helps to capture the long-term impact of the parameters and helps the model learn more generalized features during training, thereby improving the generalization ability of the model; according to the importance weight of the parameter, the step size or learning rate of the parameter update can be adjusted. For parameters with higher importance, a smaller step size can be used for fine adjustment; for parameters with lower importance, a larger step size can be used for rapid update, which helps to balance the training speed and stability of the model; it also provides additional explanatory information for the model. By analyzing the importance weight of the parameters, it is possible to understand which features or information the model mainly relies on when making predictions, which helps researchers better understand the decision-making process of the model and make targeted improvements.

[0088] In one embodiment, the step of selecting a plurality of elite agents from among the agents comprises:

[0089] Normalizing the reward values corresponding to the agents to obtain standardized reward values;

[0090] Initializing the weight and learning rate of each of the agents, and quantifying the selection probability of each of the agents based on the initialization results;

[0091] Selecting an elite agent according to each of the selection probabilities, and updating the weight of the selected elite agent according to the standardized reward value and the operation support of the distribution network;

[0092] Based on the updated weight of itself, each agent is controlled to iteratively execute the quantization step of the selection probability and the selection step of the elite agent until a preset iteration round is reached, and a number of selected elite agents are output.

[0093] Specifically, the reward value of each agent is standardized by the following formula:

[0094]

[0095] Where, is the standardized reward value; is the reward value of the agent; It is the reward value corresponding to when the actions of the agent at all times are 0; the present invention establishes a reward value threshold. When the standardized reward value is greater than the reward value threshold, it means that the strategy trained by the agent is positive, which will increase the reward of the agent; when the standardized reward value is less than the reward value threshold, it means that the strategy trained by the agent is negative, which will reduce the reward of the agent.

[0096] Then initialize the learning rate of each agent and the corresponding self-weight so that the self-weight, where the self-weight reflects the relative importance or influence of the agent in the current iteration, and the learning rate is used to control the step size or speed of the agent when updating its own weight, and calculate the selection probability corresponding to each agent through its own weight; then select the elite agent based on the calculated selection probability, and update the self-weight of the selected elite agent through the standardized reward value and the operation support of the distribution network, and iteratively execute the quantization step of the selection probability and the selection step of the elite agent using the updated self-weight of each agent until the preset iteration round is reached, and output the selected elite agents; where the selection probability is calculated by the following formula:

[0097]

[0098] Where, is the selection probability of the agent in the nth round of iteration; is the update ratio; is the weight corresponding to agent k; N is the total number of agents; The preset iteration rounds.

[0099] The update process of its own weight is expressed as follows:

[0100]

[0101] Where D select is the selected agent; The operational support of the distribution network; is the selection probability of the selected agent in the nth round of iteration.

[0102] The present invention ensures that intelligent agents are compared and evaluated on a unified scale by standardizing reward values and iteratively updating weights. The selection process of elite intelligent agents promotes the retention and further development of excellent intelligent agents, which helps to improve the performance of the entire intelligent agent group. The operational support of the distribution network is introduced as an additional evaluation indicator, so that the intelligent agent not only considers the performance of the current task when updating the weight, but also considers its contribution or adaptability to the distribution network, which helps the intelligent agent better adapt to the specific environment and needs of the distribution network. The convergence speed of the intelligent agent performance is accelerated by iteratively executing the quantification of selection probability, the selection of elite intelligent agents and the updating of their own weights. The preset iteration rounds ensure that the algorithm reaches a stable state within a reasonable time and outputs the optimal intelligent agent group.

[0103] In one embodiment, the globally aggregating the parameters in the target update network of each elite agent according to each importance weight to obtain the aggregated parameters includes:

[0104] Each of the importance weights is used as a parameter aggregation weight of each of the elite agents to globally aggregate the parameters in the target update network of each of the elite agents to obtain the aggregated parameters.

[0105] Specifically, in reinforcement learning algorithms, parameter sharing is a key step, that is, globally aggregating the parameters of elite agents in an edge server and distributing the aggregated parameters to all agents. At the same time, parameter importance weights are used to enhance the update of key parameters and soft updates are adopted to mitigate potential adverse factors caused by parameter transfer.

[0106] S3. Sharing the aggregated parameters among all intelligent agents through a soft update method, so that each intelligent agent after sharing the parameters solves the objective function of the distribution network, obtains an optimized scheduling plan for the multi-region integrated energy system, and controls the execution of the distribution network;

[0107] In one embodiment, the sharing of the aggregated parameters, that is, the sharing mechanism of the critic network parameters of each agent, can be expressed as:

[0108]

[0109] Where, 、 Update the network parameters for the kth agent before and after sharing the aggregated parameters; is the soft update ratio; is the aggregation parameter of the jth parameter; is the jth parameter; D select For elite intelligent agents; is element-wise multiplication.

[0110] In addition, the present invention can also control each elite intelligent agent to upload its local parameter sample to a central node (such as an enhanced intelligent agent or a collaborative platform), so that the central node can aggregate these parameter samples, calculate the center point of all parameter samples, and the Mahalanobis distance of each parameter sample to the center point; then remove the parameter samples that are more than a preset distance threshold from the center point, that is, eliminate abnormal and outlier parameters, and perform weighted averaging on the remaining parameter samples, and the aggregated parameters can also be obtained; then the central node returns the aggregated parameter results to each intelligent agent. After receiving the aggregated parameters, each intelligent agent does not directly replace the local parameters, but adds the aggregated parameters to its own local parameters with its corresponding importance weight as a coefficient, thereby achieving a smooth transition of the parameters.

[0111] In one embodiment, the step of enabling each of the intelligent agents after sharing parameters to solve the objective function of the distribution network, obtaining an optimized scheduling plan for the multi-region integrated energy system, and controlling the execution of the distribution network includes:

[0112] Controlling the target update network of each of the intelligent agents to share the aggregation parameters to obtain a final target network, and performing performance evaluation on each of the final target networks;

[0113] When the evaluation result does not reach the preset target, repeatedly performing the network updating step and the parameter sharing step based on each of the final target networks until the evaluation result of each updated final target network reaches the preset target;

[0114] Under the operating constraints of the integrated energy system in each of the regions, the objective function is constructed with the total cost of the distribution network as the target, and the objective function is solved by each of the intelligent agents when the evaluation result reaches the preset target, so as to obtain an optimized scheduling plan for the multi-region integrated energy system to control the execution of the distribution network.

[0115] Specifically, after each round of parameter sharing, the present invention will evaluate the performance of its final target network in order to further optimize the strategy. If the evaluation result does not reach the preset target (such as the accuracy does not reach 90%, or the loss function value does not drop below 0.1), the network update step and the parameter sharing step will be repeated based on these final target networks; multiple iterative updates will be performed until the evaluation results of each updated final target network reach the preset target; and in the integrated energy system of each region, considering its operating constraints, the objective function is constructed with the total cost of the distribution network as the target; wherein the objective function is the total cost of the region divided by the distribution network Specifically, it includes the cost of generator power generation, unit equipment loss, the cost of maintaining the optimal point of energy storage equipment, the cost of purchasing electricity from the main power grid and the integrated energy system, and the cost of purchasing natural gas from the natural gas grid. The various costs within a single region are expressed by the following formula:

[0116]

[0117]

[0118]

[0119]

[0120]

[0121] Where a kj 、b kj and c kjare the cost coefficients of the generators in region k respectively; is the output power of the jth generator in region k at time t; is the optimal output power of the jth generator in region k; is the charge and discharge power of the i-th energy storage device in region k at time t; is the maximum service life of the i-th energy storage device; is the optimal capacity of the i-th energy storage device in region k at time t; is the electricity price at time t; is the power traded between the corresponding region in region k and the main grid at time t. A value greater than 0 indicates selling electricity, and a value less than 0 indicates purchasing electricity; is the natural gas transaction cost; is the input power of the CCHP unit in region k at time t.

[0122] Its operating constraints include:

[0123] The operating constraints of the CCHP unit are expressed as follows:

[0124]

[0125] Where, 、 and are the electricity, heat and cooling power outputs of the CCHP unit at time t respectively; 、 and They are the power generation efficiency, heat-to-electricity ratio and cooling-to-electricity ratio of the trigeneration unit respectively.

[0126] The upper and lower limit operation constraints and ramp constraints of the CCHP unit are as follows:

[0127]

[0128] Where, and They are the minimum and maximum input power limits of the CCHP unit respectively; and They are the sliding rate and climbing rate of the CCHP unit respectively.

[0129] Electric boiler operation constraints , expressed by the following formula:

[0130]

[0131] Where, is the efficiency of the electric boiler; is the output power of the electric boiler at time t.

[0132] The thermal power balance constraint of the electric boiler is expressed by the following formula:

[0133]

[0134] Where, is the heat load at time t; The operating efficiency of the electric boiler; It is the thermal power output of the CCHP unit.

[0135] Electric Refrigerator Operation Constraints , expressed by the following formula:

[0136]

[0137] Where, is the efficiency of the electric refrigerator; is the output power of the electric refrigerator at time t.

[0138] The cooling power balance constraint of the electric refrigerator is expressed as follows:

[0139]

[0140] Where, is the cooling load at time t; is the operating efficiency of the refrigerant; is the cooling power output of the CCHP unit at time t.

[0141] The upper and lower limit operation constraints and ramp constraints of the generator are expressed by the following formula:

[0142]

[0143] Where, and are the minimum and maximum output power limits of the generator respectively; and are the landslide rate and climbing rate of the generator respectively.

[0144] The state of charge constraint of the energy storage system is expressed as follows:

[0145]

[0146] Where, is the state of charge of the energy storage system at time t; and are the lower and upper limits of SoC respectively.

[0147] The SoC at the next moment is expressed as:

[0148]

[0149] Where, is the charge / discharge power at time t; is the capacity of the energy storage system; is the time interval.

[0150] The charge and discharge power limits and ramp constraints are as follows:

[0151]

[0152] Where, and are the minimum and maximum output power limits of the energy storage device, respectively; and are the landslide rate and climbing rate of energy storage equipment respectively.

[0153] Power Constraints in Internal Transaction of Integrated Energy Systems It should be expressed by the following formula:

[0154]

[0155] The upper and lower limits of the power transmitted to the integrated energy system of the main power grid are as follows.

[0156]

[0157] Where, The upper limit of power transmitted to the main grid; The lower limit of power transmitted to the main grid.

[0158] When the final target network evaluation results of each agent reach the preset target, these agents are used to solve the objective function, resulting in an optimized scheduling plan for the multi-region integrated energy system to control the execution of the distribution network. During this process, each agent can autonomously learn and make decisions based on local data and experience, and upload the model to the edge server. Information exchange is carried out through the agent-critic network parameter sharing mechanism. This sharing strategy not only enables local information exchange but also enhances the collaborative optimization effect between regions. Each agent updates the model based on its own independent learning results. The driving effect of the elite agent significantly accelerates the learning process of other agents, improving the operational efficiency and economic efficiency of the entire system.

[0159] Through parameter aggregation and sharing, the present invention enables each intelligent agent to learn from the experience and knowledge of other intelligent agents, thereby improving their collaborative capabilities, which helps to achieve more efficient energy scheduling and optimization in multi-regional integrated energy systems; through iterative updates and performance evaluation, it can ensure that the final target network of each intelligent agent reaches a high performance level, which helps to improve the stability and accuracy of the entire multi-agent system; constructing an objective function with the total cost of the distribution network as the target, and obtaining an optimized scheduling plan through intelligent agent solution, which helps to achieve the economic and sustainable development of the multi-regional integrated energy system; through parameter sharing and intelligent agent collaboration, the system can adapt to different operating requirements and constraints more flexibly, which helps to improve the flexibility and robustness of the entire system.

[0160] In one embodiment, the process of another multi-region integrated energy system optimization scheduling method is as follows: Figure 3 As shown, in order to further optimize the energy management efficiency of the integrated energy system while ensuring data privacy and improve the adaptability of the integrated energy system in complex environments, first, each area is managed by an independent agent, and the agent takes six steps to operate: 1. Upload data set, 2. Local model training, 3. Elite agent selection, 4. Parameter upload, 5. Parameter aggregation, 6. Global model update. Unlike traditional federated reinforcement learning methods, the reinforcement learning method adopted in this application does not need to aggregate the model parameters of all agents, but only selects the parameters of elite agents for aggregation. This method of only selecting the aggregated parameters of elite agents reduces the communication overhead to a certain extent, thereby allowing poorly performing agents to benefit from the strategies of elite agents and reduce the time required for independent exploration; then, by sharing high-quality parameters, the strategy convergence of inefficient agents is accelerated, thereby improving the overall learning efficiency; and parameter sharing among multiple agents helps to cope with individual environmental differences and reduce the risk of a single agent falling into a local optimum. Furthermore, parameter importance weighting enhances the optimization of key parameters, effectively preventing excessive updates to irrelevant parameters and improving the efficiency and stability of the sharing process. This approach is used to solve the scheduling problem of multi-regional integrated energy systems, effectively resolving the issue of underperforming agents lingering on suboptimal strategies. Through a reinforcement learning algorithm, while maintaining exploration and focusing on the best-performing agents, the parameters of the elite agents with the highest reward values are shared with the less performing agents. A gradient importance weighting mechanism is used to measure the contribution of the elite agent's parameters to model optimization, effectively improving the optimization efficiency of important parameters and reducing the impact of irrelevant parameters, thereby enhancing the stability and overall performance of the sharing process.

[0161] In the embodiment of the present application, based on the problem of how to improve the accuracy of the integrated energy optimization scheduling results, a multi-region integrated energy system optimization scheduling method is designed, which realizes dividing the distribution network into several regions, setting an intelligent agent for the integrated energy system of each of the said regions, so that each of the said intelligent agents updates its own target network through the reinforcement learning algorithm to obtain a target update network; quantifies the importance weights of the parameters in each of the said target update networks, and selects several elite intelligent agents from each of the said intelligent agents, and globally aggregates the parameters in the target update network of each of the said elite intelligent agents according to the said importance weights to obtain aggregated parameters; the aggregated parameters are shared among all intelligent agents through the soft update method, so that each of the said intelligent agents after sharing the parameters solves the objective function of the distribution network, obtains an optimized scheduling scheme for the multi-region integrated energy system and controls the execution of the distribution network; not only reduces the communication overhead, but also further improves the learning speed of each intelligent agent while ensuring data privacy, so that the poorly performing intelligent agent can get rid of the local optimum to a certain extent and avoid wandering in the suboptimal strategy.

[0162] It should be noted that although the steps in the above flowchart are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders.

[0163] In another embodiment, Figure 4 As shown, the second aspect of the present invention provides a multi-region integrated energy system optimization and scheduling system, comprising:

[0164] The network update module 10 is used to divide the distribution network into a plurality of regions, and set an intelligent agent for the integrated energy system of each region, so that each intelligent agent updates its own target network through a reinforcement learning algorithm to obtain a target updated network;

[0165] a parameter aggregation module 20 for quantifying the importance weights of the parameters in each target update network, selecting a number of elite agents from each of the agents, and performing global aggregation on the parameters in the target update network of each of the elite agents according to the importance weights to obtain aggregated parameters;

[0166] The optimization scheduling module 30 is used to share the aggregation parameters among all intelligent agents through a soft update method, so that each intelligent agent after sharing the parameters solves the objective function of the distribution network, obtains an optimized scheduling plan for the multi-region integrated energy system and controls the execution of the distribution network.

[0167] It should be noted that each module in the above-mentioned multi-regional integrated energy system optimization and scheduling system can be fully or partially implemented through software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. For the specific definition of a multi-regional integrated energy system optimization and scheduling system, please refer to the definition of a multi-regional integrated energy system optimization and scheduling method above. The two have the same functions and effects and will not be repeated here.

[0168] A third aspect of the present invention provides an electronic device, comprising:

[0169] processor, memory, and bus;

[0170] The bus is used to connect the processor and the memory;

[0171] The memory is used to store operation instructions;

[0172] The processor is used to call the operation instruction, and the executable instruction enables the processor to perform operations corresponding to the multi-region integrated energy system optimization scheduling method shown in the first aspect of the present application.

[0173] In an alternative embodiment, an electronic device is provided, such as Figure 5 As shown, Figure 5 The electronic device 5000 shown includes: a processor 5001 and a memory 5003. The processor 5001 and the memory 5003 are connected, for example, via a bus 5002. Optionally, the electronic device 5000 may further include a transceiver 5004. It should be noted that in actual applications, the number of transceivers 5004 is not limited to one, and the structure of the electronic device 5000 does not constitute a limitation on the embodiments of the present application.

[0174] Processor 5001 may be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 5001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0175] The bus 5002 may include a path for transmitting information between the above components. The bus 5002 may be a PCI bus or an EISA bus, etc. The bus 5002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0176] The memory 5003 may be a ROM or other type of static storage device that can store static information and instructions, a RAM or other type of dynamic storage device that can store information and instructions, or an EEPROM, a CD-ROM or other optical disk storage, an optical disc storage (including a compact disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0177] The memory 5003 is used to store application code for executing the solution of the present application, and the execution is controlled by the processor 5001. The processor 5001 is used to execute the application code stored in the memory 5003 to implement the content shown in any of the above method embodiments.

[0178] Among them, electronic devices include but are not limited to: mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc.

[0179] The fourth aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements a multi-region integrated energy system optimization scheduling method shown in the first aspect of this application.

[0180] Another embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding contents of the aforementioned method embodiments.

[0181] In addition, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0182] In summary, the present invention relates to the technical field of energy optimization and scheduling, and discloses a method, system, equipment and medium for optimizing and scheduling a multi-region integrated energy system, including dividing a distribution network into several regions, setting an intelligent agent for the integrated energy system of each of the regions, so that each of the intelligent agents updates its own target network through a reinforcement learning algorithm to obtain a target update network; quantifying the importance weights of the parameters in each of the target update networks, and selecting several elite intelligent agents from each of the intelligent agents, and globally aggregating the parameters in the target update network of each of the elite intelligent agents according to the importance weights to obtain aggregated parameters; sharing the aggregated parameters among all intelligent agents through a soft update method, so that each of the intelligent agents after sharing the parameters solves the objective function of the distribution network, obtains an optimized scheduling plan for the multi-region integrated energy system and controls the execution of the distribution network, so as to improve the accuracy of the integrated energy optimization scheduling results.

[0183] Each embodiment in this specification is described in a progressive manner, and the same or similar parts of each embodiment can be directly referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. It should be noted that the various technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the various technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0184] The above-described embodiments merely represent several preferred implementations of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art could make several improvements and substitutions without departing from the technical principles of the present invention, and these improvements and substitutions should also be considered within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be based on the scope of protection of the claims.

Claims

1. A multi-region integrated energy system optimization scheduling method, characterized in that: include: The distribution network is divided into a plurality of regions, and an intelligent agent is set for the integrated energy system of each region, so that each intelligent agent updates its own target network through a reinforcement learning algorithm to obtain a target updated network; quantifying the importance weights of the parameters in each target update network, selecting a number of elite agents from each of the agents, and globally aggregating the parameters in the target update network of each of the elite agents according to the importance weights to obtain aggregated parameters; Sharing the aggregated parameters among all intelligent agents through a soft update method, so that each intelligent agent after sharing the parameters solves the objective function of the distribution network, obtains an optimized scheduling plan for the multi-region integrated energy system and controls the execution of the distribution network; The quantifying the importance weights of the parameters in each target update network includes: Performing forward propagation on each of the target update networks to quantify the loss function under the current strategy, and quantifying the gradient of each parameter in each of the target update networks according to the loss function under the current strategy; The absolute values of the gradients are averaged over multiple time steps to obtain the importance weight of each parameter in each target update network; wherein the importance weight is expressed by the following formula: Where, is the importance weight of the i-th parameter in the Critic network; T is the total duration of iteration before each parameter exchange; is the loss function of the Critic network of agent k at time t; is the i-th parameter in the Critic network; The step of selecting a plurality of elite intelligent agents from the intelligent agents comprises: Normalizing the reward values corresponding to the agents to obtain standardized reward values; Initialize the weight and learning rate corresponding to each of the agents, and quantify the selection probability corresponding to each of the agents based on the initialization results; wherein the selection probability is calculated by the following formula: Where, is the selection probability of the agent in the nth round of iteration; is the update ratio; is the weight corresponding to agent k; N is the total number of agents; It is the preset iteration round; The elite agent is selected by each selection probability, and the self-weight of the selected elite agent is updated by the standardized reward value and the operation support of the distribution network; wherein the self-weight update process is expressed by the following formula: Where, 、 are the weights of the elite agent in the nth and n+1th iterations respectively; D select For elite intelligent agents; is the standardized reward value of the elite agent in the nth iteration; The operational support of the distribution network; is the reward value threshold; is the selection probability of the selected agent in the nth round of iteration; Controlling each of the agents to iteratively execute the quantization step of the selection probability and the selection step of the elite agent based on the updated weight of the agent until a preset number of iteration rounds is reached, and outputting a number of selected elite agents; The sharing of the aggregation parameters is achieved by the following formula: Where, 、 Update the network parameters for the kth agent before and after sharing the aggregated parameters; is the soft update ratio; is the aggregation parameter of the jth parameter; is the jth parameter; is element-wise multiplication.

2. A multi-region integrated energy system optimization scheduling method according to claim 1, characterized in that: The method of enabling each of the intelligent agents to update its own target network through a reinforcement learning algorithm to obtain a target updated network includes: Initializing the environment and algorithm parameters of each agent, and constructing a playback buffer for each agent; Constructing a state set and an action set based on the integrated energy system of each of the regions, and constructing a reward function based on the economic cost of the distribution network, so that each of the intelligent agents interacts with the initialized environment through a deep deterministic policy gradient algorithm based on the initialized algorithm parameters, obtains a state value, an action value, and a reward value, and stores them in the playback buffer; The target network of each agent is trained and updated using the data in the playback buffer to obtain a target updated network of each agent.

3. The multi-region integrated energy system optimization scheduling method according to claim 1, characterized in that: The globally aggregating the parameters in the target update network of each elite agent according to each importance weight to obtain the aggregated parameters includes: Each of the importance weights is used as a parameter aggregation weight of each of the elite agents to globally aggregate the parameters in the target update network of each of the elite agents to obtain the aggregated parameters.

4. A multi-region integrated energy system optimization scheduling method according to claim 1, characterized in that: The method of enabling each of the intelligent agents after sharing parameters to solve the objective function of the distribution network, obtaining an optimized scheduling plan for the multi-region integrated energy system and controlling the execution of the distribution network includes: Controlling the target update network of each of the intelligent agents to share the aggregation parameters to obtain a final target network, and performing performance evaluation on each of the final target networks; When the evaluation result does not reach the preset target, repeatedly performing the network updating step and the parameter sharing step based on each of the final target networks until the evaluation result of each updated final target network reaches the preset target; Under the operating constraints of the integrated energy system in each of the regions, the objective function is constructed with the total cost of the distribution network as the target, and the objective function is solved by each of the intelligent agents when the evaluation result reaches the preset target, so as to obtain an optimized scheduling plan for the multi-region integrated energy system to control the execution of the distribution network.

5. A multi-region integrated energy system optimization and scheduling system, characterized in that: include: A network update module is used to divide the distribution network into several areas and set an intelligent agent for the integrated energy system of each area, so that each intelligent agent updates its own target network through a reinforcement learning algorithm to obtain a target updated network; a parameter aggregation module, configured to quantify the importance weights of the parameters in each target update network, select a number of elite agents from each of the agents, and globally aggregate the parameters in the target update network of each of the elite agents according to the importance weights to obtain aggregated parameters; An optimization scheduling module is used to share the aggregated parameters among all intelligent agents through a soft update method, so that each intelligent agent after sharing the parameters solves the objective function of the distribution network, obtains an optimized scheduling plan for the multi-region integrated energy system, and controls the execution of the distribution network; The quantifying the importance weights of the parameters in each target update network includes: Performing forward propagation on each of the target update networks to quantify the loss function under the current strategy, and quantifying the gradient of each parameter in each of the target update networks according to the loss function under the current strategy; The absolute values of the gradients are averaged over multiple time steps to obtain the importance weight of each parameter in each target update network; wherein the importance weight is expressed by the following formula: Where, is the importance weight of the i-th parameter in the Critic network; T is the total duration of iteration before each parameter exchange; is the loss function of the Critic network of agent k at time t; is the i-th parameter in the Critic network; The step of selecting a plurality of elite intelligent agents from the intelligent agents comprises: Normalizing the reward values corresponding to the agents to obtain standardized reward values; Initialize the weight and learning rate corresponding to each of the agents, and quantify the selection probability corresponding to each of the agents based on the initialization results; wherein the selection probability is calculated by the following formula: Where, is the selection probability of the agent in the nth round of iteration; is the update ratio; is the weight corresponding to agent k; N is the total number of agents; It is the preset iteration round; The elite agent is selected by each selection probability, and the self-weight of the selected elite agent is updated by the standardized reward value and the operation support of the distribution network; wherein the self-weight update process is expressed by the following formula: Where, 、 are the weights of the elite agents in the nth and n+1th iterations respectively; D select For elite intelligent agents; is the standardized reward value of the elite agent in the nth iteration; The operational support of the distribution network; is the reward value threshold; is the selection probability of the selected agent in the nth round of iteration; Controlling each of the agents to iteratively execute the quantization step of the selection probability and the selection step of the elite agent based on the updated weight of the agent until a preset number of iteration rounds is reached, and outputting a number of selected elite agents; The sharing of the aggregation parameters is achieved by the following formula: Where, 、 Update the network parameters for the kth agent before and after sharing the aggregated parameters; is the soft update ratio; is the aggregation parameter of the jth parameter; is the jth parameter; is element-wise multiplication.

6. An electronic device, characterized in that: It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, it implements the multi-region integrated energy system optimization scheduling method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein when the device where the computer-readable storage medium is located executes the computer program, the multi-region integrated energy system optimization scheduling method as described in any one of claims 1 to 4 is implemented.

Images