Collaborative optimization decision method, system and device of energy internet and storage medium

Abstract

A kind of collaborative optimization decision method, system, equipment and storage medium of energy internet, method includes setting reward mechanism to find optimal strategy, with optimal strategy as the training target of agent deep reinforcement learning, to solve regional energy internet operation optimization model, obtain optimal solution as action strategy;Agent according to the current operating state of energy internet, take the action strategy of the minimum value of similarity measure corresponding operating state as the initial value of deep reinforcement learning training, give corresponding action strategy after training;After action strategy is passed to environment, operating state transitions to the state of next time, by setting reward mechanism and action constraint condition, the good and bad of action strategy is rewarded, by iterative training, obtain deep reinforcement learning model;Collaborative optimization decision is made to each time action based on operating state similarity using deep reinforcement learning model.The present application can realize the fast and accurate optimization scheduling of energy internet.

Description

Technical Field

[0001] This invention belongs to the field of energy internet technology, specifically relating to a collaborative optimization decision-making method, system, device, and storage medium for energy internet. Background Technology

[0002] The Energy Internet refers to a new type of power network that integrates advanced power electronics technology, information technology, and intelligent management technology to interconnect a large number of distributed energy harvesting devices, distributed energy storage devices, and various types of loads, so as to realize bidirectional energy flow, peer-to-peer energy exchange and sharing.

[0003] Collaborative optimization in the energy internet refers to optimizing each sub-objective function while comprehensively considering the results of other sub-objective functions, ensuring consistency in the optimization results among the sub-objective functions. In the energy internet, common optimization objectives include: reducing system operating costs, improving system energy efficiency, and reducing system environmental costs.

[0004] With electricity as its core, new energy as its main body, and multi-energy complementarity as its advantage, the energy internet has become an inevitable choice for energy supply and consumption. However, because the energy internet includes multiple energy production, transmission, storage and consumption, it has a complex structure, numerous control variables, and typical nonlinear stochastic and multi-scale dynamic characteristics, making its coordinated optimization difficult.

[0005] Traditional methods based on energy mechanism models and mathematical optimization control, while highly accurate, are difficult to solve and slow, making them unsuitable for the requirements of online real-time collaborative optimization in the energy internet. Currently, the academic community is exploring the application of reinforcement learning in collaborative optimization of the energy internet. Reinforcement learning (RL), also known as rewarded learning, evaluation learning, or enhancement learning, is a paradigm and methodology in machine learning used to describe and solve problems where agents learn strategies to maximize rewards or achieve specific goals through interaction with the environment. Reinforcement learning involves the agent learning through trial and error, using rewards obtained from interaction with the environment to guide behavior, aiming to maximize the agent's reward. In reinforcement learning, the environment evaluates the quality of the agent's actions (usually a scalar signal), rather than telling the agent how to produce the correct action. Since the external environment provides limited information, the agent must learn through its own experience. In this way, the agent acquires knowledge in an action-evaluation environment and improves its action plans to adapt to the environment. Reinforcement learning views learning as an exploration and evaluation process. An agent selects an action for the environment; upon receiving the action, the environment's state changes, generating a reinforcement signal (reward or penalty) as feedback to the agent. The agent then selects its next action based on the reinforcement signal and the current state of the environment, aiming to increase the probability of receiving a positive reinforcement (reward). The chosen action affects not only the immediate reward but also the environment's state at the next moment and the final reward. The reward is an evaluation of the quality of the agent's action in reinforcement learning. However, when facing large-scale state and action spaces during training, optimization becomes extremely difficult, potentially leading to convergence problems. The action space refers to the set of actions output by the agent in reinforcement learning, while the state space refers to the set of environmental states input by the agent. Traditional mathematical optimization algorithms struggle to solve large-scale examples within a reasonable timeframe and find optimal solutions. Simply using deep reinforcement learning can result in lengthy training times, difficulty in achieving fast convergence, or even non-convergence. Summary of the Invention

[0006] The purpose of this invention is to provide a collaborative optimization decision-making method, system, device, and storage medium for the energy internet. By analyzing the similarity of power grid states, it integrates mathematical optimization algorithms and deep reinforcement learning algorithms, combining the advantages of both in terms of high accuracy in solving optimization strategies and fast model decision generation speed, to achieve rapid and accurate optimization scheduling of the energy internet.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] Firstly, a collaborative optimization decision-making method for the energy internet is provided, including:

[0009] Set up a reward mechanism to find the optimal strategy. * With the optimal strategy π * The optimal solution is used as the training target for deep reinforcement learning of the agent, and is then solved to obtain the optimal solution as the action policy.

[0010] The intelligent agent, based on the current operating state S of the energy internet t The action strategy corresponding to the minimum similarity metric λ is taken. The initial values ​​are used for training in deep reinforcement learning, and the corresponding action strategies are given after training. Action strategy After being passed to the environment, the running state transitions to the state S of the next moment. t+1 By setting reward mechanisms and action constraints, action strategies are given... The model is trained iteratively, and rewards are given according to the quality of the training.

[0011] We utilize a deep reinforcement learning model to make collaborative optimization decisions on actions at each time step based on the similarity of running states.

[0012] As an optional embodiment of the method of the present invention, it further includes an environmental setting step, wherein the environmental setting includes setting the state space and action space information of the regional energy internet, wherein the state space is the environmental information perceived by the intelligent agent, and the action space is the relevant decision variables, and the specific setting method is as follows:

[0013] When setting up the state space, the state of the agent at time t is represented by the following expression:

[0014]

[0015] In the formula, P i,t Let Q be the conventional output active power injected into node i at time t. i,t Let t be the traditional output reactive power injected into node i. Let Φ be the square of the gas pressure at node i at time t. i,t Let be the injected heat power at node i at time t. Let be the photovoltaic power output injected into node i at time t. The wind power output injected into node i at time t;

[0016] The action space of an intelligent agent is the output of controllable devices within its control area, expressed as the following expression:

[0017]

[0018] In the formula, P tmt For the active power output of the gas turbine, For reactive power output of gas turbines;

[0019] P t es Power output for electrical energy storage It provides reactive power for electrical energy storage.

[0020] As an optional embodiment of the method of the present invention, it further includes a similarity measurement setting step, wherein the energy internet environment is denoted by the following expression:

[0021]

[0022] In the formula, Let be the active power from node i to node j at time t in the network environment. Let be the reactive power from node i to node j in the network environment at time t, where node i and j are arbitrary node numbers in the network and i > j; and let be the state s at times t1 and t2. t1 With state s t2 The similarity measure between them is defined as the mean squared deviation of the state components, and the calculation expression is as follows:

[0023]

[0024] When λ≤24.5%, state s t1 With state s t2 They are similar;

[0025] When λ > 24.5%, state s t1 With state s t2 Not similar.

[0026] As an optional solution to the method of the present invention, the step of setting a reward mechanism to find the optimal strategy π * The steps, based on maximizing the expected cumulative return, are expressed as follows:

[0027]

[0028] In the formula, ρ π The trajectory formed by strategy π; T is the total number of scheduling periods per day; r t For instant rewards.

[0029] As an optional solution to the method of the present invention, the instant reward r t Each item includes operating cost items and node voltage constraint items, which are as follows:

[0030] The operating cost item aims to minimize the daily operating cost, and the objective function is:

[0031] C mt (t)=αP t mt

[0032] In the formula, α is the cost coefficient;

[0033] If we define the cost per kilowatt-hour as ρ, then the operating cost of energy storage is:

[0034] C b (t)=ρ|P t es |

[0035] The cost of purchasing electricity from the upper-level power grid is:

[0036]

[0037] P t grid =P t pv +P t es +P t mt -P t load

[0038] In the formula, λ buy (t) and λ sell (t) represent the electricity purchase and sales prices of the upstream power grid during time period t, respectively;

[0039] P t grid P represents the electrical power purchased from the upstream power grid. t grid >0 indicates electricity purchase, P t grid <0 indicates electricity sales;

[0040] Set instant rewards t The first term is as follows:

[0041] F1(t)=C mt (t)+C b (t)-C grid (t)

[0042] The lower the system operating cost during the scheduling period t, the greater the reward given to each agent during that period.

[0043] The node voltage constraint term indicates that the voltage amplitude of each node must be limited to a safe range, and its expression is:

[0044] 0.95u N ≤u j ≤1.05u N

[0045] In the formula, u j,t Let u be the per-unit value of the voltage at node j at time t; N This is the system's rated voltage;

[0046] After the computational agent makes a scheduling decision, the system power flow distribution is calculated, and the agent receives the following reward based on the voltage of each node:

[0047]

[0048] In the formula, k is the penalty coefficient;

[0049] The environment provides the same global reward to the regional agent at each time interval, as shown in the following formula:

[0050] r t =F1(t)+F2(t).

[0051] As an optional embodiment of the method of the present invention, the step of solving the pre-constructed regional energy internet operation optimization model to obtain the optimal solution is used as the action strategy. The steps include:

[0052] The expression for the regional energy internet operation optimization model is as follows:

[0053]

[0054]

[0055] In the formula: X is the state variable, and Y is the control variable. Y , These are the upper and lower bounds of the control variable, respectively.

[0056] f(X,Y) is the objective function, h(X,Y) is the power flow equation, and g(X,Y) is the constraint condition;

[0057] The objective function is a weighted average of multiple objective functions, including: economic objectives, power curtailment objectives, environmental objectives, and energy conversion efficiency objectives. Constraints include: equipment operating parameter constraints, production equipment parameter constraints, energy storage equipment constraints, and energy balance constraints. The regional energy internet operation optimization model is a nonlinear, nonconvex model, which is transformed into a convex optimization problem through convex relaxation. The optimal solution is obtained using the MOSEK optimization solver and denoted as .

[0058] As an optional embodiment of the method of the present invention, the step of using a deep reinforcement learning model to make collaborative optimization decisions on actions at each time step based on the similarity of running states includes:

[0059] Let the current state of the environment E be s.t , determine s t The optimal set of state-action pairs obtained through reward mechanisms and action constraints All states Similarity measure {λ i};

[0060] If λ exists i ≤24.5% indicates that a certain state in the optimal solution has been obtained. With the current state s t Similarity leads to the next action. To select the optimal action obtained through mathematical methods for the next action;

[0061] If all λ i >24.5%, meaning there is no state s that is related to the current state. t If a similar state has already yielded the optimal solution, then proceed to the next action. The next action should be selected based on the optimal action obtained through reinforcement learning.

[0062] Then the action is passed to the environment E, updating the next state s. t+1 And so on.

[0063] Secondly, a collaborative optimization decision-making system for the energy internet is provided, including:

[0064] The action strategy solving module is used to set up a reward mechanism to find the optimal strategy π. * With the optimal strategy π * The optimal solution is used as the training target for deep reinforcement learning of the agent, and is then solved to obtain the optimal solution as the action policy.

[0065] The deep reinforcement learning model training module is used by the agent to train the model based on the current operating state S of the energy internet. t The action strategy corresponding to the minimum similarity metric λ is taken as the running state. The initial values ​​are used for training in deep reinforcement learning, and the corresponding action strategies are given after training. Action strategy After being passed to the environment, the running state transitions to the state S of the next moment. t+1 By setting reward mechanisms and action constraints, action strategies are given... The model is trained iteratively, and rewards are given according to the quality of the training.

[0066] The action decision-making collaborative optimization module is used to make collaborative optimization decisions on actions at each time step based on the similarity of the running states using a deep reinforcement learning model.

[0067] As a preferred embodiment of the system of the present invention, it also includes an environment setting module. The environment setting module sets the state space and action space information of the regional energy internet. The state space is the environmental information perceived by the intelligent agent, and the action space is the relevant decision variables. The specific setting method is as follows:

[0068] When setting up the state space, the state of the agent at time t is represented by the following expression:

[0069]

[0070] In the formula, P i,t Let Q be the conventional output active power injected into node i at time t. i,t Let t be the traditional output reactive power injected into node i. Let Φ be the square of the gas pressure at node i at time t. i,t Let be the injected heat power at node i at time t. Let be the photovoltaic power output injected into node i at time t. The wind power output injected into node i at time t;

[0071] The action space of an intelligent agent is the output of controllable devices within its control area, expressed as the following expression:

[0072]

[0073] In the formula, P t mt For the active power output of the gas turbine, For reactive power output of gas turbines;

[0074] P t es Power output for electrical energy storage It provides reactive power for electrical energy storage.

[0075] As a preferred embodiment of the system of the present invention, it further includes a similarity measurement setting module, wherein the energy internet environment is represented by the following expression:

[0076]

[0077] In the formula, Let be the active power from node i to node j at time t in the network environment. Let be the reactive power from node i to node j in the network environment at time t, where node i and j are arbitrary node numbers in the network and i > j; and let be the state s at times t1 and t2. t1 With state s t2 The similarity measure between them is defined as the mean squared deviation of the state components, and the calculation expression is as follows:

[0078]

[0079] When λ≤24.5%, state s t1 With state s t2 They are similar;

[0080] When λ > 24.5%, state s t1 With state s t2 Not similar.

[0081] As a preferred embodiment of the system of the present invention, the action strategy solving module sets a reward mechanism to find the optimal strategy π. * The steps, based on maximizing the expected cumulative return, are expressed as follows:

[0082]

[0083] In the formula, ρ π The trajectory formed by strategy π; T is the total number of scheduling periods per day; r t For instant rewards.

[0084] As a preferred embodiment of the system of the present invention, the instantaneous reward r set by the action strategy solving module is... t Each item includes operating cost items and node voltage constraint items;

[0085] The operating cost item aims to minimize the daily operating cost, and the objective function is:

[0086] C mt (t)=αP t mt

[0087] In the formula, α is the cost coefficient;

[0088] If we define the cost per kilowatt-hour as ρ, then the operating cost of energy storage is:

[0089] C b (t)=ρ|P t es |

[0090] The cost of purchasing electricity from the upper-level power grid is:

[0091]

[0092] P t grid =P t pv +P t es +P t mt -P tload

[0093] In the formula, λ buy (t) and λ sell (t) represent the electricity purchase and sales prices of the upstream power grid during time period t, respectively;

[0094] P t grid P represents the electrical power purchased from the upstream power grid. t grid >0 indicates electricity purchase, P t grid <0 indicates electricity sales;

[0095] Set instant rewards t The first term is as follows:

[0096] F1(t)=C mt (t)+C b (t)-C grid (t)

[0097] The lower the system operating cost during the scheduling period t, the greater the reward given to each agent during that period.

[0098] The node voltage constraint term indicates that the voltage amplitude of each node must be limited to a safe range, and its expression is:

[0099] 0.95u N ≤u j ≤1.05u N

[0100] In the formula, u j,t Let u be the per-unit value of the voltage at node j at time t; N This is the system's rated voltage;

[0101] After the computational agent makes a scheduling decision, the system power flow distribution is calculated, and the agent receives the following reward based on the voltage of each node:

[0102]

[0103] In the formula, k is the penalty coefficient;

[0104] The environment provides the same global reward to the regional agent at each time interval, as shown in the following formula:

[0105] r t =F1(t)+F2(t).

[0106] As a preferred embodiment of the system of the present invention, the action strategy solving module solves the pre-constructed regional energy internet operation optimization model to obtain the optimal solution as the action strategy. The steps outlined in the regional energy internet operation optimization model expression are as follows:

[0107]

[0108]

[0109] In the formula: X is the state variable, and Y is the control variable. Y , These are the upper and lower bounds of the control variable, respectively.

[0110] f(X,Y) is the objective function, h(X,Y) is the power flow equation, and g(X,Y) is the constraint condition;

[0111] The objective function is a weighted average of multiple objective functions, including: economic objectives, power curtailment objectives, environmental objectives, and energy conversion efficiency objectives. Constraints include: equipment operating parameter constraints, production equipment parameter constraints, energy storage equipment constraints, and energy balance constraints. The regional energy internet operation optimization model is a nonlinear, nonconvex model, which is transformed into a convex optimization problem through convex relaxation. The optimal solution is obtained using the MOSEK optimization solver and denoted as .

[0112] As a preferred embodiment of the system of the present invention, the action decision-making collaborative optimization module sets the current state of the environment E as s. t , determine s t The optimal set of state-action pairs obtained through reward mechanisms and action constraints All states Similarity measure {λ i};

[0113] If λ exists i ≤24.5% indicates that a certain state in the optimal solution has been obtained. With the current state s t Similarity leads to the next action. To select the optimal action obtained through mathematical methods for the next action;

[0114] If all λ i >24.5%, meaning there is no state s that is related to the current state. t If a similar state has already yielded the optimal solution, then proceed to the next action. The next action should be selected based on the optimal action obtained through reinforcement learning.

[0115] Then the action is passed to the environment E, updating the next state s. t+1 And so on.

[0116] Thirdly, an electronic device is provided, the electronic device comprising:

[0117] At least one processor; and,

[0118] A memory communicatively connected to the at least one processor; wherein,

[0119] The memory stores instructions that can be executed by the at least one processor, which enables the at least one processor to perform steps in the collaborative optimization decision-making method of the energy internet.

[0120] Fourthly, a computer-readable storage medium is provided, storing a computer program that, when executed by a processor, implements the steps in the collaborative optimization decision-making method of the energy internet.

[0121] Compared with the prior art, the first aspect of the present invention has at least the following beneficial effects:

[0122] Due to the complex constraints, large scale, and multiple operating states of schedulable devices within large-scale systems, as well as strong nonlinearity, traditional mathematical optimization algorithms struggle to solve large-scale examples within a limited time and find optimal solutions. Furthermore, the large action space makes deep reinforcement learning time-consuming and difficult to converge quickly during training. This invention proposes a collaborative optimization decision-making method. Firstly, it accelerates the reinforcement learning optimization process by assigning the mathematically optimal state decision solution to the initial values ​​of the reinforcement learning action variables through state similarity calculation, ultimately obtaining a numerical state decision solution. This improves training speed while maintaining the model's output accuracy. Secondly, when optimizing decisions for real-time online output, for systems distributed across... For states near or similar to the sample, the optimal action is directly obtained using mathematical methods derived during model building. As for distance For states where the sample is distant and dissimilar, the deep reinforcement learning model built during the training phase outputs the action policy. This process also improves the speed of output decision-making by effectively utilizing the mathematically optimal action solution.

[0123] It is understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0124] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0125] Figure 1 A schematic diagram illustrating the principle of the collaborative optimization decision-making method for the energy internet according to an embodiment of the present invention;

[0126] Figure 2 A schematic diagram illustrating the deep reinforcement learning optimization principle of existing technical solutions for regional energy internet;

[0127] Figure 3 A flowchart illustrating the solution process for the regional energy internet operation optimization model in this embodiment of the invention;

[0128] Figure 4 A flowchart illustrating the training of a deep reinforcement learning model according to an embodiment of the present invention. Detailed Implementation

[0129] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0130] Based on the embodiments of the present invention, those skilled in the art can make several simple modifications and refinements without creative effort, and all other embodiments obtained are within the scope of protection of the present invention.

[0131] In this invention, reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this invention can be combined with other embodiments.

[0132] Existing technologies offer two main solutions for the coordinated optimization of the energy internet's operation:

[0133] Option 1: Regional Energy Internet Mathematical Optimization Model. This approach starts with a mathematical model, addressing non-convex optimization problems in the regional energy internet by employing linearization or convex relaxation techniques to approximate the original optimization problem as a linear programming or mixed-integer programming problem. If the objective function in a mathematical optimization problem is non-convex, or the feasible region formed by constraints is a non-convex set, then this optimization problem is called a non-convex optimization problem. Non-convex optimization problems are generally NP-hard, complex, and difficult to solve. Convex relaxation optimization problems are relaxation (approximation) models of non-convex problems, which can be solved quickly using optimization solvers. Linearization refers to linearizing the nonlinear equations describing the relationship between pipeline pressure and flow in natural gas systems and the changes in temperature and mass flow rate in thermal systems. Incremental linearization is a commonly used method, transforming the problem into a mixed-integer linear programming problem by introducing 0-1 variables. Convex relaxation techniques are also widely used to handle nonlinear component and network constraints.

[0134] Option 2, Regional Energy Internet Deep Reinforcement Learning Optimization Model: This model combines deep learning and reinforcement learning. The agent learns through trial and error, receiving rewards that guide its behavior based on interactions with the regional energy internet environment. The goal is to maximize the agent's reward. In reinforcement learning, the regional energy internet environment evaluates the quality of the agent's collaborative optimization decisions. The agent learns from its own experiences. In this way, the agent acquires knowledge in an action-evaluation environment and improves its collaborative optimization strategies to adapt to the environment. Figure 2 As shown.

[0135] The drawbacks of Scheme 1's mathematical optimization model: The operation of a regional energy internet involves the coupling of multiple energy forms such as electricity, gas, and heat. The mathematical essence of its operational optimization is a large-scale non-convex optimization problem. Currently, algorithms for solving non-convex optimization problems are divided into global optimal algorithms and non-global optimal algorithms. If a globally optimal solution is to be obtained, the scale of the solution is limited, making it difficult to solve large-scale examples within a reasonable timeframe. Methods that seek local optima generally simplify the model, losing accuracy and causing a mismatch between the solution and practical applications. However, due to the complex constraints, large scale, multiple operating states, and strong nonlinearity of the schedulable equipment operation model within a large-scale system, traditional optimization methods struggle to meet operational requirements. Furthermore, the operation of a regional energy internet involves multiple uncertainties, making it difficult to accurately characterize its uncertain states using mathematical formulas. Traditional mathematical programming methods are highly dependent on the model; inaccurate modeling makes it difficult to apply the optimal solution to practical applications.

[0136] Disadvantages of Scheme 2 (Deep Reinforcement Learning Model): With the large-scale integration of distributed power sources, the effective utilization of distributed renewable energy is fully satisfied, thereby meeting the diverse energy needs of users. Therefore, the regional energy internet collaborative optimization decision-making based on deep reinforcement learning mainly faces the following problems: First, the integration of renewable energy into the energy internet increases the complexity of the energy internet environment model, leading to an increase in the dimensionality of the deep reinforcement learning state space. Second, since the regional energy internet involves multiple controllable energy variables such as electricity, gas, and heat, the increased dimensionality of the action space makes the algorithm more prone to the curse of dimensionality, resulting in long training times and difficulty in achieving rapid convergence. Finally, since the optimization of the regional energy internet involves multi-energy collaborative optimization, setting the reward function to guide the agent in making optimal decision-making actions presents certain challenges.

[0137] Traditional methods based on energy mechanism models and mathematical optimization control, while highly accurate, are difficult and slow to solve, making them unsuitable for the requirements of online real-time collaborative optimization in the energy internet. While deep reinforcement learning models, after training, can quickly output decision actions upon inputting new states, they may struggle to converge when faced with large-scale state and action spaces during training. This invention presents a runtime optimization decision-making method that combines the advantages of mathematical programming optimization and deep reinforcement learning models, possessing both the high accuracy of mathematical methods and the rapid decision output capability of deep reinforcement learning models.

[0138] like Figure 1 As shown, the collaborative optimization decision-making method for an energy internet according to an embodiment of the present invention mainly includes the following six parts: (1) environment setting; (2) state similarity measurement; (3) reward mechanism setting; (4) mathematical optimization solution method; (5) reinforcement learning solution method based on the initial value of the optimal solution; and (6) collaborative optimization decision-making method based on the similarity of operating states. In the model training stage, the optimal decision solution is first obtained for some specific operating states of the energy internet through mathematical optimization method; then, when training the reinforcement learning model, the mathematical optimal decision solution obtained by its neighboring states is used as the initial action for model training. In the model decision-making stage, for states similar to those with mathematical optimal decision solutions, the mathematical optimal decision solution obtained by the similar states is used as the decision action; for other states, the action output by the reinforcement learning model is used as the decision action.

[0139] The objectives and related operations of the above six parts are as follows: (1) Environment setting, specifically the state space and action space information of the regional energy internet. The state space is the environmental information perceived by the agent, and the action space is the relevant decision variables. (2) State similarity measurement, specifically the similarity measurement method of different operating states of the energy internet, which provides judgment conditions for selecting the optimal initial value of the reinforcement learning solution method based on the optimal initial value in part (5) and the action selection in the collaborative optimization decision method based on the similarity of operating states in part (6). (3) Reward mechanism setting, the training objective of reinforcement learning is to find the optimal policy π. * To maximize the expected cumulative return, the instant reward r designed in this embodiment of the invention... t Each item includes operating cost and node voltage constraint. (4) Mathematical optimization solution method, which is the process of optimizing the operation of the regional energy internet based on the operations research optimization model. (5) Reinforcement learning solution method based on the initial value of the optimal solution, which is the process of optimizing the regional energy internet based on deep reinforcement learning. (6) Collaborative optimization decision method based on similarity of operating states, which gives how to select the optimal collaborative optimization action of the energy internet under different operating states.

[0140] Example 1

[0141] A collaborative optimization decision-making method for the energy internet includes:

[0142] S1. Set up a reward mechanism to find the optimal strategy π * With the optimal strategy π * The optimal solution is used as the training target for deep reinforcement learning of the agent, and is then solved to obtain the optimal solution as the action policy.

[0143] S2, the intelligent agent, based on the current operating status of the energy internet, S t The action strategy corresponding to the minimum similarity metric λ is taken as the running state. The initial values ​​are used for training in deep reinforcement learning, and the corresponding action strategies are given after training. Action strategy After being passed to the environment, the running state transitions to the state S of the next moment. t+1 By setting reward mechanisms and action constraints, action strategies are given... The model is trained iteratively, and rewards are given according to the quality of the training.

[0144] S3. Utilize a deep reinforcement learning model to make collaborative optimization decisions on actions at each time step based on the similarity of running states.

[0145] In an optional implementation, the collaborative optimization decision-making method for the energy internet of the present invention further includes an environmental setting step. The environmental setting includes setting the state space and action space information of the regional energy internet. The state space is the environmental information perceived by the agent, and the action space consists of relevant decision variables. The specific setting method is as follows:

[0146] When setting up the state space, the state of the agent at time t is represented by the following expression:

[0147]

[0148] In the formula, P i,t Let Q be the conventional output active power injected into node i at time t. i,t Let t be the traditional output reactive power injected into node i. Let Φ be the square of the gas pressure at node i at time t. i,t Let be the injected heat power at node i at time t. Let be the photovoltaic power output injected into node i at time t. The wind power output injected into node i at time t;

[0149] The action space of an intelligent agent is the output of controllable devices within its control area, expressed as the following expression:

[0150]

[0151] In the formula, P t mt For the active power output of the gas turbine, For reactive power output of gas turbines;

[0152] P t es Power output for electrical energy storage It provides reactive power for electrical energy storage.

[0153] In an optional implementation, the collaborative optimization decision-making method for the energy internet of the present invention further includes a similarity metric setting step, wherein the energy internet environment is denoted by the following expression:

[0154]

[0155] In the formula, Let be the active power from node i to node j at time t in the network environment. Let be the reactive power from node i to node j at time t in the network environment, where node i and j are arbitrary node numbers in the network and i>j;

[0156] The states s at time t1 and time t2 t1 With state s t2The similarity measure between them is defined as the mean square value of the deviation of the state components (the deviation of each component needs to be normalized), and the calculation expression is as follows:

[0157]

[0158] When λ≤24.5%, state s t1 With state s t2 They are similar;

[0159] When λ > 24.5%, state s t1 With state s t2 Not similar.

[0160] In an optional implementation, step S1 involves setting a reward mechanism to find the optimal strategy π. * The steps, based on maximizing the expected cumulative return, are expressed as follows:

[0161]

[0162] In the formula, ρ π The trajectory formed by strategy π; T is the total number of scheduling periods per day; r t For instant rewards.

[0163] In one alternative implementation, an instant reward r is set. t Each item includes operating cost items and node voltage constraint items.

[0164] The operating cost item and the node voltage constraint item are as follows:

[0165] The operating cost item aims to minimize the daily operating cost, and the objective function is:

[0166] C mt (t)=αP t mt

[0167] In the formula, α is the cost coefficient;

[0168] If we define the cost per kilowatt-hour as ρ, then the operating cost of energy storage is:

[0169] C b (t)=ρ|P t es |

[0170] The cost of purchasing electricity from the upper-level power grid is:

[0171]

[0172] P t grid =P tpv +P t es +P t mt -P t load

[0173] In the formula, λ buy (t) and λ sell (t) represent the electricity purchase and sales prices of the upstream power grid during time period t, respectively;

[0174] P t grid P represents the electrical power purchased from the upstream power grid. t grid >0 indicates electricity purchase, P t grid <0 indicates electricity sales;

[0175] Set instant rewards t The first term is as follows:

[0176] F1(t)=C mt (t)+C b (t)-C grid (t)

[0177] The lower the system operating cost during the scheduling period t, the greater the reward given to each agent during that period.

[0178] The node voltage constraint term indicates that the voltage amplitude of each node must be limited to a safe range, and its expression is:

[0179] 0.95u N ≤u j ≤1.05u N

[0180] In the formula, u j,t Let u be the per-unit value of the voltage at node j at time t; N This is the system's rated voltage;

[0181] After the computational agent makes a scheduling decision, the system power flow distribution is calculated, and the agent receives the following reward based on the calculated voltage of each node:

[0182]

[0183] In the formula, k is the penalty coefficient;

[0184] The environment provides the same global reward to the regional agent at each time interval, as shown in the following formula:

[0185] r t =F1(t)+F2(t).

[0186] In one optional implementation, step S1 involves solving a pre-built regional energy internet operation optimization model to obtain the optimal solution as the action strategy. The steps include:

[0187] The expression for the regional energy internet operation optimization model is as follows:

[0188]

[0189]

[0190] In the formula: X is the state variable, and Y is the control variable. Y , These are the upper and lower bounds of the control variable, respectively.

[0191] f(X,Y) is the objective function, h(X,Y) is the power flow equation, and g(X,Y) is the constraint condition;

[0192] The objective function is a weighted average of multiple objective functions, including: economic objectives, curtailment objectives, environmental objectives, and energy conversion efficiency objectives; constraints include: equipment operating parameter constraints, production equipment parameter constraints, energy storage equipment constraints, and energy balance constraints; see [link to relevant documentation]. Figure 3 The regional energy internet operation optimization model is a nonlinear, nonconvex model. It is transformed into a convex optimization problem through convex relaxation, and solved using the MOSEK optimization solver to obtain the optimal solution, denoted as .

[0193] In one alternative implementation, the process of training the deep reinforcement learning model in step S2 is as follows: Figure 4 As shown, step S3, which involves using a deep reinforcement learning model to make collaborative optimization decisions on actions at each time step based on the similarity of running states, includes:

[0194] Let the current state of the environment E be s. t , determine s t The optimal set of state-action pairs obtained through reward mechanisms and action constraints All states Similarity measure {λ i};

[0195] If λ exists i ≤24.5% indicates that a certain state in the optimal solution has been obtained. With the current state s t Similarity leads to the next action. To select the optimal action obtained through mathematical methods for the next action;

[0196] If all λ i>24.5%, meaning there is no state s that is related to the current state. t If a similar state has already yielded the optimal solution, then proceed to the next action. The next action should be selected based on the optimal action obtained through reinforcement learning.

[0197] Then the action is passed to the environment E, updating the next state s. t+1 And so on.

[0198] Example 2

[0199] An embodiment of the present invention provides a collaborative optimization decision-making system for the energy internet, comprising:

[0200] Action strategy solving module 1 is used to set up a reward mechanism to find the optimal strategy π. * With the optimal strategy π * The optimal solution is used as the training target for deep reinforcement learning of the agent, and is then solved to obtain the optimal solution as the action policy.

[0201] Deep reinforcement learning model training module 2 is used by the agent to train the model based on the current operating state S of the energy internet. t The action strategy corresponding to the minimum similarity metric λ is taken. The initial values ​​are used for training in deep reinforcement learning, and the corresponding action strategies are given after training. Action strategy After being passed to the environment, the running state transitions to the state S of the next moment. t+1 By setting reward mechanisms and action constraints, action strategies are given... The model is trained iteratively, and rewards are given according to the quality of the training.

[0202] Action Decision Collaborative Optimization Module 3 is used to make collaborative optimization decisions on actions at each time step based on the similarity of running states using a deep reinforcement learning model.

[0203] In one possible implementation, the collaborative optimization decision-making system for the energy internet of the present invention further includes an environment setting module. This module sets the state space and action space information of the regional energy internet. The state space is the environmental information perceived by the agent, and the action space consists of relevant decision variables. The specific setting method is as follows:

[0204] When setting up the state space, the state of the agent at time t is represented by the following expression:

[0205]

[0206] In the formula, P i,tLet Q be the conventional output active power injected into node i at time t. i,t Let t be the traditional output reactive power injected into node i. Let Φ be the square of the gas pressure at node i at time t. i,t Let be the injected heat power at node i at time t. Let be the photovoltaic power output injected into node i at time t. The wind power output injected into node i at time t;

[0207] The action space of an intelligent agent is the output of controllable devices within its control area, expressed as the following expression:

[0208]

[0209] In the formula, P t mt For the active power output of the gas turbine, For reactive power output of gas turbines;

[0210] P t es Power output for electrical energy storage It provides reactive power for electrical energy storage.

[0211] In one possible implementation, the collaborative optimization decision-making system for the energy internet of the present invention further includes a similarity measurement setting module, wherein the energy internet environment is defined by the following expression:

[0212]

[0213] In the formula, Let be the active power from node i to node j at time t in the network environment. Let be the reactive power from node i to node j in the network environment at time t, where node i and j are arbitrary node numbers in the network and i > j; and let be the state s at times t1 and t2. t1 With state s t2 The similarity measure between them is defined as the mean squared deviation of the state components, and the calculation expression is as follows:

[0214]

[0215] When λ≤24.5%, state s t1 With state s t2 They are similar;

[0216] When λ > 24.5%, state s t1 With state s t2 Not similar.

[0217] In one possible implementation, the action policy solving module 1 sets up a reward mechanism to find the optimal policy π. * The steps, based on maximizing the expected cumulative return, are expressed as follows:

[0218]

[0219] In the formula, ρ π The trajectory formed by strategy π; T is the total number of scheduling periods per day; r t For instant rewards.

[0220] In one possible implementation, the action policy solving module 1 sets the instantaneous reward r. t Each item includes operating cost items and node voltage constraint items;

[0221] The operating cost item aims to minimize the daily operating cost, and the objective function is:

[0222] C mt (t)=αP t mt

[0223] In the formula, α is the cost coefficient;

[0224] If we define the cost per kilowatt-hour as ρ, then the operating cost of energy storage is:

[0225] C b (t)=ρ|P t es |

[0226] The cost of purchasing electricity from the upper-level power grid is:

[0227]

[0228] P t grid =P t pv +P t es +P t mt -P t load

[0229] In the formula, λ buy (t) and λ sell (t) represent the electricity purchase and sales prices of the upstream power grid during time period t, respectively;

[0230] P t grid P represents the electrical power purchased from the upstream power grid. t grid >0 indicates electricity purchase, P tgrid <0 indicates electricity sales;

[0231] Set instant rewards t The first term is as follows:

[0232] F1(t)=C mt (t)+C b (t)-C grid (t)

[0233] The lower the system operating cost during the scheduling period t, the greater the reward given to each agent during that period.

[0234] The node voltage constraint term indicates that the voltage amplitude of each node must be limited to a safe range, and its expression is:

[0235] 0.95u N ≤u j ≤1.05u N

[0236] In the formula, u j,t Let u be the per-unit value of the voltage at node j at time t; N This is the system's rated voltage;

[0237] After the computational agent makes a scheduling decision, the system power flow distribution is calculated, and the agent receives the following reward based on the voltage of each node:

[0238]

[0239] In the formula, k is the penalty coefficient;

[0240] The environment provides the same global reward to the regional agent at each time interval, as shown in the following formula:

[0241] r t =F1(t)+F2(t).

[0242] In one possible implementation, the action strategy solving module 1 solves the pre-built regional energy internet operation optimization model to obtain the optimal solution as the action strategy. Steps;

[0243] The expression for the regional energy internet operation optimization model is as follows:

[0244]

[0245]

[0246] In the formula: X is the state variable, and Y is the control variable. Y , These are the upper and lower bounds of the control variable, respectively.

[0247] f(X,Y) is the objective function, h(X,Y) is the power flow equation, and g(X,Y) is the constraint condition;

[0248] The objective function is a weighted average of multiple objective functions, including: economic objectives, power curtailment objectives, environmental objectives, and energy conversion efficiency objectives. Constraints include: equipment operating parameter constraints, production equipment parameter constraints, energy storage equipment constraints, and energy balance constraints. The regional energy internet operation optimization model is a nonlinear, nonconvex model, which is transformed into a convex optimization problem through convex relaxation. The optimal solution is obtained using the MOSEK optimization solver and denoted as .

[0249] In one possible implementation, the action decision-making collaborative optimization module 3 records the current state of the environment E as s. t , determine s t The optimal set of state-action pairs obtained through reward mechanisms and action constraints All states Similarity measure {λ i};

[0250] If λ exists i ≤24.5% indicates that a certain state in the optimal solution has been obtained. With the current state s t Similarity leads to the next action. To select the optimal action obtained through mathematical methods for the next action;

[0251] If all λ i >24.5%, meaning there is no state s that is related to the current state. t If a similar state has already yielded the optimal solution, then proceed to the next action. The next action should be selected based on the optimal action obtained through reinforcement learning.

[0252] Then the action is passed to the environment E, updating the next state s. t+1 And so on.

[0253] Because the constraints of schedulable device operation models in large-scale systems are complex and large-scale, and they have multiple operating states and strong nonlinearity, traditional mathematical optimization algorithms struggle to solve large-scale examples within a reasonable timeframe and find optimal solutions. Furthermore, due to the large action space, deep reinforcement learning suffers from long training times and difficulty in achieving fast convergence. This invention addresses this issue by, during model training, including the inclusion of S in the training samples... train Perform identically distributed sampling to draw a small number of samples Find the optimal state-decision pair using mathematical methods Based on the remaining training samples By calculating state similarity, the mathematically optimal state decision solution is assigned to the initial values ​​of the reinforcement learning action variables to accelerate the reinforcement learning optimization process, and finally, a numerical state decision solution is obtained through training. This improves training speed while maintaining model output accuracy. On the other hand, when optimizing decisions for real-time online output, for models distributed... For states near or similar to the sample, the optimal action is directly obtained using mathematical methods derived during model building. For distance For states where the sample is distant and dissimilar, the deep reinforcement learning model built during the training phase outputs the action policy. This process also improves the speed of output decision-making by effectively utilizing the mathematically optimal action solution.

[0254] Example 3

[0255] This invention also proposes an electronic device, the electronic device comprising:

[0256] At least one processor; and,

[0257] A memory communicatively connected to the at least one processor; wherein,

[0258] The memory stores instructions that can be executed by the at least one processor, which enables the at least one processor to perform steps in the collaborative optimization decision-making method of the energy internet.

[0259] Example 4

[0260] This invention also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the collaborative optimization decision-making method of the energy internet.

[0261] The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals. For ease of explanation, the above content only shows the parts related to the embodiments of the present invention; for specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. This computer-readable storage medium is non-transitory and can be stored in storage devices formed by various electronic devices, enabling the execution process described in the method of the embodiments of the present invention.

[0262] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0263] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0264] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0265] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0266] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A collaborative optimization decision-making method for an energy internet, characterized in that, include: Set up a reward mechanism to find the optimal strategy. With optimal strategy The optimal solution is used as the training target for deep reinforcement learning of the agent, and is then solved to obtain the optimal solution as the action policy. ; The intelligent agent is based on the current operating status of the energy internet. Take similarity measure Action strategy for the running state corresponding to the minimum value The initial values ​​are used for training in deep reinforcement learning, and the corresponding action strategies are given after training. Action strategy After being passed to the environment, the running state transitions to the state of the next moment. By setting reward mechanisms and action constraints, action strategies are given... The model is trained iteratively, and rewards are given according to the quality of the training. Utilizing a deep reinforcement learning model to make collaborative optimization decisions for actions at each time step based on the similarity of runtime states, including: The current state of environment E is set as follows: ,judge The optimal set of state-action pairs obtained through reward mechanisms and action constraints All states Similarity measurement ; If it exists This indicates that a certain state in the optimal solution has been obtained. Compared with the current state Similarity leads to the next action. This allows the next action to be selected using mathematical methods to obtain the optimal action. If all That is, there is no state that is the same as the current state. If a similar state has already yielded the optimal solution, then proceed to the next action. This allows the next action to be selected using the optimal action obtained through reinforcement learning. Then the action is passed to the environment E to update the next state. And so on.

2. The collaborative optimization decision-making method for the energy internet according to claim 1, characterized in that, It also includes a step of setting up the environment, which involves setting the state space and action space information of the regional energy internet. The state space is the environmental information perceived by the agent, and the action space is the relevant decision variables. The specific setting method is as follows: When setting up the state space, The state of the agent at any given time is represented by the following expression: In the formula, for Injecting nodes at all times The traditional output active power, for Injecting nodes at all times Traditional output reactive power, for Time Node The square of the gas pressure. for Time Node Injected heat power, for Time Node Injected photovoltaic power output, for Time Node The injected wind power output; The action space of an intelligent agent is the output of controllable devices within its control area, expressed as the following expression: In the formula, For the active power output of the gas turbine, For reactive power output of gas turbines; Power output for electrical energy storage It provides reactive power for electrical energy storage.

3. The collaborative optimization decision-making method for the energy internet according to claim 2, characterized in that, It also includes steps for setting up similarity metrics, denoted as the Energy Internet environment as follows: In the formula, For the network environment at any time t node i To the node j active power, For the network environment at all times t node i To the node j The reactive power, of which the node i and j Number any node in the network and i > j ;time With time status With state The similarity measure between them is defined as the mean squared deviation of the state components, and the calculation expression is as follows: when At that time, state With state They are similar; when At that time, state With state Not similar.

4. The collaborative optimization decision-making method for the energy internet according to claim 1, characterized in that, The setting of a reward mechanism to find the optimal strategy The steps, based on maximizing the expected cumulative return, are expressed as follows: In the formula, For strategy The trajectory formed; T This represents the total number of scheduling periods per day. For instant rewards.

5. The collaborative optimization decision-making method for the energy internet according to claim 4, characterized in that, The instant reward Each item includes operating cost items and node voltage constraint items, which are as follows: The operating cost item aims to minimize the daily operating cost, and the objective function is: In the formula, This is the cost coefficient; Define the cost per kilowatt-hour as The operating cost of energy storage is: The cost of purchasing electricity from the upper-level power grid is: In the formula, and They represent t Electricity purchase and sales prices from the upstream power grid during the specified time period; This indicates the electrical power purchased from the upstream power grid. Indicates the purchase of electricity. Indicates electricity sales; Set instant rewards The first term is as follows: Scheduling period t The lower the system operating cost, the greater the reward given to each agent during that period. The node voltage constraint term indicates that the voltage amplitude of each node must be limited to a safe range, and its expression is: In the formula, for t Time Node j Per-unit voltage value; This is the system's rated voltage; After the computational agent makes a scheduling decision, the system power flow distribution is calculated, and the agent receives the following reward based on the voltage of each node: In the formula, k This is the penalty coefficient; The environment provides the same global reward to the regional agent at each time interval, as shown in the following formula: 。 6. The collaborative optimization decision-making method for the energy internet according to claim 1, characterized in that, The optimal solution obtained by solving the pre-constructed regional energy internet operation optimization model is used as the action strategy. The steps include: The expression for the regional energy internet operation optimization model is as follows: In the formula: For state variables, To control variables, These are the upper and lower bounds of the control variable, respectively. Let be the objective function. For the current equation. These are constraints; The objective function is a weighted average of multiple objective functions, including: economic objectives, power curtailment objectives, environmental objectives, and energy conversion efficiency objectives. Constraints include: equipment operating parameter constraints, production equipment parameter constraints, energy storage equipment constraints, and energy balance constraints. The regional energy internet operation optimization model is a nonlinear, non-convex model, which is transformed into a convex optimization problem through convex relaxation. The optimal solution is obtained using the MOSEK optimization solver and denoted as . .

7. A collaborative optimization decision-making system for the energy internet, characterized in that, include: The action strategy solving module is used to set up a reward mechanism to find the optimal strategy. With optimal strategy The optimal solution is used as the training target for deep reinforcement learning of the agent, and is then solved to obtain the optimal solution as the action policy. ; The deep reinforcement learning model training module is used by the agent to train the model based on the current operating state of the energy internet. Take similarity measure Action strategy for the running state corresponding to the minimum value The initial values ​​are used for training in deep reinforcement learning, and the corresponding action strategies are given after training. Action strategy After being passed to the environment, the running state transitions to the state of the next moment. By setting reward mechanisms and action constraints, action strategies are given... The model is trained iteratively, and rewards are given according to the quality of the training. The action decision-making collaborative optimization module is used to make collaborative optimization decisions on actions at each time step based on the similarity of running states using a deep reinforcement learning model. This includes: Let the current state of the set environment E be... ,judge The optimal set of state-action pairs obtained through reward mechanisms and action constraints All states Similarity measurement ; If it exists This indicates that a certain state in the optimal solution has been obtained. Compared with the current state Similarity leads to the next action. This allows the next action to be selected using mathematical methods to obtain the optimal action. If all That is, there is no state that is the same as the current state. If a similar state has already yielded the optimal solution, then proceed to the next action. This allows the next action to be selected using the optimal action obtained through reinforcement learning. Then the action is passed to the environment E to update the next state. And so on.

8. The collaborative optimization decision-making system for the energy internet according to claim 7, characterized in that, It also includes an environment setting module, which sets the state space and action space information of the regional energy internet. The state space is the environmental information perceived by the intelligent agent, and the action space is the relevant decision variables. The specific setting method is as follows: When setting up the state space, The state of the agent at any given time is represented by the following expression: In the formula, for Injecting nodes at all times The traditional output active power, for Injecting nodes at all times Traditional output reactive power, for Time Node The square of the gas pressure. for Time Node Injected heat power, for Time Node Injected photovoltaic power output, for Time Node The injected wind power output; The action space of an intelligent agent is the output of controllable devices within its control area, expressed as the following expression: In the formula, For the active power output of the gas turbine, For reactive power output of the gas turbine; Power output for electrical energy storage It provides reactive power for electrical energy storage.

9. The collaborative optimization decision-making system for the energy internet according to claim 8, characterized in that, It also includes a similarity measurement setting module, which defines the energy internet environment as follows: In the formula, For the network environment at any time t node i To the node j active power, For the network environment at all times t node i To the node j The reactive power, of which the node i and j Number any node in the network and i > j ;time With time status With state The similarity measure between them is defined as the mean squared deviation of the state components, and the calculation expression is as follows: when At that time, state With state They are similar; when At that time, state With state Not similar.

10. The collaborative optimization decision-making system for the energy internet according to claim 7, characterized in that, The action strategy solving module sets up a reward mechanism to find the optimal strategy. The steps, based on maximizing the expected cumulative return, are expressed as follows: In the formula, For strategy The trajectory formed; T This represents the total number of scheduling periods per day. For instant rewards.

11. The collaborative optimization decision-making system for the energy internet according to claim 10, characterized in that, The instant reward set by the action strategy solving module Each item includes operating cost items and node voltage constraint items; The operating cost item aims to minimize the daily operating cost, and the objective function is: In the formula, Cost coefficient; Define the cost per kilowatt-hour as The operating cost of energy storage is: The cost of purchasing electricity from the upper-level power grid is: In the formula, and They represent t Electricity purchase and sales prices from the upstream power grid during the specified time period; This indicates the electrical power purchased from the upstream power grid. Indicates the purchase of electricity. Indicates electricity sales; Set instant rewards The first term is as follows: Scheduling period t The lower the system operating cost, the greater the reward given to each agent during that period. The node voltage constraint term indicates that the voltage amplitude of each node must be limited to a safe range, and its expression is: In the formula, for t Time Node j Per-unit voltage value; This is the system's rated voltage; After the computational agent makes a scheduling decision, the system power flow distribution is calculated, and the agent receives the following reward based on the voltage of each node: In the formula, k This is the penalty coefficient; The environment provides the same global reward to the regional agent at each time interval, as shown in the following formula: 。 12. The collaborative optimization decision-making system for the energy internet according to claim 7, characterized in that, The action strategy solving module solves the pre-built regional energy internet operation optimization model and obtains the optimal solution as the action strategy. The steps and the expression of the regional energy internet operation optimization model are as follows: In the formula: For state variables, To control variables, These are the upper and lower bounds of the control variable, respectively. Let be the objective function. For the current equation. These are constraints; The objective function is a weighted average of multiple objective functions, including: economic objectives, power curtailment objectives, environmental objectives, and energy conversion efficiency objectives. Constraints include: equipment operating parameter constraints, production equipment parameter constraints, energy storage equipment constraints, and energy balance constraints. The regional energy internet operation optimization model is a nonlinear, non-convex model, which is transformed into a convex optimization problem through convex relaxation. The optimal solution is obtained using the MOSEK optimization solver and denoted as . .

13. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the steps in the collaborative optimization decision-making method for the energy internet as described in any one of claims 1 to 6.

14. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps in the collaborative optimization decision-making method for the energy internet as described in any one of claims 1 to 6.

Images