A distributed power source collaborative optimization method, device, equipment and medium

Abstract

The application discloses a kind of distributed power collaborative optimization method device, equipment and medium.The distributed power collaborative optimization method provided by the application carries out partition to distribution network with distributed power access, respectively set with pre-trained intelligent agent neural network in each partition;Obtain the observation value of current power grid state;The observation value is used as the input of deep reinforcement learning model, and the deep reinforcement learning model generates the control strategy of the distributed power in the current belonging partition, so that the operation cost of the current belonging partition is minimized, and the operation of the distributed power meets the operation constraint condition, to minimize the daily cumulative operation cost.Each region can realize both partition autonomy and collaborative operation, on the one hand, the regional intelligent agent realizes in-situ processing of partition data through its neural network, avoiding the collection and processing of a large amount of data by the centralized control center, on the other hand, the global distributed collaborative optimization is realized through the federal average process of parameter server.

Description

Technical Field

[0001] This invention belongs to the field of distributed power source collaborative optimization, and specifically relates to a distributed power source collaborative optimization method, device, equipment and medium. Background Technology

[0002] It is predicted that by 2050, more than half of my country's energy production and consumption will rely on the power system. The uncertainty of power grid source and load is becoming increasingly prominent, and traditional distribution network operation methods have many shortcomings in improving energy utilization efficiency. Coordinated control of distribution networks under high distributed generation penetration can fully absorb renewable energy and mitigate the impact of intermittent distributed generation on the distribution network. However, solving its coordinated optimization problem is a complex task, and current research mainly focuses on model-based distributed optimization methods.

[0003] The concept of distributed control was initially proposed in the field of multi-agent systems. With the increasing integration of distributed energy sources, the power grid scale gradually expands, placing new demands on real-time analysis and scheduling decisions for system operation, rendering traditional centralized methods inapplicable. Distributed control, however, can achieve optimal scheduling of the entire system through weak communication between adjacent agents and has been widely used to solve the problem of coordinated optimization of distribution networks under high distributed power penetration. However, as the number of distributed power sources continues to increase, these methods need to solve large-scale optimization problems with numerous nonlinear constraints, resulting in high computational costs, impacting real-time decision-making, and failing to adapt to continuously updated system operating conditions.

[0004] With the gradual integration of distributed energy resources and the large increase in the number of autonomous control points, the new generation of power systems has evolved into a giant system with complex structure, numerous devices, and complex technology. It has typical nonlinear stochastic characteristics and multi-scale dynamic characteristics. Traditional mechanism model analysis and distributed control methods are no longer able to meet the requirements of optimal scheduling, analysis and evaluation, and operation control. Summary of the Invention

[0005] The purpose of this invention is to provide a distributed power source collaborative optimization method, apparatus, device, and medium to solve the shortcomings of existing technologies, such as the need to solve large-scale optimization problems with a large number of nonlinear constraints, high computational cost, and impact on real-time decision-making when using existing distributed control methods to solve the distributed power source collaborative optimization problem under high penetration rate.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a distributed power source cooperative optimization method, comprising the following steps:

[0008] The distribution network connected to distributed power sources is divided into zones, and each zone is equipped with a pre-trained intelligent agent neural network.

[0009] The intelligent agent neural network is used to perform online optimization scheduling decisions, generating control strategies for distributed power sources within the current partition, so as to minimize the operating cost of the current partition and ensure that the operation of distributed power sources meets the operating constraints.

[0010] As an optional solution of the present invention, in the step of setting a pre-trained agent neural network in each of the partitions, the training method of the agent neural network is: to perform offline training on the agent neural network using a distributed successive convex approximation strategy optimization algorithm.

[0011] As an optional embodiment of the present invention, the step of offline training of the agent neural network using a distributed successive convex approximation strategy optimization algorithm specifically includes:

[0012] Retrieve historical data stored in the experience replay pool;

[0013] Using the historical data stored in the experience replay pool, calculate the constant term and the first derivative term of the Taylor expansion of the objective function and the value constraint, respectively;

[0014] Based on the constant term and the first derivative term of the Taylor expansion of the objective function and the value constraint, the objective function and the value function are expanded by a second-order Taylor expansion to obtain a convex optimization problem.

[0015] The convex optimization problem is solved iteratively using the primal dual gradient method to obtain the final value of the iteration.

[0016] The final value of the iteration is softly updated to obtain the original variables at the next time step, which are used as the learning parameters of the deep reinforcement learning model.

[0017] The neural network of the intelligent agent is trained using the aforementioned learning parameters.

[0018] As an optional solution of the present invention, the convex optimization problem is determined to be a target update and a feasible update. When solving the optimization problem, the target update is solved first; when solving the target update is not feasible, the feasible update is solved.

[0019] As an optional embodiment of the present invention, the convex optimization problem is defined as follows:

[0020]

[0021] stJ nm (θ n )≤d nm

[0022] in, For a convex optimization problem, θ n J represents the learning parameters of the agent neural network, where n is the agent's index, n = 1...N. n0 J is the value function of the objective function. nm Let d be the value function under the constraints. nm This represents the upper limit of the reward value.

[0023] As an optional embodiment of the present invention, the convex optimization problem is defined as follows:

[0024]

[0025] stJ nm (θ n )≤x n +d nm

[0026] in, For a convex optimization problem, θ n J represents the learning parameters of the agent neural network, where n is the agent's index, n = 1...N. nm Let d be the value function under the constraints. nm This represents the upper limit of the reward value.

[0027] As an optional solution of the present invention, the value constraint is obtained in the following way:

[0028] The value constraints are divided into global constraint values ​​and local constraint values;

[0029] Get the global constraint values ​​for all partitions;

[0030] The global constraint values ​​of all partitions are federated and averaged, and then fed back to each agent as the actual values, while the local constraint values ​​are not changed.

[0031] The actual value of the global constraint value is combined with the local constraint value to obtain the value constraint.

[0032] In a second aspect, the present invention provides a distributed power supply cooperative optimization device, comprising:

[0033] The partitioning module is used to partition the distribution network connected to distributed power sources into partitions, and each partition is equipped with a pre-trained intelligent agent neural network.

[0034] The decision-making module is used to perform online optimization scheduling decisions using the intelligent agent neural network, generate control strategies for distributed power sources within the current partition, minimize the operating cost of the current partition, and ensure that the operation of the distributed power sources meets the operating constraints.

[0035] As an optional solution of the present invention, in the partitioning module, the training method of the agent neural network is: to perform offline training of the agent neural network using a distributed successive convex approximation strategy optimization algorithm.

[0036] Specifically, the method for offline training of the agent neural network using a distributed successive convex approximation strategy optimization algorithm is as follows:

[0037] Retrieve historical data stored in the experience replay pool;

[0038] Using the historical data stored in the experience replay pool, calculate the constant term and the first derivative term of the Taylor expansion of the objective function and the value constraint, respectively;

[0039] Based on the constant term and the first derivative term of the Taylor expansion of the objective function and the value constraint, the objective function and the value function are expanded by a second-order Taylor expansion to obtain a convex optimization problem.

[0040] The convex optimization problem is solved iteratively using the primal dual gradient method to obtain the final value of the iteration.

[0041] The final value of the iteration is softly updated to obtain the original variables at the next time step, which are used as the learning parameters of the deep reinforcement learning model.

[0042] The neural network of the intelligent agent is trained using the aforementioned learning parameters.

[0043] In a third aspect, the present invention provides an electronic device including a processor and a memory, the processor being configured to execute a computer program stored in the memory to implement the above-described distributed power supply cooperative optimization method.

[0044] In a fourth aspect, the present invention provides a computer-readable medium storing at least one instruction that, when executed by a processor, implements the above-described distributed power supply cooperative optimization method.

[0045] The beneficial effects of this invention are as follows:

[0046] 1) The distributed power source collaborative optimization method provided by this invention divides the distribution network connected to distributed power sources into partitions, with each partition containing a pre-trained intelligent agent neural network. It acquires observations of the current grid state and uses these observations as input to a deep reinforcement learning model. The deep reinforcement learning model generates control strategies for the distributed power sources within the current partition, minimizing the operating cost of the current partition and ensuring that the operation of the distributed power sources meets operational constraints, thereby minimizing the daily cumulative operating cost. This allows each region to achieve both autonomy and collaborative operation. On one hand, each region's intelligent agent processes its data locally through its neural network, avoiding the need for a centralized control center to collect and process large amounts of data. On the other hand, global distributed collaborative optimization is achieved through a federated averaging process using parameter servers.

[0047] 2) The distributed power source collaborative optimization method provided by this invention comprehensively considers the optimization objective and various operational constraints, designs the state space, action space, observation space, reward function and constraint value, and uses a distributed successive convex approximation strategy optimization algorithm for model training. The trained model can provide an optimization scheduling strategy in real time according to the random changes in the source and load states, which greatly improves the online decision-making speed. Attached Figure Description

[0048] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0049] Figure 1 This is a schematic diagram illustrating the principle of the distributed power supply collaborative optimization method according to an embodiment of the present invention.

[0050] Figure 2 This is a schematic diagram of the structure of the federated policy learning framework in an embodiment of the present invention.

[0051] Figure 3 This is a schematic diagram of the training algorithm for the deep reinforcement learning model in an embodiment of the present invention.

[0052] Figure 4 This is a structural block diagram of a distributed power supply collaborative optimization device according to an embodiment of the present invention.

[0053] Figure 5 This is a structural block diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0054] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0055] The following detailed description is exemplary and intended to provide further detailed explanation of the invention. Unless otherwise specified, all technical terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this invention is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention.

[0056] Example 1

[0057] Embodiment 1 of this invention provides a distributed power source collaborative optimization method. This method is mainly aimed at distribution networks with high penetration of distributed power sources. It combines federated learning and deep reinforcement learning to establish a multi-agent deep reinforcement learning model that can realize the optimized scheduling of distributed power sources.

[0058] like Figure 1 As shown, a distributed power source collaborative optimization method has the following specific steps:

[0059] S01, Constructing a deep reinforcement learning model: Divide the distribution network under high penetration of distributed power sources into zones, set up intelligent agents in each zone, comprehensively consider the overall optimization goal of the distribution network and various operational constraints, and construct the state space, action space, observation space, reward function and constraint value of the deep reinforcement learning model based on the constrained Markov decision process.

[0060] S02. Construct a distributed optimization scheduling framework based on federated learning, and perform federated averaging of the learning parameters of the neural networks of each agent to achieve distributed collaborative optimization of each region of the distribution network.

[0061] S03, a distributed successive convex approximation strategy optimization algorithm is used to train the neural networks of each agent offline;

[0062] S04. The trained agent neural network model is used to make online optimization scheduling decisions. Each agent calculates the optimal output of controllable equipment in its own partition based on the status of the equipment in its partition, generates a control strategy, and issues scheduling instructions.

[0063] The following is a detailed explanation of steps S01 to S04 above.

[0064] Step S01: Construct a deep reinforcement learning model. Divide the distribution network under high distributed power penetration into zones, and set up intelligent agents in each zone. Considering the overall optimization goal of the distribution network and various operational constraints, design the state space, action space, observation space, reward function, and constraint value based on constrained Markov decision process. The intelligent agent is responsible for collecting renewable energy generation forecasts, load forecasts, and equipment status information within its zone, generating control strategies, and issuing scheduling instructions to controllable equipment within its zone according to the generated control strategies.

[0065] Specifically, step S01 includes the following steps:

[0066] Step 1.1: Construct the state space.

[0067] In this embodiment, the state space constructed represents the environmental information perceived by the agent. The state vector of agent n at time t is defined as... in and These are the predicted total photovoltaic output and total load of region n at time t, respectively.

[0068] It should be noted that the state variables are unrelated to the agent's control behavior, and the agent's strategy does not need to be considered. However, there are uncertainties in photovoltaic output prediction and load prediction. To represent the uncertainty in the prediction process, prediction error values ​​are superimposed on historical photovoltaic and load data. In this embodiment, it is assumed that the prediction error follows a Gaussian distribution.

[0069]

[0070]

[0071] in, and It is the actual historical data value, e PV and e D It is the standard deviation of the Gaussian distribution estimation error.

[0072] Step 1.2: Construct the observation space.

[0073] In this embodiment, the observation vector of the agent at time t is defined as O t This includes the node voltage V of the power grid at time t. t and injection current I t O t =[V t ,I t It should be noted that, unlike the state space, the observation space is implicitly determined by the agent's control behavior and therefore cannot be analyzed independently of the agent's policy. Node sensors or the distribution network's state estimation module will provide the latest observations. The global node voltages and currents satisfy the following constraints:

[0074]

[0075]

[0076] Where i and j are node numbers. ij is the branch number The superscripts M and m represent the upper and lower limits of a physical quantity, respectively.

[0077] It is important to note that the power distribution system operates under normal conditions, which assumes that the switch states and network topology remain unchanged during operation.

[0078] Step 1.3: Construct the action space.

[0079] The control behavior of agent n at time t is defined as follows: The scheduling decision variables within this region are:

[0080]

[0081] Among them, P DG Q DG These represent the active and reactive power outputs of the thermal power unit, respectively. Ch ,P Dis Q represents the charging and discharging power of the energy storage system. ESS This represents the reactive power of the energy storage system.

[0082] Thermal power units satisfy output constraints (6)-(7) and ramping constraints (8):

[0083]

[0084]

[0085]

[0086] in, t is the upper limit of climbing power, i is the node number, n is the agent number, and t is the time.

[0087] The constraints of the energy storage system are shown in equations (9)-(13), including the definition of the state of charge of the energy storage system (9) and the upper and lower limit constraints (10), the charging and discharging power constraints (11)-(12) of the energy storage system, and the upper limit constraint of the reactive power of the energy storage system (13):

[0088]

[0089]

[0090]

[0091]

[0092]

[0093] Where SOC represents the state of charge of the energy storage system, P Ch ,P Dis Q represents the charging power and discharging power of the energy storage system, respectively. ESS η represents the reactive power of the energy storage system.Ch and η Dis These represent the charging and discharging efficiency, respectively. Let be the total capacity of the energy storage system, i be the node number, n be the agent number, and t be the time.

[0094] Step 1.4: Select a control strategy.

[0095] In this embodiment, the control strategy of agent n is modeled as a multivariate Gaussian distribution, and the control strategy π of agent n is... n Defined as: Control behavior a n On D n D-dimensional multivariate Gaussian distribution n This is the dimension controlling behavior 'a', and in this embodiment, we take D. n =5.

[0096] The policy function determines the probability that the agent will take the optimal control behavior after training, as shown in equation (14):

[0097]

[0098] in, It is the mean vector. Let be the covariance matrix of a multivariate Gaussian distribution.

[0099] It should be noted that the Gaussian policy function explicitly determines the expected value and uncertainty of the optimal control behavior for each agent. In this embodiment, the learning parameter vector θ of each agent... n Consists of two parameter subsets and The components correspond to the mean vector and covariance matrix of the agent's policy function, respectively. Therefore, each agent uses two deep neural networks (DNNs) as parameter learning functions to achieve an approximate representation of the policy function.

[0100] The DNN receives the agent's state S. n As input, the mean vector and covariance matrix of the agent's control behavior are fully quantified:

[0101]

[0102]

[0103] Step 1.5: Design the reward function.

[0104] The intelligent agent changes its control behavior a n,t To minimize the total operating cost, which includes the electricity sales revenue of each region at time t and the power generation cost of the thermal power units:

[0105]

[0106]

[0107] in, It's the electricity price from the power grid. It refers to the fuel price for thermal power units. This represents the active power output from region n to the grid at the point of common coupling (PCC). The fuel cost F of the thermal power unit at node i is also mentioned. i,n,t Defined as its output a quadratic polynomial function, These are polynomial coefficients.

[0108] The switching power constraints of the PCC are shown in equations (19)-(20):

[0109]

[0110]

[0111] Therefore, for agent n, the reward function R at time t is constructed using the operating cost. n,t And using the average value, we obtain the value function for the objective function:

[0112]

[0113]

[0114] Here, t′ represents each time point from t to t+L.

[0115] Step 1.6: Design constraint value.

[0116] In this embodiment, to reflect the degree to which the solution process satisfies the constraints, a constraint value is added to the reward function. Assume the collaborative optimization problem contains M constraints, with global constraints as shown in equations (3)-(4), and the remainder as local constraints, denoted as C. nm (π n )≤d nm , n∈{1,...,N}, m∈{1,...,M}, where C nm (π n ) represents agent n in its control policy π n The reward value for the m-th constraint, d nm It is the upper limit of the return value.

[0117] To reflect the degree to which FPL adheres to the constraints, the same processing method as the reward value function is adopted to obtain the value function for the constraints:

[0118]

[0119] Here, t′ represents each time point from t to t+L.

[0120] According to the definition of constraint value, for local constraints, each agent only needs local information to calculate the reward value. However, for global constraints, agents cannot obtain the voltage and current conditions within other agents, and therefore cannot obtain an accurate global constraint value in one go. Therefore, this embodiment introduces a federated learning process to process global constraints in two stages:

[0121] (i) Each agent only counts the constraint value of its own partition and does not consider other partitions, and calculates the constraint value J. nm Including global constraint value and local constraint value

[0122] (ii) Each agent will calculate the global constraint value. The data is uploaded to a parameter server, where it performs a unified federated average and sends the result back to each agent. Each agent then uses this average as the final global constraint value and substitutes it into the algorithm for iterative calculation. This leads to the modeling and algorithm flow of the federated policy learning method proposed in this embodiment.

[0123] S02. Construct a distributed optimization scheduling framework based on federated learning, and perform federated averaging of the learning parameters of the neural networks of each agent to achieve distributed collaborative optimization in each region.

[0124] The framework of the Federated Policy Learning (FPL) method proposed in this embodiment is as follows: Figure 2 As shown. Each agent is responsible for a region and uses a private control policy for optimized scheduling. For agent n, the control policy π is approximated using a DNN. n At this time, the control strategy π n It is represented as the DNN learning parameter θ n The relevant probability distribution function is then used to select the actual control behavior 'a' by sampling the random policy. n,t Note the control strategy π n It is the state variable S n,t The function. Meanwhile, to ensure the safety of the control strategy, the agent receives observed variables O from the environment, i.e., its local partition. n,t Calculate the value function J and the value function with respect to the learning parameter θ. n gradient factor It is integrated into the multi-agent training algorithm, and secure information exchange between agents is achieved through federated learning, satisfying all global and local constraints. Federated learning reflects the interaction between regions, demonstrating how regional decisions affect each other.

[0125] To meet the global constraints of the entire network (i.e., limitations on node voltage and branch current), all regions need to operate in a coordinated manner. Therefore, the method proposed in this embodiment seeks collaborative optimization control among multiple regions. Since the global constraints are affected by the response of all regions, this embodiment designs a collaborative policy learning to ensure the safe operation of the entire power grid. Federated learning is used to weighted average the constraint functions to coordinate the policy optimization of each region, while a constraint policy optimization method is introduced to ensure safety during the data-driven process.

[0126] S03 employs a distributed successive convex approximation strategy optimization algorithm to perform offline training on the neural networks of each agent.

[0127] After the distributed optimization scheduling framework is established, the specific optimization problem is determined as follows: at time t, maximize the reward value function, that is, minimize the negative value function, i.e., equation (22), to update the agent's control policy, while satisfying the constraint value upper limit:

[0128]

[0129] StJ nm (π n )≤d nm

[0130]

[0131] Where π = {π1,...,π} N} represents the set of control policies for all agents, and Δ(·,·) is the KL divergence function used to measure previous policies. and the updated strategy The distance between them, and constrained by the step size δ, ensures that continuous strategies are close to each other, consistent with the formula. correspond.

[0132] It should be noted that for the Constrained Markov Decision Process (CMDP) problem, there are two existing solution methods: Primitive Dual Optimization (PDO) and Constrained Policy Optimization (CPO). PDO relaxes constraints using Lagrange multipliers, forming an unconstrained saddle point optimization problem. Due to the non-convexity and high computational complexity of reinforcement learning problems, existing methods often approximate the solution to the primal problem at the expense of stability, which is clearly undesirable. CPO utilizes a trust region framework to update the policy, guaranteeing performance improvement and approximate constraint satisfaction in each iteration, but theoretically, none of the CPO-based methods can guarantee strictly feasible results.

[0133] Therefore, this embodiment proposes to use the successive convex approximation strategy optimization (SCAOPO) to solve the CMDP problem. This method inherits the advantages of CPO and has been proven to converge to the KKT point of the original CMDP problem.

[0134] At the start of each iteration at time t, the objective function and the constant term of the Taylor expansion of the value constraint are first approximated using the historical data stored in the experience replay pool. and the first derivative of the Taylor expansion of the objective function and value constraints and In this embodiment, and For example.

[0135] First calculate

[0136]

[0137]

[0138] in, k1∈(0.5,1). T and k are constants, and there are a total of 2T time points. k1 and k2 are the constants required for the algorithm to converge.

[0139] Then calculate

[0140]

[0141]

[0142] For the reward value function and the constraint value function in Perform a Taylor second-order expansion at this point:

[0143]

[0144]

[0145] We obtain the convex optimization problem (31):

[0146]

[0147] stJ nm (θ n )≤d nm (31)

[0148] If problem (31) is proven infeasible, then we turn to solving the convex optimization problem (32):

[0149]

[0150] stJnm (θ n )≤x n +d nm (32)

[0151] This minimizes the possibility of constraint violations. Problems (31) and (32) are referred to as the objective update and feasible update, respectively. Then, the KL divergence constraint is satisfied using a soft update to obtain the optimal solution at time t, which is used as the initial value at time t+1.

[0152]

[0153] in, k2∈(0.5,1] and k2>k1. The obtained optimal solution is used as the optimal learning parameters of the neural network. The obtained optimal learning parameters are used to learn the neural network to obtain the neural network model, which is used to generate a control strategy to minimize the total cost of the entire distribution network. The control strategy satisfies the operating constraints.

[0154] The optimization problem is now fully defined and can be solved in the policy update iteration of the policy learning framework. However, the dimension of the DNN parameter θ may be very high, leading to increased computational costs during training. At the same time, regional agents need to protect the privacy of their control policies during training, which means that agents cannot access each other's control policies, cost functions, and local constraints. To address these two issues, this embodiment improves the centralized solution process of SCAOPO using the distributed dual subgradient method and introduces federated learning to achieve information interaction between regions, satisfying global constraints (3)-(4) while protecting privacy.

[0155] like Figure 3 As shown, taking target update as an example, the proposed distributed dual subgradient method includes the following stages:

[0156] Phase I (Initialization): Input parameters, maximum iteration time t max Maximum number of iterations k max Taylor expansion of the first derivative term and Update step size α t , original variable θ t update step size β t Reward value function Taylor expansion quadratic coefficients Constraint value function Taylor expansion quadratic coefficients Convergence Criterion Initialize the primitive variable θ n For θ n，to When time t is less than the maximum iteration time t maxAt that time, each agent uses historical data stored in the experience replay pool to approximate calculations. and Lagrange multiplier λ n (0)=[λ n1 ,…,λ nM ] T Initialize to a zero vector.

[0157] Phase II (Update of Original Variables): Minimize the Lagrange function to obtain the original variables. Solution:

[0158]

[0159] Where, Θ i Let a be the feasible region of the one-dimensional learning parameters. n and Calculated by the following formula:

[0160]

[0161]

[0162] The superscript 'i' represents the element in the i-th row. Then, smoothing is performed:

[0163]

[0164] And solve for the constraint value:

[0165]

[0166] Phase III (Federated Average): Taking into account global constraint value It is the gradient of the dual objective function with respect to the dual variables. Inspired by the FedAvg algorithm, each agent will... The value is uploaded to the parameter server, which performs a federated average of the parameter value and sends it back to each agent as the actual value.

[0167]

[0168]

[0169] The local constraint values ​​remain unchanged:

[0170]

[0171] Stage IV (Dual Gradient Update): Solving the dual problem using the dual ascent method:

[0172]

[0173] in, dnm =[d n1 ,…,d nM ] T γ is a constant, and the operator [·] + Returns the non-negative part of the input value. k is the number of iterations, λ is the dual variable in the Lagrangian function, and the primal dual gradient method is used to solve the convex optimization problem.

[0174] Phase V (convergence criterion): using The convergence of the algorithm is checked by observing changes in the parameters; when the change in parameters falls below a threshold... Stop if the time is right, otherwise return to stage II.

[0175] Phase VI (Soft Update): Utilizing the Iterative Final Value Soft updates yield the original variables for the next time step:

[0176]

[0177] In this process, the original variables at the next time step serve as the final learning parameters for the neural network.

[0178] S04. The trained model is used to make online optimization scheduling decisions. During each scheduling period, each agent calculates the optimal output of controllable equipment in its partition based on the status of the equipment in its partition, generates a control strategy, and issues scheduling instructions.

[0179] Example 2

[0180] like Figure 4 As shown, based on the same inventive concept as Embodiment 1, Embodiment 2 of the present invention provides a distributed power supply cooperative optimization device, comprising:

[0181] The partitioning module is used to partition the distribution network connected to distributed power sources into partitions, and each partition is equipped with a pre-trained intelligent agent neural network.

[0182] The decision-making module is used to perform online optimization scheduling decisions using the intelligent agent neural network, generate control strategies for distributed power sources within the current partition, minimize the operating cost of the current partition, and ensure that the operation of the distributed power sources meets the operating constraints.

[0183] In the partitioning module, the training method for the agent neural network is as follows: the agent neural network is trained offline using a distributed successive convex approximation strategy optimization algorithm, specifically including:

[0184] Retrieve historical data stored in the experience replay pool;

[0185] Using the historical data stored in the experience replay pool, calculate the constant term and the first derivative term of the Taylor expansion of the objective function and the value constraint, respectively;

[0186] Based on the constant term and the first derivative term of the Taylor expansion of the objective function and the value constraint, the objective function and the value function are expanded by a second-order Taylor expansion to obtain a convex optimization problem.

[0187] The convex optimization problem is solved iteratively using the primal dual gradient method to obtain the final value of the iteration.

[0188] The final value of the iteration is softly updated to obtain the original variables at the next time step, which are used as the learning parameters of the deep reinforcement learning model.

[0189] The neural network of the intelligent agent is trained using the aforementioned learning parameters.

[0190] In the partitioning module, the value constraint is obtained in the following way:

[0191] The value constraints are divided into global constraint values ​​and local constraint values;

[0192] Get the global constraint values ​​for all partitions;

[0193] The global constraint values ​​of all partitions are federated and averaged, and then fed back to each agent as the actual values, while the local constraint values ​​are not changed.

[0194] The actual value of the global constraint value is combined with the local constraint value to obtain the value constraint.

[0195] Example 3

[0196] like Figure 5As shown, based on the same inventive concept as Embodiment 1, Embodiment 3 of the present invention provides an electronic device 100 for implementing the distributed power supply cooperative optimization method of Embodiment 1. The electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on the at least one processor 102, and at least one communication bus 104. The memory 101 can be used to store the computer program 103. The processor 102 implements the steps of the distributed power supply cooperative optimization method of Embodiment 1 by running or executing the computer program stored in the memory 101 and calling data stored in the memory 101. The memory 101 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device 100 (such as audio data), etc. Furthermore, memory 101 may include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. At least one processor 102 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 102 may be a microprocessor or any conventional processor, etc. Processor 102 is the control center of electronic device 100, connecting various parts of electronic device 100 via various interfaces and lines.

[0197] The memory 101 in the electronic device 100 stores multiple instructions to implement a distributed power supply cooperative optimization method, and the processor 102 can execute multiple instructions to achieve the following:

[0198] The distribution network connected to distributed power sources is divided into zones, and each zone is equipped with a pre-trained intelligent agent neural network. The intelligent agent neural network is used to perform online optimization scheduling decisions and generate control strategies for the distributed power sources in the current zone, so as to minimize the operating cost of the current zone and ensure that the operation of the distributed power sources meets the operating constraints.

[0199] Example 4

[0200] If the modules / units integrated in the electronic device 100 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, and read-only memory (ROM).

[0201] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0202] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0203] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0204] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0205] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A distributed power source collaborative optimization method, characterized in that, Comprising the following steps: The power distribution network with distributed power supply is partitioned, and a pre-trained agent neural network is arranged in each partition; a deep reinforcement learning model is constructed, the power distribution network under high penetration of distributed power supply is partitioned, an agent is arranged in each region, the overall optimization target of the power distribution network and various operation constraints are comprehensively considered, the state space, action space, observation space, reward function and constraint value are designed based on the constrained Markov decision process, the agent is responsible for collecting renewable energy generation prediction, load prediction and device state information in the partition, generating a control strategy, and issuing scheduling instructions to controllable devices in the partition according to the generated control strategy; The agent neural network is used for online optimization and scheduling decision to generate a control strategy of the distributed power supply in the current partition, so that the operation cost of the current partition is minimized and the operation of the distributed power supply meets the operation constraint condition; In the step of arranging a pre-trained agent neural network in each partition, the training method of the agent neural network is as follows: a distributed successive convex approximation strategy optimization algorithm is used to perform offline training on the agent neural network; The step of using the distributed successive convex approximation strategy optimization algorithm to perform offline training on the agent neural network specifically comprises: Obtaining historical data stored in an experience replay pool; Using the historical data stored in the experience replay pool, the constant term and the first derivative term of the Taylor expansion of the objective function and the value constraint are calculated respectively; Based on the constant term and the first derivative term of the Taylor expansion of the objective function and the value constraint, the objective function and the value constraint are second-order Taylor expanded to obtain a convex optimization problem; The original dual gradient method is used to iteratively solve the convex optimization problem to obtain an iterative final value; in the step of using the original dual gradient method to iteratively solve the convex optimization problem, a federal average step is included: each agent uploads the global constraint value to a parameter server; the parameter server performs federal average on the global constraint value; the parameter server returns the global constraint value after federal average to each agent as an actual value, and the local constraint value is not changed; The iterative final value is soft-updated to obtain the original variable at the next time as the learning parameter of the deep reinforcement learning model; The learning parameter is used to train the agent neural network; The convex optimization problem is determined as a target update and a feasible update, and when the target update is infeasible, the feasible update is solved; The constraint value is obtained as follows: the constraint value is divided into a global constraint value and a local constraint value; the global constraint values of all partitions are obtained; the global constraint values of all partitions are uploaded to a parameter server for federal average, and are returned to each agent as an actual value, and the local constraint value is not changed; the actual value of the global constraint value and the local constraint value are combined to obtain the constraint value.

2. The distributed power source collaborative optimization method according to claim 1, characterized in that, The convex optimization problem is defined as follows: wherein, is a convex optimization problem, is a learning parameter of the agent neural network, n is a serial number of the agent, n = 1 … N, is a value function of the objective function, is a value function of the constraint condition, is an upper limit of the return value.

3. The distributed power source collaborative optimization method of claim 1, wherein, The convex optimization problem is defined as follows: wherein, is a convex optimization problem, are learning parameters of the agent neural network, n is a serial number of the agent, n = 1 … N, is a value function of the constraint condition, is an upper limit of the return value.

4. A distributed power source collaborative optimization apparatus for implementing the distributed power source collaborative optimization method of claim 1, characterized by, Comprising: A partition module is configured to partition a power distribution network with distributed power sources, and each partition is provided with a pre-trained agent neural network; A decision module is configured to make online optimization scheduling decisions by using the agent neural network, and generate a control strategy for the distributed power sources in the current partition to minimize the operation cost of the current partition and make the operation of the distributed power sources meet the operation constraint conditions.

5. An electronic device, comprising: The computer readable medium stores at least one instruction, and the at least one instruction is executed by the processor to implement the distributed power source collaborative optimization method according to any one of claims 1 to 3.

6. A computer readable medium characterized by The computer readable medium stores at least one instruction, and the at least one instruction is executed by the processor to implement the distributed power source collaborative optimization method according to any one of claims 1 to 3.

Images