Reinforcement learning method and system for source network load storage collaborative multi-scene optimization

Abstract

The invention discloses a reinforcement learning method and system for source network load storage collaborative multi-scene optimization, and the method comprises the steps: constructing a power distribution network optimization model with the minimum cost, and converting a mixed integer nonlinear problem into a solvable mixed integer second-order cone optimization problem; converting a mixed integer second-order cone optimization problem into a reinforcement learning decision model, and performing multi-round assignment on boundary condition parameters by using different operation scene data to form a multi-scene training task set; designing a reinforcement learning decision model multi-scene training loss function, combining the reinforcement learning decision model to construct a strategy neural network, an evaluation network neural network and a scene representation embedded neural network, and completing reinforcement learning decision adaptive to multi-scene optimization; and training a power distribution network multi-scene optimization decision model based on the multi-scene training task set, the multi-scene training reinforcement learning loss function and the neural network structure, and deploying the power distribution network multi-scene optimization decision model to an actual system to complete reinforcement learning decision model application oriented to source network load storage collaborative multi-scene optimization.

Description

Technical Field

[0001] The present invention belongs to the field of electric power artificial intelligence technology, and specifically relates to a reinforcement learning method and system for multi-scenario optimization of source-grid-load-storage collaboration. Background Art

[0002] As the construction of new power systems advances, the spatiotemporal dynamic characteristics of power grid source-load scenarios pose severe challenges to the cross-scenario migration of reinforcement learning (RL) optimization models. Although deep reinforcement learning technologies (such as model-free algorithms and meta-reinforcement learning) have demonstrated advantages in static scenarios, they face complex challenges in dynamic environment adaptation, multi-objective coordination, and safety constraint assurance. The present invention conducts analysis from four dimensions: environmental dynamism, data offset, target conflict, and constraint rigidity. Combining the optimal power flow calculation of power systems (linear programming, mixed integer programming, etc.) with the characteristics of deep neural networks, it reveals the essence of the model migration dilemma and explores a solution path that integrates technologies from multiple fields.

[0003] 1. The Adaptation Contradiction Between Dynamic Environments and Strategy Generalization

[0004] Minute-by-minute fluctuations in grid source and load scenarios (e.g., wind power fluctuations of up to 15% / min) cause time-varying parameters in Markov decision processes (MDPs). Traditional model-free reinforcement learning algorithms (e.g., DDPG and PPO) rely on policy networks trained in static environments. This lack of online learning makes them ineffective when dealing with topology reconfigurations or sudden changes in renewable energy penetration. Meta-reinforcement learning (Meta-RL) extracts meta-features of environmental dynamics to provide adaptive adjustment capabilities for policy networks. Research has shown that combining the Meta-RL algorithm with a long short-term memory network (LSTM) can improve voltage control response speed by 40% in scenarios with sudden changes in photovoltaic output.

[0005] 2. The conflict between data distribution shift and feature reconstruction

[0006] Changes in source-load scenarios can cause covariate shifts in the input data, leading to ineffective representation of hidden features in deep neural networks (DNNs). For example, when the peak-to-valley difference in load exceeds 30% of the training set, the error in estimating the action-value function Q(s,a) in the traditional DQN algorithm increases to 58%. By introducing domain-adversarial training (DAT), a feature discriminator is constructed in the hidden layer of the policy network, effectively aligning the feature distributions of the source and target domains. Experiments show that this method improves the success rate of policy transfer to 82% in scenarios with shifted load distribution.

[0007] 3. The dilemma of collaboration between multi-objective optimization and dynamic weighting

[0008] In power system optimal power flow calculations, linear programming (LP) and mixed integer second-order cone programming (MISOCP) achieve multi-objective optimization through rigorous mathematical modeling, but struggle to cope with real-time weight adjustments. Deep reinforcement learning, through end-to-end training, flexibly balances economic and safety objectives, but its fixed reward function design leads to rigid objective weights. The attention-based Dynamic Reward Allocation Network (DRAN) can analyze operating status online and control the weight allocation error between network loss minimization and voltage compliance within ±5%.

[0009] 4. Rigid barriers to security constraints and strategy exploration

[0010] While convex relaxation methods such as second-order cone programming (SOCP) can guarantee the feasibility of optimal power flow solutions, they cannot be directly embedded in the policy exploration process of reinforcement learning. The constrained policy optimization (CPO) algorithm encodes rigid conditions such as node voltage constraints and line capacity limits into the output layer of the policy network through a Lagrangian dual space mapping. In a distribution network with 50% distributed generation, the CPO algorithm reduced the constraint violation rate from 23% to 4.7% compared to traditional PPO, while maintaining an 89% economic efficiency index.

[0011] The contradiction between the dynamic evolution of source-load scenarios and the static knowledge representation of reinforcement learning models is essentially a paradigm conflict between the physical characteristics of power systems and data-driven methods. To overcome this dilemma, a "three-level fusion" framework must be constructed:

[0012] The algorithm layer integrates the scene perception capability of meta-reinforcement learning with the strict constraint modeling of MISOCP to achieve online strategy adaptation;

[0013] The model layer uses a graph convolutional network (GCN) to encode the topological features of the power grid and combines it with domain adversarial training to resist data offset;

[0014] The system layer uses the digital twin platform to implement strategy rehearsal and safety verification, and establishes a closed-loop mechanism of "simulation training-online migration-feedback correction".

[0015] In the future, it is necessary to focus on developing a reinforcement learning architecture for power information-physics integration, combining the mathematical rigor of optimal power flow calculations with the representation learning advantages of deep neural networks, and promoting the evolution of the new power system intelligent decision-making paradigm towards a dynamic, portable, secure and verifiable direction. Summary of the Invention

[0016] The technical problem to be solved by the present invention is to address the deficiencies in the above-mentioned existing technologies and provide a reinforcement learning method and system for multi-scenario optimization of source-grid-load-storage collaboration, which is used to solve the technical problem that the reinforcement learning optimization model is difficult to migrate and apply due to changes in the source-load scenarios of the power grid.

[0017] The present invention adopts the following technical solutions:

[0018] A reinforcement learning method for source-grid-load-storage collaborative multi-scenario optimization, characterized by comprising the following steps:

[0019] Construct a cost-minimizing distribution network optimization model, define the objective function and constraints, and transform the mixed-integer nonlinear problem into a solvable mixed-integer second-order cone optimization problem through convex relaxation.

[0020] The obtained mixed integer second-order cone optimization problem is transformed into a reinforcement learning decision model, and the action space, state space and reward function of the reinforcement learning decision model are defined;

[0021] Using different operating scenario data, the boundary condition parameters in the constructed distribution network optimization model are assigned multiple times, and a multi-scenario training task set is formed based on the defined reinforcement learning decision model.

[0022] Design multi-scenario training loss functions for the defined reinforcement learning decision model, including the policy network loss function, the evaluation network loss function, and the scenario representation loss function; construct the policy neural network, the evaluation network neural network, and the scenario representation embedding neural network based on the defined reinforcement learning decision model as parameter objects for updating the reinforcement learning strategy, and complete the reinforcement learning decision that adapts to multi-scenario optimization;

[0023] Based on the constructed multi-scenario training task set, the defined multi-scenario training reinforcement learning loss function and neural network structure, the distribution network multi-scenario optimization decision model is trained and deployed to the actual system to complete the application of the reinforcement learning decision model for multi-scenario optimization of source-grid-load-storage collaboration.

[0024] Preferably, for the distribution network optimization model, the energy balance constraint is processed by using the second-order cone relaxation constraint, which is converted into a mixed integer second-order cone optimization problem after processing. The second-order cone relaxation constraint is specifically as follows:

[0025]

[0026] Among them, l ij is the square of the current modulus corresponding to the ij branch, v i is the square of the voltage modulus of node i, P ij ,Q ij is the active and reactive power flowing through branch ij.

[0027] Preferably, the objective function of the distribution network optimization model is as follows:

[0028]

[0029] Where C is the cost function of distributed units, energy storage, demand response and power purchase from the grid, is the operating cost of the i-th unit, is the operating cost of the j-th energy storage device, is the operating cost of the kth demand response unit, is the cost of curtailing electricity from the nth renewable energy unit, C Ext is the cost of electricity purchase and sale by the upper power grid, P is the power of the corresponding equipment, and P i DG (t) is the operating power of the i-th unit at time t, is the operating power of the jth energy storage device at time t, is the operating power of the kth demand response unit at time t, is the power abandonment of the nth new energy unit at time t, P Ext (t) is the interaction power of the upper power grid at time t, represents the set of system node numbers where the corresponding equipment is located, α, β, c correspond to the second-order, first-order and constant coefficients respectively, ζ, ρ, λ correspond to their cost coefficients respectively, T represents the total optimization time, t is the current time, Δt is the time interval, i.e., the optimization step size, C GL is the network loss parameter, is the expectation operator.

[0030] Preferably, the distribution network optimization model is time-series decoupled, and the reinforcement learning action space is defined as the charging and discharging power of the energy storage unit and the power of the unit with a small climbing capability. The time-series coupling constraints associated with the reinforcement learning action space are transformed, where the power constraints of the unit operation are modeled as follows:

[0031]

[0032] Among them, P DG (t) is the unit power at time t, η DG is the climbing coefficient, is the set of units whose power values are determined based on the optimization solution, P DG , Respectively represent the upper and lower bounds of active power, η DG Indicates the ratio of the maximum active power change to the upper bound;

[0033] The operating constraints of energy storage operating power are modeled as follows:

[0034]

[0035] SoC min ≤SoC(t)≤SoC max

[0036] Among them, the energy storage charging and discharging power P S,ch ,P S,dis Together determine the external apparent power of energy storage, ηS,ch ,η S,dis is the charge and discharge efficiency coefficient, Maximum charging capacity for energy storage, SoC min ,SoC max are the minimum and maximum values of the energy storage state of charge, respectively;

[0037] The state space is defined as follows:

[0038] s env ={P RE (t),P Load (t),λ(t),P DG (t-1),Q DG (t-1),P DR (t-1),SoC(t-1)}

[0039] Among them, P RE Represents the new energy power of each node in the system, P Load represents the load on each node of the system, λ represents the electricity purchase price at the current moment, P DG Represents the sum of the active power of the units at each node, Q DG Represents the sum of reactive power at each node in the system, P DR It represents the sum of the demand response power at each node in the system, and SoC represents the state of charge of the energy storage at each node in the system;

[0040] The immediate reward part of the reward function is expressed as follows:

[0041]

[0042] Among them, r t Represents the immediate reward value, Represents the set of node numbers of the distributed unit system, Represents the set of energy storage system node numbers, Represents a set of node numbers that meet the demand response system requirements. Represents the set of node numbers of the power grid purchasing system; i represents the set Elements in, j represents the set Elements in, k represents the set The elements in n represent the set The elements in m represent the set Elements determined by reinforcement learning in C DG Represents the unit operation cost function, C S represents the energy storage depreciation cost function, C DR Denotes the load demand response cost function, C Ext represents the interaction cost function of the upper power grid, Represents the cost function of renewable energy curtailment, C GL Indicates the network loss converted to cost; P represents the power of the corresponding equipment, P * represents the variable value determined by reinforcement learning; SoC represents the state of charge percentage, and t represents the current time.

[0043] The baseline part of the reward function is expressed as follows:

[0044]

[0045] Among them, r t env represents the baseline reward value, Represents the energy storage and unit output values in the baseline strategy. The baseline reward setting benchmark is the overall operating cost corresponding to setting the energy storage and unit output values determined by reinforcement learning to 0.

[0046] Preferably, the multi-scenario training task set is specifically:

[0047] Based on daily operation scenarios, the continuous operation data of wind power, photovoltaic power, load and upper-level grid prices at each node are decomposed to form the time series characteristics of operation factors;

[0048] The decomposed time series features of the operating elements are classified based on the DBSCAN clustering method. Each class contains multiple approximate time series features and is regarded as a task with identical distribution characteristics.

[0049] The temporal characteristics of the operating elements of each distribution network are matched with their physical nodes to form multiple feature combinations, and each feature combination is defined as a task; each round of reinforcement learning training is performed under the same task. During the reinforcement learning iterative solution process, the operating state corresponding to each moment is determined by the temporal characteristics of the environmental elements at the current moment.

[0050] Preferably, the reinforcement learning decision-making process adapted to multi-scenario optimization is specifically as follows:

[0051] Based on the core elements of reinforcement learning and the training task set, a task distribution P(τ) is constructed. Each task is a Markov decision process, including a set of states, actions and a finite reward function, represented as task τ = {p(s0), p(s t+1 |s t ,a t ),r(s t ,a t )}, p(s0) is the initial state distribution, p(s t+1 |s t ,a t ) is the transfer distribution and r(s t ,a t) is the reward function;

[0052] Given a set of training tasks sampled from P(τ), the meta-training process learns a policy that adapts to the current task through the context c;

[0053] set up is a state transition in task τ, which corresponds to a time sequence state change in power grid optimization. The mission experience that constitutes the present collection;

[0054] Define the task context feature vector as Z, which is used to characterize the characteristics of the current running scene, and is represented by π in the strategy function. θ Feature embedding is achieved in the form of (a|s,z);

[0055] By constructing an inference network q φ (z|c) constructs a reinforcement learning value function that minimizes the estimated deviation, φ is the neural network parameter, and at the same time trains q for different scene features in a way that minimizes the amount of feature information. φ (z|c) parameters;

[0056] The distribution network optimization model is trained using a deep reinforcement learning algorithm, and the reparameterization method is used to calculate the sampling feature vector z versus q φ The gradient of the parameters in (z|c) is used to optimize the inference network q φ Parameters of (z|c), policy network π θ (a|s,z) and evaluation network Parameters.

[0057] Preferably, the multi-scenario loss function is as follows:

[0058]

[0059] in, represents the expectation of task τ, represents the expectation between the sample sets, D KL Represents the KL divergence operator, q φ represents the scene element representation embedded in the neural network, τ represents the number of a task, c τ is the context information of the corresponding task, z is the task context feature vector; g(τ,z) represents the optimization main function served by feature z, which will be designed in detail in the future in combination with the reinforcement learning algorithm; β is the target weight coefficient; p(z) is the unit Gaussian prior distribution.

[0060] Preferably, the updated loss function of the evaluation network with scene element representation embedding is as follows:

[0061]

[0062] Among them, L critic To evaluate the network loss function symbol, represents the evaluation network function, V represents the baseline evaluation network function, s′ represents the next state of migration, Indicates the z vector without gradient update;

[0063] The loss function of the policy network with scene meta-representation embedding is as follows:

[0064]

[0065] Among them, L actor is the symbol of the policy network loss function, D KL (·||·) represents the KL divergence function between two distributions, exp represents the exponential function technique, K θ Represents the state normalization factor.

[0066] Preferably, the executable neural network model adopts an Actor-Critic architecture, including a policy neural network, an evaluation neural network, and a scene meta-representation neural network;

[0067] (s n ,a n ,r n ,s′ n ) Inputting the context information fragments composed of four tuples into the scene element representation neural network to obtain the corresponding context fragment feature distribution, and obtaining n distributions for n four tuples; multiplying the n fragment feature distributions by Gaussian distribution products to obtain the scene element representation distribution, and using distribution sampling to obtain the scene element representation one-dimensional tensor;

[0068] The one-dimensional tensor representing the scene element is concatenated with the action space and state space defined by the content as the input of the evaluation network, and the final output is the value function estimate of the corresponding state-action combination;

[0069] The input of the policy network is the concatenation of the state space and the one-dimensional tensor representing the scene element, and the output is the value distribution of each variable in the action space, which is used to calculate the optimization strategy and update the policy neural network.

[0070] Preferably, the evaluation network, policy network and scene element representation network adopt a fully connected network or Transformer structure.

[0071] In a second aspect, an embodiment of the present invention provides a reinforcement learning system for optimizing source-grid-load-storage collaboration in multiple scenarios, including:

[0072] Problem module, which builds a cost-minimizing distribution network optimization model, defines the objective function and constraints, and transforms the mixed-integer nonlinear problem into a solvable mixed-integer second-order cone optimization problem through convex relaxation;

[0073] The conversion module converts the obtained mixed integer second-order cone optimization problem into a reinforcement learning decision model and defines the action space, state space and reward function of the reinforcement learning decision model;

[0074] The generation module uses different operating scenario data to perform multiple rounds of assignments on the boundary condition parameters in the constructed distribution network optimization model, and forms a multi-scenario training task set based on the defined reinforcement learning decision model;

[0075] The training module designs multi-scenario training loss functions for the defined reinforcement learning decision model, including the policy network loss function, the evaluation network loss function, and the scenario representation loss function. It also constructs the policy neural network, the evaluation network neural network, and the scenario representation embedding neural network based on the defined reinforcement learning decision model as parameter objects for updating the reinforcement learning strategy, completing reinforcement learning decision-making that adapts to multi-scenario optimization.

[0076] The output module trains the distribution network multi-scenario optimization decision model based on the constructed multi-scenario training task set, the defined multi-scenario training reinforcement learning loss function and the neural network structure, and deploys it to the actual system to complete the application of the reinforcement learning decision model for multi-scenario optimization of source-grid-load-storage collaboration.

[0077] Preferably, in the problem module, for the distribution network optimization model, the energy balance constraint is processed by using the second-order cone relaxation constraint, which is converted into a mixed integer second-order cone optimization problem after processing; the second-order cone relaxation constraint is specifically as follows:

[0078]

[0079] Among them, l ij is the square of the current modulus corresponding to the ij branch, v i is the square of the voltage modulus of node i, P ij ,Q ij is the active and reactive power flowing through branch ij;

[0080] The objective function of the distribution network optimization model is as follows:

[0081]

[0082]

[0083] Where C is the cost function of distributed units, energy storage, demand response and power purchase from the grid, is the operating cost of the i-th unit, is the operating cost of the j-th energy storage device, is the operating cost of the kth demand response unit, is the cost of curtailing electricity from the nth renewable energy unit, C Ext is the cost of electricity purchase and sale by the upper power grid, P is the power of the corresponding equipment, and P i DG (t) is the operating power of the i-th unit at time t, is the operating power of the jth energy storage device at time t, is the operating power of the kth demand response unit at time t, is the power abandonment of the nth new energy unit at time t, P Ext (t) is the interaction power of the upper power grid at time t, represents the set of system node numbers where the corresponding equipment is located, α, β, c correspond to the second-order, first-order and constant coefficients respectively, ζ, ρ, λ correspond to their cost coefficients respectively, T represents the total optimization time, t is the current time, Δt is the time interval, i.e., the optimization step size, C GL is the network loss parameter, is the expectation operator.

[0084] Preferably, in the conversion module, the distribution network optimization model is time-series decoupled, and the reinforcement learning action space is defined as the charging and discharging power of the energy storage unit and the power of the unit with a small climbing capability. The time-series coupling constraints associated with the reinforcement learning action space are converted, wherein the power constraint of the unit operation is modeled as follows:

[0085]

[0086] Among them, P DG (t) is the unit power at time t, η DG is the climbing coefficient, is the set of units whose power values are determined based on the optimization solution, P DG , Respectively represent the upper and lower bounds of active power, η DG Indicates the ratio of the maximum active power change to the upper bound;

[0087] The operating constraints of energy storage operating power are modeled as follows:

[0088]

[0089] SoC min ≤SoC(t)≤SoC max

[0090] Among them, the energy storage charging and discharging power P S,ch ,P S,dis Together determine the external apparent power of energy storage, η S,ch ,η S,dis is the charge and discharge efficiency coefficient, Maximum charging capacity for energy storage, SoC min,SoC max are the minimum and maximum values of the energy storage state of charge, respectively;

[0091] The state space is defined as follows:

[0092] s env ={P RE (t),P Load (t),λ(t),P DG (t-1),Q DG (t-1),P DR (t-1),SoC(t-1)}

[0093] Among them, P RE Represents the new energy power of each node in the system, P Load represents the load on each node of the system, λ represents the electricity purchase price at the current moment, P DG Represents the sum of the active power of the units at each node, Q DG Represents the sum of reactive power at each node in the system, P DR It represents the sum of the demand response power at each node in the system, and SoC represents the state of charge of the energy storage at each node in the system;

[0094] The immediate reward part of the reward function is expressed as follows:

[0095]

[0096] Among them, r t Represents the immediate reward value, Represents the set of node numbers of the distributed unit system, Represents the set of energy storage system node numbers, Represents a set of node numbers that meet the demand response system requirements. Represents the set of node numbers of the power grid purchasing system; i represents the set Elements in, j represents the set Elements in, k represents the set The elements in n represent the set The elements in m represent the set Elements determined by reinforcement learning in C DG Represents the unit operation cost function, C S represents the energy storage depreciation cost function, C DR Denotes the load demand response cost function, C Ext represents the interaction cost function of the upper power grid, Represents the cost function of renewable energy curtailment, C GL Indicates the network loss converted to cost; P represents the power of the corresponding equipment, P *represents the variable value determined by reinforcement learning; SoC represents the state of charge percentage, and t represents the current time.

[0097] The baseline part of the reward function is expressed as follows:

[0098]

[0099] Among them, r t env represents the baseline reward value, Represents the energy storage and unit output values in the baseline strategy. The baseline reward setting benchmark is the overall operating cost corresponding to setting the energy storage and unit output values determined by reinforcement learning to 0.

[0100] Preferably, in the generation module, the diverse training task set is specifically:

[0101] Based on daily operation scenarios, the continuous operation data of wind power, photovoltaic power, load and upper-level grid prices at each node are decomposed to form the time series characteristics of operation factors;

[0102] The decomposed time series features of the operating elements are classified based on the DBSCAN clustering method. Each class contains multiple approximate time series features and is regarded as a task with identical distribution characteristics.

[0103] The temporal characteristics of the operating elements of each distribution network are matched with their physical nodes to form multiple feature combinations, and each feature combination is defined as a task; each round of reinforcement learning training is performed under the same task. During the reinforcement learning iterative solution process, the operating state corresponding to each moment is determined by the temporal characteristics of the environmental elements at the current moment.

[0104] Preferably, in the training module, the reinforcement learning strategy model adapted to multiple scenarios is obtained as follows:

[0105] Based on the core elements of reinforcement learning and the training task set, a task distribution P(τ) is constructed. Each task is a Markov decision process, including a set of states, actions and a finite reward function, represented as task τ = {p(s0), p(s t+1 |s t ,a t ),r(s t ,a t )}, p(s0) is the initial state distribution, p(s t+1 |s t ,a t ) is the transfer distribution and r(s t ,a t ) is the reward function;

[0106] Given a set of training tasks sampled from P(τ), the meta-training process learns a policy that adapts to the current task through the context c;

[0107] set up is a state transition in task τ, which corresponds to a time sequence state change in power grid optimization. The mission experience that constitutes the present collection;

[0108] Define the task context feature vector as Z, which is used to characterize the characteristics of the current running scene, and is represented by π in the strategy function. θ Feature embedding is achieved in the form of (a|s,z);

[0109] By constructing an inference network q φ (z|c) constructs a reinforcement learning value function that minimizes the estimated deviation, φ is the neural network parameter, and at the same time trains q for different scene features in a way that minimizes the amount of feature information. φ (z|c) parameters;

[0110] The distribution network optimization model is trained using the deep reinforcement learning SAC algorithm, and the reparameterization method is used to calculate the sampling feature vector z versus q φ The gradient of the parameters in (z|c) is used to optimize the inference network q φ Parameters of (z|c), policy network π θ (a|s,z) and evaluation network Parameters;

[0111] The multi-scene loss function is as follows:

[0112]

[0113] in, represents the expectation of task τ, represents the expectation between the sample sets, D KL Represents the KL divergence operator, q φ represents the scene element representation embedded in the neural network, τ represents the number of a task, c τ is the context information of the corresponding task, z is the task context feature vector; g(τ,z) represents the optimization main function served by feature z, which will be designed in detail in the future in combination with the reinforcement learning algorithm; β is the target weight coefficient; p(z) is the unit Gaussian prior distribution;

[0114] The updated loss function of the evaluation network with scene meta-representation embedding is as follows:

[0115]

[0116] Among them, L critic To evaluate the network loss function symbol, represents the evaluation network function, V represents the baseline evaluation network function, s′ represents the next state of migration, Indicates the z vector without gradient update;

[0117] The loss function of the policy network with scene meta-representation embedding is as follows:

[0118]

[0119] Among them, L actor is the symbol of the policy network loss function, D KL (·||·) represents the KL divergence function between two distributions, exp represents the exponential function technique, K θ Represents the state normalization factor.

[0120] Preferably, in the output module, the executable neural network model adopts an Actor-Critic architecture, including a policy neural network, an evaluation neural network, and a scene meta-representation neural network;

[0121] (s n ,a n ,r n ,s′ n ) Inputting the context information fragments composed of four tuples into the scene element representation neural network to obtain the corresponding context fragment feature distribution, and obtaining n distributions for n four tuples; multiplying the n fragment feature distributions by Gaussian distribution products to obtain the scene element representation distribution, and using distribution sampling to obtain the scene element representation one-dimensional tensor;

[0122] The one-dimensional tensor representing the scene element is concatenated with the action space and state space defined by the content as the input of the evaluation network, and the final output is the value function estimate of the corresponding state-action combination;

[0123] The input of the policy network is the concatenation of the state space and the one-dimensional tensor representing the scene element, and the output is the value distribution of each variable in the action space, which is used to calculate the optimization strategy and update the policy neural network;

[0124] The evaluation network, policy network, and scene element representation network adopt a fully connected network or Transformer structure.

[0125] In the third aspect, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned reinforcement learning method for multi-scenario optimization of source-grid-load-storage collaboration are implemented.

[0126] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned reinforcement learning method for multi-scenario optimization of source-grid-load-storage collaboration.

[0127] In the fifth aspect, a chip includes a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, the steps of the above-mentioned reinforcement learning method for multi-scenario optimization of source-grid-load-storage collaboration are implemented.

[0128] In a sixth aspect, an embodiment of the present invention provides an electronic device, including a computer program, which, when executed by the electronic device, implements the steps of the above-mentioned reinforcement learning method for multi-scenario optimization of source-grid-load-storage collaboration.

[0129] Compared with the prior art, the present invention has at least the following beneficial effects:

[0130] A reinforcement learning method for multi-scenario optimization of source-grid-load-storage collaboration uses convex relaxation technology to transform complex mixed integer nonlinear problems (MINLPs) into efficiently solvable MISOCP problems, balancing solution efficiency and accuracy. By generating a diverse set of training tasks and meta-learning strategies, the model can adapt to the dynamic changes of different source-grid-load-storage scenarios. It reduces the computational complexity of distribution network optimization problems, improves real-time decision-making capabilities, and is suitable for multi-scenario requirements in actual systems.

[0131] Furthermore, a second-order cone relaxation constraint (SOCP) is introduced to handle energy balance constraints, avoiding the solution difficulties caused by non-convex nonlinearity. This ensures that the relaxed model is physically consistent with the original problem, improves the efficiency and stability of solving the optimization problem, and avoids the problem of local optimal solutions caused by non-convexity.

[0132] Furthermore, by clarifying the constraint formula, the model's interpretability is enhanced. The relaxed second-order cone constraints still reflect the physical operating laws of the power grid, providing a reliable optimization model foundation for subsequent reinforcement learning tasks.

[0133] Furthermore, the objective function encompasses multiple cost categories, including distributed generation, energy storage, demand response, and grid power purchase, achieving global economic optimization. Sub-item coefficients flexibly adapt to the cost characteristics of different equipment. This refined modeling of distribution network operating costs avoids suboptimal solutions resulting from single-cost optimization.

[0134] Furthermore, ramp coefficients and power upper and lower bounds are used to ensure unit operation within a safe range and prevent equipment overload. Multi-dimensional dynamic information, including renewable energy power, load, electricity price, and energy storage status, is included to comprehensively characterize the system state. The strategy is guided towards global optimization by integrating multiple objectives, including unit operation, energy storage depreciation, demand response, and grid interaction. Through the rational design of reinforcement learning elements, a balance between strategy feasibility and economic efficiency is achieved.

[0135] Furthermore, through time series feature decomposition and DBSCAN clustering, diverse training tasks that approximate the true distribution are generated. By combining distribution network nodes with operational factors, the correspondence between tasks and real-world scenarios is strengthened. This improves the model's generalization capabilities, avoids overfitting to a single scenario, and allows it to adapt to complex and changing operational environments.

[0136] Furthermore, through task distribution and contextual feature embedding, the policy can be adapted to multiple scenarios. This minimizes the bias in value function estimation, improving the policy's stability across different tasks. By minimizing the amount of information required to train contextual features, the model's generalization and computational efficiency are enhanced. This addresses the "catastrophic forgetting" problem in multi-scenario reinforcement learning and improves the policy's rapid adaptability.

[0137] Furthermore, mathematical formulas are used to explicitly constrain the consistency between contextual features and task objectives. A Gaussian prior distribution is used to normalize the distribution range of feature vectors to avoid overfitting. This improves the representational power of contextual features and ensures the robustness of the strategy across different tasks.

[0138] Furthermore, through the target network (baseline) and temporal difference error constraints, the value function estimation is stabilized. This balances exploration and exploitation, avoiding drastic fluctuations during policy updates. This accelerates reinforcement learning convergence and improves the stability of the training process.

[0139] Furthermore, the strategy and value function estimation are separated, balancing strategy optimization and state value assessment. Using a feature extraction network, scenario information is embedded into the strategy and evaluation networks, enhancing the model's multi-scenario perception. This improves the neural network's ability to model complex inputs and supports real-time decision-making.

[0140] Furthermore, it supports fully connected or Transformer architectures to adapt to the input characteristics of distribution networks of varying scales. The Transformer structure captures long-term temporal dependencies, while the fully connected network is suitable for fast processing of low-dimensional features. This modular design adapts to the needs of different scenarios and improves the scalability of the model.

[0141] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0142] In summary, the present invention combines convex relaxation with reinforcement learning to efficiently solve complex optimization problems; uses meta-learning and multi-scenario training to achieve dynamic strategy adaptation; and integrates the cost minimization model and the actor-critic architecture to ensure economy and real-time performance, making it suitable for the optimization of distribution networks with high penetration of new energy.

[0143] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0144] Figure 1 It is a schematic diagram of the process of the present invention;

[0145] Figure 2 This is a schematic diagram of the main structure of the evaluation neural network of the present invention;

[0146] Figure 3 The convergence effect of the tasks corresponding to the training of the source-grid-load-storage multi-scenario optimization model using this patented method is demonstrated;

[0147] Figure 4 To improve the effect of applying this patented method to train the reward function corresponding to the source-grid-load-storage multi-scenario optimization model;

[0148] Figure 5 A schematic diagram of a computer device provided in accordance with an embodiment of the present invention;

[0149] Figure 6 The present invention is a block diagram of an electronic device according to an embodiment of the present invention.

[0150] Among them, 60. Computer device; 61. Processor; 62. Memory; 63. Computer program; 600. Electronic device; 610. Processing unit; 620. Storage unit; 6201. Random access memory unit; 6202. Cache memory unit; 6203. Read-only memory unit; 6204. Program / Utility; 6205. Program module; 630. Bus; 640. Display unit; 650. Input / output interface; 660. Network adapter; 700. External device. DETAILED DESCRIPTION

[0151] The present invention provides a reinforcement learning method for multi-scenario optimization of source-grid-load-storage collaboration, which converts unstable problems into a set of multiple stable tasks, expresses their common implicit features based on contextual features, and introduces task feature tensors in the reinforcement learning state space to achieve dynamic adaptation of source-load boundaries.

[0152] Example 1

[0153] See also Figure 1 The present invention provides a reinforcement learning method for optimizing the source-grid-load-storage collaboration in multiple scenarios, comprising the following steps:

[0154] S1. Distribution network optimization modeling and model conversion;

[0155] To apply reinforcement learning algorithms to train intelligent agents to make decisions about the coordinated implementation of power generation, grid, load, and storage, an interactive simulation environment is required. Inputting current state information and action strategies into the simulation environment will then provide feedback with a corresponding reward value and the next state information. Typically, this simulation environment is centered around power flow calculation methods. Once all grid node information is provided by the intelligent agent or determined by the external environment, the power flow calculations provide reward feedback, including indicators such as grid operational stability, economic efficiency, and reliability. However, as the scale of the grid expands and the number of controllable elements increases, learning and exploration methods become difficult to quickly find feasible strategies that meet complex and diverse safety constraints. Therefore, a reinforcement learning decision-making modeling method for power generation, grid, load, and storage coordination based on optimal power flow calculations is proposed.

[0156] Distribution network optimization objectives include:

[0157] 1) The sum of the equivalent economic costs of all optimizable elements of the system during the entire optimization period;

[0158] 2) The equivalent cost of the system power network loss is expressed as:

[0159]

[0160] Among them, C represents the distributed units, energy storage, demand response and grid power purchase cost function, and P represents the power of the corresponding equipment. Represents the set of system node numbers where the corresponding equipment is located, and i, j, k, and n represent the elements in the set. The DG operation cost adopts a quadratic model, and α, β, and c correspond to the second-order, first-order, and constant coefficients respectively. S The calculation is based on the change of energy storage charge state, SoC represents the percentage of charge state, and σ represents the loss coefficient. The demand response cost, power curtailment cost and power purchase cost of the upper power grid are all represented by linear models, and ζ, ρ, and λ correspond to their cost coefficients respectively. The optimization objective function is to minimize the cumulative operating cost, T represents the total optimization time, t represents the current time, and Δt represents the time interval, i.e., the optimization step size. For the network loss parameter C GL , η GL is the target coefficient of network loss. ij is the resistance of branch ij, l ij is the square of the current amplitude flowing through branch ij.

[0161] Variable equality constraints include KCL constraints, KVL constraints, and energy balance constraints; variable inequality constraints include controllable device output limit constraints and controllable device power timing ramping constraints; integer variable constraints are constraints on discontinuous variables with special physical meanings, including energy storage device charging and discharging constraints, grid power purchase and sales constraints, and load switching behavior constraints. For distributed units, in addition to rated power constraints and active power limit constraints, ramping constraints for unit regulation are also considered, as shown in Equations (8) and (9).

[0162]

[0163] Among them, P DG , Respectively represent the upper and lower bounds of active power, η DG Indicates the ratio of the maximum active power change to the upper bound. Indicates the rated power of the unit.

[0164] New energy equipment is modeled using a constant power factor model, where its active power is equal to the maximum theoretical output minus the amount of wasted power, and its reactive power is equal to the theoretical output multiplied by the power factor.

[0165]

[0166] in, is the theoretical upper bound of the output of new energy equipment at time t, η RE is the constant power factor coefficient.

[0167] The interconnection line of the upper-level power grid is mainly responsible for purchasing electricity from the outside. Since the output of renewable energy can be adjusted, the model is limited and cannot supply electricity to the upper-level power grid.

[0168]

[0169] in, It represents the upper limit of the amount of electricity purchased. Q Ext Represents the upper and lower bounds of reactive power interaction with the upper power grid.

[0170] For energy storage device constraints, it is necessary to construct auxiliary integer variables to ensure that the energy storage is not charged and discharged at the same time.

[0171]

[0172] in, are the upper bounds of energy storage charging and discharging power, I S,ch ,I S,dis is an auxiliary 0-1 variable, η S,ch ,η S,dis are the energy storage charging and discharging efficiency, E S is the total capacity of the energy storage device, SoC max,SoC min are the upper and lower bounds of the energy storage state of charge.

[0173] The load demand response constraint is shown in the formula, which sets the load that can be regulated at each moment to always account for a fixed proportion of the current load demand.

[0174]

[0175] in, is the upper bound of demand response load, η DR is the adjustable proportional coefficient, P L (t),Q L (t) are the load demand values at the current moment. The power flow constraint of the distribution network is often constructed by branch power flow balance relaxation constraint, as shown in formula (15) (16):

[0176]

[0177] Among them, l ij (t)=|I ij (t)| 2 represents the square of the branch current modulus, v i (t)=|V i (t)| 2 Represents the square of the node voltage modulus. ij ,x ij Represents branch resistance and reactance respectively. j (t),Q j (t) represents the sum of active and reactive power of all elements connected to node j. Since Equation (15) does not include the power balance constraint of the first node, i.e., the upper tie line, an additional equality constraint is set to ensure that the external power purchase power is equal to the sum of the power of the extended branches of the first node.

[0178] Generally speaking, the entire optimization problem belongs to a mixed integer nonlinear optimization problem, which needs to be linearized or convexly relaxed to make it a solvable problem. The main factor that the original model cannot be solved is the nonlinear nonconvex constraint of the power grid flow, which is convexly relaxed using the second-order cone relaxation technology. In the present invention, the energy balance constraint is processed using the second-order cone relaxation constraint shown in the following formula, as shown in formula (17) (18). After processing, the problem is converted into a mixed integer second-order cone optimization problem, and the optimal solution of the problem can be obtained using a solver.

[0179]

[0180] Among them, l ij is the square of the current modulus corresponding to the ij branch, v i is the square of the voltage modulus of node i, P ij ,Q ijis the active and reactive power flowing through branch ij. Since the constraint form of equation (17) is non-convex, the convex constraint form corresponding to equation (18) can be obtained by using second-order cone relaxation. Introducing the network loss parameter in the target is to regularize the convex relaxation of equation (18), so as to minimize the energy balance error caused by relaxation. Especially when the power generation of renewable energy exceeds the total load of the system, it is necessary to prevent the system from being overloaded by infinitely increasing l ij To meet the relaxation constraints and enable the optimization of new energy output to work effectively.

[0181] S2. Reinforcement learning accelerated optimization solution logic and its state space and decision variable settings

[0182] S201. Setting the action space

[0183] Reinforcement learning is suitable for handling time-series decision-making problems. When there are strong time-series-related constraints in the optimization model, the corresponding main variables can be used as reinforcement learning decision variables. In the source-grid-load-storage collaborative optimization problem, the charging and discharging power of the energy storage unit and the power of the unit with small climbing ability are suitable variables for reinforcement learning decision-making. The relevant constraints are shown in Equations (19) and (20). At this time, for the original optimization model, the variables in the action space are regarded as fixed values, which greatly reduces the complexity of the original model.

[0184]

[0185] In formula (19), the energy storage charging and discharging power P S,ch ,P S,dis Jointly determine the external apparent power P of energy storage S , involving the integer variable I of the charge and discharge behavior S,ch ,I S,dis , while the energy storage state of charge SoC expressed by formula (20) is a strong timing coupling equality constraint, η S,ch ,η S,dis is the charge and discharge efficiency coefficient. S Once determined by the neural network, all of the above constraints are removed from the optimization problem, significantly simplifying the constraint form. Furthermore, the power constraints corresponding to the unit power determined by reinforcement learning are also processed in the same way.

[0186] Corresponding formulas (21) and (22)

[0187]

[0188] Among them, P DG , Respectively represent the upper and lower bounds of active power, η DG Indicates the ratio of the maximum active power change to the upper bound. Indicates the rated power of the unit. is a collection of units whose power values are determined based on reinforcement learning.

[0189] After the reinforcement learning decision variables are determined, since most of the strong temporal coupling constraints are eliminated, the temporal correlation of the entire optimization problem is weakened, and it is transformed into multiple single-section optimization problems through the decoupling method. The variable definition of the single-section optimization problem does not contain the time variable t, that is, all optimization variables are limited to one moment. The objective function of the original problem also needs to be split. The goal of the single-section optimization problem at time t is the sum of the equivalent economic costs of all optimizable elements of the system during the optimization period. Some weak temporal coupling constraints are decoupled in a greedy manner, as shown in Equation (23). The power ramping capability of the unit is greater than the external source and load change capability. At this time, the upper and lower bounds of the power at time t are the power at time t-1 plus the upper and lower bounds of the ramp change rate.

[0190]

[0191] Among them, P DG (t) is the unit power at time t, η DG is the climbing coefficient, It is the set of units whose power values are determined based on the optimization solution.

[0192] S202, setting state space

[0193] Based on the above problem transformation method, the state characteristics of the problem are constructed and the basic decision-making framework is constructed. At the time t of the decision, the section parameters at time t are extracted to form a single section optimization problem. At the same time, the parameter information at time t and its historical time (t-1, t-2, ...) is used to construct the reinforcement learning decision state space. The state space is established based on the operation data of the power grid nodes. The state of the graph nodes is divided into uncontrollable environmental state quantities and controllable state quantities within the system. Among them, the uncontrollable environmental state quantities include new energy output, load demand, market electricity price, etc., and the controllable state quantities within the system include the charge state of the energy storage equipment, the current output level of the unit, etc., which are expressed as formula (24):

[0194] s env ={P RE (t),P Load (t),λ(t),P DG (t-1),Q DG (t-1),P DR (t-1),SoC(t-1)} (24)

[0195] Among them, P RE Represents the new energy power of each node in the system, P Load represents the load on each node of the system, λ represents the electricity purchase price at the current moment, P DG Represents the sum of the active power of the units at each node, QDG Represents the sum of reactive power at each node in the system, P DR It represents the sum of the demand response power at each node in the system, and SoC represents the state of charge of the energy storage at each node in the system.

[0196] A neural network is used to establish a functional mapping between the state space and the action space, that is, to calculate the specific action value from the specific action space. The specific model design needs to be introduced in conjunction with the reinforcement learning algorithm.

[0197] S203. Setting reward function

[0198] Assigning specific action space values to the single-section optimization problem and solving the optimization problem (implemented using a commercial solver) yields the decision objective function at time t, which determines the minimum cost under the premise of energy storage power, as well as the power values of other factors to be solved.

[0199]

[0200] This minimum value reflects the immediate cost value corresponding to the strategy at time t and is the basis for constructing the reward function. t There are parts of the value of that are not affected by the strategy, which causes deviations in the strategy returns at each moment and affects the strategy evaluation.

[0201] Assume that the immediate reward r t It is expressed as the part affected by the strategy and the part not affected by the strategy, as shown in formula (26).

[0202]

[0203] The immediate reward value corresponding to the baseline strategy is removed from the immediate reward to obtain the strategy-strongly related reward feedback for reinforcement learning training. The immediate reward value includes:

[0204]

[0205] Among them, r t Represents the immediate reward value, Represents the set of node numbers of the distributed unit system, Represents the set of energy storage system node numbers, Represents a set of node numbers that meet the demand response system requirements. Represents the set of node numbers of the power grid purchasing system; i represents the set Elements in, j represents the set Elements in, k represents the set The elements in n represent the set The elements in m represent the set Elements determined by reinforcement learning in CDG Represents the unit operation cost function, C S represents the energy storage depreciation cost function, C DR Denotes the load demand response cost function, C Ext represents the interaction cost function of the upper power grid, Represents the cost function of renewable energy curtailment, C GL Indicates the network loss converted to cost; P represents the power of the corresponding equipment, P * represents the variable value determined by reinforcement learning; SoC represents the state of charge percentage, and t represents the current time t.

[0206] The baseline reward values include:

[0207]

[0208] Among them, r t env represents the baseline reward value, Represents the energy storage and unit output values in the baseline strategy. The baseline reward setting benchmark is the overall operating cost corresponding to setting the energy storage and unit output values determined by reinforcement learning to 0.

[0209] Finally, the decision variables of the reinforcement learning action space are integrated with the decision variables obtained by the optimization solution to obtain the complete decision result for the current section at time t. The above loop is executed until the final section at time T is reached, thus achieving the decision of the entire optimization problem.

[0210] S3. Set up training task set

[0211] Because the state space contains a large number of policy-independent features, each of which exhibits temporal correlations, different time-varying patterns of these features emerge. Certain elements also exhibit different patterns at the same moment. For example, renewable energy and loads exhibit different behaviors at different grid nodes. Under these different time-varying patterns, the distribution network optimization strategy will also change accordingly. To improve reinforcement learning's adaptability to these time-varying patterns, it is necessary to construct a training task set to support model training.

[0212] The clustering and amplification method of the main time series features of the training task set is implemented. Taking the daily operation scenario as the boundary, the continuous operation actual data (monthly, quarterly, annual, etc.) of wind power, photovoltaic, load and upper grid prices at each node are decomposed to form a time series feature with 96 values (or more) (one data point every 15 minutes, 96 data per day). The decomposed time series features are classified based on the DBSCAN clustering method. Each class contains multiple approximate time series features and is regarded as a task with identical distribution characteristics. Data augmentation technology (GAN, proportional residual) and other technologies are used to amplify data for tasks with fewer samples to enrich the scenario modeling description.

[0213] The time series characteristics of each distribution network operating element are mapped to its physical node, forming multiple feature combinations. Each feature combination is defined as a task. Each round of reinforcement learning training is performed under the same task. During the reinforcement learning iterative solution process, the operating state at each moment is determined by the time series characteristics of the environmental elements at that moment.

[0214] S4. Reinforcement Learning Training Algorithm

[0215] Based on the training task set, the present invention proposes a meta-reinforcement learning training method to achieve effective reinforcement learning in different scenarios.

[0216] While meta-reinforcement learning (meta-RL) algorithms can, in principle, enable agents to learn new skills from small amounts of experience, several major challenges hinder their practical application. Current methods rely heavily on online, on-policy experience, limiting their sample efficiency. They also lack mechanisms for reasoning about task uncertainty when adapting to new tasks, limiting their effectiveness in problems with sparse rewards. In this paper, we address these challenges by developing an off-policy meta-RL algorithm that decouples task inference and control.

[0217] Based on the core elements of reinforcement learning constructed from the previous content and the training task set, a task distribution P(τ) is constructed, where each task is a Markov decision process, including a set of states, actions and a finite reward function, represented as task τ={p(s0),p(s t+1 |s t ,a t ),r(s t ,a t )}, where p(s0) is the initial state distribution, p(s t+1 |s t ,a t ) is the transfer distribution and r(s t ,a t ) is the reward function. Note that this problem definition includes task distributions with different transfer functions and different reward functions. Given a set of training tasks sampled from P(τ), the meta-training process learns a policy that adapts to the current task through the context c. Let is a state transition in task τ, which corresponds to a time sequence state change in power grid optimization. It constitutes the task experience collected so far and is the basis for extracting the potential context features of the task.

[0218] The task context feature vector is defined as Z, which is used to characterize the characteristics of the current running scene and is represented by π in the strategy function. θFeature embedding is achieved in the form of (a|s,z). Contains the experience collected so far, by building an inference network q φ (z|c) where φ is the neural network parameter. Construct a method to minimize the estimation deviation of the reinforcement learning value function and minimize the amount of feature information to train q for different scene features. φ (z|c) parameter. Its optimization objective is described as in formula (29):

[0219]

[0220] Among them, τ represents the number of a task, c τ is the context information of the corresponding task, z is the task context feature vector; g(τ,z) represents the optimization main function served by feature z, which will be designed in detail in the future in combination with the reinforcement learning algorithm; β is the target weight coefficient; p(z) is the unit Gaussian prior distribution, and the standard normal distribution function is used here.

[0221] Function q φ (z|c τ ) is composed of multiple state transition sections The aggregated features are expressed as formula (30):

[0222]

[0223] in, represents the aggregation function that integrates the 1st to mth features, Ψ φ (z|c n ) indicates that the nth state transition sample is input into the inference network q φ The output distribution obtained by (z|c). In the present invention, Ψ adopts the normal distribution function, The Gaussian distribution product form is used.

[0224] Based on the context features, the deep reinforcement learning SAC algorithm is used to train the distribution network optimization model. The SAC algorithm has good sample efficiency and stability and can be well integrated with the potential context features. The reparameterization method is used to calculate the sampling feature vector z for q φ The gradient of the parameters in (z|c) is used to optimize the inference network q φ Parameters of (z|c), policy network π θ (a|s,z) and evaluation network Parameters. Where s is the same as s in formula (x) env The updated loss function of the evaluation network is shown in formula (31):

[0225]

[0226] Among them, L critic To evaluate the network loss function symbol, represents the evaluation network function, V represents the baseline evaluation network function, s′ represents the next state of migration, Represents the z vector without gradient update.

[0227] The policy network loss function is shown in formula (32):

[0228]

[0229] Among them, L actor is the symbol of the policy network loss function, D KL (·||·) represents the KL divergence function between two distributions, exp represents the exponential function technique, K θ Represents the state normalization factor.

[0230] Based on the context inference loss function defined in formula (31), since the context features are mainly used for value function prediction, the function g(τ,z) can be replaced by the evaluation loss function defined in formula (33), which is expressed as:

[0231]

[0232] S5. Neural Network Construction Method Supporting Reinforcement Learning for Distribution Network Operation Optimization

[0233] This method is primarily used to design the neural network architecture for reinforcement learning strategies, evaluation, and task representation, supporting step S4, the strategy update training for the reinforcement learning algorithm. A neural network input layer is constructed to effectively extract the spatiotemporal correlation characteristics of the distribution network's source, grid, load, and storage states. A neural network output layer is constructed that corresponds to the action space strategy distribution. The intermediate layer parameter link structure is designed to achieve the input-to-output mapping, forming the reinforcement learning strategy, evaluation, and task representation architecture.

[0234] The Actor-Critic architecture is used, which includes a policy neural network and an evaluation neural network. First, the main structure of the evaluation neural network is introduced, such as Figure 2 As shown:

[0235] Among them, (s n ,a n ,r n ,s′ nContext information fragments consisting of four tuples are input into the feature extraction neural network to obtain the corresponding context fragment feature distributions. For n tuples, n distributions will be obtained. The n fragment feature distributions are multiplied by Gaussian distributions to obtain the global context feature distribution. The distribution is then sampled to obtain a one-dimensional tensor of the global context feature. This one-dimensional tensor is concatenated with the action space and state space defined above and used as the input to the evaluation network. The final output is the value function estimate for the corresponding state-action combination.

[0236] For the policy network, since the gradient update does not involve the feature extraction network parameters, its input is the concatenation of the state space and the one-dimensional tensor of the context global feature, and the output is the value distribution of each variable in the action space, which is used to calculate the optimization strategy and update the neural network.

[0237] The evaluation network, the policy network, and the feature extraction network may adopt a fully connected network or a Transformer structure, etc., which is not specifically limited in the present invention, and any update algorithm is applicable.

[0238] In summary, the complete reinforcement learning training method for multi-scenario optimization of power generation, grid, load and storage collaboration is presented as follows:

[0239] 1) Construct a training task set containing i tasks {τ i}; 2) Initialize a sample playback buffer B for each task i ; 3) Let j be the task number, take j = 1; 4) for task τ j , initialize the context feature cache 5) Let k be the number of task training cycles, and take k = 1; 6) From Sample a context global feature tensor z; 7) Based on the tensor z, use the policy network π θ (a|s,z) from task τ j Get the interactive trajectory data and store it in B j 8) B j The data in is stored in the form of (s,a,r,s′) 9) Task training cycle number k+1, if k is less than the maximum task training cycle number K, then return to step 6), otherwise continue to step 10); 10) Task training cycle number j+1, if j is greater than the number of tasks, continue to step 11), otherwise return to step 4); 11) Let step be the number of training processes, take step = 1; 12) Replay the sample buffer B from each task i The training sample set b of the sampling algorithm i , Conghua context feature cache Sampling context information c i ; 13) from q φ (z|c i) to sample a context global feature tensor z; 14) Based on formula (31) and formula (32), respectively calculate 15) Calculate the feature network loss function 16) Calculate the total number of tasks The sum of the two, respectively update the policy network θ and the evaluation network and feature network φ; 17) training process number step+1, if step is less than the upper limit of the update step number, return to step 12), otherwise continue to step 18); 18) determine whether to exit the loop based on the training effect, if continue training, return to step 3), otherwise end training.

[0240] Based on the multi-scenario optimization reinforcement learning algorithm, the distribution network operation optimization strategy neural network is trained to obtain the distribution network operation adaptive optimization intelligent agent that is compatible with source and load scenario changes.

[0241] Example 2

[0242] The present invention provides a reinforcement learning system for source-grid-load-storage collaborative multi-scenario optimization. The system can be used to implement the above-mentioned reinforcement learning method for source-grid-load-storage collaborative multi-scenario optimization. Specifically, the reinforcement learning system for source-grid-load-storage collaborative multi-scenario optimization includes a problem module, a transformation module, a generation module, a training module and an output module.

[0243] Among them, the problem module builds a distribution network optimization model with cost minimization, defines the objective function and constraints, and transforms the mixed integer nonlinear problem into a solvable mixed integer second-order cone optimization problem through convex relaxation;

[0244] For the distribution network optimization model, the second-order cone relaxation constraint is used to deal with the energy balance constraint, which is then transformed into a mixed integer second-order cone optimization problem. The second-order cone relaxation constraint is as follows:

[0245]

[0246] Among them, l ij is the square of the current modulus corresponding to the ij branch, v i is the square of the voltage modulus of node i, P ij ,Q ij is the active and reactive power flowing through branch ij;

[0247] The objective function of the distribution network optimization model is as follows:

[0248]

[0249] Where C is the cost function of distributed units, energy storage, demand response and power purchase from the grid, is the operating cost of the i-th unit, is the operating cost of the j-th energy storage device, is the operating cost of the kth demand response unit, is the cost of curtailing electricity from the nth renewable energy unit, C Ext is the cost of electricity purchase and sale by the upper power grid, P is the power of the corresponding equipment, and P i DG (t) is the operating power of the i-th unit at time t, is the operating power of the jth energy storage device at time t, is the operating power of the kth demand response unit at time t, is the power abandonment of the nth new energy unit at time t, P Ext (t) is the interaction power of the upper power grid at time t, represents the set of system node numbers where the corresponding equipment is located, α, β, c correspond to the second-order, first-order and constant coefficients respectively, ζ, ρ, λ correspond to their cost coefficients respectively, T represents the total optimization time, t is the current time, Δt is the time interval, i.e., the optimization step size, C GL is the network loss parameter, is the expectation operator.

[0250] The conversion module converts the obtained mixed integer second-order cone optimization problem into a reinforcement learning decision model and defines the action space, state space and reward function of the reinforcement learning decision model;

[0251] The distribution network optimization model is decoupled in time series. The reinforcement learning action space is defined as the charging and discharging power of the energy storage unit and the power of the unit with a small climbing capability. The time series coupling constraints associated with the reinforcement learning action space are transformed. The power constraint of the unit operation is modeled as follows:

[0252]

[0253] Among them, P DG (t) is the unit power at time t, η DG is the climbing coefficient, is the set of units whose power values are determined based on the optimization solution, P DG , Respectively represent the upper and lower bounds of active power, η DG Indicates the ratio of the maximum active power change to the upper bound;

[0254] The operating constraints of energy storage operating power are modeled as follows:

[0255]

[0256] SoC min ≤SoC(t)≤SoC max

[0257] Among them, the energy storage charging and discharging power P S,ch ,P S,dis Together determine the external apparent power of energy storage, η S,ch ,η S,dis is the charge and discharge efficiency coefficient, Maximum charging capacity for energy storage, SoC min ,SoC max are the minimum and maximum values of the energy storage state of charge, respectively;

[0258] The state space is defined as follows:

[0259] s env ={P RE (t),P Load (t),λ(t),P DG (t-1),Q DG (t-1),P DR (t-1),SoC(t-1)}

[0260] Among them, P RE Represents the new energy power of each node in the system, P Load represents the load on each node of the system, λ represents the electricity purchase price at the current moment, P DG Represents the sum of the active power of the units at each node, Q DG Represents the sum of reactive power at each node in the system, P DR It represents the sum of the demand response power at each node in the system, and SoC represents the state of charge of the energy storage at each node in the system;

[0261] The immediate reward part of the reward function is expressed as follows:

[0262]

[0263] Among them, r t Represents the immediate reward value, Represents the set of node numbers of the distributed unit system, Represents the set of energy storage system node numbers, Represents a set of node numbers that meet the demand response system requirements. Represents the set of node numbers of the power grid purchasing system; i represents the set Elements in, j represents the set Elements in, k represents the set The elements in n represent the set The elements in m represent the set Elements determined by reinforcement learning in C DG Represents the unit operation cost function, C S represents the energy storage depreciation cost function, C DRDenotes the load demand response cost function, C Ext represents the interaction cost function of the upper power grid, Represents the cost function of renewable energy curtailment, C GL Indicates the network loss converted to cost; P represents the power of the corresponding equipment, P * represents the variable value determined by reinforcement learning; SoC represents the state of charge percentage, and t represents the current time.

[0264] The baseline part of the reward function is expressed as follows:

[0265]

[0266] Among them, r t env represents the baseline reward value, Represents the energy storage and unit output values in the baseline strategy. The baseline reward setting benchmark is the overall operating cost corresponding to setting the energy storage and unit output values determined by reinforcement learning to 0.

[0267] The generation module uses different operating scenario data to perform multiple rounds of assignments on the boundary condition parameters in the constructed distribution network optimization model, and forms a multi-scenario training task set based on the defined reinforcement learning decision model;

[0268] The diverse set of training tasks is as follows:

[0269] Based on daily operation scenarios, the continuous operation data of wind power, photovoltaic power, load and upper-level grid prices at each node are decomposed to form the time series characteristics of operation factors;

[0270] The decomposed time series features of the operating elements are classified based on the DBSCAN clustering method. Each class contains multiple approximate time series features and is regarded as a task with identical distribution characteristics.

[0271] The temporal characteristics of the operating elements of each distribution network are matched with their physical nodes to form multiple feature combinations, and each feature combination is defined as a task; each round of reinforcement learning training is performed under the same task. During the reinforcement learning iterative solution process, the operating state corresponding to each moment is determined by the temporal characteristics of the environmental elements at the current moment.

[0272] The training module designs multi-scenario training loss functions for the defined reinforcement learning decision model, including the policy network loss function, the evaluation network loss function, and the scenario representation loss function. It also constructs the policy neural network, the evaluation network neural network, and the scenario representation embedding neural network based on the defined reinforcement learning decision model as parameter objects for updating the reinforcement learning strategy, completing reinforcement learning decision-making that adapts to multi-scenario optimization.

[0273] The reinforcement learning strategy model adapted to multiple scenarios is obtained as follows:

[0274] Based on the core elements of reinforcement learning and the training task set, a task distribution P(τ) is constructed. Each task is a Markov decision process, including a set of states, actions and a finite reward function, represented as task τ = {p(s0), p(s t+1 |s t ,a t ),r(s t ,a t )}, p(s0) is the initial state distribution, p(s t+1 |s t ,a t ) is the transfer distribution and r(s t ,a t ) is the reward function;

[0275] Given a set of training tasks sampled from P(τ), the meta-training process learns a policy that adapts to the current task through the context c;

[0276] set up is a state transition in task τ, which corresponds to a time sequence state change in power grid optimization. The mission experience that constitutes the present collection;

[0277] Define the task context feature vector as Z, which is used to characterize the characteristics of the current running scene, and is represented by π in the strategy function. θ Feature embedding is achieved in the form of (a|s,z);

[0278] By constructing an inference network q φ (z|c) constructs a reinforcement learning value function that minimizes the estimated deviation, φ is the neural network parameter, and at the same time trains q for different scene features in a way that minimizes the amount of feature information. φ (z|c) parameters;

[0279] The distribution network optimization model is trained using the deep reinforcement learning SAC algorithm, and the reparameterization method is used to calculate the sampling feature vector z versus q φ The gradient of the parameters in (z|c) is used to optimize the inference network q φ Parameters of (z|c), policy network π θ (a|s,z) and evaluation network Parameters;

[0280] The multi-scene loss function is as follows:

[0281]

[0282] in, represents the expectation of task τ, represents the expectation between the sample sets, D KL Represents the KL divergence operator, q φ represents the scene element representation embedded in the neural network, τ represents the number of a task, c τ is the context information of the corresponding task, z is the task context feature vector; g(τ,z) represents the optimization main function served by feature z, which will be designed in detail in the future in combination with the reinforcement learning algorithm; β is the target weight coefficient; p(z) is the unit Gaussian prior distribution;

[0283] The updated loss function of the evaluation network with scene meta-representation embedding is as follows:

[0284]

[0285] Among them, L critic To evaluate the network loss function symbol, represents the evaluation network function, V represents the baseline evaluation network function, s′ represents the next state of migration, Indicates the z vector without gradient update;

[0286] The loss function of the policy network with scene meta-representation embedding is as follows:

[0287]

[0288] Among them, L actor is the symbol of the policy network loss function, D KL (·||·) represents the KL divergence function between two distributions, exp represents the exponential function technique, K θ Represents the state normalization factor.

[0289] The output module trains the distribution network multi-scenario optimization decision model based on the constructed multi-scenario training task set, the defined multi-scenario training reinforcement learning loss function and the neural network structure, and deploys it to the actual system to complete the application of the reinforcement learning decision model for multi-scenario optimization of source-grid-load-storage collaboration.

[0290] The executable neural network model adopts the Actor-Critic architecture, which includes a policy neural network, an evaluation neural network, and a scene meta-representation neural network;

[0291] (s n ,a n ,r n ,s′ n) Inputting the context information fragments composed of four tuples into the scene element representation neural network to obtain the corresponding context fragment feature distribution, and obtaining n distributions for n four tuples; multiplying the n fragment feature distributions by Gaussian distribution products to obtain the scene element representation distribution, and using distribution sampling to obtain the scene element representation one-dimensional tensor;

[0292] The one-dimensional tensor representing the scene element is concatenated with the action space and state space defined by the content as the input of the evaluation network, and the final output is the value function estimate of the corresponding state-action combination;

[0293] The input of the policy network is the concatenation of the state space and the one-dimensional tensor representing the scene element, and the output is the value distribution of each variable in the action space, which is used to calculate the optimization strategy and update the policy neural network;

[0294] The evaluation network, policy network, and scene element representation network adopt a fully connected network or Transformer structure.

[0295] Example 3

[0296] The present invention provides a terminal device comprising a processor and memory, the memory being used to store a computer program including program instructions, and the processor being used to execute the program instructions stored on the computer storage medium. The processor is the computing and control core of the terminal, adapted to implement one or more instructions, specifically adapted to load and execute one or more instructions to implement a corresponding method flow or function. The processor described in an embodiment of the present invention can be used to operate a reinforcement learning method for optimizing multi-scenario collaboration between source, grid, load, and storage, including:

[0297] A cost-minimizing distribution network optimization model is constructed, the objective function and constraints are defined, and the mixed-integer nonlinear problem is converted into a solvable mixed-integer second-order cone optimization problem through convex relaxation; the obtained mixed-integer second-order cone optimization problem is converted into a reinforcement learning decision model, and the action space, state space and reward function of the reinforcement learning decision model are defined; different operating scenario data are used to perform multiple rounds of assignment of boundary condition parameters in the constructed distribution network optimization model, and a multi-scenario training task set is formed based on the defined reinforcement learning decision model; a multi-scenario training loss function is designed for the defined reinforcement learning decision model, including the policy network loss function, the evaluation network loss function and the scenario representation loss function; combined with the defined reinforcement learning decision model, a policy neural network, an evaluation network neural network and a scenario representation embedded neural network are constructed as parameter objects for the reinforcement learning strategy update, completing the reinforcement learning decision adapted to multi-scenario optimization; based on the constructed multi-scenario training task set, the defined multi-scenario training reinforcement learning loss function and the neural network structure, the distribution network multi-scenario optimization decision model is trained and deployed to the actual system, completing the application of the reinforcement learning decision model for multi-scenario optimization of source-grid-load-storage collaboration.

[0298] See also Figure 5 The terminal device is a computer device. The computer device 60 of this embodiment includes: a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When the computer program 63 is executed by the processor 61, it implements the reinforcement learning method for source-grid-load-storage collaborative multi-scenario optimization in the embodiment. To avoid repetition, it is not described in detail here. Alternatively, when the computer program 63 is executed by the processor 61, it implements the functions of each model / unit in the reinforcement learning system for source-grid-load-storage collaborative multi-scenario optimization in the embodiment. To avoid repetition, it is not described in detail here.

[0299] See also Figure 6 The terminal device is an electronic device 600, which is implemented as a general-purpose computing device. The components of the electronic device may include, but are not limited to, at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), and a display unit 640.

[0300] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0301] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0302] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0303] Bus 630 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0304] The electronic device 600 may also communicate with one or more external devices 700 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., a router, a modem). Such communication may occur via an input / output interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network, a wide area network, and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0305] Example 4

[0306] The present invention also provides a storage medium, specifically a computer-readable storage medium, which is a memory device in a terminal device for storing programs and data. It is understood that the computer-readable storage medium herein may include both built-in storage media in the terminal device and, of course, extended storage media supported by the terminal device. It may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for being loaded and executed by a processor. These instructions may be one or more computer programs (including program code). It should be noted that more specific examples of the computer-readable storage medium herein include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical fiber, a portable compact disk read-only memory, an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0307] Computer-readable storage media also include data signals propagated in baseband or as part of a carrier wave, which carry readable program code. Such propagated data signals can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium can also be any readable medium other than a readable storage medium, which can send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, radio frequency, etc., or any suitable combination of the above.

[0308] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, and the like, as well as conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network or a wide area network, or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0309] The processor may load and execute one or more instructions stored in a computer-readable storage medium to implement the corresponding steps of the reinforcement learning method for multi-scenario optimization of source-grid-load-storage collaboration in the above embodiment; the processor may load and execute the following steps:

[0310] A cost-minimizing distribution network optimization model is constructed, the objective function and constraints are defined, and the mixed-integer nonlinear problem is converted into a solvable mixed-integer second-order cone optimization problem through convex relaxation; the obtained mixed-integer second-order cone optimization problem is converted into a reinforcement learning decision model, and the action space, state space and reward function of the reinforcement learning decision model are defined; different operating scenario data are used to perform multiple rounds of assignment of boundary condition parameters in the constructed distribution network optimization model, and a multi-scenario training task set is formed based on the defined reinforcement learning decision model; a multi-scenario training loss function is designed for the defined reinforcement learning decision model, including the policy network loss function, the evaluation network loss function and the scenario representation loss function; combined with the defined reinforcement learning decision model, a policy neural network, an evaluation network neural network and a scenario representation embedded neural network are constructed as parameter objects for the reinforcement learning strategy update, completing the reinforcement learning decision adapted to multi-scenario optimization; based on the constructed multi-scenario training task set, the defined multi-scenario training reinforcement learning loss function and the neural network structure, the distribution network multi-scenario optimization decision model is trained and deployed to the actual system, completing the application of the reinforcement learning decision model for multi-scenario optimization of source-grid-load-storage collaboration.

[0311] The database involved in each embodiment provided by the present invention may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchain. The processor involved in each embodiment provided by the present invention may be, but is not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, etc.

[0312] See also Figure 3 and Figure 4 ,The context feature extraction method designed by the present invention can effectively extract the similarities between scenes. Figure 3 The figure shows the convergence of contextual feature encodings z for four different source-load scenarios during algorithm training. The source-load scenarios for Tasks 0 and 2 (the upper and lower figures on the left) were taken from the same month, and the source-load scenarios for Tasks 1 and 3 (the upper and lower figures on the right) were taken from the same month. As can be seen, under the same topological conditions, the distribution of contextual feature encodings z obtained for tasks corresponding to the same month is similar, indicating that the meta-representation effectively captures both commonalities and differences across scenarios.

[0313] The reinforcement learning algorithm designed by the present invention can realize effective strategy training in multiple task sets. Figure 4 The cumulative reward curves for the model on different tasks during training are shown. The reward values fluctuate as training progresses. This fluctuation is due to inherent biases in reward feedback across different tasks. However, the average reward value is on the rise, demonstrating that the meta-reinforcement learning algorithm effectively overcomes the difficulty in effectively training strategies due to differences in task scenarios.

[0314] In summary, the present invention provides a reinforcement learning method and system for optimizing the coordinated multi-scenario of power generation, grid, load, and storage. By combining convex relaxation with reinforcement learning, it solves the MINLP problem that is difficult to handle with traditional optimization methods. Based on meta-learning and context awareness, it enables rapid adaptation of strategies in diverse scenarios. It integrates multiple cost functions to reduce the operating costs of the distribution network. Through neural network model compression and feature extraction, it supports real-time decision-making and edge computing. Through loss function design, KL divergence constraints, and prior distribution, it improves the stability of the training process.

[0315] The above content is only for explaining the technical idea of the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A reinforcement learning method for multi-scenario optimization of source-grid-load-storage collaboration, characterized by: The following steps are involved: Construct a cost-minimizing distribution network optimization model, define the objective function and constraints, and transform the mixed-integer nonlinear problem into a solvable mixed-integer second-order cone optimization problem through convex relaxation. The obtained mixed integer second-order cone optimization problem is transformed into a reinforcement learning decision model, and the action space, state space and reward function of the reinforcement learning decision model are defined; Using different operating scenario data, the boundary condition parameters in the constructed distribution network optimization model are assigned multiple times, and a multi-scenario training task set is formed based on the defined reinforcement learning decision model. Design multi-scenario training loss functions for the defined reinforcement learning decision model, including the policy network loss function, the evaluation network loss function, and the scenario representation loss function; construct the policy neural network, the evaluation network neural network, and the scenario representation embedding neural network based on the defined reinforcement learning decision model as parameter objects for updating the reinforcement learning strategy, and complete the reinforcement learning decision that adapts to multi-scenario optimization; Based on the constructed multi-scenario training task set, the defined multi-scenario training reinforcement learning loss function and neural network structure, the distribution network multi-scenario optimization decision model is trained and deployed to the actual system to complete the application of the reinforcement learning decision model for multi-scenario optimization of source-grid-load-storage collaboration.

2. The reinforcement learning method for multi-scenario optimization of source-grid-load-storage collaboration according to claim 1 is characterized in that: For the distribution network optimization model, the second-order cone relaxation constraint is used to deal with the energy balance constraint, which is then transformed into a mixed integer second-order cone optimization problem. The second-order cone relaxation constraint is as follows: Among them, l ij is the square of the current modulus corresponding to the ij branch, v i is the square of the voltage modulus of node i, P ij ,Q ij is the active and reactive power flowing through branch ij.

3. The reinforcement learning method for multi-scenario optimization of source-grid-load-storage collaboration according to claim 2 is characterized in that: The objective function of the distribution network optimization model is as follows: Where C is the cost function of distributed units, energy storage, demand response and power purchase from the grid, is the operating cost of the i-th unit, is the operating cost of the j-th energy storage device, is the operating cost of the kth demand response unit, is the cost of curtailing electricity from the nth renewable energy unit, C Ext is the cost of electricity purchase and sale by the upper power grid, P is the power of the corresponding equipment, and P i DG (t) is the operating power of the i-th unit at time t, is the operating power of the jth energy storage device at time t, is the operating power of the kth demand response unit at time t, is the power abandonment of the nth new energy unit at time t, P Ext (t) is the interaction power of the upper power grid at time t, represents the set of system node numbers where the corresponding equipment is located, α, β, c correspond to the second-order, first-order and constant coefficients respectively, ζ, ρ, λ correspond to their cost coefficients respectively, T represents the total optimization time, t is the current time, Δt is the time interval, i.e., the optimization step size, C GL is the network loss parameter, is the expectation operator.

4. The reinforcement learning method for multi-scenario optimization of source-grid-load-storage collaboration according to claim 1 is characterized in that: The distribution network optimization model is decoupled in time series. The reinforcement learning action space is defined as the charging and discharging power of the energy storage unit and the power of the unit with a small climbing capability. The time series coupling constraints associated with the reinforcement learning action space are transformed. The power constraint of the unit operation is modeled as follows: Among them, P DG (t) is the unit power at time t, η DG is the climbing coefficient, is the set of units whose power values are determined based on the optimization solution, P DG , Respectively represent the upper and lower bounds of active power, η DG Indicates the ratio of the maximum active power change to the upper bound; The operating constraints of energy storage operating power are modeled as follows: SoC min ≤SoC(t)≤SoC max Among them, the energy storage charging and discharging power P S,ch ,P S,dis Together determine the external apparent power of energy storage, η S,ch ,η S,dis is the charge and discharge efficiency coefficient, Maximum charging capacity for energy storage, SoC min ,SoC max are the minimum and maximum values of the energy storage state of charge, respectively; The state space is defined as follows: yes env ={P RE (t),P Load (t),λ(t),P DG (t-1),Q DG (t-1),P DR (t-1),SoC(t-1)} Among them, P RE Represents the new energy power of each node in the system, P Load represents the load on each node of the system, λ represents the electricity purchase price at the current moment, P DG Represents the sum of the active power of the units at each node, Q DG Represents the sum of reactive power at each node in the system, P DR It represents the sum of the demand response power at each node in the system, and SoC represents the state of charge of the energy storage at each node in the system; The immediate reward part of the reward function is expressed as follows: Among them, r t Represents the immediate reward value, Represents the set of node numbers of the distributed unit system, Represents the set of energy storage system node numbers, Represents a set of node numbers that meet the demand response system requirements. Represents the set of node numbers of the power grid purchasing system; i represents the set Elements in, j represents the set Elements in, k represents the set The elements in n represent the set The elements in m represent the set Elements determined by reinforcement learning in C DG Represents the unit operation cost function, C S represents the energy storage depreciation cost function, C DR Denotes the load demand response cost function, C Ext represents the interaction cost function of the upper power grid, Represents the cost function of renewable energy curtailment, C GL Indicates the network loss converted to cost; P represents the power of the corresponding equipment, P * represents the variable value determined by reinforcement learning; SoC represents the state of charge percentage, and t represents the current time; The baseline part of the reward function is expressed as follows: Among them, r t env represents the baseline reward value, Represents the energy storage and unit output values in the baseline strategy. The baseline reward setting benchmark is the overall operating cost corresponding to setting the energy storage and unit output values determined by reinforcement learning to 0.

5. The reinforcement learning method for multi-scenario optimization of source-grid-load-storage collaboration according to claim 1 is characterized in that: The specific multi-scenario training task set is: Based on daily operation scenarios, the continuous operation data of wind power, photovoltaic power, load and upper-level grid prices at each node are decomposed to form the time series characteristics of operation factors; The decomposed time series features of the operating elements are classified based on the DBSCAN clustering method. Each class contains multiple approximate time series features and is regarded as a task with identical distribution characteristics. The temporal characteristics of the operating elements of each distribution network are matched with their physical nodes to form multiple feature combinations, and each feature combination is defined as a task; each round of reinforcement learning training is performed under the same task. During the reinforcement learning iterative solution process, the operating state corresponding to each moment is determined by the temporal characteristics of the environmental elements at the current moment.

6. The reinforcement learning method for multi-scenario optimization of source-grid-load-storage collaboration according to claim 1 is characterized in that: The specific reinforcement learning decision-making process adapted to multi-scenario optimization is as follows: Based on the core elements of reinforcement learning and the training task set, a task distribution P(τ) is constructed. Each task is a Markov decision process, including a set of states, actions and a finite reward function, represented as task τ = {p(s0), p(s t+1 |s t ,a t ),r(s t ,a t )}, p(s0) is the initial state distribution, p(s t+1 |s t ,a t ) is the transfer distribution and r(s t ,a t ) is the reward function; Given a set of training tasks sampled from P(τ), the meta-training process learns a policy that adapts to the current task through the context c; set up is a state transition in task τ, which corresponds to a time sequence state change in power grid optimization. The mission experience that constitutes the present collection; Define the task context feature vector as Z, which is used to characterize the characteristics of the current running scene, and is represented by π in the strategy function. θ Feature embedding is achieved in the form of (a|s,z); By constructing an inference network q φ (z|c) constructs a reinforcement learning value function that minimizes the estimated deviation, φ is the neural network parameter, and at the same time trains q for different scene features in a way that minimizes the amount of feature information. φ (z|c) parameters; The distribution network optimization model is trained using a deep reinforcement learning algorithm, and the reparameterization method is used to calculate the sampling feature vector z versus q φ The gradient of the parameters in (z|c) is used to optimize the inference network q φ Parameters of (z|c), policy network π θ (a|s,z) and evaluation network Parameters.

7. The reinforcement learning method for multi-scenario optimization of source-grid-load-storage collaboration according to claim 6 is characterized in that: The multi-scene loss function is as follows: in, represents the expectation of task τ, represents the expectation between the sample sets, D KL Represents the KL divergence operator, q φ represents the scene element representation embedded in the neural network, τ represents the number of a task, c τ is the context information of the corresponding task, z is the task context feature vector; g(τ,z) represents the optimization main function served by feature z, which will be designed in detail in the future in combination with the reinforcement learning algorithm; β is the target weight coefficient; p(z) is the unit Gaussian prior distribution.

8. The reinforcement learning method for multi-scenario optimization of source-grid-load-storage collaboration according to claim 6 is characterized in that: The updated loss function of the evaluation network with scene meta-representation embedding is as follows: Among them, L critic To evaluate the network loss function symbol, Represents the evaluation network function, V represents the baseline evaluation network function, s′ represents the next state of migration, and z represents the z vector without gradient update; The loss function of the policy network with scene meta-representation embedding is as follows: Among them, L actor is the symbol of the policy network loss function, D KL (·||·) represents the KL divergence function between two distributions, exp represents the exponential function technique, K θ Represents the state normalization factor.

9. The reinforcement learning method for source-grid-load-storage collaborative multi-scenario optimization according to claim 6 is characterized in that: The executable neural network model adopts the Actor-Critic architecture, which includes a policy neural network, an evaluation neural network, and a scene meta-representation neural network; (s n ,a n ,r n ,s′ n ) Inputting the context information fragments composed of four tuples into the scene element representation neural network to obtain the corresponding context fragment feature distribution, and obtaining n distributions for n four tuples; multiplying the n fragment feature distributions by Gaussian distribution products to obtain the scene element representation distribution, and using distribution sampling to obtain the scene element representation one-dimensional tensor; The one-dimensional tensor representing the scene element is concatenated with the action space and state space defined by the content as the input of the evaluation network, and the final output is the value function estimate of the corresponding state-action combination; The input of the policy network is the concatenation of the state space and the one-dimensional tensor representing the scene element, and the output is the value distribution of each variable in the action space, which is used to calculate the optimization strategy and update the policy neural network.

10. The reinforcement learning method for multi-scenario optimization of source-grid-load-storage collaboration according to claim 9 is characterized in that: The evaluation network, policy network, and scene element representation network adopt a fully connected network or Transformer structure.

11. A reinforcement learning system for multi-scenario optimization of source-grid-load-storage collaboration, characterized by: include: Problem module, which builds a cost-minimizing distribution network optimization model, defines the objective function and constraints, and transforms the mixed-integer nonlinear problem into a solvable mixed-integer second-order cone optimization problem through convex relaxation; The conversion module converts the obtained mixed integer second-order cone optimization problem into a reinforcement learning decision model and defines the action space, state space and reward function of the reinforcement learning decision model; The generation module uses different operating scenario data to perform multiple rounds of assignments on the boundary condition parameters in the constructed distribution network optimization model, and forms a multi-scenario training task set based on the defined reinforcement learning decision model; The training module designs multi-scenario training loss functions for the defined reinforcement learning decision model, including the policy network loss function, the evaluation network loss function, and the scenario representation loss function. It also constructs the policy neural network, the evaluation network neural network, and the scenario representation embedding neural network based on the defined reinforcement learning decision model as parameter objects for updating the reinforcement learning strategy, completing reinforcement learning decision-making that adapts to multi-scenario optimization. The output module trains the distribution network multi-scenario optimization decision model based on the constructed multi-scenario training task set, the defined multi-scenario training reinforcement learning loss function and the neural network structure, and deploys it to the actual system to complete the application of the reinforcement learning decision model for multi-scenario optimization of source-grid-load-storage collaboration.

12. The reinforcement learning system for source-grid-load-storage collaborative multi-scenario optimization according to claim 11 is characterized in that: In the problem module, for the distribution network optimization model, the energy balance constraint is processed using the second-order cone relaxation constraint, which is then transformed into a mixed integer second-order cone optimization problem. The second-order cone relaxation constraint is as follows: Among them, l ij is the square of the current modulus corresponding to the ij branch, v i is the square of the voltage modulus of node i, P ij ,Q ij is the active and reactive power flowing through branch ij; The objective function of the distribution network optimization model is as follows: Where C is the cost function of distributed units, energy storage, demand response and power purchase from the grid, is the operating cost of the i-th unit, is the operating cost of the j-th energy storage device, is the operating cost of the kth demand response unit, is the cost of curtailing electricity from the nth renewable energy unit, C Ext is the cost of electricity purchase and sale by the upper power grid, P is the power of the corresponding equipment, and P i DG (t) is the operating power of the i-th unit at time t, is the operating power of the jth energy storage device at time t, is the operating power of the kth demand response unit at time t, is the power abandonment of the nth new energy unit at time t, P Ext (t) is the interaction power of the upper power grid at time t, represents the set of system node numbers where the corresponding equipment is located, α, β, c correspond to the second-order, first-order and constant coefficients respectively, ζ, ρ, λ correspond to their cost coefficients respectively, T represents the total optimization time, t is the current time, Δt is the time interval, i.e., the optimization step size, C GL is the network loss parameter, is the expectation operator.

13. The reinforcement learning system for source-grid-load-storage collaborative multi-scenario optimization according to claim 11 is characterized in that: In the conversion module, the distribution network optimization model is decoupled in time series. The reinforcement learning action space is defined as the charging and discharging power of the energy storage unit and the power of the unit with a small climbing capability. The time series coupling constraints associated with the reinforcement learning action space are converted. The power constraints for the unit operation are modeled as follows: Among them, P DG (t) is the unit power at time t, η DG is the climbing coefficient, is the set of units whose power values are determined based on the optimization solution, P DG , Respectively represent the upper and lower bounds of active power, η DG Indicates the ratio of the maximum active power change to the upper bound; The operating constraints of energy storage operating power are modeled as follows: SoC min ≤SoC(t)≤SoC max Among them, the energy storage charging and discharging power P S,ch ,P S,dis Together determine the external apparent power of energy storage, η S,ch ,η S,dis is the charge and discharge efficiency coefficient, Maximum charging capacity for energy storage, SoC min ,SoC max are the minimum and maximum values of the energy storage state of charge, respectively; The state space is defined as follows: yes env ={P RE (t),P Load (t),λ(t),P DG (t-1),Q DG (t-1),P DR (t-1),SoC(t-1)} Among them, P RE Represents the new energy power of each node in the system, P Load represents the load on each node of the system, λ represents the electricity purchase price at the current moment, P DG Represents the sum of the active power of the units at each node, Q DG Represents the sum of reactive power at each node in the system, P DR It represents the sum of the demand response power at each node in the system, and SoC represents the state of charge of the energy storage at each node in the system; The immediate reward part of the reward function is expressed as follows: Among them, r t Represents the immediate reward value, Represents the set of node numbers of the distributed unit system, Represents the set of energy storage system node numbers, Represents a set of node numbers that meet the demand response system requirements. Represents the set of node numbers of the power grid purchasing system; i represents the set Elements in, j represents the set Elements in, k represents the set The elements in n represent the set The elements in m represent the set Elements determined by reinforcement learning in C DG Represents the unit operation cost function, C S represents the energy storage depreciation cost function, C DR Denotes the load demand response cost function, C Ext represents the interaction cost function of the upper power grid, Represents the cost function of renewable energy curtailment, C GL Indicates the network loss converted to cost; P represents the power of the corresponding equipment, P * represents the variable value determined by reinforcement learning; SoC represents the state of charge percentage, and t represents the current time; The baseline part of the reward function is expressed as follows: Among them, r t env represents the baseline reward value, Represents the energy storage and unit output values in the baseline strategy. The baseline reward setting benchmark is the overall operating cost corresponding to setting the energy storage and unit output values determined by reinforcement learning to 0.

14. The reinforcement learning system for source-grid-load-storage collaborative multi-scenario optimization according to claim 11 is characterized in that: In the generation module, the diverse training task set is as follows: Based on daily operation scenarios, the continuous operation data of wind power, photovoltaic power, load and upper-level grid prices at each node are decomposed to form the time series characteristics of operation factors; The decomposed time series features of the operating elements are classified based on the DBSCAN clustering method. Each class contains multiple approximate time series features and is regarded as a task with identical distribution characteristics. The temporal characteristics of the operating elements of each distribution network are matched with their physical nodes to form multiple feature combinations, and each feature combination is defined as a task; each round of reinforcement learning training is performed under the same task. During the reinforcement learning iterative solution process, the operating state corresponding to each moment is determined by the temporal characteristics of the environmental elements at the current moment.

15. The reinforcement learning system for source-grid-load-storage collaborative multi-scenario optimization according to claim 11 is characterized in that: In the training module, the reinforcement learning strategy model adapted to multiple scenarios is obtained as follows: Based on the core elements of reinforcement learning and the training task set, a task distribution P(τ) is constructed. Each task is a Markov decision process, including a set of states, actions and a finite reward function, represented as task τ = {p(s0), p(s t+1 |s t ,a t ),r(s t ,a t )}, p(s0) is the initial state distribution, p(s t+1 |s t ,a t ) is the transfer distribution and r(s t ,a t ) is the reward function; Given a set of training tasks sampled from P(τ), the meta-training process learns a policy that adapts to the current task through the context c; set up is a state transition in task τ, which corresponds to a time sequence state change in power grid optimization. The mission experience that constitutes the present collection; Define the task context feature vector as Z, which is used to characterize the characteristics of the current running scene, and is represented by π in the strategy function. θ Feature embedding is achieved in the form of (a|s,z); By constructing an inference network q φ (z|c) constructs a reinforcement learning value function that minimizes the estimated deviation, φ is the neural network parameter, and at the same time trains q for different scene features in a way that minimizes the amount of feature information. φ (z|c) parameters; The distribution network optimization model is trained using the deep reinforcement learning SAC algorithm, and the reparameterization method is used to calculate the sampling feature vector z versus q φ The gradient of the parameters in (z|c) is used to optimize the inference network q φ Parameters of (z|c), policy network π θ (a|s,z) and evaluation network Parameters; The multi-scene loss function is as follows: in, represents the expectation of task τ, represents the expectation between the sample sets, D KL Represents the KL divergence operator, q φ represents the scene element representation embedded in the neural network, τ represents the number of a task, c τ is the context information of the corresponding task, z is the task context feature vector; g(τ,z) represents the optimization main function served by feature z, which will be designed in detail in the future in combination with the reinforcement learning algorithm; β is the target weight coefficient; p(z) is the unit Gaussian prior distribution; The updated loss function of the evaluation network with scene meta-representation embedding is as follows: Among them, L critic To evaluate the network loss function symbol, represents the evaluation network function, V represents the baseline evaluation network function, s′ represents the next state of migration, Indicates the z vector without gradient update; The loss function of the policy network with scene meta-representation embedding is as follows: Among them, L actor is the symbol of the policy network loss function, D KL (·||·) represents the KL divergence function between two distributions, exp represents the exponential function technique, K θ Represents the state normalization factor.

16. The reinforcement learning system for source-grid-load-storage collaborative multi-scenario optimization according to claim 11 is characterized in that: In the output module, the executable neural network model adopts the Actor-Critic architecture, which includes a policy neural network, an evaluation neural network, and a scene meta-representation neural network; (s n ,a n ,r n ,s′ n ) Inputting the context information fragments composed of four tuples into the scene element representation neural network to obtain the corresponding context fragment feature distribution, and obtaining n distributions for n four tuples; multiplying the n fragment feature distributions by Gaussian distribution products to obtain the scene element representation distribution, and using distribution sampling to obtain the scene element representation one-dimensional tensor; The one-dimensional tensor representing the scene element is concatenated with the action space and state space defined by the content as the input of the evaluation network, and the final output is the value function estimate of the corresponding state-action combination; The input of the policy network is the concatenation of the state space and the one-dimensional tensor representing the scene element, and the output is the value distribution of each variable in the action space, which is used to calculate the optimization strategy and update the policy neural network; The evaluation network, policy network, and scene element representation network adopt a fully connected network or Transformer structure.

17. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions that, when executed by a computing device, cause the computing device to perform the method of any one of claims 1 to 10 .

18. A computing device, characterized in that include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include steps for executing the method according to any one of claims 1 to 10.

Images