Metareinforcement learning method and system oriented to source network load storage collaboration

Abstract

The invention discloses a meta reinforcement learning method and system oriented to source network load storage cooperation, and belongs to the technical field of power system dispatching optimization. The method comprises the following steps: constructing an optimization objective function containing economic cost and network loss cost, defining various constraints, and converting the constraints into a mixed integer second-order cone optimization model through second-order cone relaxation; determining a reinforcement learning action space, a state space and a reward function, and decoupling into a single-section optimization problem; constructing a source load scene and a topological structure task subset, and combining the source load scene and the topological structure task subset to form multiple subtasks; performing model training based on a graph attention neural network, a Transform architecture and an SAC algorithm; interaction-free state space embedding vector calculation is realized through KL divergence constraint; and selecting two meta-test forms to output an optimization decision result. According to the method, the adaptive capacity of the model to the source load and topological change is improved, the strategy safety and economy are ensured, and the method is suitable for novel power system source-network-load-storage collaborative optimization scheduling.

Description

Technical Field

[0001] This invention belongs to the field of power system dispatch optimization technology, specifically involving a meta-reinforcement learning method and system for source-grid-load-storage coordination. Background Technology

[0002] Reinforcement learning aims to optimize decision-making for agents through interactive trial and error. It learns by maximizing cumulative rewards rather than approximating feature labels. Its theory is based on Markov stochastic processes and represents a third learning paradigm independent of supervised and unsupervised learning. Deep reinforcement learning combines reinforcement learning with deep learning, leveraging the feature extraction and function fitting capabilities of deep learning to improve the learning efficiency of reinforcement learning.

[0003] Deep reinforcement learning training comprises four key steps: state perception, action selection, environmental feedback, and model update. In the state perception phase, the agent comprehensively and accurately acquires environmental information, forming the basis for its action selection. In the action selection phase, based on the state perception results, the agent selects and executes an action from the feasible domain according to a certain policy. The policy constructs a mapping from the state space to the action space and is the practical carrier of the reinforcement learning objective. In the environmental feedback phase, after receiving the action executed by the agent, the environment performs state transitions based on the operating rules and simultaneously feeds back a reward value as a quantitative evaluation of the policy function, intuitively reflecting the policy's contribution to achieving the final goal. In the model update phase, the agent trains a deep neural network based on the action trajectories obtained through multiple rounds of state perception, action selection, and environmental feedback, aiming to maximize the cumulative reward. Different model update methods constitute diverse reinforcement learning algorithms. Based on value-based and policy-based update algorithms, major algorithms such as DQN, DDPG, PPO, and SAC have been developed.

[0004] A source-network-load-storage system optimization method based on the Deep Deterministic Policy Gradient Algorithm (DDPG) of reinforcement learning. DDPG is a type of reinforcement learning algorithm. In reinforcement learning, an agent continuously interacts with the surrounding environment, perceives the environment's state, and performs corresponding actions according to a certain policy. The environment, upon receiving an action, provides the agent with a reward and moves to the next state. This process is repeated until an optimal policy is learned, maximizing the cumulative reward. This includes... 1) Sensing environmental state: including power flow status data of elements such as power plants, photovoltaics, wind power, and loads; parameters of the power network include active power, reactive power, voltage, and power factor. 2) Actions: These include load reduction, generator output adjustment, and energy storage charging / discharging adjustment; 3) Rewards: These include generation costs, operating costs, and penalties for exceeding power limits; 4) Agent Training: See the overall training framework below. Figure 1 .

[0005] Existing training techniques employ soft constraints, which cannot guarantee that the policy always meets safety constraints; furthermore, training different scenarios together lacks differentiation of scenario-specific features and the ability to dynamically adapt to different scenarios; and there is no modeling of the impact of topology, resulting in a lack of topology-adaptive capabilities. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a meta-reinforcement learning method and system for source-grid-load-storage coordination, which addresses the shortcomings of the prior art. This method solves the technical problems of poor model generalization ability, difficult policy transfer, and inability to meet real-time security constraints caused by the dynamic changes of source-load scenarios and the adjustment of power grid topology in the optimization of source-grid-load-storage coordination.

[0007] The present invention adopts the following technical solution: A meta-reinforcement learning method for source-network-load-storage coordination includes the following steps: S1. Construct an optimization objective function containing system economic cost and network loss cost, define variable equality constraints, inequality constraints and integer variable constraints, and use second-order cone relaxation technology to perform convex relaxation treatment on power flow nonlinear nonconvex constraints, transforming the original mixed integer nonlinear optimization problem into a solvable mixed integer second-order cone optimization problem, forming a source-grid-load-storage collaborative optimization model; S2. Determine the energy storage charging and discharging power and the unit power as decision variables in the reinforcement learning action space, eliminate strong temporal coupling constraints, and decouple the original problem into a single-section optimization problem; construct a state space containing uncontrollable environment state variables and controllable system state variables based on grid node operation data; construct a reward function containing policy-related rewards and baseline rewards through the solution results of the single-section optimization problem; S3. Based on the state space, action space and reward function obtained in step S2, construct a subset of source-load operation scenario tasks using DBSCAN clustering and data augmentation technology, and construct a subset of topology task tasks based on fixed nodes and different line connections; combine the subset of source-load operation scenario tasks with the subset of topology task tasks to form m×n different subtasks; S4. Based on the m×n different sub-tasks formed in step S3, construct a task distribution and inference network; collect interaction samples of each sub-task, process the graph structure data in the interaction samples through a graph attention neural network, and then use the Transformer architecture to aggregate the state transition features at different times to obtain the context feature distribution; based on the SAC algorithm, fuse the context features and perform parameter update training on the policy network, evaluation network and inference network. S5. Based on the prediction data output by the source-grid-load-storage collaborative optimization model obtained in step S1, construct the task prior features; construct the task representation network using the same neural network structure as in step S4, and align the distribution of the task prior features with the distribution of the context features obtained in step S4 through KL divergence constraints to achieve interactive state space embedding vector calculation. S6. Based on the trained policy network, evaluation network, and inference network obtained in step S4 and the state space embedding vector obtained in step S5, select either the non-interactive meta-testing form or the training-free adaptive meta-testing form based on actual application conditions, and output the source-network-load-storage collaborative optimization decision result.

[0008] Preferably, in step S1, the optimization objective function includes the sum of the equivalent economic costs of all optimizable elements of the system over the entire optimization time period and the equivalent cost of the system power network loss value; the variable equality constraints include KCL constraints, KVL constraints and energy balance constraints; the variable inequality constraints include controllable equipment output limit constraints and controllable equipment power time-series ramping constraints; the integer variable constraints include energy storage device charging and discharging constraints, grid power purchase and sale constraints and load switching behavior constraints.

[0009] Preferably, in step S2, the state space is established based on the grid node operation data. The uncontrollable environmental state quantities include new energy output, load demand and market electricity price. The controllable system state quantities include the energy storage device's state of charge and the unit's current output level. The strategy-related reward in the reward function is obtained by removing the instant reward value corresponding to the baseline strategy from the instant reward.

[0010] Preferably, the strategy-related reward is calculated by the difference between the immediate reward value and the baseline reward value; the immediate reward value satisfies:

[0011] The baseline reward value is:

[0012] in, Represents the instant reward value. This represents a set of node numbers in a distributed unit system. This represents a set of node numbers for an energy storage system. This represents a set of node numbers that meet the requirements of the demand response system. A set representing the node numbers of the power grid purchasing system; Represents a set The elements in Represents a set The elements in Represents a set The elements in Represents a set The elements in Represents a set Elements determined by reinforcement learning; This represents the unit operating cost function. This represents the energy storage depreciation cost function. This represents the load demand response cost function. This represents the interaction cost function of the upper-level power grid. The cost function representing the curtailment of renewable energy sources. This indicates the cost of network loss. P This represents the power of the corresponding device. This represents the variable values ​​determined by reinforcement learning; SoC Indicates the percentage of state of charge. Indicates the current time; Represents the baseline reward value. This represents the energy storage and generator output values ​​in the baseline strategy.

[0013] Preferably, in step S3, the construction of the source load running scenario task subset specifically includes: Using daily operation scenarios as boundaries, the continuous operation data of wind power, photovoltaics, load and upstream grid price at each node are decomposed to form time series features. The DBSCAN clustering method is used for classification, and data augmentation technology is used to augment the data of tasks with fewer samples. The construction of the topology task subset specifically includes: Various topologies are constructed based on fixed nodes and different line connections. Each topology is regarded as a subtask, and it is guaranteed that all nodes are connected by edges and the number of connected edges is the total number of nodes minus one.

[0014] Preferably, in step S4, the inference network is trained by minimizing the estimation bias of the reinforcement learning value function and the amount of feature information; the graph attention neural network processes the graph structure data in the interaction samples and performs tensor merging on the node features of each graph according to the number to obtain state transition features; the Transformer architecture aggregates the state transition features at different times to obtain the context feature distribution.

[0015] Preferably, when training for parameter updates based on the SAC algorithm, the network update loss function is used to evaluate the network as follows:

[0016] in, To evaluate the sign of the network loss function, Indicates the evaluation of the network function, Indicates the baseline evaluation network function. Indicates the next state after the transition. This represents the z-vector that is not updated using gradients. The policy network loss function is:

[0017] in, Let the symbol be the loss function of the policy network. The KL divergence function represents the distance between two distributions. Techniques for representing exponential functions, This represents the state normalization factor.

[0018] Preferably, in step S5, the task representation network is constructed based on the sample data provided by the optimization model, uses GNN to process the physical parameter features of the power grid on a single time segment, and uses the Transformer structure to fuse features from multiple time segments to form a prior context feature distribution; the prior feature distribution is aligned with the context feature distribution through KL divergence constraints, specifically by minimizing the KL divergence between the two types of distributions.

[0019] Preferably, in step S6, the non-interactive meta-testing method specifically includes: based on the temporal features of scene topology state and node power prediction, matching with tasks in the training task set, obtaining offline task representation samples and initializing task meta-representation vectors, constructing a state space in combination with current actual running data, inputting the policy network to output temporal decisions; the training-free adaptation meta-testing method based on actual information specifically includes: on the basis of non-interactive meta-testing, adding the temporal trajectory of the decision to the context sample set, and applying actual interaction information through the context meta-representation model to complete scene adaptation.

[0020] Secondly, embodiments of the present invention provide a meta-reinforcement learning system for source-network-load-storage coordination, comprising: The modeling module constructs an optimization objective function containing system economic cost and network loss cost, defines variable equality constraints, inequality constraints and integer variable constraints, and uses second-order cone relaxation technology to perform convex relaxation treatment on power flow nonlinear nonconvex constraints, transforming the original mixed integer nonlinear optimization problem into a solvable mixed integer second-order cone optimization problem, forming a source-grid-load-storage collaborative optimization model; The learning module identifies energy storage charging and discharging power and unit power as decision variables in the reinforcement learning action space, eliminates strong temporal coupling constraints, and decouples the original problem into a single-section optimization problem; it constructs a state space containing uncontrollable environmental state variables and controllable system state variables based on grid node operation data; and it constructs a reward function containing policy-related rewards and baseline rewards through the solution results of the single-section optimization problem. The task module, based on the state space, action space, and reward function obtained by the learning module, constructs a subset of source-load operation scenario tasks using DBSCAN clustering and data augmentation technology, and constructs a subset of topology task tasks based on fixed nodes and different line connections; the subset of source-load operation scenario tasks and the subset of topology task tasks are combined to form m×n different subtasks; The training module constructs a task distribution and inference network based on m×n different sub-tasks formed by the task module; it collects interaction samples of each sub-task, processes the graph structure data in the interaction samples through a graph attention neural network, and then uses the Transformer architecture to aggregate state transition features at different times to obtain the context feature distribution; based on the SAC algorithm, it fuses the context features and performs parameter update training on the policy network, evaluation network, and inference network. The representation module constructs task prior features based on the prediction data output by the source-grid-load-storage collaborative optimization model obtained from the modeling module. It uses the same neural network structure as the training module to construct the task representation network, and uses KL divergence constraints to align the distribution of the task prior features with the distribution of the context features obtained from the training module, thereby realizing interactive state space embedding vector calculation. The output module, based on the trained policy network, evaluation network, and inference network obtained by the training module and the state space embedding vector obtained by the representation module, selects either the non-interactive meta-testing form or the training-free adaptable meta-testing form based on actual application conditions, and outputs the source-network-load-storage collaborative optimization decision results.

[0021] Preferably, in the modeling module, the optimization objective function includes the sum of the equivalent economic costs of all optimizable elements of the system over the entire optimization time period and the equivalent cost of the system power network loss value; the variable equality constraints include KCL constraints, KVL constraints, and energy balance constraints; the variable inequality constraints include controllable equipment output limit constraints and controllable equipment power time-series ramping constraints; the integer variable constraints include energy storage device charging and discharging constraints, grid power purchase and sale constraints, and load switching behavior constraints.

[0022] Preferably, in the learning module, the state space is established based on the grid node operation data, the uncontrollable environment state quantities include new energy output, load demand and market electricity price, and the controllable system state quantities include the energy storage device's state of charge and the unit's current output level; the strategy-related reward in the reward function is obtained by removing the immediate reward value corresponding to the baseline strategy from the immediate reward. The strategy-related reward is calculated by the difference between the immediate reward value and the baseline reward value; the immediate reward value satisfies:

[0023] The baseline reward value is:

[0024] in, Represents the instant reward value. This represents a set of node numbers in a distributed unit system. This represents a set of node numbers for an energy storage system. This represents a set of node numbers that meet the requirements of the demand response system. A set representing the node numbers of the power grid purchasing system; Represents a set The elements in Represents a set The elements in Represents a set The elements in Represents a set The elements in Represents a set Elements determined by reinforcement learning; This represents the unit operating cost function. This represents the energy storage depreciation cost function. This represents the load demand response cost function. This represents the interaction cost function of the upper-level power grid. The cost function representing the curtailment of renewable energy sources. This indicates the cost of network loss. P This represents the power of the corresponding device. This represents the variable values ​​determined by reinforcement learning; SoC Indicates the percentage of state of charge. Indicates the current time; Represents the baseline reward value. This represents the energy storage and generator output values ​​in the baseline strategy.

[0025] Preferably, in the task module, the construction of the source load running scenario task subset specifically includes: Using daily operation scenarios as boundaries, the continuous operation data of wind power, photovoltaics, load and upstream grid price at each node are decomposed to form time series features. The DBSCAN clustering method is used for classification, and data augmentation technology is used to augment the data of tasks with fewer samples. The construction of the topology task subset specifically includes: Various topologies are constructed based on fixed nodes and different line connections. Each topology is regarded as a subtask, and it is guaranteed that all nodes are connected by edges and the number of connected edges is the total number of nodes minus one.

[0026] Preferably, in the training module, the inference network is trained by minimizing the estimation bias of the reinforcement learning value function and the amount of feature information; the graph attention neural network processes the graph structure data in the interaction samples and performs tensor merging on the node features of each graph according to the number to obtain state transition features; the Transformer architecture aggregates the state transition features at different times to obtain the context feature distribution. When training for parameter updates based on the SAC algorithm, the update loss function for evaluating the network is:

[0027] in, To evaluate the sign of the network loss function, Indicates the evaluation of the network function, Indicates the baseline evaluation network function. Indicates the next state after the transition. This represents the z-vector that is not updated using gradients. The policy network loss function is:

[0028] in, Let the symbol be the loss function of the policy network. The KL divergence function represents the distance between two distributions. Techniques for representing exponential functions, This represents the state normalization factor.

[0029] Preferably, in the representation module, the task representation network is constructed based on the sample data provided by the optimization model, uses GNN to process the physical parameter features of the power grid on a single time segment, and uses the Transformer structure to fuse features from multiple time segments to form a prior context feature distribution; the prior feature distribution is aligned with the context feature distribution through KL divergence constraints, specifically by minimizing the KL divergence between the two types of distributions.

[0030] Preferably, in the output module, the non-interactive meta-testing method specifically includes: based on the temporal features of scene topology state and node power prediction, matching with tasks in the training task set, obtaining offline task representation samples and initializing task meta-representation vectors, constructing a state space in combination with current actual running data, inputting the policy network to output temporal decisions; the training-free adaptation meta-testing method based on actual information specifically includes: on the basis of non-interactive meta-testing, adding the temporal trajectory of the decision to the context sample set, and applying actual interaction information through the context meta-representation model to complete scene adaptation.

[0031] Thirdly, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the above-described meta-reinforcement learning method for source-network-load-storage coordination.

[0032] Fourthly, embodiments of the present invention provide a computer-readable storage medium including a computer program, which, when executed by a processor, implements the steps of the above-described meta-reinforcement learning method for source-network-load-storage coordination.

[0033] Fifthly, a chip includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the aforementioned meta-reinforcement learning method for source-network-load-storage collaboration.

[0034] In a sixth aspect, embodiments of the present invention provide an electronic device, including a computer program, which, when executed by the electronic device, implements the steps of the above-described meta-reinforcement learning method for source-network-load-storage coordination.

[0035] Compared with the prior art, the present invention has at least the following beneficial effects: A meta-reinforcement learning method for source-grid-load-storage coordination is proposed. This method addresses the challenge of existing reinforcement learning models struggling to adapt to source-load variations and topological differences in source-grid-load-storage scenarios by constructing an optimization model, setting reinforcement learning elements, building a multi-task training set, meta-training and task representation, and ultimately achieving adaptive decision-making. The method transforms the complex optimization problem into a solvable form using second-order cone relaxation techniques, ensuring the hard satisfaction of safety constraints. It combines DBSCAN clustering and data augmentation to construct multi-scenario, multi-topology task sets, endowing the model with dynamic adaptive capabilities. Temporal and graph structure features are extracted using graph attention neural networks and the Transformer architecture, coupled with the SAC algorithm to improve training stability and sample efficiency. KL divergence constraints enable non-interactive embedding vector computation, and two meta-testing methods adapt to different practical application scenarios, significantly improving model transferability and practicality.

[0036] Furthermore, the distributed generation unit operating cost adopts a quadratic model, the energy storage loss cost is calculated based on changes in state of charge, and the demand response cost uses a linear model, closely reflecting the actual operating characteristics of the equipment and ensuring the scientific nature of cost accounting. The grid loss cost is calculated based on the product of branch resistance and the square of the current amplitude, accurately reflecting the grid loss situation. This scheme avoids ambiguity in cost calculation, makes the optimization objective more operable, provides accurate cost data support for the subsequent construction of the reward function for reinforcement learning, and improves the economy and rationality of collaborative optimization decision-making.

[0037] Furthermore, the convex constraint form after second-order cone relaxation is clarified, overcoming the deficiency in existing technologies where soft constraints cannot guarantee strategy safety. Variable equality constraints, inequality constraints, and integer variable constraints comprehensively cover the core safety boundaries of power grid operation, ensuring that decisions satisfy physical laws and equipment operating limitations. By transforming the nonlinear, nonconvex constraints of power flow into convex constraints through second-order cone relaxation, the solvability of the model is guaranteed, and relaxation errors are reduced through network loss parameter regularization. This scheme makes the constraint system of the optimization model more complete and explicit, providing a solid guarantee for strategy safety and avoiding power grid operation risks caused by missing constraints.

[0038] Furthermore, the uncontrollable environmental state variables encompass key external factors such as renewable energy output and load demand, while the controllable system state variables include core controllable parameters such as energy storage state of charge and unit output, comprehensively covering all variables affecting decision-making. The mathematical expressions of the state space make feature quantification more standardized, providing structured data support for the state perception of the reinforcement learning agent. This scheme avoids the omission or redundancy of state features, improves the agent's accuracy in perceiving the environment, lays the foundation for the accuracy of subsequent action decisions, and enhances the robustness of the model.

[0039] Furthermore, by using the difference between the immediate reward value and the baseline reward value, costs unaffected by the strategy are eliminated, making the reward feedback more accurately reflect the strategy's merits. The immediate reward value encompasses various costs such as unit operation, energy storage depreciation, and network loss converted costs, comprehensively considering the economic and technical benefits of the decision. The baseline reward value is set as the total operating cost when the reinforcement learning decision variable is 0, providing a reasonable benchmark for reward calculation. This scheme solves the problem that existing reward functions are difficult to accurately evaluate the strategy's contribution, making the training direction of reinforcement learning clearer, accelerating the convergence of the optimal strategy, and improving the model's decision optimization capability.

[0040] Furthermore, the time-series data is decomposed on a daily basis, forming features with 96 or more values, aligning with the actual needs of the power grid's 15-minute-level dispatch cycle. DBSCAN clustering categorizes similar scenarios, while GAN or proportional residual techniques amplify tasks with fewer samples, avoiding the one-sidedness of scenario features. This approach enables the training task set to fully cover different time-varying source-load patterns, improving the model's adaptability to changes in source-load scenarios and providing ample training data support for the cross-scenario generalization of meta-reinforcement learning.

[0041] Furthermore, the construction rules for the topology task subset were clarified, taking into account both normal and abnormal operating states, thus enriching the diversity of topology scenarios. Multiple topology scenarios are constructed based on fixed nodes and connections to different lines. During normal operation, a radial structure is maintained, while an island structure is formed during abnormal operation, closely reflecting the topology changes in actual power grid operation. A directed branching tree is maintained to adapt the distribution network branch power flow structure, ensuring that the topology model is consistent with the physical structure of the power grid. This scheme solves the problem of the lack of topology adaptive capability in existing technologies, enabling the training task set to contain diverse topology features, improving the model's transfer and adaptation capability after power grid topology changes, and broadening the model's application scope.

[0042] Furthermore, specific loss functions for the evaluation network and policy network are given, providing a clear optimization direction for model parameter updates. The evaluation network loss function is calculated using the deviation between the true and predicted values, ensuring the accuracy of the value function estimation. The policy network loss function introduces KL divergence constraints, balancing the relationship between policy exploration and exploitation, and preventing the policy from converging to local optima. This approach makes the parameter updates during model training more scientific and feasible, improves training stability and sample efficiency, ensures that the model can learn better collaborative optimization strategies, and enhances the model's decision-making performance.

[0043] Furthermore, the specific execution flow of the two meta-testing formats is refined, making the decision output more targeted for different practical application scenarios. The non-interactive meta-testing format, based on predictive temporal feature matching training tasks, achieves rapid temporal decision-making, meeting real-time requirements. The training-free adaptation meta-testing format improves scenario adaptation accuracy by incorporating actual decision trajectory correction features, solving the problem of discrepancies between predictions and actual scenarios. This approach enables the model to flexibly adapt to application scenarios with varying levels of information completeness, ensuring both real-time decision-making efficiency and improving decision-making accuracy in complex scenarios, further enhancing the model's practicality and applicability.

[0044] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0045] In summary, this invention, through a meta-reinforcement learning framework, enables rapid adaptive optimization of the source-network-load-storage system under multiple scenarios and topologies, significantly improving decision-making efficiency and system adaptability, and overcoming the problems of poor generalization ability and reliance on large amounts of interactive data in traditional methods.

[0046] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of a deep reinforcement learning algorithm model; Figure 2 For multi-topology scenarios, a fixed node and different line connection diagrams are used; Figure 3 This is a schematic diagram of a graph attention neural network; Figure 4 A diagram illustrating the interaction logic between various cached data items; Figure 5 This is the cumulative reward curve of the model on different tasks during the training process of the method of the present invention; Figure 6 The convergence process of the context feature encoding z in four different source load scenarios during the algorithm training process is shown in the figure. Figure 7 This is a flowchart of the method of the present invention. Detailed Implementation

[0048] This invention provides a meta-reinforcement learning method for source-grid-load-storage coordination, supplementing optimal power flow to ensure policy security. It transforms an unstable problem into a set of multiple stable tasks, utilizing model-free reinforcement learning for training in each sub-task space. By learning historical interaction data features for each sub-task set, a representation vector describing the features of that sub-task is constructed, serving as an additional learnable embedding structure in the reinforcement learning state space to support multi-task description, thereby achieving dynamic adaptation of multiple source-load boundaries and multiple topologies. Furthermore, prior features of sub-tasks are constructed using global scenario prediction information. By comparative learning, the prior features are approximated with the sub-task vector features, achieving interaction-free state space embedding vector computation. The trained model can be directly applied to online source-grid-load-storage optimization scenarios, allowing for the selection of interactive or interaction-free application schemes based on actual application conditions.

[0049] Please see Figure 7 This invention discloses a meta-reinforcement learning method for source-network-load-storage collaboration, comprising the following steps: S1. Distribution network optimization modeling and model transformation; To train an agent using reinforcement learning algorithms for source-grid-load-storage coordinated decision-making, an interactive simulation environment needs to be constructed. After inputting the current state information and action strategy into the simulation environment, the environment will provide a corresponding reward value and the state information for the next moment. Typically, this simulation environment is based on power flow calculation methods. Once all node information of the power grid is provided by the agent or determined by the external environment, power flow calculation provides indicators such as grid operational stability, economy, and reliability as reward feedback. However, as the power grid scales up and the number of controllable factors increases, it becomes difficult to quickly find feasible strategies that meet complex and diverse security constraints through learning and exploration. Therefore, this invention proposes a source-grid-load-storage coordinated reinforcement learning decision-making modeling method based on optimal power flow calculation. The specific steps are as follows: The objectives of distribution network optimization include: 1) the sum of the equivalent economic costs of all optimizable elements of the system over the entire optimization period; and 2) the equivalent cost of system power losses, expressed as: (1) (2) (3) (4) (5) (6) (7) in, C This represents a function for distributed generation, energy storage, demand response, and grid purchase costs. P This represents the power of the corresponding device. This represents the set of system node numbers corresponding to the device. This represents the elements within the set. The DG operating cost uses a quadratic model. These correspond to second-order, first-order, and constant coefficients, respectively. Calculation based on changes in the state of charge of energy storage SoC Indicates the percentage of state of charge. This represents the loss factor. Demand response cost, curtailment cost, and upstream grid purchase cost are all represented using a linear model. Each corresponds to its cost coefficient.

[0050] The objective function is to minimize the cumulative running cost, where T represents the total optimization time and t represents the current time step. This indicates the time interval, i.e., the optimization step size. (For network loss parameters...) , This represents the target coefficient for network loss. Let be the resistance of branch ij. Let be the square of the current amplitude flowing through branch ij.

[0051] Variable equality constraints include KCL constraints, KVL constraints, energy balance constraints, etc.; variable inequality constraints include controllable equipment output limit constraints, controllable equipment power timing ramping constraints, etc.; integer variable constraints are constraints on discontinuous variables with special physical meaning, including energy storage equipment charging and discharging constraints, grid power purchase and sale constraints, load switching behavior constraints, etc. For distributed units, in addition to rated power constraints and active power limit constraints, ramping constraints for unit regulation are also considered, as shown in equations (8) and (9).

[0052] (8) (9) in, These represent the upper and lower bounds of active power, respectively. This indicates the proportion of the maximum change in active power to the upper bound. This indicates the rated power of the unit. New energy equipment is modeled using a constant power factor model, where active power equals the theoretical maximum output minus the amount of waste, and reactive power equals the theoretical output multiplied by the power factor.

[0053] (10) in, The theoretical upper bound of the force output of new energy equipment at time t. This is the constant power factor coefficient. The upstream grid interconnection line mainly handles the purchase of electricity from external sources. Since the output of new energy sources can be adjusted, the model limits its ability to transmit electricity to the upstream grid.

[0054] (11) in, Indicates the upper limit of the purchase volume. This represents the upper and lower bounds of reactive power interaction with the upstream power grid. For energy storage device constraints, auxiliary integer variables need to be constructed to ensure that energy storage devices do not charge and discharge simultaneously.

[0055] (12) (13) in, These represent the upper limits of energy storage charging and discharging power. To assist 0-1 variables, These are the energy storage charging and discharging efficiencies, The total capacity of the energy storage equipment The upper and lower bounds of the energy storage state of charge are defined. The load demand response constraint is shown in the equation, which sets the amount of load that can participate in regulation at any given time to always be a fixed proportion of the current load demand.

[0056] (14) in, This is the upper bound of the demand response load. This is an adjustable scaling factor. These represent the current load demand values. Distribution network power flow constraints are often constructed using branch power flow balance relaxation constraints, as shown in equations (15) and (16) below.

[0057] (15) (16) in, This represents the square of the branch current magnitude. This represents the square of the node voltage magnitude. These represent the branch resistance and reactance, respectively. It represents the sum of active and reactive power of all elements connected to node j. Since equation (15) does not include the power balance constraint of the first node, i.e. the upper-level tie line, an additional equality constraint is set to ensure that the external power purchase is equal to the sum of the power of the extended branches of the first node.

[0058] Generally, the entire optimization problem belongs to the mixed-integer nonlinear optimization problem, which needs to be linearized or transformed into a solvable problem form through convex relaxation. The main factor that makes the original model unsolvable is the nonlinear and nonconvex constraint of the power grid flow. A second-order cone relaxation technique is used to perform convex relaxation on it. In this patent, the energy balance constraint is handled by the second-order cone relaxation constraint as shown in equations (17) and (18). After processing, the problem is transformed into a mixed-integer second-order cone optimization problem, which can be solved by a solver to obtain the optimal solution.

[0059] (17) (18) in, Let be the square of the current magnitude corresponding to branch ij. The square of the voltage magnitude at node i. Let be the active and reactive power flowing through branch ij. Since the constraint form of equation (17) is non-convex, the convex constraint form corresponding to equation (18) can be obtained by using second-order cone relaxation. Introducing the network loss parameter into the target is a regularization of the convex relaxation of equation (18), so as to minimize the energy balance error caused by relaxation. Especially when the power generation of new energy exceeds the total system load, it prevents the power generation from increasing indefinitely. This satisfies the slack constraints, enabling the optimization of new energy output to function effectively.

[0060] S2. Set up reinforcement learning to accelerate the optimization solution logic and its state space and decision variables; 1) Motion space settings Reinforcement learning is suitable for handling time-series decision problems. When there are strong time-series constraints in the optimization model, the corresponding main variables can be used as reinforcement learning decision variables. In the source-grid-load-storage coordinated optimization problem, the charging and discharging power of the energy storage unit and the power of the unit with a small ramp-up capability are suitable variables for reinforcement learning decision, and the relevant constraints are shown in equations (19) and (20). At this time, for the original optimization model, the variables in the action space are regarded as deterministic values, which greatly reduces the complexity of the original model.

[0061] (19) (20) In equation (19), the energy storage charging and discharging power Together they determine the external apparent power of energy storage Integer variables involving charging and discharging behavior Meanwhile, the energy storage state of charge SoC expressed by equation (20) is a strongly time-coupled equality constraint. This is the charge / discharge efficiency coefficient. When the external apparent power... After being determined by the neural network, all the above constraints will be removed from the optimization problem, greatly simplifying the constraint form. Furthermore, the power constraints corresponding to the unit power determined by reinforcement learning are also processed in the same way, corresponding to equations (21) and (22).

[0062] (twenty one) (twenty two) in, These represent the upper and lower bounds of active power, respectively. This indicates the proportion of the maximum change in active power to the upper bound. This indicates the rated power of the generator unit. This refers to the set of generator sets whose power values ​​are determined based on reinforcement learning.

[0063] After determining the reinforcement learning decision variables, since most of the strongly temporally coupled constraints are eliminated, the temporal correlation of the entire optimization problem is weakened, and it can be transformed into multiple single-section optimization problems through decoupling methods. The variable definition of the single-section optimization problem does not include the time variable t, that is, all optimization variables are limited to a single time. The objective function of the original problem also needs to be split. The objective of the single-section optimization problem at time t is the sum of the equivalent economic costs of all optimizable elements of the system during the optimization period. Some weakly temporally coupled constraints can be decoupled in a greedy manner, as shown in equation (23), where Let be the unit power at time t. This is the ramp-up coefficient. The unit's power ramp-up capability is greater than the external load variation capability. In this case, the upper and lower bound constraints of the power at time t can be considered as the power at time t-1 plus the upper and lower bounds of the ramp-up variation rate.

[0064] (twenty three) in, This refers to the set of units whose power values ​​are determined based on optimization solutions.

[0065] 2) State space settings Based on the above problem transformation method, the state characteristics of the problem can be constructed and a basic decision framework can be built. At time t of the decision, the cross-sectional parameters at time t are extracted to form a single cross-sectional optimization problem. At the same time, the parameter information at time t and its historical times (t-1, t-2, ...) is used to construct the reinforcement learning decision state space. The state space is established based on the grid node operation data, where the graph node state is mainly divided into uncontrollable environment state variables and controllable state variables within the system. Among them, uncontrollable environment state variables include new energy output, load demand, market electricity price, etc., and controllable state variables within the system include energy storage equipment charge status, current unit output level, etc. It can be expressed as equation (24): (twenty four) in, This represents the power of new energy sources at each node of the system. This represents the load on each node of the system. This indicates the current electricity price. This represents the sum of the active power of the units at each node. This represents the sum of reactive power at each node of the system. This represents the sum of the demand response power at each node of the system. This indicates the energy storage charge state at each node of the system.

[0066] Neural networks are used to establish a functional mapping relationship between the state space and the action space, that is, to calculate specific action values ​​through specific action spaces. Specific model design will be discussed in more detail in conjunction with reinforcement learning algorithms.

[0067] 3) Reward function settings Assigning specific action space values ​​to a single-section optimization problem and solving this optimization problem (using a commercial solver), as shown in the equation, yields the decision objective function at time t, which determines the minimum cost under the premise of energy storage power, as well as the power values ​​of other factors to be solved.

[0068] (25) This minimum value reflects the immediate cost of the policy at time t and is the basis for constructing the reward function. Considering The value of has a portion unaffected by the strategy, causing deviations in the strategy's payoff at each time step, thus impacting strategy evaluation. Let's define an immediate reward. The part affected by the strategy and the part not affected by the strategy are represented as shown in equation (26).

[0069] (26) The immediate reward value corresponding to the baseline policy is removed from the immediate reward, resulting in a policy-strongly correlated reward feedback, which is used for reinforcement learning training. The immediate reward value includes: (27) in, Represents the instant reward value. This represents a set of node numbers in a distributed unit system. This represents a set of node numbers for an energy storage system. This represents a set of node numbers that meet the requirements of the demand response system. A set representing the node numbers of the power grid purchasing system; Represents a set The elements in Represents a set The elements in Represents a set The elements in Represents a set The elements in Represents a set Elements determined by reinforcement learning; This represents the unit operating cost function. This represents the energy storage depreciation cost function. This represents the load demand response cost function. This represents the interaction cost function of the upper-level power grid. The cost function representing the curtailment of renewable energy sources. This indicates the cost of network loss. P This represents the power of the corresponding device. This represents the variable values ​​determined by reinforcement learning; SoC Indicates the percentage of state of charge. Indicates the current time .

[0070] The baseline reward value includes: (28) in, Represents the baseline reward value. This represents the energy storage and generator output values ​​in the baseline strategy. The baseline reward setting benchmark is the total operating cost corresponding to setting the energy storage and generator output values ​​determined by reinforcement learning to 0.

[0071] Finally, the decision variables of the reinforcement learning action space are integrated with the decision variable values ​​obtained from the optimization solution to obtain the complete decision result for the current time segment t. This process is repeated until the final time segment T is reached, thus completing the decision for the entire optimization problem.

[0072] S3. Set up the training task set The training task set is divided into two parts, which are constructed for the source load running scenario and the running topology respectively.

[0073] In the source-load operation scenario: The state space contains numerous policy-independent features, each exhibiting time-dependent properties, forming different time-varying patterns. Certain elements may also exhibit different patterns simultaneously, such as renewable energy sources and loads displaying different behaviors at different grid nodes. Under these different time-varying patterns, the distribution network optimization operation strategy will also change. To improve the adaptability of reinforcement learning under different time-varying patterns, it is necessary to construct a training task set to support model training.

[0074] The training task set's main temporal features are clustered and augmented using specific methods. Taking daily operational scenarios as boundaries, the continuous operational data (monthly, quarterly, annually, etc.) of wind power, photovoltaics, load, and upstream grid prices at each node are decomposed to form temporal features with 96 (or more) values ​​(one data point every 15 minutes, 96 data points per day). These decomposed temporal features are then classified using the DBSCAN clustering method. Each cluster contains multiple approximate temporal features, considered as a task with identical distribution characteristics. Data augmentation techniques (GAN, proportional residuals, etc.) are used to augment tasks with fewer samples, enriching the scenario modeling description.

[0075] The temporal characteristics of each power distribution network operation element are mapped to its physical nodes, forming multiple feature combinations. Each feature combination is defined as a task. Each round of reinforcement learning training needs to be executed under the same task. During the reinforcement learning iterative solution process, the operating state at each time step is determined by the temporal characteristics of the environmental elements at the current time step.

[0076] Topology Structure: The multi-topology scenarios addressed in this invention are based on fixed nodes and different line connections, such as... Figure 2As shown, solid lines represent connected lines, and dashed lines represent disconnected lines. Each topology is considered a subtask. In practical applications, a region whose topology remains unchanged can be considered a node, and the scalable lines between multiple regions can be considered edges, thus obtaining a similar heterogeneous topology set. Under normal operating conditions, considering the radial constraints of the distribution network, when constructing the topology, it is necessary to ensure that all nodes are connected by edges, and the number of connected edges is the total number of nodes minus one. Under abnormal operating conditions, the number of connected edges can be less than the total number of nodes minus one, thus forming an island structure and enriching the topology. Since the signed distribution network optimization model adapts to the branching power flow structure, a directed graph with the balancing node as the root node and the end power saving node as the leaf node needs to be maintained in the multi-topology set, forming a directed branching tree.

[0077] Based on the task division of source-load operation scenarios and topology structures, specific sub-tasks are formed by combining different operation scenarios and topology structures. Generally speaking, if there are m different identically distributed source-load scenarios and n different topology structures, then m*n different sub-tasks can be formed.

[0078] S4, reinforcement learning training algorithm; Based on a training task set, this invention proposes a meta-reinforcement learning training method to achieve effective learning of reinforcement learning in different scenarios. Although meta-reinforcement learning algorithms allow agents to learn new skills from limited experience, several major challenges hinder their practicality. Current methods rely too heavily on online policy experience, limiting their sample efficiency. They also lack mechanisms for inferring task uncertainty when adapting to new tasks, limiting their effectiveness in sparse reward problems. In this patent, we address these challenges by developing an offline policy meta-reinforcement learning algorithm that separates task inference and control.

[0079] Based on the core elements of reinforcement learning constructed from the prior content and the training task set, a task distribution is built. Each task is a Markov decision process, consisting of a set of states, actions, and a finite reward function, denoted as a task. ,in For the initial state distribution, For the transition distribution and Let be the reward function. It's important to note that this problem definition includes task distributions with different transition functions and different reward functions. Given from For a set of training tasks sampled in the middle, the meta-training process learns a policy, which is learned through context. To adapt to the current task. (Set) For the task A state transition in the process corresponds to a time-series state change in power grid optimization. This constitutes the task experience currently collected and forms the basis for extracting potential contextual features of the task.

[0080] Task context feature vector is defined as Used to characterize the features of the current running scenario, in the policy function as Feature embedding is implemented in the form of [formula / method]. Because [details / method] Incorporating the experience gathered so far, a reasoning network is constructed. Among them, For the neural network parameters, a method is constructed to minimize the estimation bias of the reinforcement learning value function while simultaneously minimizing the feature information content, thus training a neural network suitable for features in different scenarios. Parameters. Its optimization objective can be described in the form of equation (29): (29) in, A number representing a specific task. This is the context information for the corresponding task. This is the task context feature vector; Representative characteristics The optimized main function will be designed in detail later in conjunction with reinforcement learning algorithms; The target weight coefficient; The prior distribution is a unit Gaussian distribution; here, the standard normal distribution function is used.

[0081] function It consists of multiple state transition sections The aggregate characteristics constituted are expressed as equation (30): (30) in, This represents the aggregation function that integrates features from the 1st to the mth features. This indicates that the nth state transition sample is input into the inference network. The resulting output distribution. The instability of the source-grid-load-storage coordination problem stems from the temporal changes in environmental states. When constructing the context feature distribution, state transition samples cannot be assumed to be independent of each other. Instead, the temporal relative positions of state transitions must be comprehensively considered to recover the temporal characteristic description and achieve the extraction of context features.

[0082] Reinforcement learning training for source-grid-load-storage collaborative optimization typically employs a fixed number of steps to constrain the training process. Let the maximum number of interaction steps be... The initial state of each state transition The relative time is Then obtained through random sampling from the cache The time distribution set of network input samples is .in, Number the trajectory samples. This represents the sampling batch size. The cumulative amount of sample cache increases with the number of interactions. It will be much larger than the size of the sampling batch. The probability of obtaining a sample set covering all interaction trajectories at all times through random sampling is significantly reduced, making it impossible to present a complete picture of the temporal characteristics of the current source-grid-load-storage collaborative task environment. Therefore, a relative time t is introduced into the sample buffer, i.e. And during sampling, a limited number of samples with no repetition at relative time t are selected, until the sample size reaches the upper limit. Then, random sampling is used to fill the sample gaps. Based on this sampling method, a task representation is constructed. Neural network structures such as Figure 3 As shown, by introducing a graph attention mechanism and a Transformer architecture to process the temporal correlation features of the source-network-load-storage graph structure domain, mean pooling is used to achieve symmetry in model input exchange and reduce representation learning complexity, ultimately obtaining the corresponding context feature distribution.

[0083] Please see Figure 3 ,Will The context information fragment composed of interactive sample quadruples is input into the feature extraction neural network, where each state... The graph structure is sampled from the task set. A graph neural network (graph attention neural network is used in the illustrated example, but the actual structure is not limited) is used to process the graph structure data in each set of interaction samples. The node features of each graph are merged into tensors according to their numbers. The merged features represent the state transition features at the corresponding sample time t. Thus, n quadruplets will yield n state transition features. Then, a Transformer structure is applied to aggregate the state transition features at different times t to obtain the context feature distribution representing the global features of the task. The one-dimensional tensor of the global context features is obtained by sampling from the distribution.

[0084] Based on a one-dimensional context tensor, a distribution network optimization model is trained using the deep reinforcement learning SAC algorithm. The SAC algorithm demonstrates good sample efficiency and stability, and can be well integrated with latent context features. A reparameterization method is used to compute the sampled feature vector z pairs. The gradient of the parameters is used to optimize the inference network. Parameters and policy networks and evaluation network The parameters. Where s is related to equation (x). The same applies. The update loss function for evaluating the network is shown in equation (31): (31) in, To evaluate the sign of the network loss function, Indicates the evaluation of the network function, Indicates the baseline evaluation network function. Indicates the next state after the transition. This represents the z vector that is not updated with gradients.

[0085] The policy network loss function is shown in equation (32): (32) in, Let the symbol be the loss function of the policy network. The KL divergence function represents the distance between two distributions. Techniques for representing exponential functions, This represents the state normalization factor.

[0086] Based on the context inference loss function defined in equation (31), since the context features are mainly used for value function prediction, the function... It can be replaced by the evaluation loss function defined by equation (33), expressed as: (33) S5. A strategy training method for non-interactive task representations supported by meta-reinforcement learning.

[0087] Contextual feature representation methods based on cached sampling imply that exploratory interactions must be performed during the meta-testing phase to obtain the model's basic cognitive ability regarding the environment. Without actual exploratory data input, the contextual feature z can only be obtained by sampling from the estimated distribution, i.e., the standard normal distribution. In the absence of any prior knowledge of environmental information, especially when the inherent physical information of the environment cannot be accurately estimated, obtaining environmental features through exploration is almost the only task identification method. However, in the context of the source-grid-load-storage coordination problem in new power systems, the decision-making environment is not entirely unknown. Controllable operational elements have complete model parameters, the main structure of the power grid topology can be accurately understood, and node operating power can be reliably modeled through high-precision short-term predictions. Although this information may differ from the posterior real data, it is sufficient to construct a concrete description of the future decision-making environment of the system.

[0088] Specifically, the source-grid-load-storage collaborative optimization model constructed in the first part includes the controllability of equipment and the safety and stability boundaries of the system, accurately characterizing the feasible range of the strategy; it also includes the power boundaries of the operating scenarios across T consecutive time segments, effectively describing the environmental transfer properties. The scenario power boundaries at this point are based on prior values ​​obtained through prediction, ensuring high accuracy on short-term time scales. When constructing the task set as described above, the clustering center curve of the source-load scenario is used as the task representation benchmark (it can be based on actual prediction data, without specific restrictions here).

[0089] Let the task representation neural network used to optimize model transformation be represented as follows: The sample data provided by the optimization model is represented as A neural network structure similar to that described above is used to construct... First, GNN is used to process the physical parameter features of the power grid at a single time point. Then, referring to the aforementioned meta-knowledge construction method, the Transformer structure is used to fuse features from multiple time points to form the context features corresponding to the model. The difference is that the samples obtained by optimizing the model transformation cover the entire decision range, and there is no need to consider the sample caching sampling strategy.

[0090] Regarding training methods, neural networks and Maintaining consistency, network parameters can be optimized using a loss function. Theoretically, this can be achieved by utilizing... direct replacement Its role in state embedding, but considering The input is a fixed sequence of features, ignoring the difference between the actual scenario and the expectation, while The network input includes real-world scenario information from the testing phase, giving it a stronger ability to represent the true characteristics of the environment. Because... and The network outputs are all Gaussian distributions corresponding to the meta-knowledge features. The fusion feature distribution of the two types of meta-knowledge is obtained through a product of high-dimensional Gaussian distributions, and its distribution parameters are related to the sub-distribution coefficients. , The distribution form is as follows:

[0091]

[0092]

[0093] in, This is the meta-representation vector corresponding to the fused feature distribution. Let be the mean vector and covariance matrix corresponding to the Gaussian distribution. Let be the inverse of the covariance matrix, when When the matrix is ​​diagonal, the value corresponds to the reciprocal of the variance of each component. In the high-dimensional feature distribution constructed by this algorithm, each component is relatively independent, and the covariance between elements is ignored when calculating the variance of the fused distribution. The fused feature constructed in this way... With neural networks , All are related, and the whole is represented as Utilizing fusion features When training with an embedded reinforcement learning policy, the loss function of equation (x) is updated to the following form:

[0094] Ideally, the relationship between distributions is called subdistribution. and High overlap or even complete coverage. To achieve this effect, additional KL divergence constraints are introduced. Approaching As shown in the following formula:

[0095] in, These are two classes of elements representing the cumulative expectation of the KL divergence distribution across all tasks, when using... Approaching When calculating KL divergence, it is necessary to consider... Placed as the first term. From the form of equation (x), we know that... and With equivalent expectation operators, the two loss functions can be combined to obtain the form shown below:

[0096] Unlike traditional model-free off-policy reinforcement learning algorithms, the aforementioned meta-reinforcement learning update process requires extensive use of interactive data with task classification. It necessitates maintaining independent sample caches for each task in the training set to update task meta-knowledge representations and corresponding policies. During training, three different sample caches are maintained: a task representation cache, a context meta-representation cache, and an experience replay cache. The task representation cache stores relevant samples transformed from the source-network-load-storage co-optimization model corresponding to all tasks. The samples are based on offline initialization and remain unchanged during training. A meta-representation context, meta-representation cache, and experience replay cache are constructed for each task for storage, corresponding to the policy trajectory of the task's interactions. The state features in the meta-representation cache are full features containing topological graph information, while the state features in the experience cache only contain one-dimensional tensor state features and do not include graph structure information. The interaction logic between the various cached data is as follows: Figure 4 As shown.

[0097] As shown in the diagram, during the environment exploration phase, the interaction environment is initialized for a specific task. The corresponding task representation sample is fused with the context sample to calculate the meta-representation feature vector value. This value serves as information expansion in the state space to support the Actor network's policy calculation until the task's decision is finalized. When a new round of interaction begins, the meta-representation feature vector value needs to be recalculated. If the initial context cache for the task is empty, only the task representation sample is used as input for calculating the meta-representation feature vector value. The policy trajectory information of this round of interaction is organized into tensors for the context cache and experience replay cache, and stored in their respective caches. Note that this sample does not contain meta-representation feature information. The above steps are repeated for different tasks to enrich the cache data for each task until the interaction rounds required for algorithm training are met. During the sample set construction phase of algorithm training, on the one hand, a new meta-representation feature vector needs to be recalculated for each task; on the other hand, an off-policy policy trajectory sample set needs to be sampled from the experience replay cache. The new meta-representation feature vector is embedded into each sample in the policy trajectory set, forming the training sample for that task together with the new meta-representation feature vector. Repeat the above steps to sample all tasks, construct the full sample set, and then update the parameters according to the loss function in equation (x).

[0098] Complete training method: 1) Construct a training task set containing i tasks. ; 2) Initialize a sample replay cache for each task. ; 3) Let j be the task number, and set j=1; 4) Regarding the task Initialize context feature cache ; 5) Let k be the number of training iterations for the task, and take k=1; 6) From A context-global feature tensor z is sampled from the sample. 7) Based on tensor z, using policy networks From the task Obtain and store the interaction trajectory data. ; 8) The data in Stored in the form ; 9) The number of training iterations for the task is k+1. If k is less than the maximum number of training iterations for the task, return to step 6; otherwise, continue to step 10. 10) Increment the training loop count by j by 1. If j is greater than the number of tasks, continue to step 11); otherwise, return to step 4. 11) Let step be the number of training processes, and take step=1; 12) Replay cache from samples of each task Medium sampling algorithm training sample set Conghua context feature cache Mid-sampling context information ; 13) From A context-global feature tensor z is sampled from the sample. 14) From Sampling a context global feature tensor ; 15) Calculate based on equations (31) and (32) respectively , ; 16) Calculate the loss function of the feature network. ; 17) Calculate the alignment loss function representing the task. ; 18) Calculate the total number of tasks. , , The sum is used to update the policy network. Evaluation network and feature networks , ; 19) Increment the training process count by 1. If step is less than the maximum number of update steps, return to step 12; otherwise, continue to step 18. 20) Determine whether to exit the loop based on the training effect. If to continue training, return to step 3; otherwise, end the training.

[0099] In another embodiment of the present invention, a meta-reinforcement learning system for source-network-load-storage coordination is provided. This system can be used to implement the above-mentioned meta-reinforcement learning method for source-network-load-storage coordination. Specifically, the meta-reinforcement learning system for source-network-load-storage coordination includes a modeling module, a learning module, a task module, a training module, a representation module, and an output module.

[0100] The modeling module constructs an optimization objective function containing system economic cost and network loss cost, defines variable equality constraints, inequality constraints and integer variable constraints, and uses second-order cone relaxation technology to perform convex relaxation treatment on power flow nonlinear nonconvex constraints, transforming the original mixed integer nonlinear optimization problem into a solvable mixed integer second-order cone optimization problem, forming a source-grid-load-storage collaborative optimization model. The learning module identifies energy storage charging and discharging power and unit power as decision variables in the reinforcement learning action space, eliminates strong temporal coupling constraints, and decouples the original problem into a single-section optimization problem; it constructs a state space containing uncontrollable environmental state variables and controllable system state variables based on grid node operation data; and it constructs a reward function containing policy-related rewards and baseline rewards through the solution results of the single-section optimization problem. The task module, based on the state space, action space, and reward function obtained by the learning module, constructs a subset of source-load operation scenario tasks using DBSCAN clustering and data augmentation technology, and constructs a subset of topology task tasks based on fixed nodes and different line connections; the subset of source-load operation scenario tasks and the subset of topology task tasks are combined to form m×n different subtasks; The training module constructs a task distribution and inference network based on m×n different sub-tasks formed by the task module; it collects interaction samples of each sub-task, processes the graph structure data in the interaction samples through a graph attention neural network, and then uses the Transformer architecture to aggregate state transition features at different times to obtain the context feature distribution; based on the SAC algorithm, it fuses the context features and performs parameter update training on the policy network, evaluation network, and inference network. The representation module constructs task prior features based on the prediction data output by the source-grid-load-storage collaborative optimization model obtained from the modeling module. It uses the same neural network structure as the training module to construct the task representation network, and uses KL divergence constraints to align the distribution of the task prior features with the distribution of the context features obtained from the training module, thereby realizing interactive state space embedding vector calculation. The output module, based on the trained policy network, evaluation network, and inference network obtained by the training module and the state space embedding vector obtained by the representation module, selects either the non-interactive meta-testing form or the training-free adaptable meta-testing form based on actual application conditions, and outputs the source-network-load-storage collaborative optimization decision results.

[0101] In the modeling module, the optimization objective function includes the sum of the equivalent economic costs of all optimizable elements of the system over the entire optimization period and the equivalent cost of the system power loss value; the variable equality constraints include KCL constraints, KVL constraints, and energy balance constraints; the variable inequality constraints include controllable equipment output limit constraints and controllable equipment power time-series ramping constraints; the integer variable constraints include energy storage device charging and discharging constraints, grid power purchase and sale constraints, and load switching behavior constraints.

[0102] In the learning module, the state space is established based on the grid node operation data. Uncontrollable environmental state quantities include renewable energy output, load demand, and market electricity price. Controllable system state quantities include the state of charge of energy storage devices and the current output level of generating units. The strategy-related reward in the reward function is obtained by removing the instant reward value corresponding to the baseline strategy from the instant reward. The strategy-related reward is calculated by the difference between the immediate reward value and the baseline reward value; the immediate reward value satisfies:

[0103] The baseline reward value is:

[0104] in, Represents the instant reward value. This represents a set of node numbers in a distributed unit system. This represents a set of node numbers for an energy storage system. This represents a set of node numbers that meet the requirements of the demand response system. A set representing the node numbers of the power grid purchasing system; Represents a set The elements in Represents a set The elements in Represents a set The elements in Represents a set The elements in Represents a set Elements determined by reinforcement learning; This represents the unit operating cost function. This represents the energy storage depreciation cost function. This represents the load demand response cost function. This represents the interaction cost function of the upper-level power grid. The cost function representing the curtailment of renewable energy sources. This indicates the cost of network loss. P This represents the power of the corresponding device. This represents the variable values ​​determined by reinforcement learning; SoC Indicates the percentage of state of charge. Indicates the current time; Represents the baseline reward value. This represents the energy storage and generator output values ​​in the baseline strategy.

[0105] In the task module, the construction of the source load running scenario task subset specifically includes: Using daily operation scenarios as boundaries, the continuous operation data of wind power, photovoltaics, load and upstream grid price at each node are decomposed to form time series features. The DBSCAN clustering method is used for classification, and data augmentation technology is used to augment the data of tasks with fewer samples. The construction of the topology task subset specifically includes: Various topologies are constructed based on fixed nodes and different line connections. Each topology is regarded as a subtask, and it is guaranteed that all nodes are connected by edges and the number of connected edges is the total number of nodes minus one.

[0106] In the training module, the inference network is trained by minimizing the estimation bias of the reinforcement learning value function and the amount of feature information; the graph attention neural network processes the graph structure data in the interaction samples and performs tensor merging on the node features of each graph according to the number to obtain the state transition features; the Transformer architecture aggregates the state transition features at different times to obtain the context feature distribution. When training for parameter updates based on the SAC algorithm, the update loss function for evaluating the network is:

[0107] in, To evaluate the sign of the network loss function, Indicates the evaluation of the network function, Indicates the baseline evaluation network function. Indicates the next state after the transition. This represents the z-vector that is not updated using gradients. The policy network loss function is:

[0108] in, Let the symbol be the loss function of the policy network. The KL divergence function represents the distance between two distributions. Techniques for representing exponential functions, This represents the state normalization factor.

[0109] In the representation module, the task representation network is constructed based on the sample data provided by the optimization model. It uses GNN to process the physical parameter features of the power grid on a single time segment and uses the Transformer structure to fuse features from multiple time segments to form a prior context feature distribution. The prior feature distribution is aligned with the context feature distribution through KL divergence constraints, specifically by minimizing the KL divergence between the two distributions.

[0110] In the output module, the non-interactive meta-testing method specifically includes: based on the temporal features of scene topology state and node power prediction, matching with tasks in the training task set, obtaining offline task representation samples and initializing task meta-representation vectors, constructing a state space in combination with current actual running data, inputting it into the policy network to output temporal decisions; the training-free adaptation meta-testing method based on actual information specifically includes: on the basis of non-interactive meta-testing, adding the temporal trajectory of the decision to the context sample set, and applying actual interaction information through the context meta-representation model to complete scene adaptation.

[0111] This invention provides a terminal device comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve corresponding method flows or corresponding functions. The processor described in this embodiment can be used for operations of meta-reinforcement learning methods oriented towards source-network-load-storage coordination, including: An optimization objective function incorporating system economic costs and network loss costs is constructed. Equality constraints, inequality constraints, and integer variable constraints are defined. A second-order cone relaxation technique is employed to convexly relax the nonlinear and nonconvex constraints of the power flow, transforming the original mixed-integer nonlinear optimization problem into a solvable mixed-integer second-order cone optimization problem, thus forming a source-grid-load-storage collaborative optimization model. The energy storage charging and discharging power and unit power are determined as decision variables in the reinforcement learning action space. Strong temporal coupling constraints are eliminated, and the original problem is decoupled into a single-section optimization problem. A state space containing uncontrollable environmental state variables and controllable system state variables is constructed based on grid node operation data. The results of solving the single-section optimization problem are used to construct... A reward function containing policy-related rewards and baseline rewards is used. Based on the obtained state space, action space, and reward function, a subset of source-load operation scenario tasks is constructed using DBSCAN clustering and data augmentation techniques, and a subset of topology task tasks is constructed based on fixed nodes and different line connections. The subset of source-load operation scenario tasks and the subset of topology task tasks are combined to form m×n different subtasks. Based on the formed m×n different subtasks, a task distribution and inference network is constructed. Interaction samples of each subtask are collected, and the graph structure data in the interaction samples is processed using a graph attention neural network, and then a Transformer is used. The architecture aggregates state transition features at different times to obtain a context feature distribution. Based on the SAC algorithm, these context features are fused to update and train the policy network, evaluation network, and inference network parameters. Based on the predicted data output by the obtained source-network-load-storage collaborative optimization model, task prior features are constructed. A task representation network is constructed using the same neural network structure, and KL divergence constraints are used to align the distribution of the task prior features with the obtained context feature distribution, achieving interactive state space embedding vector computation. Based on the obtained trained policy network, evaluation network, and inference network, and the obtained state space embedding vector, an interactive meta-testing format or a training-free adaptive meta-testing format based on actual information is selected according to the actual application conditions, and the source-network-load-storage collaborative optimization decision results are output.

[0112] Example 4 The present invention also provides a storage medium, specifically a computer-readable storage medium, which can be loaded and executed by a processor to implement one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the meta-reinforcement learning method for source-network-load-storage coordination in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by the processor to perform the following steps: An optimization objective function incorporating system economic costs and network loss costs is constructed. Equality constraints, inequality constraints, and integer variable constraints are defined. A second-order cone relaxation technique is employed to convexly relax the nonlinear and nonconvex constraints of the power flow, transforming the original mixed-integer nonlinear optimization problem into a solvable mixed-integer second-order cone optimization problem, thus forming a source-grid-load-storage collaborative optimization model. The energy storage charging and discharging power and unit power are determined as decision variables in the reinforcement learning action space. Strong temporal coupling constraints are eliminated, and the original problem is decoupled into a single-section optimization problem. A state space containing uncontrollable environmental state variables and controllable system state variables is constructed based on grid node operation data. The results of solving the single-section optimization problem are used to construct... A reward function containing policy-related rewards and baseline rewards is used. Based on the obtained state space, action space, and reward function, a subset of source-load operation scenario tasks is constructed using DBSCAN clustering and data augmentation techniques, and a subset of topology task tasks is constructed based on fixed nodes and different line connections. The subset of source-load operation scenario tasks and the subset of topology task tasks are combined to form m×n different subtasks. Based on the formed m×n different subtasks, a task distribution and inference network is constructed. Interaction samples of each subtask are collected, and the graph structure data in the interaction samples is processed using a graph attention neural network, and then a Transformer is used. The architecture aggregates state transition features at different times to obtain a context feature distribution. Based on the SAC algorithm, these context features are fused to update and train the policy network, evaluation network, and inference network parameters. Based on the predicted data output by the obtained source-network-load-storage collaborative optimization model, task prior features are constructed. A task representation network is constructed using the same neural network structure, and KL divergence constraints are used to align the distribution of the task prior features with the obtained context feature distribution, achieving interactive state space embedding vector computation. Based on the obtained trained policy network, evaluation network, and inference network, and the obtained state space embedding vector, an interactive meta-testing format or a training-free adaptive meta-testing format based on actual information is selected according to the actual application conditions, and the source-network-load-storage collaborative optimization decision results are output.

[0113] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0114] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0115] Source-grid-load-storage collaborative element reinforcement learning application method Please see Figure 5 Based on the policy neural network and meta-representation neural network constructed through meta-training, the application of source-source network load-storage collaborative optimization has two decision-making forms. The first is a non-interactive meta-testing form. Based on the temporal characteristics of scene topology state and node power prediction, it is matched with a certain task in the training task set to obtain its offline task representation sample and initialize the task meta-representation vector. At this point, the state space s is constructed using the current actual operating data, combined with the meta-representation vector. The first approach involves inputting the data into a policy network to achieve temporal decision-making, possessing the same real-time decision-making capabilities as traditional reinforcement learning. The second approach is a training-free meta-testing method based on real-world information. Since the predictive temporal features of real-world scenarios are difficult to perfectly match the trained hypothetical model, real-world scenario data is introduced to construct contextual representations to correct task feature differences, enabling efficient adaptation to new scenarios without training. Building upon the non-interactive meta-testing, the entire temporal trajectory of the decision is added to the contextual sample set. At this point, relying on the contextual meta-representation model… With no limit on the length of the input sequence and the symmetry of position exchange, it eliminates the need for trajectory caching and local sampling to obtain actual human features, and can more comprehensively apply the strategy information obtained from actual interactions to complete scene adaptation.

[0116] The reinforcement learning algorithm designed in this invention can achieve effective policy training across multiple task sets. The figure above shows the cumulative reward curves of the model on different tasks during the algorithm's training process. It can be seen that the reward value oscillates and increases as training progresses. This oscillation is due to inherent biases in the reward feedback for different tasks, but the average reward shows an upward trend, indicating that the meta-reinforcement learning algorithm effectively overcomes the problem of ineffective policy training caused by differences in task scenarios.

[0117] The context feature extraction method designed in this invention can effectively extract the similarity between scenes. Figure 6This paper demonstrates the convergence process of the context feature encoding z in four different source payload scenarios during algorithm training. Specifically, the source payload scenarios for Task 0 and Task 2 (top and bottom left images) are taken from the same month, as are those for Task 1 and Task 3 (top and bottom right images). Task 0 and Task 1 are taken from the same topology, as are Task 2 and Task 3. It is evident that, under the same topological conditions, the context feature encoding z obtained by tasks corresponding to the same month has a similar distribution, indicating that the meta-representation effectively extracts both the commonalities and differences between scenarios.

[0118] Simulation experimental data A distribution network simulation model with 10 nodes and 15 branches was built, including 2 distributed units, 3 energy storage devices, 5 flexible loads and 1 upstream grid interconnection line. Four source-load scenarios (typical days in spring, summer, autumn and winter) and three topologies (normal radial, branch disconnection forming islands, and new interconnection line) were set up, forming 12 training sub-tasks and 3 testing sub-tasks. Simulation results show that when the model of this invention is trained to 1M steps, the average cumulative reward for each task is 38.6% higher than that in the initial stage, with a 42.1% increase in reward for the high-load winter scenario and a 35.8% increase in reward for the isolated topology scenario. The context feature encoding z has a distribution similarity of 89.7% for scenarios in the same topology and season, and a similarity of 41.3% for scenarios in different seasons, effectively distinguishing scenario differences. Compared with the traditional DDPG model, the model of this invention improves the decision convergence speed by 67.2% in the new topology test task, reduces the system economic cost by 15.3%, and reduces network loss by 21.7%. Moreover, all decisions meet safety constraints such as voltage and power, and no limit violations occur.

[0119] In summary, this invention presents a meta-reinforcement learning method and system for source-grid-load-storage coordination. By constructing a source-grid-load-storage coordination optimization model and combining it with meta-reinforcement learning methods, it achieves efficient adaptive decision-making for multiple power grid scenarios and topologies. The method handles nonlinear constraints through second-order cone relaxation, extracts spatiotemporal features using graph attention networks and Transformers, constructs a multi-task training set, and aligns prior and real-time features using KL divergence, supporting interactive reasoning and online adaptation. Experiments show that the method of this invention outperforms traditional reinforcement learning methods in reward convergence, scene recognition, and policy generalization, effectively addressing source-load fluctuations and topology changes, and improving the economy and security of power grid operation.

Claims

1. A meta-reinforcement learning method oriented to source network load storage collaboration, characterized in that, The method comprises the following steps: S1, constructing an optimization objective function containing system economic cost and loss cost, defining variable equality constraints, inequality constraints and integer variable constraints, using second-order cone relaxation technology to convexly relax the non-linear non-convex constraints of power flow, converting the original mixed integer nonlinear optimization problem into a solvable mixed integer second-order cone optimization problem, and forming a source-grid-load-storage collaborative optimization model; S2, determining the storage charging and discharging power and the unit power as the action space decision variables of reinforcement learning, eliminating the strong time coupling constraints and decoupling the original problem into a single-section optimization problem; constructing a state space containing uncontrollable environmental state variables and system controllable state variables based on grid node operation data; constructing a reward function containing strategy strong correlation reward and baseline reward through the single-section optimization problem solving result; S3, based on the state space, the action space and the reward function obtained in step S2, using DBSCAN clustering and data augmentation technology to construct a source-load operation scene task subset, and based on fixed nodes and different line connections to construct a topological structure task subset; Combining the source-load operation scene task subset and the topological structure task subset to form m*n different sub-tasks; S4, based on the m*n different sub-tasks formed in step S3, constructing a task distribution and reasoning network; collecting interaction samples of each sub-task, processing the graph structure data in the interaction samples through a graph attention neural network, and then using a Transformer architecture to aggregate state transition features at different times to obtain context feature distribution; based on the SAC algorithm, the context features are fused to update and train the parameters of the strategy network, the evaluation network and the reasoning network; S5, based on the predicted data output by the source-grid-load-storage collaborative optimization model obtained in step S1, constructing task prior features; using the same neural network structure as step S4 to construct a task representation network, and aligning the distribution of the task prior features with the context feature distribution obtained in step S4 through KL divergence constraint to realize the calculation of state space embedding vectors without interaction; S6, based on the trained strategy network, evaluation network and reasoning network obtained in step S4 and the state space embedding vectors obtained in step S5, selecting an interactive-free meta-test form or a training-free adaptive meta-test form based on actual information according to the actual application conditions, and outputting a source-grid-load-storage collaborative optimization decision result.

2. The meta-reinforcement learning method for source network load storage coordination according to claim 1, wherein, In step S1, the optimization objective function includes the sum of the equivalent economic costs of all optimizable elements of the system within the entire optimization period and the equivalent cost of system power loss; the variable equality constraints include KCL constraints, KVL constraints and energy balance constraints; the variable inequality constraints include controllable device output boundary constraints and controllable device power time sequence climbing constraints; the integer variable constraints include storage device charging and discharging constraints, grid power purchase and sale constraints and load switching behavior constraints.

3. The meta-reinforcement learning method of claim 1, wherein, In step S2, the state space is established based on grid node operation data, the uncontrollable environment state quantity includes new energy output, load demand and market electricity price, the system controllable state quantity includes energy storage device state of charge and current output level of the unit, and the strategy strong correlation reward in the reward function is obtained by removing the immediate reward value corresponding to the baseline strategy from the immediate reward.

4. The meta-reinforcement learning method of claim 3, wherein, The strategy strong correlation reward is calculated by the difference between the immediate reward value and the baseline reward value; the immediate reward value satisfies: The baseline reward value is: wherein, represents an instant reward value, represents a set of distributed unit system node numbers, represents a set of energy storage system node numbers, represents a set of demand response system node numbers, represents a set of grid electricity purchase system node numbers; represents an element in the set represents an element in the set represents an element in the set represents an element in the set represents an element in the set represents an element in the set represents an element in the set represents an element in the set represents an element in the set determined by reinforcement learning; represents a unit operation cost function, represents an energy storage loss cost function, represents a load demand response cost function, represents a superior grid interaction cost function, represents a cost function of new energy curtailment, represents a grid loss equivalent cost; P represents the power of the corresponding device, represents a variable value determined by reinforcement learning; SoC represents a state of charge percentage, represents the current time; represents a baseline reward value, represents an energy storage and unit output value in the baseline strategy.

5. The meta-reinforcement learning method of claim 1, wherein, In step S3, the construction of the source-load operation scene task subset specifically includes: The continuous operation actual data of wind power, photovoltaic, load and the price of the upper grid on each node are decomposed to form time sequence characteristics based on the daily operation scene as a boundary, classification is performed based on the DBSCAN clustering method, and data augmentation technology is used to perform data augmentation on the tasks containing fewer samples; The construction of the topology structure task subset specifically includes: A plurality of topology structures are formed based on fixed nodes and different line connections, each topology structure is regarded as a subtask, and it is ensured that all nodes are connected by edges and the number of connected edges is one less than the total number of nodes.

6. The meta-reinforcement learning method for source network load storage coordination according to claim 1, characterized in that, In step S4, the inference network is trained by minimizing the reinforcement learning value function estimation bias and feature information quantity; the graph attention neural network processes the graph structure data in the interaction sample, and tensor merging is performed on the node features of each graph according to the number to obtain state transition features; The Transformer architecture aggregates the state transition features at different times to obtain context feature distribution.

7. The meta-reinforcement learning method of claim 6, wherein, When the SAC algorithm is used for parameter update training, the update loss function of the evaluation network is: wherein, to evaluate the network loss function sign, represents the evaluation network function, represents the baseline evaluation network function, represents the next state of the transition, represents the z vector without gradient update; The policy network loss function is: where, is the policy network loss function notation, denotes the KL divergence function between two distributions, denotes the exponential function technique, denotes the state normalization factor.

8. The meta-reinforcement learning method of claim 1, wherein, In step S5, the task representation network is constructed based on sample data provided by the optimization model, GNN is used to process the grid physical parameter features at a single time section, and Transformer structure is used to fuse multi-time section features to form prior context feature distribution; the prior feature distribution is aligned with the context feature distribution through KL divergence constraint, and the alignment is realized by minimizing the KL divergence between the two types of distributions.

9. The meta-reinforcement learning method of claim 1, wherein, In step S6, the interaction-free meta testing form specifically includes: based on the scene topology state and node power prediction time sequence characteristics, matching with the tasks in the training task set, obtaining offline task representation samples and initializing task meta representation vectors, constructing a state space combined with the current actual operation data, inputting the policy network to output time sequence decisions; the training-free adaptive meta testing form based on actual information specifically includes: on the basis of the interaction-free meta testing, the time sequence trajectory of the decision is added to the context sample set, and the actual interaction information is applied through the context meta representation model to complete the scene adaptation.

10. A meta-reinforcement learning system oriented to source network load storage collaboration, characterized in that, It includes: The modeling module constructs an optimization objective function containing system economic cost and network loss cost, defines variable equation constraints, inequality constraints and integer variable constraints, performs convex relaxation processing on the non-linear non-convex constraints of power flow by using the second-order cone relaxation technology, converts the original mixed integer non-linear optimization problem into a solvable mixed integer second-order cone optimization problem, and forms a source-grid-load-storage collaborative optimization model; The learning module determines the energy storage charging and discharging power and the unit power as the reinforcement learning action space decision variables, eliminates the strong time coupling constraints and decouples the original problem into a single-section optimization problem; a state space including uncontrollable environmental state variables and system controllable state variables is constructed based on grid node operation data; a reward function including a strategy strong correlation reward and a baseline reward is constructed through the single-section optimization problem solution results; The task module, based on the state space, the action space and the reward function obtained by the learning module, uses DBSCAN clustering and data augmentation technology to construct a source load operation scene task subset, and constructs a topological structure task subset based on fixed nodes and different line connections; The source load operation scene task subset and the topological structure task subset are combined to form m×n different sub-tasks; The training module, based on the m×n different sub-tasks formed by the task module, constructs a task distribution and an inference network; interaction samples of each sub-task are collected, the graph structure data in the interaction samples are processed through a graph attention neural network, and then the state transition features at different times are aggregated through a Transformer architecture to obtain a context feature distribution; the context features are fused based on the SAC algorithm to update and train the parameters of the strategy network, the evaluation network and the inference network; The representation module, based on the predicted data output by the source grid load storage collaborative optimization model obtained by the modeling module, constructs task prior features; a task representation network is constructed using the same neural network structure as the training module, and the distribution of the task prior features is aligned with the context feature distribution obtained by the training module through KL divergence constraint, realizing the calculation of the state space embedding vector without interaction; The output module, based on the trained strategy network, evaluation network and inference network obtained by the training module and the state space embedding vector obtained by the representation module, selects an interaction-free unit test form or an interaction-free adaptive unit test form based on actual information according to the actual application conditions, and outputs a source grid load storage collaborative optimization decision result. 11.The source network load storage oriented collaborative meta-reinforcement learning system according to claim 10, characterized in that, In the modeling module, the optimization objective function includes the sum of the equivalent economic costs of all optimizable elements of the system within the entire optimization period and the equivalent cost of system power loss; the variable equation constraints include KCL constraints, KVL constraints and energy balance constraints; the variable inequality constraints include controllable device output boundary constraints and controllable device power time sequence climbing constraints; the integer variable constraints include energy storage device charging and discharging constraints, grid power purchase and sale constraints and load switching behavior constraints.

12. The meta-reinforcement learning system of claim 10, wherein, In the learning module, the state space is established based on grid node operation data, the uncontrollable environmental state variables include new energy output, load demand and market electricity price, and the system controllable state variables include energy storage device state of charge and current unit output level; the strategy strong correlation reward in the reward function is obtained by removing the immediate reward value corresponding to the baseline strategy from the immediate reward; The strategy strong correlation reward is calculated by the difference between the immediate reward value and the baseline reward value; The immediate reward value satisfies: The baseline reward value is: wherein, represents an instant reward value, represents a set of distributed unit system node numbers, represents a set of energy storage system node numbers, represents a set of demand response system node numbers, represents a set of grid electricity purchase system node numbers; represents an element in the set represents an element in the set represents an element in the set represents an element in the set represents an element in the set represents an element in the set represents an element in the set represents an element in the set represents an element in the set determined by reinforcement learning; represents a unit operation cost function, represents an energy storage loss cost function, represents a load demand response cost function, represents a senior grid interaction cost function, represents a cost function of new energy curtailment, represents a grid loss equivalent cost; P represents the power of the corresponding device, represents a variable value determined by reinforcement learning; SoC represents a state of charge percentage, represents the current time; represents a baseline reward value, represents an energy storage and unit output value in the baseline strategy.

13. The meta-reinforcement learning system of claim 10, wherein, In the task module, the construction of the source load operation scene task subset specifically includes: The continuous operation actual data of the wind power, photovoltaic, load and price of the upper power grid on each node are divided based on a daily operation scene as a boundary to form a time sequence feature, classified based on a DBSCAN clustering method, and data augmentation technology is used to perform data augmentation on a task containing fewer samples; The constructing of the topology structure task subset specifically includes: A plurality of topology structures are formed based on fixed nodes and different line connections, each topology structure is regarded as a subtask, and all nodes are connected by edges and the number of connected edges is one less than the total number of nodes. 14.The source-oriented network-payload-storage collaborative meta-reinforcement learning system of claim 10, wherein, In the training module, the inference network is trained by minimizing the reinforcement learning value function estimation bias and feature information amount; the graph attention neural network processes the graph structure data in the interaction sample, and tensor merging is performed on the node features of each graph according to the number to obtain state transition features; The Transformer architecture aggregates the state transition features at different times to obtain a context feature distribution; When the SAC algorithm is used for parameter update training, the update loss function of the evaluation network is: wherein, to evaluate the network loss function sign, represents the evaluation network function, represents the baseline evaluation network function, represents the next state of the transition, represents the z vector without gradient update; The policy network loss function is: where, is the policy network loss function notation, denotes the KL divergence function between two distributions, denotes the exponential function technique, denotes the state normalization factor.

15. The meta-reinforcement learning system of claim 10, wherein, In the representation module, the task representation network is constructed based on sample data provided by the optimization model, uses GNN to process power grid physical parameter features at a single time section, uses a Transformer structure to fuse multi-time section features to form a prior context feature distribution, and aligns the prior feature distribution with the context feature distribution through KL divergence constraint, and the alignment is specifically realized by minimizing the KL divergence between the two types of distributions. 16.The source-oriented meta-reinforcement learning system of claim 10, wherein, In the output module, the interaction-free test form specifically includes: based on the scene topology state and the node power prediction time sequence feature, matching with the tasks in the training task set, obtaining the offline task representation sample and initializing the task meta-representation vector, combining the current actual operation data to construct a state space, inputting the policy network to output a time sequence decision; the training-free adaptive meta-test form based on actual information specifically includes: on the basis of the interaction-free meta-test, adding the time sequence trajectory of the decision to the context sample set, and applying the actual interaction information through the context meta-representation model to complete the scene adaptation.

17. A computer-readable storage medium storing one or more programs, the one or more programs comprising instructions for: The one or more programs include instructions that, when executed by a computing device, cause the computing device to perform the method of any one of claims 1-9.

18. A computing device, comprising: Including: One or more processors, memories and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs include steps for performing the method of any one of claims 1-9.

Images