Intelligent power grid dispatching algorithm based on reinforcement learning

Abstract

The invention discloses an intelligent power grid dispatching algorithm based on reinforcement learning, and the algorithm comprises the following steps: constructing a basic model of intelligent power grid dispatching, inputting pre-collected power grid data, and describing the environment and target of algorithm application; pre-training the pre-collected data by using the constructed model, so that the model masters basic safety behaviors and environment rules; on the basis of the online reinforcement learning dynamic adjustment capability, performing fine adjustment on the pre-training model according to actual environment feedback; a layered reinforcement learning idea is introduced to process a continuous and safe scene, a complex action space including continuous and discrete conditions is subdivided into different hierarchical structures, and each hierarchy is responsible for a specific type of decision; a heuristic rule is introduced, and the hierarchical structure is cut and optimized; and outputting a decision by the optimized model for power grid dispatching. According to the scheme, through self-adjustment and learning of online reinforcement learning, manual intervention is reduced, and the participation degree of the model in power dispatching of the smart grid is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of power grid dispatching, and in particular to a smart grid dispatching algorithm based on reinforcement learning. Background Art

[0002] Electricity, as the lifeblood of modern society, its efficient and reliable dispatching constitutes the lifeline to maintain the smooth operation of social production and life. In today's world, electricity is not just a simple energy transfer, it carries multiple missions of digital transformation, economic vitality and sustainable development, and is an indispensable cornerstone of modern society. With the acceleration of the global electrification process, the demand for electricity is showing an exponential growth trend. Coupled with the large-scale access of renewable energy, the power grid dispatching system is facing a profound change and challenge.

[0003] Although the current power grid dispatching system has entered the era of automation, significantly improving dispatching efficiency and accuracy, the limitations of traditional dispatching models are increasingly prominent when dealing with the increasingly complex and changeable power supply and demand environment. In the field of power grid dispatching, action space refers to the set of all possible operations that can be selected by dispatching decision makers or dispatching algorithms. In the dispatching and control of power systems, the definition of action space is crucial to the implementation of effective dispatching strategies, because it is directly related to how to make optimal decisions in a complex and changeable power supply and demand environment. The huge complexity of the action space and the inherent rigidity of the dispatching strategy mean that it is difficult for power grid dispatching to make optimal decisions in a rapidly changing market environment. In addition, although the heuristic dispatching strategy that relies on expert design performs well in specific scenarios, the difficulty and complexity of continuous optimization have become a bottleneck restricting its further development. Especially in the face of emergencies, such as a sudden surge in power demand or power grid failures caused by natural disasters, although manual intervention is a necessary remedial measure, its time-consuming and easily affected by subjective judgment often makes the power grid recovery process slow and full of uncertainty, and even exacerbates the instability of the power grid due to human errors. Summary of the invention

[0004] Purpose of the invention: In order to solve the above-mentioned problems, the present invention provides a smart grid dispatching algorithm based on reinforcement learning, which reduces manual intervention and enhances the model's participation in smart grid power dispatching through self-adjustment and learning of online reinforcement learning.

[0005] The technical solution, a smart grid dispatching algorithm based on reinforcement learning, includes the following steps:

[0006] Step S1, construct a basic model for smart grid dispatching, input pre-collected grid data, and describe the environment and objectives of algorithm application, including defining the state space and action space in the environment;

[0007] Step S2: Use the constructed model to pre-train the pre-collected data so that the model can grasp basic safety behaviors and environmental rules;

[0008] Step S3: Based on the dynamic adjustment capability of online reinforcement learning, the pre-trained model is fine-tuned according to actual environment feedback to improve model performance;

[0009] Step S4: Introduce the concept of hierarchical reinforcement learning to deal with the scenario of continuous safety, subdividing the complex action space including continuous and discrete situations into different hierarchical structures, each of which is responsible for a specific type of decision;

[0010] Step S5: introducing heuristic rules to trim and optimize the hierarchical structure;

[0011] Step S6: The optimized model outputs a decision, which is adopted by the power grid dispatcher.

[0012] According to another aspect of the present application, the state space and action space in the environment defined in step S1 are specifically:

[0013] Step S11, defining the state space in the environment, including:

[0014] Step S111, modeling the state space of the environment as a heterogeneous graph, represented by a quintuple G = (V, E, τ, φ, ψ), where V represents a vertex set, i.e., each node in the power grid, and E represents a set of edges, i.e., a topological structure of connections between each node;

[0015] Step S112: For the node type, a mapping function τ:V→N is used to map each node to a node type, where V represents a vertex set and N represents a corresponding node type set in the environment.

[0016] N={line,substation,generator,loader,storage};

[0017] Step S113, using a mapping function φ:E→R for edge types, mapping each node to a corresponding feature, where R represents a corresponding edge type set in the environment, R={1,2} represents the bus number to which it is connected, and ψ:V→F represents a feature mapping function;

[0018] Step S12, defining the action space in the environment, including: for the action space A in the environment, decomposing it into the following 6 parts, namely

[0019] Do-nothing: do nothing at the current time node;

[0020] Rewiring: Reconnecting a broken linei ;

[0021] Disconnection of wires: disconnecting a line that is not disconnected j ;

[0022] Reconfigure the substation busbar structure: Change a substation i Busbar configuration;

[0023] Restore the substation busbar structure: Restore a substation i busbar configuration, that is, connect all nodes to busbar 1;

[0024] Redistribute the power generation of generators and the charging and discharging of storage structures: continuous action type, adjust the power generation of each generator {g i} or storage unit {st i} power generation or charging power.

[0025] According to another aspect of the present application, the reward function used in the training in step S2 adopts the following method:

[0026]

[0027] According to another aspect of the present application, the model infrastructure and the structures of each head constructed in step S1 are as follows:

[0028] Backbone network: Use heterogeneous graph neural network HN to represent and learn the original features of each node, and obtain abstract features {f1, f2, f3, ...} of the same dimension for each node. Then, specific decisions are made based on these abstract features. The whole process is described by the mathematical relationship G' = HN (G), where G is the original input graph and G' is the output graph. Only the feature mapping function of the two graphs changes.

[0029] Top-level decision node: The decision goal of the top-level decision node is to select a broad action type, that is, from the 6 subsets Select one from the above, the node structure accepts 6 feature vectors representing different action spaces After MLP, a 6-dimensional logical value logits is output to represent different actions;

[0030] Wire reconnection / disconnection unit: The model input is the feature vector of normal / disconnected lines, that is, input = {f i |i∈L r} or input = {f i |i∈L d}, where L r With L dThey represent the set of disconnected and normal line nodes respectively. The decision-making neural network model is the Transformers model or directly uses the feature vector. The obtained vector is used for target positioning to obtain the probability of selecting a certain line, and the uncooled lines are marked;

[0031] Busbar configuration unit: Use substation nodes to obtain the characteristic vector {f i |i∈S}, select a specific substation node i, where S is the set of substation nodes;

[0032] Generator and storage structure unit: Use the characteristic vector {f i |i∈ST∪GE}, and use the two-dimensional vector value output by MLP to make decisions, where ST and GE represent the storage structure node and the generator node, respectively.

[0033] According to another aspect of the present application, the reinforcement learning fine-tuning stage in step S3 is further as follows:

[0034] Step S31, select some parameters of the fixed backbone network Backbone to maintain the distribution stability of dynamic changes corresponding to the reinforcement learning goal;

[0035] Step S32, selecting and optimizing the top decision nodes and other decision nodes alternately to reduce the difficulty of training and decision making;

[0036] Step S33, adopting the PPO algorithm and introducing the generalized action option criterion, making certain corrections to GAE to make the PPO algorithm compatible with option;

[0037] The corrected GAE looks like this:

[0038]

[0039] Among them, k i Represents the duration of the i-th action. For ordinary actions k i =1; for option, k i ≥1; if for any i there are k i =1, the modified formula degenerates to the original GAE formula.

[0040] According to another aspect of the present application, the decision-making process in step S4 is further as follows:

[0041] Step S41: First, construct the input structure G of the graph model according to the environment input, and then obtain the feature vector {f i};

[0042] Step S42: Using the feature vector, sample the probability distribution corresponding to the top-level decision node output and select a specific action type A. i ;

[0043] Step S43: After determining the selected action type, obtain a specific action distribution on a specific node, and then perform sampling to obtain a final action.

[0044] Beneficial effects: Through early imitation learning, the model quickly masters the expert behavior pattern, reduces the time and resources required to explore the environment from scratch, and improves the stability and efficiency of power grid dispatching; in the ever-changing real environment, the model combines online reinforcement learning to adjust itself, improves the adaptability and robustness to new situations, and realizes effective control and optimization of power grid dispatching; the present invention uses the idea of ​​hierarchical reinforcement learning, especially the introduction of generalized actions, which strengthens the model to take more efficient decision-making paths in a safe environment, reduces unnecessary computing burden, and ensures the system's rapid response capability in the face of emergencies; designs multi-layer neural network outputs for complex environments, effectively manages continuous and discrete mixed action spaces and hierarchical structures, and at the same time, trims the hierarchical structure through heuristic rules to simplify the depth of the decision tree, ensuring that the decision logic of the neural network is clear and non-redundant, which can fully explore the deep laws of the environment and ensure the efficiency and simplicity of the model training and decision-making process. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is the overall network structure diagram of the present invention.

[0046] Figure 2 It is a decision flow chart of the present invention. DETAILED DESCRIPTION

[0047] like Figure 1 As shown, a smart grid dispatching algorithm based on reinforcement learning includes the following steps:

[0048] Step S1, construct a basic model for smart grid dispatching, input pre-collected grid data, and describe the environment and objectives of algorithm application, including defining the state space and action space in the environment;

[0049] Step S2: Use the constructed model to pre-train the pre-collected data so that the model can grasp basic safety behaviors and environmental rules;

[0050] Step S3: Based on the dynamic adjustment capability of online reinforcement learning, the pre-trained model is fine-tuned according to actual environment feedback to improve model performance;

[0051] Step S4: Introduce the concept of hierarchical reinforcement learning to deal with the scenario of continuous safety, subdividing the complex action space including continuous and discrete situations into different hierarchical structures, each of which is responsible for a specific type of decision;

[0052] Step S5: introducing heuristic rules to trim and optimize the hierarchical structure;

[0053] Step S6: The optimized model outputs a decision, which is adopted by the power grid dispatcher.

[0054] First, the model is pre-trained on a large amount of data sampled from a rule-based expert model through imitation learning methods; then, with the help of the dynamic adjustment capabilities of online reinforcement learning, reinforcement learning is used to fine-tune the model in a specific environment.

[0055] In view of the fact that the environment is mostly in a safe state under normal circumstances and does not require frequent intervention actions, the present invention introduces the concept of hierarchical reinforcement learning and creatively defines a concept of "generalized action" (or option) specifically for handling scenarios of continuous safety.

[0056] Due to the complexity of the environmental action space, which contains both continuous and discrete situations, and the action has a hierarchical structure, the output module of the neural network is carefully designed, which is subdivided into multiple levels, each level focuses on a specific type of decision, thereby achieving the hierarchical and refined decision-making process. However, too much action layering is also likely to cause problems such as increased training difficulty and decreased decision-making stability. To this end, the present invention further introduces heuristic rules to cut and optimize the hierarchical structure to ensure that the decision logic of the neural network is clear and non-redundant, which can fully explore the deep laws of the environment and ensure the efficiency and simplicity of model training and decision-making processes.

[0057] Through the pre-training stage, the model can quickly master the basic rules and safety behaviors of power grid dispatching, so as to respond quickly in practical applications and improve dispatching efficiency; by utilizing the dynamic adjustment capability of online reinforcement learning, the model can be fine-tuned according to real-time environmental feedback, adapt to various changes and uncertainties in power grid operation, and enhance its adaptability and robustness; the concept of hierarchical reinforcement learning is introduced, especially when dealing with continuous safety scenarios, by defining "generalized actions" (Option) to reduce unnecessary intervention and ensure the safety of power grid operation; the complex action space is subdivided into different hierarchical structures, each level focuses on a specific type of decision, realizing the hierarchical and refined decision-making process, Improves the accuracy and efficiency of decision-making; trims and optimizes the hierarchical structure through heuristic rules to ensure that the decision logic of the neural network is clear and non-redundant, which can fully explore the deep laws of the environment and ensure the efficiency and simplicity of the model training and decision-making process; the design of this solution allows flexible application in different power grid environments, whether it is a large power grid or a small power grid, it can be adjusted and optimized according to specific circumstances; the characteristics of the reinforcement learning algorithm enable the model to continue to improve in continuous learning and adjustment, and the performance of the model will continue to improve over time; the automated decision-making process reduces human intervention, reduces the risks caused by human errors, and improves the reliability and stability of power grid dispatch. In summary, this solution provides an efficient, secure, flexible and continuously improving smart grid dispatching solution by combining pre-training, online reinforcement learning, hierarchical reinforcement learning and heuristic optimization.

[0058] According to another aspect of the present application, the state space and action space in the environment defined in step S1 are specifically:

[0059] Step S11, defining the state space in the environment, including:

[0060] Step S111, modeling the state space of the environment as a heterogeneous graph, represented by a quintuple G = (V, E, τ, φ, ψ), where V represents a vertex set, i.e., each node in the power grid, and E represents a set of edges, i.e., a topological structure of connections between each node;

[0061] Step S112: for the node type, a mapping function τ:V→N is used to map each node to a node type, where V represents a vertex set, N represents a corresponding node type set in the environment, and N={line, substation, generator, loader, storage};

[0062] Step S113, using a mapping function φ:E→R for edge types, mapping each node to a corresponding feature, where R represents a corresponding edge type set in the environment, R={1,2} represents the bus number to which it is connected, and ψ:V→F represents a feature mapping function;

[0063] Step S12, defining the action space in the environment, including: for the action space A in the environment, decomposing it into the following 6 parts, namely

[0064] Do-nothing: do nothing at the current time node;

[0065] Rewiring: Reconnecting a broken line i ;

[0066] Disconnection of wires: disconnecting a line that is not disconnected j ;

[0067] Reconfigure the substation busbar structure: Change a substation i Busbar configuration;

[0068] Restore the substation busbar structure: Restore a substation i busbar configuration, that is, connect all nodes to busbar 1;

[0069] Redistribute the power generation of generators and the charging and discharging of storage structures: continuous action type, adjust the power generation of each generator {g i} or storage unit {st i} power generation or charging power.

[0070] According to another aspect of the present application, the reward function used in the training in step S2 adopts the following method:

[0071]

[0072] According to another aspect of the present application, the model infrastructure and the structures of each head constructed in step S1 are as follows:

[0073] Backbone network: Use heterogeneous graph neural network HN to represent and learn the original features of each node, and obtain abstract features {f1, f2, f3, ...} of the same dimension for each node. Then, specific decisions are made based on these abstract features. The whole process is described by the mathematical relationship G' = HN (G), where G is the original input graph and G' is the output graph. Only the feature mapping function of the two graphs changes.

[0074] Top-level decision node: The decision goal of the top-level decision node is to select a broad action type, that is, from the 6 subsets Select one from the above, the node structure accepts 6 feature vectors representing different action spaces After MLP, a 6-dimensional logical value logits is output to represent different actions;

[0075] Wire reconnection / disconnection unit: The model input is the feature vector of normal / disconnected lines, that is, input = {f i |i∈L r} or input = {f i |i∈L d}, where L r With L d They represent the set of disconnected and normal line nodes respectively. The decision-making neural network model is the Transformers model or directly uses the feature vector. The obtained vector is used for target positioning to obtain the probability of selecting a certain line, and the uncooled lines are marked;

[0076] Busbar configuration unit: Use substation nodes to obtain the characteristic vector {f i |i∈S}, select a specific substation node i, where S is the set of substation nodes;

[0077] Generator and storage structure unit: Use the characteristic vector {f i |i∈ST∪GE}, and use the two-dimensional vector value output by MLP to make decisions, where ST and GE represent the storage structure node and the generator node, respectively.

[0078] According to another aspect of the present application, the reinforcement learning fine-tuning stage in step S3 is further as follows:

[0079] Step S31, select some parameters of the fixed backbone network Backbone to maintain the distribution stability of dynamic changes corresponding to the reinforcement learning goal;

[0080] Step S32, selecting and optimizing the top decision nodes and other decision nodes alternately to reduce the difficulty of training and decision making;

[0081] Step S33, adopting the PPO algorithm and introducing the generalized action option criterion, making certain corrections to GAE to make the PPO algorithm compatible with option;

[0082] The corrected GAE looks like this:

[0083]

[0084] Among them, k i Represents the duration of the i-th action. For ordinary actions k i =1; for option, ki ≥1; if for any i there are k i =1, the modified formula degenerates to the original GAE formula.

[0085] It should be noted that the reinforcement learning algorithm HPPO in the hybrid action space was tried in the early days, but it was found that its training effect was not as good as directly expanding the PPO algorithm to the layered mode. Therefore, the solution of expanding the PPO algorithm to the layered mode was adopted.

[0086] This solution maintains the stability of the distribution of dynamic changes corresponding to the reinforcement learning objective by fixing some parameters of the Backbone. This helps to maintain the stability of the model during fine-tuning and avoid learning instability or performance degradation caused by large parameter changes; the method of alternately optimizing the top-level decision nodes and other decision nodes reduces the difficulty of training and decision-making. This hierarchical optimization strategy enables the model to focus more on decisions at a specific level, improving training efficiency and decision quality; the generalized action option criterion is introduced, and GAE (Generalized Advantage Estimation) is modified to make the PPO (Proximal Policy Optimization) algorithm compatible with the concept of option. This correction allows the model to better utilize the advantages of options when dealing with complex decisions, improving the flexibility and adaptability of decisions; the revised GAE formula takes into account the duration of actions, especially for options, which enables the model to more accurately evaluate the impact of long-term decisions and make more reasonable decisions; when the duration of all actions is equal, the revised formula degenerates to the original GAE formula, which simplifies the implementation and understanding of the algorithm while also ensuring computational efficiency in specific situations; through optimization in the reinforcement learning fine-tuning phase, the model can continuously learn and improve in practical applications, and over time, the performance of the model will continue to improve, achieving continuous optimization and improvement.

[0087] In summary, this solution provides a stable, efficient, compatible and continuously improving smart grid dispatching solution through strategies such as fixed parameters, alternating optimization, and algorithm correction in the reinforcement learning fine-tuning stage.

[0088] According to another aspect of the present application, Figure 2 As shown, the model decision in step S4 follows a top-down decision process. This step-by-step decision process is adopted instead of treating all actions equally, in order to utilize the information in this decision structure. Specifically, most actions in the same category have similar functions, or actions in the same category have the same information. The decision process is as follows:

[0089] Step S41: First, construct the input structure G of the graph model according to the environment input, and then obtain the feature vector {f i};

[0090] Step S42: Using the feature vector, sample the probability distribution corresponding to the top-level decision node output and select a specific action type A. i ;

[0091] Step S43: After determining the selected action type, obtain a specific action distribution on a specific node, and then perform sampling to obtain a final action.

[0092] For the Do-nothing type action, there is no need to use neural network decision making, which represents an option. Use the triple <I,π,β> to describe an option, where: Indicates the state set at the beginning of an option, only the current state s t ∈I, this option can be selected; π:S×A→[0,1] represents the strategy executed in this option; β:S + →[0,1] represents the end condition. If the reinforcement learning agent selects an option, it will continue to execute according to the strategy π until it stops randomly according to β. If it is a non-terminal state, the agent selects the next option. In the power grid environment, the starting state set I = S is defined, that is, all states; the internal strategy π is defined as a rule that always outputs do-nothing, and the termination condition β is heuristically defined as a deterministic function that determines whether the environment is dangerous (such as the line load situation);

[0093] To select the line on / off type action, simply sample the probability of the corresponding neural network output. The action of selecting the substation bus configuration recovery is similar to the line action type.

[0094] For selecting the action type of substation bus reconfiguration, a series of action sets are screened based on rules for different substations in advance. si |i∈S}, where A si represents the action set selected on substation node i. After selecting the substation node, you only need to select the corresponding action set A si Just select an action a from the set. Experiments have shown that adding another neural network decision node to this part will make training difficult and the final effect is not good. Therefore, a rule-based method of selecting an action is adopted. Specifically, these actions are traversed, and the simulation function of the environment is used to simulate these actions on a certain substation node, and the action with the best effect is selected as the final action.

[0095] For the action type of changing the power of the generator and storage unit, the output of the neural network is used as the Beta distribution parameter, and the value in the interval (0, 1) is sampled from the distribution, and then the value is converted to the range of each node action space. Alternatively, due to the complexity of training continuous and discrete action spaces at the same time, this part is replaced by a solver-based or rule-based solution.

[0096] It should be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present invention will not further describe various possible combinations.

Claims

1. A smart grid dispatching algorithm based on reinforcement learning, characterized in that: The steps include: Step S1, construct a basic model for smart grid dispatching, input pre-collected grid data, and describe the environment and objectives of algorithm application, including defining the state space and action space in the environment; Step S2: Use the constructed model to pre-train the pre-collected data so that the model can grasp basic safety behaviors and environmental rules; Step S3: Based on the dynamic adjustment capability of online reinforcement learning, the pre-trained model is fine-tuned according to actual environment feedback to improve model performance; Step S4: Introduce the concept of hierarchical reinforcement learning to deal with the scenario of continuous safety, subdividing the complex action space including continuous and discrete situations into different hierarchical structures, each of which is responsible for a specific type of decision; Step S5: introducing heuristic rules to trim and optimize the hierarchical structure; Step S6: The optimized model outputs a decision, which is adopted by the power grid dispatcher.

2. The smart grid dispatching algorithm based on reinforcement learning as claimed in claim 1, characterized in that: The state space and action space in the environment defined in step S1 are specifically: Step S11, defining the state space in the environment, including: Step S111, modeling the state space of the environment as a heterogeneous graph, represented by a quintuple G = (V, E, τ, φ, ψ), where V represents a vertex set, i.e., each node in the power grid, and E represents a set of edges, i.e., a topological structure of connections between each node; Step S112: for the node type, a mapping function τ:V→N is used to map each node to a node type, where V represents a vertex set, N represents a corresponding node type set in the environment, and N={line, substation, generator, loader, storage}; Step S113, using a mapping function φ:E→R for edge types, mapping each node to a corresponding feature, where R represents a corresponding edge type set in the environment, R={1,2} represents the bus number to which it is connected, and ψ:V→F represents a feature mapping function; Step S12, defining the action space in the environment, including: for the action space A in the environment, decomposing it into the following 6 parts, namely Do-nothing: do nothing at the current time node; Rewiring: Reconnecting a broken line i ; Disconnection of wires: disconnecting a line that is not disconnected j ; Reconfigure the substation busbar structure: Change a substation i Busbar configuration; Restore the substation busbar structure: Restore a substation i busbar configuration, that is, connect all nodes to busbar 1; Redistribute the power generation of generators and the charging and discharging of storage structures: continuous action type, adjust the power generation of each generator {g i } or storage unit {st i } power generation or charging power.

3. The smart grid dispatching algorithm based on reinforcement learning as claimed in claim 1, characterized in that: The reward function used in the training in step S2 adopts the following method: Among them, r represents the reward function. The input of the reward function contains three items. s and s' represent the states at the current moment and the next moment respectively, and a represents the action taken at the current moment.

4. The smart grid dispatching algorithm based on reinforcement learning as claimed in claim 1, characterized in that: The model infrastructure and the structures of each head constructed in step S1 are as follows: Backbone network: Use heterogeneous graph neural network HN to represent and learn the original features of each node, and obtain abstract features {f1, f2, f3, ...} of the same dimension for each node. Then, specific decisions are made based on these abstract features. The whole process is described by the mathematical relationship G' = HN (G), where G is the original input graph and G' is the output graph. Only the feature mapping function of the two graphs changes. Top-level decision node: The decision goal of the top-level decision node is to select a broad action type, that is, from the 6 subsets Select one from the above, the node structure accepts 6 feature vectors representing different action spaces After MLP, a 6-dimensional logical value logits is output to represent different actions; Wire reconnection / disconnection unit: The model input is the feature vector of normal / disconnected lines, that is, input = {f i |i∈L r } or input = {f i |i∈L d }, where L r With L d They represent the set of disconnected and normal line nodes respectively. The decision-making neural network model is the Transformers model or directly uses the feature vector. The obtained vector is used for target positioning to obtain the probability of selecting a certain line, and the uncooled lines are marked; Busbar configuration unit: Use substation nodes to obtain the characteristic vector {f i |i∈S}, select a specific substation node i, where S is the set of substation nodes; Generator and storage structure unit: Use the characteristic vector {f i |i∈ST∪GE}, and use the two-dimensional vector value output by MLP to make decisions, where ST and GE represent the storage structure node and the generator node, respectively.

5. The smart grid dispatching algorithm based on reinforcement learning as claimed in claim 1, characterized in that: The reinforcement learning fine-tuning stage in step S3 is further as follows: Step S31, select some parameters of the fixed backbone network Backbone to maintain the distribution stability of dynamic changes corresponding to the reinforcement learning goal; Step S32, selecting and optimizing the top decision nodes and other decision nodes alternately to reduce the difficulty of training and decision making; Step S33, adopting the PPO algorithm and introducing the generalized action option criterion, making certain corrections to GAE to make the PPO algorithm compatible with option; The corrected GAE looks like this: Among them, k i Represents the duration of the i-th action. For ordinary actions k i =1; for option, k i ≥1; if for any i there are k i =1, the modified formula degenerates to the original GAE formula.

6. The smart grid dispatching algorithm based on reinforcement learning as claimed in claim 1, characterized in that: The decision-making process in step S4 is further as follows: Step S41: First, construct the input structure G of the graph model according to the environment input, and then obtain the feature vector {f i }; Step S42: Using the feature vector, sample the probability distribution corresponding to the top-level decision node output and select a specific action type A. i ; Step S43: After determining the selected action type, obtain a specific action distribution on a specific node, and then perform sampling to obtain a final action.

Images