Smart power grid resource allocation method and device based on Steinberg game strategy

By applying the Steinberg game strategy and multi-agent reinforcement learning model in the smart grid, and optimizing resource allocation, the problem of low resource allocation efficiency in the smart grid is solved, and the effective allocation of resources and the maximum benefit is achieved.

CN119090202APending Publication Date: 2024-12-06GUANGZHOU POWER SUPPLY BUREAU GUANGDONG POWER GRID CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411129203.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-16
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing smart grid lacks an effective economic interaction model to guide resource transactions and interest allocation between grid operators and energy suppliers, resulting in low resource allocation efficiency and failure to fully consider dynamic changes in the market environment, such as supply and demand fluctuations and Price changes.

Method used

Using Steinberg game strategy and multi-agent reinforcement learning model, we use the node state set of smart grids, generate resource interaction strategies, and optimize resource configuration and reward allocation between virtual operation nodes and resource provision nodes.

Benefits of technology

It improves the efficiency of resource allocation of smart grids, helps all participants find the optimal strategy in the ever-changing environment, and achieves effective allocation of resources and maximizes benefits.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090202A_ABST
    Figure CN119090202A_ABST
Patent Text Reader

Abstract

The invention relates to a smart power grid resource allocation method and device based on a Steinberg game strategy, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring a node state set of a smart power grid at the current moment based on a Steinberg game strategy; the node state set comprises a state corresponding to each node in the smart power grid at the current moment; the nodes comprise virtual operation nodes and resource providing nodes; the virtual operation nodes provide network services based on the network resources provided by the resource providing nodes; inputting the node state set into a multi-agent reinforcement learning model, and processing the node state set through the multi-agent reinforcement learning model to obtain a resource interaction strategy corresponding to each node in the smart power grid; and performing resource configuration on each node in the smart power grid based on the resource interaction strategy corresponding to each node in the smart power grid. By adopting the method, the resource configuration efficiency of each participant in the smart power grid can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of smart grid technology, and in particular to a smart grid resource configuration method, device, computer equipment, storage medium and computer program product based on Steinberg game strategy. Background Art

[0002] The construction and operation of smart grids involve multiple parties, including grid operators, energy suppliers, equipment manufacturers, etc. These parties have complex economic interactions in the use, maintenance and profit distribution of grid resources. For example, energy suppliers may need to pay grid operators to obtain cache space closer to users, thereby improving the efficiency of energy distribution and the user's electricity experience. Grid operators need to set reasonable prices to maximize their profits and encourage the participation of energy suppliers.

[0003] At present, the existing smart grid lacks an effective economic interaction model to guide the resource transactions and benefit distribution between grid operators and energy suppliers. In addition, grid operators and energy suppliers often face the problem of incomplete information when making economic decisions. At the same time, the existing strategies fail to fully consider the dynamic changes in the market environment, such as supply and demand fluctuations, price changes, etc. These problems lead to the inefficiency of smart grids in resource allocation.

[0004] Therefore, traditional technologies have the problem of low efficiency in resource allocation for various participants in the smart grid. Summary of the invention

[0005] Based on this, it is necessary to provide a smart grid resource configuration method, device, computer equipment, computer-readable storage medium and computer program product based on the Steinberg game strategy, which can improve the resource allocation efficiency of each participant in the smart grid in response to the above technical problems.

[0006] A smart grid resource configuration method based on Steinberg game strategy, the method comprising:

[0007] Based on the Steinberg game strategy, the node state set of the smart grid at the current moment is obtained; the node state set includes the state corresponding to each node in the smart grid at the current moment; the nodes include virtual operation nodes and resource provision nodes; the virtual operation nodes provide network services based on the network resources provided by the resource provision nodes;

[0008] The node state set is input into the multi-agent reinforcement learning model, and the node state set is processed by the multi-agent reinforcement learning model to obtain the resource interaction strategy corresponding to each node in the smart grid;

[0009] Based on the resource interaction strategy corresponding to each node in the smart grid, resources are configured for each node in the smart grid.

[0010] In one embodiment, based on the Steinberg game strategy, the node state set of the smart grid at the current moment is obtained, including:

[0011] In the first stage of the game, the storage space occupation corresponding to each resource providing node at the previous moment is obtained, and the unit resource attribute value corresponding to each virtual operation node at the current moment is determined according to the storage space occupation corresponding to each resource providing node at the previous moment;

[0012] In the second stage of the game, the storage space occupied by each resource providing node at the current moment is determined according to the unit resource attribute value corresponding to each virtual operation node at the current moment;

[0013] According to the unit resource attribute value corresponding to each virtual operation node at the current moment and the storage space occupancy corresponding to each resource providing node at the current moment, the node state set of the smart grid at the current moment is generated.

[0014] In one embodiment, the multi-agent reinforcement learning model includes a strategy output network and a strategy evaluation network; a node state set is input into the multi-agent reinforcement learning model, and the node state set is processed by the multi-agent reinforcement learning model to obtain a resource interaction strategy corresponding to each node in the smart grid, including:

[0015] For any node, the state corresponding to the node is input into the strategy output network, and the action selection strategy corresponding to the node is obtained based on the state corresponding to the node through the strategy output model in the strategy output network;

[0016] The state and action selection strategy corresponding to each node are input into the strategy evaluation network, and the action value determination model in the strategy evaluation network outputs the action selection strategy value corresponding to each node based on the state and action selection strategy corresponding to each node;

[0017] According to the action selection strategy value corresponding to each node, the resource interaction strategy corresponding to each node is determined.

[0018] In one embodiment, the strategy selection model is expressed as: ;

[0019] in, represents the state of the node at time t, represents the strategy of the node at time t, represents the action chosen by the node at time t.

[0020] In one embodiment, the action value determination model is expressed as:

[0021] ;

[0022] in, Indicates the action selection strategy value corresponding to the node, represents the reward at time t, represents the discount factor, Indicates status The target Q value output by the next strategy evaluation network is, Indicates in status The policy output model corresponding to the next policy, θ π′ represents the network parameters of the policy output network, Represents the network parameters θ of the output network of the strategy π′ The next state reached by the node in the case of represents the network parameters of the policy evaluation network, Indicates in status The action selection strategy of the next node is The model used to output the target Q value in the policy evaluation network.

[0023] A training method for the multi-agent reinforcement learning model as described above, the method comprising:

[0024] Based on the Steinberg game strategy, the sample node state set, sample node action set and sample node reward set of the smart grid are obtained; the sample node state set includes the state corresponding to each sample node in the smart grid; the sample node action set includes the action corresponding to each sample node in the smart grid; the sample node reward set includes the reward corresponding to each sample node in the smart grid;

[0025] The multi-agent reinforcement learning model is trained using a sample node state set, a sample node action set and a sample node reward set to obtain a trained multi-agent reinforcement learning model; the trained multi-agent reinforcement learning model is used to output resource configuration information for the smart grid.

[0026] A smart grid resource configuration device based on Steinberg game strategy, the device comprising:

[0027] The acquisition module is used to acquire the node state set of the smart grid at the current moment based on the Steinberg game strategy; the node state set includes the state corresponding to each node in the smart grid at the current moment; the node includes a virtual operation node and a resource providing node; the virtual operation node provides network services based on the network resources provided by the resource providing node;

[0028] An input module is used to input the node state set into the multi-agent reinforcement learning model, and the node state set is processed by the multi-agent reinforcement learning model to obtain the resource interaction strategy corresponding to each node in the smart grid;

[0029] The configuration module is used to configure resources for each node in the smart grid based on the resource interaction strategy corresponding to each node in the smart grid.

[0030] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0031] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0032] A computer program product comprises a computer program, which implements the steps of the above method when executed by a processor.

[0033] The above-mentioned smart grid resource configuration method, device, computer equipment, storage medium and computer program product based on the Steinberg game strategy obtain the node state set of the smart grid at the current moment based on the Steinberg game strategy; the node state set includes the state corresponding to each node in the smart grid at the current moment; the node includes a virtual operation node and a resource providing node; the virtual operation node provides network services based on the network resources provided by the resource providing node; the node state set is input into the multi-agent reinforcement learning model, and the node state set is processed by the multi-agent reinforcement learning model to obtain the resource interaction strategy corresponding to each node in the smart grid; based on the resource interaction strategy corresponding to each node in the smart grid, resources are configured for each node in the smart grid; in this way, based on the Steinberg game strategy and the multi-agent reinforcement learning model, reference information can be provided for the resource interaction and reward distribution between the virtual operation node and the resource providing node, which can improve the resource allocation efficiency of the smart grid, help all participants find the optimal strategy in a constantly changing environment, and achieve effective resource allocation and benefit maximization. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] FIG1 is an application environment diagram of a smart grid resource configuration method based on a Steinberg game strategy in one embodiment;

[0036] Figure 2 A schematic diagram of a process of a smart grid resource configuration method based on a Steinberg game strategy in one embodiment;

[0037] Figure 3A schematic diagram of a flow chart of a method for obtaining a node state set of a smart grid at a current moment in an embodiment;

[0038] Figure 4 is a schematic diagram of a MADDPG framework in one embodiment;

[0039] Figure 5 A flowchart of a training method for a multi-agent reinforcement learning model in one embodiment;

[0040] Figure 6 A smart grid resource configuration method based on the Steinberg game strategy in another embodiment;

[0041] Figure 7 It is a structural block diagram of a smart grid resource configuration device based on the Steinberg game strategy in one embodiment;

[0042] Figure 8 is a structural block diagram of a training device for a multi-agent reinforcement learning model in one embodiment;

[0043] FIG. 9 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] The smart grid resource configuration method based on the Steinberg game strategy provided in the embodiment of the present application can be applied in the application environment shown in Figure 1. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Based on the Steinberg game strategy, the server 104 obtains the node state set of the smart grid at the current moment; the node state set includes the state corresponding to each node in the smart grid at the current moment; the node includes a virtual operation node and a resource providing node; the virtual operation node provides network services based on the network resources provided by the resource providing node; the server 104 inputs the node state set into the multi-agent reinforcement learning model, and processes the node state set through the multi-agent reinforcement learning model to obtain the resource interaction strategy corresponding to each node in the smart grid; the server 104 performs resource configuration for each node in the smart grid based on the resource interaction strategy corresponding to each node in the smart grid. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0046] In an exemplary embodiment, Figure 2 As shown, a smart grid resource configuration method based on the Steinberg game strategy is provided, and the method is applied to the server 104 in FIG. 1 as an example for description, including the following steps 202 to 206. Among them:

[0047] Step 202, based on the Steinberg game strategy, obtain the node status set of the smart grid at the current moment; the node status set includes the status corresponding to each node in the smart grid at the current moment; the nodes include virtual operation nodes and resource provision nodes; the virtual operation nodes provide network services based on the network resources provided by the resource provision nodes.

[0048] Among them, the Stackelberg game strategy corresponds to the Stackelberg equilibrium (SE) model, which involves at least two participants with different decision-making time points, in which the leader makes a decision first and the followers then respond according to the leader's decision. In this application, the leader is a virtual operation node and the followers are resource provision nodes.

[0049] Among them, the virtual operation node can be a mobile network virtual operator (Mobile Network Virtual Operator, MVNO), that is, a node in the smart grid that does not have its own network infrastructure but rents network resources from other operators to provide services.

[0050] Among them, the resource providing node can be a content provider (CP), that is, an entity in the smart grid that provides content or services to mobile network virtual operators, such as video streaming services, user demand data, or scheduling and control of power loads.

[0051] Among them, the smart grid includes multiple mobile network virtual operators MVNO and content providers CP. The mobile network virtual operator MVNO provides network services based on the network resources provided by the content provider CP. The smart grid is an electric power system that uses advanced information and communication technologies to achieve efficient, reliable and intelligent management of the electric power system.

[0052] Among them, the node can be a virtual operation node or a resource provision node.

[0053] Optionally, the server obtains the node status set of the smart grid at the current moment based on the Steinberg game strategy, that is, obtains the status corresponding to each virtual operation node and each resource provision node in the smart grid at the current moment.

[0054] Step 204, input the node state set into the multi-agent reinforcement learning model, process the node state set through the multi-agent reinforcement learning model, and obtain the resource interaction strategy corresponding to each node in the smart grid.

[0055] Among them, the multi-agent reinforcement learning model can be a model used to solve the problem of multiple agents learning and optimizing strategies in a common environment. The model is implemented based on the MADDPG algorithm of Stackelberg-MADDPG, which can achieve the Stackelberg game equilibrium of the formulated game.

[0056] Among them, the resource interaction strategy can refer to the economic interaction between the virtual operation nodes and the resource providing nodes in the smart grid, and the process of resource interaction corresponds to the interaction and decision-making process between the virtual operation nodes and the resource providing nodes in resource trading and benefit distribution.

[0057] Optionally, the server inputs the node state set into the multi-agent reinforcement learning model, processes the node state set through the multi-agent reinforcement learning model, and obtains the resource interaction strategy corresponding to each virtual operation node and each resource providing node in the smart grid.

[0058] Step 206: Based on the resource interaction strategy corresponding to each node in the smart grid, perform resource configuration for each node in the smart grid.

[0059] Optionally, the server configures resources for each node in the smart grid based on the resource interaction strategy corresponding to each node in the smart grid.

[0060] In the above-mentioned smart grid resource configuration method based on the Steinberg game strategy, the node state set of the smart grid at the current moment is obtained based on the Steinberg game strategy; the node state set includes the state corresponding to each node in the smart grid at the current moment; the node includes a virtual operation node and a resource providing node; the virtual operation node provides network services based on the network resources provided by the resource providing node; the node state set is input into the multi-agent reinforcement learning model, and the node state set is processed by the multi-agent reinforcement learning model to obtain the resource interaction strategy corresponding to each node in the smart grid; based on the resource interaction strategy corresponding to each node in the smart grid, resources are configured for each node in the smart grid; in this way, based on the Steinberg game strategy and the multi-agent reinforcement learning model, reference information can be provided for the resource interaction and reward distribution between the virtual operation node and the resource providing node, which can improve the resource allocation efficiency of the smart grid, help all participants find the optimal strategy in a constantly changing environment, and achieve effective resource allocation and benefit maximization.

[0061] In an exemplary embodiment, Figure 3 As shown, step 202 includes steps 302 to 306. Among them:

[0062] Step 302, in the first stage of the game, obtain the storage space occupancy corresponding to each resource providing node at the previous moment of the current moment, and determine the unit resource attribute value corresponding to each virtual operation node at the current moment based on the storage space occupancy corresponding to each resource providing node at the previous moment.

[0063] The first stage of the game may refer to the first stage in the Steinberg game strategy, corresponding to the stage in which the leader virtual operation node MVNOv makes a decision first.

[0064] The storage space occupation may refer to the storage space purchase amount of the resource providing node, i.e., the content provider CPu. The storage space occupation refers to the storage capacity purchased by the CPu from the MVNOv in order to provide services in the smart grid. This storage space can be used to cache data, such as user demand data or other information related to grid operations, to improve service efficiency and response speed. The storage space purchase amount is usually measured in units of data storage, such as GB (gigabytes), TB (terabytes), etc.

[0065] The unit resource attribute value may refer to a virtual operation node, namely a mobile network virtual operator MVNO. The unit price is the fee that the MVNO charges the CP for providing storage space services, calculated based on the unit of storage space.

[0066] Optionally, in the first stage of the game, the server obtains the storage space purchase amount corresponding to each resource providing node CPu at the previous time t-1 , and provide the storage space purchase amount corresponding to the node CPu based on the resources at the previous time t-1 , determine the current virtual operation node MVNO Corresponding unit price .

[0067] Step 304, in the second stage of the game, the storage space occupancy corresponding to each resource providing node at the current moment is determined according to the unit resource attribute value corresponding to each virtual operation node at the current moment.

[0068] The second stage of the game may refer to the second stage in the Steinberg game strategy, corresponding to the stage in which the follower makes a decision for the resource providing node CPu according to the decision made by the leader in the first stage.

[0069] Optionally, in the second stage of the game, the server currently has each virtual operation node MVNO Corresponding unit price , determine the CP of each resource provider node For each virtual operation node MVNO The corresponding unit price Set storage space purchase amount .

[0070] Step 306, generating a node state set of the smart grid at the current moment according to the unit resource attribute value corresponding to each virtual operation node at the current moment and the storage space occupancy corresponding to each resource providing node at the current moment.

[0071] Optionally, the server can configure the virtual operation nodes MVNO according to the current time. Corresponding unit price . And each resource provider node CP For each virtual operation node MVNO The corresponding unit price Set storage space purchase amount , generate MVNO , CP At the present moment The state set .

[0072] In this embodiment, in the first stage of the game, the storage space occupancy corresponding to each resource providing node at the previous moment is obtained, and the unit resource attribute value corresponding to each virtual operation node at the current moment is determined according to the storage space occupancy corresponding to each resource providing node at the previous moment; in the second stage of the game, the storage space occupancy corresponding to each resource providing node at the current moment is determined according to the unit resource attribute value corresponding to each virtual operation node at the current moment; according to the unit resource attribute value corresponding to each virtual operation node at the current moment and the storage space occupancy corresponding to each resource providing node at the current moment, the node state set of the smart grid at the current moment is generated; in this way, the decision-making process of virtual operation nodes and resource providing nodes in resource utilization and benefit distribution of smart grids can be effectively simulated based on the Steinberg game strategy. Through this hierarchical decision-making model, the relationship and interaction mechanism of each participant in the smart grid can be better reflected, which is conducive to more accurate resource allocation of the smart grid.

[0073] In one embodiment, the multi-agent reinforcement learning model includes a policy output network and a policy evaluation network; the node state set is input into the multi-agent reinforcement learning model, and the node state set is processed by the multi-agent reinforcement learning model to obtain the resource interaction strategy corresponding to each node in the smart grid, including: for any node, the state corresponding to the node is input into the policy output network, and the action selection strategy corresponding to the node is obtained based on the state corresponding to the node through the policy output model in the policy output network; the state and action selection strategy corresponding to each node are input into the policy evaluation network, and the action value determination model in the policy evaluation network outputs the action selection strategy value corresponding to each node based on the state and action selection strategy corresponding to each node; according to the action selection strategy value corresponding to each node, the resource interaction strategy corresponding to each node is determined.

[0074] Among them, the strategy output network is a neural network used to predict the node action selection strategy in the multi-agent reinforcement learning model, and the strategy evaluation network is a neural network used to predict the action value of the node action selection strategy in the multi-agent reinforcement learning model.

[0075] To facilitate understanding by those skilled in the art, Figure 4 An exemplary MADDPG framework is provided, corresponding to a multi-agent reinforcement learning algorithm based on DDPG, which adopts the Actor-Critic framework and the CTDE training mode. The policy output network involved in this application is the Actor network in the Actor-Critic framework, and the policy evaluation network is the Critic network in the Actor-Critic framework.

[0076] The Actor-Critic framework combines the advantages of policy-based and value-based methods. Through the policy function in the Actor network, each node can generate an action under a given state. The Actor network is associated with the Critic network. Through the action value function in the Critic network, the value of the action output by the Actor network can be output, and the loss function is used to evaluate the action selection of the Actor network. Then, the Actor network uses the Deterministic Policy Gradient (DPG) method to update its network parameters. The policy gradient method DPG uses the action value output by the Critic network to guide the update of the policy. The deterministic policy gradient DPG can directly generate a deterministic action selection strategy, avoiding frequent action sampling. The Critic network uses the gradient descent method to update the action value function.

[0077] Among them, the action selection strategy refers to the action selected by the strategy output network for the node.

[0078] Among them, the action selection strategy value refers to the action value of the action selected by the strategy output network for the node.

[0079] Optionally, for any node, the server inputs the state corresponding to the node into the policy output network, obtains the action selection strategy corresponding to the node based on the state corresponding to the node through the policy output model in the policy output network, and then inputs the state and action selection strategy corresponding to each node into the policy evaluation network, and outputs the action selection strategy value corresponding to each node based on the state and action selection strategy corresponding to each node through the action value determination model in the policy evaluation network. The server then determines the resource interaction strategy corresponding to each node according to the action selection strategy value corresponding to each node.

[0080] In this embodiment, for any node, the state corresponding to the node is input into the policy output network, and the action selection strategy corresponding to the node is obtained based on the state corresponding to the node through the policy output model in the policy output network; the state and action selection strategy corresponding to each node are input into the policy evaluation network, and the action value determination model in the policy evaluation network outputs the action selection strategy value corresponding to each node based on the state and action selection strategy corresponding to each node; according to the action selection strategy value corresponding to each node, the resource interaction strategy corresponding to each node is determined; in this way, through the collaborative work of the policy output network and the policy evaluation network, the stability of the value-based reinforcement learning method and the flexibility of the policy-based reinforcement learning method can be effectively combined, so as to learn more accurate resource interaction strategies, which is conducive to providing accurate guidance for resource allocation of smart grids.

[0081] In one embodiment, the strategy selection model is expressed as: ;in, represents the state of the node at time t, represents the strategy of the node at time t, represents the action chosen by the node at time t.

[0082] In one embodiment, the action value determination model is expressed as: ;in, Indicates the action selection strategy value corresponding to the node, represents the reward at time t, represents the discount factor, Indicates status The following strategy evaluates the target Q value of the network output, Indicates that in the state The strategy output model corresponding to the strategy described below, θ π′ represents the network parameters of the strategy output network, Represents the network parameters θ of the output network of the strategy π′ The next state reached by the node in the case of represents the network parameters of the strategy evaluation network, Indicates that in the state The action selection strategy of the node described below is is a model in the strategy evaluation network for outputting the target Q value; belong or , and Respectively represent MVNO and CP A collection of strategies.

[0083] Among them, the target Q value is the numerical value of the action value of the action selection strategy of the policy output network used in the policy evaluation network.

[0084] It should be noted that any node in the smart grid involved in this application corresponds to an agent in the multi-intelligence reinforcement learning model, and each node, that is, each agent, corresponds to a policy output network Actor and a policy evaluation network Critic. Each policy output network Actor network and each policy evaluation network Critic contain an online network and a target network. In practical applications, it is necessary to maintain the stability of the information output by the online network by setting the network parameters in the target network.

[0085] In an exemplary embodiment, Figure 5As shown, a training method for a multi-agent reinforcement learning model is provided, and the method is applied to the server 104 in FIG. 1 as an example for explanation, including the following steps 502 to 504. Among them:

[0086] Step 502, based on the Steinberg game strategy, obtain the sample node state set, sample node action set and sample node reward set of the smart grid; the sample node state set includes the state corresponding to each sample node in the smart grid; the sample node action set includes the action corresponding to each sample node in the smart grid; the sample node reward set includes the reward corresponding to each sample node in the smart grid.

[0087] The sample node state set may refer to a set consisting of states of each sample node during the Steinberg game process.

[0088] The sample node action set may refer to a set of actions of each sample node in the Steinberg game process.

[0089] The sample node reward set may refer to a set of rewards obtained by each sample node during the Steinberg game.

[0090] Optionally, the server obtains a sample node state set, a sample node action set, and a sample node reward set of the smart grid based on the Steinberg game strategy.

[0091] The following exemplary method provides a method for determining the state, action and reward of a sample node based on the Steinberg game strategy, wherein the sample node includes a virtual operation node MVNO and resource provider node CP ,in:

[0092] When determining the state of the sample node: at the moment of the first stage of the game , MVNO First, based on the previous moment CP Storage space purchased , set the unit price ; In the second stage of the game, CP According to MVNO Set the unit price Setting to determine the amount of storage space to purchase MVNO , CP At the moment The state sets are represented as .

[0093] When determining the action of a sample node: given the state , the reinforcement learning agent performs actions based on the observations. Depending on the actions chosen, the reinforcement learning agent affects each entity (i.e., MVNO or CP ) utility changes. MVNO , CP At the moment The action sets are represented as Each reinforcement learning agent corresponds to a sample node.

[0094] When determining the rewards for sample nodes: you need to seek immediate rewards , maximizing system utility. MVNO , CP At the moment The rewards are expressed as , the system utility is the aggregate reward that the reinforcement learning agent obtains from the environment, expressed as .

[0095] Step 504, use the sample node state set, the sample node action set and the sample node reward set to train the multi-agent reinforcement learning model to obtain a trained multi-agent reinforcement learning model; the trained multi-agent reinforcement learning model is used to output resource configuration information for the smart grid.

[0096] Optionally, the server uses a sample node state set, a sample node action set, and a sample node reward set to train the multi-agent reinforcement learning model to obtain a trained multi-agent reinforcement learning model.

[0097] The following exemplary method provides a method for training a multi-agent reinforcement learning model using a sample node state set, a sample node action set, and a sample node reward set, corresponding to Figure 4 . Specifically:

[0098] First, we need to obtain training data, including local observation information obtained by each agent from observing the state of the environment, expressed as , and the observation information, action information, and reward information obtained by each agent independently interacting with the environment, expressed as , represents the state space of the environment at time t, It represents the action space of the agent at time t. It represents the reward space of the agent at time t. represents the state space of the environment at time t+1, is the state space at time t+1 predicted by the Steinberg game strategy based on the state space, action space, and reward space at time t; then, the state Input to the Actor network and get the predicted action output by the Actor network ,Will Input a batch of experience data into the experience pool D and randomly sample it , using empirical data To update the weights of each agent's Actor network and Critic network; then, the predicted action output by the Actor network Input to the Critic network, the Critic network predicts the action Output action value , and calculate the loss information corresponding to the Actor network and the Critic network respectively; finally, the model parameters are updated according to the loss information, thus completing one training.

[0099] Among them, the status After input into the Actor network, the Actor network adopts the strategy function Based on the current state and current strategies Select Action ,Right now ;in Corresponds to Figure 4 Status in It represents the local observation information obtained by the sample node, i.e. the agent, when observing the environment at time t. represents the strategy of the sample node, i.e. the agent, at time t, Action representing sample node selection.

[0100] Among them, the Critic network predicts the action Output action value When , the Critic network calculates the action based on the Bellman equation, an action value function Action Value ; The Bellman equation is used to calculate the expected return for each state or state-action pair.

[0101] In the first stage of training, the Critic network can be expressed by minimizing the loss function:

[0102] ;

[0103] ;

[0104] in, is in state The reward under the Steinberg game equilibrium point under , yi represents the target Q value of the ith agent, ri represents the reward of the ith agent, η represents the discount factor, represents the new state reached by the agent after executing the action, Represents the parameter set of the policy output network (i.e., Actor network) of the i-th agent; the policy gradient corresponding to the deterministic policy gradient DPG objective function is expressed as:

[0105] ;

[0106] Where ▽ is the gradient operator, which is used to calculate the gradient of the objective function with respect to the parameters; J is the objective function (or performance indicator), usually expressed in terms of the policy π i The expected return under the state distribution D is maximized, which is the main goal of reinforcement learning; E is the expectation operator, which means i The expected value below.

[0107] This embodiment uses soft update to update The target network is expressed as:

[0108]

[0109]

[0110] in Represents the learning rate.

[0111] In the second phase of training, the formula similar to the first phase is used to calculate the CP related functions.

[0112] The training method of the multi-agent reinforcement learning model mentioned above obtains the sample node state set, sample node action set and sample node reward set of the smart grid based on the Steinberg game strategy; the sample node state set includes the state corresponding to each of the sample nodes in the smart grid; the sample node action set includes the action corresponding to each of the sample nodes in the smart grid; the sample node reward set includes the reward corresponding to each of the sample nodes in the smart grid; the sample node state set, sample node action set and sample node reward set are used to train the multi-agent reinforcement learning model to obtain a trained multi-agent reinforcement learning model; the trained multi-agent reinforcement learning model is used to output resource configuration information for the smart grid; in this way, the multi-agent reinforcement learning model can be trained based on the sample node state set, sample node action set and sample node reward set obtained by the Steinberg game strategy, so as to train a multi-agent reinforcement learning model that can accurately output resource configuration information for the smart grid, which is conducive to accurately allocating resources to each participant of the smart grid in the prediction stage.

[0113] In another embodiment, Figure 6As shown, a smart grid resource configuration method based on the Steinberg game strategy is provided, and the method is applied to the server 104 in FIG. 1 as an example for explanation, including the following steps:

[0114] Step S602, based on the Steinberg game strategy, obtain the node status set of the smart grid at the current moment; the node status set includes the status corresponding to each node in the smart grid at the current moment; the nodes include virtual operation nodes and resource provision nodes; the virtual operation nodes provide network services based on the network resources provided by the resource provision nodes.

[0115] Step S604: for any node, the state corresponding to the node is input into the policy output network, and the action selection strategy corresponding to the node is obtained based on the state corresponding to the node through the policy output model in the policy output network.

[0116] Step S606, input the state and action selection strategy corresponding to each node into the strategy evaluation network, and output the action selection strategy value corresponding to each node based on the state and action selection strategy corresponding to each node through the action value determination model in the strategy evaluation network.

[0117] Step S608, determining the resource interaction strategy corresponding to each node according to the action selection strategy value corresponding to each node.

[0118] Step S610: performing resource configuration for each node in the smart grid based on the resource interaction strategy corresponding to each node in the smart grid.

[0119] It should be noted that the specific limitations of the above steps can refer to the specific limitations of a smart grid resource allocation method based on the Steinberg game strategy mentioned above.

[0120] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0121] Based on the same inventive concept, the embodiment of the present application also provides a smart grid resource configuration device based on the Steinberg game strategy for implementing the smart grid resource configuration method based on the Steinberg game strategy involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the smart grid resource configuration device based on the Steinberg game strategy provided below can refer to the limitations of the smart grid resource configuration method based on the Steinberg game strategy above, and will not be repeated here.

[0122] In an exemplary embodiment, Figure 7 As shown, a smart grid resource configuration device based on the Steinberg game strategy is provided, including: an acquisition module 702, an input module 704 and a configuration module 706, wherein:

[0123] The acquisition module 702 is used to acquire the node state set of the smart grid at the current moment based on the Steinberg game strategy; the node state set includes the state corresponding to each node in the smart grid at the current moment; the node includes a virtual operation node and a resource providing node; the virtual operation node provides network services based on the network resources provided by the resource providing node;

[0124] An input module 704 is used to input the node state set into the multi-agent reinforcement learning model, and process the node state set through the multi-agent reinforcement learning model to obtain the resource interaction strategy corresponding to each node in the smart grid;

[0125] The configuration module 706 is used to perform resource configuration for each node in the smart grid based on the resource interaction strategy corresponding to each node in the smart grid.

[0126] In one of the embodiments, the acquisition module 702 is specifically used to obtain, in the first stage of the game, the storage space occupancy corresponding to each resource providing node at the previous moment of the current moment, and determine the unit resource attribute value corresponding to each virtual operation node at the current moment based on the storage space occupancy corresponding to each resource providing node at the previous moment; in the second stage of the game, determine the storage space occupancy corresponding to each resource providing node at the current moment based on the unit resource attribute value corresponding to each virtual operation node at the current moment; generate the node status set of the smart grid at the current moment based on the unit resource attribute value corresponding to each virtual operation node at the current moment and the storage space occupancy corresponding to each resource providing node at the current moment.

[0127] In one embodiment, the multi-agent reinforcement learning model includes a policy output network and a policy evaluation network; the input module 704 is specifically used to input the state corresponding to any node into the policy output network, and obtain the action selection strategy corresponding to the node based on the state corresponding to the node through the policy output model in the policy output network; the state and action selection strategy corresponding to each node are input into the policy evaluation network, and the action value determination model in the policy evaluation network outputs the action selection strategy value corresponding to each node based on the state and action selection strategy corresponding to each node; and the resource interaction strategy corresponding to each node is determined according to the action selection strategy value corresponding to each node.

[0128] In one embodiment, the strategy selection model involved in the input module 704 is expressed as:

[0129] ;

[0130] in, represents the state of the node at time t, represents the strategy of the node at time t, An action representing node selection.

[0131] In one embodiment, the action value determination model involved in the input module 604 is expressed as:

[0132] ;

[0133] in, Indicates the action selection strategy value corresponding to the node, represents the reward at time t, represents the discount factor, Indicates status The target Q value output by the next strategy evaluation network is, Indicates in status The policy output model corresponding to the next policy, θ π′ represents the network parameters of the policy output network, Represents the network parameters θ of the output network of the strategy π′ The next state reached by the node in the case of represents the network parameters of the policy evaluation network, Indicates in status The action selection strategy of the next node is The model used to output the target Q value in the policy evaluation network

[0134] Each module in the smart grid resource configuration device based on the Steinberg game strategy can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0135] In an exemplary embodiment, Figure 8 As shown, a training device for a multi-agent reinforcement learning model is provided, including: a sample acquisition module 802 and a model training module 804, wherein:

[0136] The sample acquisition module 802 is used to acquire a sample node state set, a sample node action set and a sample node reward set of the smart grid based on the Steinberg game strategy; the sample node state set includes the state corresponding to each sample node in the smart grid; the sample node action set includes the action corresponding to each sample node in the smart grid; the sample node reward set includes the reward corresponding to each sample node in the smart grid;

[0137] The model training module 804 is used to train the multi-agent reinforcement learning model using a sample node state set, a sample node action set and a sample node reward set to obtain a trained multi-agent reinforcement learning model; the trained multi-agent reinforcement learning model is used to output resource configuration information for the smart grid.

[0138] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG9. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store smart grid resource configuration data based on the Steinberg game strategy. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a smart grid resource configuration method based on the Steinberg game strategy is implemented.

[0139] Those skilled in the art will understand that the structure shown in FIG. 9 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0140] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the smart grid resource configuration method based on the Steinberg game strategy. The steps of the smart grid resource configuration method based on the Steinberg game strategy here can be the steps of the smart grid resource configuration method based on the Steinberg game strategy in each of the above embodiments.

[0141] In one embodiment, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the processor executes the steps of the smart grid resource configuration method based on the Steinberg game strategy. The steps of the smart grid resource configuration method based on the Steinberg game strategy here can be the steps of the smart grid resource configuration method based on the Steinberg game strategy in each of the above embodiments.

[0142] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the processor executes the steps of the smart grid resource configuration method based on the Steinberg game strategy. The steps of the smart grid resource configuration method based on the Steinberg game strategy here can be the steps of the smart grid resource configuration method based on the Steinberg game strategy in each of the above embodiments.

[0143] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0144] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0145] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A smart grid resource configuration method based on Steinberg game strategy, characterized in that: The method comprises: Based on the Steinberg game strategy, a node state set of the smart grid at the current moment is obtained; the node state set includes the state corresponding to each node in the smart grid at the current moment; the nodes include virtual operation nodes and resource provision nodes; the virtual operation nodes provide network services based on the network resources provided by the resource provision nodes; Inputting the node state set into a multi-agent reinforcement learning model, processing the node state set through the multi-agent reinforcement learning model, and obtaining a resource interaction strategy corresponding to each of the nodes in the smart grid; Based on the resource interaction strategy corresponding to each of the nodes in the smart grid, resources are configured for each of the nodes in the smart grid.

2. The method according to claim 1, characterized in that The step of obtaining the node state set of the smart grid at the current moment based on the Steinberg game strategy includes: In the first stage of the game, the storage space occupancy corresponding to each resource providing node at the previous moment of the current moment is obtained, and the unit resource attribute value corresponding to each virtual operation node at the current moment is determined according to the storage space occupancy corresponding to each resource providing node at the previous moment; In the second stage of the game, the storage space occupation corresponding to each of the resource providing nodes at the current moment is determined according to the unit resource attribute value corresponding to each of the virtual operation nodes at the current moment; A node status set of the smart grid at the current moment is generated according to the unit resource attribute value corresponding to each of the virtual operation nodes at the current moment and the storage space occupancy corresponding to each of the resource providing nodes at the current moment.

3. The method according to claim 1, characterized in that The multi-agent reinforcement learning model includes a strategy output network and a strategy evaluation network; the node state set is input into the multi-agent reinforcement learning model, and the node state set is processed by the multi-agent reinforcement learning model to obtain the resource interaction strategy corresponding to each node in the smart grid, including: For any of the nodes, inputting the state corresponding to the node into the strategy output network, and obtaining the action selection strategy corresponding to the node based on the state corresponding to the node through the strategy output model in the strategy output network; Inputting the state and action selection strategy corresponding to each of the nodes into the strategy evaluation network, and outputting the action selection strategy value corresponding to each of the nodes based on the state and action selection strategy corresponding to each of the nodes through the action value determination model in the strategy evaluation network; According to the action selection strategy value corresponding to each of the nodes, the resource interaction strategy corresponding to each of the nodes is determined.

4. The method according to claim 3, characterized in that The strategy selection model is expressed as: ; in, represents the state of the node at time t, represents the strategy of the node at the time t, represents the action selected by the node at the time t.

5. The method according to claim 3, characterized in that: The action value determination model is expressed as: ; in, Indicates the action selection strategy value corresponding to the node, represents the reward at time t, represents the discount factor, Indicates status The following strategy evaluates the target Q value of the network output, Indicates that in the state The strategy output model corresponding to the strategy described below, θ π′ represents the network parameters of the strategy output network, Represents the network parameters θ of the output network of the strategy π′ The next state reached by the node in the case of represents the network parameters of the strategy evaluation network, Indicates that in the state The action selection strategy of the node described below is is a model used in the strategy evaluation network to output the target Q value.

6. A training method for a multi-agent reinforcement learning model as claimed in claim 1, characterized in that: The method comprises: Based on the Steinberg game strategy, a sample node state set, a sample node action set and a sample node reward set of a smart grid are obtained; the sample node state set includes the state corresponding to each of the sample nodes in the smart grid; the sample node action set includes the action corresponding to each of the sample nodes in the smart grid; the sample node reward set includes the reward corresponding to each of the sample nodes in the smart grid; The sample node state set, the sample node action set and the sample node reward set are used to train the multi-agent reinforcement learning model to obtain a trained multi-agent reinforcement learning model; the trained multi-agent reinforcement learning model is used to output resource configuration information for the smart grid.

7. A smart grid resource configuration device based on Steinberg game strategy, characterized in that: The device comprises: An acquisition module is used to acquire a node state set of the smart grid at the current moment based on a Steinberg game strategy; the node state set includes states corresponding to each of the nodes in the smart grid at the current moment; the nodes include virtual operation nodes and resource provision nodes; the virtual operation nodes provide network services based on network resources provided by the resource provision nodes; An input module, used for inputting the node state set into a multi-agent reinforcement learning model, processing the node state set through the multi-agent reinforcement learning model, and obtaining a resource interaction strategy corresponding to each of the nodes in the smart grid; A configuration module is used to perform resource configuration on each of the nodes in the smart grid based on the resource interaction strategy corresponding to each of the nodes in the smart grid.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.