Method, device and equipment for solving supply and demand matching problem of multiple smart grids

Abstract

The embodiment of the invention relates to the technical field of power system automation, and provides a multi-smart grid supply and demand matching problem solving method, device and equipment, and the method comprises the steps: taking each smart grid as an agent, constructing a shared action value table of a plurality of agents, the shared action value table comprises an action value mean value, an action value variance and sampling times of each state action pair; according to a preset sampling strategy, sampling the action value of each agent in different state action pairs to obtain an action value sampling value of each agent in each state action pair, and updating the shared action value table; and determining an optimal strategy according to the updated action value table, and taking the optimal strategy as an optimal solution of each intelligent power grid supply and demand matching problem. According to the embodiment of the invention, the optimal action can be selected through information sharing among the intelligent agents under limited computing resources, and the efficiency of learning the optimal strategy by multiple intelligent agents is improved.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of power system automation technology, and in particular to a method, device and equipment for solving the problem of matching supply and demand of multiple smart grids. Background Art

[0002] With the latest technological developments in renewable energy generation, fuel cells, and storage devices, the power grid is transforming into multiple interconnected microgrids. As a large number of new energy sources, distributed energy sources, electric vehicles, and other devices continue to be connected, the growing demand for electricity and the intermittent nature of electricity supply require coordination between supply and demand, which poses new challenges to solving the supply and demand matching problem of smart grids. The supply and demand matching of traditional power grids is mainly completed through one-time scheduling, while smart grids need to adjust the supply and demand relationship in real time and dynamically to adapt to the access of new energy sources and fluctuations in electricity demand. In the supply and demand matching problem of smart grids, multiple entities need to make real-time decisions on similar problems at the same time. Therefore, the supply and demand matching problem of smart grids can be modeled as a multi-agent reinforcement learning problem. Each agent represents an entity in the smart grid. Each agent interacts with the main grid to learn how to optimize its decision-making to maximize the overall benefit.

[0003] When multiple smart grids have similar geographical locations, functions, and energy storage devices, their cost functions and state transition matrices are often similar. However, traditional independent optimization methods ignore this similarity, resulting in each smart grid having to learn and optimize its strategy independently, which wastes limited computing resources and time. Therefore, how to select the optimal action through information sharing between intelligent agents under limited computing resources and improve the efficiency of learning the optimal strategy is an urgent problem to be solved. Summary of the invention

[0004] In view of the above-mentioned problems in the prior art, the purpose of the embodiments of this specification is to provide a method, device and equipment for solving the supply and demand matching problem of multiple smart grids, so as to select the optimal action through information sharing between intelligent agents under limited computing resources and improve the efficiency of learning the optimal strategy.

[0005] In order to solve the above technical problems, the specific technical solutions of the embodiments of this specification are as follows:

[0006] On the one hand, an embodiment of the present specification provides a method for solving a supply-demand matching problem of multiple smart grids, wherein the action value function and the state transition matrix of each smart grid are the same, and the method includes:

[0007] Taking each smart grid as an intelligent agent, constructing a shared action value table for multiple intelligent agents, wherein the shared action value table includes the action value mean, action value variance and sampling times of existing state-action pairs;

[0008] The action value of each agent in different state-action pairs is sampled according to a preset sampling strategy to obtain a sampling value of the action value of each agent in each state-action pair, wherein the sampling strategy is used to determine the sampling times of each state-action pair according to the action value mean and action value variance;

[0009] Update the shared action value table according to the action value sampling value and sampling times of each agent in each state-action pair;

[0010] The optimal strategy is determined according to the updated shared action value table, and the optimal strategy is used as the optimal solution to each smart grid supply and demand matching problem.

[0011] Furthermore, sampling the action value of each agent in different state-action pairs according to a preset sampling strategy includes:

[0012] S1: Determine the upper limit of the total number of sampling times for an agent to act in a certain state;

[0013] S2: Determine the initial sampling strategy of the agent according to the shared action value table;

[0014] S3: sampling the action values ​​of different actions of the agent in this state according to the initial sampling strategy to obtain sampling values ​​of the action values ​​of different actions of the agent in this state;

[0015] S4: re-determine a new sampling strategy for the agent based on the action value sampling values ​​of different actions of the agent in this state, re-sample based on the new sampling strategy and obtain new action value sampling values ​​of different actions of the agent in this state;

[0016] S5: updating the action value variance, action value mean and current total sampling times according to the new action value sampling value;

[0017] S6: Determine whether the current total number of sampling times is less than the upper limit of the total number of sampling times. If so, repeat steps S4-S5;

[0018] S7: If not, stop sampling and determine the optimal action in this state.

[0019] Furthermore, determining the initial sampling strategy of the agent according to the shared action value table includes:

[0020] Determining the initial sampling times according to the shared action value table;

[0021] The sampling times of each action of the agent in the current state are set as the initial sampling times.

[0022] Furthermore, re-determining a new sampling strategy of the agent according to the action value sampling values ​​of different actions of the agent in this state includes:

[0023] Determine the optimal action corresponding to the state in this round of sampling according to the action value sampling values ​​of different actions of the agent in this state;

[0024] Calculate the total number of samplings after this round of sampling in this state, the difference between the mean action value of each action and the optimal action, and the variance of the action value of each action;

[0025] According to the total number of samplings after this round of sampling in this state, the difference between the mean action value of each action and the optimal action, and the variance of the action value of each action, the distribution ratio of the number of samplings corresponding to each action is calculated;

[0026] The number of sampling times for each action after the current round of sampling is determined according to the allocation ratio.

[0027] Furthermore, according to the total number of samplings in this state after the current round of sampling, the difference between the mean action value of each action and the optimal action, and the action value variance of each action, the distribution ratio of the number of samplings corresponding to each action is calculated, including:

[0028] The following formula is used to calculate the distribution ratio of sampling times for each action:

[0029]

[0030] Among them, α a is the distribution ratio of the number of sampling times corresponding to each action a, a and a′ represent any pair of non-optimal actions, and b represents the optimal action. represents the number of times agent i samples action a in this round of sampling, is the total number of actions sampled in state s after this round of sampling, δ a is the difference between the mean action value of each action a and the optimal action b, is the action value variance of each action a, is the action value variance of the optimal action b, It represents the number of times agent i samples the optimal action b in this round of sampling.

[0031] Further, the updating of the current total number of sampling times according to the new action value sampling value includes:

[0032] Use the following formula to update the current total number of sampling times:

[0033]

[0034] in, is the current total sampling times, a is an action in this state, is the number of times agent i samples action a in this round of sampling, and Δ is the increment of the number of single-step sampling times.

[0035] Furthermore, the shared action value table is updated independently, synchronously or asynchronously by multiple agents.

[0036] Furthermore, updating the shared action value table according to the action value sampling value and sampling times of each agent in each state-action pair includes:

[0037] For each state of each agent, determining whether the shared action value table contains relevant records of the state;

[0038] If not, then calculate the action value mean and action value variance of the state-action pair according to the action value sampling value of each state-action pair of the agent in the state; record the action value mean, action value variance and sampling times of the state-action pair in the shared action table;

[0039] If yes, the shared action value table is updated using the following formula:

[0040]

[0041] Among them, μ s,a represents the mean action value of action a in state s, μ s ' ,a represents the mean action value of action a under the updated state s, represents the action value variance of action a in state s, represents the variance of the action value of action a under the updated state s, n s,a represents the total number of sampling times of action a in state s, n s ' ,a represents the total number of sampling times of action a in the updated state s, represents the number of times agent i samples action a in this round of sampling, It represents the action value of agent i in state s when taking action a in the n+1th round, Var represents the variance, and E represents the expectation.

[0042] On the other hand, an embodiment of the present specification provides a device for solving the supply and demand matching problem of multiple smart grids, wherein the action value function and the state transition matrix of each smart grid are the same, and the device comprises:

[0043] A construction module, used to take each smart grid as an intelligent agent and construct a shared action value table for multiple intelligent agents, wherein the shared action value table includes the action value mean, action value variance and sampling times of the existing state-action pairs;

[0044] A sampling module is used to sample the action value of each agent in different state-action pairs according to a preset sampling strategy to obtain the action value sampling value of each agent in each state-action pair, wherein the sampling strategy is used to determine the sampling times of each state-action pair according to the action value mean and action value variance;

[0045] An updating module, configured to update the shared action value table according to the action value sampling value and sampling times of each agent in each state-action pair;

[0046] The determination module is used to determine the optimal strategy according to the updated shared action value table, and use the optimal strategy as the optimal solution to each smart grid supply and demand matching problem.

[0047] On the other hand, an embodiment of the present specification further provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein when the computer program is executed by the processor, the instructions of any one of the above methods are executed.

[0048] On the other hand, an embodiment of the present specification further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor of a computer device, the computer program executes instructions of any one of the above-described methods.

[0049] On the other hand, the embodiments of this specification further provide a computer program product, which, when executed by a processor of a computer device, executes instructions of any one of the above methods.

[0050] By adopting the above technical solution, the method for solving the supply and demand matching problem of multiple smart grids provided in the embodiment of this specification firstly regards each smart grid as an intelligent agent according to the similarity of the problems solved by different intelligent agents, and constructs a shared action value table of multiple intelligent agents, thereby increasing the utilization rate of data and thus improving the efficiency of learning the optimal strategy. Then, the action value of each intelligent agent in different state-action pairs is sampled according to the pre-set sampling strategy, and the action value sampling value of each intelligent agent in each state-action pair is obtained. According to the action value sampling value of each intelligent agent in each state-action pair, the shared action value table is updated to obtain the final action value table, which can maximize the probability of correctly selecting the optimal action by allocating the number of sampling times of different actions under the condition of limited computing resources, thereby improving the sampling efficiency when using multi-agent reinforcement learning to solve the supply and demand matching problem of multiple smart grids, and can achieve a more accurate estimation of the action value function of each state-action pair.

[0051] The above description is only an overview of the technical solutions of some embodiments of this specification. In order to more clearly understand the technical means of some embodiments of this specification, they can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiments of this specification more obvious and easy to understand, the following specifically cites the preferred embodiments and describes them in detail in conjunction with the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 A schematic flow chart of a method for solving a multi-smart grid supply and demand matching problem in some embodiments of this specification is shown;

[0054] Figure 2 A schematic diagram showing a flow chart of sampling the action value of each agent in different state-action pairs according to a pre-set sampling strategy in some embodiments of this specification;

[0055] Figure 3 A schematic diagram of a process for re-determining a new sampling strategy for the agent in some embodiments of this specification is shown;

[0056] Figure 4 A schematic diagram showing the process of multi-agent information sharing in some embodiments of this specification;

[0057] Figure 5A schematic diagram of the module structure of a device for solving the supply and demand matching problem of multiple smart grids in some embodiments of this specification is shown;

[0058] Figure 6 A schematic diagram of the structure of a computer device in this specification is shown.

[0059] Description of the accompanying symbols:

[0060] 501. Building blocks;

[0061] 502, sampling module;

[0062] 503. Update module;

[0063] 504. Determine module;

[0064] 602. Computer equipment;

[0065] 604, processor;

[0066] 606. Memory;

[0067] 608, driving mechanism;

[0068] 610, input / output module;

[0069] 612. Input devices;

[0070] 614. Output device;

[0071] 616. Presentation equipment;

[0072] 618. Graphical user interface;

[0073] 620, network interface;

[0074] 622, communication link;

[0075] 624. Communication bus. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0077] In order to solve the above problems, an embodiment of the present specification provides a method for solving the supply and demand matching problem of multiple smart grids. Figure 1It is a flowchart of a method for solving a multi-smart grid supply and demand matching problem provided by an embodiment of this specification. This specification provides method operation steps as described in the embodiment or flowchart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the order of many steps and does not represent the only execution order. When the system or device product is executed in practice, it can be executed in the order of the method shown in the embodiment or the accompanying drawings or in parallel.

[0078] It should be noted that the terms "first", "second", etc. in this specification and claims and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this specification described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0079] Reference Figure 1 As shown, the embodiment of this specification provides a method for solving the supply and demand matching problem of multiple smart grids, wherein the action value function and the state transfer matrix of each smart grid are the same, and the method includes:

[0080] S101: Taking each smart grid as an intelligent agent, constructing a shared action value table for multiple intelligent agents, wherein the shared action value table includes the action value mean, action value variance and sampling times of existing state-action pairs.

[0081] In a smart grid system, when the cost functions and state transfer matrices of several micro-smart grids are similar, it can be considered that the problems to be solved by each smart grid are similar. Considering that each micro-smart grid has an intelligent agent for deciding the charging and discharging of energy storage equipment, some information should be shared between multiple intelligent agents to improve the learning efficiency of the optimal strategy. Specifically, if multiple micro-smart grids are located in similar geographical locations (such as office buildings or residential buildings), have similar functions (similar photovoltaic and wind power generation curves) and similar energy storage devices, then their state transfer matrices can be considered similar. If the cost of multiple microgrids buying electricity from the main grid is not much different, their cost functions can be considered similar. In this embodiment, the shared action value table is a two-dimensional array for storing the action value mean, action value variance and sampling number of each action in different states, thereby guiding the decision-making process of the intelligent agent, where the action value mean represents the expected return of the intelligent agent choosing an action in the current state, the action value variance is used to measure the uncertainty of the estimate, and the sampling number represents the sampling experience of the intelligent agent under the state action pair.

[0082] S102: Sample the action value of each agent in different state-action pairs according to a pre-set sampling strategy to obtain the action value sampling value of each agent in each state-action pair, wherein the sampling strategy is used to determine the sampling times of each state-action pair according to the action value mean and action value variance.

[0083] In reinforcement learning, the agent needs to try different actions to collect enough data to update its strategy. However, due to time and resource constraints, the agent cannot try all actions indefinitely. If the agent samples all actions the same number of times, this uniform distribution may be inefficient for those actions that are obviously worse or obviously better. Therefore, with a limited total number of samplings, a sampling strategy is needed to determine which actions should be sampled more. In this embodiment, a sampling number is allocated for each state-action pair through a pre-set sampling strategy, aiming to allocate the sampling number according to the potential value or uncertainty of each state-action pair, so that the agent can learn which actions are effective more quickly, thereby maximizing the efficiency of the process of learning the optimal strategy under limited computing resources. By giving priority to those actions that are most likely to provide new information or the greatest improvement, the agent can converge to the optimal strategy faster.

[0084] S103: updating the shared action value table according to the action value sampling value and sampling times of each agent in each state-action pair.

[0085] In this embodiment, multiple agents sample and update the shared action value table independently, synchronously or asynchronously, that is, all agents access and update the same action value table, thereby being able to share information about which actions are beneficial under which states, thereby increasing data utilization and avoiding repeated calculations and waste of computing resources.

[0086] S104: Determine an optimal strategy according to the updated shared action value table, and use the optimal strategy as the optimal solution to each smart grid supply and demand matching problem.

[0087] By adopting the above technical solution, the method for solving the supply and demand matching problem of multiple smart grids provided in the embodiment of this specification can firstly take each smart grid as an intelligent agent according to the similarity of the problems solved by different intelligent agents, and construct a shared action value table of multiple intelligent agents, thereby increasing the utilization rate of data and thus improving the efficiency of learning the optimal strategy. Then, the action value of each intelligent agent in different state-action pairs is sampled according to the pre-set sampling strategy, and the action value sampling value of each intelligent agent in each state-action pair is obtained. According to the action value sampling value of each intelligent agent in each state-action pair, the shared action value table is updated to obtain the final action value table, so that the probability of correctly selecting the optimal action can be maximized by allocating the number of sampling times of different actions under limited computing resources, thereby improving the sampling efficiency when using multi-agent reinforcement learning to solve the supply and demand matching problem of multiple smart grids, and realizing a more accurate estimation of the action value function of each state-action pair.

[0088] In the embodiments of this specification, for each intelligent agent, the state may include the state of charge of the energy storage device, the power generation of new energy, the power consumption of the power-consuming device, the use time and price of the power grid, etc. The actions may include charging, discharging and neither charging nor discharging of the energy storage device, and may also include specific charging and discharging power, which may have different definitions according to different application scenarios.

[0089] In the embodiment of this specification, in step S102, refer to Figure 2 , the action value of each agent in different state-action pairs is sampled according to a preset sampling strategy, including:

[0090] S1: Determine the upper limit of the total number of samples that an agent can take in a certain state.

[0091] In this embodiment, the upper limit N of the total number of sampling times of actions in state s is determined by the time and computing resources for the current agent to make decisions. In some embodiments, the upper limit of the total number of sampling times can be dynamically adjusted according to the historical decision-making speed of the agent and the current system load to ensure efficient use of computing resources.

[0092] S2: Determine the initial sampling strategy of the agent according to the shared action value table.

[0093] In this embodiment, for the current agent i, according to the shared action value table, determining the initial sampling strategy of the agent may include:

[0094] Determining the initial sampling times according to the shared action value table;

[0095] The sampling times of each action of the agent in the current state are set as the initial sampling times.

[0096] It can be understood that when the data in the shared action value table are all initialized data, for example, when the action value mean and action value variance are randomly initialized to 0, the sampling times of all actions of the agent in the current state are set to the initial sampling times n0, and n0 can be determined by computing resources. This is because the action value and instability of each action in the current state cannot be understood based on the initialization data, so the initial sampling times of each action can be set to the same n0. This strategy can ensure that the agent has a certain exploration of all actions in the early stage, avoiding missing other actions that may be better because of being too biased towards a certain action. In the learning process, as the agent's understanding of the value of different actions deepens, the sampling times of each action can be dynamically adjusted according to the actual learning progress and effect. For example, for actions with higher value and relatively stable, the sampling times can be reduced to save resources; for actions with large value fluctuations or actions that have not been fully explored, the sampling times can be increased to obtain more information.

[0097] When there is already a real action value sampling value in the shared action value table, the sampling strategy needs to be determined based on the real action value sampling value. Where S is the state space, so that the initial sampling times n0 of each action can be determined according to the real action value sampling value. For each action a∈Α, according to the strategy d Perform n0 samplings.

[0098] S3: Sampling the action values ​​of different actions of the agent in this state according to the initial sampling strategy to obtain sampling values ​​of the action values ​​of different actions of the agent in this state.

[0099] In this embodiment, for each action a∈Α, where A is the action space, the action value of the action is calculated according to the initial sampling strategy. Perform n0 samplings to obtain the action value sampling value of each action in state s in this round of sampling, including the action value mean and action value variance

[0100] S4: Re-determine the new sampling strategy of the agent based on the action value sampling values ​​of different actions of the agent in this state, re-sample based on the new sampling strategy and obtain new action value sampling values ​​of different actions of the agent in this state.

[0101] S5: Update the action value variance, action value mean and current total sampling times according to the new action value sampling value.

[0102] S6: Determine whether the current total number of sampling times is less than the upper limit of the total number of sampling times;

[0103] If yes, repeat steps S4-S5.

[0104] S7: If not, stop sampling and determine the optimal action in this state.

[0105] In this embodiment, based on the first action value sampling value obtained in the first round of sampling, the action value and uncertainty of each action of agent i in this round of sampling can be obtained, so as to redetermine the new sampling strategy of agent i based on the first action value sampling value. The new sampling strategy will reallocate the sampling times for each action based on the action value sampling value obtained in this round of sampling, that is, increase the sampling times for high-value actions and reduce the sampling times for low-value actions. Agent i performs a new round of sampling on each action in state s according to the new sampling strategy (that is, the sampling times for each action corresponding to the new sampling strategy), re-obtains a new action value sampling value for each action, and updates the action value variance, action value mean and the total number of times the action in state s has been sampled after the new round of sampling for each action in state s according to the new action value sampling value. Afterwards judgement Is it less than the upper limit N of the total number of sampling times of actions in state s? If it is less than, it indicates that there are still computing resources left, and the sampling strategy can be iteratively optimized according to steps S4-S5. If it is greater than, it indicates that there are no available computing resources. At this time, the optimal action in state s is determined according to the mean value of the action value output by the latest round of sampling. In this way, the basic sampling strategy provides a preliminary exploration direction for the agent, and by continuously adjusting the sampling strategy, the agent can more effectively explore the value of different actions, so that when computing resources are limited, the probability of correctly selecting the optimal action is maximized by allocating the number of sampling times for different actions, thereby improving the sampling efficiency when using multi-agent reinforcement learning to solve the supply and demand matching problem of multiple smart grids.

[0106] In the embodiment of this specification, updating the current total number of sampling times according to the new action value sampling value includes:

[0107] Use the following formula to update the current total number of sampling times:

[0108]

[0109] in, is the current total sampling times, a is an action in this state, is the number of times agent i samples action a in this round of sampling, and Δ is the increment of the number of single-step sampling times.

[0110] In the embodiments of this specification, refer to Figure 3 In step S4, the new sampling strategy of the agent is re-determined according to the action value sampling values ​​of different actions of the agent in this state, including:

[0111] S301: Determine the optimal action corresponding to the state in this round of sampling based on the action value sampling values ​​of different actions of the agent in this state.

[0112] S302: Calculate the total number of sampling times after the current round of sampling in this state, the difference between the mean action value of each action and the optimal action, and the action value variance of each action.

[0113] In this embodiment, according to the first action value average Determine the optimal action under this round of sampling, that is, determine the action with the largest mean value of the first action as the optimal action, and then calculate the total number of samplings under state s after this round of sampling, the difference between the mean value of each action and the optimal action, and the variance of the action value of each action according to the following formula:

[0114]

[0115] Among them, δ a is the difference between the mean action value of each action and the optimal action b, is the action value variance of each action, represents the average action value of the optimal action b of agent i in state s sampled in the n+1th round, represents the mean action value of action a of agent i in state s sampled in round n+1, represents the total number of samples taken in state s after this round of sampling, n0 represents the initial number of samples, |A i | represents the action space of agent i in state s, Δ represents the increment of single-step sampling times, and Var represents the variance.

[0116] S303: Calculate the distribution ratio of the number of samplings corresponding to each action according to the total number of samplings after the current round of sampling in this state, the difference between the mean action value of each action and the optimal action, and the action value variance of each action.

[0117] S304: Determine the number of sampling times for each action after the current round of sampling according to the allocation ratio.

[0118] In this embodiment, the distribution ratio of the number of sampling times corresponding to each action is calculated according to the following formula:

[0119]

[0120] Among them, α a is the distribution ratio of the number of sampling times corresponding to each action a, a and a′ represent any pair of non-optimal actions, and b represents the optimal action. represents the number of times agent i samples action a in this round of sampling, is the total number of actions sampled in state s after this round of sampling, δ a is the difference between the mean action value of each action a and the optimal action b, is the action value variance of each action a, It represents the number of times agent i samples the optimal action b in this round of sampling.

[0121] In this embodiment, the following formula is used to calculate the number of sampling times for each action after this round of sampling:

[0122]

[0123] in, is the number of sampling times for each action in this round of sampling, is the current total number of sampling times, α a is the distribution ratio of the number of sampling times corresponding to each action a, Represents the number of times agent i samples action a in this round of sampling.

[0124] After obtaining the required sampling times for each action after this round of sampling, additional sampling is performed on each action according to the required sampling times to obtain the action value sampling value of each action.

[0125] In this embodiment, for each action a, Additional sampling is performed to obtain the action value sampled value of each action, and δ is updated according to the action value sampled value obtained by additional sampling. a and

[0126] In this embodiment, after obtaining the optimal action of agent i in state s, the action is performed in the actual microgrid system, and the state transfer is implemented, the shared action value table is updated, and then the above steps are repeated to iteratively update the sampling strategy, and finally the optimal action for each state is obtained.

[0127] In this embodiment, the specific steps of updating the shared action value table are as follows:

[0128] For each state of each agent, determine whether the shared action value table contains relevant records of the state:

[0129] (1) If there is no record of state s in the shared action value table, then the action value mean of each state-action pair is calculated based on the action value sample value of each state-action pair of the agent in that state. and action value variance The action value of the state-action pair is averaged Action Value Variance And sampling times Recorded on the Shared Actions form.

[0130] (2) Otherwise, record the action value mean, action value variance, and sampling times for the state pair (s, a) in the shared action value table as μ s ' ,a , and n s ' ,a , and update the shared action value table using the following formula:

[0131]

[0132] Among them, μ s,a represents the mean action value of action a in state s, μ s ' ,a represents the mean action value of action a under the updated state s, represents the action value variance of action a in state s, represents the variance of the action value of action a under the updated state s, n s,a represents the total number of sampling times of action a in state s, n s ' ,a represents the total number of sampling times of action a in the updated state s, represents the number of times agent i samples action a in this round of sampling, represents the action value of action a of agent i in state s in the n+1th round of sampling, Var represents the variance, and E represents the expectation. In this way, the embodiment of this specification can gradually maximize the probability of correctly selecting the optimal action by allocating the number of sampling times for different actions under limited computing resources, thereby achieving a more accurate estimation of the action value and accelerating the convergence of the algorithm, thereby improving the sampling efficiency.

[0133] In this embodiment, Figure 4As shown in the figure, the shared action value table is updated independently, synchronously or asynchronously by multiple agents. For another agent j, when the shared action value table has not been updated, sampling can be performed according to the initial shared action value table. If the shared action value table has been updated by other agents, sampling is performed according to the updated shared action value table. In this way, as the number of agents increases, the shared action value table can be continuously improved. Multiple agents share information about which actions are beneficial in which states, thereby increasing data utilization, avoiding repeated calculations and waste of computing resources, and improving sampling efficiency.

[0134] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. And the acquisition, storage, use, and processing of data in the technical solutions described in the embodiments of this application are in compliance with relevant regulations.

[0135] Based on the above-mentioned method for solving the problem of matching supply and demand of multiple smart grids, the embodiment of this specification also provides a device for solving the problem of matching supply and demand of multiple smart grids. The device may include a system (including a distributed system), software (application), module, component, server, client, etc. using the method described in the embodiment of this specification and a device combined with necessary implementation hardware. Based on the same innovative concept, the device in one or more embodiments provided in the embodiment of this specification is as described in the following embodiments. Since the implementation scheme and method of the device to solve the problem are similar, the implementation of the specific device in the embodiment of this specification can refer to the implementation of the aforementioned method, and the repetitions will not be repeated. As used below, the term "unit" or "module" can implement a combination of software and / or hardware with predetermined functions. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0136] Specifically, Figure 5 This is a schematic diagram of the module structure of an embodiment of a multi-smart grid supply and demand matching problem solving device provided in the embodiment of this specification, referring to Figure 5 As shown, an embodiment of this specification provides a device for solving the supply and demand matching problem of multiple smart grids, wherein the action value function and the state transfer matrix of each smart grid are the same, and the device includes:

[0137] A construction module 501 is used to construct a shared action value table for multiple intelligent entities by treating each smart grid as an intelligent entity, wherein the shared action value table includes the action value mean, action value variance and sampling times of existing state-action pairs;

[0138] A sampling module 502 is used to sample the action value of each agent in different state-action pairs according to a preset sampling strategy to obtain a sampling value of the action value of each agent in each state-action pair, wherein the sampling strategy is used to determine the sampling times of each state-action pair according to the action value mean and action value variance;

[0139] An updating module 503 is used to update the shared action value table according to the action value sampling value and sampling times of each agent in each state-action pair;

[0140] The determination module 504 is used to determine the optimal strategy according to the updated shared action value table, and use the optimal strategy as the optimal solution to each smart grid supply and demand matching problem.

[0141] The beneficial effects obtained by the device provided in the embodiments of this specification are consistent with the beneficial effects obtained by the above method and will not be repeated here.

[0142] Reference Figure 6 As shown, based on the above-mentioned method for solving the supply and demand matching problem of multiple smart grids, a computer device 602 is also provided in an embodiment of this specification, wherein the above-mentioned method runs on the computer device 602. The computer device 602 may include one or more processors 604, such as one or more central processing units (CPUs), and each processing unit may implement one or more hardware threads. The computer device 602 may also include any memory 606, which is used to store any kind of information such as code, settings, data, etc. Non-limitingly, for example, the memory 606 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Further, any memory can provide volatile or non-volatile retention of information. Further, any memory can represent a fixed or removable component of the computer device 602. In one case, when the processor 604 executes an associated instruction stored in any memory or combination of memories, the computer device 602 can perform any operation of the associated instruction. The computer device 602 also includes one or more drive mechanisms 608 for interacting with any storage, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0143] The computer device 602 may also include an input / output module 610 (I / O) for receiving various inputs (via input devices 612) and for providing various outputs (via output devices 614). A specific output mechanism may include a presentation device 616 and an associated graphical user interface (GUI) 618. In other embodiments, the input / output module 610 (I / O), input device 612, and output device 614 may not be included, and the computer device 602 may be used as a computer device in a network. The computer device 602 may also include one or more network interfaces 620 for exchanging data with other devices via one or more communication links 622. One or more communication buses 624 couple the components described above together.

[0144] The communication link 622 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 622 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.

[0145] Corresponding to Figures 1 to 4 In addition to the method shown, an embodiment of the specification also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are executed.

[0146] The embodiment of the present specification also provides a computer-readable instruction, wherein when the processor executes the instruction, the program therein causes the processor to execute the following Figures 1 to 4 The method shown.

[0147] The embodiments of the present specification also provide a computer program product, including at least one instruction or at least one program, wherein the at least one instruction or the at least one program is loaded and executed by a processor to implement the following Figures 1 to 4 The method shown.

[0148] It should be understood that in the various embodiments of this specification, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this specification.

[0149] It should also be understood that in the embodiments of this specification, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this specification generally indicates that the associated objects before and after are in an "or" relationship.

[0150] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.

[0151] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0152] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0153] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this specification.

[0154] In addition, each functional unit in each embodiment of this specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0155] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0156] Specific embodiments are used in this specification to illustrate the principles and implementation methods of this specification. The description of the above embodiments is only used to help understand the methods and core ideas of this specification. At the same time, for those skilled in the art, according to the ideas of this specification, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on this specification.

Claims

1. A method for solving the supply and demand matching problem of multiple smart grids, characterized in that: The action value function and state transfer matrix of each smart grid are the same, and the method includes: Taking each smart grid as an intelligent agent, constructing a shared action value table for multiple intelligent agents, wherein the shared action value table includes the action value mean, action value variance and sampling times of existing state-action pairs; The action value of each agent in different state-action pairs is sampled according to a preset sampling strategy to obtain a sampling value of the action value of each agent in each state-action pair, wherein the sampling strategy is used to determine the sampling times of each state-action pair according to the action value mean and action value variance; Update the shared action value table according to the action value sampling value and sampling times of each agent in each state-action pair; The optimal strategy is determined according to the updated shared action value table, and the optimal strategy is used as the optimal solution to each smart grid supply and demand matching problem.

2. The method according to claim 1, characterized in that The sampling of the action value of each agent in different state-action pairs according to a preset sampling strategy includes: S1: Determine the upper limit of the total number of sampling times for an agent to act in a certain state; S2: Determine the initial sampling strategy of the agent according to the shared action value table; S3: sampling the action values ​​of different actions of the agent in this state according to the initial sampling strategy to obtain sampling values ​​of the action values ​​of different actions of the agent in this state; S4: re-determine a new sampling strategy for the agent based on the action value sampling values ​​of different actions of the agent in this state, re-sample based on the new sampling strategy and obtain new action value sampling values ​​of different actions of the agent in this state; S5: updating the action value variance, action value mean and current total sampling times according to the new action value sampling value; S6: Determine whether the current total number of sampling times is less than the upper limit of the total number of sampling times. If so, repeat steps S4-S5; S7: If not, stop sampling and determine the optimal action in this state.

3. The method according to claim 2, characterized in that Determining the initial sampling strategy of the agent according to the shared action value table includes: Determining the initial sampling times according to the shared action value table; The sampling times of each action of the agent in the current state are set as the initial sampling times.

4. The method according to claim 2, characterized in that: The re-determining a new sampling strategy of the agent according to the action value sampling values ​​of different actions of the agent in this state includes: Determine the optimal action corresponding to the state in this round of sampling according to the action value sampling values ​​of different actions of the agent in this state; Calculate the total number of samplings after this round of sampling in this state, the difference between the mean action value of each action and the optimal action, and the variance of the action value of each action; According to the total number of samplings after this round of sampling in this state, the difference between the mean action value of each action and the optimal action, and the variance of the action value of each action, the distribution ratio of the number of samplings corresponding to each action is calculated; The number of sampling times for each action after the current round of sampling is determined according to the allocation ratio.

5. The method according to claim 4, characterized in that The distribution ratio of the number of samplings corresponding to each action is calculated based on the total number of samplings after the current round of sampling in this state, the difference between the mean value of the action value of each action and the optimal action, and the variance of the action value of each action, including: The following formula is used to calculate the distribution ratio of sampling times for each action: Among them, α a is the distribution ratio of the number of sampling times corresponding to each action a, a and a′ represent any pair of non-optimal actions, and b represents the optimal action. represents the number of times agent i samples action a in this round of sampling, is the total number of actions sampled in state s after this round of sampling, δ a is the difference between the mean action value of each action a and the optimal action b, is the action value variance of each action a, is the action value variance of the optimal action b, It represents the number of times agent i samples the optimal action b in this round of sampling.

6. The method according to claim 2, characterized in that The updating of the current total number of sampling times according to the new action value sampling value comprises: Use the following formula to update the current total number of sampling times: in, is the current total sampling times, a is an action in this state, is the number of times agent i samples action a in this round of sampling, and Δ is the increment of the number of single-step sampling times.

7. The method according to claim 1, characterized in that The shared action value table is updated independently, synchronously or asynchronously by multiple agents.

8. The method according to claim 1, characterized in that The updating of the shared action value table according to the action value sampling value and sampling times of each agent in each state-action pair includes: For each state of each agent, determining whether the shared action value table contains relevant records of the state; If not, then calculate the action value mean and action value variance of the state-action pair according to the action value sampling value of each state-action pair of the agent in the state; record the action value mean, action value variance and sampling times of the state-action pair in the shared action table; If yes, the shared action value table is updated using the following formula: Among them, μ s,a represents the mean action value of action a in state s, μ s ' ,a represents the mean action value of action a under the updated state s, represents the action value variance of action a in state s, represents the variance of the action value of action a under the updated state s, n s,a represents the total number of sampling times of action a in state s, n s ' ,a represents the total number of sampling times of action a in the updated state s, represents the number of times agent i samples action a in this round of sampling, It represents the action value of agent i in state s when taking action a in the n+1th round, Var represents the variance, and E represents the expectation.

9. A device for solving the problem of supply and demand matching of multiple smart grids, characterized in that: The action value function and state transfer matrix of each smart grid are the same, and the device comprises: A construction module, used to take each smart grid as an intelligent agent and construct a shared action value table for multiple intelligent agents, wherein the shared action value table includes the action value mean, action value variance and sampling times of the existing state-action pairs; A sampling module is used to sample the action value of each agent in different state-action pairs according to a preset sampling strategy to obtain the action value sampling value of each agent in each state-action pair, wherein the sampling strategy is used to determine the sampling times of each state-action pair according to the action value mean and action value variance; An updating module, configured to update the shared action value table according to the action value sampling value and sampling times of each agent in each state-action pair; The determination module is used to determine the optimal strategy according to the updated shared action value table, and use the optimal strategy as the optimal solution to each smart grid supply and demand matching problem.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product, characterized in that The method comprises at least one instruction or at least one program, wherein the at least one instruction or the at least one program is loaded and executed by a processor to implement the method according to any one of claims 1 to 8.

Images