Method and device for determining micro-grid planning scheme, storage medium and electronic equipment

Through the deep Q-learning algorithm based on priority experience replay training of the target Markov decision-making process model, the problem of low accuracy in microgrid planning is solved, and a more accurate microgrid power capacity planning is achieved.

CN118944073APending Publication Date: 2024-11-12GUANGDONG POWER GRID CO LTD +1
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410992571.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The accuracy of the existing technology in microgrid planning is low in the accuracy of determining the scheme, and it is difficult to reasonably plan the power capacity of fans, photovoltaics and other power sources.

Method used

The target Markov decision-making process model is trained using a deep Q-learning algorithm based on priority experience playback. By obtaining the status data of the microgrid, inputting it into the target Markov decision-making process model, and determining the installation capacity planning scheme of the microgrid.

Benefits of technology

It improves the accuracy of the microgrid planning scheme, reduces the randomness of empirical sampling, and enhances the learning stability and processing accuracy of the model in complex environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118944073A_ABST
    Figure CN118944073A_ABST
Patent Text Reader

Abstract

The invention discloses a microgrid planning scheme determination method and device, a storage medium and electronic equipment. The method comprises the steps that state data of a micro-grid are acquired, and the state data at least comprise a system load power value, an electricity value of the micro-grid purchasing electricity from a power distribution network, an electricity value of the micro-grid selling electricity to the power distribution network and a charge state value of micro-grid energy storage; the state data of the micro-grid are input to a target Markov decision process model, a planning scheme of the installation capacity of the micro-grid is obtained, the target Markov decision process model is obtained by training an initial Markov decision process model based on a depth Q learning algorithm of priority experience playback, and the installation capacity of the micro-grid is obtained. The initial Markov decision process model is constructed based on a micro-grid multi-target planning model. According to the invention, the technical problem of low accuracy of the determined micro-grid planning scheme in the prior art is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of microgrid planning, and in particular to a method, device, storage medium and electronic device for determining a microgrid planning scheme. Background Art

[0002] Microgrid planning is based on meeting system load requirements, and the optimal solution is obtained by planning the power type and capacity. Due to its unique geographical location, microgrids are often very complicated when planning capacity. How to reasonably plan the capacity of power sources such as wind turbines and photovoltaics is the primary problem that needs to be solved in microgrid planning. At present, related technologies have the problem of low accuracy when determining microgrid planning solutions.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present invention provide a method, device, storage medium and electronic device for determining a microgrid planning scheme, so as to at least solve the technical problem of low accuracy of the microgrid planning scheme determined in the related art.

[0005] According to one aspect of an embodiment of the present invention, a method for determining a microgrid planning scheme is provided, comprising: obtaining status data of the microgrid, wherein the status data at least includes: a system load power value, the value of electricity purchased by the microgrid from the distribution network, the value of electricity sold by the microgrid to the distribution network, and a state of charge value of energy storage in the microgrid; inputting the status data of the microgrid into a target Markov decision process model to obtain a planning scheme for the installation capacity of the microgrid, wherein the target Markov decision process model is obtained by training an initial Markov decision process model based on a deep Q learning algorithm with priority experience replay, and the initial Markov decision process model is constructed based on a multi-objective planning model of the microgrid.

[0006] Furthermore, the method for determining the microgrid planning scheme also includes: determining the state space according to the parameters of the objective function in the multi-objective planning model and the parameters in the constraint conditions, wherein the parameters in the objective function include at least: the system load power, the electricity price at which the microgrid purchases electricity from the distribution network, and the electricity price at which the microgrid sells electricity to the distribution network, and the parameters in the constraint conditions include at least: the charge state of the microgrid energy storage; determining the action space according to the parameters to be solved in the multi-objective planning model, wherein the parameters to be solved include: the installation capacity of multiple devices required for the microgrid; determining the reward function according to the objective function in the multi-objective planning model, wherein the reward function is used to calculate the reward obtained after performing an action on the state; and constructing an initial Markov decision process model according to the state space, the action space, and the reward function.

[0007] Furthermore, the method for determining the microgrid planning scheme also includes: obtaining the current state of the state space through the intelligent agent in the initial Markov decision process model, and generating an experience tuple based on the current state, wherein the experience tuple includes the current state, the current action, the reward obtained by executing the current action on the current state, and the next state obtained by executing the current action on the current state; storing the experience tuple in the experience replay pool based on the priority experience replay by the intelligent agent, and sampling the updated experience replay pool to obtain the target experience tuple; processing the target experience tuple based on the Q network by the intelligent agent, calculating the loss function value based on the processing result, and updating the network parameter value of the Q network according to the loss function value, wherein the Q network is used to calculate the reward value obtained after executing the action on the state, and the target strategy in the intelligent agent is updated as the network parameter value of the Q network is updated; when the Q network does not meet the preset iteration condition, the intelligent agent determines the next state in the target experience tuple as the updated current state of the state space, and repeatedly executes the step of generating the experience tuple according to the updated current state until the preset iteration condition is met to obtain the target Markov decision process model.

[0008] Furthermore, the method for determining the microgrid planning scheme also includes: determining a current action from the action space based on the current state; executing the current action on the current state to obtain a reward and a next state corresponding to the current state; generating an experience tuple based on the current state, the current action, the reward obtained by executing the current action on the current state, and the next state.

[0009] Furthermore, the method for determining the microgrid planning scheme also includes: processing the current state and the current action in the experience tuple based on the Q network to obtain a first reward value; processing the next state and the first target action in the experience tuple based on the target Q network to obtain a second reward value, wherein the target Q network is a delayed updated copy of the Q network; determining the sampling probability of the experience tuple based on the first reward value and the second reward value; storing the experience tuple in the experience replay pool based on the sampling probability, and sampling according to the sampling probability of each experience tuple in the updated experience replay pool to obtain the target experience tuple.

[0010] Furthermore, the method for determining the microgrid planning scheme also includes: processing the current state and the current action in the target experience tuple according to the Q network to obtain a third reward value; processing the next state and the second target action in the target experience tuple according to the target Q network to obtain a fourth reward value, wherein the target Q network is a delayed update copy of the Q network; calculating the fifth reward value based on the discount factor, the fourth reward value and the reward in the target experience tuple; substituting the third reward value and the fifth reward value into the minimization mean square error function to calculate the loss function value.

[0011] Furthermore, the method for determining the microgrid planning scheme also includes: after updating the network parameter value of the Q network according to the loss function value, determining the iteration round of the last update of the network parameter value of the target Q network to obtain the target iteration round; judging whether the number of rounds between the current iteration round and the target iteration round reaches the preset number of rounds; when the number of rounds between the current iteration round and the target iteration round reaches the preset number of rounds, updating the network parameter value of the target Q network according to the network parameter value of the Q network.

[0012] According to another aspect of an embodiment of the present invention, a device for determining a microgrid planning scheme is also provided, including: an acquisition module, used to acquire status data of the microgrid, wherein the status data at least includes: a system load power value, the value of electricity purchased by the microgrid from the distribution network, the value of electricity sold by the microgrid to the distribution network, and a charge state value of the microgrid energy storage; a first processing module, used to input the status data of the microgrid into a target Markov decision process model to obtain a planning scheme for the installation capacity of the microgrid, wherein the target Markov decision process model is obtained by training an initial Markov decision process model based on a deep Q learning algorithm with priority experience replay, and the initial Markov decision process model is constructed based on a multi-objective planning model of the microgrid.

[0013] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned method for determining a microgrid planning scheme when running.

[0014] According to another aspect of an embodiment of the present invention, an electronic device is also provided, the electronic device comprising one or more processors; a memory for storing one or more programs, which, when the one or more programs are executed by the one or more processors, enables the one or more processors to run the programs, wherein the programs are configured to execute the above-mentioned method for determining the microgrid planning scheme when running.

[0015] In an embodiment of the present invention, a target Markov decision process model obtained by training a deep Q learning algorithm based on priority experience replay is used to determine a microgrid planning scheme. By acquiring the state data of the microgrid, the state data of the microgrid is input into the target Markov decision process model to obtain a planning scheme for the installation capacity of the microgrid. The state data at least includes: system load power value, the value of electricity purchased by the microgrid from the distribution network, the value of electricity sold by the microgrid to the distribution network, and the state of charge value of the microgrid energy storage. The target Markov decision process model is obtained by training the initial Markov decision process model based on the deep Q learning algorithm based on priority experience replay. The initial Markov decision process model is constructed based on the multi-objective planning model of the microgrid.

[0016] In the above process, an initial Markov decision process model is constructed by a multi-objective planning model based on a microgrid, which facilitates the solution of the multi-objective planning model of the microgrid based on the Markov decision process model. Since the priority experience replay assigns a priority to each experience according to the importance of the experience, and selects experience for model training according to this priority, the target Markov decision process model is obtained by training the deep Q learning algorithm based on priority experience replay, which can effectively reduce the randomness of experience sampling during the training process, thereby helping the model to better learn generalized knowledge from experience, and helping the model to improve the stability of the learning process in a complex environment, thereby improving the processing accuracy of the target Markov decision process model and the accuracy of the determined microgrid planning scheme.

[0017] It can be seen that the solution provided in the present application achieves the purpose of determining the microgrid planning scheme based on the target Markov decision process model obtained by training the deep Q learning algorithm with priority experience replay, thereby achieving the technical effect of improving the accuracy of the determined microgrid planning scheme, and further solving the technical problem of low accuracy of the microgrid planning scheme determined in the related technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0019] Figure 1 is a schematic diagram of an optional method for determining a microgrid planning scheme according to an embodiment of the present invention;

[0020] Figure 2 is a schematic diagram of a corresponding relationship between an optional planning scheme and a microgrid working effect according to an embodiment of the present invention;

[0021] Figure 3 is a schematic diagram of an optional target determination Markov decision process model according to an embodiment of the present invention;

[0022] Figure 4 is a schematic diagram of an optional device for determining a microgrid planning scheme according to an embodiment of the present invention;

[0023] Figure 5 is a schematic diagram of an optional electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards in the relevant regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0027] Example 1

[0028] According to an embodiment of the present invention, an embodiment of a method for determining a microgrid planning scheme is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0029] Figure 1 is a schematic diagram of an optional method for determining a microgrid planning scheme according to an embodiment of the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0030] Step S101, obtaining the status data of the microgrid, wherein the status data at least includes: the system load power value, the value of electricity purchased by the microgrid from the distribution network, the value of electricity sold by the microgrid to the distribution network, and the charge state value of the microgrid energy storage.

[0031] Optionally, electronic devices, application systems, servers and other devices may be used as the execution subject of the present application. In this embodiment, the target processing system is used as the execution subject to obtain the status data of the microgrid.

[0032] In some embodiments, the state data of the microgrid includes at least the system load power value P L (t), the value of electricity purchased by the microgrid from the distribution network ρ in (t), the value of electricity sold by the microgrid to the distribution network ρ out (t), the state of charge value SOC(t) of the microgrid energy storage. The state data of the microgrid can be input into the target processing system by the staff for acquisition by the target processing system.

[0033] Step S102, inputting the state data of the microgrid into the target Markov decision process model to obtain a planning scheme for the installation capacity of the microgrid, wherein the target Markov decision process model is obtained by training the initial Markov decision process model based on the deep Q learning algorithm with priority experience replay, and the initial Markov decision process model is constructed based on the multi-objective planning model of the microgrid.

[0034] In some embodiments, the target processing system can input the state data of the microgrid into the target Markov decision process model to obtain a planning scheme for the installation capacity of the microgrid, wherein the target Markov decision process is obtained by training the initial Markov decision process model, and the initial Markov decision process model is constructed based on the multi-objective planning model of the microgrid.

[0035] Optionally, the economic performance and power supply reliability of the microgrid are comprehensively considered to construct a multi-objective planning model that minimizes the total annual cost of the microgrid system and minimizes the annual load power outage rate. In some embodiments, the power loss inside the microgrid is ignored, and a multi-objective planning model for the economic and reliable microgrid is established with 365 days a year as a cycle T. The planning objectives include minimizing the total cost of the microgrid system and minimizing the annual load power outage rate. The objective function and constraints in the multi-objective planning model can be as follows:

[0036] 1. Objective function f1

[0037] The objective function f1 is expressed as minimizing the total annual cost of the microgrid system. The total annual cost includes the initial equipment investment cost, operation and maintenance cost, interaction cost between the microgrid and the distribution network, and environmental treatment cost.

[0038] f1=min(C inv +C om +C g +C en )

[0039]

[0040] Among them, C inv The initial equipment investment cost of the construction; C om is the annual operation and maintenance cost; C g is the annual interaction cost between the microgrid and the distribution network; C en is the annual cost of environmental treatment; i is wind turbine, diesel engine, photovoltaic panel, and energy storage battery; c is i is the unit capacity investment cost of the i-th type of equipment, which includes the purchase cost and installation cost; S i is the installed capacity of the i-th type of equipment; k i is the annual maintenance and operation cost per unit capacity of the i-th type of equipment; ρ in (t),ρ out (t) are the electricity prices of the microgrid purchased from the distribution network and the electricity prices of the microgrid purchased from the distribution network during period t; P in (t), P out (t) are the power purchased by the microgrid from the distribution network and the power sold by the microgrid to the distribution network during period t; j is the pollutant gas CO2, SO2, NO x Types; C j is the processing cost of the jth gas; m j (t) is the mass of polluted gas generated by the microgrid during period t. In some embodiments, the length of period t is one day.

[0041] 2. Objective function f2

[0042] The annual load power shortage rate is used to evaluate the power supply reliability of the microgrid, and the objective function f2 is the minimum annual load power shortage rate μ. The annual load power shortage rate is the ratio of the annual load power shortage to the annual load power. The smaller the value, the smaller the system power shortage and the higher the power supply reliability.

[0043]

[0044] Among them, μ is the annual load power shortage rate; P L (t) is the load power during period t; LPS(t) is the system power shortage during period t.

[0045] 3. Constraints

[0046] The constraints of the multi-objective planning model of the microgrid include system capacity constraints, equipment operation constraints, power balance constraints, electricity purchase and sales price constraints, and power supply reliability constraints.

[0047] (1) System capacity constraints

[0048] P imin ≤P i ≤P i max

[0049] P WT +P PV +P DE +P BES ≥P L (t)

[0050] Among them, P i is the power corresponding to the installed capacity of the i-th type of equipment; P i min , P i max are the maximum power and minimum power corresponding to the installed capacity of the i-th type of equipment respectively; P WT , P PV , P DE , P BES They are the powers corresponding to the installed capacities of wind turbines, diesel engines, photovoltaic panels and energy storage batteries.

[0051] (2) Equipment operation constraints

[0052] P WTmin ≤P WT (t)≤P WTmax

[0053] P DEmin ≤P DE (t)≤P DEmax

[0054] P PVmin ≤P PV (t)≤P PVmax

[0055]

[0056] SOC min ≤SOC(t)≤SOC max

[0057] SOC(0)=SOC(t′)

[0058] Among them, P WT (t) is the output power of the fan during period t; P WTmax , P WTmin P is the upper and lower limits of the fan output power; DE (t) is the output power of the diesel engine during period t; P DEmax , P DEmin The upper and lower limits of diesel engine output power; P PV (t) is the output power of photovoltaic power generation during period t; P PVmax, P PVmin are the upper and lower limits of the photovoltaic output power; SOC(t) is the state of charge of the energy storage during the period t; SOC(t+Δt) is the state of charge of the energy storage during the period t+Δt; P ch (t), P dis (t) is the charging and discharging power of the energy storage during the period t; η c , η d SOC is the charging and discharging efficiency of energy storage; max , SOC min SOC(0) is the state of charge of the energy storage at time 0; SOC(t') is the state of charge of the energy storage at time t' in the t period. In some embodiments, t' is the 24th hour.

[0059] (3) Power balance constraints

[0060] The power of the microgrid system meets the following balance constraints:

[0061] P WT (t)+P PV (t)+P DE (t)+P dis (t)+P in (t) = P ch (t)+P L (t)+P out (t)

[0062] (4) Constraints on electricity purchase and sales prices

[0063] The electricity purchase and sale price ρ(t) between the microgrid and the distribution network satisfies the following relationship:

[0064] ρ out (t)<ρ(t)<ρ in (t)

[0065] Among them, ρ in (t),ρ out (t) are the electricity prices that the microgrid purchases from the distribution network and the electricity prices that the microgrid purchases from the distribution network during period t, respectively.

[0066] (5) Power supply reliability constraints

[0067] In order to ensure the feasibility and practical significance of the multi-objective planning model constructed by the present invention and to ensure the power supply reliability of the microgrid, the annual load power shortage rate μ should satisfy:

[0068] μ≤μ max

[0069] Among them, μ max It is the maximum load power failure rate allowed throughout the year during power supply.

[0070] In some embodiments, the target processing system can determine the state space according to the objective function and constraints in the multi-objective programming model, determine the action space according to the parameters to be solved in the multi-objective programming model, and thus construct an initial Markov decision process model according to the state space and the action space. After the target Markov decision process model is trained, the target processing system can input the above-mentioned state data as the state into the target Markov decision process model. The target Markov decision process model determines the action to be performed in the state from the action space according to the state, that is, determines the parameter value of the parameter to be solved, and outputs the result. Therefore, the action output by the target Markov decision process model can be determined as the planning scheme for the installation capacity of the microgrid.

[0071] In some embodiments, in the process of training to obtain the target Markov decision process model, the target processing system can train the initial Markov decision process model based on the deep Q learning algorithm of priority experience replay to obtain the target Markov decision process model. The core idea of ​​priority experience replay is to assign a priority to each experience according to its importance, and select experience for model training according to this priority. In this way, experiences with higher priorities are more likely to be selected, thereby effectively improving the efficiency of model learning.

[0072] In some embodiments, the parameters to be solved in the multi-objective planning model are described. As shown in Table 1, three planning schemes for setting the installation capacity of the microgrid are as follows: Scheme 1: wind turbine / photovoltaic / energy storage; Scheme 2: wind turbine / diesel engine / photovoltaic; Scheme 3: wind turbine / diesel engine / photovoltaic / energy storage. The planned installation capacity of wind turbines, diesel engines, photovoltaics and energy storage under the three schemes are shown in Table 1. In Scheme 1, renewable energy output is greater in order to improve energy consumption capacity; in Scheme 3, the capacity of the diesel engine has decreased, and the investment in energy storage plays an important role in valley cutting and peak filling in the microgrid.

[0073] Table 1

[0074] plan Fan / kW Diesel engine / kW Photovoltaic / kW Energy storage / kW 1 2500 0 560 390 2 2400 120 300 0 3 2400 100 240 120

[0075] Figure 2 is a schematic diagram of the corresponding relationship between an optional planning scheme according to an embodiment of the present invention and the working effect of the microgrid, such as Figure 2 As shown in Table 1, the microgrid under Scheme 1 is not equipped with diesel engines, and the total annual cost of the microgrid is the highest, with a total cost of 28.1319 million yuan. This is because the planned capacity of photovoltaic and energy storage is larger than that of the latter three schemes, resulting in increased investment costs. Due to the lack of stable power supply from diesel engines, the system has the highest load power shortage rate throughout the year. Therefore, planning a certain number of diesel engines in the microgrid can greatly improve the economy and power supply reliability of the microgrid. According to Figure 2Comparing Scheme 2 and Scheme 3 in Table 1, although the planning object of Scheme 3 has an additional energy storage device and the investment cost increases by 50,000 yuan, the power generation of the diesel engine decreases due to the use of the energy storage device, which reduces the emission of polluting gases and the environmental treatment cost. The final annual total cost is reduced by 2.84%, and the annual load power shortage rate is reduced by 8.57%. Therefore, adding energy storage to the planning scheme helps to increase the penetration rate of new energy, reduce environmental pollution and improve power supply reliability. Among the three planning schemes, the annual total cost of Scheme 3 is 21.39% and 2.84% lower than that of Schemes 1 and 2, respectively. Therefore, when planning a microgrid, considering wind turbines, diesel engines, photovoltaics and energy storage equipment can improve the economy of the system while ensuring the reliability of the system power supply. Therefore, the installed capacity of wind turbines, diesel engines, photovoltaics and energy storage batteries in the microgrid can be determined as the parameters to be solved in the multi-objective planning model of the microgrid.

[0076] Based on the scheme defined by the above steps S101 to S102, it can be known that in an embodiment of the present invention, a target Markov decision process model obtained by training a deep Q learning algorithm based on priority experience replay is used to determine the microgrid planning scheme, and the state data of the microgrid is obtained, thereby inputting the state data of the microgrid into the target Markov decision process model to obtain a planning scheme for the installation capacity of the microgrid. Among them, the state data at least includes: system load power value, the value of electricity purchased by the microgrid from the distribution network, the value of electricity sold by the microgrid to the distribution network, and the state of charge value of the microgrid energy storage. The target Markov decision process model is obtained by training the initial Markov decision process model based on the deep Q learning algorithm based on priority experience replay, and the initial Markov decision process model is constructed based on the multi-objective planning model of the microgrid.

[0077] It is easy to notice that in the above process, the initial Markov decision process model is constructed by the multi-objective planning model based on the microgrid, which facilitates the solution of the multi-objective planning model of the microgrid based on the Markov decision process model. Since the priority experience replay assigns a priority to each experience according to the importance of the experience, and selects the experience to train the model according to this priority, the target Markov decision process model is obtained by training the deep Q learning algorithm based on the priority experience replay, which can effectively reduce the randomness of experience sampling during the training process, thereby helping the model to better learn generalized knowledge from experience, and helping the model to improve the stability of the learning process in a complex environment, thereby improving the processing accuracy of the target Markov decision process model and improving the accuracy of the determined microgrid planning scheme.

[0078] It can be seen that the solution provided in the present application achieves the purpose of determining the microgrid planning scheme based on the target Markov decision process model obtained by training the deep Q learning algorithm with priority experience replay, thereby achieving the technical effect of improving the accuracy of the determined microgrid planning scheme, and further solving the technical problem of low accuracy of the microgrid planning scheme determined in the related technology.

[0079] In an optional embodiment, Figure 3 is a schematic diagram of an optional target determination Markov decision process model according to an embodiment of the present invention, such as Figure 3 As shown, the initial Markov decision process model is obtained in the following manner: determine the state space according to the parameters of the objective function in the multi-objective programming model and the parameters in the constraint conditions, wherein the parameters in the objective function include at least: system load power, the price of electricity purchased by the microgrid from the distribution network, and the price of electricity sold by the microgrid to the distribution network, and the parameters in the constraint conditions include at least: the charge state of the microgrid energy storage; determine the action space according to the parameters to be solved in the multi-objective programming model, wherein the parameters to be solved include: the installation capacity of multiple devices required for the microgrid; determine the reward function according to the objective function in the multi-objective programming model, wherein the reward function is used to calculate the reward obtained after performing an action on the state; construct the initial Markov decision process model according to the state space, the action space and the reward function.

[0080] Optionally, in this embodiment, the state space may be expressed as:

[0081] S=[SOC(t),P L (t),ρ in (t),ρ out (t)]

[0082] In some embodiments, the state space includes multiple states, and one state is composed of the parameter values ​​corresponding to the parameters of the above-mentioned objective function and the parameters in the constraint conditions. After the parameters are determined, the target processing system can determine the parameter values ​​that can be selected for each parameter according to the parameter value selection rules corresponding to each parameter, thereby determining multiple states of the state space according to the parameter values ​​that can be selected for each parameter. The parameter values ​​corresponding to different states are not exactly the same.

[0083] Optionally, in this embodiment, the action space can be expressed as:

[0084] A=[S WT ,S PV ,S DE ,S BES ]

[0085] That is, the multiple devices required for the microgrid are wind turbines, diesel engines, photovoltaic panels, and energy storage batteries. In some embodiments, the action space includes multiple actions, and one action is composed of the parameter values ​​corresponding to the above-mentioned parameters to be solved. After the parameters to be solved are determined, the target processing system can determine the parameter values ​​that can be selected for the parameters to be solved according to the parameter value selection rules corresponding to the parameters to be solved, thereby determining multiple actions in the action space according to the parameter values ​​that can be selected for the parameters to be solved. The parameter values ​​corresponding to different actions are not exactly the same.

[0086] In some embodiments, in the microgrid state st, the agent in the Markov decision process model performs action a t Then the reward R is obtained. The purpose of the multi-objective planning of the microgrid is to minimize the total annual cost of the microgrid system and the annual load power shortage rate. Therefore, in this embodiment, the reward function can be expressed as:

[0087] R=-(C inv +C om +C g +C en +βμ)

[0088] Among them, β is the penalty coefficient of the microgrid's annual load power shortage rate.

[0089] After determining the state space, action space and reward function, the target processing system can construct an initial Markov decision process model based on the state space, action space and reward function.

[0090] It should be noted that through the above process, an effective conversion of the multi-objective planning model is achieved, so that the target Markov decision process model obtained by training the initial Markov decision process model is essentially equivalent to solving the planning scheme based on the state data in the microgrid when solving the action according to the state, thereby improving the stability and accuracy of determining the planning scheme.

[0091] In an optional embodiment, the target Markov decision process model can be trained in the following manner: the agent in the initial Markov decision process model obtains the current state of the state space, and generates an experience tuple based on the current state, wherein the experience tuple includes the current state, the current action, the reward obtained by executing the current action on the current state, and the next state obtained by executing the current action on the current state; the agent stores the experience tuple in the experience replay pool based on the priority experience replay, and samples the updated experience replay pool to obtain the target experience tuple; the agent processes the target experience tuple based on the Q network, calculates the loss function value based on the processing result, and updates the network parameter value of the Q network according to the loss function value, wherein the Q network is used to calculate the reward value obtained after executing the action on the state, and the target strategy in the agent is updated as the network parameter value of the Q network is updated; when the Q network does not meet the preset iteration condition, the agent determines the next state in the target experience tuple as the updated current state of the state space, and repeats the step of generating the experience tuple based on the updated current state until the preset iteration condition is met, thereby obtaining the target Markov decision process model.

[0092] Optionally, after the initial Markov decision process model is constructed, the target processing system can initialize the Q network and the experience replay pool through the agent in the initial Markov decision process model. The experience replay pool is used to store a series of experience tuples.

[0093] Afterwards, if Figure 3 As shown in the figure, the agent can obtain the current state of the state space. In the first iteration, the current state of the state space can be a random state in the state space or a state specified by the user. After that, the agent can generate an experience tuple based on the current state. For example, the current action is determined based on the current state, and then the current action is performed on the current state to obtain the reward and the next state.

[0094] Then, if Figure 3 As shown, the agent can store the experience tuples in the experience replay pool based on the priority experience replay, and sample the updated experience replay pool to obtain the target experience tuples. For example, the agent determines the sampling probability of the experience tuple, and then stores the experience tuples in the experience replay pool based on the sampling probability, and samples based on the sampling probability of each experience tuple in the updated experience replay pool to obtain the target tuple. Among them, the priority experience replay is to assign a priority to each experience according to the importance of the experience, and select the experience for model training according to this priority, that is, the agent implements the process of storing the experience tuples in the experience replay pool and the process of sampling the updated experience replay pool based on the priority experience replay.

[0095] Optional, such as Figure 3 As shown in FIG, after the target experience tuple is determined, the target experience tuple can be processed by the agent based on the Q network, the loss function value is calculated based on the processing result, and the network parameter value of the Q network is updated according to the loss function value. For example, the agent calculates the reward value obtained after performing the current action on the current state through the Q network, and the reward value is the processing result, so as to calculate the loss function value according to the reward value, and update the network parameter value of the Q network according to the loss function value.

[0096] Optionally, the target strategy in the intelligent agent is updated as the network parameter values ​​of the Q network are updated. The target strategy is used to select actions from the action space according to the state, that is, the target strategy is used to determine the planning scheme for the installation capacity of the microgrid according to the above-mentioned state data.

[0097] In some embodiments, the preset iteration condition is that the number of iterations of the Q network reaches a preset number of iterations. Figure 3 As shown, when the number of iterations of the Q network does not reach the preset number of iterations, the intelligent agent can determine the next state in the target experience tuple as the updated current state of the state space, and repeat the step of generating a new experience tuple according to the updated current state until the number of iterations of the Q network reaches the preset number of iterations, thereby obtaining the target Markov decision process model.

[0098] It should be noted that, through the above process, the deep Q learning algorithm based on priority experience replay is implemented to effectively train the initial Markov decision process model, thereby effectively improving the model performance of the target Markov decision process model.

[0099] In an optional embodiment, in the process of generating an experience tuple based on the current state, the intelligent agent can determine the current action from the action space based on the current state, and then perform the current action on the current state to obtain a reward and the next state corresponding to the current state, thereby generating an experience tuple based on the current state, the current action, the reward obtained by performing the current action on the current state, and the next state.

[0100] In some embodiments, the agent determines the current action from the action space according to the target strategy and the current state. Optionally, in the first iteration round, the target strategy is a greedy strategy, and the agent can determine the current action from the action space according to the greedy strategy and the current state. For example, the agent evaluates the reward that can be obtained by executing each action in the action space under the current state, and thus determines the action with the highest corresponding reward as the current action.

[0101] Optionally, in the Nth iteration, the target strategy is updated as the network parameter value of the Q network is updated, and the target strategy may no longer be a greedy strategy. Wherein N is a positive integer greater than 1.

[0102] After determining the current action, the agent performs the current action at on the current state st and obtains the reward R and the next state s t+ 1. Thus, the experience tuple is generated based on the current state, the current action, the reward obtained by performing the current action on the current state, and the next state. That is, the experience tuple can be expressed as (s t ,a t ,r t ,s t+1 ).

[0103] It should be noted that through the above process, accurate generation of experience tuples is achieved, which facilitates effective model training and improves model performance.

[0104] In an optional embodiment, in the process of storing experience tuples in an experience replay pool based on priority experience replay by the intelligent agent, and sampling the updated experience replay pool to obtain a target experience tuple, the intelligent agent can process the current state and current action in the experience tuple based on the Q network to obtain a first reward value, and then process the next state and the first target action in the experience tuple based on the target Q network to obtain a second reward value, wherein the target Q network is a delayed updated copy of the Q network, and then determine the sampling probability of the experience tuple based on the first reward value and the second reward value, thereby storing the experience tuple in the experience replay pool based on the sampling probability, and sampling is performed according to the sampling probability of each experience tuple in the updated experience replay pool to obtain the target experience tuple.

[0105] In some embodiments, the agent can process the current state and current action in the experience tuple based on the Q network to obtain a first Q value, that is, a first reward value.

[0106] In some embodiments, the first target action may be the current action in the experience tuple.

[0107] In some embodiments, the first target action may also be selected by the agent from the action space. For example, for each action in the action space, the agent may process the next state in the experience tuple and the action based on the Q network to obtain the first candidate reward value corresponding to the action. Afterwards, the agent may determine the action with the highest first candidate reward value as the first target action, and then process the next state in the experience tuple and the first target action based on the target Q network to obtain the second reward value.

[0108] In some embodiments, the target Q network is a delayed update copy of the Q network, and the agent will also initialize the target Q network when initializing the Q network and the experience replay pool. Optionally, the agent periodically (e.g., every C time steps) copies the network parameter values ​​of the Q network to the target Q network in a certain proportion to achieve delayed update of the parameter values ​​of the target Q network.

[0109] In some embodiments, in the process of determining the sampling probability of the experience tuple based on the first reward value and the second reward value, the agent may determine the first reward value as the first Q value, and determine the first target Q value according to the second reward value, thereby determining the sampling probability of the experience tuple according to the first Q value and the first target Q value. For example, the agent may calculate the error between the first target Q value and the first Q value to obtain the TD error (also known as the time series error), thereby determining the sampling probability of the experience tuple according to the time series error.

[0110] For example, the agent can calculate the sampling probability of the experience tuple according to the following formula:

[0111]

[0112] Where D(m) is the sampling probability of experience tuple m; d m is the priority of experience tuple m; e represents the size of the experience replay pool, σ∈[0,1] is the degree of control priority usage, and rank(m) is the experience number corresponding to the mth experience tuple when the experience tuples are sorted from large to small according to the absolute value of the timing error.

[0113] In some embodiments, in the process of determining the first target Q value based on the second reward value, the agent can directly determine the second reward value as the first target Q value, or the agent can multiply the second reward value by a preset conversion factor, and then determine the sum of the obtained product and the reward in the experience tuple as the first target Q value.

[0114] In some embodiments, after the sampling probability is determined, the agent can store the experience tuples into the experience replay pool according to the sampling probability, and perform sampling according to the updated sampling probability of each experience tuple in the experience replay pool to obtain the target experience tuple.

[0115] It should be noted that by determining the sampling probability of the experience tuple based on the processing results of the Q network and the target Q network, the accuracy of the determined sampling probability is improved. By sampling in the updated experience replay pool according to the sampling probability, the model can extract relatively important or high potential value experience tuples, thereby improving the learning efficiency of the model and then improving the model performance.

[0116] In an optional embodiment, in the process of processing the target experience tuple based on the Q network by the intelligent agent and calculating the loss function value based on the processing result, the intelligent agent can process the current state and the current action in the target experience tuple according to the Q network to obtain a third reward value, and then process the next state and the second target action in the target experience tuple according to the target Q network to obtain a fourth reward value, wherein the target Q network is a delayed updated copy of the Q network, and then calculate the fifth reward value based on the discount factor, the fourth reward value and the reward in the target experience tuple, so as to substitute the third reward value and the fifth reward value into the minimization mean square error function to calculate the loss function value.

[0117] Optionally, the agent can process the current state and current action in the target experience tuple according to the Q network to obtain a second Q value, that is, a third reward value.

[0118] In some embodiments, the second target action may be the current action in the target experience tuple.

[0119] In some embodiments, the second target action may also be selected by the agent from the action space. For example, for each action in the action space, the agent may process the next state in the target experience tuple and the action based on the Q network to obtain the second candidate reward value corresponding to the action. Afterwards, the agent may determine the action with the highest second candidate reward value as the second target action, and then process the next state in the target experience tuple and the second target action based on the target Q network to obtain a fourth reward value.

[0120] Afterwards, the agent can determine the second target Q value according to the fourth reward value. For example, the fourth reward value is multiplied by a preset discount factor, and then the sum of the obtained product and the reward in the target experience tuple is determined as the second target Q value (that is, the aforementioned fifth reward value).

[0121] Furthermore, the agent can substitute the third reward value and the fifth reward value into the minimization mean square error function, that is, substitute the second Q value and the second target Q value into the minimization mean square error function, so as to calculate the loss function value. For example, the loss function value is calculated according to the following formula:

[0122] L(w i )=E[(TargetQ-Q(s t ,a t ;w i )) 2 ]

[0123] Among them, the aforementioned w i is the network parameter value of the Q network at the i-th iteration, L(w i ) indicates the use of wi The loss function value corresponding to the Q network, E is the expectation; TargetQ is the second target Q value (that is, the fifth reward value), Q(s t ,a t ;w i ) is the second Q value.

[0124] It should be noted that, through the above process, the accurate calculation of the loss function value is achieved, thereby improving the accuracy of the parameter update of the Q network, and further effectively improving the training effect of the Markov decision process.

[0125] In an optional embodiment, after updating the network parameter values ​​of the Q network according to the loss function value, the intelligent agent can determine the iteration round of the last update of the network parameter values ​​of the target Q network, obtain the target iteration round, and then judge whether the number of rounds between the current iteration round and the target iteration round reaches the preset number of rounds, thereby updating the network parameter values ​​of the target Q network according to the network parameter values ​​of the Q network when the number of rounds between the current iteration round and the target iteration round reaches the preset number of rounds.

[0126] In some embodiments, the agent may periodically (eg, every C time steps) update the network parameter values ​​of the target Q network based on the network parameter values ​​of the Q network.

[0127] Optionally, the agent may update the network parameter values ​​of the target Q network according to the network parameter values ​​of the Q network when the number of rounds between the current iteration round and the target iteration round reaches a preset number of rounds. For example, the network parameter values ​​of the Q network are directly used as the network parameter values ​​of the target Q network, or the network parameter values ​​of the Q network are copied to the target Q network at a certain ratio.

[0128] Optionally, the agent may not update the network parameter values ​​of the target Q network when the number of rounds between the current iteration round and the target iteration round does not reach a preset number of rounds.

[0129] It should be noted that, through the above process, an effective delayed update of the parameters of the target Q network is achieved.

[0130] It can be seen that the solution provided in the present application achieves the purpose of determining the microgrid planning scheme based on the target Markov decision process model obtained by training the deep Q learning algorithm with priority experience replay, thereby achieving the technical effect of improving the accuracy of the determined microgrid planning scheme, and further solving the technical problem of low accuracy of the microgrid planning scheme determined in the related technology.

[0131] Example 2

[0132] According to an embodiment of the present invention, an embodiment of a device for determining a microgrid planning scheme is provided, wherein: Figure 4 is a schematic diagram of an optional device for determining a microgrid planning scheme according to an embodiment of the present invention, such as Figure 4 As shown, the device comprises:

[0133] The acquisition module 401 is used to acquire the status data of the microgrid, wherein the status data at least includes: the system load power value, the value of electricity purchased by the microgrid from the distribution network, the value of electricity sold by the microgrid to the distribution network, and the charge state value of the microgrid energy storage;

[0134] The first processing module 402 is used to input the state data of the microgrid into the target Markov decision process model to obtain a planning scheme for the installation capacity of the microgrid, wherein the target Markov decision process model is obtained by training the initial Markov decision process model based on the deep Q learning algorithm of priority experience replay, and the initial Markov decision process model is constructed based on the multi-objective planning model of the microgrid.

[0135] It should be noted that the acquisition module 401 and the first processing module 402 correspond to steps S101 to S102 in the above embodiment. The examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment 1.

[0136] Optionally, the device for determining the microgrid planning scheme also includes: a first determination module, used to determine the state space according to the parameters of the objective function in the multi-objective planning model and the parameters in the constraint conditions, wherein the parameters in the objective function include at least: system load power, the electricity price at which the microgrid purchases electricity from the distribution network, and the electricity price at which the microgrid sells electricity to the distribution network, and the parameters in the constraint conditions include at least: the charge state of the microgrid energy storage; a second determination module, used to determine the action space according to the parameters to be solved in the multi-objective planning model, wherein the parameters to be solved include: the installation capacity of multiple devices required for the microgrid; a third determination module, used to determine the reward function according to the objective function in the multi-objective planning model, wherein the reward function is used to calculate the reward obtained after performing an action on the state; a construction module, used to construct an initial Markov decision process model based on the state space, the action space and the reward function.

[0137] Optionally, the device for determining the microgrid planning scheme also includes: a generation module, which is used to obtain the current state of the state space through the intelligent agent in the initial Markov decision process model, and generate an experience tuple based on the current state, wherein the experience tuple includes the current state, the current action, the reward obtained by executing the current action on the current state, and the next state obtained by executing the current action on the current state; a sampling module, which is used to store the experience tuple in the experience replay pool based on the priority experience replay through the intelligent agent, and sample the updated experience replay pool to obtain the target experience tuple; a second processing module, which is used to perform the Q network based on the target The experience tuple is processed, the loss function value is calculated based on the processing result, and the network parameter value of the Q network is updated according to the loss function value, wherein the Q network is used to calculate the reward value obtained after performing an action on the state, and the target strategy in the intelligent agent is updated as the network parameter value of the Q network is updated; the third processing module is used to determine the next state in the target experience tuple as the updated current state of the state space through the intelligent agent when the Q network does not meet the preset iteration condition, and repeat the step of generating the experience tuple according to the updated current state until the preset iteration condition is met, so as to obtain the target Markov decision process model.

[0138] Optionally, the generation module also includes: a first determination submodule, used to determine the current action from the action space based on the current state; a first processing submodule, used to execute the current action on the current state, obtain a reward and a next state corresponding to the current state; a generation submodule, used to generate an experience tuple based on the current state, the current action, the reward obtained by executing the current action on the current state, and the next state.

[0139] Optionally, the sampling module also includes: a second processing submodule, used to process the current state and current action in the experience tuple based on the Q network to obtain a first reward value; a third processing submodule, used to process the next state and the first target action in the experience tuple based on the target Q network to obtain a second reward value, wherein the target Q network is a delayed updated copy of the Q network; a second determination submodule, used to determine the sampling probability of the experience tuple based on the first reward value and the second reward value; a sampling submodule, used to store the experience tuple in the experience replay pool based on the sampling probability, and sample according to the sampling probability of each experience tuple in the updated experience replay pool to obtain a target experience tuple.

[0140] Optionally, the second processing module also includes: a fourth processing sub-module, which processes the current state and the current action in the target experience tuple according to the Q network to obtain a third reward value; a fifth processing sub-module, which processes the next state and the second target action in the target experience tuple according to the target Q network to obtain a fourth reward value, wherein the target Q network is a delayed updated copy of the Q network; a first calculation sub-module, which is used to calculate the fifth reward value based on the discount factor, the fourth reward value and the reward in the target experience tuple; and a second calculation sub-module, which is used to substitute the third reward value and the fifth reward value into the minimization mean square error function to calculate the loss function value.

[0141] Optionally, the device for determining the microgrid planning scheme also includes: a fourth determination module, used to determine the iteration round of the last update of the network parameter value of the target Q network to obtain the target iteration round; a judgment module, used to judge whether the number of rounds between the current iteration round and the target iteration round reaches a preset number of rounds; an update module, used to update the network parameter value of the target Q network according to the network parameter value of the Q network when the number of rounds between the current iteration round and the target iteration round reaches a preset number of rounds.

[0142] Example 3

[0143] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned method for determining a microgrid planning scheme when running.

[0144] Example 4

[0145] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, wherein: Figure 5 is a schematic diagram of an optional electronic device according to an embodiment of the present invention, such as Figure 5 As shown, the electronic device includes one or more processors; a memory for storing one or more programs, which, when executed by one or more processors, enables the one or more processors to run the programs, wherein the programs are configured to execute the above-mentioned method for determining the microgrid planning scheme when running.

[0146] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0147] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0148] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0149] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0150] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0151] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0152] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for determining a microgrid planning scheme, It is characterized in that include: Acquire the status data of the microgrid, wherein the status data at least includes: the system load power value, the value of electricity purchased by the microgrid from the distribution network, the value of electricity sold by the microgrid to the distribution network, and the state of charge value of the energy storage of the microgrid; The state data of the microgrid is input into the target Markov decision process model to obtain a planning scheme for the installation capacity of the microgrid, wherein the target Markov decision process model is obtained by training an initial Markov decision process model based on a deep Q learning algorithm with priority experience replay, and the initial Markov decision process model is constructed based on a multi-objective planning model of the microgrid.

2. The method according to claim 1, It is characterized in that The initial Markov decision process model is obtained by: Determine the state space according to the parameters of the objective function in the multi-objective programming model and the parameters in the constraint conditions, wherein the parameters in the objective function include at least: system load power, the electricity price at which the microgrid purchases electricity from the distribution network, and the electricity price at which the microgrid sells electricity to the distribution network, and the parameters in the constraint conditions include at least: the state of charge of the microgrid energy storage; Determining an action space according to parameters to be solved in the multi-objective programming model, wherein the parameters to be solved include: installation capacities of multiple devices required by the microgrid; Determining a reward function according to the objective function in the multi-objective programming model, wherein the reward function is used to calculate a reward obtained after performing an action on a state; The initial Markov decision process model is constructed according to the state space, the action space and the reward function.

3. The method according to claim 2, It is characterized in that The target Markov decision process model is trained in the following way: Obtaining the current state of the state space through the agent in the initial Markov decision process model, and generating an experience tuple according to the current state, wherein the experience tuple includes the current state, the current action, the reward obtained by executing the current action on the current state, and the next state obtained by executing the current action on the current state; The agent stores the experience tuple in an experience replay pool based on the priority experience replay, and samples the updated experience replay pool to obtain a target experience tuple; The target experience tuple is processed by the agent based on the Q network, a loss function value is calculated based on the processing result, and a network parameter value of the Q network is updated according to the loss function value, wherein the Q network is used to calculate a reward value obtained after performing an action on a state, and the target strategy in the agent is updated as the network parameter value of the Q network is updated; When the Q network does not meet the preset iteration condition, the intelligent agent determines the next state in the target experience tuple as the updated current state of the state space, and repeats the step of generating experience tuples according to the updated current state until the preset iteration condition is met, thereby obtaining the target Markov decision process model.

4. The method according to claim 3, It is characterized in that Generate an experience tuple according to the current state, including: Determining the current action from the action space according to the current state; Execute the current action on the current state to obtain a reward and a next state corresponding to the current state; The experience tuple is generated according to the current state, the current action, a reward obtained by executing the current action on the current state, and a next state.

5. The method according to claim 3, It is characterized in that The agent stores the experience tuple in an experience replay pool based on the priority experience replay, and samples the updated experience replay pool to obtain a target experience tuple, including: Processing the current state and the current action in the experience tuple based on the Q network to obtain a first reward value; Processing the next state and the first target action in the experience tuple based on the target Q network to obtain a second reward value, wherein the target Q network is a delayed updated copy of the Q network; Determining a sampling probability of the experience tuple based on the first reward value and the second reward value; The experience tuple is stored in the experience replay pool based on the sampling probability, and sampling is performed according to the sampling probability of each experience tuple in the updated experience replay pool to obtain the target experience tuple.

6. The method according to claim 3, It is characterized in that The target experience tuple is processed by the agent based on the Q network, and a loss function value is calculated based on the processing result, including: Processing the current state and the current action in the target experience tuple according to the Q network to obtain a third reward value; Processing the next state and the second target action in the target experience tuple according to the target Q network to obtain a fourth reward value, wherein the target Q network is a delayed update copy of the Q network; A fifth reward value is calculated based on the conversion factor, the fourth reward value, and the reward in the target experience tuple; Substitute the third reward value and the fifth reward value into the minimization mean square error function to calculate the loss function value.

7. The method according to claim 6, It is characterized in that After updating the network parameter value of the Q network according to the loss function value, the method further includes: Determine the iteration round of the last update of the network parameter value of the target Q network to obtain a target iteration round; Determine whether the number of rounds between the current iteration round and the target iteration round reaches a preset number of rounds; When the number of rounds between the current iteration round and the target iteration round reaches the preset number of rounds, the network parameter value of the target Q network is updated according to the network parameter value of the Q network.

8. A device for determining a microgrid planning scheme, It is characterized in that include: An acquisition module is used to acquire the status data of the microgrid, wherein the status data at least includes: a system load power value, an electricity value of the microgrid purchased from the distribution network, an electricity value of the microgrid sold to the distribution network, and a charge state value of the microgrid energy storage; The first processing module is used to input the state data of the microgrid into the target Markov decision process model to obtain a planning scheme for the installation capacity of the microgrid, wherein the target Markov decision process model is obtained by training an initial Markov decision process model based on a deep Q learning algorithm with priority experience replay, and the initial Markov decision process model is constructed based on a multi-objective planning model of the microgrid.

9. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the method for determining the microgrid planning scheme described in any one of claims 1 to 7 when running.

10. An electronic device, It is characterized in that The electronic device includes one or more processors; A memory for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to run the programs, wherein the program is configured to execute the method for determining the microgrid planning scheme described in any one of claims 1 to 7 when running.

Citation Information

Cited By

  • Microgrid planning processing method and device

    CN119204619A

  • Microgrid planning and processing method and device

    CN119204619B