Battery energy storage capacity estimation method and system based on reinforcement learning algorithm

Abstract

The present invention relates to a battery energy storage capacity estimation method and system based on a reinforcement learning algorithm. It includes: according to the strategy network under different battery capacities, simulate the charging and discharging actions of the energy storage battery after adding the energy storage battery, obtain the optimal strategy, and calculate the annual electricity bill savings; calculate the internal rate of return during the investment payback period under different battery capacities; within the acceptable internal rate of return range, calculate the financial cost during the investment payback period under different battery capacities according to the loan interest rate; calculate the profit under different battery capacities according to the initial investment and financial cost; output the battery capacity with the highest profit as the final battery capacity selection plan. The present invention can assist in making decisions on energy storage plans, and can provide more accurate calculation results based on historical data to support investment decisions. Various conditions can be flexibly changed, and can provide accurate calculation results of corresponding changes. The characteristic setting based on electricity price fluctuations can provide calculation uncertainty and investment return risk.

Description

Technical Field

[0001] The present invention relates to the field of power demand side energy storage design, and in particular to a battery energy storage capacity estimation method and system based on a reinforcement learning algorithm. Background Art

[0002] With the reform of the electricity market, the electricity price in the future will be determined by the competition between power generation and power consumption. The electricity price will change in real time according to time and region (every 15 minutes), and the volatility of the electricity price will increase significantly. Therefore, making full use of the fluctuating electricity price to adjust the internal electricity consumption of the building can not only help the electricity user to reduce the total electricity cost, but also help to reduce the peak power consumption pressure on the power generation side. Among them, the transformation and increase of energy storage on the demand side can bring considerable flexibility to the regulation of building electricity consumption, but the current cost of energy storage batteries is not low, so it is necessary to accurately evaluate the economic value brought by the energy storage design scheme to assist in judging the energy storage transformation decision. The general energy storage design evaluation method has poor accuracy, resulting in low or even negative return on investment. The present invention makes full use of the historical real-time operation data of the building and the historical real-time fluctuating electricity price to provide quantitative auxiliary decision-making for each building instance and different external conditions. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present invention provides a battery energy storage capacity estimation method and system based on a reinforcement learning algorithm.

[0004] According to a first aspect of an embodiment of the present invention, a battery energy storage capacity estimation method based on a reinforcement learning algorithm is provided, comprising:

[0005] According to the strategy network under different battery capacities, the charging and discharging actions of the energy storage battery after adding the energy storage battery are simulated to obtain the optimal strategy, and the annual electricity cost saved under the optimal strategy is calculated;

[0006] Taking the annual electricity savings as the net cash flow, calculate the internal rate of return during the investment payback period for different battery capacities;

[0007] Within the acceptable internal rate of return, calculate the financial cost of different battery capacities during the investment payback period based on the loan interest rate;

[0008] Calculate the profit for different battery capacities based on the initial investment and the stated financial costs;

[0009] The battery capacity with the highest profit is output as the final battery capacity selection plan.

[0010] Furthermore, the method also includes the step of using a Q-Learning algorithm of reinforcement learning to train the policy network, specifically including:

[0011] Step 1, initialize algorithm parameters and Q table;

[0012] Step 2: Enter the environmental status and query all possible actions that the battery can take;

[0013] Step 3: Query the Q value of each action that the battery can take in the current state in the Q table, and select an action based on the queried Q value;

[0014] Step 4: Calculate the reward based on the selected action;

[0015] Step 5: Update the electricity price in the battery;

[0016] Step 6: Update the Q table;

[0017] Step 7: Update the battery energy storage status and environmental status;

[0018] Repeat steps 2-7 until the algorithm converges.

[0019] Furthermore, in step 3, an action is selected according to the queried Q value, specifically including:

[0020] If the Q values ​​are all 0, assign the same probability to all possible actions, and then select an action based on the probability; otherwise, increase the probability of the action with the highest Q value, and then select an action based on the probability.

[0021] Further, the step 4 specifically includes:

[0022] When the battery is in the discharge state, if the price of electricity in the battery is higher than the price of electricity in the grid, a negative reward is calculated; if the price of electricity in the battery is not higher than the price of electricity in the grid, a positive reward is calculated;

[0023] When the battery is in charging state, if the price of electricity in the battery is not greater than the price of electricity from the grid, a positive reward is calculated, otherwise no reward is given;

[0024] No reward will be given if the battery is neither charged nor discharged.

[0025] Furthermore, the internal rate of return is calculated as:

[0026]

[0027] Among them, NPV is the net present value, CF is i is the net cash flow in the i-th year, n is the investment payback period, and IRR is the internal rate of return.

[0028] According to a second aspect of an embodiment of the present invention, a battery energy storage capacity estimation system based on a reinforcement learning algorithm is provided, comprising:

[0029] The first calculation module is used to simulate the charging and discharging actions of the energy storage battery after adding the energy storage battery according to the strategy network under different battery capacities, obtain the optimal strategy, and calculate the annual electricity cost saved under the optimal strategy;

[0030] The second calculation module is used to calculate the internal rate of return during the investment payback period under different battery capacities by taking the electricity cost saved each year as the net cash flow;

[0031] A third calculation module is used to calculate the financial cost of different battery capacities within the investment payback period according to the loan interest rate within an acceptable internal rate of return;

[0032] A fourth calculation module, used to calculate the profit under different battery capacities according to the initial investment and the financial cost;

[0033] The scheme selection module is used to output the battery capacity with the highest profit as the final battery capacity selection scheme.

[0034] Furthermore, the system also includes a training module for obtaining the policy network by training using a Q-Learning algorithm of reinforcement learning, and the training module specifically includes:

[0035] Initialization unit, used to initialize algorithm parameters and Q table;

[0036] A query unit, used to input the environmental status and query all possible actions that the battery can take;

[0037] A selection unit, used to query the Q value of each action that the battery can take in the current state in the Q table, and select an action according to the queried Q value;

[0038] A calculation unit, used to calculate the reward based on the selected action;

[0039] A first updating unit, used for updating the electricity price in the battery;

[0040] A second updating unit, used for updating the Q table;

[0041] A third updating unit, used for updating the battery energy storage state and the environmental state;

[0042] The calling unit is used to repeatedly call the initialization unit, the first query unit, the second query unit, the calculation unit, the first update unit, the second update unit and the third update unit until the algorithm converges.

[0043] Furthermore, the selection unit is specifically used for:

[0044] If the Q values ​​are all 0, assign the same probability to all possible actions, and then select an action based on the probability; otherwise, increase the probability of the action with the highest Q value, and then select an action based on the probability.

[0045] According to a third aspect of an embodiment of the present invention, a terminal device is provided, including:

[0046] Processor; and

[0047] The memory stores executable codes thereon, and when the executable codes are executed by the processor, the processor is caused to execute the method as described above.

[0048] According to a fourth aspect of an embodiment of the present invention, there is provided a non-transitory machine-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the method as described above.

[0049] The technical solution provided by the embodiments of the present invention may have the following beneficial effects:

[0050] 1. It can assist users in making decisions on energy storage solutions in a step-by-step manner.

[0051] 2. Based on historical data on the construction demand side, more accurate calculation results can be given for each project to support investment decisions.

[0052] 3. Various conditions (such as electricity price volatility, unit battery cost, battery model selection, strategy constraints, etc.) can be changed flexibly, and accurate calculation results of corresponding changes can be given.

[0053] 4. Characteristic settings based on electricity price fluctuations (manually set ranges) can give calculation uncertainty and investment return risk.

[0054] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The above and other objects, features and advantages of the present invention will become more apparent through a more detailed description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present invention.

[0056] Figure 1 is a flow chart of a battery energy storage capacity estimation method based on a reinforcement learning algorithm according to an exemplary embodiment of the present invention;

[0057] Figure 2 It is the algorithm principle diagram of the method of the present invention;

[0058] Figure 3 It is a flowchart of the training steps of the Q-Learning algorithm;

[0059] Figure 4 This is a diagram showing the calculation results of annual electricity savings and percentages for different battery capacities;

[0060] Figure 5 This is a schematic diagram of the calculation results of the internal rate of return under different battery capacities;

[0061] Figure 6 This is a schematic diagram of 10-year income and expenditure under different battery capacities;

[0062] Figure 7 Schematic diagram of total profit for 10 years under different battery capacities. DETAILED DESCRIPTION

[0063] The preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0064] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0065] It should be understood that although the terms "first", "second", "third", etc. may be used to describe various information in the present invention, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0066] The present invention is a method for regulating energy storage operation by combining real-time operation data of a building throughout the year with real-time electricity prices. The energy storage scheme is divided into two-step decision-making. First, the internal rate of return is calculated, and then profit charts of different schemes are given based on the acceptable internal rate of return to provide people with decision-making. In addition, the long-term return is considered through the reinforcement learning algorithm to calculate the optimal strategy for the energy storage battery.

[0067] The technical solutions of the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0068] like Figure 2 As shown, the principle of this method includes four steps: inputting basic information, energy storage battery strategy, profit calculation and output decision.

[0069] First of all, before making a decision on a battery energy storage solution, you need to input two types of basic information. The first is the basic condition setting, which includes battery specifications (battery theoretical charge and discharge depth, battery theoretical charge and discharge cycle efficiency, battery theoretical charge and discharge power, etc.) and time-of-use electricity prices. The second is the energy storage battery selection range setting, which is mainly the upper and lower limits of the battery capacity.

[0070] In the energy storage battery strategy link, the reinforcement learning algorithm is mainly used. In this embodiment, the Q-Learning algorithm can be used. This algorithm regards the energy storage battery as an intelligent agent and performs autonomous learning in an unknown environment. Initially, a state is given, and then an action is made. After each action is completed, a reward is given (this reward may be positive or negative), and at the same time, it is refreshed to a new state and a new action is made. Through continuous cycles, the intelligent body will use the learned rules to choose the best action in different situations. The core of the Q-Learning algorithm is to establish a state-action value Q table, continuously interact with the environment to learn, update the Q table, and finally summarize a Q table, which records the rewards of different actions under different states. When a new state is given, the Q table is queried to find the action with the highest reward.

[0071] When training the reinforcement learning algorithm, the data needs to be processed. Since the historical power consumption, battery capacity and other data are continuous, there are too many states, and the Q table method cannot save all of them. Therefore, when solving this type of random optimal control problem with continuous states and decision spaces, it is necessary to discretize the states and decision spaces of the problem first. Discretize the historical power consumption, battery capacity, and electricity price data, and bin them according to the corresponding step size.

[0072] In addition, the algorithm needs to be defined as follows:

[0073] Define intelligent agents: Energy storage batteries of different capacities are intelligent agents that can perform different actions in different situations.

[0074] Define the environment: The historical energy consumption of a building over a period of time is the environment, which includes the time-of-use electricity price, the limitations of the energy storage battery (input and output power, upper and lower limits of capacity), etc.

[0075] Define action: A decision made by the agent, which in this model includes charging the energy storage battery, discharging the energy storage battery, and keeping the energy storage battery on standby.

[0076] Define action space: a set of all actions. In this model, the charging and discharging power of the energy storage battery is binned to obtain the action space.

[0077] Status: A summary of the current environment, including the battery capacity, time-of-use electricity prices, and building electricity consumption.

[0078] State space: The set of all possible states.

[0079] Policy function: The next action output by the agent in the corresponding state.

[0080] Reward: Energy storage batteries save electricity costs under time-of-use electricity prices.

[0081] Policy network: a table of correspondence between actions and rewards.

[0082] Specifically, Figure 3 As shown, the training steps of the Q-Learning algorithm include:

[0083] S1, initialize algorithm parameters and Q table;

[0084] Among them, the algorithm parameters are initialized: γ = 0.95 (the attenuation value of future rewards. The closer γ is to 1, the more sensitive it is to future rewards), α = 0.2 (learning rate, which determines how much of the error is to be learned), ε = 0.65 (greediness, which is a strategy used in decision-making. For example, when ε = 0.9, it means that 90% of the time the behavior will be selected according to the optimal value of the Q table, and 10% of the time the behavior will be randomly selected); Initialize the Q table: the initial values ​​are all 0.

[0085] S2, input the environmental status and query all possible actions that the battery can take;

[0086] Specifically, the environmental status includes the energy storage battery power status, electricity price status, project energy consumption status, etc. The actions that the battery can take include:

[0087] a. If the battery is not full, you can charge it;

[0088] b. If the battery is not empty, it can be discharged;

[0089] c. The battery can be neither charged nor discharged in any state.

[0090] S3, querying the Q value of each action that the battery can take in the current state in the Q table, and selecting an action according to the queried Q value;

[0091] Specifically, if the Q values ​​are all 0, all possible actions are given the same probability, and then the action is selected based on the probability; otherwise, the probability of the action with the highest Q value is increased, and then the action is selected based on the probability. In the early learning stage, in order to increase the diversity of training samples and effectively update the Q table, the selected action is insensitive to the size of the reward. Later, as learning deepens, the action with the largest reward in the action set is selected with a higher probability.

[0092] S4. Calculate the reward based on the selected action;

[0093] Specifically, the reward calculation method is as follows: When the battery is discharged, the price of electricity in the battery is higher than the price of electricity on the grid, and a negative reward = electricity × price difference is obtained; when the price of electricity in the battery is not higher than the price of electricity on the grid, a positive reward = electricity × price difference is obtained. When the battery is charged: the price of electricity in the battery is not higher than the price of electricity on the grid, a positive reward = electricity × price difference is obtained, otherwise no reward is given. If the battery is not moved: no reward is given.

[0094] S5, updating the electricity price in the battery;

[0095] S6. Update Q table;

[0096] Specifically, according to the state and reward, the Q table is updated by the following formula:

[0097] Q(S t ,A t )←Q(S t ,A t )+α(R t+1 +λmax a Q(S t+1 ,a)-Q(S t ,A t ))

[0098] S7, updating battery energy storage status and environmental status;

[0099] Specifically, the environment state of the next time segment is obtained by updating.

[0100] Repeat steps S2-S7 to allow the agent to fully explore the environment, traverse various possibilities in the environment, continuously update the Q table, and gradually find the optimal solution, that is, choose the correct action in different states to save the most electricity, and stop training after convergence.

[0101] Figure 1 It is a flow chart of a method for estimating battery energy storage capacity based on a reinforcement learning algorithm according to an exemplary embodiment of the present invention.

[0102] See also Figure 1 , the method comprising:

[0103] 110. According to the strategy network under different battery capacities, simulate the charging and discharging actions of the energy storage battery after adding the energy storage battery, obtain the optimal strategy, and calculate the annual electricity cost saved under the optimal strategy;

[0104] Specifically, the strategy network can be trained by using the Q-Learning algorithm of the reinforcement learning.

[0105] According to the above method, the annual electricity cost saving of different battery capacities can be calculated. Furthermore, the annual electricity cost saving percentage of different battery capacities can be calculated, such as Figure 4 shown.

[0106] 120. Taking the electricity cost saved each year as the net cash flow, calculate the internal rate of return during the investment payback period for different battery capacities;

[0107] Specifically, the Internal Rate of Return (IRR) is the discount rate when the total present value of capital inflows is equal to the total present value of capital outflows and the net present value is zero. It is the rate of return that an investment aspires to achieve, and the larger the index, the better. Generally speaking, when the internal rate of return is greater than or equal to the benchmark rate of return, the project is feasible.

[0108] IRR calculation formula:

[0109]

[0110] Among them, NPV is the net present value, CF is i is the net cash flow in the i-th year, n is the investment payback period, and IRR is the internal rate of return.

[0111] According to the above internal rate of return calculation formula, input the investment payback period and annual net cash flow to calculate the internal rate of return under different battery capacities, such as Figure 5 shown.

[0112] 130. Within the acceptable internal rate of return, calculate the financial cost of different battery capacities during the investment payback period based on the loan interest rate;

[0113] Specifically, financial cost = principal + interest, Figure 6 This is a schematic diagram of 10-year income and expenditure under different battery capacities.

[0114] 140. Calculate the profit at different battery capacities based on the initial investment and the financial cost;

[0115] Specifically, profit = total income - principal - interest, Figure 7 Schematic diagram of total profit for 10 years under different battery capacities.

[0116] 150. Output the battery capacity with the highest profit as the final battery capacity selection plan.

[0117] The present invention provides a battery energy storage capacity estimation method based on a reinforcement learning algorithm, which has the following beneficial effects:

[0118] 1. It can assist users in making decisions on energy storage solutions in a step-by-step manner.

[0119] 2. Based on historical data on the construction demand side, more accurate calculation results can be given for each project to support investment decisions.

[0120] 3. Various conditions (such as electricity price volatility, unit battery cost, battery model selection, strategy constraints, etc.) can be changed flexibly, and accurate calculation results of corresponding changes can be given.

[0121] 4. Characteristic settings based on electricity price fluctuations (manually set ranges) can give calculation uncertainty and investment return risk.

[0122] The battery energy storage capacity estimation system based on the reinforcement learning algorithm shown in the exemplary embodiment of the present invention includes:

[0123] The first calculation module is used to simulate the charging and discharging actions of the energy storage battery after adding the energy storage battery according to the strategy network under different battery capacities, obtain the optimal strategy, and calculate the annual electricity cost saved under the optimal strategy;

[0124] The second calculation module is used to calculate the internal rate of return during the investment payback period under different battery capacities by taking the electricity cost saved each year as the net cash flow;

[0125] A third calculation module is used to calculate the financial cost of different battery capacities within the investment payback period according to the loan interest rate within an acceptable internal rate of return;

[0126] A fourth calculation module, used to calculate the profit under different battery capacities according to the initial investment and the financial cost;

[0127] The scheme selection module is used to output the battery capacity with the highest profit as the final battery capacity selection scheme.

[0128] Optionally, in this embodiment, the system further includes a training module for obtaining the policy network by training using a Q-Learning algorithm of reinforcement learning, and the training module specifically includes:

[0129] Initialization unit, used to initialize algorithm parameters and Q table;

[0130] A query unit, used to input the environmental status and query all possible actions that the battery can take;

[0131] A selection unit, used to query the Q value of each action that the battery can take in the current state in the Q table, and select an action according to the queried Q value;

[0132] A calculation unit, used to calculate the reward based on the selected action;

[0133] A first updating unit, used for updating the electricity price in the battery;

[0134] A second updating unit, used for updating the Q table;

[0135] A third updating unit, used for updating the battery energy storage state and the environmental state;

[0136] The calling unit is used to repeatedly call the initialization unit, the first query unit, the second query unit, the calculation unit, the first update unit, the second update unit and the third update unit until the algorithm converges.

[0137] Optionally, in this embodiment, the selection unit is specifically configured to:

[0138] If the Q values ​​are all 0, assign the same probability to all possible actions, and then select an action based on the probability; otherwise, increase the probability of the action with the highest Q value, and then select an action based on the probability.

[0139] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0140] An exemplary embodiment of the present invention shows a computing device, including a memory and a processor.

[0141] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0142] The memory may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage. Among them, ROM may store static data or instructions required by the processor or other modules of the computer. The permanent storage may be a readable and writable storage device. The permanent storage may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0143] The memory stores executable codes, and when the executable codes are processed by the processor, the processor can execute part or all of the methods described above.

[0144] In addition, the method according to the present invention may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing part or all of the steps in the above method of the present invention.

[0145] Alternatively, the present invention may also be implemented as a non-temporary machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or computing device, server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present invention.

[0146] The scheme of the present invention has been described in detail above with reference to the accompanying drawings. In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments. Those skilled in the art should also be aware that the actions and modules involved in the specification are not necessarily required by the present invention. In addition, it can be understood that the steps in the method of the embodiment of the present invention can be adjusted in order, merged and deleted according to actual needs, and the modules in the device of the embodiment of the present invention can be merged, divided and deleted according to actual needs.

[0147] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or a combination of both.

[0148] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system and method according to multiple embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0149] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A battery energy storage capacity estimation method based on reinforcement learning algorithm, characterized in that: include: According to the strategy network under different battery capacities, the charging and discharging actions of the energy storage battery after adding the energy storage battery are simulated to obtain the optimal strategy, and the annual electricity cost saved under the optimal strategy is calculated; Taking the annual electricity savings as the net cash flow, calculate the internal rate of return during the investment payback period for different battery capacities; Within the acceptable internal rate of return, calculate the financial cost of different battery capacities during the investment payback period based on the loan interest rate; Calculate the profit for different battery capacities based on the initial investment and the stated financial costs; Output the battery capacity with the highest profit as the final battery capacity selection plan; The step of using a Q-Learning algorithm of reinforcement learning to train the policy network is also included, specifically including: Step 1, initialize algorithm parameters and Q table; Step 2: Enter the environmental status and query all possible actions that the battery can take; Step 3: Query the Q value of each action that the battery can take in the current state in the Q table, and select an action based on the queried Q value; Step 4: Calculate the reward based on the selected action; Step 5: Update the electricity price in the battery; Step 6: Update the Q table; Step 7: Update the battery energy storage status and environmental status; Repeat steps 2-7 until the algorithm converges; In step 3, an action is selected according to the queried Q value, specifically including: If the Q values ​​are all 0, assign the same probability to all possible actions, and then select an action based on the probability; otherwise, increase the probability of the action with the highest Q value, and then select an action based on the probability; The step 4 specifically includes: When the battery is in the discharge state, if the price of electricity in the battery is higher than the price of electricity in the grid, a negative reward is calculated; if the price of electricity in the battery is not higher than the price of electricity in the grid, a positive reward is calculated; When the battery is in charging state, if the price of electricity in the battery is not greater than the price of electricity from the grid, a positive reward is calculated, otherwise no reward is given; No reward will be given if the battery is neither charged nor discharged.

2. The method according to claim 1, characterized in that The formula for calculating the internal rate of return is: Among them, NPV is the net present value, CF is i is the net cash flow in the i-th year, n is the investment payback period, and IRR is the internal rate of return.

3. A battery energy storage capacity estimation system based on reinforcement learning algorithm, characterized in that: include: The first calculation module is used to simulate the charging and discharging actions of the energy storage battery after adding the energy storage battery according to the strategy network under different battery capacities, obtain the optimal strategy, and calculate the annual electricity cost saved under the optimal strategy; The second calculation module is used to calculate the internal rate of return during the investment payback period under different battery capacities by taking the electricity cost saved each year as the net cash flow; A third calculation module is used to calculate the financial cost of different battery capacities within the investment payback period according to the loan interest rate within an acceptable internal rate of return; A fourth calculation module, used to calculate the profit under different battery capacities according to the initial investment and the financial cost; A scheme selection module is used to output the battery capacity with the highest profit as the final battery capacity selection scheme; It also includes a training module for obtaining the policy network by training using a Q-Learning algorithm of reinforcement learning, and the training module specifically includes: Initialization unit, used to initialize algorithm parameters and Q table; A query unit, used to input the environmental status and query all possible actions that the battery can take; A selection unit, used to query the Q value of each action that the battery can take in the current state in the Q table, and select an action according to the queried Q value; A calculation unit, used to calculate the reward based on the selected action; A first updating unit, used for updating the electricity price in the battery; A second updating unit, used for updating the Q table; A third updating unit, used for updating the battery energy storage state and the environmental state; A calling unit, used for repeatedly calling the initialization unit, the first query unit, the second query unit, the calculation unit, the first update unit, the second update unit and the third update unit until the algorithm converges; The selection unit is specifically used for: If the Q values ​​are all 0, assign the same probability to all possible actions, and then select an action based on the probability; otherwise, increase the probability of the action with the highest Q value, and then select an action based on the probability; The computing unit is specifically used for: When the battery is in the discharge state, if the price of electricity in the battery is higher than the price of electricity in the grid, a negative reward is calculated; if the price of electricity in the battery is not higher than the price of electricity in the grid, a positive reward is calculated; When the battery is in charging state, if the price of electricity in the battery is not greater than the price of electricity from the grid, a positive reward is calculated, otherwise no reward is given; No reward will be given if the battery is neither charged nor discharged.

4. A terminal device, characterized in that: include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 2.

5. A non-transitory machine-readable storage medium having executable code stored thereon, characterized in that: When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as claimed in any one of claims 1 to 2.

Images