An adaptive optimization energy storage method based on reinforcement learning

Abstract

The application discloses an adaptive optimization energy storage method based on reinforcement learning, comprising: acquiring power data of a user, and constructing a demand charge model of the user; combining the demand charge model, considering constraint conditions, analyzing and evaluating economic benefits of an energy storage system under current power consumption, and obtaining a preliminary allocation scheme of the energy storage system; according to the preliminary allocation scheme of the energy storage system, preprocessing energy storage action parameters, and coarsely allocating action strategies; converting the coarsely allocated energy storage system action into an action sequence corresponding to each time point, and generating an energy storage system scheduling sequence sorted according to time; and according to current environmental characteristics, dynamically adjusting the energy storage system scheduling sequence to achieve the purpose of optimal enterprise benefits. The application makes up for the defects of the lack of general applicability of traditional algorithms, reduces the dependence of traditional algorithms on scenes, can be applied in different scenes, and finally obtains an optimal scheduling strategy.

Description

Technical Field

[0001] This invention belongs to the field of battery energy storage optimization and scheduling technology, specifically involving an adaptive optimization energy storage method based on reinforcement learning, which can be used in diverse scenarios such as power system optimization, microgrid energy storage, industrial engineering, and municipal construction. Background Technology

[0002] In recent years, given the continuous development and improvement of energy storage systems, their application in power systems has expanded from peak shaving and valley filling to include frequency regulation, demand-side response, power supply reliability analysis, and mitigation of fluctuations in renewable energy generation. As an excellent carrier for peak shaving and valley filling of terminal power loads, energy storage can save electricity costs for industrial users. Local governments have begun to vigorously promote the application of energy storage systems, indicating broad commercial prospects. Therefore, research on user-side energy storage optimization scheduling has attracted much attention. Although optimized energy storage scheduling is beneficial for fully realizing the economic and environmental benefits of the energy system, the randomness and uncertainty of the power usage environment bring significant challenges to energy storage scheduling. To better promote and utilize energy storage systems, optimized scheduling is needed to achieve efficient, economical, and stable operation.

[0003] Traditional energy storage dispatching methods rely on comprehensive data analysis and manual adjustments. The entire energy storage dispatching process is a closed loop, from data analysis to manual adjustment of energy storage solutions, such as... Figure 1 As shown, this energy storage process involves multiple modules and departments, including users, functional departments, energy storage systems, and energy exchange equipment. The business processes of each component are inconsistent, and interaction between them is inconvenient. The environmental and constraint information used in energy storage planning is not real-time and has a certain lag. Furthermore, due to the lack of necessary communication between the energy storage system and the energy supplier, the current workflow between them is incomplete. Limited idle resources of the energy storage system play a crucial role in the entire process, affecting the issuance of instructions and feedback on status. Overall, the traditional energy storage planning process can respond to some energy needs, but it is no longer adequate for new demands and challenges. The drawbacks of the traditional energy storage process can be summarized as follows:

[0004] (1) The entire energy storage process is too complex and cumbersome. The time from when enterprise users put forward their needs to when they receive feedback on energy usage is often long, which cannot meet the users' requirements for timeliness. (2) Energy storage planning is generally based on offline operations, and the planning scheme cannot adapt to the real-time changing environment. (3) There is a lack of rapid auxiliary means and automated processing procedures. In many cases, manual modification of the energy storage system operation data is required. The emergency adjustment process is complex, and there are many human-computer interaction operations, which are prone to errors.

[0005] It can be seen that the energy storage system only operates according to the planning results throughout the process, and cannot respond in real time to changes in the enterprise's working environment and business acceptance. This leads to a deviation between the actual scheduling results and expectations, and further reflects that the actual operating efficiency of the energy storage system is lower than expected.

[0006] With in-depth research into intelligent optimization methods, reinforcement learning algorithms have become a major technology for solving user-side energy storage scheduling problems. This algorithm requires no prior knowledge, can adaptively adjust parameters to achieve dynamic learning and evolution of the system, and is applicable to different scenarios, ultimately obtaining the optimal scheduling strategy. Currently, for the real-time scheduling problem of hybrid energy storage systems, researchers have proposed a real-time scheduling method based on dynamic programming and genetic algorithms, effectively improving the timeliness of energy storage scheduling. Other researchers have used genetic algorithms combined with simulated annealing algorithms to optimize user-side energy storage charging and discharging strategies, effectively improving the algorithm's optimization speed and convergence performance. Furthermore, to address the poor real-time performance of peak shaving and valley filling in battery energy storage systems, dynamic programming has been proposed for real-time optimization and adjustment of the system. These technologies have significant research value and increasingly widespread application value in power system optimization, microgrid energy storage, and industrial engineering.

[0007] The optimization scheduling problem of battery energy storage systems needs to consider both the economic efficiency of the system and the optimization scheduling of its charging and discharging strategies. Current algorithms only focus on economic efficiency in solving the energy storage scheduling problem, failing to comprehensively consider both aspects. These algorithms cannot simultaneously achieve both economic efficiency and real-time performance in peak shaving and valley filling, and they also lack sufficient adaptability in adjusting system stability and scheduling strategies, thus failing to improve the economic benefits of energy storage systems. This is because the current optimization algorithms are highly sensitive to parameter configuration during iteration, requiring parameter adjustments based on the problem, resulting in low computational efficiency. Secondly, these algorithms are primarily problem-oriented, with the solution process dependent on the problem or scenario, neglecting the specific characteristics of energy storage scheduling and the applicability of the solution model. Thirdly, the objective function and constraints for energy storage optimization scheduling are extremely complex, leading to high computational complexity. Current solution models and methods exhibit slow convergence speeds and require further improvement. Summary of the Invention

[0008] To address the aforementioned problems in existing technologies, this invention provides an adaptive optimization energy storage method based on reinforcement learning. The technical problem to be solved by this invention is achieved through the following technical solution:

[0009] This invention provides an adaptive optimization energy storage method based on reinforcement learning, comprising:

[0010] S1: Acquire users' electricity data, analyze the current electricity consumption of the power system, and build a demand-based billing model for users;

[0011] S2: Combining the demand billing model and considering the constraints, analyze and evaluate the economic benefits of the energy storage system under the current electricity consumption conditions to obtain a preliminary allocation scheme for the energy storage system;

[0012] S3: Based on the preliminary allocation scheme of the energy storage system, preprocess the energy storage action parameters and coarsely allocate the energy storage system action strategies for each stage;

[0013] S4: Transform the coarsely allocated energy storage system actions into an action sequence and correspond it to each time point to generate an energy storage system scheduling sequence ordered by time;

[0014] S5: Based on the current environmental characteristics, dynamically adjust the scheduling sequence of the energy storage system using the Q-learning algorithm to achieve optimal corporate profits.

[0015] In one embodiment of the present invention, the demand billing model is represented by the maximum return on investment max(E / C), where C represents the investment cost of the user installing the energy storage system and E represents the revenue of the energy storage system.

[0016] In one embodiment of the present invention, S2 includes:

[0017] S2.1: Construct constraints, including energy storage load constraints, energy storage system capacity constraints, and energy storage rate constraints, wherein,

[0018] Energy storage load constraints:

[0019] Energy storage system capacity constraint: L min,t ≤P t +δ i,t -ρ i,t ≤L max,t

[0020] Energy storage rate constraint: E max =β*P max

[0021] Where, δ max p represents the maximum discharge power. max P represents the maximum charging power. t S represents the energy storage load at time t. t L represents the battery state at time t. min，t L max，t E represents the minimum and maximum load values ​​of the energy storage system at time t. max P represents the energy storage system capacity, β represents the energy storage charge / discharge rate, and P represents the energy storage system capacity.max Indicates the rated power of the energy storage system;

[0022] S2.2: Combining the demand billing model and the constraints, analyze and evaluate the economic benefits of the energy storage system under the current electricity consumption conditions, and obtain a preliminary allocation scheme for the energy storage system.

[0023] In one embodiment of the present invention, S2.2 includes:

[0024] S2.21: Initialize power parameters, including rated power, charging and discharging power, and charging and discharging rate settings;

[0025] S2.22: Initialize the charging and discharging actions that the user needs to perform at different times throughout the day, forming a set of charging and discharging action sequences for each time period throughout the day;

[0026] S2.23: Determine whether the current action sequence satisfies the expected value of the objective function composed of the demand billing model and the constraints. If it does, output the pre-planned sequence; otherwise, return to step S2.22.

[0027] In one embodiment of the present invention, S3 includes:

[0028] S3.1: Considering energy constraints, charging and discharging power constraints, energy storage load constraints, and energy storage rate constraints, calculate the charging and discharging operation probabilities for the three stages of peak electricity consumption, valley electricity consumption, and level electricity consumption.

[0029] S3.2: Adjust the action sequence in the preliminary allocation scheme of the energy storage system according to the charging and discharging action probability, and coarsely divide the action strategy set of the energy storage system.

[0030] In one embodiment of the present invention, during peak electricity consumption periods, the probability of discharging > the probability of being idle > the probability of charging; during off-peak electricity consumption periods, the probability of charging > the probability of being idle > the probability of discharging; and during low electricity consumption periods, the probability of being idle > the probability of charging > the probability of discharging.

[0031] In one embodiment of the present invention, S5 includes:

[0032] S5.1: Construct an objective function for optimizing the energy storage system, which is measured by two dimensions: the number of charge-discharge cycles and the economic efficiency of the energy storage system.

[0033] S5.2: Triggering conditions for the energy storage system to adaptively adjust its scheduling based on changes in the action candidate set, emergencies, or maintenance of the energy storage system;

[0034] S5.3: Optimize the action sequence of the energy storage system based on the Q-learning algorithm until the optimal action sequence is obtained.

[0035] In one embodiment of the present invention, the objective function is expressed as:

[0036]

[0037] in, α1 + α2 = 1, This indicates that if the energy storage system k is in a charging state during the time period t, otherwise This indicates that if the energy storage system k is in a discharging state during the time interval t, otherwise This represents the expenditure incurred by energy storage system k during time period t when it transitions from an idle state to a charging state. C represents the benefit of energy storage system k transitioning from an idle state to a discharging state during time interval t; k T represents the investment cost of energy storage system k. k This indicates the time period during which the energy storage system k needs to schedule charging and discharging operations.

[0038] In one embodiment of the present invention, S5.3 includes:

[0039] S5.31: Load various initial information of the energy storage system. The initial information mainly includes the set of power time periods, the set of working states, the set of action initialization, the system operation data, the basic information of the probability of energy storage state occurrence, the Q table, and the basic parameters of the Q learning algorithm.

[0040] S5.32: Record the current operating status of the energy storage system and select the current operating mode based on the current action probability distribution;

[0041] S5.33: Evaluate based on the selected action and calculate the energy storage revenue value of the current working state;

[0042] S5.34: Using the energy storage revenue value at the current moment as the element for calculating the Q value, the maximum Q value of the next state is selected from the Q table according to the greedy strategy, thereby calculating the Q value of the current state;

[0043] S5.35: Based on the calculated Q value of the current state, update the Q value of the current state and arrange the actions at each time point according to the updated Q value, and finally maintain the optimal action sequence until the end of the whole process. The optimal Q value corresponds to the SN value in the objective function. The SN value is used as the influence parameter for solving the benefit of the energy storage system. Finally, the objective function value is calculated and the final action sequence is generated. The energy storage system executes according to the action sequence.

[0044] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0045] 1. This invention proposes an adaptive optimization energy storage method based on reinforcement learning. It considers numerous practical factors such as power, floating electricity price, user demand, and peak-valley constraints, taking into account both energy storage economics and optimization strategies. The method proposes a two-level programming model with energy storage investment return rate and energy storage revenue as objectives. Considering environmental changes and emergencies, the action sequence output of the energy storage system scheduling layer is used as the input of the planning layer. The system economic evaluation of the planning layer is fed back to the scheduling layer. This process is iterated repeatedly, thereby realizing the adaptive learning process of the energy storage system.

[0046] 2. This invention uses reinforcement learning to drive algorithm evolution, generate energy storage scheduling sequences and calculate revenue values. This algorithm does not require prior knowledge and can adaptively adjust parameters to achieve dynamic learning and evolution of the system. It is superior to traditional scheduling algorithms in terms of globality, makes up for the shortcomings of traditional algorithms in terms of generality, reduces the dependence of traditional algorithms on different scenarios, and can be applied to different scenarios, ultimately obtaining the optimal scheduling strategy.

[0047] 3. This invention introduces action sequence replanning trigger conditions, which are divided into three types: action candidate set change, sudden state, and system maintenance. This ensures that the energy storage system can make real-time adjustments in response to complex environments. In the context of optimizing the scheduling of user-side battery energy storage systems under dynamic environments, this invention studies efficient energy storage scheduling algorithms to ensure efficient solutions to dynamic and complex energy storage scheduling problems, achieving adaptive real-time dynamic energy storage scheduling.

[0048] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0049] Figure 1 This is a flowchart illustrating the implementation of a traditional energy storage planning method.

[0050] Figure 2 This is a flowchart of an adaptive optimization energy storage method based on reinforcement learning provided in an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the scheduling process of an adaptive optimization energy storage method based on reinforcement learning provided in an embodiment of the present invention;

[0052] Figure 4 This is a billing model diagram for domestic electricity prices provided in an embodiment of the present invention;

[0053] Figure 5 This is a flowchart of an energy storage planning stage provided by an embodiment of the present invention;

[0054] Figure 6 This is a pre-allocation relationship diagram of charging and discharging actions at various time points provided in an embodiment of the present invention;

[0055] Figure 7 This is a diagram illustrating the relationship between actions and energy storage system constraints, provided in an embodiment of the present invention.

[0056] Figure 8 This is a flowchart of a scheduling layer processing provided in an embodiment of the present invention;

[0057] Figure 9 This is a flowchart of an energy storage system action sequence optimization provided by an embodiment of the present invention;

[0058] Figure 10 This is a schematic diagram of energy storage planning after the action candidate set is changed, provided by an embodiment of the present invention;

[0059] Figure 11 This is a schematic diagram of a replanning process under an emergency, provided by an embodiment of the present invention;

[0060] Figure 12 This is a schematic diagram of a system maintenance replanning provided in an embodiment of the present invention;

[0061] Figure 13 This is a reinforcement learning model diagram provided in an embodiment of the present invention;

[0062] Figure 14 This is a flowchart of an energy storage optimization scheduling algorithm based on Q-reinforcement learning provided in an embodiment of the present invention;

[0063] Figure 15 This is the electricity load of a certain enterprise from May to August, as provided in an embodiment of the present invention;

[0064] Figure 16 This is a comparison chart of enterprise power load between the method of this invention and the traditional TES algorithm;

[0065] Figure 17 This is a performance analysis of the scheduling strategies of the method in this embodiment of the invention and the traditional TES algorithm;

[0066] Figure 18 This is a convergence speed curve of the method in this embodiment of the invention and the traditional TES algorithm. Detailed Implementation

[0067] To further illustrate the technical means and effects of this invention in achieving its intended purpose, the following detailed description of the adaptive optimization energy storage method based on reinforcement learning proposed in accordance with this invention is provided in conjunction with the accompanying drawings and specific embodiments.

[0068] The foregoing and other technical contents, features, and effects of the present invention will be clearly presented in the following detailed description of specific embodiments in conjunction with the accompanying drawings. Through the description of the specific embodiments, a more in-depth and concrete understanding can be gained of the technical means and effects adopted by the present invention to achieve its intended purpose. However, the accompanying drawings are for reference and illustration only and are not intended to limit the technical solutions of the present invention.

[0069] The implementation of this invention comprises two parts: a planning layer and a scheduling layer. The planning layer initializes parameters, including enterprise electricity consumption parameters and energy storage system action parameters, and performs economic evaluation and optimized energy storage allocation for the energy storage scheduling system. The scheduling layer performs action strategy preprocessing, initial state setting, and, considering charging and discharging constraints, generates a charging and discharging sequence that meets the conditions according to the charging and discharging scheduling strategy. Please refer to [link to relevant documentation]. Figure 2 and Figure 3 The adaptive optimization energy storage method in this embodiment includes:

[0070] S1: Obtain users' electricity data, analyze the current electricity consumption of the power system, and build a demand-based billing model for users.

[0071] Specifically, the first step is to import electricity data from users (e.g., businesses) and analyze their electricity consumption patterns and characteristics. These patterns refer to dynamic changes in user load and usage scenarios. Currently, domestic electricity pricing methods are divided into two categories: a combination of "basic electricity price + energy consumption pricing" and a combination of "basic electricity price + demand pricing," as shown below. Figure 4 As shown, different billing models incur different costs. The electricity-based billing model uses actual electricity usage as the charging standard, while the demand-based billing model dynamically calculates charges based on the electricity demand reported by the enterprise. Demand exceeding the reported demand is charged at double the existing rate. Given the flexibility and dynamic adjustability of the demand-based billing model, enterprises can effectively control their electricity consumption by choosing this method. Therefore, in the initial planning stage, it is necessary to start with the enterprise's electricity consumption characteristics and first construct the enterprise's demand-based billing model.

[0072] Specifically, let C represent the investment cost for a user to install an energy storage system, and E represent the revenue from the energy storage system. Then, the enterprise's demand-based billing model can be represented by the maximum return on investment, max(E / C). This embodiment of the invention defines the parameters affecting the return on investment of the energy storage system, including: the usable lifespan of the energy storage system (N), and the investment payback period of the energy storage system (N...). T The monthly savings in electricity demand (C1) after installing the energy storage system, the monthly savings in basic electricity cost (C2) after installing the energy storage system, and the revenue generated by the energy storage system (S) T ) and monthly basic electricity fee (B T The benefits of energy storage systems can be specifically described as follows:

[0073]

[0074] in, C2=(S T -B T T represents the number of days, n represents the time point (1 to 24 hours), m represents the electricity cost for different time periods, and δ i，t ρ represents the discharge rate of the energy storage system at time t on day i. i，t This represents the charging rate of the energy storage system at time t on day i.

[0075] S2: Combining the demand-based billing model and considering the constraints, analyze and evaluate the economic benefits of the energy storage system under the current electricity consumption conditions to obtain a preliminary allocation scheme for the energy storage system.

[0076] In this embodiment, step S2 specifically includes the following steps:

[0077] S2.1: Construct constraints, including energy storage load constraints, energy storage system capacity constraints, and energy storage rate constraints.

[0078] Given the varying peak-valley electricity prices across different regions, and the existence of areas with small peak-valley price differences, an evaluation model with return on investment as the objective function is constructed based on the demand-based billing model. The revenue of the energy storage system can be represented by the revenue period of the energy storage system and the electricity cost savings for users. Considering constraints such as energy storage load constraints, energy storage system capacity constraints, and energy storage ratio constraints, and incorporating energy storage strategies into the planning process, an economic evaluation result of the energy storage system and a preliminary allocation plan for the energy storage system are generated.

[0079] Specifically, after initializing electricity consumption parameters and constructing the user demand billing model, it is necessary to assess the feasibility and economic benefits of installing energy storage systems from an economic perspective. Due to varying peak-valley electricity prices across different regions, and the existence of areas with small peak-valley price differences, the revenue of the energy storage system can be represented by the system's revenue period and user electricity cost savings, based on the demand billing model from the previous stage. Furthermore, this model needs to consider constraints such as energy storage load constraints, energy storage system capacity constraints, and energy storage ratio constraints. Simultaneously, energy storage strategies must be incorporated into the planning process, ultimately providing the economic evaluation results and preliminary allocation scheme for the energy storage system. In this process, the energy storage load constraints, energy storage system capacity constraints, and energy storage ratio constraints can be expressed by the following formulas:

[0080] Energy storage load constraints are expressed as follows:

[0081]

[0082] The capacity constraint of an energy storage system is expressed as:

[0083] L min,t ≤P t +δ i,t -ρ i,t ≤L max,t

[0084] Energy storage ratio constraint is expressed as:

[0085] E max =β*P max

[0086] Where, δ max ρ represents the maximum discharge power. max P represents the maximum charging power. t S represents the energy storage load at time t. t L represents the battery state at time t. min，t L max，t E represents the minimum and maximum load values ​​of the energy storage system at time t. max P represents the energy storage system capacity, β represents the energy storage charge / discharge rate, and P represents the energy storage system capacity. max This indicates the rated power of the energy storage system.

[0087] S2.2: Combining the demand-based billing model and the constraints, analyze and evaluate the economic benefits of the energy storage system under the current electricity consumption conditions to obtain a preliminary allocation scheme for the energy storage system. For details, please refer to... Figure 5 The steps can be represented as follows:

[0088] S2.21: Initialize power parameters, including rated power, charging / discharging power, and charging / discharging rate settings;

[0089] S2.22: Initialize the charging and discharging actions that the user needs to perform at different times throughout the day, forming a set of charging and discharging action sequences for each time period throughout the day;

[0090] S2.23: Determine whether the current action sequence satisfies the expected value of the objective function composed of the demand billing model and the constraints. If it does, output the pre-planned sequence; otherwise, return to step S2.22.

[0091] S3: Transmit the preliminary energy storage system allocation scheme obtained in step S2 to the scheduling layer, perform energy storage action parameter preprocessing, and perform coarse allocation of energy storage strategies for each stage.

[0092] The first issue to address in this step is the probability of charging and discharging actions occurring at each point in time. Constraints related to the energy storage system's operating strategy include energy constraints, charging and discharging power constraints, energy storage load constraints, and energy storage rate constraints. Among these, the energy constraint states that the charging and discharging process within any given time period cannot exceed the battery's maximum capacity EN. i The charging and discharging power constraint means that the power value at any given time cannot exceed the rated power. To complete energy storage dispatch, these four types of constraints must be met. Therefore, the first step is to pre-allocate the probability of action at each time point according to the three stages of electricity consumption: peak, valley, and flat. Figure 6 As shown, in this embodiment of the invention, p(s) t s t+1 a t ) indicates that the state of the energy storage system is determined by s t State through action a t Transfer to s t+1 The probability of a state, when the energy storage system is operating, is determined by state s. t Convert to s t+1 The probabilities of different states vary. For example, during off-peak hours, the probability of an energy storage system choosing to operate is: charging probability > idle probability > discharging probability. Conversely, during peak hours, the probabilities are reversed. Due to the difference between peak and off-peak electricity prices, p(s) varies under different conditions. t s t+1 a t The following constraints need to be met. Then, based on the allocation results, the action sequence input to the planning layer is adjusted according to the probability of occurrence.

[0093] This means that during peak electricity consumption periods, the probability of the energy storage system choosing to discharge is greater than the probability of the system being idle. Under these circumstances, the probability of the energy storage system choosing to charge is the lowest. This means that during periods of low energy consumption, the probability of the energy storage system choosing an idle action is greater than the probability of the system choosing a charging action. Under these circumstances, the probability of the energy storage system choosing a discharging action is the lowest. This indicates that during periods of low electricity demand, the probability of the energy storage system choosing to charge is greater than the probability of the system being idle. Under these circumstances, the probability of the energy storage system choosing to discharge is minimized. This indicates the charging state to be selected at time t. This indicates the selection of the discharge state at time t. This indicates that the idle state is selected at time t. Then, based on the allocation results, the action sequence input to the planning layer is adjusted according to the probability of occurrence. The specific implementation of this step is as follows:

[0094] S3.1: Considering energy constraints, charging and discharging power constraints, energy storage load constraints, and energy storage rate constraints, calculate the probability of operation in the three stages of peak electricity consumption, valley electricity consumption, and level electricity consumption.

[0095] When an energy storage system is operating, during off-peak hours, the probability of the system choosing an action is: charging probability > idle probability > discharging probability. During peak hours, the probability is: discharging probability > idle probability > charging probability. Therefore, before scheduling energy storage operations, the system needs to analyze the actions at a specific moment based on the current situation and use the appropriate actions for each moment as constraints for charging and discharging action allocation. During charging, system overhead parameters to consider include: the electricity cost for the time period, the electricity cost consumed by the charging system, and the amount of electricity charged. When the system is idle, the electricity cost for the time period is considered. During discharging, system overhead parameters to consider include: the electricity cost for the time period, the electricity cost saved by the discharging system, and the amount of electricity discharged.

[0096] S3.2: Adjust the action sequence in the preliminary allocation scheme of the energy storage system according to the charging and discharging action probability, and coarsely divide the action strategy set of the energy storage system.

[0097] The charging and discharging process satisfies multiple constraints, including energy constraints, a maximum of one charging and discharging operation per time period, minimum and maximum load constraints of the energy storage system, and rated power constraints of the energy storage system. Based on these constraints, the charging and discharging actions are pre-allocated to obtain coarse allocation results.

[0098] S4: Transform the coarsely allocated energy storage system actions into an action sequence and correspond it to each time point to generate an energy storage system scheduling sequence ordered by time.

[0099] After step S3, although the action strategy of the energy storage system has been roughly divided, the impact of each adjustment on the system's benefits remains unclear. This step transforms the adjusted actions into an action sequence and assigns it to each time point. Several constraints on the energy storage system have been defined previously. Clearly, under these constraints, the action strategy selection differs from the strategy generated during the planning process. Based on these constraints, the initial action sequence for the scheduling process can be generated. Please refer to [link to relevant documentation]. Figure 7 , Figure 7 This document describes the basic constraints that must be satisfied to generate an energy storage action sequence. During energy storage scheduling at each time step, these constraints must be used to generate the appropriate action sequence. Generally, given the probability of action occurrence, the action sequence satisfying the constraints is not unique. After generating the actions for each time step, the next step is to find a sequence from the processed actions that satisfies the constraints as the result of this scheduling operation. Figure 8 As shown, after the scheduling results are generated, the energy storage system scheduling sequence can be generated based on the current scheduling results.

[0100] S5: Energy Storage System Adaptive Learning: Based on the current environmental characteristics, the energy storage system scheduling sequence is dynamically adjusted using the Q-learning algorithm to achieve optimal corporate profits.

[0101] In the process of optimizing the scheduling of energy storage systems, considering factors such as environmental changes, unforeseen circumstances, and economic considerations, this process uses the action sequence output of the energy storage system scheduling layer as input to the planning layer. The system economic evaluation from the planning layer is then fed back to the scheduling layer, and this process is iterated repeatedly, thus achieving the adaptive learning process of the energy storage system. Please see [link to relevant documentation]. Figure 9 , Figure 9 This is a flowchart of an energy storage system action sequence optimization provided by an embodiment of the present invention. In the flowchart, the dashed line A represents the initial action sequence, and the solid line B represents the action sequence updated after the energy storage system scheduling layer and planning layer are combined and re-planned and scheduled.

[0102] After the energy storage system planning layer, the energy storage system needs to execute response actions according to the scheduling sequence. However, due to environmental changes, unforeseen circumstances, and economic considerations, the pre-set action sequence may fail to meet actual usage needs, resulting in excessive economic investment for the enterprise. To avoid such economic losses during the operation of the energy storage system, dynamic adjustments to the actions are necessary based on current environmental characteristics to achieve optimal enterprise benefits. The specific implementation of this step is as follows:

[0103] S5.1: Construct an objective function for optimizing the energy storage system, which is measured by two dimensions: the number of charge-discharge cycles and the economic efficiency of the energy storage system.

[0104] The optimization objective of the user-side energy storage planning problem is to minimize user electricity costs while considering the balanced power consumption of each module in the user's industry chain. The operational status of user energy storage is measured by recording the number of charge-discharge cycles (SN) of the energy storage system, while the overall benefit of user energy storage is determined based on the economics of the energy storage system. The objective function of user-side energy storage consists of the above two dimensions; therefore, the objective function of the energy storage dispatch system can be expressed as:

[0105]

[0106] in, α1 + α2 = 1, This indicates that if the energy storage system k is in a charging state during the time period t, otherwise This indicates that if the energy storage system k is in a discharging state during the time interval t, otherwise This represents the expenditure incurred by energy storage system k during time period t when it transitions from an idle state to a charging state. The value of energy storage system k is represented by its gain from transitioning from an idle state to a discharged state during time interval t; k∈S indicates that energy storage system k is considered; C k T represents the investment cost of energy storage system k. k This indicates the time period during which the energy storage system k needs to schedule charging and discharging operations.

[0107] S5.2: Triggering conditions for the energy storage system to adaptively adjust its scheduling based on changes in the action candidate set, emergencies, or maintenance of the energy storage system.

[0108] When an energy storage system formulates its action sequence, it first requires an initial sequence input. This input serves as the trigger for optimized scheduling, and the triggering conditions need to be set in advance before the energy storage system's planning and scheduling. During the optimized scheduling process, once the conditions for replanning are met, the system will complete the energy storage planning and action sequence rescheduling response according to the system settings. Generally, the situations requiring replanning during energy storage system scheduling can be categorized into the following three types:

[0109] 1) Action candidate set change. This involves adjusting the action candidate set at a specific moment. This could be due to changes in hourly electricity prices or adjustments in corporate strategies. For action candidate set changes, assume the energy storage system at a certain moment T... k The original action candidate set was C = {C1, C2, C3}. Due to some special reason, T k If a certain action C1 is no longer considered in the candidate set, the energy storage system needs to replan and schedule based on the new candidate set C' = {C2, C3} to finally obtain an action strategy that meets the conditions. Please refer to [link to relevant documentation]. Figure 10 , Figure 10 This is a schematic diagram of energy storage planning after the action candidate set is changed, provided by an embodiment of the present invention.

[0110] 2) Unexpected Situations. During operation, the energy storage system may encounter unforeseen circumstances such as sudden changes in enterprise power demand, leading to unforeseen issues in the action sequence strategy. In such cases, the action sequence needs to be replanned to ensure maximum benefit under those circumstances. Please refer to [link / reference]. Figure 11 , Figure 11 This is a schematic diagram of a replanning process under emergency conditions provided by an embodiment of the present invention. Figure 11 As shown, T k When a company's electricity consumption suddenly changes, its action strategy changes from C1 to C3. Resources need to be allocated to ensure the company's operation under these circumstances. At this time, the company determines its action strategy for this moment, and the energy storage system needs to adjust the entire energy storage strategy based on the known situation to maximize the company's benefits.

[0111] 3) Energy Storage System Maintenance. During operation, energy storage systems may be scheduled for maintenance, resulting in a loss of peak-shaving and valley-filling capabilities. In this case, it's necessary to consider the action strategy for the periods when the system is not yet scheduled for maintenance. Based on system maintainability indicators, the system can be restored and switched to normal operation within the scheduled maintenance time t. During this process, the energy storage status for the intermediate time period t needs to be set to a uniform idle state, and rescheduling should be performed based on this premise. Please refer to [link to relevant documentation]. Figure 12 , Figure 12 This is a schematic diagram of a system maintenance and replanning provided by an embodiment of the present invention. During the operation of the energy storage system, due to T k The maintenance work needs to be maintained for t hours. The energy storage system adjusts the scheduling strategy based on the known situation to ensure the most efficient use of resources.

[0112] S5.3: Optimize the action sequence of the energy storage system based on the Q-learning algorithm until the optimal action sequence is obtained.

[0113] Q-learning, a typical offline control strategy in reinforcement learning, is based on Markov theory and uses a state-action value function for iterative solution, effectively solving sequential decision-making problems through learning. However, in solving energy storage scheduling problems, traditional scheduling methods rely on greedy strategies for iterative solutions, which have significant shortcomings in global optimization and are prone to getting trapped in readily available optima.

[0114] Q-learning algorithms have the following advantages over general mathematical optimization:

[0115] 1) Q-learning algorithms based on reinforcement learning are model-free algorithms, meaning they do not require highly accurate models during the optimization process. For example, in energy storage optimization, the coupling relationships between various modules can make it difficult to establish a solution model. Therefore, reinforcement learning offers a degree of versatility.

[0116] 2) The reward function is crucial in reinforcement learning strategies, as it avoids complex operations such as differentiation and matrix inversion caused by the objective function, reducing computation time. Furthermore, based on the idea of ​​iterative trial and error, the Q-learning algorithm enables the agent to possess a certain level of reasoning and learning ability by continuously exploring unknown domains and utilizing existing experience, demonstrating good generalization ability and showing certain advantages over traditional heuristic algorithms.

[0117] Please see Figure 13 , Figure 13This invention provides a reinforcement learning model diagram. In this model, the agent connects to the environment through perception and action. Taking the current state as input, it selects an action to generate output. The action selection affects the state of the environment and is fed back to the agent. The agent's action selection generates a reward value, which accumulates with each action selection, eventually leading to system convergence through repeated iterations. For details, please refer to [link to relevant documentation]. Figure 14 , Figure 14 This is a flowchart of an energy storage optimization scheduling algorithm based on Q-reinforcement learning provided by an embodiment of the present invention. The adaptive energy storage optimization scheduling method based on Q-learning includes the following steps, and the specific process is described below:

[0118] S5.31: Initialization. Load various initial information of the energy storage system. For the energy storage planning and scheduling system, the initial information mainly includes the set of power time periods, the set of working states, the set of action initializations, system operation data, basic information on the probability of energy storage state occurrence, the Q-table, and the basic parameters of the Q-learning algorithm, etc.

[0119] S5.32: Action Selection. Record the current operating status of the energy storage system and select the current operating mode (charging, discharging, idle) based on the current action probability distribution. In this stage, if the probabilities of each action are equal, an action can be randomly selected to proceed to the next action selection.

[0120] S5.33: Evaluation Phase. In this phase, an evaluation is conducted based on the selected actions, calculating the reward value for the current operational state. For energy storage systems, this involves selecting energy storage actions, evaluating the benefits that can be obtained from executing those actions, and finally recording the evaluation results as input for subsequent phases.

[0121] S5.34: Calculate the Q-value. After evaluation, the current energy storage revenue can be used as a factor in calculating the Q-value. The maximum Q-value for the next state is selected from the Q-table using a greedy strategy (all Q-values ​​are initially 0), thus calculating the Q-value for the current state. During this process, to ensure the exploratory nature of the algorithm, the greedy strategy selection probability needs to be reasonably set. The energy storage system takes action a at time t. t The value function is expressed as follows:

[0122]

[0123]

[0124]

[0125]

[0126] Where Q(s) t a t ) indicates that at time t, the energy storage system is supplied by s t State selection action a t The subsequent profit value, Q old (s t a t () represents the Q value before the update, Q new (s t a t ) represents the Q-value generated in this update; α (0 < α < 1) represents the learning factor; r(s) t s t+1 a t ) indicates that the system state is determined by s t After action a t Transfer to s t+1 The reward value obtained from a state change represents the benefit gained from the change in the energy storage system's state. It should be noted that when the energy storage system's State of Charge (SOC) does not meet the constraints, a penalty factor for the charging and discharging states needs to be added. and μ c and μ d This represents the rate of return for choosing to charge or discharge in the current state; the rate of return varies at different times. t s t+1 a t ) indicates that the system state is determined by s t After action a t Transfer to s t+1 The probability of a state; γ represents the decay coefficient γ∈(0,1); P k Indicates the price of electricity; s t This indicates the current state of the energy storage system (charging, discharging, idle).

[0127] S5.35: Update the Q-table. Based on the calculated Q-values, update the Q-values ​​for the current state, and arrange the actions at each time point according to the updated Q-values, ultimately maintaining the optimal action sequence until the entire process ends. The optimal Q-value corresponds to the SN value in the objective function, and the SN value can be used as an influence parameter for solving the energy storage system's revenue. Finally, the objective function value is calculated, and the final action sequence is generated. The energy storage system executes according to the action sequence.

[0128] The goal of Q-learning is to adaptively learn and find the optimal policy based on iterative updates of the state-action value (Q(s, a)), thus solving the action sequence decision problem. As a model-free reinforcement learning algorithm, Q-learning is trained through interaction with the environment. Q-learning consists of four parts: state, action, policy, and environment, and can be represented as follows:<S,A,L,E> The four-tuple. In the Q-learning algorithm, the energy storage system is calculated from its current state s. t Select the next action (charging, discharging, idle) a t To transition to the next state s t+1 The reward value Q(s, a) is obtained to achieve the optimal action sequence π, thus realizing the adaptive modulation energy storage strategy. Furthermore, the action selection process at each step is based on a greedy strategy π(s, a). t Expand (its expression is as follows) and select the action corresponding to the maximum Q value.

[0129]

[0130] The effectiveness of the adaptive optimization energy storage method based on reinforcement learning in this invention can be further illustrated by the following comparative simulation experiments.

[0131] (I) Simulation Conditions

[0132] All simulation experiments in this embodiment of the invention are based on a laptop computer with a Core I7-8550 1.8GHz CPU, 8GB of memory, Windows 7 operating system, and Matlab 2020a coding environment.

[0133] (II) Simulation Content

[0134] This invention provides three different scenarios for testing. In scenario one, the enterprise does not consider the integration of energy storage systems. In scenario two, the enterprise considers the integration of multiple different types of energy storage systems and uses a traditional energy storage scheduling algorithm to solve for the charging and discharging strategies of these systems. In scenario three, the enterprise considers the integration of multiple different types of energy storage systems and uses the Q-learning-based energy storage scheduling algorithm of this invention to solve for the charging and discharging strategies of these systems. Scenarios one and two are used to verify the method proposed in this paper. Scenario three can be used simultaneously to verify the economics of the energy storage model and the effectiveness of the adaptive optimization energy storage (RLES) algorithm and the traditional energy storage (TES) algorithm proposed in this invention for energy storage scheduling.

[0135] This simulation process uses electricity data from a large machinery manufacturing enterprise in Foshan as a test case. The electricity consumption of the enterprise is statistically analyzed from two perspectives: daily time periods (0:00-24:00) and daily data collection. Figure 15 As shown. By Figure 15It can be seen that the company's electricity load exhibits obvious peak-valley characteristics from May to mid-August, with daily electricity consumption at full capacity and significant peak-valley differences. The parameters of the energy storage products and the electricity price tables for peak and off-peak periods in this region are shown in Tables 1 and 2.

[0136] Table 1 Energy Storage System Parameter Table

[0137] Rated power (kW) 30 30 50 100 200 Battery capacity (kW·h) 40 50 100 200 500 Equipment lifespan 10 years 10 years 10 years 10 years 10 years Charge and discharge efficiency 90％ 90％ 90％ 90％ 90％ Energy storage system ratio 2 2 2 2 2 Electricity cost 4 yuan / kW·h 4 yuan / kW·h 4 yuan / kW·h 4 yuan / kW·h 4 yuan / kW·h State of charge (0.1，1) (0.1，1) (0.1，1) (0.1，1) (0.1，1) Price 200,000 yuan 300,000 yuan 600,000 yuan 660,000 yuan 1.31 million yuan

[0138] Table 2 Peak-Valley Electricity Prices

[0139] peak 09:00-12:00、19:00-22:00 1.367 flat section 08:00-09:00、12:00-19:00、22:00-24:00 0.81 trough 00:00-8:00 0.33

[0140] The historical data of the company from May 2022 to September 2022 was used as a test dataset to evaluate the economics of the company's adoption of energy storage systems. The company's energy storage capacity and the economic analysis of the company's energy storage under different energy storage configurations are shown in Tables 3 to 5.

[0141] Table 3. Annual electricity cost savings for enterprises after integrating energy storage (unit: yuan)

[0142] CN-1 39119.87 58348.17 CN-2 40117.15 54335.8 CN-3 48249.35 63141.35 CN-4 48249.35 63141.35 CN-5 48249.35 63141.35

[0143] Table 4. Results of Energy Storage Economic Analysis Based on Traditional TES Algorithm

[0144]

[0145]

[0146] Table 5. Energy storage economic analysis results based on the RLES algorithm of the present invention.

[0147]

[0148] The simulation results show that after installing an energy storage system, the company's electricity costs are significantly reduced. Due to the company's own production capacity limitations, the cost savings tend to stabilize after increasing the energy storage battery capacity. The company can recover its investment costs by installing CN-1, CN-2, and CN-3 energy storage products. Under the TES algorithm, the return on investment for all three is above 20%, with CN-1 and CN-2 exceeding 70%. Under the RLES algorithm, the return on investment for all three is above 50%, with CN-1 and CN-2 exceeding 100%. Comparative analysis shows that the company's benefits are maximized after installing the CN-1 model energy storage product. Furthermore, the simulation results indicate that considering different configurations (energy storage power and energy storage capacity), changes in energy storage capacity do not significantly affect the company's economic benefits. However, changes in energy storage power significantly alter the company's economic benefits. Further analysis reveals that the curves showing the relationship between energy storage investment and returns exhibit a trend of first decreasing and then increasing investment as energy storage capacity configuration changes, while the return trend is the opposite. This indicates that when installing energy storage systems, companies need to consider appropriate parameter configurations to maximize returns.

[0149] Then, the performance of the RLES algorithm proposed in this embodiment of the invention in solving the energy storage optimization scheduling problem is verified (parameter settings are shown in Table 6). This embodiment verifies the algorithm from two aspects: performance indicators (such as battery utilization rate and annual revenue) and scheduling strategy in the energy storage scheduling process.

[0150] Table 6 Initialization parameter values ​​for the Q-learning algorithm

[0151] Value 0.1 0.9 0.9

[0152] Table 7 Performance Analysis of Algorithm for Energy Storage Scheduling

[0153] Battery utilization 66.1％ 57.1％ Annual income 41,000 yuan 21,800 yuan

[0154] The results in Table 7 show that, under the same conditions, the performance index analysis of energy storage optimization based on the RLES algorithm and the traditional energy storage optimization algorithm shows that the RLES algorithm has a greater advantage in performance, significantly improves the battery utilization rate of the system, and can obtain higher annual comprehensive income.

[0155] Regarding the verification of the algorithm scheduling strategy, we will take a specific workday of the company as an example. Please refer to [link / reference needed]. Figure 16 , Figure 17 and Figure 18 , Figure 16 This is a comparison chart of enterprise power load between the method of this invention and the traditional TES algorithm; Figure 17 This is a performance analysis of the scheduling strategies of the method in this embodiment of the invention and the traditional TES algorithm; Figure 18This is a graph showing the convergence speed of the method in this embodiment of the invention and the traditional TES algorithm. From... Figure 16 It can be seen that after enterprises install energy storage, both algorithms can achieve peak shaving and valley filling, reducing the enterprise's electricity load. Figure 17 It can be seen that both the TES and RLES algorithms can optimize scheduling to achieve peak and valley filling for enterprise energy storage. During the period from 0:00 to 8:00, the electricity price is low, and the energy storage system begins charging the batteries. Charging stops when the batteries reach their state of charge (SOC) constraint, and the system remains idle. During the periods from 9:00 to 12:00 and 19:00 to 21:00, the electricity price increases, and the energy storage system begins discharging. During the period from 22:00 to 24:00, the energy storage system's batteries reach their SOC constraint and remain idle. In the energy storage strategy optimization process, deviations exist between the RLES algorithm's battery charging and discharging action strategies and the TES algorithm's battery charging and discharging action strategies at the action sequences of 2:00, 3:00, 6:00-8:00, 11:00, 13:00-15:00, 17:00, and 20:00-21:00, respectively. Further analysis shows that the addition of the Q-learning algorithm improves the actual benefits of the energy storage system and optimizes the scheduling strategy. Experimental results verify the scheduling performance of the RLES algorithm presented in this paper. Figure 18 This figure compares the convergence curves of the RLES algorithm proposed in this invention with the traditional TES algorithm in solving the energy storage scheduling problem. As shown in the figure, the TES algorithm converges after 68 iterations, while the RLES algorithm converges after 13 iterations. The algorithm proposed in this paper can quickly obtain the optimal solution, demonstrating superior convergence performance. Experimental results show that the method of this invention can effectively solve the energy storage optimization scheduling problem. This method significantly outperforms the comparative algorithm in terms of energy storage system economy and energy storage action strategies. In the energy storage scheduling process, the application of the Q-learning algorithm improves energy storage scheduling efficiency and reduces the total electricity input for enterprises.

[0156] In summary, this invention's reinforcement learning-based adaptive optimization energy storage method considers numerous practical factors such as power, floating electricity prices, user demand, and peak-valley constraints, taking into account both energy storage economics and optimization strategies. It proposes a two-layer programming model with energy storage investment return rate and energy storage revenue as objectives. Considering environmental changes and unforeseen circumstances, the action sequence output of the energy storage system scheduling layer is used as the input to the planning layer. The system economic evaluation of the planning layer is then fed back to the scheduling layer, and this iterative process is repeated to achieve the adaptive learning process of the energy storage system. This invention uses reinforcement learning to drive algorithm evolution, generating energy storage scheduling sequences and calculating revenue values. This algorithm does not require prior knowledge and can adaptively adjust parameters to achieve dynamic system learning and evolution. Globally, it outperforms traditional scheduling algorithms, overcoming the shortcomings of traditional algorithms in terms of versatility and reducing their dependence on specific scenarios. It can be applied to different scenarios, ultimately obtaining the optimal scheduling strategy.

[0157] Furthermore, this invention introduces three types of action sequence replanning trigger conditions: action candidate set change, sudden state, and system maintenance. This ensures that the energy storage system can make real-time adjustments in response to complex environments. In the context of optimizing the scheduling of user-side battery energy storage systems under dynamic environments, this invention studies efficient energy storage scheduling algorithms to ensure efficient solutions to dynamic and complex energy storage scheduling problems, achieving adaptive real-time dynamic energy storage scheduling.

[0158] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. An adaptive optimization energy storage method based on reinforcement learning, characterized in that, include: S1: Acquire users' electricity data, analyze the current electricity consumption of the power system, and build a demand-based billing model for users; S2: Combining the demand billing model and considering the constraints, analyze and evaluate the economic benefits of the energy storage system under the current electricity consumption conditions to obtain a preliminary allocation scheme for the energy storage system; S3: Based on the preliminary allocation scheme of the energy storage system, preprocess the energy storage action parameters and coarsely allocate the energy storage system action strategies for each stage; S4: Transform the coarsely allocated energy storage system actions into an action sequence and correspond it to each time point to generate an energy storage system scheduling sequence ordered by time; S5: Based on the current environmental characteristics, dynamically adjust the scheduling sequence of the energy storage system using the Q-learning algorithm to achieve optimal corporate profits; The demand-based billing model uses the maximum return on investment (max) E / C ) indicates that, This indicates the investment cost for users to install energy storage systems. Indicates the revenue of the energy storage system; The parameters affecting the return on investment of energy storage systems include: the service life of the energy storage system. Investment payback period for energy storage systems The monthly savings in electricity costs after installing an energy storage system The basic electricity cost saved each month after installing an energy storage system Energy storage system power supply revenue And monthly basic electricity fee The benefits of energy storage systems can be specifically described as follows: in, , , , This indicates a point in time, specifically within 1 to 24 hours, with values ​​ranging from 1 to 24. Indicates the first i Heavenly t Discharge rate of the energy storage system at each time point Indicates the first i The charging rate of the energy storage system at time point t on day t; S2 includes: S2.1: Construct constraints, including energy storage load constraints, energy storage system capacity constraints, and energy storage rate constraints, wherein, in, express Energy storage load at any time, express Battery status at any time. , express The minimum and maximum load values ​​of the energy storage system at all times. Indicates the capacity of the energy storage system. Indicates the energy storage charge / discharge rate. Indicates the rated power of the energy storage system; S2.2: Combining the demand billing model and the constraints, analyze and evaluate the economic benefits of the energy storage system under the current electricity consumption conditions, and obtain a preliminary allocation scheme for the energy storage system; S5 includes: S5.1: Construct an objective function for optimizing the energy storage system, which is measured by two dimensions: the number of charge-discharge cycles and the economic efficiency of the energy storage system. S5.2: Triggering conditions for the energy storage system to adaptively adjust its scheduling based on changes in the action candidate set, emergencies, or maintenance of the energy storage system; S5.3: Optimize the action sequence of the energy storage system based on the Q-learning algorithm until the optimal action sequence is obtained; The objective function is expressed as: in, , , , , Indicates if energy storage system exist The device is in a charging state during the specified time period. =1, otherwise =0; Indicates if energy storage system exist The period is a discharge state. =1, otherwise =0; Indicates energy storage system exist Expenses incurred when an idle state transitions to a charging state during a specific time period; Indicates energy storage system exist The benefit of transitioning from an idle state to a discharging state within a certain time period; Indicates energy storage system k The investment cost, Indicates energy storage system k The time period for charging and discharging needs to be scheduled.

2. The adaptive optimization energy storage method based on reinforcement learning according to claim 1, characterized in that, S2.2 includes: S2.21: Initialize power parameters, including rated power, charging and discharging power, and charging and discharging rate settings; S2.22: Initialize the charging and discharging actions that the user needs to perform at different times throughout the day, forming a set of charging and discharging action sequences for each time period throughout the day; S2.23: Determine whether the current action sequence satisfies the expected value of the objective function composed of the demand billing model and the constraints. If it does, output the pre-planned sequence; otherwise, return to step S2.

22.

3. The adaptive optimization energy storage method based on reinforcement learning according to claim 2, characterized in that, S3 includes: S3.1: Considering energy constraints, charging and discharging power constraints, energy storage load constraints, and energy storage rate constraints, calculate the charging and discharging operation probabilities for the three stages of peak electricity consumption, valley electricity consumption, and level electricity consumption. S3.2: Adjust the action sequence in the preliminary allocation scheme of the energy storage system according to the charging and discharging action probability, and coarsely divide the action strategy set of the energy storage system.

4. The adaptive optimization energy storage method based on reinforcement learning according to claim 3, characterized in that, During peak electricity consumption periods, the probability of discharging > the probability of being idle > the probability of charging; during off-peak electricity consumption periods, the probability of charging > the probability of being idle > the probability of discharging; during low electricity consumption periods, the probability of being idle > the probability of charging > the probability of discharging.

5. The adaptive optimization energy storage method based on reinforcement learning according to claim 4, characterized in that, S5.3 includes: S5.31: Load various initial information of the energy storage system. The initial information mainly includes the set of power time periods, the set of working states, the set of action initialization, system operation data, the basic information of the probability of energy storage state occurrence, the Q table, and the basic parameters of the Q learning algorithm. S5.32: Record the current operating status of the energy storage system and select the current operating mode based on the current action probability distribution; S5.33: Evaluate based on the selected action and calculate the energy storage revenue value of the current working state; S5.34: Using the energy storage revenue value at the current moment as the element for calculating the Q value, the maximum Q value of the next state is selected from the Q table according to the greedy strategy, thereby calculating the Q value of the current state; S5.35: Based on the calculated Q-value of the current state, update the Q-value of the current state and arrange the actions at each time point according to the updated Q-value, ultimately maintaining the optimal action sequence until the entire process ends. The optimal Q-value corresponds to the objective function... SN value, SN The value is used as an impact parameter for solving the benefits of the energy storage system. Finally, the objective function value is calculated, and the final action sequence is generated. The energy storage system executes according to the action sequence.

Images