Light storage power station optimization control method based on reinforcement learning
Through the optimization control method based on reinforcement learning, the Q-Learning algorithm is used to dynamically adjust the charge and discharge of energy storage batteries of the optical storage charging and charging station, which solves the problem of time-consuming, labor-intensive and flexible traditional methods, and achieves lower operating costs and higher flexibility.
Patent Information
- Application Number
- CN202411459275.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-18
- Publication Date
- 2025-02-07
AI Technical Summary
The operation and scheduling strategies of traditional optical storage charging stations rely on complex mathematical structural models, which leads to time-consuming, labor-intensive and poor flexibility, making it difficult to effectively deal with environmental changes.
The optimization control method based on reinforcement learning is adopted, and the Q-Learning algorithm is used to solve iteratively, and the optimal charging and discharge capacity SOC of the energy storage battery of the optical storage charging station is obtained at 24 hourly moments a day, and the control strategy is dynamically adjusted to reduce operating costs.
There is no need to manually define rules or construct data, and adjust the optimal charge and discharge capacity SOC adaptively through interaction with the environment, reducing the operating costs of optical storage charging and charging stations, and improving flexibility and adaptability.
Smart Images
Figure CN119401522A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of power system control, and in particular to a method for optimizing control of a photovoltaic storage and charging station based on reinforcement learning. Background Art
[0002] The popularization of electric vehicles is an effective solution to the global energy problem and environmental crisis. However, at present, my country's power generation energy is still mainly coal. If electric vehicles are charged through ordinary charging stations, most of the electricity consumed still comes from non-clean energy such as coal, which does not have a clear advantage in energy conservation and emission reduction. Using new energy as the main power source for electric vehicle charging stations can achieve low carbon in the true sense.
[0003] The photovoltaic storage and charging integrated charging station is a high-tech green charging mode that realizes the coordinated support of new energy, energy storage and intelligent charging by using photovoltaic power piles. The photovoltaic storage and charging station adopts a modular design concept, including supply and distribution, rectification system, photovoltaic, energy storage, charging, monitoring and other modules. It has the functions of clean energy bidirectional routing, supply and demand side control, and grid energy friendly interaction. It can be used as a typical application scenario for new energy vehicles to use new energy electricity.
[0004] A photovoltaic storage charging station is a new energy charging station that uses photovoltaic power generation to charge electric vehicles. It is usually equipped with energy storage equipment to achieve energy storage and regulation, improve the photovoltaic absorption rate, effectively suppress the impact of fast charging loads on the power system, and play a role in peak shaving and valley filling. At the same time, it obtains economic profits through "low storage and high generation".
[0005] In order to achieve efficient utilization of photovoltaic power generation and reduce the operating costs of charging stations, it is necessary to control and optimize the operation and scheduling strategies of photovoltaic storage charging stations. In the traditional way, the operation and scheduling strategies of photovoltaic storage charging stations are mainly achieved by establishing complex mathematical structure models. Although this method can reduce the operating costs of charging stations to a certain extent, since the mathematical structure model is a fixed rule defined by humans, it is necessary to construct a large amount of data training to determine the specific model fixed parameters, which is time-consuming and labor-intensive. In addition, when the operation and scheduling strategies of photovoltaic storage charging stations are controlled and optimized through mathematical structure models, even in response to environmental changes, the optimization control is carried out in a fixed manner, which has poor flexibility. Summary of the invention
[0006] Based on this, it is necessary to provide an optimization control method for photovoltaic storage and charging stations based on reinforcement learning to address the above technical problems. There is no need to manually define rules or construct data. Through interaction with the environment, the control strategy can be gradually adjusted to maximize the optimization goal. That is, the energy storage action of the photovoltaic storage and charging station can be gradually adjusted to reduce the operating cost of the entire photovoltaic storage and charging station, which can save manpower time and is more flexible than traditional methods.
[0007] The present invention provides a method for optimizing and controlling a photovoltaic charging station based on reinforcement learning. The Q-Learning algorithm is used to iteratively solve and obtain the optimal charge and discharge capacity SOC of the energy storage battery in the photovoltaic charging station at 24 hours a day. The method includes the following steps:
[0008] S1. Get the initial Q-table;
[0009] S2. Determine the energy storage battery status and energy storage battery action at 24 hours a day based on the initial Q-table and the exploration probability threshold, and calculate the total effect function at 24 hours a day for characterizing the total operating cost of the photovoltaic storage charging station;
[0010] S3. Using the Bellman equation, the initial Q-table is updated based on the energy storage battery status and energy storage battery action at 24 hourly moments of the day and the total effect function at 24 hourly moments, wherein the updated Q-table stores multiple data from 1 to 24 moments of the day, and each data is expressed as (energy storage battery status, energy storage battery action, total effect function);
[0011] S4, determine whether the current update iteration number is less than the preset iteration number threshold, if so, use the updated Q-table as the initial Q-table and return to step S1, otherwise, use the updated Q-table as the final Q-table;
[0012] S5. Extract the optimal action corresponding to the maximum value of the total effect function at each moment from the final Q-table to establish the optimal charge and discharge capacity SOC of the energy storage battery.
[0013] In one embodiment, step S2 specifically starts from time 0 and performs the following steps:
[0014] S21, determining the current state of the energy storage battery;
[0015] S22, determining the current charge and discharge limit according to the SOC curve of the energy storage battery, and taking the overlap of the current charge and discharge limit and the total action space of the energy storage battery as the action space of the energy storage battery at the current moment;
[0016] S23, generate a random number in the range of [0, 1), when the random number is less than the exploration probability threshold, randomly select an action in the action space of the energy storage battery at the current moment as the action of the energy storage battery at the current moment, when the random number is greater than the exploration probability threshold, select the action corresponding to the maximum value of the effect function value at the current moment in the current Q-table as the action of the energy storage battery at the current moment;
[0017] S24, determine whether the current time is less than 24, if so, determine the state of the energy storage battery at the next time based on the action space of the energy storage battery at the current time, and return to step S1, otherwise output the energy storage battery state and energy storage battery action at 24 times and calculate the total effect function at 24 hourly times;
[0018] Among them, the calculation formula for the total effect function at 24 hourly moments is:
[0019] C = total electricity purchase cost + battery loss cost + transformer demand electricity fee - photovoltaic grid-connected electricity × photovoltaic grid-connected electricity fee.
[0020] In one of the embodiments, the preset iteration number threshold is 1,000,000.
[0021] In one embodiment, the Bellman equation in step S3 is:
[0022]
[0023] In the formula, Q new (s t ,a t ) indicates the updated t ,a t ) state, Q(s t ,a t ) represents the maximum effect function value in the Q-table at time t, a t is the energy storage battery action at time t, s t represents the state of the energy storage battery at time t, Indicates that in state s t+1 The lower effect function value selects the function value when the maximum action a is used, γ is the conversion coefficient, γ=1, α represents the learning rate, α=1.
[0024] In one embodiment, the exploration probability threshold is
[0025]
[0026] Where β represents the exploration probability threshold, k represents the agent parameter, k = max_epochs × 0.95, max_epochs represents the preset iteration number threshold, and epoch represents the current update iteration number.
[0027] In one embodiment, battery loss cost = battery loss kilowatt-hour cost × absolute value of energy storage battery action × maximum charge and discharge capacity of the energy storage battery × battery loss efficiency;
[0028] The calculation formula for the cost of battery loss per kilowatt-hour is:
[0029]
[0030] In the formula, C b represents the cost per kWh of battery loss, E represents the battery capacity, I represents the cost per kWh, N represents the maximum number of times the battery is used, η represents the efficiency factor of the battery, and μ represents the battery life attenuation coefficient, which are respectively μ Second attenuation μ, N μ It means that the battery life decays N times when the battery life decay coefficient decays μ.
[0031] The beneficial effects of the present invention are as follows: the present invention utilizes the advantages of the self-supervised machine learning method of the Q-Learning algorithm without the need for specific environmental model parameters and acquiring knowledge through interaction with the environment. In the iterative solution process, the solution space of the operating state of the energy storage equipment of the photovoltaic storage and charging system can be continuously explored through the objective function Bellman equation and the constraint condition maximum effect function, and the relatively optimal optimal action of the energy storage battery at 24 hours a day is found to form the optimal charge and discharge amount SOC of the energy storage battery. The optimal charge and discharge amount SOC can be adaptively adjusted according to the environmental changes of the photovoltaic storage and charging station including electric load, time-of-use electricity price, etc. to reduce the operating cost of the entire photovoltaic storage and charging station. There is no need to manually define rules and construct data, which improves the flexibility and adaptability of the operation of the photovoltaic storage and charging system. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 This is one of the flow charts of the optimization control method of the photovoltaic storage and charging station based on reinforcement learning provided by an embodiment of the present invention;
[0033] Figure 2 It is a flow chart of an optimization control method of a photovoltaic charging station based on reinforcement learning in another embodiment of the present invention. DETAILED DESCRIPTION
[0034] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0035] In one embodiment, Figure 1 As shown, Figure 1 It is a flow chart of the optimization control method of the photovoltaic storage and charging station based on reinforcement learning provided by an embodiment of the present invention. The optimization control method of the photovoltaic storage and charging station based on reinforcement learning of this embodiment uses the Q-Learning algorithm to iteratively solve and obtain the optimal charge and discharge capacity SOC of the energy storage battery in the photovoltaic storage and charging station at 24 hours a day.
[0036] Reinforcement learning (Q-Learning) algorithm is a field in machine learning that emphasizes how to act based on the environment to maximize expected benefits. The environment is usually abstracted as a Markov decision process (MDP), which is a four-tuple consisting of a state set S, an action set A, a state transition probability P, and a reward function R. The agent selects an action based on its current state, transfers to the next state after executing the action, and obtains a reward related to the state and action. The learning goal is to learn a strategy by interacting with the environment to maximize the cumulative reward.
[0037] In this embodiment, in the Q-Learning algorithm, the environment refers to the entire photovoltaic storage and charging station, specifically refers to parameters such as the power load and time-of-use electricity price of the entire photovoltaic storage and charging station, the state is the SOC of the energy storage battery at each moment, and the action is the SOC of the charge and discharge amount of the energy storage battery.
[0038] This method is applied to computer equipment, such as Figure 1 As shown, the method comprises the following steps:
[0039] S1. Get the initial Q-table. The initial Q-table is empty before updating iteration.
[0040] S2. Based on the initial Q-table and the exploration probability threshold, determine the energy storage battery status and energy storage battery action at 24 hours a day and calculate the total effect function at 24 hours a day to characterize the total operating cost of the photovoltaic storage charging station.
[0041] S3. Use the Bellman equation to update the initial Q-table based on the energy storage battery status and energy storage battery action at 24 hourly moments of the day and the total effect function at 24 hourly moments, wherein the updated Q-table stores multiple data from 1 to 24 moments of the day, and each data is expressed as (energy storage battery status, energy storage battery action, total effect function).
[0042] Specifically, the Bellman equation in step S3 is:
[0043]
[0044] In the formula, Q new (s t ,a t ) indicates the updated t ,a t ) state, Q(s t ,a t ) represents the maximum effect function value in the Q-table at time t, a t is the energy storage battery action at time t, s t represents the state of the energy storage battery at time t, Indicates that in state s t+1 The lower effect function value selects the function value when the maximum action a is used, γ is the conversion coefficient, γ=1, α represents the learning rate, α=1.
[0045] S4. Determine whether the current update iteration number is less than the preset iteration number threshold. If so, use the updated Q-table as the initial Q-table and return to step S1. Otherwise, use the updated Q-table as the final Q-table.
[0046] Specifically, the preset iteration number threshold is 1000000.
[0047] S5. Extract the optimal action corresponding to the maximum value of the total effect function at each moment from the final Q-table to establish the optimal charge and discharge capacity SOC of the energy storage battery.
[0048] In this embodiment, the Q-Learning algorithm can be used to continuously explore the solution space of the energy storage device operating state of the photovoltaic storage and charging system through the objective function Bellman equation and the constraint condition maximum effect function in the iterative solution process, and find the optimal action of the energy storage battery at 24 hours a day to form the optimal charge and discharge amount SOC of the energy storage battery. This process does not require specific environmental model parameters, and obtains decision results by interacting with the environment to reduce the operating cost of the entire photovoltaic storage and charging station. There is no need to manually define rules and construct data, which improves the flexibility and adaptability of the operation of the photovoltaic storage and charging system.
[0049] In one embodiment, if Figure 2 As shown, step S2 specifically starts from time 0 and performs the following steps:
[0050] S21. Determine the current state of the energy storage battery.
[0051] For example, at time 0, the energy storage battery charge and discharge SOC is 0, and the energy storage battery state at time 0 is state = (0, 0).
[0052] S22, determining the current charge and discharge limit according to the SOC curve of the energy storage battery, and taking the overlap of the current charge and discharge limit and the total action space of the energy storage battery as the action space of the energy storage battery at the current moment.
[0053] Exemplarily, the total action space of the energy storage battery is [-1, -0.98, -0.96, 0.96, 0.98, 1]. The charge and discharge SOC of the energy storage battery at time 0 is 0, and the charging limit is (-0.48, 0). Then, the action space of the energy storage battery at the current time is [-0.48, -0.46, -0.44, -0.04-0.02, 0].
[0054] S23. Generate a random number in the range of [0, 1). When the random number is less than the exploration probability threshold, randomly select an action in the action space of the energy storage battery at the current moment as the action of the energy storage battery at the current moment. When the random number is greater than the exploration probability threshold, select the action corresponding to the maximum value of the effect function value at the current moment in the current Q-table as the action of the energy storage battery at the current moment.
[0055] In this embodiment, the probability threshold and the generated random number are explored to determine whether the current iteration selects the energy storage battery action by selecting from the current Q-table or completely randomly selecting, which can prevent the iterative decision process from falling into a local optimal solution.
[0056] Specifically, the exploration probability threshold is
[0057]
[0058] Where β represents the exploration probability threshold, k represents the agent parameter, k = max_epochs × 0.95, max_epochs represents the preset iteration number threshold, and epoch represents the current update iteration number.
[0059] As the number of iterations epoch increases, β gradually decreases from 1 to 0.03, and the entire decision-making process tends to look up the table more and more, making the final action of the energy storage battery at the current moment more and more accurate.
[0060] For example, epoch=1, then β=1, and when the random number is less than 1, an action is randomly generated in the action space that satisfies the constraints, for example, action=-0.2 is obtained.
[0061] When the random number is greater than 1, such as the state is (0.1, 16), find the action corresponding to the maximum effect function value in the current Q-table. If the maximum effect function value is -3947.8174403601556, the corresponding action is -0.22, then the energy storage battery action at the current moment is action = -0.22.
[0062] S24, determine whether the current time is less than 24, if so, determine the energy storage battery state at the next time based on the action space of the energy storage battery at the current time, and return to step S1, otherwise output the energy storage battery state and energy storage battery action at 24 times and calculate the total effect function of 24 hours. The total effect function of 24 hours is equivalent to the operating cost of the entire charging station for one day.
[0063] Among them, the calculation formula for the total effect function at 24 hourly moments is:
[0064] C = total electricity purchase cost + battery loss cost + transformer demand electricity fee - photovoltaic grid-connected electricity × photovoltaic grid-connected electricity fee.
[0065] In this embodiment, the electricity purchase cost at 24 hourly moments is the sum of the energy storage battery operation for one hour at each hourly moment, and the electricity purchase cost at each hourly moment is the product of the battery SOC at the current moment and the power grid electricity price at the current moment.
[0066] Specifically, battery loss cost = battery loss cost per kilowatt-hour × absolute value of energy storage battery action × maximum charge and discharge capacity of energy storage battery × battery loss efficiency;
[0067] The calculation formula for the cost of battery loss per kilowatt-hour is:
[0068]
[0069] In the formula, C b represents the cost per kWh of battery loss, E represents the battery capacity, I represents the cost per kWh, N represents the maximum number of times the battery is used, η represents the efficiency factor of the battery, and μ represents the battery life attenuation coefficient, which are respectively μ Second attenuation μ, N μ It means that the battery life decays N times when the battery life decay coefficient decays μ.
[0070] For example, the energy storage battery capacity is 233kw / h, the energy storage battery cost per kilowatt-hour is 800 yuan, the maximum battery life is 6000 times, the battery efficiency factor is 0.9, the battery life attenuation coefficient is 0.02, and the battery life attenuation coefficient decays by 0.02 when the battery life attenuation coefficient decays by 600 times.
[0071] Battery consumption cost per kilowatt-hour:
[0072]
[0073] In another specific embodiment, the energy storage battery status and energy storage battery action at 24 hours a day finally determined in step S2 and the total effect function at 24 hours a day for characterizing the total operating cost of the photovoltaic storage charging station are calculated as follows:
[0074] Energy storage battery status states:[(0,0),(0.2,1),(0.54,2),(0.9,3),(0.92,4),(0.92,5),(0.98,6),(0.98,7),(1.0,8),(0.58,9),(0.94,10),(0.5,11),(0.26,12),(0.24,13),(0.02,14),(0.4,15),(0.64,16),(0.32,17),(0.1,18),(0.1,19),(0.02,20),(0.46,21),(0.28,22),(0.12,23)];
[0075] Energy storage battery actions actions:[-0.2,-0.34,-0.36,-0.02,0.0,-0.06,0.0,-0.02,0.42,-0.36,0.44,0.24,0.02,0.22,-0.38,-0.24,0.32,0.22,0.0,0.08,-0.44,0.18,0.16,-0.44];
[0076] Total effect function: -3951.2433750805735;
[0077] Then, in step S3, the initial Q-table is updated by using the Bellman equation based on the energy storage battery status and energy storage battery action at 24 hours a day and the total effect function at 24 hours a day to obtain the updated Q-table:
[0078] [{((0,0),-0.2):-3951.2433750805735},{((0.2,1),-0.34):-3951.2433750805735},{((0.54,2),-0.36):-3951.2433750805735},{((0.9,3),-0.02):-3951.2433750805735},{((0.92,4),0.0):-3951.2433750805735},{((0.92,5),-0.06):-3951.2433750805735},{((0.98,6),0.0):-3951.2433750805735},{((0.98,7),-0.02):-3951.2433750805735},{((1.0,8),0.42):-3951.2433750805735},{((0.58,9),-0.36):-3951.2433750805735},{((0.94,10),0.44):-3951.2433750805735},{((0.5,11),0.24):-3951.2433750805735},{((0.26,12),0.02):-3951.2433750805735},{((0.24,13),0.22):-3951.2433750805735},{((0.02,14),-0.38):-3951.2433750805735},{((0.4,15),-0.24):-3951.2433750805735},{((0.64,16),0.32):-3951.2433750805735},{((0.32,17),0.22):-3951.2433750805735},{((0.1,18),0.0):-3951.2433750805735},{((0.1,19),0.08):-3951.2433750805735},{((0.02,20),-0.44):-3951.2433750805735},{((0.46,21),0.18):-3951.2433750805735},{((0.28,22),0.16):-3951.2433750805735},{((0.12,23),-0.44):-3951.2433750805735}]。
[0079] In step S4, it is determined that the current update iteration number is less than the preset iteration number threshold, the updated Q-table is used as the initial Q-table and the process returns to step S1.
[0080] After multiple iterative updates, the final Q-table stores multiple data from 1 to 24 moments of the day. Each moment has multiple data with different effect functions. Then, the optimal action corresponding to the maximum value of the total effect function at each moment is extracted from the final Q-table to establish the optimal charge and discharge SOC of the energy storage battery.
[0081] The reinforcement learning-based optimization control method for the photovoltaic charging station of this embodiment updates the Q-table by interacting with the photovoltaic charging station environment, and finally outputs the optimal action at each moment when the total operating cost of the photovoltaic charging station is the lowest under the constraints of the number of iterations and the total effect function at 24 hourly moments to achieve regulation of the energy storage action of the photovoltaic charging station to reduce the operating cost of the entire photovoltaic charging station. It does not require manual definition of rules or construction of data and is more flexible.
[0082] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.
Claims
1. A method for optimizing and controlling a photovoltaic charging station based on reinforcement learning, characterized in that: The Q-Learning algorithm is used to iteratively solve and obtain the optimal charge and discharge capacity SOC of the energy storage battery in the photovoltaic charging station at 24 hours a day. The method includes the following steps: S1. Get the initial Q-table; S2. Determine the energy storage battery status and energy storage battery action at 24 hours a day based on the initial Q-table and the exploration probability threshold, and calculate the total effect function at 24 hours a day for characterizing the total operating cost of the photovoltaic storage charging station; S3. Using the Bellman equation, the initial Q-table is updated based on the energy storage battery status and energy storage battery action at 24 hourly moments of the day and the total effect function at 24 hourly moments, wherein the updated Q-table stores multiple data from 1 to 24 moments of the day, and each data is expressed as (energy storage battery status, energy storage battery action, total effect function); S4, determine whether the current update iteration number is less than the preset iteration number threshold, if so, use the updated Q-table as the initial Q-table and return to step S1, otherwise, use the updated Q-table as the final Q-table; S5. Extract the optimal action corresponding to the maximum value of the total effect function at each moment from the final Q-table to establish the optimal charge and discharge capacity SOC of the energy storage battery.
2. The optimization control method of a photovoltaic charging station based on reinforcement learning according to claim 1 is characterized in that: The step S2 specifically starts from time 0 and performs the following steps: S21, determining the current state of the energy storage battery; S22, determining the current charge and discharge limit according to the SOC curve of the energy storage battery, and taking the overlap of the current charge and discharge limit and the total action space of the energy storage battery as the action space of the energy storage battery at the current moment; S23, generate a random number in the range of [0, 1), when the random number is less than the exploration probability threshold, randomly select an action in the action space of the energy storage battery at the current moment as the action of the energy storage battery at the current moment, when the random number is greater than the exploration probability threshold, select the action corresponding to the maximum value of the effect function value at the current moment in the current Q-table as the action of the energy storage battery at the current moment; S24, determine whether the current time is less than 24, if so, determine the state of the energy storage battery at the next time based on the action space of the energy storage battery at the current time, and return to step S1, otherwise output the energy storage battery state and energy storage battery action at 24 times and calculate the total effect function at 24 hourly times; Among them, the calculation formula for the total effect function at 24 hourly moments is C = total electricity purchase cost + battery loss cost + transformer demand electricity fee - photovoltaic grid-connected electricity × photovoltaic grid-connected electricity fee.
3. The optimization control method of a photovoltaic charging station based on reinforcement learning according to claim 1 is characterized in that: The preset iteration number threshold is 1000000.
4. The optimization control method of a photovoltaic charging station based on reinforcement learning according to claim 1 is characterized in that: The Bellman equation in step S3 is In the formula, Q new (s t ,a t ) indicates the updated t ,a t ) state, Q(s t ,a t ) represents the maximum effect function value in the Q-table at time t, a t is the energy storage battery action at time t, s t represents the state of the energy storage battery at time t, Indicates that in state s t+1 The lower effect function value selects the function value when the maximum action a is used, γ is the conversion coefficient, γ=1, α represents the learning rate, α=1.
5. The optimization control method of a light storage and charging station based on reinforcement learning according to claim 2 is characterized in that: The exploration probability threshold is Where β represents the exploration probability threshold, k represents the agent parameter, k = max_epochs × 0.95, max_epochs represents the preset iteration number threshold, and epoch represents the current update iteration number.
6. The optimization control method of a light storage and charging station based on reinforcement learning according to claim 2 is characterized in that: The battery loss cost = battery loss cost per kilowatt-hour × absolute value of energy storage battery action × maximum charge and discharge capacity of the energy storage battery × battery loss efficiency; The calculation formula of the battery loss per kilowatt-hour is: In the formula, C b represents the cost per kWh of battery loss, E represents the battery capacity, I represents the cost per kWh, N represents the maximum number of times the battery is used, η represents the efficiency factor of the battery, and μ represents the battery life attenuation coefficient, which are respectively μ Second attenuation μ, N μ It means that the battery life decays N times when the battery life decay coefficient decays μ.