Industrial and commercial park shared storage and charging system intelligent scheduling method based on reinforcement learning

Abstract

The invention relates to an industrial and commercial park shared storage and charging system intelligent scheduling method based on reinforcement learning, and the method comprises the steps: carrying out the modeling of various loads in an industrial and commercial park, building a model of an optical storage and charging system, analyzing the charging and discharging characteristics of the optical storage and charging system, and calculating the degradation cost of a battery. By designing a reasonable reward function, the reinforcement learning algorithm is guided to make optimal charging and discharging decisions at different moments so as to maximize the overall benefits of the industrial and commercial park. And the reinforcement learning agent continuously selects actions according to the current state, executes a corresponding charging and discharging strategy, and optimizes the strategy by feeding back a reward signal. Through multiple iterations, the optimal scheduling mode is found step by step, the balance between power supply and demand is ensured, energy waste is reduced, and the service life of a battery is prolonged. According to the invention, the use efficiency of energy in industrial and commercial parks is improved, the energy cost can be greatly reduced, the double improvement of economy and sustainability is realized, and the safety and stability of a 750KV and above power transmission system can be improved through seamless connection with a power grid through dynamic load adjustment.

Description

Technical Field

[0001] The present invention belongs to the field of smart grid and energy management, and specifically relates to an intelligent scheduling method for a shared storage and charging system in an industrial and commercial park based on reinforcement learning. The method aims to reduce electricity costs, improve energy utilization efficiency, effectively alleviate large-scale power grid load pressure, support the safe operation of large power grids, and extend the service life of energy storage batteries by optimizing the charging and discharging strategies of energy storage systems. Background Art

[0002] With the transformation of the global energy structure and the widespread application of new energy, energy management in industrial and commercial parks has gradually become a hot topic of research. In particular, with the popularization of electric vehicles and the introduction of renewable energy such as photovoltaics, the complexity of energy demand and supply in industrial and commercial parks has continued to increase. The existing traditional power grid system is difficult to meet such diversified needs, resulting in a high dependence of industrial and commercial parks on the power grid, and it is difficult to achieve long-term stable operation due to the degradation of battery energy storage systems. Therefore, how to optimize the charging and discharging strategy of the energy storage system on the basis of ensuring the load of industrial and commercial parks has become a key issue in improving the economy and sustainability of the energy system. At present, although some studies have attempted to reduce electricity costs and battery losses by optimizing the scheduling of battery energy storage systems, due to the diversity and uncertainty of energy management, traditional scheduling methods are often difficult to achieve ideal results in actual scenarios. In addition, due to the large load fluctuations in industrial and commercial parks, how to reasonably schedule energy storage equipment to meet electricity demand and extend the life of the energy storage system has become a key issue that needs to be solved in this field. Traditional scheduling methods often find it difficult to balance battery life, electricity costs and overall system performance. Therefore, an intelligent scheduling method based on reinforcement learning is needed to achieve efficient optimization of power scheduling and resource sharing. Summary of the invention

[0003] The purpose of the present invention is to provide an intelligent scheduling method for a shared storage and charging system in an industrial and commercial park based on reinforcement learning. The present invention reduces electricity costs and reduces degradation losses of energy storage systems by optimizing the charging and discharging scheduling of photovoltaic storage and charging systems in industrial and commercial parks, thereby improving the economy and resource utilization efficiency of the overall system.

[0004] In the present invention, a load model of various types of electrical equipment in the industrial and commercial park is first established, taking into account the energy consumption characteristics and load requirements of different equipment. At the same time, a mathematical model of the charging and discharging characteristics, capacity and degradation cost of the shared energy storage system in the industrial and commercial park is established. By using the reinforcement learning algorithm, the charging and discharging decisions of the photovoltaic storage system are dispatched, so that the system can reduce battery degradation loss and electricity costs while meeting the load requirements.

[0005] The present invention proposes an intelligent scheduling method for a shared storage and charging system in an industrial and commercial park based on reinforcement learning, and the specific steps are as follows:

[0006] (1) By modeling various loads in the industrial and commercial park that includes shared solar-storage-charging, the energy balance of the shared solar-storage-charging system is controlled to achieve intelligent scheduling and optimal management of energy, focusing on the electricity demand of park users, the charging demand of electric vehicles, and the supply of photovoltaic power generation;

[0007] (2) Establish a model of the shared solar-storage-charging system, analyze its charging and discharging characteristics, and calculate the degradation cost of the battery;

[0008] (3) By designing a reasonable reward function, the reinforcement learning algorithm is guided to make the best charging and discharging decisions at different times to maximize the overall benefits of the industrial and commercial park. In this process, the reinforcement learning agent continuously selects actions based on the current state, executes the corresponding charging and discharging strategy, and optimizes the strategy through feedback reward signals. Through multiple iterations, the system gradually finds the optimal scheduling method to ensure the balance between power supply and demand, reduce energy waste, and extend battery life.

[0009] In the present invention, the modeling in step (1) is to establish the interactive relationship between energy storage equipment, electric vehicles and other loads. Specifically, the energy storage equipment can be used as an emergency power supply in the modeling, and can also be charged during peak electricity price periods to reduce dependence on the distribution network. The loads of various industrial and commercial parks include:

[0010] Energy storage equipment is used to store electrical energy that cannot be consumed immediately in industrial and commercial parks. The mathematical model of energy storage equipment is shown in formula (1):

[0011]

[0012] in: represents the state of the energy storage device at time t, P d，t is the charge and discharge power, ΔT is the time step;

[0013] The capacity range of energy storage equipment is subject to the following constraints:

[0014]

[0015] Where: p dod is the lower limit of the device discharge power, is the maximum storage capacity of the device, e d，t is the charge and discharge state of the storage device at time t, is the upper limit of charging power. and are the states of the device at t=19 and t=7 respectively, p st is the upper limit coefficient of charging power, p edis the charging state of the electric vehicle; Formula (2)-Formula (6) are the charging and discharging states and capacity limits of the energy storage device in different time periods;

[0016] Electric vehicles are an indispensable load in industrial and commercial parks. Depending on the charging mode, the charging schedule of electric vehicles will vary. Electric vehicles adopt a home charging mode, that is, the owner goes home to charge after get off work and stops charging when going out to work; the start time of electric vehicle charging follows a normal distribution, and its probability density function is:

[0017]

[0018] Among them: parameter σ s and μ s They represent the standard deviation and mean of the normal distribution that controls the start time of electric vehicle charging, σ s =3.3,μ s =18; According to formula (7), the charging schedule of the electric vehicle is obtained, and then the owner is charged according to the corresponding charging mode;

[0019] The capacity of electric vehicles in industrial and commercial parks follows a uniform distribution, as shown in formula (8); where 20-60 kWh represents the common capacity range of electric vehicles on the market; the probability that electric vehicles in industrial and commercial parks have a capacity within this range is uniformly distributed:

[0020]

[0021] The charging formula of electric vehicles is similar to that of energy storage devices, following the charging process of lithium-ion batteries; the charging process is shown in formula (9) and formula (10):

[0022]

[0023] SOC min ≤SOC i，t ≤SOC max (10)

[0024] in: is the charging power of the electric vehicle, is the battery capacity of the electric vehicle, determined by the uniform distribution equation for determining the capacity of electric vehicles in industrial and commercial parks; SOC min Ensuring that electric vehicles are charged to the minimum SOC required for operation is mainly to ensure that the battery of the electric vehicle will not be deeply discharged during the discharge process, thereby extending the life of the electric vehicle battery; and SOC max This corresponds to the maximum SOC that the electric vehicle can achieve, avoiding overcharging of the electric vehicle;

[0025] There are two charging modes for electric vehicles: fast charging mode and smart charging mode. The fast charging mode is suitable for emergency charging needs and can fully charge in a shorter time, but the charging power is higher, which may affect the battery life. The smart charging mode is suitable for users who are not in a hurry to charge. Its charging power is lower and can be dynamically adjusted according to the overall load of the industrial and commercial park and the fluctuation of electricity prices to reduce charging costs and avoid excessive load on the power grid.

[0026] The charging power is limited as follows:

[0027]

[0028] in: It is the charging power of electric vehicles. In fast charging mode, the charging power is equal to the maximum allowable power of the charging pile in the industrial and commercial park, which is 20kWh. In smart charging mode, the charging power is between 0 and the maximum power, and changes with the overall load curve of the industrial and commercial park and the fluctuation of real-time electricity prices, thereby achieving two purposes. One is that the charging of electric vehicles in the industrial and commercial park will not be too high during peak hours, thereby putting pressure on the distribution network of the industrial and commercial park. The second is to avoid the peak period of electricity prices to meet the charging needs, thereby reducing the charging costs of car owners.

[0029] In the present invention, the industrial and commercial park described in step (2) is equipped with a shared photovoltaic storage and charging system, which needs to give priority to meeting the internal load of the industrial and commercial park, especially during the peak load period; so as to effectively reduce the dependence of the industrial and commercial park load on the distribution network, because during the non-peak period, the load pressure of the distribution network is relatively small; therefore, various loads in the industrial and commercial park need to meet the power balance constraint;

[0030] The energy balance constraint formula is as follows:

[0031]

[0032] Where: L t is the total power demand of the industrial and commercial park, which consists of two parts. The first part is That is, the basic load of the park, including daily electricity consumption, such as electricity demand for infrastructure such as lights and air conditioners; the second part is That is, the electric vehicle charging load in the industrial and commercial park. The size of this part is determined by the penetration rate of electric vehicles in the industrial and commercial park. That is, the charging power of an electric vehicle is The total fleet power of the industrial and commercial park basically depends on how many electric vehicles are equipped in the park; the charging load of electric vehicles is not included in the basic load and requires independent optimization and scheduling.

[0033] In the present invention, the reward function in the reinforcement learning described in step (3) plays a core role in guiding the agent's decision-making, determines how the agent evaluates its behavior, and guides the agent to approach the target behavior; the charging and discharging decisions of the industrial and commercial park energy storage system are optimized through reinforcement learning, especially during the peak load period, the energy storage system should bear part of the load pressure;

[0034] In order to constrain the agent to behave in a way that does not meet the goal, a penalty function is used to reduce such behavior; the penalty function is defined as:

[0035]

[0036] in: It represents the energy waste generated at this time step, specifically the surplus electric energy that cannot be fully utilized by photovoltaic power generation, or the energy discharged by the energy storage system that exceeds the demand of the industrial and commercial park;

[0037] The energy output of the photovoltaic power generation system needs to meet the power demand of the industrial and commercial park and minimize waste; the energy balance formula (14) of the photovoltaic system is as follows:

[0038]

[0039] Where: G t It is the electricity generated by the photovoltaic system at each moment. Part of this electricity is used by users in the industrial and commercial park, marked as A portion is stored by energy storage, marked as If there is any remaining photovoltaic power, it is marked as This indicates that this part of the electricity is wasted;

[0040] The power demand of industrial and commercial park users also needs to meet the power balance constraints to ensure the reasonable distribution of industrial and commercial park loads among various energy inputs; the energy balance formula of users is as follows:

[0041]

[0042] Where: L t is the total electricity demand of the industrial and commercial park, part of which comes from photovoltaic power generation, i.e. Part of it comes from the discharge of the energy storage system, that is Another part depends on the power grid, i.e. If the energy storage system discharge far exceeds the industrial and commercial park load demand, it is marked as This indicates that this part of energy is wasted;

[0043] The charging power source of the energy storage system includes the purchase of electricity from the power grid and the power generation of the photovoltaic system; the charging power formula of the energy storage system is as follows:

[0044]

[0045] in: is the charging power from the grid, The charging power comes from the photovoltaic power generation system;

[0046]

[0047] Energy waste generated by the entire industrial and commercial park The source and photovoltaic power generation have not been fully utilized, resulting in photovoltaic power waste And the energy storage discharge far exceeds the demand of industrial and commercial parks, resulting in waste of energy storage electricity This part of electricity waste not only affects the power supply efficiency of the industrial and commercial park, but also increases the energy consumption burden of the system. Therefore, punishment is needed in reinforcement learning to reduce unnecessary wasteful behavior.

[0048] (14) The reward function ensures that the shared solar-storage-charging system can bear the load of the industrial and commercial park through the power balance of the industrial and commercial park;

[0049] The electricity cost of the industrial and commercial park needs to take into account the cost of purchasing electricity from the power grid and the energy storage system, which is calculated as shown in formula (18):

[0050]

[0051] in: is the electricity cost of the entire industrial and commercial park, It is the real electricity price, which is based on the local electricity price standard. It is the electricity purchased by the industrial and commercial park from the power grid. It is the electricity purchased by the energy storage system from the grid. The electricity cost is closely related to the load fluctuation of the industrial and commercial park and the electricity price of the grid. Therefore, the use of optimized electricity purchase methods can significantly reduce the electricity cost of the industrial and commercial park.

[0052] The charging behavior of the energy storage system not only involves the storage of electrical energy, but also brings cost losses, especially the wear and tear of the battery. The cost loss caused by the charging behavior of the energy storage system is calculated as shown in formula (19):

[0053]

[0054] Where: C E is the unit energy cost of the battery, which can be the initial cost of the battery or the replacement cost; D is the depth of discharge of the battery, indicating the degree of discharge; L CThe cycle life of the battery refers to the number of charge and discharge cycles that the battery can experience under specific discharge depth and conditions. The wear cost of the battery is related to energy cost, efficiency, discharge depth, cycle life, and charge and discharge power. By properly managing the charge and discharge strategy, the battery life can be effectively extended and the wear cost can be reduced.

[0055] The reward function of the industrial and commercial park is shown in formula (20):

[0056]

[0057] in: is the reward of the industrial and commercial park at time t; It is the electricity cost of the industrial and commercial park. The goal is to reduce the electricity cost and thus reduce the industrial and commercial park's dependence on the power grid; It is the wear and tear cost of the energy storage system, which aims to take into account the long-term health of the energy storage system and optimize the use of the battery; It is a penalty function, which aims to reduce energy waste and ensure efficient operation of the system;

[0058] Through the reward function, the shared solar-storage-charging system can self-adjust the charging and discharging strategy during the reinforcement learning process, which not only optimizes the economic cost, but also extends the service life of the energy storage system and reduces energy waste, thereby achieving effective management of the load of industrial and commercial parks and reducing dependence on the distribution network.

[0059] In the present invention, the reinforcement learning algorithm described in step (3) is used to control the charging and discharging decisions of the energy storage system of the industrial and commercial park, so that the system forms an optimal usage strategy according to different time conditions, thereby maximizing the long-term rewards of the industrial and commercial park;

[0060] The core task of reinforcement learning is to enable the agent (in this scenario, the dispatch controller of the industrial and commercial park energy storage system) to learn to maximize its long-term rewards through a series of actions in the environment; in this application scenario, the agent's "action" refers to the decision on the charging and discharging degree of the energy storage system, and the goal is to maximize the reward function of the industrial and commercial park through these decisions, ensuring that while meeting the load demand of the industrial and commercial park, the electricity cost is reduced, the life of the energy storage system is extended, and energy waste is reduced;

[0061] In the reinforcement learning framework, the state of the environment is a random variable at each step; in this case, the state of the environment is determined by multiple factors such as the current charging state of the energy storage system, load demand, and photovoltaic power generation; assuming that at a certain moment, the agent is in state s t , take action t , and then transfer to the next state s t+1 And get rewards The state transfer process can be expressed as:

[0062] s t+1 =f(s t , a t ) (twenty one)

[0063] Among them: f is the state transfer function, which indicates the change of the environment state after taking a certain action in a certain state;

[0064] The goal of the agent is to maximize its cumulative reward by choosing appropriate action sequences; this is usually achieved through a value function; given the current time t, the cumulative reward G t It is the weighted sum of all rewards from the current moment to the end moment:

[0065]

[0066] Where: γ is the discount factor, usually between 0≤γ≤1; the discount factor controls the weight of future rewards. The smaller the value, the less impact future rewards have on current decisions; and the reward function r in the formula t+1 In fact, A simplified representation of , that is, evaluating the impact of the current decision on the agent through the reward function;

[0067] The policy π defines the probability distribution of the agent's actions in each state, which can be deterministic or stochastic;

[0068] π(a|s)=P(a t =a|s t =s)(23)

[0069] The random strategy is shown in formula (23), which represents the probability of taking action a in state s. In practical applications, random strategies can help explore different decision paths to find the optimal strategy.

[0070] The reinforcement learning method adopts the Q-learning method, which is based on value iteration and approaches the optimal strategy by continuously updating the action value function. In the Q-learning method, each pair of states s t and action a t There is a corresponding Q value, which represents the expected return of taking the action in that state; the update rule of the Q-learning method is as follows:

[0071] Q(s t , a t )←Q(s t , a t )+α(r t+1 +γmax a′ Q(s t+1 , a′)-Q(s t , at )) (twenty four)

[0072] Among them: α is the learning rate, which controls the step size of each update; max a′ Q(s t+1 , a′) is in the new state s t+1 Next, take the Q value corresponding to the best action among all possible actions a′;

[0073] By continuously iteratively updating the Q value, the Q-learning method can gradually find the best strategy, that is, taking the best action in each state to maximize the cumulative return.

[0074] The beneficial effects of the present invention are as follows: the present invention maximizes the utilization of renewable energy and reduces dependence on traditional power grids by optimizing the charging and discharging strategies of the energy storage system. By optimizing the reinforcement learning algorithm, the cost of purchasing electricity from the power grid is minimized, and the energy storage system is reasonably scheduled to reduce the overall electricity cost. While optimizing the electricity cost, the battery degradation problem is considered to ensure the healthy use of the energy storage battery. Through the penalty mechanism, the waste of photovoltaic power and energy storage system power is reduced, further improving the overall economy of the system. This framework can be promoted and applied to industrial and commercial parks of different sizes and types, has strong adaptability, and is particularly suitable for implementation in smart grid and multi-microgrid environments.

[0075] The present invention provides an innovative intelligent scheduling method for shared storage and charging systems in industrial and commercial parks based on reinforcement learning, which can effectively solve various challenges in power scheduling under a multi-microgrid shared architecture, optimize the operation of energy systems in industrial and commercial parks, improve energy utilization efficiency and economy, and enhance the safety and stability of the power grid. It has broad application prospects and promotion value. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0077] Figure 1 This is a flowchart of the intelligent scheduling method for a shared storage and charging system in an industrial and commercial park based on reinforcement learning involved in the present invention.

[0078] Figure 2 This is a framework diagram of the energy balance of an industrial and commercial park in the intelligent scheduling method of a shared storage and charging system for an industrial and commercial park based on reinforcement learning involved in the present invention. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0080] Example 1: In the intelligent scheduling method of shared storage and charging system in industrial and commercial parks based on reinforcement learning, different types of load equipment are equipped in the industrial and commercial park, including basic power loads and electric vehicles. The industrial and commercial park is also equipped with a shared photovoltaic storage and charging system, including photovoltaic panels and centralized energy storage. The collaborative work between these devices can provide sufficient energy supply. Through the shared photovoltaic storage and charging system, the load demand and energy supply in different areas are optimized and scheduled.

[0081] The main goal of this invention is to achieve the optimal scheduling of shared energy storage systems in industrial and commercial parks to minimize electricity costs and degradation costs of energy storage systems and improve the comprehensive utilization efficiency of energy. Through reinforcement learning algorithms, based on actual load demand and renewable energy supply, the charging and discharging decisions of the photovoltaic storage system are adjusted to optimize the performance and economy of the overall system.

[0082] In the present invention, a reinforcement learning algorithm is used to dynamically adjust the charging and discharging strategy of the energy storage system. First, a reward function is designed to take into account the power demand of various types of equipment in the industrial and commercial park, the energy cost, and the degradation loss of the energy storage system. By setting the objective function, the algorithm aims to minimize the weighted sum of the power cost and the battery degradation cost. To achieve this goal, the system dynamically adjusts the charge and discharge of the energy storage system through real-time monitoring of the load, energy production, and energy storage system status.

[0083] This method uses the Q-learning strategy in the reinforcement learning algorithm to improve the accuracy and efficiency of charging and discharging decisions by continuously optimizing the strategy. Specifically, at each time step, the state of the industrial and commercial park energy storage system (such as remaining battery power, load demand, energy supply, etc.) is used as input. The reinforcement learning agent selects the best charging and discharging operation based on the current state and evaluates the effect of the current decision through the reward function. As the training progresses, the agent gradually learns the optimal charging and discharging strategy in various different scenarios.

[0084] Figure 1 This is a flowchart of the intelligent scheduling method for a shared storage and charging system in an industrial and commercial park based on reinforcement learning involved in the present invention.

[0085] Figure 2This is a framework diagram of the energy balance of the industrial and commercial park in the intelligent scheduling method of the shared storage and charging system of the industrial and commercial park based on reinforcement learning involved in the present invention. It can be seen that the industrial and commercial park is equipped with user basic power load, fleet charging demand and industrial and commercial park shared photovoltaic storage and charging system. The method includes:

[0086] Modeling of various loads in industrial and commercial parks that include shared solar storage and charging, especially the interaction between energy storage equipment, electric vehicles and other loads. Specifically, energy storage equipment can be used as an emergency power supply in this system, and can also be charged during peak electricity price periods to reduce dependence on the distribution network. Various industrial and commercial park loads include: Energy storage equipment is used to store electrical energy that cannot be consumed immediately in industrial and commercial parks. The mathematical model of energy storage is as follows:

[0087]

[0088] in: represents the state of the energy storage device at time t, P d，t is the charge and discharge power, and ΔT is the time step.

[0089] The capacity range of energy storage equipment is subject to the following constraints:

[0090]

[0091]

[0092] Where: p dod is the lower limit of the device discharge power, is the maximum storage capacity of the device, e d，t is the charge and discharge state of the storage device at time t, is the upper limit of charging power. and are the states of the device at t=19 and t=7 respectively, p st is the upper limit coefficient of charging power, p ed is the charging state of the electric vehicle; the above formula describes the charging and discharging state and capacity limit of the energy storage device in different time periods. Electric vehicles have become an indispensable load in industrial and commercial parks. Depending on the charging mode, the charging schedule of electric vehicles will vary. Generally, electric vehicles adopt a home charging mode, that is, the owner goes home to charge after get off work and stops charging when going out to work. The start time of electric vehicle charging follows a normal distribution, and its probability density function is:

[0093]

[0094] Among them: parameter σ s and μ srepresent the standard deviation and mean of the normal distribution controlling the start time of electric vehicle charging, where σ s =3.3,μ s = 18. According to the above formula, the timetable for the start of charging of electric vehicles in the industrial and commercial park can be obtained, and then the owners can be charged according to the charging mode of the industrial and commercial park.

[0095] The capacity of electric vehicles in industrial and commercial parks follows a uniform distribution, as shown in the formula below. 20-60kWh represents the common capacity range of electric vehicles on the market. The probability that electric vehicles in industrial and commercial parks have a capacity within this range is uniformly distributed:

[0096]

[0097] The charging formula for electric vehicles is similar to that for energy storage devices, and is essentially described in terms of the process for lithium-ion batteries. The charging process is as follows:

[0098]

[0099] SOC min ≤SOC i，t ≤SOC max

[0100] in: is the charging power of the electric vehicle, is the battery capacity of the electric vehicle, determined by the uniform distribution equation for determining the capacity of electric vehicles in industrial and commercial parks. SOC min Ensuring that electric vehicles are charged to the minimum SOC required for operation is mainly to ensure that the battery of the electric vehicle will not be deeply discharged during the discharge process, thereby extending the life of the electric vehicle battery. max This corresponds to the maximum SOC that the electric vehicle can achieve, avoiding overcharging of the electric vehicle.

[0101] There are two charging modes for electric vehicles: fast charging mode and smart charging mode. Fast charging mode is suitable for emergency charging needs and can fully charge in a shorter time, but the charging power is higher, which may affect the battery life. Smart charging mode is suitable for users who are not in a hurry to charge. Its charging power is lower and can be dynamically adjusted according to the overall load of the industrial and commercial park and electricity price fluctuations to reduce charging costs and avoid excessive load on the grid. The charging power is limited as follows:

[0102]

[0103] in: It is the charging power of electric vehicles. In fast charging mode, the charging power is equal to the maximum allowable power of the charging pile in the industrial and commercial park, which is generally 20kWh. In smart charging mode, this charging power is between 0 and the maximum power, and can change with the overall load curve of the industrial and commercial park and the fluctuation of real-time electricity prices, thereby achieving two purposes. One is that the charging of electric vehicles in the industrial and commercial park will not be too high during peak hours, thereby putting pressure on the distribution network of the industrial and commercial park. The second is to avoid the peak period of electricity prices to meet the charging needs, thereby reducing the charging costs of car owners.

[0104] The industrial and commercial park is equipped with a shared solar storage and charging system. The system needs to give priority to meeting the internal load of the industrial and commercial park, especially during peak load periods. This can effectively reduce the dependence of the industrial and commercial park load on the distribution network, because during non-peak periods, the load pressure on the distribution network is relatively small. In order to achieve this goal, all types of loads in the industrial and commercial park need to meet the power balance constraints.

[0105] The energy balance formula is as follows:

[0106]

[0107] Where: L t is the total power demand of the industrial and commercial park, which consists of two parts: That is, the basic load of the industrial and commercial park, which generally includes the daily electricity consumption of users, such as the power demand of infrastructure such as lights and washing machines. That is, the electric vehicle charging load in the industrial and commercial park. The size of this part is determined by the penetration rate of electric vehicles in the industrial and commercial park. That is, the charging power of an electric vehicle is The total fleet power of the industrial and commercial park basically depends on how many electric vehicles are equipped in the industrial and commercial park. At present, the distribution network of most industrial and commercial parks is sufficient to meet the basic power demand, but as the number of electric vehicles increases, the load pressure of the power grid will gradually increase. Therefore, the charging load of electric vehicles should be regarded as a separate object of concern, not included in the basic load, and needs to be independently optimized and dispatched.

[0108] The reward function in reinforcement learning plays a core role in guiding agent decision-making, determining how the agent evaluates its behavior and guiding the agent toward the target behavior. To achieve this goal, we hope to optimize the charging and discharging decisions of the energy storage system in the industrial and commercial park through reinforcement learning, especially during peak load periods, when the energy storage system should bear part of the load pressure.

[0109] The penalty function is designed to constrain the agent to reduce behaviors that do not meet the goal. The penalty function is defined as:

[0110]

[0111] in: It represents the energy waste generated at this time step, specifically the surplus electric energy that cannot be fully utilized by photovoltaic power generation, or the energy discharged by the energy storage system that exceeds the demand of the industrial and commercial park.

[0112] The energy output of the photovoltaic power generation system needs to meet the power demand of the industrial and commercial park and minimize waste. The energy balance formula of the photovoltaic system is as follows:

[0113]

[0114] Where: G t It is the electricity generated by the photovoltaic system at each moment. Part of this electricity is used by users in the industrial and commercial park, marked as A portion is stored by energy storage, marked as If there is any remaining photovoltaic power, it is marked as This indicates that this part of the electricity is wasted.

[0115] The power demand of industrial and commercial park users also needs to meet the power balance constraints to ensure the reasonable distribution of industrial and commercial park loads among various energy inputs. The energy balance formula of users is as follows:

[0116]

[0117] L t is the total electricity demand of the industrial and commercial park, part of which comes from photovoltaic power generation, i.e. Part of it comes from the discharge of the energy storage system, that is Another part depends on the power grid, i.e. If the energy storage system discharge far exceeds the industrial and commercial park load demand, it is marked as This indicates that this part of energy is wasted.

[0118] The charging power source of the energy storage system includes the purchase of electricity from the power grid and the power generation of the photovoltaic system. The charging power formula of the energy storage system is as follows:

[0119]

[0120] in: is the charging power from the grid, It is the charging power from the photovoltaic power generation system.

[0121]

[0122] Energy waste generated by the entire industrial and commercial park The source and photovoltaic power generation have not been fully utilized, resulting in photovoltaic power waste And the energy storage discharge far exceeds the demand of industrial and commercial parks, resulting in energy storage power waste This part of energy waste not only affects the power supply efficiency of the industrial and commercial park, but also increases the energy consumption burden of the system. Therefore, it is necessary to punish it in reinforcement learning to reduce unnecessary wasteful behavior. By designing the energy balance of the industrial and commercial park, it can be ensured that the shared photovoltaic storage and charging system bears the load of the industrial and commercial park. However, the successful implementation of this process depends on the guidance of the reward function. Therefore, in order to effectively guide the operation of the shared photovoltaic storage and charging system, it is necessary to design a reasonable industrial and commercial park reward function.

[0123] First, the electricity cost of the industrial and commercial park needs to take into account the cost of purchasing electricity from the power grid and the energy storage system. The calculation formula is as follows:

[0124]

[0125] in: is the electricity cost of the entire industrial and commercial park, It is the real electricity price, which is based on the local electricity price standard. It is the electricity purchased by the industrial and commercial park from the power grid. It is the electricity purchased by the energy storage system from the power grid. The electricity cost is closely related to the fluctuation of the industrial and commercial park load and the electricity price of the power grid. Therefore, rationally optimizing the electricity purchase method can significantly reduce the electricity cost of the industrial and commercial park.

[0126] The charging behavior of the energy storage system not only involves the storage of electrical energy, but also brings certain cost losses, especially the wear and tear of the battery. The cost calculation formula is as follows:

[0127]

[0128] This part is the cost loss caused by the charging behavior of the energy storage system. E L is the unit energy cost of the battery, which can be the initial cost of the battery or the replacement cost. D is the depth of discharge of the battery, which indicates the degree of discharge. C is the cycle life of the battery, which indicates the number of charge and discharge cycles that the battery can experience under specific discharge depth and conditions. The formula shows that the wear cost of the battery is related to multiple factors, including unit energy cost, efficiency, discharge depth, cycle life, and charge and discharge power. By properly managing the charge and discharge strategy, the battery life can be effectively extended and the wear cost can be reduced.

[0129] Taking the above factors into consideration, the reward function of the industrial and commercial park can be designed as follows:

[0130]

[0131] in: is the reward of the industrial and commercial park at time t. It is the electricity cost of the industrial and commercial park. The goal is to reduce the electricity cost and thus reduce the industrial and commercial park’s dependence on the power grid. It is the wear and tear cost of the energy storage system, which aims to take into account the long-term health of the energy storage system and optimize the use of the battery. It is a penalty function that aims to reduce energy waste and ensure efficient operation of the system.

[0132] By designing such a reward function, the system can self-adjust the charging and discharging strategy during the reinforcement learning process, which not only optimizes the economic cost, but also extends the service life of the energy storage system and reduces energy waste, thereby achieving effective management of the load of industrial and commercial parks and reducing dependence on the distribution network.

[0133] After designing the reward function for the industrial and commercial park, the next step is to design a reinforcement learning algorithm to control the charging and discharging decisions of the industrial and commercial park’s energy storage system, so that the system can form the optimal usage strategy according to different time conditions, thereby maximizing the long-term rewards of the industrial and commercial park.

[0134] The core task of reinforcement learning is to enable the agent (in this scenario, the dispatch controller of the industrial and commercial park energy storage system) to learn to maximize its long-term rewards through a series of actions in the environment. In this application scenario, the agent's "action" refers to the decision on the degree of charging and discharging of the energy storage system, and the goal is to maximize the reward function of the industrial and commercial park through these decisions, ensuring that while meeting the load demand of the industrial and commercial park, the electricity cost is reduced, the life of the energy storage system is extended, and energy waste is reduced.

[0135] In the reinforcement learning framework, the state of the environment is a random variable at each step. In this case, the state of the environment is determined by multiple factors such as the current charging state of the energy storage system, load demand, photovoltaic power generation, etc. Assume that at a certain moment, the agent is in state s t , take action t , and then transfer to the next state s t+1 And get rewards The state transfer process can be expressed as:

[0136] s t+1 =f(s t , a t )

[0137] Among them: f is the state transfer function, which indicates the change of the environment state after taking a certain action in a certain state.

[0138] The goal of the agent is to maximize its cumulative reward by choosing appropriate action sequences. This is usually achieved through a value function. Given the current time t, the cumulative reward G tIt is the weighted sum of all rewards from the current moment to the end moment:

[0139]

[0140] Where: γ is the discount factor, usually between 0≤γ≤1. The discount factor controls the weight of future rewards. The smaller the value, the less impact future rewards have on current decisions. The reward function r in the formula t+1 In fact, A simplified representation of , where the impact of the current decision on the agent is evaluated through a reward function.

[0141] The policy π defines the probability distribution of the agent's actions at each state. It can be either deterministic or stochastic.

[0142] π(a|s)=P(a t =a|s t =s)

[0143] This is the formula for the random strategy, which represents the probability of taking action a in state s. In practical applications, random strategies can help explore different decision paths to find the optimal strategy.

[0144] Q-learning is a commonly used reinforcement learning method. It is based on the idea of ​​value iteration and approaches the optimal strategy by continuously updating the action value function. In Q-learning, each pair of states s t and action a t Each state has a corresponding Q value, which represents the expected return of taking that action in that state. The update rule of Q-learning is as follows:

[0145]

[0146] Where: α is the learning rate, which controls the step size of each update. max a′ Q(s t+1 , a′) is in the new state s t+1 Next, take the Q value corresponding to the best action among all possible actions a′.

[0147] By continuously iteratively updating the Q value, the Q-learning method can gradually find the best strategy, that is, taking the best action in each state to maximize the cumulative return.

[0148] The various technical features in the above-mentioned embodiments can be flexibly combined as needed. In order to keep the description concise, all possible combinations of the technical features are not described in detail in the above-mentioned embodiments. However, as long as the combination of these technical features does not cause contradictions in practical applications, they should be deemed to be included in the scope recorded in this specification.

[0149] It is worth noting that this does not mean any limitation on the scope of the invention patent. According to the understanding of ordinary technicians in this field, the above embodiments can still be modified and improved without departing from the concept of the present invention, and these changes and improvements belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the scope defined by the attached claims.

Claims

1. Intelligent scheduling method for shared storage and charging system in industrial and commercial parks based on reinforcement learning, characterized by The specific steps are as follows: (1) By modeling various loads in the industrial and commercial park that includes shared solar-storage-charging, the energy balance of the shared solar-storage-charging system is controlled to achieve intelligent scheduling and optimal management of energy, focusing on the electricity demand of park users, the charging demand of electric vehicles, and the supply of photovoltaic power generation; (2) Establish a model of the shared solar-storage-charging system, analyze its charging and discharging characteristics, and calculate the degradation cost of the battery; (3) By designing a reward function, the reinforcement learning algorithm is guided to make the best charging and discharging decisions at different times to maximize the overall benefits of the industrial and commercial park. In this process, the reinforcement learning agent continuously selects actions based on the current state, executes the corresponding charging and discharging strategies, and optimizes the charging and discharging decisions by feedback reward signals. Through multiple iterations, the shared solar storage and charging system gradually finds the optimal scheduling method to ensure the balance between power supply and demand, reduce energy waste, and extend battery life.

2. The method according to claim 1, characterized in that The modeling described in step (1) is to establish the interactive relationship between energy storage equipment, electric vehicles and other loads. That is, the energy storage equipment can be used as an emergency power supply in the modeling, and can also be charged during peak electricity price periods to reduce dependence on the distribution network. The loads of various industrial and commercial parks include: Energy storage equipment is used to store electrical energy that cannot be consumed immediately in industrial and commercial parks. The mathematical model of energy storage equipment is shown in formula (1): in: represents the state of the energy storage device at time t, P d，t is the charge and discharge power, ΔT is the time step; The capacity range of energy storage equipment is subject to the following constraints: Where: p dod is the lower limit of the device discharge power, is the maximum storage capacity of the device, e d，t is the charge and discharge state of the storage device at time t, is the upper limit of charging power. and are the charge and discharge states of the device at t=19 and t=7, respectively. st is the upper limit coefficient of charging power, p ed is the charging state of the electric vehicle; Formula (2)-Formula (6) are the charging and discharging states and capacity limits of the energy storage device in different time periods; The charging schedule of electric vehicles will vary depending on the charging mode. Electric vehicles adopt the home charging mode, that is, the owner goes home to charge after get off work and stops charging when going out to work; the start time of electric vehicle charging follows a normal distribution, and its probability density function is: Among them: parameter σ s and μ s They represent the standard deviation and mean of the normal distribution that controls the start time of electric vehicle charging, σ s =3.3,μ s =18; According to formula (7), the charging schedule of the electric vehicle is obtained, and then the owner is charged according to the corresponding charging mode; The capacity of electric vehicles in industrial and commercial parks follows a uniform distribution, as shown in formula (8); where 20-60 kWh represents the common capacity range of electric vehicles on the market; the probability that electric vehicles in industrial and commercial parks have a capacity within this range is uniformly distributed: The charging formula of electric vehicles is similar to that of energy storage devices, following the charging process of lithium-ion batteries; the charging process is shown in formula (9) and formula (10): SOC min ≤SOC i，t ≤SOC max (10) in: is the charging power of the electric vehicle, is the battery capacity of the electric vehicle, determined by the uniform distribution equation for determining the capacity of electric vehicles in industrial and commercial parks; SOC min Ensure that the electric vehicle is charged to the minimum SOC required for operation, and ensure that the battery of the electric vehicle will not be deeply discharged during the discharge process, thereby extending the life of the electric vehicle battery; SOCmax corresponds to the maximum SOC that the electric vehicle can reach, avoiding overcharging of the electric vehicle; The charging modes of electric vehicles include fast charging mode and smart charging mode; the fast charging mode is suitable for emergency charging needs and can be fully charged in a shorter time; the smart charging mode is suitable for users who are not in a hurry to charge, with a lower charging power, which is dynamically adjusted according to the overall load of the industrial and commercial park and the fluctuation of electricity prices to reduce charging costs and avoid excessive load on the power grid; The charging power is limited as follows: in: It is the charging power of electric vehicles. In fast charging mode, the charging power is equal to the maximum allowable power of the charging pile in the industrial and commercial park, that is, 20kWh. In smart charging mode, the charging power is between 0 and the maximum power, and changes with the overall load curve of the industrial and commercial park and the fluctuation of real-time electricity prices, thereby achieving two purposes. One is that the charging of electric vehicles in the industrial and commercial park will not be too high during peak hours, thereby putting pressure on the distribution network of the industrial and commercial park. The second is to avoid the peak period of electricity prices to meet the charging needs, thereby reducing the charging costs of car owners.

3. The method according to claim 1, characterized in that: The industrial and commercial park described in step (2) is equipped with a shared photovoltaic storage and charging system, which needs to give priority to meeting the internal load of the industrial and commercial park, that is, the peak load period; in order to effectively reduce the dependence of the industrial and commercial park load on the distribution network, therefore, all types of loads in the industrial and commercial park need to meet the power balance constraint; The power balance constraint formula is as follows: Where: L t is the total power demand of the industrial and commercial park, which consists of two parts. The first part is That is, the basic load of the park, including the power demand for daily electricity consumption; the second part is That is, the electric vehicle charging load in the industrial and commercial park. The size of this part is determined by the penetration rate of electric vehicles in the industrial and commercial park. That is, the charging power of an electric vehicle is The total fleet power of the industrial and commercial park basically depends on how many electric vehicles are equipped in the park; the charging load of electric vehicles is not included in the basic load and requires independent optimization and scheduling.

4. The method according to claim 1, characterized in that: The reward function in reinforcement learning described in step (3) plays a core role in guiding the agent's decision-making, determines how the agent evaluates its behavior, and guides the agent to move closer to the target behavior; Reinforcement learning is used to optimize the charging and discharging decisions of energy storage systems in industrial and commercial parks, especially during peak load periods, when the energy storage system should bear part of the load pressure; A penalty function is used to reduce the occurrence of behaviors that do not meet the target of the constraint agent; the penalty function is defined as: in: It represents the energy waste generated at this time step, specifically the surplus electric energy that cannot be fully utilized by photovoltaic power generation, or the energy discharged by the energy storage system that exceeds the demand of the industrial and commercial park; The energy output of the photovoltaic power generation system needs to meet the power demand of the industrial and commercial park and minimize waste; the energy balance formula of the photovoltaic system is as follows: Where: G t It is the electricity generated by the photovoltaic system at each moment. Part of this electricity is used by users in the industrial and commercial park, marked as A portion is stored by energy storage, marked as If there is any remaining photovoltaic power, it is marked as This indicates that this part of the electricity is wasted; The power demand of industrial and commercial park users also needs to meet the power balance constraints to ensure the reasonable distribution of industrial and commercial park loads among various energy inputs; the energy balance formula of users is as follows: Where: L t is the total electricity demand of the industrial and commercial park, part of which comes from photovoltaic power generation, i.e. Part of it comes from the discharge of the energy storage system, that is Another part depends on the power grid, i.e. If the energy storage system discharge far exceeds the industrial and commercial park load demand, it is marked as This indicates that this part of energy is wasted; The charging power source of the energy storage system includes the purchase of electricity from the power grid and the power generation of the photovoltaic system; the charging power formula of the energy storage system is as follows: in: is the charging power from the grid, The charging power comes from the photovoltaic power generation system; Energy waste generated by the entire industrial and commercial park The source and photovoltaic power generation have not been fully utilized, resulting in photovoltaic power waste And the energy storage discharge far exceeds the demand of industrial and commercial parks, resulting in waste of energy storage electricity This part of electricity waste not only affects the power supply efficiency of the industrial and commercial park, but also increases the energy consumption burden of the system. Therefore, punishment is needed in reinforcement learning to reduce unnecessary wasteful behavior.

5. The method according to claim 4, characterized in that: The reward function ensures that the shared solar storage and charging system can bear the load of the industrial and commercial park through the balance of power in the industrial and commercial park; The electricity cost of the industrial and commercial park needs to take into account the cost of purchasing electricity from the power grid and the energy storage system, which is calculated as shown in formula (18): in: is the electricity cost of the entire industrial and commercial park, It is the real electricity price, which is based on the local electricity price standard. It is the electricity purchased by the industrial and commercial park from the power grid. It is the electricity purchased by the energy storage system from the power grid. The electricity cost is closely related to the load fluctuation of the industrial and commercial park and the electricity price of the power grid. Therefore, the use of optimized electricity purchase methods can significantly reduce the electricity cost of the industrial and commercial park. The charging behavior of the energy storage system not only involves the storage of electrical energy, but also brings cost losses, especially the wear and tear of the battery. The cost loss caused by the charging behavior of the energy storage system is calculated as shown in formula (19): Where: C E is the unit energy cost of the battery, which is the initial cost or replacement cost of the battery; D is the depth of discharge of the battery, which indicates the degree of discharge; L C The cycle life of the battery refers to the number of charge and discharge cycles that the battery can experience under specific discharge depth and conditions. The wear cost of the battery is related to energy cost, efficiency, discharge depth, cycle life, and charge and discharge power. By properly managing the charge and discharge strategy, the battery life can be effectively extended and the wear cost can be reduced. The reward function of the industrial and commercial park is shown in formula (20): in: is the reward of the industrial and commercial park at time t; It is the electricity cost of the industrial and commercial park. The goal is to reduce the electricity cost and thus reduce the industrial and commercial park's dependence on the power grid; It is the wear and tear cost of the energy storage system, taking into account the long-term health of the energy storage system and optimizing the use of the battery; It is a penalty function that reduces energy waste and ensures efficient operation of the system; Through the reward function, the shared solar-storage-charging system self-adjusts the charging and discharging strategy during the reinforcement learning process, which not only optimizes the economic cost, but also extends the service life of the energy storage system and reduces energy waste, thereby achieving effective management of the load of industrial and commercial parks and reducing dependence on the distribution network.

6. The method according to claim 1, characterized in that: The reinforcement learning algorithm described in step (3) is used to control the charging and discharging decisions of the industrial and commercial park energy storage system, so that the system forms an optimal usage strategy according to different time conditions, thereby maximizing the long-term rewards of the industrial and commercial park; The core task of reinforcement learning is to enable the agent, in this scenario, the dispatch controller of the energy storage system of the industrial and commercial park, to learn to maximize its long-term rewards through a series of actions in the environment; in this application scenario, the agent's "action" refers to the decision on the charging and discharging degree of the energy storage system, and the goal is to maximize the reward function of the industrial and commercial park through these decisions, ensuring that while meeting the load demand of the industrial and commercial park, the electricity cost is reduced, the life of the energy storage system is extended, and energy waste is reduced; In the reinforcement learning framework, the state of the environment is a random variable at each step; in this case, the state of the environment is determined by multiple factors such as the current charging state of the energy storage system, load demand, and photovoltaic power generation; assuming that at a certain moment, the agent is in state s t , take action t , and then transfer to the next state s t+1 and get rewarded The state transfer process can be expressed as: s t+1 =f(s t ,a t ) (21) Among them: f is the state transfer function, which indicates the change of the environment state after taking a certain action in a certain state; The goal of the agent is to maximize its cumulative reward by choosing appropriate action sequences; this is usually achieved through a value function; given the current time t, the cumulative reward G t It is the weighted sum of all rewards from the current moment to the end moment: Where: γ is the discount factor, usually between 0≤γ≤1; the discount factor controls the weight of future rewards. The smaller the value, the less impact future rewards have on current decisions. The reward function r in formula (22) t+1 yes A simplified representation of , that is, evaluating the impact of the current decision on the agent through the reward function; The policy π defines the probability distribution of the agent's actions in each state, which can be deterministic or stochastic; π(a|s)=P(a t =a|s t =s) (23) The random strategy is shown in formula (23), which represents the probability of taking action a in state s. In practical applications, random strategies help explore different decision paths to find the optimal strategy. The reinforcement learning method adopts the Q-learning method, which is based on value iteration and approaches the optimal strategy by continuously updating the action value function. In the Q-learning method, each pair of states s t and action a t There is a corresponding Q value, which represents the expected return of taking the action in that state; the update rule of the Q-learning method is as follows: Q(s t ,a t )←Q(s t ,a t )+α(r t+1 +γmax a′ Q(s t+1 ,a′)-Q(s t ,a t )) (24) Among them: α is the learning rate, which controls the step size of each update; max a′ Q(s t+1 , a ′ ) is in the new state s t+1 Next, take all possible actions a ′ The Q value corresponding to the best action in; By continuously iteratively updating the Q value, the Q-learning method can gradually find the best strategy, that is, taking the best action in each state to maximize the cumulative return.

Images