Intelligent scheduling method for industrial and commercial park shared storage and charging system based on reinforcement learning

Abstract

The present application relates to a kind of based on the intelligent scheduling method of industrial and commercial park shared storage and filling system of reinforcement learning, the modeling of various loads in industrial and commercial park is carried out, the model of light storage and filling system is established, its charge-discharge characteristics are analyzed, and the degradation cost of battery is calculated.Through designing reasonable reward function, guide reinforcement learning algorithm to make optimal charge-discharge decision at different time, to maximize the overall benefit of industrial and commercial park.Reinforcement learning agent continuously selects action according to current state, executes corresponding charge-discharge strategy, and optimizes the strategy through feedback reward signal.Through multiple iterations, gradually find the optimal scheduling mode, ensure the balance of power supply and demand, reduce energy waste, prolong the service life of battery.The present application not only improves the use efficiency of energy in industrial and commercial park, but also greatly reduces energy cost, realizes the double promotion of economy and sustainability, and can be seamlessly connected with power grid through dynamic load adjustment, improves the safety and stability of 750 kV and above level power transmission system.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of smart grid and energy management, and specifically relates to a kind of intelligent scheduling method of industrial and commercial park shared storage and charging system based on reinforcement learning, which aims to reduce electricity cost and improve energy utilization efficiency by optimizing the charge-discharge strategy of energy storage system, effectively alleviate large-scale power grid load pressure, support the safe operation of large power grid, and prolong the service life of energy storage battery. BACKGROUND

[0002] With the transformation of global energy structure and the wide application of new energy, industrial and commercial park energy management has gradually become a research hotspot. Especially with the popularity of electric vehicles and the introduction of renewable energy such as photovoltaic, the complexity of energy demand and supply in industrial and commercial park is increasing. The existing traditional power grid system is difficult to meet such diversified demand, resulting in high dependence of industrial and commercial park on power grid, and due to the degradation problem of battery energy storage system, it is difficult to realize long-term stable operation. Therefore, how to optimize the charge-discharge strategy of energy storage system on the basis of guaranteeing the load of industrial and commercial park has become a key problem to improve the economy and sustainability of energy system. At present, although some researches have tried to reduce electricity cost and battery loss by optimizing the scheduling of battery energy storage system, due to the diversity and uncertainty of energy management, traditional scheduling methods often fail to achieve ideal results in actual scenarios. In addition, due to the large fluctuation of load in industrial and commercial park, how to reasonably schedule energy storage equipment to meet electricity demand and prolong the service life of energy storage system has become a key problem to be solved in this field. Traditional scheduling methods often fail to balance battery life, electricity cost and overall system performance, so an intelligent scheduling method based on reinforcement learning is needed to achieve efficient optimization of energy scheduling and resource sharing. SUMMARY

[0003] The purpose of the present application is to provide an intelligent scheduling method of industrial and commercial park shared storage and charging system based on reinforcement learning. By optimizing the charge-discharge scheduling of photovoltaic storage and charging system in industrial and commercial park, the present application reduces power cost and reduces degradation loss of energy storage system, thereby improving the economy and resource utilization efficiency of the overall system.

[0004] In the present application, first, the load model of various types of electrical equipment in industrial and commercial park is established, considering the energy consumption characteristics and load demand of different equipment. At the same time, for the shared energy storage system in industrial and commercial park, mathematical models of its charge-discharge characteristics, capacity and degradation cost are established. By using reinforcement learning algorithm, the charge-discharge decision of photovoltaic storage and charging system is scheduled, so that the system can reduce battery degradation loss and electricity cost while meeting load demand.

[0005] The intelligent scheduling method of industrial and commercial park shared storage and charging system based on reinforcement learning proposed in the present application has the following specific steps:

[0006] (1) By modeling various loads in the industrial and commercial park containing shared light storage and charging, the energy balance of the shared light storage and charging system is controlled, the intelligent scheduling and optimized management of energy are realized, and the power demand of the park users, the charging demand of electric vehicles and the supply of photovoltaic power generation are mainly considered;

[0007] (2) A model of the shared light storage and charging system is established, the charging and discharging characteristics are analyzed, and the degradation cost of the battery is calculated;

[0008] (3) By designing a reasonable reward function, the reinforcement learning algorithm is guided to make the optimal charging and discharging decision at different times to maximize the overall benefit of the industrial and commercial park; In this process, the reinforcement learning agent continuously selects actions according to the current state, executes the corresponding charging and discharging strategy, and optimizes the strategy through feedback reward signals; Through multiple iterations, the system gradually finds the optimal scheduling mode to ensure the balance between power supply and demand, reduce energy waste and prolong the service life of the battery.

[0009] In the present application, the modeling in step (1) establishes the interaction relationship of energy storage devices, electric vehicles and other loads. Specifically, the energy storage device can be used as an emergency power supply in modeling, and can also be charged during peak electricity price period to reduce dependence on the power distribution network. Various industrial and commercial park loads include:

[0010] The energy storage device is used to store electric energy that cannot be immediately consumed in the industrial and commercial park. The mathematical model of the energy storage device is shown in formula (1):

[0011]

[0012] Among them: represents the state of the energy storage device at time t, P d，t is the charging and discharging power, and ΔT is the time step;

[0013] The capacity range of the energy storage device is subject to the following constraints:

[0014]

[0015] Among them: p dod is the lower limit of the device discharging power, is the maximum storage capacity of the device, e d，t is the charging and discharging state of the storage device at time t, is the upper limit of the charging power, and are the states of the device at t=19 and t=7, respectively, p st is the upper limit coefficient of the charging power, and p edis the state of charge of the electric vehicle; formula (2) - formula (6) are the state of charge and capacity limit of the energy storage device in different time periods;

[0016] Electric vehicles are indispensable loads in industrial and commercial parks. Depending on the charging mode, the charging schedule of electric vehicles will vary. Electric vehicles use the home charging mode, which means that the owner charges the vehicle after work and stops charging when leaving for work. The start time of electric vehicle charging follows a normal distribution, and its probability density function is:

[0017]

[0018] wherein: parameters σ s and μ s respectively represent the standard deviation and mean value of the normal distribution of the control electric vehicle charging start time, σ s = 3.3, μ s = 18; According to formula (7), the electric vehicle charging start time table is obtained, and then the owner is charged according to the corresponding charging mode;

[0019] The capacity of electric vehicles in industrial and commercial parks follows a uniform distribution, as shown in formula (8); wherein 20-60kWh represents the common capacity range of electric vehicles in the market; The probability of electric vehicles in industrial and commercial parks having a capacity in this range is uniformly distributed:

[0020]

[0021] The charging formula of electric vehicles is similar to that of energy storage devices, according to the charging process of lithium-ion batteries; The charging process is shown in formula (9) and formula (10):

[0022]

[0023] SOC min ≤ SOC i，t ≤ SOC max (10)

[0024] wherein: is the charging power of the electric vehicle, is the battery capacity of the electric vehicle, which is determined by the uniform distribution equation for determining the capacity of electric vehicles in industrial and commercial parks; SOC min ensures that the electric vehicle is charged to the minimum SOC required for operation, mainly to ensure that the battery of the electric vehicle is not deeply discharged during discharging, thereby prolonging the life of the battery of the electric vehicle; And SOC max corresponds to the maximum SOC that the electric vehicle can reach, to avoid overcharging the electric vehicle;

[0025] Electric vehicle charging methods include fast charging mode and smart charging mode. Fast charging mode is suitable for emergency charging needs and can fully charge the battery in a short time, but the charging power is high, which may affect battery life. Smart charging mode is suitable for users who are not in a hurry to charge. Its charging power is lower and can be dynamically adjusted according to the overall load of industrial and commercial parks and electricity price fluctuations to reduce charging costs and avoid excessive grid load.

[0026] The charging power is limited as follows:

[0027]

[0028] in: This refers to the charging power of electric vehicles. In fast charging mode, the charging power is equal to the maximum allowable power of charging piles in the industrial and commercial park, which is 20kWh. In smart charging mode, the charging power is between 0 and the maximum power, and varies with the overall load curve of the industrial and commercial park and the fluctuation of real-time electricity prices. This achieves two purposes: first, to prevent the charging of electric vehicles in the industrial and commercial park from being too high during peak hours, thus putting pressure on the power distribution network of the industrial and commercial park; and second, to avoid peak electricity price periods to meet charging needs, thereby reducing the charging costs for car owners.

[0029] In this invention, the industrial and commercial park mentioned in step (2) is equipped with a shared photovoltaic energy storage and charging system. The shared photovoltaic energy storage and charging system needs to prioritize the load within the industrial and commercial park, especially during peak load periods, in order to effectively reduce the dependence of the industrial and commercial park load on the distribution network, because the load pressure on the distribution network is relatively small during off-peak periods. Therefore, various loads within the industrial and commercial park need to meet the power balance constraints.

[0030] The formula for the energy balance constraint is as follows:

[0031]

[0032] Where: L t The total electricity demand of the industrial and commercial park consists of two parts, the first part being... The first part is the park's basic load, including daily electricity consumption such as the power demand for infrastructure like lights and air conditioning; the second part is... This refers to the charging load of electric vehicles in industrial and commercial parks. The size of this portion is determined by the penetration rate of electric vehicles within the industrial and commercial parks, i.e., the charging power of a single electric vehicle is... The total fleet capacity of the industrial and commercial park depends primarily on the number of electric vehicles in the park; the charging load of electric vehicles is not included in the base load and requires independent optimization and scheduling.

[0033] In this invention, the reward function in reinforcement learning described in step (3) plays a core role in guiding agent decision-making, determining how the agent evaluates its behavior and guiding the agent toward the target behavior; reinforcement learning is used to optimize the charging and discharging decisions of the energy storage system in industrial and commercial parks, especially during peak load periods, when the energy storage system should bear part of the load pressure;

[0034] To constrain agents from engaging in behaviors that do not conform to the objectives, a penalty function is used to reduce such behaviors; the penalty function is defined as:

[0035]

[0036] in: This refers to the energy waste generated at that time step, specifically the surplus electrical energy that photovoltaic power generation fails to fully utilize, or the energy discharged by the energy storage system that exceeds the demand of the industrial and commercial park.

[0037] The energy output of the photovoltaic power generation system needs to meet the electricity demand of the industrial and commercial park and minimize waste; the energy balance formula (14) of the photovoltaic system is as follows:

[0038]

[0039] Among them: G t This refers to the electrical energy generated by the photovoltaic system at every moment. A portion of this electrical energy is used by users in industrial and commercial parks and is labeled as... Part of it is stored in energy storage, marked as If there is any remaining photovoltaic power, it is marked as... This indicates that this portion of electrical energy is being wasted;

[0040] The electricity demand of users in industrial and commercial parks also needs to meet power balance constraints to ensure a reasonable distribution of load among various energy inputs. The energy balance formula for users is as follows:

[0041]

[0042] Where: L t This represents the total electricity demand of the industrial and commercial park, part of which comes from photovoltaic power generation, i.e. Part of it comes from the discharge of the energy storage system, that is Another part relies on the power grid, that is If the discharge capacity of the energy storage system far exceeds the load demand of the industrial and commercial park, it will be marked as... This indicates that this portion of energy is wasted;

[0043] The charging power of an energy storage system comes from electricity purchased from the grid and electricity generated by a photovoltaic system; the charging power formula for an energy storage system is as follows:

[0044]

[0045] in: For charging power from the grid, The charging power comes from the photovoltaic power generation system;

[0046]

[0047] Energy waste generated throughout the industrial and commercial park The source of photovoltaic power generation is not fully utilized, resulting in a waste of photovoltaic power. Furthermore, the discharge of energy storage far exceeds the demand of industrial and commercial parks, resulting in a waste of energy storage power. This waste of electricity not only affects the power supply efficiency of industrial and commercial parks, but also increases the energy consumption burden of the system. Therefore, it is necessary to impose penalties in reinforcement learning to reduce unnecessary waste.

[0048] (14) The reward function, through the power balance of the industrial and commercial park, ensures that the shared photovoltaic energy storage and charging system can bear the load of the industrial and commercial park;

[0049] The electricity cost of industrial and commercial parks needs to take into account the cost of purchasing electricity from the grid and energy storage systems, which is calculated as shown in formula (18):

[0050]

[0051] in: This is the electricity cost for the entire industrial and commercial park. This is the actual electricity price, based on local electricity price standards. The electricity that the industrial and commercial park purchases from the power grid. The energy storage system purchases electricity from the grid; the cost of electricity is closely related to the fluctuations in the load of the industrial and commercial park and the electricity price of the grid. Therefore, adopting an optimized way of purchasing electricity can significantly reduce the electricity cost of the industrial and commercial park.

[0052] The charging behavior of an energy storage system not only involves the storage of electrical energy, but also incurs cost losses, especially battery wear and tear; the cost losses caused by the charging behavior of an energy storage system are calculated as shown in formula (19):

[0053]

[0054] Where: C E This is the unit energy cost of the battery, which can be the initial cost or replacement cost; D is the depth of discharge, indicating the degree of discharge; L CBattery cycle life refers to the number of charge-discharge cycles a battery can undergo under specific depth of discharge and conditions. Battery wear and tear costs are related to energy costs, efficiency, depth of discharge, cycle life, and charge / discharge power. By rationally managing charge and discharge strategies, battery life can be effectively extended and wear and tear costs reduced.

[0055] The reward function for industrial and commercial parks is given by formula (20):

[0056]

[0057] in: It is a reward for industrial and commercial parks at time t; It refers to the electricity costs of industrial and commercial parks, with the goal of reducing electricity costs and thus reducing the dependence of industrial and commercial parks on the power grid; It is the wear and tear cost of the energy storage system, which aims to balance the long-term health of the energy storage system and optimize the usage of the battery. It is a penalty function designed to reduce energy waste and ensure the efficient operation of the system;

[0058] Through the reward function, the shared photovoltaic-storage-charging system can self-adjust its charging and discharging strategies during reinforcement learning, which not only optimizes economic costs but also extends the lifespan of the energy storage system and reduces energy waste, thereby achieving effective management of the load in industrial and commercial parks and reducing dependence on the distribution network.

[0059] In this invention, the reinforcement learning algorithm described in step (3) is used to control the charging and discharging decisions of the energy storage system in the industrial and commercial park, so that the system can form the optimal usage strategy according to different time conditions, thereby maximizing the long-term reward of the industrial and commercial park.

[0060] The core task of reinforcement learning is to enable the agent (in this scenario, the dispatch controller of the energy storage system in the industrial and commercial park) to learn to maximize its long-term rewards through a series of actions in the environment. In this application scenario, the agent's "actions" refer to decisions on the charging and discharging levels of the energy storage system, and the goal is to maximize the reward function of the industrial and commercial park through these decisions, so as to ensure that the load demand of the industrial and commercial park is met while reducing electricity costs, extending the life of the energy storage system, and reducing energy waste.

[0061] In the reinforcement learning framework, the state of the environment is a random variable at each step. In this case, the state of the environment is determined by multiple factors such as the current charging state of the energy storage system, load demand, and photovoltaic power generation. Suppose that at a certain moment, the agent is in state s. t Take action t Then transition to the next state s t+1 And receive a reward The state transition process can then be represented as:

[0062] s t+1 =f(s) t a t ) (twenty one)

[0063] Where f is the state transition function, which represents the change in the environmental state after taking a certain action in a certain state;

[0064] An agent's goal is to maximize its cumulative reward by selecting an appropriate sequence of actions; this is typically achieved through a value function; given the current time t, the cumulative reward G... t It is the weighted sum of all rewards from the moment the action begins until the moment it ends:

[0065]

[0066] Where: γ is the discount factor, typically between 0 ≤ γ ≤ 1; the discount factor controls the weight of future rewards, the smaller the value, the smaller the impact of future rewards on the current decision; and the reward function r in the formula... t+1 Actually it is A simplified representation of this is that the impact of the current decision on the agent is evaluated through a reward function;

[0067] Policy π is defined as the probability distribution of the agent's action in each state; it can be deterministic or random.

[0068] π(a|s)=P(a t =a|s t =s)(23)

[0069] The stochastic policy is given by formula (23), which represents the probability of taking action a in state s. In practical applications, the stochastic policy can help explore different decision paths and thus find the optimal policy.

[0070] The reinforcement learning method employs Q-learning, which is based on value iteration and approximates the optimal policy by continuously updating the action-value function. In the Q-learning method, for each pair of states s... t and action a t Each state has a corresponding Q-value, representing the expected reward of taking that action in that state; the update rule of the Q-learning method is as follows:

[0071] Q(s t a t )←Q(s t a t )+α(r t+1 +γmax a′ Q(s t+1 ,a′)-Q(s t at )) (twenty four)

[0072] Where: α is the learning rate, controlling the step size for each update; max a′ Q(s t+1 ,a′) is in the new state s t+1 Under these conditions, take the best action among all possible actions a′ and find the corresponding Q value.

[0073] By continuously iterating and updating the Q-value, the Q-learning method can gradually find the optimal strategy, that is, to take the best action in each state to maximize the cumulative reward.

[0074] The beneficial effects of this invention are as follows: By optimizing the charging and discharging strategy of the energy storage system, this invention maximizes the utilization of renewable energy and reduces dependence on the traditional power grid. Through reinforcement learning algorithm optimization, it minimizes the cost of purchasing electricity from the grid and rationally schedules the energy storage system, thereby reducing overall electricity costs. While optimizing electricity costs, it also considers battery degradation to ensure the healthy use of the energy storage batteries. Through a penalty mechanism, it reduces the waste of photovoltaic and energy storage system power, further improving the overall economic efficiency of the system. This framework can be widely applied to industrial and commercial parks of different sizes and types, exhibiting strong adaptability, and is particularly suitable for implementation in smart grid and multi-microgrid environments.

[0075] This invention provides an innovative intelligent scheduling method for shared energy storage and charging systems in industrial and commercial parks based on reinforcement learning. It can effectively solve various challenges in power dispatching under a multi-microgrid shared architecture, optimize the operation of energy systems in industrial and commercial parks, improve energy efficiency and economy, and enhance the safety and stability of the power grid. It has broad application prospects and promotional value. Attached Figure Description

[0076] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0077] Figure 1 This is a flowchart of the intelligent scheduling method for a shared storage and charging system in industrial and commercial parks based on reinforcement learning, which is involved in this invention.

[0078] Figure 2 This is a framework diagram of energy balance in industrial and commercial parks in the intelligent scheduling method for shared energy storage and charging systems in industrial and commercial parks based on reinforcement learning, which is involved in this invention. Detailed Implementation

[0079] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0080] Example 1: In the intelligent scheduling method for a shared energy storage and charging system in industrial and commercial parks based on reinforcement learning, the industrial and commercial parks are equipped with different types of load equipment, including basic electricity loads and electric vehicles. The industrial and commercial parks are also equipped with a shared photovoltaic-energy storage and charging system, which includes photovoltaic panels and centralized energy storage. The collaborative work between these devices can provide sufficient energy supply. Through the shared photovoltaic-energy storage and charging system, the load demand and energy supply in different areas are optimized and scheduled.

[0081] The main objective of this invention is to achieve optimized scheduling of shared energy storage systems in industrial and commercial parks, minimizing electricity costs and energy storage system degradation costs, and improving overall energy utilization efficiency. Through reinforcement learning algorithms, based on actual load demand and renewable energy supply, the charging and discharging decisions of the photovoltaic-storage-charging system are adjusted, thereby optimizing the overall system performance and economy.

[0082] In this invention, reinforcement learning algorithms are used to dynamically adjust the charging and discharging strategies of an energy storage system. First, a reward function is designed to consider the electricity demand of various devices within the industrial and commercial park, energy costs, and the degradation losses of the energy storage system. By setting an objective function, the algorithm aims to minimize the weighted sum of electricity costs and battery degradation costs. To achieve this goal, the system dynamically adjusts the charging and discharging amounts of the energy storage system through real-time monitoring of load, energy production, and the energy storage system's status.

[0083] This method utilizes the Q-learning strategy in reinforcement learning algorithms to improve the accuracy and efficiency of charging and discharging decisions through continuous strategy optimization. Specifically, at each time step, the state of the industrial and commercial park energy storage system (such as remaining battery capacity, load demand, and energy supply) serves as input. The reinforcement learning agent selects the optimal charging and discharging operation based on the current state and evaluates the effectiveness of the current decision through a reward function. As training progresses, the agent gradually learns the optimal charging and discharging strategies under various different scenarios.

[0084] Figure 1 This is a flowchart of the intelligent scheduling method for a shared storage and charging system in industrial and commercial parks based on reinforcement learning, which is involved in this invention.

[0085] Figure 2This is a framework diagram of energy balance in industrial and commercial parks, based on the reinforcement learning-based intelligent scheduling method for shared energy storage and charging systems. It shows the user's basic electricity load, fleet charging needs, and the shared photovoltaic-energy storage and charging system within the industrial and commercial park. The method includes:

[0086] Modeling is performed on various loads in industrial and commercial parks that include shared photovoltaic, energy storage, and charging systems, particularly the interactions between energy storage devices, electric vehicles, and other loads. Specifically, energy storage devices in this system can serve as both emergency power and charging during peak electricity price periods, reducing dependence on the distribution network. Various loads in the industrial and commercial parks include: energy storage devices used to store electrical energy that cannot be consumed immediately within the industrial and commercial parks; the mathematical model for energy storage is shown below:

[0087]

[0088] in: P represents the state of the energy storage device at time t. d，t ΔT represents the charging / discharging power, and ΔT represents the time step.

[0089] The capacity range of energy storage devices is subject to the following constraints:

[0090]

[0091]

[0092] Where: p dod This is the lower limit of the equipment's discharge power. It is the device's maximum storage capacity, e d，t It represents the charging and discharging state of the storage device at time t. It is the upper limit of charging power. and These are the states of the device at times t=19 and t=7, respectively, p st It is the upper limit coefficient of charging power, p ed This refers to the charging status of electric vehicles; the formula above describes the charging and discharging status and capacity limitations of energy storage devices at different times. Electric vehicles have become an indispensable load in industrial and commercial parks. The charging schedule for electric vehicles varies depending on the charging mode. Generally, electric vehicles use a home charging mode, meaning the owner charges at home after get off work and stops charging when leaving for work. The start time of electric vehicle charging follows a normal distribution, with the probability density function as follows:

[0093]

[0094] Where: parameter σ s and μ sLet σ and σ' represent the standard deviation and mean of the normal distribution controlling the start time of electric vehicle charging, respectively. s =3.3,μ s =18. Based on the formula above, the charging schedule for electric vehicles in the industrial and commercial park can be obtained, and then charging will be provided to vehicle owners according to the charging mode of the industrial and commercial park.

[0095] The capacity of electric vehicles in industrial and commercial parks follows a uniform distribution, as shown in the formula below. Here, 20-60 kWh represents the common capacity range of electric vehicles on the market. The probability that electric vehicles in industrial and commercial parks have capacities within this range is uniformly distributed:

[0096]

[0097] The charging formula for electric vehicles is similar to that for energy storage devices, essentially describing the process of lithium-ion batteries. This charging process is as follows:

[0098]

[0099] SOC min ≤SOC i，t ≤SOC max

[0100] in: It refers to the charging power of electric vehicles. The battery capacity of electric vehicles is determined by a uniform distribution equation that defines the capacity of trolleybuses in industrial and commercial parks. SOC min Ensuring that an electric vehicle is charged to the minimum SOC required for operation primarily prevents the battery from being deeply discharged during the discharge process, thereby extending the battery's lifespan. SOC... max This corresponds to the maximum SOC that an electric vehicle can achieve, thus preventing overcharging of electric vehicles.

[0101] Electric vehicle charging methods include fast charging and smart charging. Fast charging is suitable for urgent charging needs, capable of fully charging the battery in a short time, but its high charging power may impact battery life. Smart charging is suitable for users who are not in a hurry to charge; its charging power is lower and can be dynamically adjusted based on the overall load of industrial and commercial parks and electricity price fluctuations to reduce charging costs and avoid excessive grid load. The limitations on charging power are as follows:

[0102]

[0103] in: This refers to the charging power of electric vehicles. In fast charging mode, the charging power is equal to the maximum allowable power of charging piles in industrial and commercial parks, which is generally 20kWh. In smart charging mode, this charging power is between 0 and the maximum power, and can change with the overall load curve of the industrial and commercial park and the fluctuation of real-time electricity prices. This achieves two purposes: first, it prevents electric vehicle charging in industrial and commercial parks from being too high during peak hours, thus putting pressure on the power distribution network of the industrial and commercial parks; second, it avoids peak electricity price periods to meet charging needs, thereby reducing the charging costs for car owners.

[0104] Industrial and commercial parks are equipped with shared photovoltaic, energy storage, and charging systems. These systems must prioritize the load within the industrial and commercial parks, especially during peak load periods. This effectively reduces the parks' dependence on the power distribution network, as the network experiences less load during off-peak hours. To achieve this, all types of loads within the industrial and commercial parks must meet power balance constraints.

[0105] The energy balance formula is as follows:

[0106]

[0107] Where: L t The total electricity demand of the industrial and commercial park consists of two parts, one of which is This refers to the basic load of industrial and commercial parks, which generally includes the daily electricity consumption of users, such as the power demand of infrastructure like lights and washing machines. The second part is... This refers to the charging load of electric vehicles in industrial and commercial parks. The size of this portion is determined by the penetration rate of electric vehicles within the industrial and commercial parks, i.e., the charging power of a single electric vehicle is... The total power capacity of the electric vehicle fleet in an industrial and commercial park largely depends on the number of electric vehicles deployed within the park. Currently, the power distribution network in most industrial and commercial parks is sufficient to meet basic electricity needs, but as the number of electric vehicles increases, the load on the power grid will gradually increase. Therefore, the charging load of electric vehicles should be considered separately, not included in the basic load, and requires independent optimization and scheduling.

[0108] In reinforcement learning, the reward function plays a central role in guiding agent decision-making, determining how the agent evaluates its behavior and guiding it towards the target behavior. To achieve this goal, we aim to optimize the charging and discharging decisions of energy storage systems in industrial and commercial parks through reinforcement learning, especially during peak load periods when the energy storage system should bear a portion of the load pressure.

[0109] The penalty function is designed to constrain the agent to reduce behaviors that do not conform to the objective. This penalty function is defined as follows:

[0110]

[0111] in: This refers to the energy waste generated at that time step, specifically the surplus electrical energy that photovoltaic power generation fails to fully utilize, or the energy discharged by the energy storage system exceeding the demand of the industrial and commercial park.

[0112] The energy output of a photovoltaic power generation system needs to meet the electricity demand of the industrial and commercial park while minimizing waste. The energy balance formula for a photovoltaic system is as follows:

[0113]

[0114] Among them: G t This refers to the electrical energy generated by the photovoltaic system at every moment. A portion of this electrical energy is used by users in industrial and commercial parks and is labeled as... Part of it is stored in energy storage, marked as If there is any remaining photovoltaic power, it is marked as... This indicates that this portion of electrical energy is being wasted.

[0115] The electricity demand of users in industrial and commercial parks also needs to meet power balance constraints to ensure a reasonable distribution of load among various energy inputs. The energy balance formula for users is as follows:

[0116]

[0117] L t This represents the total electricity demand of the industrial and commercial park, part of which comes from photovoltaic power generation, i.e. Part of it comes from the discharge of the energy storage system, that is Another part relies on the power grid, that is If the discharge capacity of the energy storage system far exceeds the load demand of the industrial and commercial park, it will be marked as... This indicates that this portion of energy was wasted.

[0118] The charging power for energy storage systems comes from electricity purchased from the grid and electricity generated by photovoltaic systems. The charging power formula for an energy storage system is as follows:

[0119]

[0120] in: For charging power from the grid, The charging power comes from the photovoltaic power generation system.

[0121]

[0122] Energy waste generated throughout the industrial and commercial park The source of photovoltaic power generation is not fully utilized, resulting in a waste of photovoltaic power. And the waste of energy storage power caused by energy storage discharge far exceeding the demand of industrial and commercial parks. This energy waste not only affects the power supply efficiency of industrial and commercial parks but also increases the energy consumption burden of the system. Therefore, it needs to be penalized in reinforcement learning to reduce unnecessary waste. By designing the power balance of industrial and commercial parks, it can be ensured that the shared photovoltaic-storage-charging system can handle the load of the parks. However, the successful implementation of this process depends on the guidance of the reward function. Therefore, in order to effectively guide the operation of the shared photovoltaic-storage-charging system, a reasonable reward function for industrial and commercial parks needs to be designed.

[0123] First, the electricity costs for industrial and commercial parks need to consider the cost of purchasing electricity from the grid and energy storage systems. The calculation formula is as follows:

[0124]

[0125] in: This is the electricity cost for the entire industrial and commercial park. This is the actual electricity price, based on local electricity price standards. The electricity that the industrial and commercial park purchases from the power grid. This refers to the electricity purchased by the energy storage system from the grid. Electricity costs are closely related to fluctuations in the load of industrial and commercial parks and the grid's electricity price; therefore, optimizing electricity purchase methods can significantly reduce the electricity costs of industrial and commercial parks.

[0126] The charging process of an energy storage system not only involves the storage of electrical energy but also incurs certain costs, particularly battery wear and tear. The cost calculation formula is as follows:

[0127]

[0128] This portion represents the cost losses incurred due to the charging activities of the energy storage system. Among them, C... E This is the unit energy cost of the battery, which can be the initial cost or the replacement cost. D is the depth of discharge, indicating the extent of discharge. L C Cycle life is the number of charge-discharge cycles a battery can withstand under specific depths of discharge and conditions. This formula shows that battery wear and tear costs are related to multiple factors, including cost per unit energy, efficiency, depth of discharge, cycle life, and charge / discharge power. By rationally managing charge and discharge strategies, battery life can be effectively extended and wear and tear costs reduced.

[0129] Taking all the above factors into account, the reward function for industrial and commercial parks can be designed in the following form:

[0130]

[0131] in: It is a reward for industrial and commercial parks at time t. It refers to the electricity costs of industrial and commercial parks, with the goal of reducing electricity costs and thus reducing the parks' dependence on the power grid. It is the wear and tear cost of the energy storage system, which aims to balance the long-term health of the energy storage system and optimize the usage of the battery. It is a penalty function designed to reduce energy waste and ensure the efficient operation of the system.

[0132] By designing such a reward function, the system can self-adjust its charging and discharging strategy during reinforcement learning, which not only optimizes economic costs but also extends the lifespan of the energy storage system and reduces energy waste, thereby achieving effective management of the load in industrial and commercial parks and reducing dependence on the power distribution network.

[0133] After designing the reward function for the industrial and commercial park, the next step is to design a reinforcement learning algorithm. This algorithm will control the charging and discharging decisions of the energy storage system in the industrial and commercial park, enabling the system to form the optimal usage strategy based on different time-of-day conditions, thereby maximizing the long-term reward of the industrial and commercial park.

[0134] The core task of reinforcement learning is to enable an agent (in this scenario, the dispatch controller of an industrial park energy storage system) to learn to maximize its long-term rewards through a series of actions in the environment. In this application scenario, the agent's "actions" refer to decisions regarding the charging and discharging levels of the energy storage system, and the goal is to maximize the reward function of the industrial park through these decisions, ensuring that the load demand of the industrial park is met while reducing electricity costs, extending the lifespan of the energy storage system, and reducing energy waste.

[0135] In reinforcement learning frameworks, the state of the environment is a random variable at each step. In this case, the state of the environment is determined by multiple factors, including the current charging state of the energy storage system, load demand, and photovoltaic power generation. Suppose that at a certain moment, the agent is in state s. t Take action t Then transition to the next state s t+1 And receive a reward The state transition process can then be represented as:

[0136] s t+1 =f(s) t a t )

[0137] Where f is the state transition function, which represents the change in the environmental state after taking a certain action in a certain state.

[0138] An agent's goal is to maximize its cumulative reward by choosing an appropriate sequence of actions. This is typically achieved through a value function. Given the current time t, the cumulative reward G... tIt is the weighted sum of all rewards from the moment the action begins until the moment it ends:

[0139]

[0140] Where γ is the discount factor, typically between 0 and 1. The discount factor controls the weight of future rewards; the smaller the value, the less influence future rewards have on the current decision. The reward function r in the formula... t+1 Actually it is A simplified representation of this is that the impact of the current decision on the agent is evaluated through a reward function.

[0141] Policy π defines the probability distribution of the agent's actions in each state. It can be deterministic or random.

[0142] π(a|s)=P(a t =a|s t =s)

[0143] This is the formula for a stochastic policy, representing the probability of taking action a in state s. In practical applications, stochastic policies can help explore different decision paths, thereby finding the optimal policy.

[0144] Q-learning is a commonly used reinforcement learning method that is based on the idea of ​​value iteration. It approximates the optimal policy by continuously updating the action-value function. In Q-learning, for each pair of states s... t and action a t Each state has a corresponding Q-value, representing the expected reward of taking that action in that state. The update rule for Q-learning is as follows:

[0145]

[0146] Where: α is the learning rate, which controls the step size for each update. max a′ Q(s t+1 ,a′) is in the new state s t+1 Given the best possible action a′ among all possible actions, take the Q value corresponding to that action.

[0147] By continuously iterating and updating the Q-value, the Q-learning method can gradually find the optimal strategy, that is, to take the best action in each state to maximize the cumulative reward.

[0148] The technical features in the above embodiments can be flexibly combined as needed. To maintain brevity, not all possible combinations of the technical features are described in detail in the above embodiments. However, any combination of these technical features that does not contradict each other in practical application should be considered as included within the scope of this specification.

[0149] It is important to note that this does not imply any limitation on the scope of the invention patent. As understood by those skilled in the art, modifications and improvements can still be made to the above embodiments without departing from the inventive concept, and all such modifications and improvements fall within the protection scope of this invention. Therefore, the protection scope of this invention patent should be determined by the scope defined in the appended claims.

Claims

1. A method for intelligent scheduling of shared storage and charging systems in industrial and commercial parks based on reinforcement learning, characterized in that... The specific steps are as follows: (1) By modeling various loads in industrial and commercial parks that include shared photovoltaic storage and charging systems, the energy balance of the shared photovoltaic storage and charging system is controlled to achieve intelligent scheduling and optimized management of energy, with a focus on the electricity demand of park users, the charging demand of electric vehicles and the supply of photovoltaic power generation. The modeling involves establishing the interaction relationships between energy storage devices, electric vehicles, and other loads. In this modeling, energy storage devices can function as both emergency power sources and charging equipment during peak electricity price periods, reducing reliance on the distribution network. Loads in various industrial and commercial parks include: Energy storage devices are used to store electrical energy that cannot be consumed immediately within industrial and commercial parks. The mathematical model of the energy storage device is shown in formula (1): ； in: Indicates time The status of energy storage devices For charging and discharging power, For time step; The capacity range of energy storage devices is subject to the following constraints: ； ； ； ； ； in: This is the lower limit of the equipment's discharge power. This is the device's maximum storage capacity. It represents the charging and discharging state of the storage device at time t. It is the upper limit of charging power. and The equipment is in and The charging and discharging state at any given moment. It is the upper limit coefficient of charging power. It represents the charging state of electric vehicles; Formulas (2) to (6) represent the charging and discharging states and capacity limits of energy storage devices in different time periods; The charging schedule for electric vehicles varies depending on the charging mode. For example, in a home charging mode, the vehicle is charged when the owner returns home from get off work and stops charging when leaving for work. The start time of electric vehicle charging follows a normal distribution, with the following probability density function: （7）； Where: parameters and Let represent the standard deviation and mean of the normal distribution controlling the charging start time of electric vehicles, respectively. , According to formula (7), the timetable for starting to charge electric vehicles is obtained, and then the owners are charged according to the corresponding charging mode. The capacity of electric vehicles in industrial and commercial parks follows a uniform distribution, as shown in formula (8); where 20 − 60kJ represents the common capacity range of electric vehicles on the market; the probability that electric vehicles in industrial and commercial parks have a capacity within this range is uniformly distributed: （8）； The charging formulas for electric vehicles and energy storage devices are both based on the charging process of lithium-ion batteries; the charging process is shown in formulas (9) and (10): ； （10）； in: It refers to the charging power of electric vehicles. The battery capacity of electric vehicles is determined by a uniform distribution equation that defines the capacity of trolleybuses in industrial and commercial parks. Ensure that the electric vehicle is charged to the minimum SOC required for operation, and prevent the electric vehicle battery from being deeply discharged during the discharge process, thereby extending the life of the electric vehicle battery; 𝑆𝑂𝐶𝑚𝑎𝑥 corresponds to the maximum SOC that the electric vehicle can reach, and avoids overcharging the electric vehicle; Electric vehicles can be charged in two ways: fast charging and smart charging. Fast charging is suitable for emergency charging needs and can fully charge the battery in a short time. Smart charging is suitable for users who are not in a hurry to charge. It has a lower charging power and is dynamically adjusted according to the overall load of industrial and commercial parks and electricity price fluctuations to reduce charging costs and avoid excessive grid load. The charging power is limited as follows: （11）； in: This refers to the charging power of electric vehicles. In fast charging mode, the charging power is equal to the maximum allowable power of the charging piles in the industrial and commercial park, which is 20kWh. In smart charging mode, the charging power is between 0 and the maximum power, which varies with the overall load curve of the industrial and commercial park and the fluctuation of real-time electricity prices. This achieves two purposes: first, the charging of electric vehicles in the industrial and commercial park will not be too high during peak hours, thus putting pressure on the power distribution network of the industrial and commercial park; second, the charging demand will be met by avoiding peak electricity price periods, thereby reducing the charging costs for car owners. (2) Establish a model of a shared photovoltaic energy storage and charging system, analyze its charging and discharging characteristics, and calculate the degradation cost of the battery; The industrial and commercial park is equipped with a shared photovoltaic, energy storage and charging system. The shared photovoltaic, energy storage and charging system needs to prioritize the load within the industrial and commercial park, i.e. the peak load period, in order to effectively reduce the dependence of the industrial and commercial park load on the distribution network. Therefore, all types of loads within the industrial and commercial park need to meet the power balance constraints. (3) By designing a reward function, the reinforcement learning algorithm is guided to make the optimal charging and discharging decisions at different times to maximize the overall benefits of the industrial and commercial park. In this process, the reinforcement learning agent continuously selects actions based on the current state, executes the corresponding charging and discharging strategies, and optimizes the charging and discharging decisions through feedback reward signals. The shared photovoltaic energy storage and charging system gradually finds the optimal scheduling method through multiple iterations to ensure the balance between power supply and demand, reduce energy waste, and extend battery life. The reward function in the reinforcement learning plays a core role in guiding agent decision-making, determining how the agent evaluates its behavior and guiding it toward the target behavior; by using reinforcement learning to optimize the charging and discharging decisions of the energy storage system in industrial and commercial parks, the energy storage system should bear a portion of the load pressure during peak load periods.

2. The method according to claim 1, characterized in that: The formula for the energy balance constraint in step (2) is as follows: （12）； in: The total electricity demand of the industrial and commercial park consists of two parts, the first part being... The first part is the park's basic load, including daily electricity demand; the second part is... This refers to the charging load of electric vehicles in industrial and commercial parks. The size of this load is determined by the penetration rate of electric vehicles within the industrial and commercial parks; that is, the charging power of a single electric vehicle is... The total fleet power of the industrial and commercial park depends on how many electric vehicles are equipped in the park; the charging load of electric vehicles is not included in the base load and needs to be independently optimized and scheduled.

3. The method according to claim 1, characterized in that: Step (3) employs a penalty function to reduce the occurrence of behaviors by the constraint agent that do not conform to the objective; the penalty function is defined as: （13）； in: This refers to the energy waste generated at that time step, specifically the surplus electrical energy that photovoltaic power generation fails to fully utilize, or the energy discharged by the energy storage system that exceeds the demand of the industrial and commercial park. The energy output of the photovoltaic power generation system needs to meet the electricity demand of the industrial and commercial park while minimizing waste; the energy balance formula for the photovoltaic system is as follows: （14）； in: This refers to the electrical energy generated by the photovoltaic system at every moment, specifically the portion of the photovoltaic power generated that is used by users in industrial and commercial parks. The portion of the electricity generated by photovoltaic power that is stored by energy storage is labeled as... If there is any remaining photovoltaic power, it will be marked as... This indicates that this portion of electrical energy is being wasted; The electricity demand of users in industrial and commercial parks also needs to meet power balance constraints to ensure a reasonable distribution of load among various energy inputs. The energy balance formula for users is as follows: （15）； This refers to the total electricity demand of industrial and commercial parks, a portion of which comes from the electricity generated by photovoltaic power generation and is used by users within the industrial and commercial parks. Part of it comes from the discharge of the energy storage system, that is... Another part relies on the power grid, namely... If the discharge capacity of the energy storage system far exceeds the load demand of the industrial and commercial park, it will be marked as... This indicates that this portion of energy is wasted; The charging power of an energy storage system comes from electricity purchased from the grid and electricity generated by a photovoltaic system; the charging power formula for an energy storage system is as follows: （16） in: For charging power from the grid, The portion of the electricity generated by photovoltaic power that is stored by energy storage; （17）； Energy waste generated throughout the industrial and commercial park The source of photovoltaic power generation is not fully utilized, resulting in a waste of photovoltaic power. Furthermore, the discharge of energy storage far exceeds the demand of industrial and commercial parks, resulting in a waste of energy storage power. This waste of electricity not only affects the power supply efficiency of industrial and commercial parks, but also increases the energy consumption burden of the system. Therefore, it is necessary to impose penalties in reinforcement learning to reduce unnecessary waste.

4. The method according to claim 1, characterized in that: The reward function described in step (3) ensures that the shared photovoltaic energy storage and charging system can handle the load of the industrial and commercial park by balancing the power supply of the park. The electricity cost of industrial and commercial parks needs to take into account the cost of purchasing electricity from the grid and energy storage systems, which is calculated as shown in formula (18): （18）； in This is the electricity cost for the entire industrial and commercial park. This is the actual electricity price, based on local electricity price standards. This refers to the portion of electricity generated by photovoltaic power that is used by users in industrial and commercial parks. The energy storage system purchases electricity from the grid; the cost of electricity is closely related to the fluctuations in the load of industrial and commercial parks and the electricity price of the grid. Therefore, adopting optimized electricity purchase methods can significantly reduce the electricity costs of industrial and commercial parks. The charging behavior of an energy storage system not only involves the storage of electrical energy, but also brings cost losses and battery wear; the cost losses caused by the charging behavior of an energy storage system are calculated as shown in formula (19): （19）； in: It is the unit energy cost of the battery, which is the initial cost or replacement cost of the battery; It is the depth of discharge of the battery, indicating the degree of discharge; Battery cycle life refers to the number of charge-discharge cycles a battery can undergo under specific depth of discharge and conditions. Battery wear and tear costs are related to energy costs, efficiency, depth of discharge, cycle life, and charge / discharge power. By rationally managing charge and discharge strategies, battery life can be effectively extended and wear and tear costs reduced. The reward function for industrial and commercial parks is given by formula (20): （20）； in: It is an industrial and commercial park at all times The reward; It refers to the electricity costs of industrial and commercial parks, with the goal of reducing electricity costs and thus reducing the dependence of industrial and commercial parks on the power grid; It is the wear and tear cost of the energy storage system, taking into account the long-term health of the energy storage system and optimizing the usage of the battery. It is a penalty function that reduces energy waste and ensures efficient system operation; Through the reward function, the shared photovoltaic-storage-charging system self-adjusts its charging and discharging strategies during reinforcement learning, which not only optimizes economic costs but also extends the lifespan of the energy storage system and reduces energy waste. This enables effective management of the load in industrial and commercial parks and reduces dependence on the power distribution network.

5. The method according to claim 1, characterized in that: The reinforcement learning algorithm described in step (3) is used to control the charging and discharging decisions of the energy storage system in the industrial and commercial park, so that the system can form the optimal usage strategy according to different time conditions, thereby maximizing the long-term reward of the industrial and commercial park. The core task of reinforcement learning is to enable the agent, which is the dispatch controller of the energy storage system in the industrial and commercial park in this scenario, to learn to maximize its long-term returns through a series of actions in the environment. In this application scenario, the agent's "actions" refer to decisions on the charging and discharging levels of the energy storage system, and the goal is to maximize the reward function of the industrial and commercial park through these decisions, so as to ensure that the load demand of the industrial and commercial park is met while reducing electricity costs, extending the life of the energy storage system, and reducing energy waste. In the reinforcement learning framework, the state of the environment is a random variable at each step; in this case, the state of the environment is determined by the current charging state of the energy storage system, load demand, and photovoltaic power generation. Assume that at a certain moment, the agent is in state... Take action Then transition to the next state. And receive a reward The state transition process is then represented as: （21）； in: It is a state transition function, which represents the change in the state of the environment after taking a certain action in a certain state; The agent's goal is to maximize its cumulative reward by selecting an appropriate sequence of actions; this is achieved through a value function; given the current moment... Cumulative returns It is the weighted sum of all rewards from the moment the action begins until the moment it ends: （22）； in: It is a discount factor, at 0 Between; the discount factor controls the weight of future rewards, the smaller the value, the smaller the impact of future rewards on the current decision; while the reward function in formula (22) yes A simplified representation of this is that the impact of the current decision on the agent is evaluated through a reward function; Strategy Define the probability distribution of the agent's actions in each state; it can be deterministic or random. （23）； The random policy is given by formula (23), which indicates that in state... At that time, take action The probability of; in practical applications, stochastic strategies help explore different decision paths, thereby finding the optimal strategy; The reinforcement learning method employs Q-learning, which is based on value iteration and approximates the optimal policy by continuously updating the action-value function. In the Q-learning method, for each pair of states... and actions Each state has a corresponding Q-value, representing the expected reward of taking that action in that state; the update rule of the Q-learning method is as follows: （24）； in: It is the learning rate, which controls the step size of each update; In the new state Next, take all actions. The Q value corresponding to the best action in the process; By continuously iterating and updating the Q-value, the Q-learning method can gradually find the optimal strategy, that is, to take the best action in each state to maximize the cumulative reward.

Images