Optical storage charging station coordinated operation optimization method

Abstract

The invention relates to an optical storage charging station coordinated operation optimization method, and belongs to the technical field of charging station optimized operation. According to the technical scheme, the method comprises the following steps: performing refined mathematical modeling on an EPE coordinated optimization operation problem; describing an EPE coordinated optimization operation model through a Markov process; the SAC algorithm is improved based on a conservative Q network; performing hyper-parameter optimization by using a parallel Bayesian algorithm to obtain an optimal hyper-parameter group, and obtaining a PBO-SAC-CQL algorithm for training of EV-PV-ESS coordinated optimization operation; and training the model, providing test data based on photovoltaic output and short-term prediction of arrival of the electric vehicle, and obtaining an EV-PV-ESS coordinated optimization operation strategy. The method has the beneficial effects that an optimal strategy can be quickly converged, a more suitable hyper-parameter is obtained in a high-dimensional hyper-parameter space, the system is endowed with stronger autonomous performance, and the decision speed can be greatly improved.

Description

Technical Field

[0001] The present invention relates to a method for optimizing the coordinated operation of a photovoltaic storage charging station, and belongs to the technical field of optimized operation of charging stations. Background Art

[0002] Solar-energy-storage charging stations are comprehensive energy systems that integrate photovoltaic (PV) power generation, energy storage systems (ESS), and electric vehicle (EV) charging. With the increasing popularity of electric vehicles and the rapid development of renewable energy, solar-energy-storage charging stations have become crucial infrastructure for balancing energy supply and demand and improving energy efficiency. However, the operation of solar-energy-storage charging stations involves multiple complex factors, including the uncertainty of photovoltaic power generation, the randomness of EV charging demand, and the charge and discharge management of energy storage systems. These factors make the coordinated and optimized operation of solar-energy-storage charging stations a challenging problem.

[0003] Compared to traditional model-based approaches, reinforcement learning (RL) learns optimal strategies through the interaction between an agent and its environment. It offers advantages such as adaptability, flexibility, and efficiency, effectively addressing the uncertainty of photovoltaic power generation and the randomness of electric vehicle charging demand. However, traditional RL algorithms still face two challenges in the coordinated optimization of PV-storage charging stations: in complex environments, they are prone to overoptimistic Q-value estimation, leading to suboptimal action selection; and whether RL algorithms achieve a balance between exploration and exploitation often depends on the choice of hyperparameters. Inappropriate hyperparameters can cause the strategy to fall into a local optimum. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for optimizing the coordinated operation of a photovoltaic charging station. Through PBO parameter optimization combined with the SAC-CQL solution algorithm, it can quickly converge to the optimal strategy. The proposed parallel parameter optimization method can significantly improve the optimization efficiency and obtain more suitable hyperparameters in the high-dimensional hyperparameter space. The introduction of the parameter optimization method gives the system stronger autonomy, and the PBO-SAC-CQL algorithm can greatly improve the decision-making speed, which has significant advantages over traditional methods and effectively solves the above-mentioned problems existing in the background technology.

[0005] The technical solution of the present invention is: a method for optimizing the coordinated operation of a photovoltaic charging station, comprising the following steps:

[0006] (1) Mathematical modeling of the EPE (EV-PV-ESS) coordinated optimization operation problem of photovoltaic charging stations;

[0007] (2) Describing the EPE coordinated optimization operation model through Markov process;

[0008] (3) The traditional SAC algorithm (Soft Actor–Critic) is improved based on the conservative Q network, combining the efficient exploration capability of SAC with the conservatism of CQL to avoid the strategy from falling into overly optimistic Q-value estimation;

[0009] (4) Using the Parallel Bayesian Optimization (PBO) algorithm to optimize the hyperparameters of the proposed SAC-CQL algorithm, the optimal hyperparameter set is obtained, and the PBO-SAC-CQL algorithm is obtained for training the coordinated optimization operation of EV-PV-ESS.

[0010] (5) The model is trained using actual historical data, and the short-term prediction of PV output and EV arrival is used to provide test data to obtain the EV-PV-ESS coordinated optimization operation strategy.

[0011] In step (1), the objective function of the photovoltaic charging station operation optimization is described as:

[0012]

[0013] Where: C GRID,t F is the cost of purchasing electricity from the grid for the photovoltaic storage charging station during period t; GRID,t C is the revenue of the solar-storage charging station selling electricity to the grid during period t; ESS,t is the energy storage mileage cost in the solar-storage charging station during period t, which is used to measure the loss and aging of equipment; F EV,t is the charging revenue of the solar-storage charging station for electric vehicles during period t; T is the number of periods in the set cycle;

[0014] The various costs and benefits are expressed as follows:

[0015] C GRID,t =λ c,t P c,t Δt (2)

[0016] F GRID,t =λ f,t P f,t Δt (3)

[0017] C ESS,t =L mil |P ESS,t -P ESS,t-1 | (4)

[0018] F EV,t =λ EV,t P EV,t Δt (5)

[0019] Where: P c,t and P f,tare the power purchased and sold by the photovoltaic charging station from the grid during period t; P ESS,t P is the energy storage operating power of the solar-storage charging station during period t; EV,t is the total charging power demanded by electric vehicles in the solar-storage charging station during period t; c,t and λ f,t are the prices of electricity purchased and sold from the grid by the photovoltaic storage charging station during period t, EV,t L is the price of charging electric vehicles from the photovoltaic charging station during period t; mil is the mileage cost coefficient of the solar-storage charging station, and Δt is the unit time duration;

[0020] Considering power balance, energy storage constraints, and charging constraints, the constraints for optimizing the operation of a solar-storage charging station are described as follows:

[0021] P ESS,t =P DIS,t -P CH,t (6)

[0022] P PV,t +P ESS,t +P c,t =P EV,t +P f,t (7)

[0023]

[0024] E min ≤E t ≤E max (9)

[0025]

[0026] Where: P DIS,t and P CH,t are the discharging and charging powers of the solar-storage charging station during period t; E t is the energy stored in the energy storage system at time t; E min and E max are the minimum and maximum capacities of the energy storage system respectively; η CH and η DIS are the charging and discharging efficiencies of the energy storage system, respectively; and are the maximum charge and discharge power of the energy storage system respectively; and are the maximum power purchase and sales of the photovoltaic storage charging station system; P EV,t,i is the charging power of electric vehicle i at time t; 0,i and T i The charging start time and the charging end time are respectively; R i is the total demand for electric vehicles i; is the maximum charging power of electric vehicle i.

[0027] In step (2), according to the current state s t Output the optimal action a t , the reward r of the action returned by the environment t and the new state s t+1 According to the characteristics of the EPE coordinated optimization operation problem of the photovoltaic charging station, the corresponding state space, action space and reward function are designed.

[0028] 1) State Space

[0029] Status t It is composed of all the features that reflect the current and future conditions. In the EPE coordinated optimization operation problem, the state space S t It consists of the charging price, purchase price, sales price, photovoltaic output, charging power, power demand and current time of the photovoltaic charging station, and is specifically expressed as follows:

[0030]

[0031] 2) Action Space

[0032] Set the energy storage output and electric vehicle charging power as action space A t , the action space range is limited by the constraint condition formula (5), as shown in formula (20),

[0033] A t ={P EV,t,i ,P CH,t ,P DIS,t} (18)

[0034] 3) Reward Function

[0035] The reward function defines the value of taking an action in a specific state and is the core mechanism for guiding the learning of the intelligent agent. It helps the intelligent agent optimize its decision-making strategy by providing immediate feedback on its behavior.

[0036] R t =F GRID,t +F EV,t -C ESS,t -C GRID,t (19).

[0037] In step (3), SAC uses two Q networks, Critic, and a policy network, Actor, where Critic is used to estimate the Q value and Actor is used to generate actions.

[0038] The loss function of Critic is:

[0039] L(Q)=E (s,a,r,s′)～D [(Q(s,a)-(r+γ(V(s′)))) 2 ] (20)

[0040] Where V(s′)=E a′～π [Q(s′,a′)-αlogπ(a′|s′)], α is the entropy regularization coefficient;

[0041] The loss function of Actor is:

[0042] L(π)=E s～D,a～π [αlogπ(a|s)-Q(s,a)] (21)

[0043] In addition, SAC balances exploration and utilization by adjusting the entropy regularization coefficient α, and the α loss function is:

[0044] L(α)=E s～D,a～π [-αlogπ(a|s)-αH0] (22)

[0045] Where H0 is the target entropy;

[0046] The conservatism of CQL is introduced on the basis of SAC to form the SAC-CQL algorithm, whose objective function is:

[0047]

[0048] L(Q) is the traditional Bellman error term, which is the Critic's loss function; κ is the weight of the conservative regularization term, which is used to control the strength of conservatism. It is an exponentially weighted sum of the Q values ​​of all actions and then takes the logarithm to punish overestimated Q values. a～π [Q(s,a)] is the expectation of the Q value under the current strategy, which is used to balance conservatism.

[0049] The specific implementation steps of step (4) are as follows:

[0050] 1) Gaussian process modeling

[0051] Assume that the objective function f(x) obeys a Gaussian process

[0052] f(x)~GP(m(x),k(x,x′))(24)

[0053] Where: m(x) is the mean function, set to a constant; k(x,x′) is the covariance function; x=[x1,x2,…,x d ] is a set of hyperparameters, also called input points or candidate points; the covariance function is used to measure the similarity between two input points x, x′:

[0054]

[0055] Where: x i is the value of the i-th hyperparameter; d is the dimension of the hyperparameter, is the signal variance, which controls the overall amplitude of the function; is the length scale, which controls the decay rate of the similarity of the input points in this dimension;

[0056] 2) Parallel acquisition function

[0057] Improved acquisition function using batch upper confidence bounds:

[0058]

[0059] Where: μ(x i ) and σ(x i ) are Gaussian processes at x i The mean and standard deviation of , q is the number of candidate points in each iteration; β is a trade-off parameter that controls the balance between exploration and utilization;

[0060] 3) Parallel evaluation

[0061] In each iteration, the objective function values ​​of q candidate points {y1,…,y q}, and add these data points to the training set D

[0062] D←D∪{(x1,y1),…,(x q ,y q )}(27)

[0063] 4) Update the Gaussian process model

[0064] Update the posterior distribution of the Gaussian process model using the new data points:

[0065] f(x)|D~N(μ(x),σ 2 (x))(28)

[0066] Where: mean function μ(x) and variance function σ 2 (x) is calculated according to the posterior formula of the Gaussian process.

[0067] The beneficial effects of the present invention are as follows: through PBO parameter optimization combined with the SAC-CQL solution algorithm, it is possible to quickly converge to the optimal strategy; the proposed parallel parameter optimization method can significantly improve the optimization efficiency and obtain more suitable hyperparameters in the high-dimensional hyperparameter space; the introduction of the parameter optimization method gives the system stronger autonomous performance, and the PBO-SAC-CQL algorithm can greatly improve the decision-making speed, which has significant advantages over traditional methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solutions and advantages of the invention implementation cases clearer, the technical solutions in the invention implementation cases will be clearly and completely described below in conjunction with the drawings in the implementation cases. Obviously, the implementation cases described are only a small part of the implementation cases of the present invention, rather than all the implementation cases. Based on the implementation cases in the present invention, all other implementation cases obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0070] A method for optimizing the coordinated operation of a photovoltaic charging station comprises the following steps:

[0071] (1) Mathematical modeling of the EPE coordinated optimization operation problem of the photovoltaic storage charging station;

[0072] (2) Describing the EPE coordinated optimization operation model through Markov process;

[0073] (3) Improve the traditional SAC algorithm based on the conservative Q network, combining the efficient exploration ability of SAC with the conservatism of CQL to avoid the strategy from falling into overly optimistic Q-value estimation;

[0074] (4) The proposed SAC-CQL algorithm is optimized for hyperparameters using the parallel Bayesian algorithm to obtain the optimal hyperparameter set, and the PBO-SAC-CQL algorithm is obtained for training the coordinated optimization operation of EV-PV-ESS;

[0075] (5) The model is trained using actual historical data, and the short-term prediction of PV output and EV arrival is used to provide test data to obtain the EV-PV-ESS coordinated optimization operation strategy.

[0076] In step (1), the objective function of the photovoltaic charging station operation optimization is described as:

[0077]

[0078] Where: C GRID,t F is the cost of purchasing electricity from the grid for the photovoltaic storage charging station during period t; GRID,t C is the revenue of the solar-storage charging station selling electricity to the grid during period t; ESS,t is the energy storage mileage cost in the solar-storage charging station during period t, which is used to measure the loss and aging of equipment; F EV,t is the charging revenue of the solar-storage charging station for electric vehicles during period t; T is the number of periods in the set cycle;

[0079] The various costs and benefits are expressed as follows:

[0080] C GRID,t =λ c,t P c,t Δt (2)

[0081] F GRID,t =λ f,t P f,t Δt (3)

[0082] C ESS,t =L mil |P ESS,t -P ESS,t-1 | (4)

[0083] F EV,t =λ EV,t P EV,t Δt (5)

[0084] Where: P c,t and P f,t are the power purchased and sold by the photovoltaic charging station from the grid during period t; P ESS,t P is the energy storage operating power of the solar-storage charging station during period t; EV,t is the total charging power demanded by electric vehicles in the solar-storage charging station during period t; c,t and λ f,t are the prices of electricity purchased and sold from the grid by the photovoltaic storage charging station during period t, EV,t L is the price of charging electric vehicles from the photovoltaic charging station during period t; mil is the mileage cost coefficient of the solar-storage charging station, and Δt is the unit time duration;

[0085] Considering power balance, energy storage constraints, and charging constraints, the constraints for optimizing the operation of a solar-storage charging station are described as follows:

[0086] P ESS,t =P DIS,t -P CH,t (6)

[0087] P PV,t +P ESS,t +P c,t =P EV,t +P f,t (7)

[0088]

[0089] E min ≤E t ≤E max (9)

[0090]

[0091]

[0092] Where: P DIS,t and P CH,t are the discharging and charging powers of the solar-storage charging station during period t; E t is the energy stored in the energy storage system at time t; E min and E max are the minimum and maximum capacities of the energy storage system respectively; η CH and η DIS are the charging and discharging efficiencies of the energy storage system, respectively; and are the maximum charge and discharge power of the energy storage system respectively; and are the maximum power purchase and sales of the photovoltaic storage charging station system; P EV,t,i is the charging power of electric vehicle i at time t; 0,i and T i The charging start time and the charging end time are respectively; R i is the total demand for electric vehicles i; is the maximum charging power of electric vehicle i.

[0093] In step (2), according to the current state s t Output the optimal action a t , the reward r of the action returned by the environment t and the new state s t+1 According to the characteristics of the EPE coordinated optimization operation problem of the photovoltaic charging station, the corresponding state space, action space and reward function are designed.

[0094] 1) State Space

[0095] Status t It is composed of all the features that reflect the current and future conditions. In the EPE coordinated optimization operation problem, the state space S t It consists of the charging price, purchase price, sales price, photovoltaic output, charging power, power demand and current time of the photovoltaic charging station, and is specifically expressed as follows:

[0096]

[0097] 2) Action Space

[0098] Set the energy storage output and electric vehicle charging power as action space A t , the action space range is limited by the constraint condition formula (5), as shown in formula (20),

[0099] A t ={PEV,t,i ,P CH,t ,P DIS,t} (18)

[0100] 3) Reward Function

[0101] The reward function defines the value of taking an action in a specific state and is the core mechanism for guiding the learning of the intelligent agent. It helps the intelligent agent optimize its decision-making strategy by providing immediate feedback on its behavior.

[0102] R t =F GRID,t +F EV,t -C ESS,t -C GRID,t (19).

[0103] In step (3), SAC uses two Q networks, Critic, and a policy network, Actor, where Critic is used to estimate the Q value and Actor is used to generate actions.

[0104] The loss function of Critic is:

[0105] L(Q)=E (s,a,r,s′)～D [(Q(s,a)-(r+γ(V(s′)))) 2 ] (20)

[0106] Where V(s′)=E a′～π [Q(s′,a′)-αlogπ(a′|s′)], α is the entropy regularization coefficient;

[0107] The loss function of Actor is:

[0108] L(π)=E s～D,a～π [αlogπ(a|s)-Q(s,a)] (21)

[0109] In addition, SAC balances exploration and utilization by adjusting the entropy regularization coefficient α, and the α loss function is:

[0110] L(α)=E s～D,a～π [-αlogπ(a|s)-αH0] (22)

[0111] Where H0 is the target entropy;

[0112] The conservatism of CQL is introduced on the basis of SAC to form the SAC-CQL algorithm, whose objective function is:

[0113]

[0114] L(Q) is the traditional Bellman error term, which is the Critic's loss function; κ is the weight of the conservative regularization term, which is used to control the strength of conservatism. It is an exponentially weighted sum of the Q values ​​of all actions and then takes the logarithm to punish overestimated Q values. a～π [Q(s,a)] is the expectation of the Q value under the current strategy, which is used to balance conservatism.

[0115] The specific implementation steps of step (4) are as follows:

[0116] 1) Gaussian process modeling

[0117] Assume that the objective function f(x) obeys a Gaussian process

[0118] f(x)~GP(m(x),k(x,x′)) (24)

[0119] Where: m(x) is the mean function, set to a constant; k(x,x′) is the covariance function; x=[x1,x2,…,x d ] is a set of hyperparameters, also called input points or candidate points; the covariance function is used to measure the similarity between two input points x, x′:

[0120]

[0121] Where: x i is the value of the i-th hyperparameter; d is the dimension of the hyperparameter, is the signal variance, which controls the overall amplitude of the function; is the length scale, which controls the decay rate of the similarity of the input points in this dimension;

[0122] 2) Parallel acquisition function

[0123] Improved acquisition function using batch upper confidence bounds:

[0124]

[0125] Where: μ(x i ) and σ(x i ) are Gaussian processes at x i The mean and standard deviation of , q is the number of candidate points in each iteration; β is a trade-off parameter that controls the balance between exploration and utilization;

[0126] 3) Parallel evaluation

[0127] In each iteration, the objective function values ​​of q candidate points {y1,…,y q}, and add these data points to the training set D

[0128] D←D∪{(x1,y1),…,(x q ,y q )}(27)

[0129] 4) Update the Gaussian process model

[0130] Update the posterior distribution of the Gaussian process model using the new data points:

[0131] f(x)|D~N(μ(x),σ 2 (x))(28)

[0132] Where: mean function μ(x) and variance function σ 2 (x) is calculated according to the posterior formula of the Gaussian process.

[0133] In practical application, the present invention includes steps 1 to 4:

[0134] Step 1: mathematically model the coordinated optimization operation problem of electric vehicle-photovoltaic-energy storage (EV-PV-ESS, EPE) at the photovoltaic charging station.

[0135] Solar-energy storage charging stations can generate economic benefits by charging electric vehicles and trading with the power grid. The optimization goal is to maximize the economic benefits of solar-energy storage charging stations, that is, to minimize the total cost. Therefore, the objective function for the optimization of solar-energy storage charging stations can be described as:

[0136]

[0137] Where: C GRID,t F is the cost of purchasing electricity from the grid for the photovoltaic storage charging station during period t; GRID,t C is the revenue of the solar-storage charging station selling electricity to the grid during period t; ESS,t is the energy storage mileage cost in the solar-storage charging station during period t, which is used to measure the loss and aging of equipment; F EV,t is the charging revenue of the solar-storage charging station for electric vehicles in period t; T is the number of periods in the set cycle.

[0138] The various costs and benefits can be expressed as:

[0139] C GRID,t =λ c,t P c,t Δt (2)

[0140] F GRID,t =λ f,t P f,t Δt (3)

[0141] C ESS,t =L mil |P ESS,t -P ESS,t-1| (4)

[0142] F EV,t =λ EV,t P EV,t Δt (5)

[0143] Where: P c,t 、P f,t are the power of electricity purchased and sold by the photovoltaic charging station from the grid during period t; P ESS,t P is the energy storage operating power of the solar-storage charging station during period t; EV,t is the total charging power demanded by electric vehicles in the solar-storage charging station during period t; c,t ,λ f,t are the prices of electricity purchased and sold from the grid by the photovoltaic charging station during period t, EV,t L is the price of charging electric vehicles from the photovoltaic charging station during period t; mil is the mileage cost coefficient of the solar-storage charging station, and Δt is the unit time duration.

[0144] Taking into account power balance, energy storage constraints, and charging constraints, the constraints for optimizing the operation of a solar-storage charging station can be described as follows:

[0145] P ESS,t =P DIS,t -P CH,t (6)

[0146] P PV,t +P ESS,t +P c,t =P EV,t +P f,t (7)

[0147]

[0148] E min ≤E t ≤E max (9)

[0149]

[0150] Where: P DIS,t 、P CH,t are the discharging and charging powers of the solar-storage charging station during period t; E t is the energy stored in the energy storage system at time t; E min 、E max are the minimum and maximum capacities of the energy storage system respectively; η CH ,η DIS are the charging and discharging efficiency of the energy storage system; are the maximum charging and discharging power of the energy storage system respectively; are the maximum power purchase and sales of the photovoltaic storage charging station system; P EV,t,i is the charging power of electric vehicle i at time t; 0,i 、T i The charging start time and the charging end time are respectively; R i is the total demand for electric vehicles i; is the maximum charging power of electric vehicle i.

[0151] Step 2: Describe the EPE coordinated optimization operation model through Markov process. The reinforcement learning algorithm training process can be regarded as a trial-and-error process by interacting with the training environment representing the physical system. The agent is based on the current state s t Output the optimal action a t , the environment returns the reward r for the action t and the new state s t+1 According to the characteristics of the coordinated optimization operation problem of electric vehicle-photovoltaic-energy storage (EV-PV-ESS, EPE) in a photovoltaic charging station, the corresponding state space, action space and reward function are designed.

[0152] 1) State Space

[0153] Status t It consists of all the features that reflect the current and future conditions. In the EPE coordinated optimization operation problem, the state space S t It consists of the charging price of the photovoltaic storage charging station, the purchase price of electricity, the sales price of electricity, the photovoltaic output, the charging power, the power demand, and the current time, and is specifically expressed as follows:

[0154]

[0155] 2) Action Space

[0156] Set the energy storage output and electric vehicle charging power as action space A t , the action space range is limited by the constraint condition formula (5), as shown in formula (20).

[0157] A t ={P EV,t,i ,P CH,t ,P DIS,t} (18)

[0158] 3) Reward Function

[0159] The reward function defines the value of taking an action in a specific state and is the core mechanism for guiding the agent's learning. It helps the agent optimize its decision-making strategy by providing immediate feedback on its behavior.

[0160] R t =F GRID,t +FEV,t -C ESS,t -C GRID,t (19)

[0161] Step 3: Improve the traditional SAC algorithm based on the conservative Q network, combining the efficient exploration ability of SAC and the conservatism of CQL to avoid the strategy from falling into overly optimistic Q-value estimation.

[0162] SAC (Soft Actor-Critic) is an algorithm based on maximum entropy reinforcement learning that aims to maximize the entropy of a policy while optimizing cumulative rewards. Combining the actor-critic framework with entropy regularization, SAC can prevent the policy from falling into local optima while maintaining efficient exploration. SAC uses two Q-networks (critic) and a policy network (actor), where the critic is used to estimate Q-values ​​and the actor is used to generate actions.

[0163] The loss function of Critic is:

[0164] L(Q)=E (s,a,r,s′)～D [(Q(s,a)-(r+γ(V(s′)))) 2 ] (20)

[0165] Where V(s′)=E a′～π [Q(s′,a′)-αlogπ(a′|s′)], α is the entropy regularization coefficient.

[0166] The loss function of Actor is:

[0167] L(π)=E s～D,a～π [αlogπ(a|s)-Q(s,a)] (21)

[0168] In addition, SAC balances exploration and utilization by adjusting the entropy regularization coefficient α. The α loss function is:

[0169] L(α)=E s～D,a～π [-αlogπ(a|s)-αH0] (22)

[0170] Where H0 is the target entropy.

[0171] However, traditional SAC algorithms can suffer from overly optimistic Q-value estimates, leading to suboptimal strategies during training. Especially given the high uncertainty of photovoltaic output, overly optimistic Q-value estimates can lead strategies to choose risky actions. The conservative nature of CQL can prevent this. To address this issue, we propose an improved approach that incorporates the conservative nature of CQL (Conservative Q-Learning) on ​​top of SAC, resulting in the SAC-CQL algorithm.

[0172] Conservative Q-learning (CQL) is an algorithm designed to address the problem of overoptimistic Q-value estimates in reinforcement learning. In many reinforcement learning tasks, Q-value estimates can deviate from the true values ​​due to function approximation errors, sampling bias, or environmental uncertainty. This can cause the policy to overestimate the Q-values ​​of certain actions during training, leading to suboptimal solutions. CQL introduces conservative constraints to make Q-value estimates more conservative, thus avoiding this overoptimistic problem.

[0173] The core of CQL is to introduce a conservative regularization term into the traditional Q-learning objective function. The traditional Q-learning objective is to minimize the Bellman error. CQL adds a conservative regularization term to this objective function, and its objective function is:

[0174]

[0175] L(Q) is the traditional Bellman error term, which is also the critic's loss function. κ is the weight of the conservative regularization term, which is used to control the strength of conservatism. It is an exponentially weighted sum of the Q values ​​of all actions and then takes the logarithm to penalize overestimated Q values. a～π [Q(s,a)] is the expectation of the Q value under the current strategy, which is used to balance conservatism.

[0176] Step 4: Use the parallel Bayesian algorithm to perform hyperparameter optimization on the proposed SAC-CQL algorithm to obtain the optimal hyperparameter group, and obtain the PBO-SAC-CQL algorithm for training the coordinated optimization operation of EV-PV-ESS. Bayesian optimization is a sequential optimization method based on a probability model, which is suitable for scenarios where the objective function is expensive to calculate and difficult to derive. The core idea is to construct a probabilistic model of the objective function (usually using a Gaussian process) and select the hyperparameter combination that is most likely to improve the objective function for evaluation in each iteration, thereby gradually approaching the optimal solution. The core of the PBO-SAC-CQL algorithm is to use parallel Bayesian optimization to optimize the hyperparameters of the SAC-CQL algorithm to obtain the optimal hyperparameter combination. The specific implementation steps are as follows:

[0177] 1) Gaussian process modeling

[0178] Assume that the objective function f(x) obeys a Gaussian process

[0179] f(x)~GP(m(x),k(x,x′))(24)

[0180] Where: m(x) is the mean function, set to a constant; k(x,x′) is the covariance function; x=[x1,x2,…,x d] is a set of hyperparameters, also called input points or candidate points. The covariance function is used to measure the similarity between two input points x and x′:

[0181]

[0182] Where: x i is the value of the i-th hyperparameter; d is the dimension of the hyperparameter, is the signal variance, which controls the overall amplitude of the function; is the length scale, which controls the decay rate of the similarity of input points in this dimension.

[0183] 2) Parallel acquisition function

[0184] The traditional acquisition function can only select one candidate point. In order to achieve parallelization, the acquisition function improved by batch upper confidence bound can be used:

[0185]

[0186] Where: μ(x i ),σ(x i ) are Gaussian processes at x i The mean and standard deviation of , q is the number of candidate points in each iteration; β is a trade-off parameter that controls the balance between exploration and utilization.

[0187] 3) Parallel evaluation

[0188] In each iteration, the objective function values ​​of q candidate points {y1,…,y q}, and add these data points to the training set D

[0189] D←D∪{(x1,y1),…,(x q ,y q )} (27)

[0190] 4) Update the Gaussian process model

[0191] Update the posterior distribution of the Gaussian process model using the new data points:

[0192] f(x)|D~N(μ(x),σ 2 (x)) (28)

[0193] Where: mean function μ(x) and variance function σ 2 (x) is calculated according to the posterior formula of the Gaussian process.

[0194] The advantages of the present invention are as follows:

[0195] 1) Characterize the dynamic characteristics of electric vehicles, photovoltaics and energy storage systems in photovoltaic charging stations, and combine the Markov decision process to describe the system state transition and optimization objectives, which can accurately capture the randomness and uncertainty in complex environments.

[0196] 2) An improved SAC algorithm based on CQL is adopted, combining the efficient exploration capability of SAC with the conservative constraints of CQL to effectively avoid the problem of over-optimism in Q-value estimation, improve the stability and robustness of the strategy, and ensure that the algorithm can converge to a better solution in complex environments.

[0197] 3) PBO is used to optimize the hyperparameters of the SAC-CQL algorithm. Through efficient probability models and acquisition functions, the optimal hyperparameter combination is found within a relatively small number of iterations, significantly improving the algorithm performance while reducing the cost of manual parameter adjustment and achieving automated tuning.

[0198] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for optimizing the coordinated operation of a photovoltaic charging station, characterized in that The following steps are involved: (1) Mathematical modeling of the EPE coordinated optimization operation problem of the photovoltaic storage charging station; (2) Describing the EPE coordinated optimization operation model through Markov process; (3) Improve the traditional SAC algorithm based on the conservative Q network, combining the efficient exploration ability of SAC with the conservatism of CQL to avoid the strategy from falling into overly optimistic Q-value estimation; (4) The proposed SAC-CQL algorithm is optimized for hyperparameters using the parallel Bayesian algorithm to obtain the optimal hyperparameter set, and the PBO-SAC-CQL algorithm is obtained for training the coordinated optimization operation of EV-PV-ESS; (5) The model is trained using actual historical data, and the short-term prediction of PV output and EV arrival is used to provide test data to obtain the EV-PV-ESS coordinated optimization operation strategy.

2. The method for optimizing coordinated operation of a photovoltaic charging station according to claim 1, characterized in that: In step (1), the objective function of the photovoltaic charging station operation optimization is described as: Where: C GRID,t F is the cost of purchasing electricity from the grid for the photovoltaic storage charging station during period t; GRID,t C is the revenue of the solar-storage charging station selling electricity to the grid during period t; ESS,t is the energy storage mileage cost in the solar-storage charging station during period t, which is used to measure the loss and aging of equipment; F EV,t is the charging revenue of the solar-storage charging station for electric vehicles during period t; T is the number of periods in the set cycle; The various costs and benefits are expressed as follows: C GRID,t =λ c,t P c,t Δt (2) F GRID,t =λ f,t P f,t Δt (3) C ESS,t =L mil |P ESS,t -P ESS,t-1 | (4) F EV,t =λ EV,t P EV,t Δt (5) Where: P c,t and P f,t are the power purchased and sold by the photovoltaic charging station from the grid during period t; P ESS,t P is the energy storage operating power of the solar-storage charging station during period t; EV,t is the total charging power demanded by electric vehicles in the solar-storage charging station during period t; c,t and λ f,t are the prices of electricity purchased and sold from the grid by the photovoltaic storage charging station during period t, EV,t L is the price of charging electric vehicles from the photovoltaic charging station during period t; mil is the mileage cost coefficient of the solar-storage charging station, and Δt is the unit time duration; Considering power balance, energy storage constraints, and charging constraints, the constraints for optimizing the operation of a solar-storage charging station are described as follows: P ESS,t =P DIS,t -P CH,t (6) P PV,t +P ESS,t +P c,t =P EV,t +P f,t (7) AND min ≤E t ≤E max (9) Where: P DIS,t and P CH,t are the discharging and charging powers of the solar-storage charging station during period t; E t is the energy stored in the energy storage system at time t; E min and E max are the minimum and maximum capacities of the energy storage system, respectively; η CH and η DIS are the charging and discharging efficiencies of the energy storage system, respectively; and are the maximum charge and discharge power of the energy storage system respectively; and are the maximum power purchase and sales of the photovoltaic storage charging station system; P EV,t,i is the charging power of electric vehicle i at time t; 0,i and T i The charging start time and the charging end time are respectively; R i is the total demand for electric vehicles i; is the maximum charging power of electric vehicle i.

3. The method for optimizing coordinated operation of a photovoltaic charging station according to claim 1, characterized in that: In step (2), according to the current state s t Output the optimal action a t , the reward r of the action returned by the environment t and the new state s t+1 , according to the characteristics of the EPE coordinated optimization operation problem of the photovoltaic storage charging station, the corresponding state space, action space and reward function are designed; 1) State Space Status t It is composed of all the features that reflect the current and future conditions. In the EPE coordinated optimization operation problem, the state space S t It consists of the charging price, purchase price, sales price, photovoltaic output, charging power, power demand and current time of the photovoltaic charging station, and is specifically expressed as follows: 2) Action Space Set the energy storage output and electric vehicle charging power as action space A t , the action space range is limited by the constraint condition formula (5), as shown in formula (20), A t ={P EV,t,i ,P CH,t ,P DIS,t } (18) 3) Reward Function The reward function defines the value of taking an action in a specific state and is the core mechanism for guiding the learning of the intelligent agent. It helps the intelligent agent optimize its decision-making strategy by providing immediate feedback on its behavior. R t =F GRID,t +F EV,t -C ESS,t -C GRID,t (19)。 4. The method for optimizing the coordinated operation of a photovoltaic charging station according to claim 1, characterized in that: In step (3), SAC uses two Q networks, Critic, and a policy network, Actor, where Critic is used to estimate the Q value and Actor is used to generate actions. The loss function of Critic is: L(Q)=E (s,a,r,s′)～D [(Q(s,a)-(r+γ(V(s′)))) 2 ] (20) Where V(s′)=E a′～π [Q(s′,a′)-αlogπ(a′|s′)], α is the entropy regularization coefficient; The loss function of Actor is: L(π)=E s～D,a～π [αlogπ(a∣s)−Q(s,a)] (21) In addition, SAC balances exploration and utilization by adjusting the entropy regularization coefficient α, and the α loss function is: L(a)=E s～D,a～π [-αlogπ(a∣s)-αH0] (22) Where H0 is the target entropy; The conservatism of CQL is introduced on the basis of SAC to form the SAC-CQL algorithm, whose objective function is: L(Q) is the traditional Bellman error term, which is also the Critic's loss function; κ is the weight of the conservative regularization term, which is used to control the strength of conservatism; It is an exponentially weighted sum of the Q values ​​of all actions and then takes the logarithm to punish overestimated Q values. a～π [Q(s,a)] is the expectation of the Q value under the current strategy, which is used to balance conservatism.

5. The method for optimizing coordinated operation of a photovoltaic charging station according to claim 1, characterized in that: The specific implementation steps of step (4) are as follows: 1) Gaussian process modeling Assume that the objective function f(x) obeys a Gaussian process f(x)~GP(m(x),k(x,x′))(24) Where: m(x) is the mean function, set to a constant; k(x,x′) is the covariance function; x=[x1,x2,…,x d ] is a set of hyperparameters, also called input points or candidate points; the covariance function is used to measure the similarity between two input points x, x′: Where: x i is the value of the i-th hyperparameter; d is the dimension of the hyperparameter, is the signal variance, which controls the overall amplitude of the function; is the length scale, which controls the decay rate of the similarity of the input points in this dimension; 2) Parallel acquisition function Improved acquisition function using batch upper confidence bounds: Where: μ(x i ) and σ(x i ) are Gaussian processes at x i The mean and standard deviation of , q is the number of candidate points in each iteration; β is a trade-off parameter that controls the balance between exploration and utilization; 3) Parallel evaluation In each iteration, the objective function values ​​of q candidate points {y1,...,y q }, and add these data points to the training set D D←D∪{(x1,y1),...,(x q ,and q )} (27) 4) Update the Gaussian process model Update the posterior distribution of the Gaussian process model using the new data points: f(x)∣D~N(μ(x),σ 2 (x))(28) Where: mean function μ(x) and variance function σ 2 (x) is calculated according to the posterior formula of Gaussian process.

Images