Photovoltaic-energy storage-charging multi-stage scheduling and market bidding optimization method and device

Abstract

The invention discloses a photovoltaic-energy storage-charging multi-stage scheduling and market bidding optimization method and a photovoltaic-energy storage-charging multi-stage scheduling and market bidding optimization device. The method comprises the following steps: constructing a data-driven random environment model reflecting photovoltaic output, electricity price fluctuation and charging load uncertainty by adopting a mode of combining time sequence clustering and a non-homogeneous Markov chain based on historical operation data; modeling a scheduling and bidding problem of the optical storage and charging integrated station into a multi-stage Markov decision process model which comprises day-ahead decision and joint optimization of multiple intra-day rolling adjustment; a deep reinforcement learning algorithm is utilized to train the network, and a strategy regulation and control network which can adapt to various uncertain scenes and meet equipment physical constraints is obtained; and deploying the trained strategy regulation and control network in an energy management system to realize global coordinated scheduling and bidding of the optical storage and charging integrated station. According to the method, the economic benefit is remarkably improved, the robustness is greatly enhanced, the decision is globally coordinated and optimized, the real-time decision capability is strong, and the expandability and portability are good.

Description

Technical Field

[0001] This invention relates to the field of smart grid and energy system optimization control technology, specifically to a multi-stage scheduling and market bidding optimization method and apparatus for photovoltaic-energy storage-charging. Background Technology

[0002] According to the latest statistics from the International Energy Agency (IEA), the global cumulative installed capacity of photovoltaic power generation has exceeded 1 TW, and the number of electric vehicles has surpassed 26 million, with a compound annual growth rate of over 40%. Against this backdrop, integrated photovoltaic (PV), energy storage (ESS), and electric vehicle charging facilities have emerged as important components of distributed energy systems.

[0003] As a new type of integrated energy system, the photovoltaic-storage-charging station has significant technical and economic advantages: (1) the photovoltaic system provides clean and renewable electricity, effectively reducing carbon emission intensity; (2) the energy storage system, through energy time-shifting function, smooths out the intermittency and fluctuation of photovoltaic output, achieving peak shaving and valley filling; (3) the electric vehicle charging load, as a flexible and controllable mobile energy storage unit, can participate in demand-side response; (4) the entire station, through unified coordination and control, can maximize the local photovoltaic consumption rate and reduce grid transmission losses. According to literature statistics, a reasonably configured photovoltaic-storage-charging station can increase the photovoltaic utilization rate to over 90%, improve the overall system energy efficiency by 15-25%, and reduce operating costs by 20-30%.

[0004] With the deepening of power market reform, my country has established a multi-timescale spot market system encompassing day-ahead, intraday, and real-time markets, as well as a variety of ancillary service markets such as frequency regulation, peak shaving, and reserve. In this market environment, integrated photovoltaic-storage-charging stations, as emerging market players, face complex decision-making optimization problems: they need to determine the baseline operating plan for the entire day during the day-ahead phase (including grid-connected electricity, energy storage charging and discharging plans, and ancillary service capacity applications), and dynamically adjust strategies based on real-time information at multiple intraday phases to maximize overall daily revenue.

[0005] Existing technologies mainly include the following types: I. Deterministic Optimization Methods Deterministic optimization methods are based on deterministic forecasting (such as photovoltaic power output forecasting, load forecasting, and electricity price forecasting). They establish mixed-integer linear programming (MILP) or nonlinear programming (NLP) models and use commercial solvers for solving. A typical model's objective function is in the form of: (1); Wherein, Π is the total profit, λt is the electricity price for time period t, Pgrid,t is the power exchanged with the grid (positive value is electricity sales, negative value is electricity purchase), Cop,t is the operation and maintenance cost, Ras,t is the revenue from ancillary services, and Δt is the length of time period. The advantages of this method are that the model is clear and the solution speed is fast, but it has significant defects: (1) It depends entirely on the prediction accuracy. When the actual operation deviates from the prediction, the optimization results may become seriously invalid; (2) It cannot handle prediction uncertainty and lacks robustness to extreme weather; (3) The day-ahead and intraday decision-making are separated, and the linkage effect between stages is not considered.

[0006] II. Stochastic Optimization and Robust Optimization Methods To overcome the shortcomings of deterministic methods, researchers have introduced stochastic programming and robust optimization theory. Stochastic programming methods generate multiple uncertain scenarios to construct two-stage or multi-stage stochastic programming models: (2); Where x is the decision variable for the first stage (daily plan), ys is the decision variable for the second stage under scenario s (intraday adjustment), ξs is the realized value of the uncertain parameters in scenario s, and Es represents the expected value over all scenarios. The robust optimization method uses an uncertainty set to describe the range of parameter fluctuations. min max f(x, ξ)st ξ ∈ U (3); Where U represents the set of uncertainties. However, both types of methods still have the following limitations: scenario dependence, temporal coupling complexity, model assumption limitations, and computational efficiency bottlenecks. The dynamic evolution equation of the state of charge (SOC) of an energy storage system is: SOC t+1 = SOC t + (η c · P c,t - P d,t / η d ) · Δt / E rated(4); Where ηc and ηd are the charge / discharge efficiencies, Pc,t and Pd,t are the charge / discharge power, and Erated is the rated capacity. This results in strong coupling of decision-making across different time periods.

[0007] Third, preliminary research based on reinforcement learning In recent years, some researchers have attempted to apply reinforcement learning methods to energy system optimization. In particular, reinforcement learning methods have shown unique advantages in handling sequential decision-making and uncertainty. However, existing RL-based research still has significant shortcomings: limited application scenarios, insufficient constraint handling, simplistic environment modeling, and challenges in algorithm selection and parameter tuning.

[0008] Based on the above analysis, a new technical solution is urgently needed to address the scheduling and bidding optimization problem of integrated photovoltaic, energy storage, and charging stations in a multi-stage power market environment. This solution should organically integrate the adaptive learning capabilities of deep reinforcement learning with domain knowledge of power system optimization, overcoming the limitations of traditional optimization methods in terms of prediction accuracy and scenario assumptions, while also compensating for the shortcomings of existing RL methods in constraint handling and environmental modeling. Summary of the Invention

[0009] To address the aforementioned issues, this invention provides a multi-stage scheduling and market bidding optimization method and apparatus for photovoltaic-energy storage-charging. The aim is to deeply integrate deep reinforcement learning algorithms with knowledge from the electricity market domain, constructing a Markov decision process model for day-ahead and intraday joint decision-making. Through extensive interactive training between the agent and the environment, the optimal scheduling and bidding strategies under uncertain conditions are learned, thus overcoming or at least partially overcoming the shortcomings of existing technologies.

[0010] The embodiments of the present invention adopt the following technical solutions: In a first aspect, the present invention provides a multi-stage scheduling and market bidding optimization method for photovoltaic-energy storage-charging, including: A data-driven stochastic environment model is constructed based on historical operating data to reflect the uncertainty of photovoltaic output, electricity price fluctuations and charging load. The data-driven stochastic environment model is a Monte Carlo stochastic scenario constructed by combining time series clustering and non-homogeneous Markov chains. The scheduling and bidding problem of integrated photovoltaic, energy storage and charging stations is modeled as a multi-stage Markov decision process model that includes day-ahead decision and multiple intraday rolling adjustments. The multi-stage Markov decision process model is trained using a deep reinforcement learning algorithm to obtain a policy control network that can adapt to various uncertain scenarios and meet the physical constraints of the equipment. The trained strategy control network is deployed in the energy management system to realize the global coordinated scheduling and bidding action commands of the photovoltaic-storage-charging integrated station.

[0011] Secondly, the present invention also provides a multi-stage scheduling and market bidding optimization device for photovoltaic-energy storage-charging, comprising: The scenario construction unit is used to construct a data-driven stochastic environment model based on historical operating data, which reflects the uncertainty of photovoltaic output, electricity price fluctuations and charging load. The data-driven stochastic environment model is a Monte Carlo stochastic scenario constructed by combining time series clustering and non-homogeneous Markov chains. The decision model building unit is used to model the scheduling and bidding problem of the integrated photovoltaic-storage-charging station into a multi-stage Markov decision process model that includes day-ahead decision and multiple intraday rolling adjustments. The training unit is used to train the multi-stage Markov decision process model using a deep reinforcement learning algorithm to obtain a policy control network that can adapt to various uncertain scenarios and meet the physical constraints of the equipment. The deployment unit is used to deploy the trained strategy control network into the energy management system to realize the global coordinated scheduling and bidding action commands of the photovoltaic-storage-charging integrated station.

[0012] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described photovoltaic-energy storage-charging multi-stage scheduling and market bidding optimization method.

[0013] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program that, when instructed by a processor, implements the steps of the above-described photovoltaic-energy storage-charging multi-stage scheduling and market bidding optimization method.

[0014] The above-described at least one technical solution adopted in the embodiments of the present invention can achieve the following beneficial effects: First, economic benefits are significantly improved. The integrated energy station can autonomously learn bidding and scheduling strategies that approximate the optimal level, thereby maximizing operational revenue. Simulation results show that compared with traditional two-stage optimization, applying deep RL can increase the total profit of the station by 15-20% and significantly reduce losses such as default penalties. In the simulation example, the daily total profit achieved by this scheme is close to the theoretical optimum, with an optimization gap of less than 1%, demonstrating excellent optimization performance.

[0015] Second, robustness is significantly enhanced. Because a large number of random scenarios are generated using Markov chains and fully explored during training, the strategy obtained by this method exhibits strong adaptability and robustness to various weather changes, price fluctuations, and uncertainties in charging demand. This means that in actual operation, even if the environment deviates from certain predictions, this method can still adjust the strategy to maintain a near-optimal return level, reducing reliance on accurate predictions.

[0016] Third, the invention achieves global coordination and optimization in decision-making. Through multi-stage joint optimization and RL policy training, it ensures consistent goals and coordination between day-ahead and multiple intraday decisions, avoiding suboptimal results that may occur in traditional methods where each stage operates independently. When market conditions change, the strategy is dynamically adjusted during intraday stages, but always with the goal of achieving global optimum, thus balancing long-term benefits with short-term flexibility.

[0017] Fourth, it boasts strong real-time decision-making capabilities. Once the deep reinforcement learning strategy is trained, its computation is highly efficient during actual deployment, providing optimization decisions for the current state in just milliseconds. This allows energy stations to easily handle frequent intraday market clearing cycles, achieving near real-time intelligent control. Traditional mathematical programming-based methods may struggle to solve complex problems in a timely manner, especially when frequent rolling operations are required, making the computational speed advantage of this invention even more pronounced.

[0018] Fifth, it exhibits good scalability and portability. This method demonstrates excellent scalability, allowing for the smooth introduction of other distributed energy sources or loads into the state space, in addition to photovoltaics, energy storage, and electric vehicles. The agent can automatically adapt to the decisions of newly added modules through training. Furthermore, since the strategy is presented in the form of a neural network, it is also easy to port to embedded energy management systems, enabling rapid deployment of intelligent control modules. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1 A flowchart illustrating a multi-stage scheduling and market bidding optimization method for photovoltaic-energy storage-charging according to an embodiment of the present invention is shown. Figure 2 A schematic diagram of a multi-stage scheduling and market bidding optimization device for photovoltaic-energy storage-charging according to an embodiment of the present invention is shown; Figure 3 A schematic diagram of an electronic device according to an embodiment of the present invention is shown. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0021] The purpose of this invention is to provide a multi-stage scheduling and market bidding optimization method for photovoltaic-energy storage-charging. This method can be used for integrated energy stations that combine photovoltaic power generation, electrochemical energy storage systems and electric vehicle charging facilities, in an electricity trading environment that includes day-ahead market and multiple intraday markets, to achieve autonomous optimization decision-making for multi-stage joint scheduling and market bidding strategies.

[0022] The technical solution of this invention revolves around a core approach integrating Markov uncertainty modeling, multi-stage MDP optimization, and PPO deep reinforcement learning solution. The overall architecture is divided into an offline training phase and an online deployment phase: the offline training phase constructs an uncertainty environment simulator based on historical data and trains the optimal policy network using deep reinforcement learning algorithms; the online deployment phase embeds the trained policy network into the energy management system, outputting scheduling decisions in real time. The technical details are elaborated below.

[0023] Figure 1 This diagram illustrates a flowchart of a multi-stage scheduling and market bidding optimization method for photovoltaic-energy storage-charging according to an embodiment of the present invention. Figure 1 It can be seen that, from Figure 1 As can be seen, this embodiment includes steps S110 to S120: Step S110: Construct a data-driven stochastic environment model based on historical operating data, reflecting the uncertainty of photovoltaic output, electricity price fluctuations and charging load. The data-driven stochastic environment model is a Monte Carlo stochastic scenario constructed by combining time series clustering and non-homogeneous Markov chains.

[0024] Uncertainty modeling is a crucial foundation of this invention. Unlike traditional scene generation methods based on parametric distribution assumptions, this invention proposes a data-driven, non-parametric method that automatically extracts the temporal evolution patterns of uncertainty from historical data through a combination of time series clustering and Markov chains. This method eliminates the need for manually specifying the probability distribution type and can adaptively learn the joint distribution of multiple variables and temporal correlations.

[0025] Specifically, in some embodiments of the present invention, the data-driven stochastic environment model is constructed using a combination of time series clustering and non-homogeneous Markov chains, including: Step S111: Collect historical operational data from multiple dimensions and preprocess it to obtain multidimensional time series variables.

[0026] First, historical operational data is collected.

[0027] The following multi-source heterogeneous data were collected from the historical operation database of the energy station and the power grid dispatch system. The heterogeneity of each dimension is arranged in chronological order to form time-series variables, which may include, but are not limited to, photovoltaic power, time-of-use electricity price, charging load and meteorological factors.

[0028] Photovoltaic power output data (PV power): such as PV power generation (kW) every 15 minutes, with a time span of ≥1 year and a total sample size of ≥35,040 data points. The data is preferably selected to cover all four seasons and different weather types (sunny, cloudy, rainy, snowy) to ensure scenario diversity.

[0029] Electricity price data (time-of-use pricing): such as the day-ahead market time-of-use pricing and the intraday real-time market pricing (RMB / kWh), with a time resolution of 15 minutes. Electricity price data needs to distinguish between peak, mid-peak, flat, and off-peak periods, and include abnormal market events (such as price spikes caused by tight supply and demand, transmission congestion, etc.).

[0030] Charging load data (charging load): such as electric vehicle arrival time, departure time, initial SOC, target SOC, charging power demand, etc., forming a charging event sequence. The data needs to cover weekdays and weekends, holidays and non-holidays, and differences in arrival distribution at different times.

[0031] Auxiliary meteorological data (meteorological factors): ambient temperature (°C), humidity (%), wind speed (m / s), global horizontal irradiance (W / m²), etc., are used to analyze the influencing factors of photovoltaic power output and improve the accuracy of forecasts.

[0032] Then, the historical raw data for each dimension are preprocessed, and the preprocessed data are recorded as time-series variables for each dimension. In some embodiments of the present invention, preprocessing includes, but is not limited to: outlier detection and cleaning, missing value imputation, and data normalization.

[0033] Outlier detection and cleaning: Outliers are detected using the 3σ criterion and box plot method. For photovoltaic power output, if the data Pt for a certain period satisfies |Pt - μ|>3σ (where μ is the historical mean for the same period and σ is the standard deviation), it is marked as an outlier. Data that is continuously missing due to sensor failure or communication interruption, or that clearly violates physical laws (such as negative power or exceeding rated capacity), is discarded.

[0034] Missing value imputation: For short-term missing values ​​(<30 minutes), linear interpolation is used: Pt = Pt-1 + (Pt+k -Pt-1) / (k+1), where k is the number of missing time periods. For long-term missing values ​​(≥30 minutes), imputation is based on the similarity day method: K neighboring days (K=5) with similar weather conditions and date types are found from historical data, and the median of their corresponding time periods is taken as the imputation value.

[0035] Data normalization: Mapping variables to the [0,1] interval facilitates clustering and training. The normalization formula is: (5); in, x min and x `max` represents the minimum and maximum values ​​of the variable in historical data, respectively. The normalization parameters need to be saved for data preprocessing and inverse normalization during online deployment.

[0036] Step S112: Extract contemporaneous feature vectors from the multidimensional time series variables according to time periods.

[0037] By time alignment and feature construction, contemporaneous feature vectors are extracted from the time-series variables of the above dimensions according to time periods.

[0038] To ensure consistent timestamps (UTC or local time) across all data sources, a uniform 15-minute time resolution can be used. Construct a multidimensional feature vector: xt = [PPV,t, λt, DEV,t, Tt, It, ht, dt]T (6); Where PPV,t represents photovoltaic output, λt represents electricity price, DEV,t represents charging demand, Tt represents temperature, It represents irradiance, ht represents hourly code (values ​​range from 0 to 23), and dt represents weekday code (values ​​range from 1 to 7). These temporal characteristics help to capture periodic patterns.

[0039] Step S113: The continuous state space is discretized into a finite number of state clusters using the density clustering algorithm DBSCAN.

[0040] To construct a Markov chain model, continuous uncertain variables need to be discretized into a finite set of states. In some embodiments of the present invention, the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) density clustering algorithm is used. Its core advantages are: (1) automatically determining the number of clusters without pre-specifying the K value; (2) discovering clusters of arbitrary shapes to adapt to the non-convexity of actual data distribution; and (3) identifying noise points to improve clustering quality and robustness.

[0041] The following is a brief explanation of the DBSCAN algorithm principle. DBSCAN defines clustering based on density reachability, and its key concepts include: ε-neighborhood: The ε-neighborhood of a point p is defined as Nε(p) = {q ∈ D | dist(p,q) ≤ ε}, which is the set of all points whose distance from p does not exceed ε. The distance metric used is Euclidean distance.

[0042] Core point: If the ε-neighborhood of point p contains at least MinPts points, i.e., |Nε(p)| ≥ MinPts, then p is a core point. A core point represents the center of a dense region. Core(p) |Nε(p)| ≥ MinPts(7).

[0043] Boundary point: If a point q is not a core point, but lies within the ε-neighborhood of some core point p, then q is a boundary point. Boundary points belong to a cluster but are located at the edge of the cluster.

[0044] Noise points: Points that are neither core points nor boundary points are considered noise points, representing outliers or abnormal data.

[0045] Density accessibility: If p is a core point and q ∈ Nε(p), then q is said to be density accessible from p.

[0046] Density reachability: If there exists a chain of points p1, p2, ..., pn such that pi+1 is density-reachable from pi (i=1,...,n-1), then pn is density-reachable from p1. Points belonging to the same cluster are density-reachable from each other.

[0047] The following describes the process of discretizing a continuous state space into a finite number of state clusters using the density clustering algorithm DBSCAN. In some embodiments of the present invention, this process includes: Step S113-1: For each scheduling period, construct a normalized multidimensional feature vector set based on historical operation data. The feature vector includes at least photovoltaic power generation output, grid electricity price and electric vehicle charging load, and may optionally include meteorological and time coding information.

[0048] For each time period t (e.g., 8:00-8:15), extract the normalized feature vector set Xt = {xt(d) | d=1,...,D} for all historical days corresponding to the time period, where D is the number of historical days (usually D≥365). Each vector contains key variables such as photovoltaic output, electricity price, and charging demand. Optionally, meteorological and temporal features can be added to improve clustering accuracy.

[0049] Step S113-2: The DBSCAN density clustering algorithm is used to cluster the feature vector sets of each time period. The neighborhood radius ε is determined by the inflection point of the k-distance curve, and the minimum number of points MinPts is set to the feature dimension plus 1.

[0050] DBSCAN's performance depends on the selection of ε and MinPts. This invention employs the following heuristic method for hyperparameter selection: MinPts setting: As a rule of thumb, MinPts = dim + 1 (dim is the feature dimension). For a 7-dimensional feature vector, MinPts = 8. Alternatively, MinPts = 2 × dim can be used as a slightly more lenient choice.

[0051] ε selection: Calculate the distance from each point to its MinPts nearest neighbor and plot a distance-number curve (k-distance graph). The distance value corresponding to the inflection point (elbow) on the curve is the appropriate ε. This method can adapt to the data density distribution, with a typical value range of ε ∈ [0.1, 0.3] (after normalization).

[0052] Step S113-3: Based on the clustering results, divide the typical operating modes, assign a unique state number to each valid cluster, and calculate its cluster center and covariance matrix for subsequent scene generation.

[0053] The DBSCAN algorithm is run on the aforementioned feature vector set Xt. The pseudocode is as follows: Algorithm 1: DBSCAN Clustering Input: Dataset Xt, parameters ε, MinPts Output: Cluster labels 1: Initialize all points as UNVISITED, cluster number C = 0 2: for each point p ∈ Xt do 3: if p has been visited then continue 4: Mark p as visited 5: Nε = {q ∈ Xt | dist(p,q) ≤ ε} / / Find the ε-neighborhood 6:if |Nε| <MinPts then 7: Mark p as a noise point 8:else 9: C = C + 1 / / Create a new cluster 10:ExpandCluster(p, Nε, C, ε, MinPts) 11: end for 12: return labels The function ExpandCluster(p, Nε, C, ε, MinPts): 1: Add p to cluster C 2: for each point q ∈ Nε do 3: if q is not visited then 4: Mark q as visited 5:Nε' = {r ∈ Xt | dist(q,r) ≤ ε} 6: if |Nε'| ≥ MinPts then 7: Nε = Nε ∪ Nε' / / Extended neighborhood 8: if q does not belong to any cluster then 9: Add q to cluster C After clustering, the following clustering results are obtained: Kt valid clusters Ct,1, Ct,2, ..., Ct,Kt (excluding noise points). A unique state number st,k ∈ {1,2,...,Kt} is assigned to each cluster. The center vector (mean of all points within the cluster) and variance matrix of each cluster are calculated and saved for subsequent scene generation. (8).

[0054] For noisy points, the following options are available: (1) remove them from subsequent modeling; (2) assign them to the nearest cluster; (3) group them separately as low-probability events. This invention recommends strategy (1) to maintain the simplicity of the Markov chain.

[0055] Step S113-4: Use the silhouette coefficient to evaluate the clustering quality. If the average value is lower than the preset threshold, adjust the parameters and re-cluster to ensure the rationality and discriminability of the state division.

[0056] To ensure the accuracy of the results, cluster quality can also be assessed using the silhouette coefficient. For a point x∈Ct,k, the silhouette coefficient is defined as: s(x) = (b(x) - a(x)) / max{a(x), b(x)} (9); Where a(x) is the average distance between x and other points in the same cluster (intra-cluster distance), and b(x) is the average distance between x and all points in the nearest neighbor cluster (minimum inter-cluster distance); the silhouette coefficient s(x)∈[-1,1], the closer the value is to 1, the better the clustering quality; the global average silhouette coefficient should be ≥0.5 (preset threshold), otherwise ε or MinPts needs to be adjusted and re-clustered.

[0057] Step S114: Estimate the transition probability matrix corresponding to each time period based on the historical state transition frequency, and introduce Laplace smoothing to enhance the model's generalization ability.

[0058] Before constructing the non-homogeneous Markov chain model, the transition probability matrix is ​​estimated, specifically including steps S114-1 to S114-3.

[0059] Step S114-1: Count the frequency of historical state transitions.

[0060] Based on the discrete state sequence obtained after clustering, for each time period t, we iterate through all historical day data and count the number of times a state transitions from state i to state j in time period t+1. ; and the total number of occurrences of state i in time period t. .

[0061] For each time period t∈[1,95] (a total of 96 time periods, only 95 transition matrices are needed), the frequency of historical state transitions is counted based on the historical state sequence.

[0062] Step S114-2: Based on the historical state transition frequencies obtained from statistics, the original transition probability is calculated using the maximum likelihood method.

[0063] Based on the following formula, the transition probability matrix is ​​calculated using maximum likelihood estimation: Pt(i,j) = (St+1 = j | St = i) = Nt(i→j) / Nt(i)(10); Where Nt(i→j) represents the number of times time period t transitions from state i to state j in time period t+1 in the historical data, and Nt(i) represents the total number of times time period t is in state i. Clearly, Σj Pt(i,j) = 1, satisfying the probability normalization condition.

[0064] Step 114-3: Apply Laplace's algorithm to smooth the transition probability matrix.

[0065] Furthermore, Laplace smoothing is introduced to enhance the model's generalization ability.

[0066] To avoid the zero-probability problem (where some transitions did not appear in historical data but could actually occur), Laplace smoothing (additive smoothing) is used: Pt(i,j) = (Nt(i→j) + α) / (Nt(i) + α·Kt+1)(11); Where α is the smoothing parameter (usually taken as α = 0.1-1.0), and Kt+1 is the number of states in time interval t+1. After smoothing, even if Nt(i→j) = 0, Pt(i,j) is a small positive number α / (Nt(i) + α·Kt+1), ensuring the traversal of the chain.

[0067] Step S115: Construct a non-homogeneous Markov chain model. Its initial distribution is obtained from the state frequency statistics of the first time period and is used to describe the joint dynamic characteristics of uncertainty evolution over time.

[0068] Based on the discrete state sequences obtained from clustering, a non-homogeneous Markov chain (NHMC) is constructed to describe the temporal evolution of uncertainty. Non-homogeneity refers to the change of the transition probability matrix over time, i.e., Pt(i,j) ≠ Pt'(i,j) for t≠t', to reflect different transition patterns at different time periods.

[0069] The construction of a non-homogeneous Markov chain specifically includes steps S115-1 to S115-3.

[0070] Step S115-1: Construct a first-order Markov chain.

[0071] The Markov property assumptions are as follows: The state at time t+1 depends only on the state at time t and is independent of earlier times. The following is the expression for a first-order Markov property chain: (12); This assumption holds true in most cases because the 15-minute granularity of the state already contains sufficient historical information. If longer-term dependencies need to be considered, it can be extended to a higher-order Markov chain (such as a second-order chain: St+1 depends on St and St-1), but this will significantly increase the model complexity.

[0072] Step S115-2: Construct an estimate of the initial state distribution of the first-order Markov chain.

[0073] This invention uses the above-mentioned first-order Markov property chain and initializes it accordingly.

[0074] The initial state distribution is estimated by statistically analyzing the state distribution during the first time interval (0:00-0:15) as the initial distribution of the Markov chain: (13); Where N1(i) is the number of days in state i during the first period of historical data, and D is the total number of days. The initial distribution can be regarded as the zero-order transition probability, reflecting the typical state distribution at the beginning of a day (usually, photovoltaic output is zero, electricity price is low, and charging demand is low).

[0075] Step S115-3: Store the smoothed transition probability matrix Pt corresponding to each time period as model parameters, and combine it with the initial state distribution of the first-order Markov chain to form a non-homogeneous Markov chain.

[0076] Continuing from the previous step, the smoothed transition probability matrix Pt corresponding to each time period is stored as model parameters, which, together with the initial state distribution, constitute a non-homogeneous Markov chain: (14).

[0077] Non-homogeneous Markov chains are used for subsequent Monte Carlo scene generation and reinforcement learning training.

[0078] To ensure the accuracy of the constructed non-homogeneous Markov chain model, any one or a combination of the following methods can be used for verification.

[0079] Steady-state distribution test method: For a homogeneous Markov chain, there exists a steady-state distribution π satisfying πP=π. Although this model is a non-homogeneous chain, the long-term average transition matrix can be tested. The steady-state distribution is compared with the frequency distribution of historical states.

[0080] Reverse generation verification method: Use the constructed Markov chain to generate N=1000 scene sequences, calculate the statistical characteristics (mean, variance, autocorrelation coefficient) of the generated data, and compare it with historical data. The relative error should be <10%.

[0081] Chi-square test method: Perform a chi-square test on the transition frequency for each time period t, testing the hypothesis H0: the actual transition follows model Pt. If the p-value > 0.05, then accept H0, and consider the model to be well fitted.

[0082] Step S116: Generate uncertain Monte Carlo random scenarios based on the trained non-homogeneous Markov chain as a data-driven random environment model.

[0083] Monte Carlo scene generation is based on a trained non-homogeneous Markov chain model to generate multivariate uncertainty scene sequences, specifically including steps S116-1 to S116-4: Step S116-1: For a daily operation scenario, determine the discrete state of each scenario in the first time period by randomly sampling according to the initial state distribution.

[0084] Step S116-2: In chronological order, use the transition probability matrix P corresponding to each time period. t Based on the current state, the next state is sampled step by step to construct a complete state transition path for 96 time periods.

[0085] Step S116-3: After obtaining the entire discrete state sequence, the state number of each time period is mapped to the center vector of its corresponding cluster, and a Gaussian perturbation that conforms to the statistical characteristics of the cluster is superimposed to generate continuous feature values.

[0086] Step S116-4: Truncate the obtained values ​​in the range of [0,1] and convert them into actual physical quantities through inverse normalization. The actual physical quantities include photovoltaic output, electricity price and charging demand.

[0087] The above process is repeated to generate N independent daily operating scenarios, which constitute a scenario set for reinforcement learning training.

[0088] Specifically, based on the trained Markov chain model = {π1, P1, P2, ..., P95}, using the Monte Carlo method to generate a large number of random scene sequences for reinforcement learning training and policy evaluation. The core idea of ​​scene generation is: sampling the initial state from the initial distribution, then progressively sampling subsequent states according to the time-varying transition probability matrix, and finally mapping the discrete state sequence back to continuous values. The following is a specific algorithm for generating random scene sequences based on Markov chains: Algorithm 2: Monte Carlo Scene Generation Input: Markov chain model Cluster parameters {μt,k, Σt,k}, number of scenes N Output: Scene set = {ξ (1) , ξ (2) , ..., ξ ( )} 1: for n = 1 to N do 2: / / Initial sampling state 3:S1 (n) ~ Categorical(π1) 4: / / Time-by-time sampling state sequence 5: for t = 1 to 95 do 6:i = S t (n) / / Current status number 7:S t+1 (n) ~ Categorical(P t (i, :)) / / Sample the next state based on the transition probability 8:end for 9: / / State sequence to continuous value 10: for t = 1 to 96 do 11:k = S t (n) / / Status number for the current time period 12:ε t ~ (0, Σt,k) / / Sample perturbation from cluster variance matrix 13:x t (n) = μt,k + ε t / / Add perturbation to cluster center 14:x t (n) = clip(x t (n) , 0, 1) / / Truncate to the range [0,1] 15:end for 16:ξ (n) = {x1 (n) , x2 (n) , ..., x 96 (n) / / This constitutes a complete scene 17: end for 18: return .

[0089] The key steps of the above algorithm are explained below: Line 3: Perform classification sampling based on the initial distribution π1. Implementation method: Generate uniform random numbers u ~ U(0,1), and find the values ​​that satisfy Σi'=1. i If π1(i') ≥ the smallest i of u, then S1 = i.

[0090] Line 7: Perform classification sampling based on the current state i and the i-th row of the transition matrix Pt to determine the next state. This step reflects the Markov property, that is, the sampling of St+1 depends only on St.

[0091] Lines 12-13: To increase scene diversity, Gaussian noise is added to the cluster centers. The noise covariance matrix Σt,k is obtained from historical data statistics and reflects the intra-cluster variability. If the rank of Σt,k is not full, a diagonal matrix approximation can be used or a regularization term λI can be added.

[0092] Line 14: Since adding noise may cause the value to exceed the range [0,1], truncation is required. After truncation, the original dimensions are restored by using the inverse normalization formula x_origin = x_norm × (xmax - xmin) + xmin.

[0093] The following is an analysis of the computational complexity of the above algorithm: The time complexity of the scene generation algorithm is O(N×T×K), where N is the number of scenes, T=96 is the number of time periods, and K is the average number of states. For N=10,000, T=96, K=5, the total number of samples is approximately 4.8×10^6. 6 This can be completed in just a few seconds on a modern computer. The space complexity is O(N×T×d), where d is the feature dimension. Storing 10,000 scenes requires approximately 0.5GB of memory, which can be easily accommodated.

[0094] This Monte Carlo scenario with uncertainty offers the following advantages: First, it is data-driven: it does not require assumptions about the parameter distribution (such as normal or Beta distribution), learning directly from historical data and adapting to any distribution. Second, it has temporal correlation: Markov chains naturally capture time dependencies, avoiding unreasonable scenarios generated by independent sampling (such as photovoltaic systems oscillating violently on the same day). Third, it is highly efficient: the scenario generation speed is fast, and it can generate massive amounts of scenarios (10^10^10). 4 -10 5 Fourth, interpretability: Each scenario corresponds to a clear state transition path, facilitating the analysis of the probability of occurrence and coping strategies for extreme events (such as prolonged rainy days). Fifth, scalability: Supports multivariate joint modeling (photovoltaics, electricity prices, load) and non-stationary processes (seasonal, trend-based), achieved by adding state dimensions or using time-varying transition probability matrices.

[0095] Step S120: The scheduling and bidding problem of the integrated photovoltaic-storage-charging station is modeled as a multi-stage Markov decision process model that includes day-ahead decision and multiple intraday rolling adjustments.

[0096] The scheduling and bidding problem of integrated photovoltaic, energy storage, and charging stations is modeled as a multi-stage Markov decision process (joint optimization MDP model) that includes day-ahead decision-making and multiple intraday rolling adjustments, defining the following elements: State space S: includes the real-time output of photovoltaic power at the current moment, the state of charge (SOC) of the energy storage system, the number of charging loads connected and their power demand, the current market price, the trading session, and the current uncertain state label from the Markov chain.

[0097] Action Space A: This includes charging and discharging power commands for energy storage systems, declared photovoltaic power generation, capacity pricing for participating in ancillary services such as frequency regulation, and power allocation strategies for charging piles.

[0098] The reward function R is designed as a single-step economic benefit and can be calculated using the following formula: (15); in, For real-time electricity prices, For auxiliary service unit price, To provide ancillary service capacity, This represents the equivalent lifetime loss cost of the i-th device (such as a battery or inverter) during time period t.

[0099] The state transition mechanism T is driven by the actual physical dynamic equations and the Markov chain, and satisfies the following conditions: (16); in, For the system dynamics model, Indicates based on the current state S t The Markov state transition distribution.

[0100] Step S130: The multi-stage Markov decision process model is trained using a deep reinforcement learning algorithm to obtain a policy control network that can adapt to various uncertain scenarios and meet the physical constraints of the equipment.

[0101] In some embodiments, the agent is trained using the PPO algorithm with 10,000 generated random Monte Carlo scenarios. Training parameters are set as follows: learning rate lr = 1e-4, batch size 64, 2048 samples per update, 10 training epochs, and discount factor γ = 0.99. After approximately 2000 training episodes (equivalent to 2000 simulated days), the policy network converges. After training, performance is evaluated on 100 independent test scenarios.

[0102] Step S140: Deploy the trained strategy control network in the energy management system to realize the global coordinated scheduling and bidding action commands of the photovoltaic-storage-charging integrated station.

[0103] The following uses a practical application example to further illustrate the effects of the present invention. In some embodiments of the present invention, a photovoltaic-storage-charging integrated energy station is configured with a 500kW photovoltaic array, a 250kW / 500kWh battery energy storage system, and 20 high-speed charging piles (capable of charging up to 20 electric vehicles simultaneously). This energy station is connected to the regional power grid and can participate in the local day-ahead energy market and frequency regulation ancillary service market, as well as intraday energy trading every 4 hours. The method of the present invention is applied to optimize the 24-hour operation of this energy station under a typical day. Table 1 shows the system parameter configuration.

[0104] Table 1

[0105] Training Results and Performance Analysis: The agent was trained using the PPO algorithm with 10,000 generated random scenarios. Training parameters were set as follows: learning rate lr = 1e-4, batch size 64, 2048 samples per update, 10 training epochs, and discount factor γ = 0.99. After approximately 2000 training episodes (equivalent to 2000 simulated days), the policy network converged. Performance was evaluated on 100 independent test scenarios after training. Table 2 shows the performance metrics comparison.

[0106] Table 2

[0107] As shown in Table 2, the method of this invention significantly outperforms traditional methods in three indicators: average daily profit, photovoltaic grid integration rate, and solution time. Compared with deterministic optimization methods, the profit is increased by 27.7%, and the photovoltaic grid integration rate is increased by 8.5 percentage points; compared with two-stage stochastic optimization methods, the profit is increased by 16.7%, and the solution time is shortened by 99.996%, fully verifying the effectiveness and superiority of this invention.

[0108] As can be seen from the above, the key points of implementing the embodiments of the present invention are: (1) fully collecting historical data to ensure the accuracy of the Markov chain model; (2) reasonably designing the state space and action space to balance the model complexity and computational efficiency; (3) carefully designing the reward function to guide the policy to learn feasible solutions that satisfy the constraints; (4) fully training the PPO algorithm to ensure that the policy converges to a high-quality solution; and (5) conducting sufficient simulation tests and verifications before actual deployment.

[0109] As can be seen from the above, this application has at least the following beneficial effects: Fourth, economic benefits are significantly improved. The integrated energy station can autonomously learn bidding and scheduling strategies that approximate the optimal level, thereby maximizing operational revenue. Simulation results show that compared with traditional two-stage optimization, applying deep RL can increase the total profit of the station by 15-20% and significantly reduce losses such as default penalties. In the simulation example, the daily total profit achieved by this scheme is close to the theoretical optimum, with an optimization gap of less than 1%, demonstrating excellent optimization performance.

[0110] Fifth, robustness is significantly enhanced. By employing Markov chains to generate a large number of random scenarios and fully exploring them during training, the strategy obtained by this method exhibits strong adaptability and robustness to various weather changes, price fluctuations, and uncertainties in charging demand. This means that in actual operation, even if the environment deviates from certain predictions, this method can still adjust the strategy to maintain a near-optimal return level, reducing reliance on accurate predictions.

[0111] Sixth, the invention achieves global coordination and optimization in decision-making. Through multi-stage joint optimization and RL policy training, it ensures consistent goals and coordination between day-ahead and multiple intraday decisions, avoiding suboptimal results that may occur in traditional methods where each stage operates independently. When market conditions change, the strategy is dynamically adjusted during intraday stages, but always with the goal of achieving global optimum, thus balancing long-term benefits and short-term flexibility.

[0112] Fourth, it boasts strong real-time decision-making capabilities. Once the deep reinforcement learning strategy is trained, its computation is highly efficient during actual deployment, providing optimization decisions for the current state in just milliseconds. This allows energy stations to easily handle frequent intraday market clearing cycles, achieving near real-time intelligent control. Traditional mathematical programming-based methods may struggle to solve complex problems in a timely manner, especially when frequent rolling operations are required, making the computational speed advantage of this invention even more pronounced.

[0113] Fifth, it exhibits good scalability and portability. This method demonstrates excellent scalability, allowing for the smooth introduction of other distributed energy sources or loads into the state space, in addition to photovoltaics, energy storage, and electric vehicles. The agent can automatically adapt to the decisions of newly added modules through training. Furthermore, since the strategy is presented in the form of a neural network, it is also easy to port to embedded energy management systems, enabling rapid deployment of intelligent control modules.

[0114] Figure 2 This diagram illustrates the structure of a multi-stage scheduling and market bidding optimization device for photovoltaic-energy storage-charging according to an embodiment of the present invention. Figure 2 It can be seen that the multi-stage scheduling and market bidding optimization device 200 for photovoltaic-energy storage-charging includes: The scenario construction unit 210 is used to construct a data-driven stochastic environment model based on historical operating data, which reflects the uncertainty of photovoltaic output, electricity price fluctuations and charging load. The data-driven stochastic environment model is a Monte Carlo stochastic scenario constructed by combining time series clustering and non-homogeneous Markov chains. The decision model building unit 220 is used to model the scheduling and bidding problem of the integrated photovoltaic storage and charging station into a multi-stage Markov decision process model that includes day-ahead decision and multiple intraday rolling adjustments. Training unit 230 is used to train the multi-stage Markov decision process model using a deep reinforcement learning algorithm to obtain a policy control network that can adapt to various uncertainty scenarios and meet the physical constraints of the equipment. Deployment unit 240 is used to deploy the trained strategy control network in the energy management system to realize the global coordinated scheduling and bidding action commands of the photovoltaic-storage-charging integrated station.

[0115] In some embodiments of this application, in the above-described apparatus, the scene construction unit 210 is used to collect historical operational data in multiple dimensions and preprocess it to obtain multidimensional time-series variables; extract contemporaneous feature vectors from the multidimensional time-series variables according to time periods; discretize the continuous state space into a finite number of state clusters using the density clustering algorithm DBSCAN; estimate the transition probability matrix corresponding to each time period based on the historical state transition frequency, and introduce Laplace smoothing to enhance the model's generalization ability; construct a non-homogeneous Markov chain model, whose initial distribution is obtained from the state frequency statistics of the first time period, to describe the joint dynamic characteristics of uncertainty evolving over time; and generate an uncertain Monte Carlo random scene based on the trained non-homogeneous Markov chain as a data-driven random environment model.

[0116] In some embodiments of this application, in the above-described apparatus, the scene construction unit 210 is used to construct a normalized multi-dimensional feature vector set based on historical operating data for each scheduling period. The feature vectors include at least photovoltaic power generation output, grid electricity price, and electric vehicle charging load, and may optionally include meteorological and time coding information. The DBSCAN density clustering algorithm is used to cluster the feature vector set for each period, wherein the neighborhood radius ε is determined by the inflection point of the k-distance curve, and the minimum number of points MinPts is set to the feature dimension plus 1. Typical operating modes are divided according to the clustering results, a unique state number is assigned to each effective cluster, and its cluster center and covariance matrix are calculated for subsequent scene generation. The silhouette coefficient is used to evaluate the clustering quality. If the average value is lower than a preset threshold, the parameters are adjusted and re-clustering is performed to ensure the rationality and discriminability of the state division.

[0117] In some embodiments of this application, in the above-described apparatus, the scene construction unit 210 is used to count the frequency of historical state transitions; based on the counted frequency of historical state transitions, the original transition probability is calculated using the maximum likelihood method; and the transition probability matrix is ​​smoothed using Laplace.

[0118] In some embodiments of this application, in the above-described apparatus, the scene construction unit 210 is used to construct a first-order Markov chain; construct an initial state distribution estimate of the first-order Markov chain; store the smoothed transition probability matrix Pt corresponding to each time period as model parameters, and combine it with the initial state distribution of the first-order Markov chain to form a non-homogeneous Markov chain.

[0119] In some embodiments of this application, in the above-described apparatus, the scene construction unit 210 is used to determine the discrete state of each scene in the first time period based on random sampling of the initial state distribution for a daily operating scene; and to use the transition probability matrix P corresponding to each time period in chronological order. tBased on the current state, the next state is sampled step by step to construct a complete state transition path for 96 time periods. After obtaining the entire discrete state sequence, the state number of each time period is mapped to the center vector of its corresponding cluster, and a Gaussian perturbation that conforms to the statistical characteristics of the cluster is superimposed to generate continuous feature values. The obtained values ​​are truncated in the range of [0,1] and converted into actual physical quantities through inverse normalization. The actual physical quantities include photovoltaic power output, electricity price and charging demand.

[0120] In some embodiments of this application, in the above-described apparatus, when training a multi-stage Markov decision process model, the agent is trained using the PPO algorithm with 10,000 generated random Monte Carlo scenarios. The training parameters are set as follows: learning rate lr=1e-4, batch size 64, 2048 samples per update, 10 training epochs, and discount factor γ=0.99. After approximately 2000 training episodes, the performance is evaluated on 100 independent test scenarios.

[0121] It should be noted that the aforementioned photovoltaic-energy storage-charging multi-stage scheduling and market bidding optimization device can realize the aforementioned photovoltaic-energy storage-charging multi-stage scheduling and market bidding optimization method, which will not be elaborated further.

[0122] Figure 3 A schematic diagram of the structure of an electronic device according to an embodiment of the present invention is shown. Figure 3 As shown, the electronic device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used for communication with external devices via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a multi-stage scheduling and market bidding optimization method for photovoltaic-energy storage-charging.

[0123] In one embodiment, the electronic device provided by the present invention includes a memory and a processor. The memory stores a database and a computer program that can run on the processor. When the processor executes the computer program, it implements the steps of the aforementioned photovoltaic-energy storage-charging multi-stage scheduling and market bidding optimization method.

[0124] The above is as described in the present invention. Figure 2The method for multi-stage scheduling and market bidding optimization of photovoltaic-energy storage-charging disclosed in the illustrated embodiments can be applied to a processor or implemented by a processor. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by software instructions. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The steps of the method disclosed in the embodiments of this invention can be directly implemented by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0125] In one embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the aforementioned photovoltaic-energy storage-charging multi-stage scheduling and market bidding optimization method.

[0126] It should be noted that the functions or steps that the above-mentioned electronic devices or computer-readable storage media can achieve can be referred to the relevant descriptions in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0127] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0128] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0129] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A multi-stage scheduling and market bidding optimization method for photovoltaic-energy storage-charging, characterized in that, include: A data-driven stochastic environment model is constructed based on historical operating data to reflect the uncertainty of photovoltaic output, electricity price fluctuations and charging load. The data-driven stochastic environment model is a Monte Carlo stochastic scenario constructed by combining time series clustering and non-homogeneous Markov chains. The scheduling and bidding problem of integrated photovoltaic, energy storage and charging stations is modeled as a multi-stage Markov decision process model that includes day-ahead decision and multiple intraday rolling adjustments. The multi-stage Markov decision process model is trained using a deep reinforcement learning algorithm to obtain a policy control network that can adapt to various uncertain scenarios and meet the physical constraints of the equipment. The trained strategy control network is deployed in the energy management system to realize the global coordinated scheduling and bidding action commands of the photovoltaic-storage-charging integrated station.

2. The method according to claim 1, characterized in that, The data-driven stochastic environment model is constructed according to the following method: Collect historical operational data from multiple dimensions and preprocess it to obtain multidimensional time-series variables; Extract contemporaneous feature vectors from multidimensional time-series variables by time period; The density clustering algorithm DBSCAN is used to discretize the continuous state space into a finite number of state clusters. The transition probability matrix for each time period is estimated based on the historical state transition frequency, and Laplace smoothing is introduced to enhance the model's generalization ability. A non-homogeneous Markov chain model is constructed, whose initial distribution is obtained from the state frequency statistics of the first time period, and is used to describe the joint dynamic characteristics of uncertainty evolution over time. Uncertainty-based Monte Carlo random scenarios are generated using trained non-homogeneous Markov chains as a data-driven random environment model.

3. The method according to claim 2, characterized in that, The discretization of a continuous state space into a finite number of state clusters using the density clustering algorithm DBSCAN includes: For each scheduling period, a normalized multidimensional feature vector set is constructed based on historical operation data. The feature vector includes at least photovoltaic power generation output, grid electricity price and electric vehicle charging load, and may optionally include meteorological and time coding information. The DBSCAN density clustering algorithm is used to cluster the feature vector sets of each time period. The neighborhood radius ε is determined by the inflection point of the k-distance curve, and the minimum number of points MinPts is set to the feature dimension plus 1. Based on the clustering results, typical operating modes are divided, a unique state number is assigned to each effective cluster, and its cluster center and covariance matrix are calculated for subsequent scene generation; The silhouette coefficient is used to evaluate the clustering quality. If the average value is lower than the preset threshold, the parameters are adjusted and the clustering is re-organized to ensure the rationality and discriminativeness of the state division.

4. The method according to claim 2, characterized in that, The method of estimating the transition probability matrix corresponding to each time period based on the historical state transition frequency, and introducing Laplace smoothing to enhance the model's generalization ability, includes: Statistical analysis of historical state transition frequency; Based on the historical state transition frequencies obtained from statistics, the original transition probabilities are calculated using the maximum likelihood method. The transition probability matrix is ​​smoothed using Laplace.

5. The method according to claim 2, characterized in that, The construction of the non-homogeneous Markov chain model includes: Construct a first-order Markov property chain; Construct an initial state distribution estimate for a first-order Markovian chain; The smoothed transition probability matrix Pt corresponding to each time period is stored as the model parameters, and together with the initial state distribution of the first-order Markov chain, they form a non-homogeneous Markov chain.

6. The method according to claim 2, characterized in that, The Monte Carlo random scene based on the generation of uncertainty from a trained non-homogeneous Markov chain includes: For a daily operation scenario, the discrete state of each scenario in the first time period is determined by random sampling based on the initial state distribution; In chronological order, using the transition probability matrix P corresponding to each time period t Based on the current state, the next state is sampled step by step to construct a complete state transition path for 96 time periods; After obtaining the entire discrete state sequence, the state number of each time period is mapped to the center vector of its corresponding cluster, and a Gaussian perturbation that conforms to the statistical characteristics of the cluster is superimposed to generate continuous feature values. The obtained values ​​are truncated within the range of [0,1] and converted into actual physical quantities through inverse normalization. The actual physical quantities include photovoltaic output, electricity price, and charging demand.

7. The method according to claim 1, characterized in that, When training the multi-stage Markov decision process model, the agent was trained using the PPO algorithm with 10,000 generated random Monte Carlo scenarios. The training parameters were set as follows: learning rate lr=1e-4, batch size 64, 2048 samples per update, 10 training epochs, and discount factor γ=0.

99. After about 2000 training episodes, the performance was evaluated on 100 independent test scenarios.

8. A multi-stage scheduling and market bidding optimization device for photovoltaic-energy storage-charging, characterized in that, include: The scenario construction unit is used to construct a data-driven stochastic environment model based on historical operating data, which reflects the uncertainty of photovoltaic output, electricity price fluctuations and charging load. The data-driven stochastic environment model is a Monte Carlo stochastic scenario constructed by combining time series clustering and non-homogeneous Markov chains. The decision model building unit is used to model the scheduling and bidding problem of the integrated photovoltaic-storage-charging station into a multi-stage Markov decision process model that includes day-ahead decision and multiple intraday rolling adjustments. The training unit is used to train the multi-stage Markov decision process model using a deep reinforcement learning algorithm to obtain a policy control network that can adapt to various uncertain scenarios and meet the physical constraints of the equipment. The deployment unit is used to deploy the trained strategy control network into the energy management system to realize the global coordinated scheduling and bidding action commands of the photovoltaic-storage-charging integrated station.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the multi-stage scheduling and market bidding optimization method for photovoltaic-energy storage-charging as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is instructed by the processor, it implements the steps of the multi-stage scheduling and market bidding optimization method for photovoltaic-energy storage-charging as described in any one of claims 1 to 7.

Images