A charging station operation scheduling optimization method based on reinforcement learning

Abstract

The application discloses a kind of based on reinforcement learning's charging station operation scheduling optimization method, comprising the following steps: relevant data in current scheduling period of charging station is collected, and original state set is formed by preprocessing;The encoding and fusion of each kind of feature in original state set are carried out, and uniform state hidden vector is generated;Determine operation evidence item, and generate scheduling confidence parameter using the method based on evidence availability and fluctuation risk joint correction;Hierarchical reinforcement learning scheduling model is constructed, and candidate scheduling strategy is output;Executable scheduling strategy is generated by the hierarchical constraint projection correction of candidate scheduling strategy;Executable scheduling strategy is executed and forms operation feedback event stream;According to operation feedback event stream, update hierarchical reinforcement learning scheduling model parameter and operation evidence item weight.The application can improve the real-time performance, self-adaptability and constraint satisfaction ability of charging station scheduling decision, reduce user waiting time and operation cost, improve service income and overall operation efficiency.

Description

Technical Field

[0001] This invention relates to the field of intelligent operation and energy scheduling optimization technology for charging stations, and in particular to a method for optimizing the operation and scheduling of charging stations based on reinforcement learning. Background Technology

[0002] With the continuous growth of new energy vehicle ownership and the expansion of charging infrastructure, operation scheduling optimization technologies for charging station operation have received widespread attention. Existing charging station scheduling methods mainly rely on fixed rules, static priority allocation, or single-layer optimization algorithms for vehicle access, power allocation, and energy storage control. However, these methods commonly suffer from the following problems in practical applications:

[0003] Vehicle arrivals within stations exhibit significant temporal fluctuations and randomness. Scheduled arrivals, temporary arrivals, equipment failures, and load changes are intertwined. Existing methods often rely solely on single-moment status for scheduling, failing to accurately reflect the evolution of operational status over continuous scheduling cycles. This leads to inaccurate reserved capacity, delayed peak-hour strategy switching, and inappropriate timing for energy storage intervention. Station-level operational data, pile-level operational data, vehicle-level status data, and external environmental data come from diverse sources, are asynchronous in timing, and exhibit significant quality fluctuations. Existing methods lack a joint processing mechanism to address the reliability of operational evidence and the risks of scenario fluctuations, easily causing distortion in scheduling credibility assessments and reducing the stability of scheduling decisions. Furthermore, the high degree of coupling between vehicle-pile matching, power allocation, and energy storage coordination means that existing methods typically lack a tiered correction process for hard and soft constraints, resulting in poor executability of output strategies and difficulty in balancing waiting time, operational revenue, equipment load balancing, and user satisfaction.

[0004] Therefore, how to provide a method for optimizing the operation and scheduling of charging stations based on reinforcement learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a charging station operation scheduling optimization method based on reinforcement learning. This invention comprehensively adopts operational evidence item processing, scheduling credibility parameter calculation based on joint correction of evidence availability and volatility risk, hierarchical reinforcement learning scheduling model, and hierarchical constraint projection correction method. It describes in detail the intelligent implementation process of charging station operation scheduling and has the advantages of high scheduling stability, strong strategy adaptability, good executability, and high overall operational efficiency.

[0006] A method for optimizing the operation scheduling of charging stations based on reinforcement learning according to an embodiment of the present invention includes the following steps:

[0007] Step 1: Collect station-level operation data, pile-level operation data, vehicle-level status data, and external environment data of the charging station within the current scheduling cycle, and preprocess them to form the original state set;

[0008] Step 2: Encode and fuse the station-level features, pile-level features, vehicle-level features, and environmental features in the original state set to generate a unified state latent vector;

[0009] Step 3: Determine the operational evidence items based on the original state set, and use the scheduling credibility parameter calculation method based on the joint correction of evidence availability and volatility risk to process the operational evidence items and generate scheduling credibility parameters;

[0010] Step 4: Construct a hierarchical reinforcement learning scheduling model. The hierarchical reinforcement learning scheduling model includes a high-level PPO policy network that introduces an operational state temporal memory unit and a low-level attention-enhanced Dueling Double DQN scheduling network. Based on the unified state latent vector and scheduling credibility parameters, it outputs station-level scheduling action primitives, continuous control parameters, vehicle-pile matching actions, power allocation actions, and energy storage coordination actions to form candidate scheduling strategies.

[0011] Step 5: Use the hierarchical constraint projection correction method to perform hard constraint verification, soft constraint optimization, and action correction on the candidate scheduling strategy to generate an executable scheduling strategy;

[0012] Step Six: Execute executable scheduling strategies, collect actual waiting time, reservation fulfillment results, power point tracking error, service revenue, electricity purchase cost, energy storage cycle count, equipment temperature rise changes, and user evaluation results to form an operational feedback event flow;

[0013] Step 7: Update the parameters of the hierarchical reinforcement learning scheduling model based on the operational feedback event flow, and update the weights of the operational evidence items.

[0014] Optionally, step one specifically includes:

[0015] Collect station-level operation data, pile-level operation data, vehicle-level status data, and external environment data of charging stations within the current scheduling cycle;

[0016] The station-level operation data, pile-level operation data, vehicle-level status data, and external environment data are timestamped according to a unified scheduling clock.

[0017] The data after timestamp alignment is preprocessed. The preprocessing includes filling in missing data items according to the corresponding values ​​of the same data source at the previous sampling time, filling in data items that are missing for more than a preset number of sampling times according to the average of the data items in multiple consecutive scheduling cycles before the current scheduling cycle, removing data items that exceed the allowable range of the corresponding physical quantity, encoding and converting categorical data, and normalizing numerical data according to their respective historical value ranges.

[0018] The preprocessed station-level operation data, pile-level operation data, vehicle-level status data, and external environment data are collected at the station, pile, vehicle, and environment levels to form the original status set.

[0019] Optionally, step two specifically includes:

[0020] The original state set is divided into a station-level feature subset, a pile-level feature subset, a vehicle-level feature subset, and an environmental feature subset;

[0021] The station-level feature subset is encoded to obtain the station-level encoding vector. The pile-level feature subset is arranged and encoded according to the charging pile identifier to obtain the pile-level encoding vector. The vehicle-level feature subset is arranged and encoded according to the vehicle arrival time and the scheduled arrival time to obtain the vehicle-level encoding vector. The environmental feature subset is encoded to obtain the environmental encoding vector.

[0022] The station-level coding vector, pile-level coding vector, vehicle-level coding vector, and environment coding vector are concatenated in sequence to form the initial fusion vector;

[0023] The initial fusion vector is subjected to dimensional unification using fully connected projection mapping to obtain a projection vector. The projection vector is then subjected to gated weighted fusion processing to generate a unified state latent vector.

[0024] Optionally, step three specifically includes:

[0025] The arrival vehicle records in the original state set are statistically analyzed by time period and the trend of change are processed to obtain the short-term arrival traffic prediction value. The reserved vehicle records in the original state set are statistically processed for reserved arrival time, actual arrival time, no-arrival records and cancellation records to obtain the reservation fulfillment probability. The charging pile operation records in the original state set are processed for normal operation time ratio, reduced operation status and fault status to obtain the equipment availability probability. The station power records in the original state set are processed by deducting the total available power in the station, real-time output power and reserved power demand to obtain the current power margin.

[0026] The energy storage operation records in the original state set are processed by the energy storage state of charge, maximum discharge power, and current discharge power to obtain the energy storage release capacity. The photovoltaic output records in the original state set are processed by the current real-time photovoltaic output, historical photovoltaic output level at the same time, and weather conditions to obtain the photovoltaic usable power. The user charging request records in the original state set are processed by the vehicle target state of charge, current state of charge, energy replenishment demand, and expected remaining stay time before departure to obtain the user urgency score. The short-term arrival traffic forecast, reservation fulfillment probability, equipment availability probability, current power margin, energy storage release capacity, photovoltaic usable power, and user urgency score are used as operational evidence items. The data integrity, variation range between adjacent scheduling cycles, and data collection time interval of each operational evidence item are processed to obtain the evidence usability.

[0027] Each operational evidence item is processed by analyzing its historical deviation from similar values ​​and the magnitude of changes in the current scheduling cycle to obtain the volatility risk. Each operational evidence item is then combined with its corresponding evidence availability for credibility correction to obtain the first corrected evidence item. The first corrected evidence item is then combined with its corresponding volatility risk for risk suppression correction to obtain the second corrected evidence item. The second corrected evidence item is then weighted and aggregated to obtain the scheduling credibility parameter.

[0028] Optionally, step four specifically includes:

[0029] The unified state latent vector and scheduling credibility parameters are input into the operation state temporal memory unit in the high-level PPO policy network. The current scheduling cycle input state and the previous scheduling cycle historical state are temporally correlated to obtain the station-level temporal state features. The operation state temporal memory unit includes an input gate, a forget gate, an output gate, and a memory state channel. The input gate performs weight control on the writing of the current scheduling cycle input state. The forget gate performs weight control on the retention of the previous scheduling cycle historical state. The memory state channel performs weighted accumulation on the current scheduling cycle input state and the previous scheduling cycle historical state. The output gate outputs the station-level temporal state features.

[0030] The station-level time-series state characteristics are input into the high-level PPO policy network for policy calculation. The high-level PPO policy network includes a fully connected layer, a nonlinear activation layer and a policy output layer connected in sequence. The policy output layer includes a discrete action output branch and a continuous parameter output branch. The discrete action output branch outputs the probability distribution of station-level scheduling action primitives, and the continuous parameter output branch outputs continuous control parameters.

[0031] The unified state latent vector, scheduling credibility parameters, station-level scheduling action primitives, and continuous control parameters are input into a low-level attention-enhanced Dueling Double DQN scheduling network. The low-level attention-enhanced Dueling Double DQN scheduling network includes an attention association unit, a feature fusion layer, a value branch, and a dominance branch. The attention association unit outputs vehicle-pile association features. The feature fusion layer concatenates and fuses the vehicle-pile association features with the unified state latent vector, scheduling credibility parameters, station-level scheduling action primitives, and continuous control parameters to obtain joint scheduling features. The value branch outputs the state value corresponding to the current scheduling state. The dominance branch outputs the action dominance value corresponding to each candidate action. The state value and the action dominance value are combined to obtain the action value corresponding to vehicle-pile matching action, power allocation action, and energy storage coordination action.

[0032] The vehicle-pile matching actions, power allocation actions, and energy storage coordination actions are sorted and screened according to their action value. The station-level scheduling action primitives, continuous control parameters, vehicle-pile matching actions, power allocation actions, and energy storage coordination actions are then combined to form candidate scheduling strategies.

[0033] Optionally, step five specifically includes:

[0034] The candidate scheduling strategies are parsed according to the station-level scheduling action primitives, continuous control parameters, vehicle-pile matching actions, power allocation actions and energy storage coordination actions to obtain the action set to be verified.

[0035] The set of actions to be verified is subjected to hard constraint verification, which includes verification of transformer capacity upper limit, verification of total power upper limit in the station, verification of rated output power of single pile, verification of energy storage charge state boundary and verification of fault pile disabling. Actions that fail the hard constraint verification are removed to obtain a set of hard constraint candidate actions.

[0036] The set of candidate actions with hard constraints is optimized with soft constraints. The soft constraint optimization includes optimization of appointment fulfillment target, waiting time target, equipment load balancing target, and peak-valley electricity price cost target. The action cost is calculated and sorted according to the action cost.

[0037] The sorted set of hard-constraint candidate actions is modified to obtain the set of executable actions.

[0038] The set of executable actions is reorganized according to the original field order of the candidate scheduling policy to generate an executable scheduling policy.

[0039] Optionally, step six specifically includes:

[0040] The executable scheduling strategy is sent to the charging station control system to execute station-level scheduling action primitives, continuous control parameters, vehicle-charging station matching actions, power allocation actions, and energy storage coordination actions.

[0041] The system collects data on the actual start time of vehicle charging, the actual arrival time of reserved vehicles, the actual output power, service order settlement data, electricity purchase settlement data, energy storage charging and discharging metering data, equipment temperature monitoring data, and user evaluation data.

[0042] The actual waiting time is obtained by comparing the actual start time of the vehicle with the arrival time of the vehicle. The actual arrival time of the reserved vehicle is compared with the reservation record to obtain the reservation fulfillment result. The actual output power is compared with the target power value to obtain the power tracking error. The service order settlement data, electricity purchase settlement data, energy storage charging and discharging metering data, equipment temperature monitoring data and user evaluation data are statistically processed to obtain service revenue, electricity purchase cost, energy storage cycle number, equipment temperature rise change and user evaluation results.

[0043] The actual waiting time, reservation fulfillment results, power tracking error, service revenue, electricity purchase cost, energy storage cycle count, equipment temperature rise changes, and user evaluation results are packaged into an operation feedback event flow.

[0044] Optionally, step seven specifically includes:

[0045] The actual waiting time, reservation fulfillment results, power tracking error, service revenue, electricity purchase cost, energy storage cycle count, equipment temperature rise change and user evaluation results in the operation feedback event stream are extracted to construct a reward signal;

[0046] The unified state latent vector, scheduling credibility parameters, candidate scheduling policies, executable scheduling policies, reward signals, and the unified state latent vector corresponding to the next scheduling cycle are associated and packaged to form training samples.

[0047] The training samples are input into the hierarchical reinforcement learning scheduling model. The policy gradient of the high-level PPO policy network is updated, and the action value of the low-level attention-enhanced Dueling Double DQN scheduling network is updated to obtain the updated parameters of the hierarchical reinforcement learning scheduling model.

[0048] The actual waiting time, reservation fulfillment results, power tracking error, service revenue, electricity purchase cost, energy storage cycle count, equipment temperature rise change, and user evaluation results in the operational feedback event stream are correlated with each operational evidence item, and the weight of the operational evidence items is adjusted according to the degree of influence of each operational evidence item on the reward signal.

[0049] The beneficial effects of this invention are:

[0050] This invention addresses the problems in charging station operation scheduling, such as reliance on single-moment state decisions, insufficient reliability of operational evidence, and poor executability of scheduling results. It constructs a complete technical chain from obtaining the original state set, generating unified state latent vectors, calculating scheduling credibility parameters, hierarchical reinforcement learning scheduling, to hierarchical constraint projection correction and operational feedback updates. Specifically, the scheduling credibility parameter calculation method based on joint correction of evidence availability and fluctuation risk improves the accuracy of operational evidence item processing and the reliability of scheduling inputs. By introducing a high-level PPO policy network with operational state temporal memory units and a low-level attention-enhanced Dueling Double DQN scheduling network to collaboratively output candidate scheduling strategies, it enhances the ability to characterize operational changes in continuous scheduling cycles and the ability to make joint decisions on multiple actions. Furthermore, the hierarchical constraint projection correction method ensures that the candidate scheduling strategies meet the actual operational boundaries, thereby effectively improving the scheduling stability, strategy adaptability, execution feasibility, and overall operational efficiency of charging stations under peak fluctuation, dense reservation, and equipment state change scenarios. This has strong engineering application value and promotional significance. Attached Figure Description

[0051] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0052] Figure 1 This is a flowchart of a charging station operation scheduling optimization method based on reinforcement learning proposed in this invention;

[0053] Figure 2 This is a schematic diagram of a charging station operation scheduling optimization method based on reinforcement learning proposed in this invention;

[0054] Figure 3 This is a framework diagram of the hierarchical reinforcement learning scheduling model in the charging station operation scheduling optimization method proposed in this invention. Detailed Implementation

[0055] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0056] refer to Figures 1-3 A method for optimizing the operation and scheduling of charging stations based on reinforcement learning includes the following steps:

[0057] Step 1: Collect station-level operation data, pile-level operation data, vehicle-level status data, and external environment data of the charging station within the current scheduling cycle, and preprocess them to form the original state set;

[0058] Step 2: Encode and fuse the station-level features, pile-level features, vehicle-level features, and environmental features in the original state set to generate a unified state latent vector;

[0059] Step 3: Determine the operational evidence items based on the original state set, and use the scheduling credibility parameter calculation method based on the joint correction of evidence availability and volatility risk to process the operational evidence items and generate scheduling credibility parameters;

[0060] Step 4: Construct a hierarchical reinforcement learning scheduling model. The hierarchical reinforcement learning scheduling model includes a high-level PPO policy network that introduces operational state temporal memory units and a low-level attention-enhanced Dueling Double DQN scheduling network. Based on the unified state latent vector and scheduling credibility parameters, it outputs station-level scheduling action primitives, continuous control parameters, vehicle-pile matching actions, power allocation actions, and energy storage coordination actions to form candidate scheduling strategies.

[0061] Step 5: Use the hierarchical constraint projection correction method to perform hard constraint verification, soft constraint optimization, and action correction on the candidate scheduling strategy to generate an executable scheduling strategy;

[0062] Step Six: Execute executable scheduling strategies, collect actual waiting time, reservation fulfillment results, power point tracking error, service revenue, electricity purchase cost, energy storage cycle count, equipment temperature rise changes, and user evaluation results to form an operational feedback event flow;

[0063] Step 7: Update the parameters of the hierarchical reinforcement learning scheduling model based on the operational feedback event flow, and update the weights of the operational evidence items.

[0064] In this embodiment, step one specifically includes:

[0065] The system collects station-level operation data, pile-level operation data, vehicle-level status data, and external environment data for the current scheduling cycle. Station-level operation data includes the number of vehicles queuing in the station, the number of vehicles with reservations, the total available power in the station, the energy storage charge status, real-time photovoltaic output, and time-of-use electricity price data. Pile-level operation data includes the occupancy status, rated output power, real-time output power, equipment temperature, and fault status data of each charging pile. Vehicle-level status data includes the vehicle's current charge status, target charge status, estimated departure time, reserved arrival time, and charging demand data. External environment data includes weather conditions, time period type, holiday markings, and regional traffic congestion data.

[0066] The station-level operation data, pile-level operation data, vehicle-level status data and external environment data are timestamped according to a unified scheduling clock. The unified scheduling clock is a unified time reference established based on the start time of the current scheduling cycle.

[0067] The data after timestamp alignment is preprocessed. The preprocessing includes filling missing data items with the corresponding values ​​of the same data source at the previous sampling time, filling missing data items that have been missing for more than a preset number of sampling times with the average value of the data item in multiple consecutive scheduling cycles before the current scheduling cycle, removing data items that exceed the allowable range of the corresponding physical quantity, encoding and converting categorical data, and normalizing numerical data according to their respective historical value ranges.

[0068] The preprocessed station-level operation data, pile-level operation data, vehicle-level status data, and external environment data are collected at the station, pile, vehicle, and environment levels to form the original status set.

[0069] In this embodiment, step two specifically includes:

[0070] The original state set is divided into a station-level feature subset, a pile-level feature subset, a vehicle-level feature subset, and an environmental feature subset;

[0071] The station-level feature subset is encoded to obtain the station-level encoding vector. The pile-level feature subset is arranged and encoded according to the charging pile identifier to obtain the pile-level encoding vector. The vehicle-level feature subset is arranged and encoded according to the vehicle arrival time and the scheduled arrival time to obtain the vehicle-level encoding vector. The environmental feature subset is encoded to obtain the environmental encoding vector.

[0072] The station-level coding vector, pile-level coding vector, vehicle-level coding vector, and environment coding vector are concatenated in sequence to form the initial fusion vector;

[0073] The initial fusion vector is subjected to dimensional unification using fully connected projection mapping to obtain a projection vector. The projection vector is then subjected to gated weighted fusion processing to generate a unified state latent vector.

[0074] In this embodiment, step three specifically includes:

[0075] The arrival vehicle records in the original state set are statistically analyzed by time period and the trend of change are processed to obtain the short-term arrival traffic prediction value. The reserved vehicle records in the original state set are statistically processed for reserved arrival time, actual arrival time, no-arrival records and cancellation records to obtain the reservation fulfillment probability. The charging pile operation records in the original state set are processed for normal operation time ratio, reduced operation status and fault status to obtain the equipment availability probability. The station power records in the original state set are processed by deducting the total available power in the station, real-time output power and reserved power demand to obtain the current power margin.

[0076] The energy storage operation records in the original state set are processed by the energy storage state of charge, maximum discharge power, and current discharge power to obtain the energy storage release capacity. The photovoltaic output records in the original state set are processed by the current real-time photovoltaic output, historical photovoltaic output level at the same time, and weather conditions to obtain the photovoltaic usable power. The user charging request records in the original state set are processed by the vehicle target state of charge, current state of charge, energy replenishment demand, and expected remaining stay time before departure to obtain the user urgency score. The short-term arrival traffic forecast, reservation fulfillment probability, equipment availability probability, current power margin, energy storage release capacity, photovoltaic usable power, and user urgency score are used as operational evidence items. The data integrity, variation range between adjacent scheduling cycles, and data collection time interval of each operational evidence item are processed to obtain the evidence usability.

[0077] Each operational evidence item is processed by analyzing its historical deviation from similar values ​​and the magnitude of changes in the current scheduling cycle to obtain the volatility risk. Each operational evidence item is then combined with its corresponding evidence availability for credibility correction to obtain the first corrected evidence item. The first corrected evidence item is then combined with its corresponding volatility risk for risk suppression correction to obtain the second corrected evidence item. The second corrected evidence item is then weighted and aggregated to obtain the scheduling credibility parameter.

[0078] In this embodiment, step four specifically includes:

[0079] The unified state latent vector and scheduling credibility parameters are input into the operation state temporal memory unit in the high-level PPO policy network. The current scheduling cycle input state and the previous scheduling cycle historical state are temporally correlated to obtain the station-level temporal state features. The operation state temporal memory unit includes an input gate, a forget gate, an output gate and a memory state channel. The input gate performs weight control on writing the current scheduling cycle input state, the forget gate performs weight control on retaining the previous scheduling cycle historical state, the memory state channel performs weighted accumulation on the current scheduling cycle input state and the previous scheduling cycle historical state, and the output gate outputs the station-level temporal state features.

[0080] The station-level time-series state features are input into the high-level PPO policy network for policy calculation. The high-level PPO policy network includes a fully connected layer, a nonlinear activation layer, and a policy output layer connected in sequence. The fully connected layer is used to perform dimensional mapping on the station-level time-series state features. The nonlinear activation layer uses a modified linear unit function for nonlinear transformation. The policy output layer includes a discrete action output branch and a continuous parameter output branch. The discrete action output branch uses an exponential normalization function to output the probability distribution of station-level scheduling action primitives. The continuous parameter output branch uses a hyperbolic tangent function and interval scaling to output continuous control parameters.

[0081] The unified state latent vector, scheduling credibility parameters, station-level scheduling action primitives, and continuous control parameters are input into the low-level attention-enhanced Dueling Double DQN scheduling network. The low-level attention-enhanced Dueling Double DQN scheduling network includes an attention association unit, a feature fusion layer, a value branch, and a dominance branch. The attention association unit uses the vehicle features corresponding to the candidate vehicle set as the query vector and the charging pile features corresponding to the candidate charging pile set as the key vector and value vector. It calculates the association score by the dot product of the query vector and the key vector, and performs exponential normalization on the association score to obtain the vehicle-charging pile association weight. Then, it performs a weighted summation of the vehicle-charging pile association weight and the value vector to obtain the vehicle-charging pile association feature. The feature fusion layer concatenates and fuses the vehicle-charging pile association feature with the unified state latent vector, scheduling credibility parameters, station-level scheduling action primitives, and continuous control parameters to obtain the joint scheduling feature. The value branch outputs the state value corresponding to the current scheduling state, and the dominance branch outputs the action dominance value corresponding to each candidate action. The state value and the action dominance value are combined to obtain the action value corresponding to the vehicle-charging pile matching action, power allocation action, and energy storage coordination action.

[0082] The vehicle-pile matching actions, power allocation actions, and energy storage coordination actions are sorted and screened according to their action value. The station-level scheduling action primitives, continuous control parameters, vehicle-pile matching actions, power allocation actions, and energy storage coordination actions are combined to form candidate scheduling strategies.

[0083] The beneficial effects of constructing a hierarchical reinforcement learning scheduling model are as follows: The high-level PPO policy network, which includes an operational state temporal memory unit, can perform temporal modeling of queuing changes, reservation changes, power load changes, energy storage state changes, and equipment operating state changes over multiple consecutive scheduling cycles. This solves the problems of existing charging station operation scheduling methods that rely solely on single-moment states for decision-making, are unable to reflect the continuous evolution trend of operational states, and result in inaccurate reservation capacity reservations, unreasonable timing of energy storage intervention, and delayed peak-hour operation strategy switching. The low-level attention-enhanced Dueling Double DQN scheduling network addresses the problems of high action coupling, inconsistency between local decisions and station-level operation strategies, and unstable action value estimation in existing scheduling methods by jointly evaluating vehicle-charging station matching actions, power allocation actions, and energy storage collaborative actions. Furthermore, the high-level and low-level collaborative output of candidate scheduling strategies enables unified linkage between station-level operation strategies and execution-level scheduling actions, thereby improving the scheduling stability, strategy adaptability, and overall operational efficiency of charging stations in peak fluctuation scenarios, reservation-intensive scenarios, and equipment state change scenarios.

[0084] In this embodiment, step five specifically includes:

[0085] The candidate scheduling strategies are parsed according to the station-level scheduling action primitives, continuous control parameters, vehicle-pile matching actions, power allocation actions and energy storage coordination actions to obtain the action set to be verified.

[0086] The set of actions to be verified is subjected to hard constraint verification, which includes verification of transformer capacity upper limit, total power upper limit within the station, rated output power of a single pile, energy storage charge state boundary, and disabled fault pile verification. Specifically, the total power demand corresponding to the candidate scheduling strategy is compared with the transformer capacity upper limit and the total power upper limit within the station; the target power corresponding to the vehicle-pile matching action and power allocation action is compared with the rated output power of a single pile; the charging and discharging request corresponding to the energy storage collaborative action is compared with the energy storage charge state boundary; the target charging pile corresponding to the vehicle-pile matching action is compared with the fault pile state; and actions that fail the hard constraint verification are removed to obtain the hard constraint candidate action set.

[0087] The set of hard-constrained candidate actions is optimized using soft constraints. Soft-constraint optimization includes optimization of appointment fulfillment targets, waiting time targets, equipment load balancing targets, and peak-valley electricity price cost targets. Specifically, the appointment fulfillment deviation, waiting time increment, equipment load deviation, and electricity purchase cost increment corresponding to each hard-constrained candidate action are weighted to obtain the action value, and then sorted according to the action value.

[0088] The sorted set of hard constraint candidate actions is modified by the following steps: interval pruning is performed on the continuous control parameters, proportional voltage drop is performed on the power distribution action, replacement charging pile redistribution is performed on the vehicle-charging pile matching action, and charging and discharging direction switching or power amplitude adjustment is performed on the energy storage coordination action to obtain the set of executable actions.

[0089] The set of executable actions is reorganized according to the original field order of the candidate scheduling policy to generate an executable scheduling policy.

[0090] In this embodiment, step six specifically includes:

[0091] The executable scheduling strategy is sent to the charging station control system to execute station-level scheduling action primitives, continuous control parameters, vehicle-charging station matching actions, power allocation actions, and energy storage coordination actions.

[0092] The system collects data on the actual start time of vehicle charging, the actual arrival time of reserved vehicles, the actual output power, service order settlement data, electricity purchase settlement data, energy storage charging and discharging metering data, equipment temperature monitoring data, and user evaluation data.

[0093] The actual waiting time is obtained by comparing the actual start time of the vehicle with the arrival time of the vehicle. The actual arrival time of the reserved vehicle is compared with the reservation record to obtain the reservation fulfillment result. The actual output power is compared with the target power value to obtain the power tracking error. The service order settlement data, electricity purchase settlement data, energy storage charging and discharging metering data, equipment temperature monitoring data and user evaluation data are statistically processed to obtain service revenue, electricity purchase cost, energy storage cycle number, equipment temperature rise change and user evaluation results.

[0094] The actual waiting time, reservation fulfillment results, power tracking error, service revenue, electricity purchase cost, energy storage cycle count, equipment temperature rise changes, and user evaluation results are packaged into an operation feedback event flow.

[0095] In this embodiment, step seven specifically includes:

[0096] The actual waiting time, reservation fulfillment results, power tracking error, service revenue, electricity purchase cost, energy storage cycle count, equipment temperature rise change and user evaluation results in the operation feedback event stream are extracted to construct a reward signal;

[0097] The unified state latent vector, scheduling credibility parameters, candidate scheduling policies, executable scheduling policies, reward signals, and the unified state latent vector corresponding to the next scheduling cycle are associated and packaged to form training samples.

[0098] The training samples are input into the hierarchical reinforcement learning scheduling model. The policy gradient of the high-level PPO policy network is updated, and the action value of the low-level attention-enhanced Dueling Double DQN scheduling network is updated to obtain the updated parameters of the hierarchical reinforcement learning scheduling model.

[0099] The actual waiting time, reservation fulfillment results, power tracking error, service revenue, electricity purchase cost, energy storage cycle count, equipment temperature rise change, and user evaluation results in the operational feedback event stream are correlated with each operational evidence item, and the weight of the operational evidence items is adjusted according to the degree of influence of each operational evidence item on the reward signal.

[0100] Example 1: To verify the feasibility of this invention in practice, it was applied to a comprehensive new energy vehicle charging station near a city transportation hub. This charging station simultaneously serves ride-hailing vehicles, taxis, private cars, and vehicles with pre-booked charging. During daily operation, it experiences typical high-fluctuation operating scenarios, including concentrated arrivals during morning and evening rush hours, overlap of pre-booked and unbooked vehicles, significant fluctuations in time-of-use electricity prices, and occasional charging pile malfunctions. The station is equipped with 24 DC fast charging piles, each with a rated output power of 120kW, 16 AC slow charging piles, each with a rated output power of 7kW, a transformer capacity of 1800kW, an energy storage system with a rated capacity of 500kWh and a maximum charging / discharging power of 300kW, and a photovoltaic installed capacity of 260kW. The scheduling cycle was set to 5 minutes, the status sampling interval was set to 30 seconds, and operational data was continuously collected for 30 days, with the first 24 days used for training and the last 6 days for testing.

[0101] During implementation, station-level operational data, pile-level operational data, vehicle-level status data, and external environmental data are first collected and processed through time alignment, missing data completion, anomaly removal, and normalization to form an initial state set. Then, station-level features, pile-level features, vehicle-level features, and environmental features are encoded and fused to generate a unified state latent vector. Next, operational evidence items are determined based on the initial state set, and scheduling credibility parameters are generated using a scheduling credibility parameter calculation method based on evidence availability and fluctuation risk joint correction. Subsequently, a hierarchical reinforcement learning scheduling model is constructed. The high-level PPO policy network, which incorporates operational state temporal memory units, outputs station-level scheduling action primitives and continuous control parameters. The low-level attention-enhanced Dueling Double DQN scheduling network outputs vehicle-pile matching actions, power allocation actions, and energy storage coordination actions, forming candidate scheduling strategies. Finally, a hierarchical constraint projection correction method is used to perform hard constraint verification, soft constraint optimization, and action correction on the candidate scheduling strategies to generate an executable scheduling strategy. Finally, the executable scheduling strategy is executed, and the actual waiting time, reservation fulfillment results, power tracking error, service revenue, electricity purchase cost, energy storage cycle count, equipment temperature rise change and user evaluation results are collected to form an operation feedback event stream. The parameters of the hierarchical reinforcement learning scheduling model and the weights of operation evidence items are updated according to the operation feedback event stream.

[0102] To objectively evaluate the effectiveness of this invention, a fixed-rule scheduling method and a single-layer PPO scheduling method were selected for comparison. The fixed-rule scheduling method employs a static rule of first-come, first-served plus reservation priority; the single-layer PPO scheduling method directly outputs scheduling actions based solely on the current state, without setting up operational evidence processing or hierarchical constraint projection correction processes; the method of this invention adopts the complete process described in the claims. The testing phase selected the three consecutive days with the most concentrated load within the last six days, serving a total of 1864 vehicles, of which 31.6% were reserved vehicles and 68.4% were temporary arrivals, which can better reflect the real-world performance under complex operating conditions.

[0103] Table 1. Overall Results of Different Methods in the Test Scenario

[0104] method Average waiting time / min Reservation fulfillment rate / % Power tracking error / % Average daily service revenue / yuan Average daily electricity purchase cost (yuan) Average daily energy storage cycle times Equipment temperature rise change / °C User satisfaction score Fixed rule scheduling 23.8 86.4 9.7 28640 17980 1.18 13.6 7.9 Single-layer PPO scheduling 17.2 91.8 7.1 30120 17090 1.46 11.4 8.5 Method of the present invention 12.9 96.7 4.8 31860 16140 1.31 9.2 9.1

[0105] As shown in Table 1, the method of this invention outperforms the comparative methods in several core indicators. The average waiting time decreased from 23.8 minutes with fixed-rule scheduling to 12.9 minutes, a reduction of 45.8%, and a further reduction of 25.0% compared to single-layer PPO. This indicates that the present invention can more accurately identify load change trends before peak hours and make advance arrangements for reserved capacity, power adjustment, and energy storage intervention, thereby effectively reducing queuing time. The reservation fulfillment rate increased from 86.4% with fixed-rule scheduling to 96.7%, an increase of 4.9 percentage points compared to single-layer PPO. This shows that the present invention has a greater advantage in ensuring the timely access of reserved vehicles and can better balance the resource allocation relationship between reserved vehicles and temporarily arriving vehicles.

[0106] The power point tracking error decreased from 9.7% with fixed-rule scheduling to 4.8%, indicating that the scheduling strategy generated by the present invention after hierarchical constraint projection correction is more consistent with the actual operating boundary of the charging station, and can avoid the problem of power deviation accumulation caused by the inability to execute output actions. In terms of economics, the present invention achieves an average daily service revenue of 31,860 yuan, which is 11.2% higher than that of fixed-rule scheduling and 5.8% higher than that of single-layer PPO; the average daily electricity purchase cost is reduced to 16,140 yuan, which is 10.2% lower than that of fixed-rule scheduling and 5.6% lower than that of single-layer PPO. This shows that the present invention can combine the available photovoltaic power, the release capacity of energy storage, and the time-of-use electricity price changes to more rationally arrange the power allocation and energy storage coordinated actions within the station, thereby achieving simultaneous optimization of revenue improvement and cost control.

[0107] From the perspective of equipment operation, the average daily energy storage cycle count of this invention is 1.31 times, lower than the 1.46 times of single-layer PPO scheduling. This indicates that this invention does not trade local scheduling benefits for frequent start-stop energy storage, but rather ensures operational effectiveness while also considering energy storage lifespan. The equipment temperature rise decreased from 13.6℃ in fixed-rule scheduling to 9.2℃, a 19.3% reduction compared to single-layer PPO. This demonstrates that this invention reduces heat accumulation caused by prolonged high-load operation of local equipment through joint optimization of vehicle-pile matching actions, power distribution actions, and energy storage coordination actions. User satisfaction improved from 7.9 points in fixed-rule scheduling to 9.1 points, further illustrating the significant advantages of this invention in service experience and operational stability.

[0108] Based on the above implementation results, it can be seen that this invention forms a complete closed loop covering "state awareness—policy generation—action correction—execution feedback—model update" through operational evidence item processing, scheduling credibility parameter calculation, hierarchical reinforcement learning scheduling, and hierarchical constraint projection correction. Compared with existing technologies, this invention can not only significantly shorten waiting time, improve reservation fulfillment rate, reduce power tracking error and electricity purchase cost, but also improve equipment heat load distribution and enhance user satisfaction. Therefore, it has high engineering application value and promotion significance.

[0109] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for optimizing the operation and scheduling of charging stations based on reinforcement learning, characterized in that, Includes the following steps: Step 1: Collect station-level operation data, pile-level operation data, vehicle-level status data, and external environment data of the charging station within the current scheduling cycle, and preprocess them to form the original state set; Step 2: Encode and fuse the station-level features, pile-level features, vehicle-level features, and environmental features in the original state set to generate a unified state latent vector; Step 3: Determine the operational evidence items based on the original state set, and use the scheduling credibility parameter calculation method based on the joint correction of evidence availability and volatility risk to process the operational evidence items and generate scheduling credibility parameters; Step 4: Construct a hierarchical reinforcement learning scheduling model. The hierarchical reinforcement learning scheduling model includes a high-level PPO policy network that introduces an operational state temporal memory unit and a low-level attention-enhanced Dueling Double DQN scheduling network. Based on the unified state latent vector and scheduling credibility parameters, it outputs station-level scheduling action primitives, continuous control parameters, vehicle-pile matching actions, power allocation actions, and energy storage coordination actions to form candidate scheduling strategies. Step 5: Use the hierarchical constraint projection correction method to perform hard constraint verification, soft constraint optimization, and action correction on the candidate scheduling strategy to generate an executable scheduling strategy; Step Six: Execute executable scheduling strategies, collect actual waiting time, reservation fulfillment results, power point tracking error, service revenue, electricity purchase cost, energy storage cycle count, equipment temperature rise changes, and user evaluation results to form an operational feedback event flow; Step 7: Update the parameters of the hierarchical reinforcement learning scheduling model based on the operational feedback event flow, and update the weights of the operational evidence items.

2. The method for optimizing charging station operation scheduling based on reinforcement learning according to claim 1, characterized in that, Step one specifically includes: Collect station-level operation data, pile-level operation data, vehicle-level status data, and external environment data of charging stations within the current scheduling cycle; The station-level operation data, pile-level operation data, vehicle-level status data, and external environment data are timestamped according to a unified scheduling clock. The data after timestamp alignment is preprocessed. The preprocessing includes filling in missing data items according to the corresponding values ​​of the same data source at the previous sampling time, filling in data items that are missing for more than a preset number of sampling times according to the average of the data items in multiple consecutive scheduling cycles before the current scheduling cycle, removing data items that exceed the allowable range of the corresponding physical quantity, encoding and converting categorical data, and normalizing numerical data according to their respective historical value ranges. The preprocessed station-level operation data, pile-level operation data, vehicle-level status data, and external environment data are collected at the station, pile, vehicle, and environment levels to form the original status set.

3. The method for optimizing charging station operation scheduling based on reinforcement learning according to claim 1, characterized in that, Step two specifically includes: The original state set is divided into a station-level feature subset, a pile-level feature subset, a vehicle-level feature subset, and an environmental feature subset; The station-level feature subset is encoded to obtain the station-level encoding vector. The pile-level feature subset is arranged and encoded according to the charging pile identifier to obtain the pile-level encoding vector. The vehicle-level feature subset is arranged and encoded according to the vehicle arrival time and the scheduled arrival time to obtain the vehicle-level encoding vector. The environmental feature subset is encoded to obtain the environmental encoding vector. The station-level coding vector, pile-level coding vector, vehicle-level coding vector, and environment coding vector are concatenated in sequence to form the initial fusion vector; The initial fusion vector is subjected to dimensional unification using fully connected projection mapping to obtain a projection vector. The projection vector is then subjected to gated weighted fusion processing to generate a unified state latent vector.

4. The method for optimizing charging station operation scheduling based on reinforcement learning according to claim 1, characterized in that, Step three specifically includes: The arrival vehicle records in the original state set are statistically analyzed by time period and the trend of change are processed to obtain the short-term arrival traffic prediction value. The reserved vehicle records in the original state set are statistically processed for reserved arrival time, actual arrival time, no-arrival records and cancellation records to obtain the reservation fulfillment probability. The charging pile operation records in the original state set are processed for normal operation time ratio, reduced operation status and fault status to obtain the equipment availability probability. The station power records in the original state set are processed by deducting the total available power in the station, real-time output power and reserved power demand to obtain the current power margin. The energy storage operation records in the original state set are processed by the energy storage state of charge, maximum discharge power, and current discharge power to obtain the energy storage release capacity. The photovoltaic output records in the original state set are processed by the current real-time photovoltaic output, historical photovoltaic output level at the same time, and weather conditions to obtain the photovoltaic usable power. The user charging request records in the original state set are processed by the vehicle target state of charge, current state of charge, energy replenishment demand, and expected remaining stay time before departure to obtain the user urgency score. The short-term arrival traffic forecast, reservation fulfillment probability, equipment availability probability, current power margin, energy storage release capacity, photovoltaic usable power, and user urgency score are used as operational evidence items. The data integrity, variation range between adjacent scheduling cycles, and data collection time interval of each operational evidence item are processed to obtain the evidence usability. Each operational evidence item is processed by analyzing its historical deviation from similar values ​​and the magnitude of changes in the current scheduling cycle to obtain the volatility risk. Each operational evidence item is then combined with its corresponding evidence availability for credibility correction to obtain the first corrected evidence item. The first corrected evidence item is then combined with its corresponding volatility risk for risk suppression correction to obtain the second corrected evidence item. The second corrected evidence item is then weighted and aggregated to obtain the scheduling credibility parameter.

5. The method for optimizing charging station operation scheduling based on reinforcement learning according to claim 1, characterized in that, Step four specifically includes: The unified state latent vector and scheduling credibility parameters are input into the operation state temporal memory unit in the high-level PPO policy network. The current scheduling cycle input state and the previous scheduling cycle historical state are temporally correlated to obtain the station-level temporal state features. The operation state temporal memory unit includes an input gate, a forget gate, an output gate, and a memory state channel. The input gate performs weight control on the writing of the current scheduling cycle input state. The forget gate performs weight control on the retention of the previous scheduling cycle historical state. The memory state channel performs weighted accumulation on the current scheduling cycle input state and the previous scheduling cycle historical state. The output gate outputs the station-level temporal state features. The station-level time-series state characteristics are input into the high-level PPO policy network for policy calculation. The high-level PPO policy network includes a fully connected layer, a nonlinear activation layer and a policy output layer connected in sequence. The policy output layer includes a discrete action output branch and a continuous parameter output branch. The discrete action output branch outputs the probability distribution of station-level scheduling action primitives, and the continuous parameter output branch outputs continuous control parameters. The unified state latent vector, scheduling credibility parameters, station-level scheduling action primitives, and continuous control parameters are input into a low-level attention-enhanced Dueling Double DQN scheduling network. The low-level attention-enhanced Dueling Double DQN scheduling network includes an attention association unit, a feature fusion layer, a value branch, and a dominance branch. The attention association unit outputs vehicle-pile association features. The feature fusion layer concatenates and fuses the vehicle-pile association features with the unified state latent vector, scheduling credibility parameters, station-level scheduling action primitives, and continuous control parameters to obtain joint scheduling features. The value branch outputs the state value corresponding to the current scheduling state. The dominance branch outputs the action dominance value corresponding to each candidate action. The state value and the action dominance value are combined to obtain the action value corresponding to vehicle-pile matching action, power allocation action, and energy storage coordination action. The vehicle-pile matching actions, power allocation actions, and energy storage coordination actions are sorted and screened according to their action value. The station-level scheduling action primitives, continuous control parameters, vehicle-pile matching actions, power allocation actions, and energy storage coordination actions are then combined to form candidate scheduling strategies.

6. The charging station operation scheduling optimization method based on reinforcement learning according to claim 1, characterized in that, Step five specifically includes: The candidate scheduling strategies are parsed according to the station-level scheduling action primitives, continuous control parameters, vehicle-pile matching actions, power allocation actions and energy storage coordination actions to obtain the action set to be verified. The set of actions to be verified is subjected to hard constraint verification, which includes verification of transformer capacity upper limit, verification of total power upper limit in the station, verification of rated output power of single pile, verification of energy storage charge state boundary and verification of fault pile disabling. Actions that fail the hard constraint verification are removed to obtain a set of hard constraint candidate actions. The set of candidate actions with hard constraints is optimized with soft constraints. The soft constraint optimization includes optimization of appointment fulfillment target, waiting time target, equipment load balancing target, and peak-valley electricity price cost target. The action cost is calculated and sorted according to the action cost. The sorted set of hard-constraint candidate actions is modified to obtain the set of executable actions. The set of executable actions is reorganized according to the original field order of the candidate scheduling policy to generate an executable scheduling policy.

7. The method for optimizing charging station operation scheduling based on reinforcement learning according to claim 1, characterized in that, Step six specifically includes: The executable scheduling strategy is sent to the charging station control system to execute station-level scheduling action primitives, continuous control parameters, vehicle-charging station matching actions, power allocation actions, and energy storage coordination actions. The system collects data on the actual start time of vehicle charging, the actual arrival time of reserved vehicles, the actual output power, service order settlement data, electricity purchase settlement data, energy storage charging and discharging metering data, equipment temperature monitoring data, and user evaluation data. The actual waiting time is obtained by comparing the actual start time of the vehicle with the arrival time of the vehicle. The actual arrival time of the reserved vehicle is compared with the reservation record to obtain the reservation fulfillment result. The actual output power is compared with the target power value to obtain the power tracking error. The service order settlement data, electricity purchase settlement data, energy storage charging and discharging metering data, equipment temperature monitoring data and user evaluation data are statistically processed to obtain service revenue, electricity purchase cost, energy storage cycle number, equipment temperature rise change and user evaluation results. The actual waiting time, reservation fulfillment results, power tracking error, service revenue, electricity purchase cost, energy storage cycle count, equipment temperature rise changes, and user evaluation results are packaged into an operation feedback event flow.

8. The method for optimizing charging station operation scheduling based on reinforcement learning according to claim 1, characterized in that, Step seven specifically includes: The actual waiting time, reservation fulfillment results, power tracking error, service revenue, electricity purchase cost, energy storage cycle count, equipment temperature rise change and user evaluation results in the operation feedback event stream are extracted to construct a reward signal; The unified state latent vector, scheduling credibility parameters, candidate scheduling policies, executable scheduling policies, reward signals, and the unified state latent vector corresponding to the next scheduling cycle are associated and packaged to form training samples. The training samples are input into the hierarchical reinforcement learning scheduling model. The policy gradient of the high-level PPO policy network is updated, and the action value of the low-level attention-enhanced Dueling Double DQN scheduling network is updated to obtain the updated parameters of the hierarchical reinforcement learning scheduling model. The actual waiting time, reservation fulfillment results, power tracking error, service revenue, electricity purchase cost, energy storage cycle count, equipment temperature rise change, and user evaluation results in the operational feedback event stream are correlated with each operational evidence item, and the weight of the operational evidence items is adjusted according to the degree of influence of each operational evidence item on the reward signal.

Images