Electric vehicle charging and discharging scheduling optimization method and device based on PPO algorithm

Abstract

The invention relates to an electric vehicle charging and discharging scheduling optimization method and device based on a PPO algorithm, and the method comprises the steps: scanning the state of a charging station in real time, and collecting the current state space data when a preset event is detected; inputting the state space data into a pre-trained strategy network, and obtaining the initial charging and discharging power of each electric vehicle; if the sum of the initial charging and discharging power of the electric vehicles exceeds the upper limit of the total power of the charging station, calculating the priority of the electric vehicles and distributing the charging power of each electric vehicle, otherwise, directly distributing the charging power; the strategy network is trained based on a PPO algorithm, a cutting mechanism and a value network are introduced into training of the strategy network, the value network provides a value basis for the strategy network to output the optimal electric vehicle discharge power, and the strategy network is promoted to converge towards the maximum aggregator income direction. Compared with the prior art, the electric vehicle charging and discharging scheduling optimization can be performed in a targeted manner according to the characteristics of the scheduling environment.

Description

Technical Field

[0001] This invention relates to the field of new energy vehicle charging station operation strategies, and in particular to a method and device for optimizing electric vehicle charging and discharging scheduling based on the PPO algorithm. Background Technology

[0002] With the accelerated transformation and upgrading of the energy structure, the installed capacity of green and low-carbon energy sources such as wind power and photovoltaics is increasing year by year, and the number of new energy vehicles is also increasing. However, the strong intermittency of wind power and photovoltaics, together with the bi-peak characteristics of electric vehicle (EV) charging load, exacerbate the uncertainty on both the power system source and load sides, significantly increasing the difficulty of supply and demand balance regulation.

[0003] In recent years, Vehicle-to-Grid (V2G) technology has used electric vehicles as mobile energy storage devices, providing crucial flexibility resources to the power grid through bidirectional charging and discharging, thus becoming an important method for regulating the grid's supply and demand balance. However, aggregators face two major challenges when scheduling the charging and discharging of dispersed electric vehicles: highly random arrival times and demand from electric vehicle users, and time-of-use fluctuations in base electricity prices during peak and off-peak periods. How to effectively schedule the charging and discharging of different electric vehicles while meeting user needs is a critical problem that urgently needs to be solved.

[0004] Chinese patent application CN119476567A discloses a V2G electric vehicle charging station charging and discharging decision optimization method and device. Addressing the problem that existing technologies struggle to cope with complex and ever-changing charging environments and user behaviors, this method constructs an interactive environment for new energy vehicle charging stations based on V2G technology. It sets up state variables, action variables, and a reward function using reinforcement learning, and employs the SD3 algorithm to build a reinforcement learning-based charging and discharging decision model for electric vehicle charging stations. The model is trained using actual historical data to obtain the optimal decision model and calculate the optimal charging and discharging scheduling scheme. While this method achieves efficient optimization of charging station charging and discharging decisions in complex environments, it does not consider the upper limit of the charging station's carrying capacity, making it prone to planning power exceeding the actual available power. Furthermore, it only solves localized problems related to real-time scheduling at the operational level and cannot provide decision support for the long-term construction and transformation of the power grid. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a method and device for optimizing electric vehicle charging and discharging scheduling based on the PPO algorithm. This method and device can adapt to charging stations of different sizes, different user ratios, and different electricity price settings. The corresponding business needs can be met simply by changing the relevant internal parameters of the algorithm.

[0006] The objective of this invention can be achieved through the following technical solutions: A method for optimizing electric vehicle charging and discharging scheduling based on the PPO algorithm, the method comprising: Real-time scanning of charging station status; when a preset event is detected, current state space data is collected. The state space data is input into a pre-trained policy network to obtain the initial charging and discharging power of each electric vehicle. If the sum of the initial charging and discharging power of the electric vehicles exceeds the total power limit of the charging station, the priority of the electric vehicles is calculated, and the charging power of each electric vehicle is allocated according to the priority. Otherwise, the charging power is directly allocated. Based on the charging power allocated to each electric vehicle, the charging piles are controlled to complete the charging and discharging scheduling optimization. The policy network is trained based on the PPO algorithm. The training of the policy network incorporates a pruning mechanism and a value network. The value network provides a value basis for the policy network to output the optimal electric vehicle discharge power.

[0007] Furthermore, the preset events include electric vehicle entry, electric vehicle departure, and electricity price time period switching.

[0008] Furthermore, the current state space data includes the overall state and the electric vehicle state; The overall status includes the current electricity price period, the type of the next electricity price period, and the current real-time total load power of the charging station; The electric vehicle status includes the remaining departure time of each electric vehicle on site, the remaining charging capacity of each electric vehicle on site, the type of charging station connected to each electric vehicle on site, and the charging / discharging task type of each electric vehicle on site.

[0009] Furthermore, the process of obtaining the initial charging and discharging power of each electric vehicle through the strategy network includes: The overall state and the electric vehicle state are encoded separately, and key information in the spatiotemporal dimensions is extracted; The state codes of each electric vehicle are merged with the overall state code to obtain the encoding vector used to determine the power of the electric vehicle in the current state. Based on the encoding vector and neural network, the beta distribution parameters of each electric vehicle are output. and ,exist Internal sampling is used to obtain beta distribution sample values, and the power is scaled according to the task type of each electric vehicle to obtain the initial charging and discharging power of each electric vehicle.

[0010] Furthermore, the task types include charging tasks and discharging tasks; If the task type of the electric vehicle is a charging task, then the expression for scaling power is: in, Let i be the initial charge / discharge power of electric vehicle i at time t. Let n be the minimum charging power of the charging pile. The maximum charging power of charging pile n is given by [the value of n]. Let i be the beta distribution sample value of electric vehicle i; If the electric vehicle's task type is a discharge task, then the expression for scaling power is: in, Let n be the maximum discharge power of the charging pile.

[0011] Furthermore, the expression for calculating the priority of electric vehicles is as follows: in, Prioritization of electric vehicle i The maximum charging power of charging pile n is given by [the value of n]. Let be the remaining departure time of electric vehicle i at time t. Let represent the remaining charge capacity of electric vehicle i at time t.

[0012] Furthermore, the pre-training process of the policy network includes: Construct a policy network and a value network and initialize their parameters. Determine the number of training rounds and the number of repetitions for each training trajectory. Each training round simulates a day's charging station scheduling. Based on preset parameters, the initial charging pile configuration, electricity price rules, and vehicle sequence are generated to simulate the scheduling environment; Using the simulated scheduling environment, the policy network is trained based on the policy network loss, and the value network is trained based on the MSE loss. During the training of the policy network and the value network, the value network scores the state space data of the simulated scheduling environment for the current and next states, outputs the current state value estimate and the next state value estimate, calculates the advantage function and uses it as the key input of the policy network loss, and promotes the policy network to converge toward maximizing the aggregator's revenue.

[0013] Furthermore, the expression for the policy network loss is: in, The loss value of the policy network. Weights are sampled to indicate the importance of the probability differences between policies. Let be the dominant function in iteration t. This refers to the clipping hyperparameters; The expression for the MSE loss is: in, MSE loss is the loss value of the value network. For the number of iterations, For the value network to predict the value of the scheduling environment in iteration t, For the immediate benefits of environmental feedback; in, For the network's temporal difference error, As a discount factor, This is a hyperparameter used to balance bias and variance.

[0014] Furthermore, after optimizing the charge and discharge scheduling, the actual benefits obtained from this round of scheduling optimization, the charging power allocation results, and the state space data are recorded and stored. If the duration of continuous charging and discharging scheduling optimization reaches the preset statistical time threshold, the charging station operation statistics are obtained based on the stored data, and the total revenue during this period is calculated.

[0015] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the electric vehicle charging and discharging scheduling optimization method based on the PPO algorithm as described above.

[0016] Compared with the prior art, the beneficial effects of the present invention include: 1. The scheduling optimization algorithm proposed in this invention maximizes charging station revenue. It systematically applies the Near-End Policy Optimization (PPO) algorithm to solve the dynamic charging and discharging scheduling problem of electric vehicle fleets, transforming a complex, stochastic, and high-dimensional optimization problem into a Markov decision process that can autonomously learn through interaction between the agent and the environment. The algorithm designed in this invention makes new action choices when EVs enter or leave and when electricity prices change, i.e., it is event-triggered to adapt to the dynamic and stochastic nature of the scheduling environment, thereby improving the solution quality of the algorithm. It can provide continuous and adaptive optimal power allocation, enabling the algorithm to accurately capture dynamic changes in the environment, avoiding redundant calculations when nothing is happening, and preventing lag in response during drastic changes, thus significantly improving the quality and efficiency of the solution. This invention also addresses the charging station power limit problem. When the total power demand exceeds the limit, it proposes a priority index based on charging urgency, considering the required power and remaining time. It can intelligently identify vehicles with tight time and heavy tasks and prioritize power allocation, achieving a good balance between meeting user needs and maximizing aggregator revenue. It is a more refined scheduling strategy that conforms to actual operational logic.

[0017] 2. This invention designs a policy network that extracts and fuses the overall state code and the electric vehicle state code separately, enabling the network to simultaneously perceive global information and individual differences; the same set of network parameters can handle charging and discharging decisions for fast charging and slow charging, simplifying the model structure and significantly improving the convergence speed and generalization ability of the algorithm.

[0018] 3. The present invention introduces a pruning mechanism in the training of the policy network, which can effectively reduce the impact of destructive updates on training.

[0019] 4. By optimizing the charging and discharging timing, this invention can guide electric vehicles to charge during off-peak hours and discharge during peak hours, thereby smoothing the grid load, shaving peaks and filling valleys, and providing flexibility support for the grid.

[0020] 5. The training and application of the policy network and value network in this invention ensures optimal or near-optimal performance under different charging station scales, different user ratios, and different electricity pricing policies, demonstrating strong generalization ability.

[0021] 6. The algorithm proposed in this invention only needs to obtain two basic pieces of information: the current electricity price and the scale of the charging station. Furthermore, through the structural design of the algorithm, it can solve for any combination of information such as EV battery capacity, maximum power of the charging station, and maximum and minimum power of the charging pile. Attached Figure Description

[0022] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a schematic diagram of V2G scenario services in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the specific implementation framework of the electric vehicle charging and discharging scheduling optimization method of the present invention. Figure 4 This is a flowchart of the PPO algorithm of the present invention; Figure 5 This is a diagram of the strategy network structure of the present invention; Figure 6 This is a schematic diagram of electricity price setting in an embodiment of the present invention; Figure 7 This is a reward value convergence graph in an embodiment of the present invention; Figure 8 This is a convergence graph of network loss values ​​in an embodiment of the present invention; Figure 9 This is a box plot comparing the returns of different algorithms in this embodiment of the invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] Example 1 This embodiment discloses an electric vehicle charging and discharging scheduling optimization method based on the PPO algorithm, the method as follows: Figure 1 As shown, it includes steps S1-S4.

[0025] In this embodiment, the scheduling scenario is as follows: A commercial charging station operating by an aggregator, supporting V2G mode, has multiple intelligent fast and slow charging piles. Electric vehicles (EVs) can both charge through these piles and discharge in reverse. The charging and discharging power of each intelligent charging pile can be adjusted between maximum and minimum values. For example... Figure 2 As shown, the aggregator provides charging and discharging services for electric vehicle users. Users can choose to charge their electric vehicles or discharge from them. The electricity purchased by the aggregator from the reverse discharge of electric vehicles by users is directly supplied to electric vehicles that are charging at the same time. Users can also choose to purchase electricity from the grid to meet the charging needs of their electric vehicles. There is a total power limit for the charging and discharging operations of the entire charging station. .

[0026] The electricity price is set as follows: the power grid adopts a time-of-use pricing mechanism, dividing the day into off-peak, flat, and peak periods, each corresponding to a different base price. Aggregators add a service fee to the base price to determine the charging price for electric vehicle users, and also determine the discharging price of electric vehicles (i.e., the price at which the aggregator purchases electricity from electric vehicles) for different periods. In this embodiment, it is assumed that both the charging and discharging prices of electric vehicles adopt time-of-use pricing and are predetermined, and the electricity price parameters for the charging and discharging scheduling problem are known.

[0027] The user behavior is as follows: Electric vehicle users arrive randomly at any time of day. This could be due to the charging or discharging needs of employees working in the area, or it could be due to private cars or commercial vehicles coming to charge. In this embodiment, it is assumed that after arriving, the user will select their preferred charging station type for charging or discharging, and the charging / discharging will automatically disconnect after completion. Furthermore, if no charging station is available, the vehicle will leave on its own. The initial State of Charge (SOC) and battery capacity of the electric vehicle upon entry are known, as are the required SOC and planned departure time for the user.

[0028] The aggregator's revenue calculation is as follows: the aggregator receives revenue from user payments based on the amount of electric vehicle charging, while also paying the cost of purchasing electricity from the grid according to time-of-use pricing rules, and paying electric vehicle users the corresponding electricity purchase cost based on the discharge price and discharge volume. This embodiment assumes that the charging user's demand must be met. If the required SOC (State of Charge) is not reached by the user's designated departure time, the user is compensated for the insufficient SOC at departure. For electric vehicle discharge services, the aggregator is the buyer; therefore, the discharge user's demand may be partially met or not met at all.

[0029] In this embodiment, the scheduling problem is transformed into a Markov decision process (MDP). Given the electricity price and the scale of the charging station, the charging and discharging power of the EVs on site is used as the decision variable, and the optimization objective is to maximize the aggregator's revenue. The charging and discharging scheduling scheme at the current moment is determined, and after a natural day ends, the total revenue of the aggregator and its total power change curve for that day are output.

[0030] The specific process of each step in the electric vehicle charging and discharging scheduling optimization method based on the PPO algorithm includes: Step S1: Scan the charging station status in real time. When a preset event is detected, collect the current state space data.

[0031] Preset events include electric vehicle entry, electric vehicle departure, and electricity price time period switching.

[0032] The current state space data includes the overall state and the electric vehicle state; The overall status includes the current electricity price period, the type of the next electricity price period, and the current real-time total load power of the charging station; The electric vehicle status includes the remaining departure time of each electric vehicle on site, the remaining charging capacity of each electric vehicle on site, the type of charging station connected to each electric vehicle on site, and the charging / discharging task type of each electric vehicle on site.

[0033] Step S2: Input the state space data into the pre-trained policy network to obtain the preliminary charging and discharging power of each electric vehicle.

[0034] A pruning mechanism and a value network are introduced into the training of the policy network. Both the policy network and the value network are pre-trained based on the PPO algorithm. The pruning mechanism introduced in the training of the policy network can effectively reduce the impact of destructive updates on training, while the value network provides a value basis for the policy network to output the optimal electric vehicle discharge power. The trained policy network and value network constitute an intelligent agent, which can be directly deployed in charging stations in practical applications and can be trained in real time during actual use.

[0035] During the training of the policy network and the value network, the value network scores the state space data of the simulated scheduling environment for the current and next states, outputs the current state value estimate and the next state value estimate, calculates the advantage function and uses it as the key input for the policy network loss, and drives the policy network to converge toward maximizing the aggregator's revenue.

[0036] The PPO algorithm exhibits good stability and adaptability. It is a policy-based approach that directly and explicitly learns a target policy based on the sequence trajectories generated by the interaction between the current policy and the environment. It employs an Actor-Critic framework, achieving efficient training through collaborative optimization of two networks. The policy network (Actor) is responsible for extracting effective features from the complex state space and outputting the action policy, while the value network (Critic) scores the states and outputs state value estimates. The two networks have independent network parameters and update methods.

[0037] The pre-training process of the policy network is as follows: Figure 4 As shown, it includes: Algorithm initialization: Construct the policy network and value network and initialize their parameters; determine the number of training rounds (num_episodes) and the number of repetitions (epochs) for each training trajectory; each training round simulates a day's charging station scheduling; based on preset parameters, generate initial charging pile configurations, electricity price rules, and vehicle sequences to simulate the scheduling environment.

[0038] Action selection: Based on the current state, the strategy network calculates the charging and discharging power of each EV under the existing network parameters. However, the decision-making process must differentiate between charging and discharging operations. In addition, each charging station has its maximum and minimum charging power, which requires controlling the range of network output. Therefore, this method is designed with the algorithm network structure and corresponding power calculation method specifically to address the above characteristics.

[0039] The structure diagram of the policy network is as follows: Figure 5 As shown, firstly, the overall state and EV state in the state space are encoded separately. Key information in the spatiotemporal dimension is extracted through a fully connected layer. Then, the encoding of each EV state is merged with the encoding of the overall state to obtain an encoding vector used to determine the EV power in the current state.

[0040] To adapt to EVs undergoing different charging and discharging tasks and using different types of charging stations, the network output is first standardized to a uniform range before being processed accordingly based on its characteristics. Considering the characteristic that beta distribution sampling always involves continuous values ​​between 0 and 1, for each encoded vector, the parameters of the beta distribution are output separately through a neural network with the same structure. and ,exist Mid-sampling It distinguishes between charging and discharging tasks and performs power scaling using different formulas.

[0041] Once the power generated by each charging station exceeds the total power limit, This requires power allocation according to certain priority rules, distributing power in descending order of priority until the sum of the currently allocated power reaches a certain level. Up to that point. Priority Defined as the maximum power of each EV on site based on the charging station it is connected to. The time required for charging accounts for a portion of its remaining time away from the site. The ratio can be used to more accurately reflect the urgency of the current charging operation for each EV.

[0042] State Update: To address dynamically changing scheduling scenarios, this method sets the state update conditions to include EV arrivals, departures, and electricity price changes. Electricity price changes occur at fixed times throughout the day, but EV arrival and departure times are random, and the number of EVs arriving in a day is uncertain, thus increasing the difficulty of solving the model. Therefore, the following settings are made: This represents the time of each state update, and the set is used before all decisions are made. The number of elements and their sizes are unknown.

[0043] After each decision on the EV's charging / discharging power, the algorithm calculates the time remaining until the next price change based on the current time and power. Time remaining for the fastest departing EV The minimum value is taken as the next state update time. If new EVs enter the market during this period, their entry time will be used. As the state update time, i.e. .

[0044] After determining the EV charging and discharging power and the next update time, the algorithm will update all information such as price, time, and SOC accordingly, and record the actual benefits obtained from this round of decision-making.

[0045] Network parameter updates: After completing a training iteration, each parameter in the training trajectory is traversed. The quadruple is used to obtain the beta distribution probability function of the current state and the next state through the policy network, and the value is calculated using the value network. The timing difference error of the network is calculated according to the following formula. : in, As a discount factor, its value determines the degree to which future rewards influence current decisions. By setting a higher discount factor... Values ​​can reduce short-sightedness in decision-making by intelligent agents. This is a hyperparameter used to balance bias and variance.

[0046] Then, the dominance function is calculated recursively as shown in the following equation. hyperparameters Used to balance bias and variance: in, For the network's temporal difference error, This is a hyperparameter used to balance bias and variance.

[0047] After obtaining the complete sequence of advantage functions, as shown in the following equation, the policy network loss is calculated based on the advantage functions, where... The importance sampling weights are used to correct for probability differences between policies; while the pruning objective function is used to limit the update magnitude of network parameters within the range... Inside, among which This refers to the clipping hyperparameter. in, The loss value of the policy network. Weights are sampled to indicate the importance of the probability differences between policies. Let be the dominant function in iteration t. This refers to the clipping hyperparameters; For the value network, the MSE loss is used to calculate its estimation error. The expression for the MSE loss is: in, MSE loss is the loss value of the value network. For the number of iterations, For the value network to predict the value of the scheduling environment in iteration t, For the immediate benefits of environmental feedback.

[0048] Finally, the obtained network loss is backpropagated to update its parameters, thus ending one round of learning.

[0049] The Episode iteration and epoch parameter update are performed repeatedly. After each Episode, the reward value (total revenue of the aggregator on that day) and the dual network loss value are recorded. When the reward value is stable in a certain range for several consecutive rounds (such as 10 rounds) and the dual network loss value continues to decrease to a stable level, it indicates that the agent has learned a stable policy and the training can be terminated.

[0050] After training is terminated, the final policy network parameters and value network parameters are saved for subsequent scheduling applications of actual charging stations. In actual applications, the dual networks with these parameters are directly called without retraining.

[0051] Step S3: If the total initial charging and discharging power of electric vehicles exceeds the total power limit of the charging station, calculate the priority of electric vehicles and allocate charging power to each electric vehicle according to the priority order; otherwise, allocate charging power directly.

[0052] The process of obtaining the initial charge and discharge power of each electric vehicle through a policy network includes: The overall state and the electric vehicle state are encoded separately, and key information in the spatiotemporal dimensions is extracted; The state codes of each electric vehicle are merged with the overall state code to obtain the encoding vector used to determine the power of the electric vehicle in the current state. Based on the encoding vector and neural network, the beta distribution parameters of each electric vehicle are output. and ,exist Internal sampling is used to obtain beta distribution sample values, and the power is scaled according to the task type of each electric vehicle to obtain the initial charging and discharging power of each electric vehicle.

[0053] Task types include charging tasks and discharging tasks; If the task type of the electric vehicle is a charging task, then the expression for scaling power is: in, Let i be the initial charge / discharge power of electric vehicle i at time t. Let n be the minimum charging power of the charging pile. The maximum charging power of charging pile n is given by [the value of n]. Let i be the beta distribution sample value of electric vehicle i; If the electric vehicle's task type is a discharge task, then the expression for scaling power is: in, Let n be the maximum discharge power of the charging pile.

[0054] The formula for calculating the priority of electric vehicles is: in, Prioritization of electric vehicle i The maximum charging power of charging pile n is given by [the value of n]. Let be the remaining departure time of electric vehicle i at time t. Let represent the remaining charge capacity of electric vehicle i at time t.

[0055] Step S4: Based on the charging power allocated to each electric vehicle, control the charging pile to complete the charging and discharging scheduling optimization.

[0056] Record and store the actual benefits, charging power allocation results, and state space data obtained in this round of scheduling optimization.

[0057] If the duration of continuous charging and discharging scheduling optimization reaches the preset statistical time threshold, the charging station operation statistics are obtained based on the stored data, and the total revenue during this period is calculated.

[0058] Example 2 This embodiment, based on Embodiment 1 above, discloses a specific implementation example of an electric vehicle charging and discharging scheduling optimization method based on the PPO algorithm, such as... Figure 3 As shown, it is divided into three parts: scheduling environment, agent, and algorithm application. First, the scheduling environment is defined to simulate the real charging station environment, mainly including the setting of state space, dynamic space, and reward function. Second, the agent, namely the policy network and value network, is trained. The agent continuously explores in the set environment to find better policies and updates the relevant network parameters according to the training. If its reward value converges in each round, it means that a stable policy has been learned. After training, the optimal network parameters are saved. When the algorithm is applied to the actual charging station scheduling, the charging and discharging scheduling scheme can be solved by calling the agent with optimized parameters.

[0059] In this embodiment, the trained agent is compared with several common benchmark algorithms to demonstrate its superiority. In addition, to verify the universality of the designed algorithm, sensitivity analysis is performed on the set examples.

[0060] The charging station is equipped with 20 fast charging stations and 10 slow charging stations, with the fast charging stations having a charging power range of [missing information]. The discharge power range is The charging power range of the slow charging station is .

[0061] Referring to a charging station in a certain city, the following time periods are defined: 0:00-6:00 and 23:00-24:00 are off-peak hours; 6:00-8:00, 11:00-18:00 and 21:00-23:00 are normal hours; and 8:00-11:00 and 18:00-21:00 are peak hours. The corresponding base electricity price, retail electricity price, and purchase electricity price are set as follows: Figure 6 As shown.

[0062] The charging station is located in a commercial area, and its users are mainly employees who arrive during rush hours and individual travelers who arrive randomly. According to relevant research, the battery capacities of the users' EVs are set to the common 60kWh, 80kWh, and 100kWh. The number of EVs entering the station at different times and the proportion of fast charging, slow charging, and discharging are shown in Table 1 below. It can be seen that except for the period from 6:00 to 8:00 when employees mainly arrive for slow charging, the rest of the time is dominated by fast charging vehicles, and discharging EVs only appear during normal times and peak hours.

[0063] Table 1 Arrival Status of Electric Vehicles According to the established calculation example, after completing all decisions for the day, the network parameters of the policy network and value network are updated based on the trajectory, such as... Figure 7 As shown, the reward value gradually increases in the early stages of exploration, reaching a plateau around round 200, thus achieving convergence. Figure 8 As shown, the corresponding network loss value also converges in the same number of rounds, indicating that the policy network and value network have learned a stable policy.

[0064] To verify the algorithm's superiority, 20 sets of the above-defined computational examples were randomly generated. The trained policy network and value network were compared with benchmark algorithms for different objectives (focusing on earliest departure, maximum battery power, earliest arrival, battery-to-time ratio, and vehicle type, respectively). Unlike the reward function, the comparison's revenue value was calculated based on actual revenue, taking into account the losses caused by unmet user needs. Figure 9 As shown, the designed PPO algorithm outperforms the best benchmark strategy by an average of 9.22% in 20 randomized trials. To further verify the algorithm's versatility, sensitivity analysis was performed on the set examples to test the algorithm's performance in different scenarios. The charging station size, the proportion of users reached, and the electricity price were varied. The experimental results are shown in Tables 2, 3, and 4. It can be seen that the designed algorithm exhibits optimal performance in all scenarios, indicating that the algorithm can adapt to charging stations of different sizes, different user proportions, and different electricity price settings. Table 2. Algorithm benefits for charging stations of different sizes Table 3 Algorithm Revenue under Different User Ratios Table 4 Algorithm Revenue under Different Price Settings Unlike previous methods that updated the state at fixed intervals, this algorithm makes new action choices when electric vehicles enter or leave the country and when electricity prices change, in order to adapt to the dynamic and random nature of the scheduling environment, thereby improving the solution quality of the algorithm.

[0065] Example 3 Based on Embodiment 1, this embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the electric vehicle charging and discharging scheduling optimization method based on the aforementioned PPO algorithm.

[0066] At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the above-mentioned electric vehicle charging and discharging scheduling optimization method based on the PPO algorithm. Of course, in addition to the software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0067] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0068] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0069] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A PPO algorithm-based electric vehicle charging and discharging scheduling optimization method, characterized in that, The method comprises: Real-time scanning of the charging station state, collecting current state space data when a preset event is detected; Inputting the state space data into a pre-trained policy network to obtain the initial charging and discharging power of each electric vehicle; if the sum of the initial charging and discharging power of the electric vehicles exceeds the upper limit of the total power of the charging station, calculating the priority of the electric vehicles, and distributing the charging power of each electric vehicle according to the priority ranking, otherwise directly distributing the charging power; Based on the charging power allocated to each electric vehicle, control the charging pile to complete the charging and discharging scheduling optimization; The policy network is trained based on the PPO algorithm, and a clipping mechanism and a value network are introduced in the training of the policy network, and the value network provides a value basis for the output of the optimal electric vehicle discharging power of the policy network. 2.The PPO algorithm-based electric vehicle charging and discharging scheduling optimization method according to claim 1, characterized in that, The preset event includes electric vehicle entry, electric vehicle exit and electricity price period switching. 3.The PPO algorithm-based electric vehicle charging and discharging scheduling optimization method of claim 1, wherein, The current state space data includes the overall state and the electric vehicle state; The overall state includes the current electricity price period, the next electricity price period type and the current real-time total load power of the charging station; The electric vehicle state includes the remaining exit time of each on-site electric vehicle, the remaining charging capacity of each on-site electric vehicle, the access charging pile type of each on-site electric vehicle, and the charging and discharging task type of each on-site electric vehicle.

4. The PPO algorithm-based electric vehicle charging and discharging scheduling optimization method according to claim 3, characterized in that, The process of obtaining the initial charging and discharging power of each electric vehicle through the policy network comprises: Encoding the overall state and the electric vehicle state respectively to extract key information in the space-time dimension; Merge each electric vehicle state encoding and overall state encoding one by one to obtain an encoding vector for determining the power size of the electric vehicle under the current state; Based on the encoding vector and the neural network, beta distribution parameters of each electric vehicle are output and Beta distribution sampling values are obtained by sampling in the , and the initial charging and discharging power of each electric vehicle is obtained by scaling the power according to the task type of each electric vehicle.

5. The PPO algorithm-based electric vehicle charging and discharging scheduling optimization method according to claim 4, characterized in that, The task type includes charging task and discharging task; If the task type of the electric vehicle is a charging task, the expression of the scaled power is: wherein, is the preliminary charging and discharging power of the electric vehicle i at time t, is the minimum charging power of the charging pile n, is the maximum charging power of the charging pile n, is the beta distribution sampling value of the electric vehicle i; If the task type of the electric vehicle is a discharging task, the expression of the scaled power is: wherein, is the maximum discharge power for the charging column n.

6. The PPO algorithm-based electric vehicle charging and discharging scheduling optimization method according to claim 1, characterized in that, The calculation expression of the electric vehicle priority is: wherein, is a priority of the electric vehicle i, is a maximum charging power of the charging pile n, is a remaining departure time of the electric vehicle i at the time t, is a remaining charging electric quantity of the electric vehicle i at the time t.

7. The PPO algorithm-based electric vehicle charging and discharging scheduling optimization method according to claim 1, characterized in that, The pre-training process of the policy network comprises: Building a policy network and a value network and initializing the parameters of the policy network and the value network, determining the training rounds of training and the number of repeated learning of each training trajectory, and simulating a day of charging station scheduling in each round of training; Based on the preset parameters, generate an initial charging pile configuration, electricity price rule and vehicle sequence to simulate the scheduling environment; Based on the simulated scheduling environment, train the policy network based on the policy network loss and train the value network based on the MSE loss, in the training of the policy network and the value network, the value network scores the state space data of the simulated scheduling environment of the current and next state, outputs the current state value estimate and the next state value estimate, calculates the advantage function and uses it as a key input of the policy network loss to promote the policy network to converge to the direction of maximizing the aggregated merchant revenue. 8.The PPO algorithm-based electric vehicle charging and discharging scheduling optimization method of claim 7, wherein, The expression of the policy network loss is: wherein, is a loss value of the policy network, is an importance sampling weight for correcting probability difference between policies, is an advantage function in iteration t, is a clipping hyperparameter; The expression of the MSE loss is: wherein, is the MSE loss, i.e. the loss value of the value network, is the number of iterations, is the value prediction of the value network for the scheduled environment in iteration t, is the immediate reward of the environment feedback; wherein, is the timing difference error of the network, is the discount factor, is a hyperparameter for balancing bias and variance. 9.The PPO algorithm-based electric vehicle charging and discharging scheduling optimization method of claim 1, wherein, After the charging and discharging scheduling optimization, record the actual revenue, charging power allocation result and state space data obtained in this round of scheduling optimization and store them; If the time of the continuous charging and discharging scheduling optimization reaches the preset statistical time threshold, a charging station operation statistical result is obtained based on the storage data, and total income at this time period is calculated. 10.An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, The processor implements the steps of the electric vehicle charging and discharging scheduling optimization method based on the PPO algorithm according to any one of claims 1-9 when executing the computer program.

Images