Electric vehicle fleet daily charging scheduling method and system based on reinforcement learning

Abstract

The application discloses a method and system for day-ahead charging scheduling of an electric vehicle fleet based on reinforcement learning, and belongs to the field of electric vehicle charging scheduling. The method comprises the following steps: obtaining charging parameters of single electric vehicles, modeling charging behaviors of the single electric vehicles as local Markov decision processes, defining a heuristic scheduling function based on charging urgency, and constructing a local control model of the single electric vehicles; modeling a day-ahead decision process of an aggregator as a global Markov decision process, mapping high-dimensional state spaces of all single electric vehicles in the fleet into low-dimensional feature states through feature extraction; constructing a global optimization strategy model at the level of the aggregator; taking the sum of day-ahead electricity purchase cost and real-time imbalance penalty cost as the target, using a reinforcement learning algorithm to solve the global optimization strategy model, and outputting a day-ahead load plan; and the state transition of the global Markov decision process is updated based on the execution feedback of the heuristic scheduling function.

Description

Technical Field

[0001] This invention belongs to the field of electric vehicle charging scheduling, specifically relating to a day-ahead charging scheduling method and system for electric vehicle fleets based on reinforcement learning. Background Technology

[0002] With the rapid growth of electric vehicle ownership, the concentrated charging behavior of electric vehicle fleets has brought significant new load pressure to the power system, especially during peak hours, which may lead to grid instability and increased costs. Therefore, how to achieve intelligent scheduling and cost optimization control of electric vehicle charging behavior without affecting the user experience has become an important research topic in the intersection of smart grid and transportation energy.

[0003] Currently, most electric vehicle charging scheduling strategies adopt centralized optimization or rule-driven approaches. While they possess certain scheduling capabilities, they suffer from the following problems: lack of sensitivity to electricity price fluctuations, making it difficult to adapt to dynamic market environments; scheduling results lack good interpretability, making it difficult to deploy in real-world operational scenarios; and high computational complexity and poor scalability, making it difficult to meet the scheduling needs of large-scale fleets.

[0004] On the other hand, in the current electricity market, operators (such as fleet aggregators) usually need to submit their charging plans and electricity demand for the next day in advance. Faced with the high uncertainty of future electricity prices, traditional optimization methods are unable to accurately formulate flexible and economical multi-stage pricing and dispatch strategies.

[0005] Therefore, there is an urgent need for a new approach that is responsive, scalable, and deployable to reduce total cost and improve grid-friendly fleet charging strategies. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the present invention aims to provide a day-ahead charging scheduling method and system for electric vehicle fleets based on reinforcement learning, thereby solving the problems in the prior art.

[0007] The objective of this invention can be achieved through the following technical solutions: A reinforcement learning-based method for day-ahead charging scheduling of electric vehicle fleets includes the following steps: The charging parameters of a single electric vehicle are obtained, the charging behavior of the single electric vehicle is modeled as a local Markov decision process, and a heuristic scheduling function based on charging urgency is defined to construct a local control model for the single electric vehicle. Based on the local control model of individual electric vehicles, the day-ahead decision-making process of the aggregator is modeled as a global Markov decision-making process. Through feature extraction, the high-dimensional state space of all individual electric vehicles in the fleet is mapped to a low-dimensional feature state, which includes at least the total charging energy of the fleet. A global optimization strategy model at the aggregator level is constructed. Based on the global Markov decision process, with the goal of minimizing the sum of day-ahead electricity purchase cost and real-time imbalance penalty cost, a reinforcement learning algorithm is used to solve the global optimization strategy model and output the day-ahead load plan; and the state transition of the global Markov decision process is updated based on the execution feedback of the heuristic scheduling function.

[0008] Furthermore, the specific process of feature extraction is as follows: Define the state of a single electric vehicle i at time t as follows: ,in This indicates the energy that the electric vehicle has been charged; Define the low-dimensional feature state The sum of the charged energy of all online electric vehicles in the fleet: ; The system state of the global Markov decision process is represented as follows: Where t is the current time, This is the set of charging parameters for all online electric vehicles at the current moment.

[0009] Furthermore, the cost function corresponding to the objective is: in, These are the power plans submitted by aggregators to the market recently. To control the cycle duration, The current market electricity price. The total energy actually charged by the vehicle fleet between time t and t+1. The unbalanced price coefficient; This represents the cost of purchasing actual charging energy in the current market. This represents the opportunity cost incurred due to the discrepancy between the actual charging amount and the current plan.

[0010] Furthermore, the execution logic of the heuristic scheduling function includes: Receive total control power request from aggregator ; Based on the current charging energy and charging parameters of a single electric vehicle, calculate the minimum charging power for each electric vehicle. and maximum charging power ; Based on the minimum charging power Prioritize charging urgency, giving priority to electric vehicles with higher minimum charging power requirements, and allocate the total control power request accordingly. Distributed to individual electric vehicles.

[0011] Furthermore, the minimum charging power The calculation formula is: The maximum charging power The calculation formula is: in, To meet the demand for energy, For the current charged energy, For departure time, Due to the physical power limitations of the battery, The number of control periods within a unit market cycle. This indicates a positive value operation.

[0012] Furthermore, the state transition equation of the global Markov decision process is defined as: in, By heuristic scheduling function The determined power allocation to individual electric vehicles. For the power response of a single electric vehicle The subsequent actual charging action; the state transition equation characterizes the macroscopic characteristic states. The dynamic evolution of the execution results of heuristic scheduling at the micro-level.

[0013] The reinforcement learning-based electric vehicle fleet day-ahead charging scheduling system executes the above methods, including: Local control module: acquires the charging parameters of individual electric vehicles, models the charging behavior of individual electric vehicles as a local Markov decision process, defines a heuristic scheduling function based on charging urgency, and constructs a local control model for individual electric vehicles. Global model construction module: Based on the local control model of individual electric vehicles, the day-ahead decision-making process of the aggregator is modeled as a global Markov decision process. Through feature extraction, the high-dimensional state space of all individual electric vehicles in the fleet is mapped to a low-dimensional feature state, which includes at least the total charging energy of the fleet; and a global optimization strategy model at the aggregator level is constructed. Strategy solving module: Based on the global Markov decision process, with the goal of minimizing the sum of day-ahead electricity purchase cost and real-time imbalance penalty cost, the module uses reinforcement learning algorithm to solve the global optimization strategy model and outputs the day-ahead load plan; and the state transition of the global Markov decision process is updated based on the execution feedback of the heuristic scheduling function.

[0014] A computer storage medium storing a readable program that, when executed, instructs a computing device to perform the reinforcement learning-based day-ahead charging scheduling method for an electric vehicle fleet as described above.

[0015] An electronic device includes: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; The memory is used to store at least one executable instruction that causes the processor to perform operations corresponding to the reinforcement learning-based day-ahead charging scheduling method for electric vehicle fleets described above.

[0016] A computer program product includes computer instructions that instruct a computing device to perform operations corresponding to the reinforcement learning-based day-ahead charging scheduling method for electric vehicle fleets described above.

[0017] The beneficial effects of this invention are: 1. Compared with the traditional centralized optimization or rule-driven approach to electric vehicle charging scheduling strategies, this invention adopts an aggregation-level policy learning framework based on reinforcement learning and a heuristic policy mapping mechanism, which can respond sensitively to electricity price fluctuations and adapt to dynamic electricity price environments; the scheduling results have good interpretability and can be deployed to real operation scenarios.

[0018] 2. By optimizing the calculation method, a multi-stage day-ahead charging scheduling scheme with minimal cost and feasibility is proposed. The proposed model provides technical support for aggregators to submit cost-effective electricity purchase quotations to the day-ahead market and obtain the best overall benefits when the charging flexibility of individual electric vehicles is unknown. It can meet the needs of large-scale fleet scheduling. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of the electric vehicle fleet day-ahead charging scheduling method of the present invention; Figure 2 This is a schematic diagram of the three-step scheduling algorithm for electric vehicle fleets based on reinforcement learning, as presented in this invention. Figure 3 This is a schematic diagram of the charge / discharge bidding curves for different time periods of the present invention; Figure 4 This is a schematic diagram of the optimal charging and discharging bidding strategy of the present invention; Figure 5This is a schematic diagram comparing the planned power output of the day with the real-time power allocation of electric vehicles according to the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Example 1 like Figure 1 As shown, the day-ahead charging scheduling method for electric vehicle fleets based on reinforcement learning includes the following steps: S1. Obtain the charging parameters of a single electric vehicle, model the charging behavior of the single electric vehicle as a local Markov decision process, define a heuristic scheduling function based on charging urgency, and construct a local control model for the single electric vehicle. The specific process for constructing the local control model of a single electric vehicle is as follows: The main stakeholders in this invention are electric vehicle fleet aggregators. The aggregator's decision-making process is divided into two stages.

[0023] In the first decision phase (D-1 day), the aggregator predicts the energy required to charge its electric vehicles on D day and purchases that amount of energy in the day-ahead market; in the second decision phase (D day), the aggregator communicates with the electric vehicles to control their charging based on the amount of energy purchased in the first decision phase in the day-ahead market.

[0024] In the first decision-making phase (Day D-1), the aggregator decides how much energy to purchase in the day-ahead market; in the day-ahead market, energy purchases are made in each market session of the following day (Day D); the energy purchased by the aggregator day-ahead... This can be summarized as the day-ahead load plan: (1) Where H represents the total number of market hours in a day.

[0025] The decision to formulate day-ahead load plans is driven by two factors. First, the cost of purchasing load plans in the day-ahead market should be based on minimizing the day-ahead price. According to the market cycle h: (2) Secondly, load planning must not allow for imbalances; that is, in the second decision-making phase, the planned energy should be able to be charged by the electric vehicle fleet without any imbalances.

[0026] At the end of the first decision-making phase, the load plan in formula (1) The load plan for day D has already been purchased in the market recently; In the first decision-making phase (Day D-1), the aggregation chamber communicates with the electric vehicle to control its charging power.

[0027] In this invention, each market cycle h is divided into multiple equally spaced control cycles; for each control cycle t, the aggregator requests power from each electric vehicle in its fleet. It can be summarized as follows: (3) in, For the first i Electric vehicles in control cycle t Controlled charging power, This represents the total number of electric vehicles in the fleet. The total number of control periods divided into the entire day; Based on these requests, the grid-connected electric vehicle determines its actual charging power locally. ,in In this invention, Defined as a strategy function that ensures the user's requirements for battery charging state are met, while adhering as closely as possible to the control power requested by the aggregator. Based on all requested power values Electric vehicle fleets in each market cycle Charging amount for: (4) Where the function The market cycle is mapped to its respective control cycle, where H is the total number of market sessions in a day.

[0028] For each control cycle, the aggregator determines the control power for each electric vehicle. To make this decision, it is assumed that the aggregator can request the current state of all electric vehicles before each control cycle.

[0029] The decision for each control cycle is based on minimizing the daytime load plan in equation (1). The imbalance between the actual load in equation (4) and the actual load. If a negative imbalance occurs (charging amount is greater than the day-ahead market purchase amount), Then the aggregator must base its decisions on the negative imbalance price. Pay additional fees. If a positive imbalance occurs (charging volume is less than the current day's market purchase volume), Then the aggregator will base its decisions on the positive imbalance price. You will receive a refund. Because the positive imbalance price is lower than the day-ahead price, aggregators can only receive a partial refund for the excess electricity they purchased.

[0030] In the current market, based on the imbalance price, a complete cost function can be defined: (5) in: (6) (7) Among them, the negative imbalance price is higher than the known day-ahead price. The positive imbalance price is lower than the known day-ahead price. . The choice is based on the typical difference between day-ahead prices and imbalance prices in the electricity market under consideration.

[0031] In the second decision phase (Day D), the aggregator controls the charging of its electric vehicles to follow the day-ahead power plan defined in the first decision phase: (8) Aggregators follow this power plan as strictly as possible by using a three-step scheduling algorithm.

[0032] The first step is for the scheduling algorithm to collect status information from the electric vehicle fleet and planned power values ​​from the aggregator. ; In the second step, the scheduling algorithm uses this information to calculate the control power value in formula (3). ; Third, each control power value is communicated to the corresponding electric vehicle, which uses the control power value as input to its local decision-making process.

[0033] Before explaining the scheduling algorithm in detail, let's first introduce the local decision-making process.

[0034] electric vehicles Its charging behavior is based on its Daily charging parameters ; (9) in, For the first The set of charging parameters for a vehicle on day D. If a vehicle does not charge on that day, it is denoted as an empty set. ; and These represent their arrival and departure times, respectively. It is at its departure time The required energy; These are power limits defined by the electric vehicle battery management system.

[0035] therefore, yes The maximum number of electric vehicles that can be reached within a day; based on , Indicates that electric vehicles are included. In time Available charging parameters: (10) To model the local decision-making of electric vehicles, their charging behavior is represented as a Markov Decision Process (MDP); the state space of the electric vehicle... Composed of charging energy, defined as: (11) in, It is at its departure time Energy required; electric vehicles A complete charging cycle begins at arrival time initial state Ending at departure time Terminal status .

[0036] Action space A consists of all charging actions, and is defined as follows: (12) in, It is a dimension of The action vector represents the real-time charging decisions for all electric vehicles in the fleet; It is a vector The first in The element represents the element. The charging power of electric vehicles at the current pace; These are power limits defined by the electric vehicle battery management system.

[0037] The system dynamics state transitions of an electric vehicle can be described as follows: (13) in This refers to the length of the control cycle. These transitions can only occur between the arrival and departure times of the electric vehicle. Electric vehicle strategy. Charge the battery before departure, while adhering as closely as possible to the charging power requested by the collector. : (14) in, This represents the set of charging parameters available for electric vehicles during that time period.

[0038] For each electric vehicle, the strategy is based on charging energy. Charging parameters and request charging power Define an action (15) (16) in, It is reached at a certain time Minimum power required and Indicates maximum power, subject to and charging energy Limitations. These constraints ensure the effective charging power of electric vehicles.

[0039] The core of the scheduling performed by the aggregator is the scheduling algorithm. Its functional form is: (17) The algorithm uses the current time Charging energy The electric vehicle parameters t and daytime power in equation (10) As input, the scheduling algorithm calculates the charging priority for each electric vehicle (EV), which serves as a heuristic to allocate power among EVs based on their charging urgency. For example, an EV with a depleted battery typically has a higher priority than one with a nearly full battery. The algorithm's output is the controlled power for each EV. This control power In the third step, this is communicated to the corresponding electric vehicle. Finally, the electric vehicle responds according to the strategy in equation (14). Calculate its actual charging power.

[0040] S2, based on the local control model of individual electric vehicles, the day-ahead decision-making process of the aggregator is modeled as a global Markov decision-making process. Through feature extraction, the high-dimensional state space of all individual electric vehicles in the fleet is mapped to a low-dimensional feature state, which includes at least the total charging energy of the fleet; and a global optimization strategy model at the aggregator level is constructed. The specific process of establishing a global charging behavior optimization strategy model at the aggregator level is as follows: In the first decision-making phase (day D-1), the aggregator determines the day-ahead plan to be submitted. This decision-making process is formalized as a Markov Decision Process (MDP), with its state space as follows: The action space is .exist In the middle, controlling the action is Discretization, where It is the maximum permissible total charging power determined by the power grid. Control actions can only affect state-space components. (Electric vehicle charging energy in Equation (11)). The uncontrollable state space component (electric vehicle charging parameters in Equation (10)) is caused by random perturbation. The determination is characterized by a joint probability distribution. . and The state transitions are as follows: (18) (19) in, Indicates that the system is in The state at time t is defined as follows: Substitute (17) into (14) and substitute (14) into (13).

[0041] The cost signal in this MDP is obtained by using state variables. and control actions Substituting into the aggregator cost function (5), we get: (20) in, Indicates the first The cost function of the step; , and These are in the corresponding market cycles The price is defined internally, and .

[0042] The goal of the first decision-making phase is to find an optimal open-loop strategy. This strategy is based solely on the initial state of the system. Selecting actions to control time t ( The resulting actions define a complete day plan. This plan minimizes the expected cost of phase T: (twenty one) in, Represents the initial state The optimal expected total cost; This indicates the optimal control power action determined recently; This represents the set of charging parameters available for electric vehicles during that time period.

[0043] For any real number of electric vehicles (N ≥ 10), the curse of dimensionality quickly makes MDPs intractable. To address this, a feature extraction method is used to transform the state... Mapped to Defined as: (twenty two) This refers to the total charging energy of the electric vehicle fleet. Therefore, the new system state can be used... This invention describes the problem by extracting features, transforming the control problem into an imperfect state information problem. This invention uses only... The current value.

[0044] Based on features Substituting into (6) and (7), the cost function in (20) can be written as: (twenty three) in, It is the electric vehicle fleet in time and The total energy consumed during charging. The first part of the formula ( This refers to the cost of purchasing actual charging energy in the day-to-day market (in the second decision phase). Part Two ( This represents the opportunity cost of buying too little (negative imbalance) or too much (positive imbalance) in the current market. Therefore, when At this point, the opportunity cost of the optimal day-ahead scheduling is zero. This is an important result because it implies the optimal policy under deterministic conditions: (twenty four) This holds approximately true in random cases, and this property of the optimal strategy will be utilized in the algorithm.

[0045] S3, based on the global Markov decision process, with the goal of minimizing the sum of day-ahead electricity purchase cost and real-time imbalance penalty cost, the global optimization strategy model is solved using a reinforcement learning algorithm to output the day-ahead load plan; and the state transition of the global Markov decision process is updated based on the execution feedback of the heuristic scheduling function.

[0046] like Figure 2 As shown, the process of finding the optimal policy includes: The first step involves the scheduling algorithm collecting state information from the electric vehicle fleet and planned power values ​​from the aggregator. In this step, the aggregator first constructs charging and discharging bidding curves for each time period based on demand bidding prediction data from market participants. The low-dimensional feature state, composed of the total charging energy of the fleet, is observed by an agent in a reinforcement learning algorithm. The strategy gradient algorithm is used to evaluate the load action for different day-ahead periods. The immediate costs incurred. The agent establishes a mapping relationship between states and optimal actions through multiple rounds of iterative training in a simulated environment. Guided by maximizing long-term cumulative returns, it determines the optimal purchase volume that can balance electricity purchase costs and imbalance risks in each time period, thereby outputting a complete day-ahead load plan.

[0047] In the second step, the scheduling algorithm uses this information to calculate the control power value in equation (3). The scheduling algorithm uses the generated day-ahead load plan as the total power reference benchmark, combined with the state of charge, expected departure time, and charging power demand of each electric vehicle in the fleet during the current control cycle. The algorithm uses a heuristic scheduling function based on charging urgency to calculate the scheduling priority of each electric vehicle, and dynamically allocates the day-ahead planned power to each individual controlled charging unit according to the priority order. This allocation process fully considers the power limit constraints of individual electric vehicles, and ensures through hierarchical control logic that while fitting the day-ahead plan, the total load of the fleet prioritizes meeting the needs of vehicles that have run out of power or are about to leave the site.

[0048] Third, each control power value is communicated to the corresponding electric vehicle, which uses this value as input to its local decision-making process. Each individual electric vehicle executes the actual charging action according to its local Markov decision process, and feeds back the actual energy state and executed power consumed during that cycle to the scheduling algorithm in real time. The scheduling algorithm updates the state transition probabilities of the global Markov decision process in real time based on the deviation between the actual load and the day-ahead plan, using the remaining energy demand of the fleet after actual execution as the starting state for the next control cycle. This closed-loop update mechanism based on execution feedback enables the reinforcement learning algorithm to correct subsequent decisions in real time based on imbalances in previous cycles, and ultimately minimizes the sum of day-ahead costs and real-time penalty costs by dynamically adjusting the load allocation logic.

[0049] Based on a similar inventive concept, embodiments of the present invention also provide a computer storage medium storing a readable program that, when run by a processor, can execute the above-described reinforcement learning-based electric vehicle fleet day-ahead charging scheduling method.

[0050] Based on a similar inventive concept, this invention provides an electronic device, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the above-described reinforcement learning-based electric vehicle fleet day-ahead charging scheduling method.

[0051] Based on a similar inventive concept, embodiments of the present invention also provide a computer program product, including computer instructions, which instruct a computing device to perform the operations corresponding to the above-described reinforcement learning-based electric vehicle fleet day-ahead charging scheduling method.

[0052] Example 2 In this embodiment, the above-mentioned reinforcement learning-based day-ahead charging scheduling method for electric vehicle fleets is adopted to adjust the charging strategy of the charging station's cooperative fleet based on internal sample data of a certain charging station company.

[0053] The solution process is as follows: The aggregator first obtains load forecasts and electricity price fluctuation data from other market participants. Based on this, it models the day-ahead electricity purchase cost using a global Markov decision process. The first step is to construct the bidding curve: by analyzing historical market clearing data, construct the charging and discharging bidding curves for each time period. Figure 3 The figure shows that the marginal electricity price increases in a stepped manner as power increases. The conclusion is that when the fleet's real-time charging load exceeds the day-ahead plan, the aggregator must bear a higher tier of load penalty cost; therefore, the bidding curve provides a cost boundary constraint for subsequent reinforcement learning.

[0054] The second step is optimal bidding determination: Based on the known bidding curve, a global optimization strategy model is solved using a reinforcement learning algorithm. This yields the final output optimal charging / discharging bidding strategy. Figure 4 The conclusion is that this strategy works in different electricity price ranges (such as...). The department found a balance between electricity purchase costs and imbalance risks, and determined the optimal planned purchase volume for each time period in the day-ahead market. .

[0055] After determining the day-ahead plan, the aggregator enters the real-time scheduling phase, using a three-step scheduling algorithm to allocate the planned power to individual electric vehicles.

[0056] Step 3, Power Comparison Analysis: Comparison chart of planned power and real-time power allocation of electric vehicles (…) Figure 5 The thick black outline represents the day-ahead planned power determined in the first phase. The scheduling algorithm is based on the charging urgency of each vehicle (as determined by...). The decision is made to make fine-grained allocations within each control cycle (1-20).

[0057] At the end of the solution process, the system compares the planned power for the day with the real-time power allocation of electric vehicles using a graph. Figure 5 The real-time allocation feedback reflected in the data continuously updates the state transition probabilities of the global Markov decision process. In this way, the model can learn the fleet behavior characteristics under different date types, enabling it to predict the bidding curve for the next day. Figure 3 ) and optimal strategy ( Figure 4 More accurate.

[0058] Example 3 In this embodiment, a day-ahead charging scheduling system for electric vehicle fleets based on reinforcement learning is proposed, comprising: Local control module: acquires the charging parameters of individual electric vehicles, models the charging behavior of individual electric vehicles as a local Markov decision process, defines a heuristic scheduling function based on charging urgency, and constructs a local control model for individual electric vehicles. Global model construction module: Based on the local control model of individual electric vehicles, the day-ahead decision-making process of the aggregator is modeled as a global Markov decision process. Through feature extraction, the high-dimensional state space of all individual electric vehicles in the fleet is mapped to a low-dimensional feature state, which includes at least the total charging energy of the fleet; and a global optimization strategy model at the aggregator level is constructed. Strategy solving module: Based on the global Markov decision process, with the goal of minimizing the sum of day-ahead electricity purchase cost and real-time imbalance penalty cost, the module uses reinforcement learning algorithm to solve the global optimization strategy model and outputs the day-ahead load plan; and the state transition of the global Markov decision process is updated based on the execution feedback of the heuristic scheduling function.

[0059] The methods of the present invention can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code originally stored on a remote recording medium or a non-transitory machine-readable medium and subsequently stored on a local recording medium, downloaded via a network. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses the code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for performing the methods shown herein.

[0060] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A day-ahead charging scheduling method for electric vehicle fleets based on reinforcement learning, characterized in that, Includes the following steps: The charging parameters of a single electric vehicle are obtained, the charging behavior of the single electric vehicle is modeled as a local Markov decision process, and a heuristic scheduling function based on charging urgency is defined to construct a local control model for the single electric vehicle. Based on the local control model of individual electric vehicles, the day-ahead decision-making process of the aggregator is modeled as a global Markov decision-making process. Through feature extraction, the high-dimensional state space of all individual electric vehicles in the fleet is mapped to a low-dimensional feature state, which includes at least the total charging energy of the fleet. A global optimization strategy model at the aggregator level is constructed. Based on the global Markov decision process, with the goal of minimizing the sum of day-ahead electricity purchase cost and real-time imbalance penalty cost, a reinforcement learning algorithm is used to solve the global optimization strategy model and output the day-ahead load plan; and the state transition of the global Markov decision process is updated based on the execution feedback of the heuristic scheduling function.

2. The day-ahead charging scheduling method for electric vehicle fleets based on reinforcement learning according to claim 1, characterized in that, The specific process of feature extraction is as follows: Define the state of a single electric vehicle i at time t as follows: ,in This indicates the energy that the electric vehicle has been charged; Define the low-dimensional feature state The sum of the charged energy of all online electric vehicles in the fleet: ; The system state of the global Markov decision process is represented as follows: Where t is the current time, This is the set of charging parameters for all online electric vehicles at the current moment.

3. The day-ahead charging scheduling method for electric vehicle fleets based on reinforcement learning according to claim 1, characterized in that, The cost function corresponding to the objective is: in, These are the power plans submitted by aggregators to the market recently. To control the cycle duration, The current market electricity price. The total energy actually charged by the vehicle fleet between time t and t+1. The unbalanced price coefficient; This represents the cost of purchasing actual charging energy in the current market. This represents the opportunity cost incurred due to the discrepancy between the actual charging amount and the current plan.

4. The day-ahead charging scheduling method for electric vehicle fleets based on reinforcement learning according to claim 1, characterized in that, The execution logic of the heuristic scheduling function includes: Receive total control power request from aggregator ; Based on the current charging energy and charging parameters of a single electric vehicle, calculate the minimum charging power for each electric vehicle. and maximum charging power ; Based on the minimum charging power Prioritize charging urgency, giving priority to electric vehicles with higher minimum charging power requirements, and allocate the total control power request accordingly. Distributed to individual electric vehicles.

5. The day-ahead charging scheduling method for electric vehicle fleets based on reinforcement learning according to claim 4, characterized in that, The minimum charging power The calculation formula is: The maximum charging power The calculation formula is: in, To meet the demand for energy, For the current charged energy, For departure time, Due to the physical power limitations of the battery, The number of control periods within a unit market cycle. This indicates a positive value operation.

6. The day-ahead charging scheduling method for electric vehicle fleets based on reinforcement learning according to claim 2, characterized in that, The state transition equation of the global Markov decision process is defined as follows: in, By heuristic scheduling function The determined power allocation to individual electric vehicles. For the power response of a single electric vehicle The subsequent actual charging action; the state transition equation characterizes the macroscopic characteristic states. The dynamic evolution of the execution results of heuristic scheduling at the micro-level.

7. A day-ahead charging scheduling system for electric vehicle fleets based on reinforcement learning, comprising the method described in any one of claims 1-6, characterized in that, include: Local control module: acquires the charging parameters of individual electric vehicles, models the charging behavior of individual electric vehicles as a local Markov decision process, defines a heuristic scheduling function based on charging urgency, and constructs a local control model for individual electric vehicles. Global model construction module: Based on the local control model of individual electric vehicles, the day-ahead decision-making process of the aggregator is modeled as a global Markov decision process. Through feature extraction, the high-dimensional state space of all individual electric vehicles in the fleet is mapped to a low-dimensional feature state, which includes at least the total charging energy of the fleet; and a global optimization strategy model at the aggregator level is constructed. Strategy solving module: Based on the global Markov decision process, with the goal of minimizing the sum of day-ahead electricity purchase cost and real-time imbalance penalty cost, the module uses reinforcement learning algorithm to solve the global optimization strategy model and outputs the day-ahead load plan; and the state transition of the global Markov decision process is updated based on the execution feedback of the heuristic scheduling function.

8. A computer storage medium storing a readable program, characterized in that, When the program runs, it can instruct the computing device to execute the reinforcement learning-based day-ahead charging scheduling method for electric vehicle fleets as described in any one of claims 1-6.

9. An electronic device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation corresponding to the day-ahead charging scheduling method for electric vehicle fleets based on reinforcement learning as described in any one of claims 1-6.

10. A computer program product comprising computer instructions, characterized in that, The computer instructions instruct the computing device to perform the operations corresponding to the day-ahead charging scheduling method for electric vehicle fleets based on reinforcement learning as described in any one of claims 1-6.

Images