An Optimization Method for Hybrid Charging Schemes of Electric Buses Based on Deep Reinforcement Learning

By optimizing the charging scheme for electric buses through deep reinforcement learning, the problems of random fluctuations in photovoltaic power generation and dynamic environmental changes have been solved, thereby increasing the proportion of photovoltaic power in the public transportation system and reducing costs.

CN118690917BActive Publication Date: 2024-11-15JILIN UNIVERSITY +1

Patent Information

Application Number
CN202411139683.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-11-15
Estimated Expiration
2044-08-20

AI Technical Summary

Technical Problem

The existing photovoltaic-energy storage-grid hybrid power supply mode has failed to effectively cope with the random fluctuations in photovoltaic power generation and the dynamic changes in the external environment, resulting in the bus charging scheme not achieving optimal results, with low photovoltaic power ratio and high charging costs.

Method used

An optimization method for hybrid charging schemes of electric buses based on deep reinforcement learning is adopted. By defining basic variables and constraints, the problem is transformed into a Markov control problem. The charging schedule is optimized using a dual-delay deep deterministic policy gradient algorithm to generate an adaptive charging strategy.

Benefits of technology

While ensuring a stable power supply, the proportion of clean energy has been increased, reducing carbon emissions and operating costs of the public transportation system and optimizing energy efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118690917B_ABST
    Figure CN118690917B_ABST
Patent Text Reader

Abstract

The present invention relates to a method for optimizing hybrid charging schemes for electric buses based on deep reinforcement learning. In order to solve the problem that the existing hybrid power supply mode of "photovoltaic-energy storage-grid" ignores the random fluctuations of photovoltaic power generation and is difficult to fully cope with the dynamic changes of the external environment, the generated bus charging scheme often cannot achieve the optimal effect in practice, resulting in a low proportion of photovoltaic power in the power consumed by the bus system and high charging costs. The process is: determining the charging scheduling objectives and constraints of the bus system; converting the charging scheduling problem of the bus system into a Markov control problem; solving the problem to determine the network parameters; inputting the state of the intelligent system to be tested in the t period and the action in the #imgabs0# period into the network, outputting a set of recommended actions, and obtaining an actual action set; converting the actual action set into a charging scheme and executing it. The present invention is used in the field of charging optimization for electric buses.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a method for optimizing a hybrid charging scheme for an electric public transport vehicle. Background Art

[0002] Photovoltaic power generation, as a clean energy supply technology, has significant advantages in reducing carbon emissions from power systems. Urban bus stations cover a large area and are suitable for the deployment of photovoltaic power generation facilities and the provision of photovoltaic power for buses. However, there are random fluctuations in photovoltaic power generation, and energy storage devices can be configured synchronously at bus stations to store some photovoltaic power. In addition, when photovoltaic power generation is insufficient to maintain the power required for electric bus operations under severe weather conditions, the power grid can also be used to charge the buses. In this case, the electric bus system has a hybrid power supply mode of "photovoltaic-energy storage-grid", which is of great significance for increasing the proportion of clean energy in the electric bus system, reducing carbon emissions and operating costs of the bus system.

[0003] However, in this mode, the charging behavior of buses is not only affected by the route schedule and route mileage, but also by various dynamic factors such as weather conditions, photovoltaic power generation, and the remaining battery power of buses. Therefore, it is necessary to consider multiple factors to optimize the charging timing and charging method of buses, so as to increase the proportion of photovoltaic power in the power consumed by the bus system and reduce the charging cost. However, the existing technology ignores the random fluctuations of photovoltaic power generation, and it is difficult to fully cope with the dynamic changes of the external environment (such as solar radiation, bus batteries, and the remaining power of energy storage batteries). The generated bus charging schemes often fail to achieve the optimal effect in practice. The present invention comprehensively considers the randomness of photovoltaic power generation and the uncertainty of external environmental factors, and proposes an optimization method for hybrid charging schemes of electric buses based on deep reinforcement learning. Summary of the invention

[0004] The purpose of the present invention is to solve the problem that the existing "photovoltaic-energy storage-grid" hybrid power supply mode ignores the random fluctuations of photovoltaic power generation and is difficult to fully cope with the dynamic changes of the external environment. The generated bus charging scheme often fails to achieve the optimal effect in practice, resulting in a low proportion of photovoltaic power in the electricity consumed by the bus system and high charging costs. A hybrid charging scheme optimization method for electric buses based on deep reinforcement learning is proposed.

[0005] The specific process of the hybrid charging scheme optimization method for electric buses based on deep reinforcement learning is as follows:

[0006] Step 1. Determine the charging scheduling objectives and constraints of the public transportation system; the specific process is as follows:

[0007] Step 1.1. Define basic variables; the specific process is:

[0008] Step 1.1.1. The number of bus routes operated by the bus station with photovoltaic facilities is I, and each route is equipped with K electric buses, which requires N trips per day;

[0009] There are O charging piles in the station, and the charging power of each charging pile is , The unit is kW;

[0010] The rated capacity of the battery of electric bus k is recorded as , The unit is kWh, ;

[0011] The battery health status of bus k is recorded as , The unit is %;

[0012] The rated capacity of the battery in the energy storage device is recorded as , The unit is kWh;

[0013] The battery health status of the energy storage device is recorded as , The unit is %;

[0014] The photovoltaic layout area is recorded as A, and the unit of A is m 2 ;

[0015] Step 1.1.2. is the time interval, and the whole day is divided into T time periods. ;

[0016] Indicates rounding down. The unit is min;

[0017] The starting time of period t is recorded as The end time is recorded as , ;

[0018] Step 1.1.3. Let the photovoltaic power generation in period t be , The unit is kW;

[0019] The solar radiation in period t is expressed as , The unit is kW / m 2 ;

[0020] Put the bus in The battery state of charge SOC at the time is recorded as ;

[0021] The energy storage device The battery state of charge SOC at the time is recorded as ;

[0022] Step 1.1.4. Place bus k in The set of shifts that have been completed before time is defined as ;

[0023] If bus k is in motion during period t, the number of bus trips during period t is recorded as ; If bus k is not in motion during period t, then ;

[0024] The first bus trip that bus k will take after time period t is recorded as ;

[0025] Step 1.1.5. In period t, there are three charging methods available for bus k: charging with photovoltaic power, charging with energy storage device, and charging with power grid. Any bus can only choose one charging method in the same period;

[0026] If bus k chooses to use photovoltaic charging during period t, then , otherwise let ;

[0027] If bus k chooses to use the energy storage device to charge during period t, then , otherwise let ;

[0028] If bus k chooses to charge from the grid during period t, then , otherwise let ;

[0029] There are two charging methods for the energy storage device: charging by photovoltaic power generation and charging by power grid. If the energy storage device chooses to charge by photovoltaic power generation during the period t, then , otherwise let ; If the energy storage device chooses to use the power grid for charging during the t period, then , otherwise let ;

[0030] Step 1.2. Determine the battery state of charge SOC of the energy storage device;

[0031] Step 1.3. Determine the battery state of charge SOC of the bus battery;

[0032] Step 1.4. Determine the charging cost of the bus system throughout the day;

[0033] Step 1.5. Determine the average photovoltaic power generation self-use rate of the public transportation system;

[0034] Step 1.6. Set the minimum total charging cost of the public transportation system and the maximum self-use rate of photovoltaic power generation as the optimization objectives of the model;

[0035] Step 1.7. Determine the number of times the bus and energy storage device are charged during period t and ;

[0036] Step 1.8. Set model constraints;

[0037] Step 2. Convert the bus system charging scheduling problem into a Markov control problem;

[0038] Step 3. Use the double-delayed deep deterministic policy gradient algorithm to solve the Markov control problem and determine the network parameters of the main critic network, the main actor network, the target critic network, and the target actor network;

[0039] Step 4. Set the state of the agent system to be tested in period t and Actions during the period Input the main Actor network, and the main Actor network outputs the recommended action set for the current period; obtain the actual action set based on the recommended action set;

[0040] Convert the actual action set into a charging scheme and execute it.

[0041] The beneficial effects of the present invention are:

[0042] The present invention takes the electric bus system which adopts three hybrid power supply modes, namely photovoltaic, energy storage and power grid, as the research object, takes the minimization of the system's all-day charging cost and the maximization of the photovoltaic power generation self-use rate as the optimization goal, and establishes the electric bus charging scheme optimization technology based on deep reinforcement learning to output the starting time, charging duration and charging method of each electric bus each time.

[0043] The deep reinforcement learning charging scheme optimization method proposed in the present invention can adaptively learn to obtain charging strategies for time-varying environmental conditions, effectively reducing the impact of uncertain factors such as photovoltaic power generation, remaining power of bus batteries and energy storage batteries. While ensuring the stability of power supply and the reliability of bus operation, the charging scheme for all vehicles in the station is quickly generated, ultimately achieving the purpose of increasing the proportion of clean energy in the electric bus system, reducing carbon emissions and operating costs of the bus system, and optimizing energy efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0045] Specific implementation method 1: The specific process of the electric bus hybrid charging scheme optimization method based on deep reinforcement learning in this implementation method is as follows:

[0046] Step 1. Determine the charging scheduling objectives and constraints of the public transportation system; the specific process is as follows:

[0047] Step 1.1. Define basic variables; the specific process is:

[0048] Step 1.1.1. The number of bus routes operated by the bus station with photovoltaic facilities is I, and each route is equipped with K electric buses, which requires N trips per day;

[0049] There are O charging piles in the station, and the charging power of each charging pile is , The unit is kW;

[0050] The rated capacity of the battery of electric bus k is recorded as , The unit is kWh, ;

[0051] The battery health status of bus k is recorded as , The unit is %;

[0052] The rated capacity of the battery in the energy storage device is recorded as , The unit is kWh;

[0053] The battery health status of the energy storage device is recorded as , The unit is %;

[0054] The photovoltaic layout area is recorded as A, and the unit of A is m 2 ;

[0055] Step 1.1.2. is the time interval, and the whole day is divided into T time periods. ;

[0056] Indicates rounding down. The unit is min, and the recommended value is 15;

[0057] The starting time of period t is recorded as The end time is recorded as , ;

[0058] Step 1.1.3. Let the photovoltaic power generation in period t be , The unit is kW;

[0059] The solar radiation in period t is expressed as , The unit is kW / m 2 ;

[0060] Assumptions and remains unchanged during the period t;

[0061] Put the bus in The battery state of charge SOC at the time is recorded as ;

[0062] The energy storage device The battery state of charge SOC at the time is recorded as ;

[0063] Step 1.1.4. Place bus k in The set of shifts that have been completed before time is defined as ;

[0064] If bus k is in motion during period t, the number of bus trips during period t is recorded as ; If bus k is not in motion during period t, then ;

[0065] The first bus trip that bus k will take after time period t is recorded as ;

[0066] Step 1.1.5. In period t, there are three charging methods available for bus k: charging with photovoltaic power, charging with energy storage device, and charging with power grid. Any bus can only choose one charging method in the same period;

[0067] If bus k chooses to use photovoltaic charging during period t, then , otherwise let ;

[0068] If bus k chooses to use the energy storage device to charge during period t, then , otherwise let ;

[0069] If bus k chooses to charge from the grid during period t, then , otherwise let ;

[0070] There are two charging methods for the energy storage device: charging by photovoltaic power generation and charging by power grid. If the energy storage device chooses to charge by photovoltaic power generation during the period t, then , otherwise let ; If the energy storage device chooses to use the power grid for charging during the t period, then , otherwise let ;

[0071] Step 1.2. Determine the battery state of charge SOC of the energy storage device;

[0072] Step 1.3. Determine the battery state of charge SOC of the bus battery;

[0073] Step 1.4. Determine the charging cost of the bus system throughout the day;

[0074] Step 1.5. Determine the average photovoltaic power generation self-use rate of the public transportation system;

[0075] Step 1.6. Set the minimum total charging cost of the public transportation system and the maximum self-use rate of photovoltaic power generation as the optimization objectives of the model;

[0076] Step 1.7. Determine the number of times the bus and energy storage device are charged during period t and ;

[0077] Step 1.8. Set model constraints;

[0078] Step 2. Convert the bus system charging scheduling problem into a Markov control problem;

[0079] Step 3. Use the double-delayed deep deterministic policy gradient algorithm to solve the Markov control problem and determine the network parameters of the main critic network, the main actor network, the target critic network, and the target actor network;

[0080] Step 4. Set the state of the multi-agent system to be tested in time period t and Actions during the period Input the main Actor network, and the main Actor network outputs the recommended action set for the current period; obtain the actual action set based on the recommended action set;

[0081] The actual action set is converted into a charging plan (charging start time, charging duration, charging method) and executed.

[0082] Specific implementation method 2: This implementation method is different from specific implementation method 1 in that the battery state of charge SOC of the energy storage device is determined in step 1.2;

[0083] The specific process is:

[0084] Step 1.2.1. Record the charge of the energy storage device during period t as , the expression is:

[0085] (1)

[0086] In the formula, is the charge amount of the energy storage device during period t, in kWh;

[0087] is a 0-1 integer variable, indicating the charging state of the energy storage device; A value of 1 indicates that the energy storage device has a charging plan within the t period. A value of 0 indicates that there is no charging plan for the energy storage device during period t;

[0088] The charging method and duration are uncertain in each time period, so the bus / agent is not necessarily in a charging state after every minute in each time period. Therefore, there is no expression "in a charging state".

[0089] Indicates the charging duration of the energy storage device in time period t, in min;

[0090] The value is taken at the minimum charging time With time interval between, The unit is min, and the value is 5;

[0091] is the charging efficiency of the charging pile, which is 0.9;

[0092] Step 1.2.2. Determine the total power consumption of the energy storage device during period t , the expression is:

[0093] (2)

[0094] In the formula, is the total power consumption of the energy storage device in period t, in kWh;

[0095] is a 0-1 integer variable, indicating the charging status of bus k; variable The value of 1 indicates that there is a charging plan for vehicle k in period t. A value of 0 means that bus k will not have any charging plan during this period;

[0096] represents the charging duration of bus k in period t, in min;

[0097] The value is taken at the minimum charging time With time interval between, The unit is min;

[0098] Step 1.2.3. Use formulas (3)-(4) to determine the energy storage device The SOC at the moment, , the expression is:

[0099] (3)

[0100] (4)

[0101] In the formula, The energy storage device is The remaining power at the moment, The unit is kWh; For energy storage devices SOC at all times, The unit is %.

[0102] The other steps and parameters are the same as those in the first embodiment.

[0103] Specific implementation method three: This implementation method is different from specific implementation methods one or two in that the battery state of charge SOC of the bus battery is determined in step 1.3;

[0104] The specific process is:

[0105] Step 1.3.1. Calculate the charge of bus k during period t , the expression is:

[0106] (5)

[0107] In the formula, is the charging amount of bus k in period t, in kWh; represents the charging duration of bus k in period t, in min; The value is taken at the minimum charging time With time interval between, The unit is min; is the charging efficiency of the charging pile, which is 0.9;

[0108] Step 1.3.2. Determine the power consumption of bus k on shift n , the expression is:

[0109] (6)

[0110] In the formula, is the power consumption of bus k in shift n, The unit is kWh; is the operating mileage of flight n, The unit is km; is the curb mass of bus k, in kg; is the travel time of flight n, The unit is min; is the average ambient temperature when bus k performs shift n, The unit is °C; is the optimal operating temperature of the bus battery (the value is 23.7), The unit is °C;

[0111] For a stationary bus, Indicates that bus k ends at Total power consumption at the moment , unit is kWh;

[0112] For buses in motion, Indicates that bus k is Total power consumption before departure, in kWh;

[0113] As shown in formula (7):

[0114] (7)

[0115] Step 1.3.3. For a stationary bus, Indicates that bus k is The battery state of charge at the moment, in %;

[0116] For buses in motion, Indicates that bus k is on the schedule The battery state of charge at the time of departure, in %;

[0117] As shown in formula (8):

[0118] (8)

[0119] The other steps and parameters are the same as those in the first or second embodiment.

[0120] Specific implementation method 4: This implementation method is different from any one of the specific implementation methods 1 to 3 in that the charging cost of the public transportation system throughout the day is determined in step 1.4;

[0121] The specific process is:

[0122] Step 1.4.1. Determine the charging cost of the energy storage device during period t , the expression is:

[0123] (9)

[0124] In the formula, is the charging cost of the energy storage device in period t, in yuan; is the cost of photovoltaic power generation, in yuan / kWh; is the unit price of the power grid in period t, in yuan / kWh;

[0125] Step 1.4.2. The unit price of power supply of the energy storage device in time period t is recorded as , the expression is:

[0126] (10)

[0127] In the formula, is the power supply unit price of the energy storage device in time period t, in yuan / kWh;

[0128] Step 1.4.3. Determine the charging cost of bus k during period t , the expression is:

[0129] (11)

[0130] In the formula, is the charging cost of bus k in period t, in yuan;

[0131] Step 1.4.4. The total charging cost of the bus system in period t is recorded as , the expression is:

[0132] (12)

[0133] In the formula, is the total charging cost of the public transportation system in period t, in yuan;

[0134] The public transportation system includes buses, photovoltaic facilities, and energy storage devices; the minimum charging cost of the entire system and the maximum photovoltaic utilization rate are my research goals;

[0135] Step 1.4.5. Charging costs of the public transport system throughout the day The calculation method of is shown in formula (13):

[0136] (13)

[0137] In the formula, is the charging cost of the public transportation system throughout the day, in yuan;

[0138] In step 1.5, the average photovoltaic power generation self-use rate of the public transportation system is determined;

[0139] The specific process is:

[0140] Step 1.5.1. Use It represents the self-use rate of photovoltaic power generation of the public transportation system in period t, and the expression is:

[0141] (14)

[0142] Among them, the photovoltaic power generation power in period t Calculated by formula (15):

[0143] (15)

[0144] In the formula, is the photovoltaic power generation efficiency (the recommended value is 0.85); is the photovoltaic rated power, in kW; is the temperature coefficient, and its value range is 0.004-0.006; is the photovoltaic cell temperature in period t, in °C, which is given by given; is the reference temperature of the photovoltaic cell, in °C, usually set to 25 °C;

[0145] (16)

[0146] In the formula, is the ambient temperature in °C; It is the battery temperature when the photovoltaic cell is working normally, in °C, usually set to 55 °C;

[0147] Step 1.5.2. Determine the average photovoltaic power generation self-use rate of the public transportation system throughout the day , the expression is:

[0148] (17).

[0149] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.

[0150] Specific implementation method 5: This implementation method is different from any one of specific implementation methods 1 to 4 in that in step 1.6, the minimum total charging cost of the public transportation system and the maximum self-use rate of photovoltaic power generation are set as the optimization objectives of the model;

[0151] As shown in formulas (18) and (19):

[0152] (18)

[0153] (19).

[0154] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0155] Specific implementation method 6: This implementation method is different from the specific implementation methods 1 to 5 in that the number of times the bus and the energy storage device are charged in the t period is determined in step 1.7. and ; The specific process is:

[0156] Step 1.7.1. Set bus k to The charging state of the minute is recorded as ;

[0157] Indicates that bus k is the first Minutes in charging state;

[0158] It means that bus k is the first Minutes not in charging state; ;

[0159] The energy storage device is placed in the t period The charging state of the minute is recorded as ;

[0160] Indicates the energy storage device in the t period Minutes in charging state;

[0161] Indicates the energy storage device in the t period Minutes not in charging state; ;

[0162] Step 1.7.2. Count the number of times the bus and energy storage device are charged during period t and :

[0163] (20)

[0164] (twenty one)

[0165] In the formula, represents the number of times the bus is charged in period t; Indicates the number of times the energy storage device is charged in time period t; Indicates that bus k is the first Minutes of charging status; represents the charging status of bus k at the first minute of period t; Indicates that the energy storage device is Minutes of charging status; Indicates the charging state of the energy storage device at the first minute of period t; , , are logical operation symbols, representing AND operation, OR operation, and NOT operation respectively;

[0166] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.

[0167] Specific implementation method 7: This implementation method is different from any one of specific implementation methods 1 to 6 in that the model constraint conditions are set in step 1.8; the model constraint conditions are as follows:

[0168] (twenty two)

[0169] (twenty three)

[0170] (twenty four)

[0171] (25)

[0172] (26)

[0173] (27)

[0174] (28)

[0175] (29)

[0176] (30)

[0177] (31)

[0178] In the formula, is the battery state of charge of bus k at the end of period t, which is numerically equivalent to the battery state of charge of bus k at the beginning of period t+1; is the battery state of charge of the energy storage device battery at the end of period t, which is numerically equivalent to the battery state of charge of the energy storage device battery at the beginning of period t+1;

[0179] and They are the upper and lower limits of the SOC of the electric bus battery, usually 80% and 20%;

[0180] and They are the upper and lower limits of the SOC of the energy storage device battery, usually 80% and 20%;

[0181] The bus k runs The power consumption is calculated as shown in formula (6). ,but ,like , then the calculation method is as shown in formula (6);

[0182] and Respectively represent the maximum number of charging times of the bus and energy storage device in a day;

[0183] Formulas (22) and (23) indicate that the bus and the energy storage device can only have one charging mode at any time;

[0184] Formula (24) indicates that the energy storage device cannot perform charging and discharging activities at the same time;

[0185] Formula (25) indicates that the number of buses charging at the same time cannot exceed the number of charging piles;

[0186] Formula (26) indicates that the total photovoltaic power supply power cannot exceed the photovoltaic power generation power at any time;

[0187] Formula (27) indicates that the SOC of the bus battery at the end of each period cannot exceed the SOC upper limit of the bus battery;

[0188] Formula (28) indicates that the bus is at the end of its trip. The SOC after the test should not be less than the lower limit of the SOC of the bus battery;

[0189] Formula (29) indicates that the SOC of the energy storage device at the end of each period needs to be within the constraints of the upper and lower limits of the SOC of the energy storage device battery;

[0190] Formula (30) indicates that the total number of charging times for a bus in a day cannot exceed the maximum number of charging times;

[0191] Formula (31) indicates that the total number of charging times of the energy storage device in a day cannot exceed the maximum number of charging times.

[0192] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0193] Specific implementation eight: This implementation differs from any one of specific implementations one to seven in that in step 2, the charging scheduling problem of the public transportation system is transformed into a Markov control problem; the specific process is as follows:

[0194] Step 2.1. Set the K buses as K independent bus agents, using k as the number;

[0195] Setting the energy storage device as an independent energy storage agent;

[0196] A multi-agent system consisting of K independent bus agents and 1 energy storage agent (K+ 1);

[0197] Step 2.2. Record the state of the multi-agent system at time t as , The expression of is as follows:

[0198] (32)

[0199] In the formula, is the basic environmental state during period t, composed as in formula (33);

[0200] is the state of bus agents 1, ..., k, ..., K in period t, composed as in formula (34);

[0201] is the state of the energy storage agent in period t, and its composition is as shown in formula (35);

[0202] (33)

[0203] (34)

[0204] (35)

[0205] In the formula, Indicates the bus k-distance frequency The remaining time for departure is numerically equal to the starting time of period t With shift The difference in departure times, min; represents the state of bus agent k at time period t;

[0206] Step 2.3. Record the action of the multi-agent system in time period t as , The expression of is as follows:

[0207] (36)

[0208] In the formula, represents the action of bus agent k during time period t, ;

[0209] represents the action of the energy storage agent during time period t;

[0210] and The composition is basically the same. and As shown in formulas (37) and (38):

[0211] (37)

[0212] (38)

[0213] In the formula, represents the charging mode of bus agent k during period t. From the above, we can see that this variable has four values: 0, 1, 2, and 3; Indicates that bus agent k does not charge during period t, Indicates that bus agent k uses photovoltaics to charge during period t, Indicates that bus agent k uses energy storage device to charge during period t, Indicates that bus agent k uses the power grid to charge during period t;

[0214] Indicates the charging mode of the energy storage agent during period t. This variable has three values: 0, 1, and 2; Indicates that the energy storage agent does not charge during period t. Indicates that the energy storage agent uses photovoltaics to charge during period t, Indicates that the energy storage agent uses the power grid to charge during period t;

[0215] and They represent the charging start time of bus agent k and energy storage agent in period t, respectively, indicating that the agent will be charged at the first / minutes to start charging, ;

[0216] Step 2.4. Let the reward of the public transportation system in period t be , The expression of is as follows:

[0217] (39)

[0218] In the formula, is the penalty function transformed from constraint (24), and its calculation method is as shown in formula (40);

[0219] is the penalty function transformed from constraints (27)-(29), and the calculation method is as shown in formula (41);

[0220] is the penalty function transformed from constraints (30) and (31), and its calculation method is as shown in formula (44);

[0221] , , , , is the coefficient in the group reward;

[0222] (40)

[0223] In the formula, It is the penalty coefficient for the energy storage device to charge and discharge at the same time;

[0224] (41)

[0225] In the formula, is the SOC penalty cost of bus agent k in period t, which is calculated as shown in formula (42);

[0226] is the SOC penalty cost of the energy storage agent in period t, calculated as in formula (43);

[0227] (42)

[0228] In the formula, is the SOC penalty coefficient of the bus agent;

[0229] (43)

[0230] In the formula, is the SOC penalty coefficient of the energy storage agent;

[0231] (44)

[0232] In the formula, and are the basic charging penalty costs of the bus agent and the energy storage agent in period t, respectively, and the calculation methods are as shown in formulas (45) and (46); and are the additional charging penalty costs of the bus agent and the energy storage agent in period t, respectively, and the calculation methods are as shown in formulas (47) and (48);

[0233] (45)

[0234] (46)

[0235] (47)

[0236] (48)

[0237] In the formula, is the basic penalty coefficient; It is the additional penalty factor when the cumulative number of charging times approaches the limit.

[0238] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.

[0239] Specific implementation method 9: This implementation method is different from any one of specific implementation methods 1 to 8 in that the double-delayed deep deterministic policy gradient algorithm is used in step 3 to solve the Markov control problem; the specific process is:

[0240] Step 3.1. Build the main network, which includes the main Actor network and the main Critic network;

[0241] The specific process is:

[0242] Step 3.1.1. Build the main Actor network, which consists of an input layer, three hidden layers, a ReLU activation function layer, and an output layer.

[0243] The specific process is:

[0244] Step 3.1.1.1. For the bus agent, the input of the main Actor network is given by The state and The 6-dimensional vector composed of the actions of the bus agent during the time period;

[0245] For the energy storage agent, the input of the main Actor network is given by The state and A 4-dimensional vector composed of the actions of the time-slot energy storage agent;

[0246] Step 3.1.1.2. Set the number of neurons in the three hidden layers to 128;

[0247] Step 3.1.1.3. The output of the main Actor network is the recommended action for the current period; the dimension of the recommended action is 3;

[0248] Each agent has a master actor network and a target actor network. All agents share the same master critic network and target critic network.

[0249] Step 3.1.2. For the bus agent, the recommended action output by the main Actor network , Enter the Softmax function; Enter the Tanh activation function respectively;

[0250] For the energy storage agent, the recommended actions output by the main Actor network , Enter the Softmax function; Enter the Tanh activation function respectively;

[0251] The Softmax function is used to convert the continuous output about the charging method into discrete choices; continuous actions such as charging duration are directly output, and the Tanh activation function is used to limit its output range.

[0252] The action is a mixed action set, that is, the action set contains both continuous actions such as (charging start time, duration) and discrete actions (charging method). The traditional double-delay deep deterministic policy gradient algorithm can only output continuous solutions, so we need to convert some continuous actions into discrete actions. The Softmax function is one of the conversion methods. For continuous actions, in order to make the output recommended actions meet the actual constraints, its range needs to be limited, and the Tanh activation function meets this condition well.

[0253] Step 3.1.3. Build the main critic network, which consists of an input layer, three hidden layers, a ReLU activation function, and an output layer;

[0254] The specific process is:

[0255] Step 3.1.3.1. Set the input of the main critic network to a 9K+13-dimensional vector consisting of the states and actions of each agent and the basic environment state ( , and );

[0256] 9K+13 { + + }

[0257] ={ + + }

[0258] =((3K+3)+(3K+3)+(6+3K+1))

[0259] Step 3.1.3.2. Set the number of neurons in the three hidden layers to 128;

[0260] Step 3.1.3.3. The critic network directly outputs the Q-value estimates of all agent state-action pairs (there is only one Q-value, and the Q-value corresponds to the state-action pairs of all agents), with a dimension of 1;

[0261] Step 3.1.4. Change the learning rate of the Actor network Set to 1e-4;

[0262] The learning rate of the Critic network Set to 5e-4;

[0263] Discount Factor Set to 0.99;

[0264] The Ornstein-Uhlenbeck process is used to generate exploration noise during action exploration; the speed at which the noise returns to the mean Set to 0.15; the amplitude of the noise Set to 0.2; the long-term mean of the noise Set to 0; initial noise value Set to 0;

[0265] Use the experience replay pool to store and randomly extract training samples, and the pool size is set to 1e-6;

[0266] Use Adam optimizer to update the main network parameters;

[0267] Step 3.2. Use the He initialization method to initialize the weights of the main Actor network and the main Critic network in the main network, that is, from a mean of 0 and a variance of Randomly obtain the initial weight of each layer of the network from the normal distribution; The calculation method is as follows ; Set the initial bias to 0; Set the maximum number of iterations to 1000;

[0268] (49)

[0269] In the formula, It is the input dimension of the main Actor network and the main Critic network in the main network. Taking the Actor network of the bus agent as an example, the number of input nodes in its input layer is 9;

[0270] Step 3.3. Build the target network, which includes the target Actor network and the target Critic network;

[0271] The target Actor network has the same structure as the main Actor network, and the parameter values ​​are consistent;

[0272] The target critic network has the same structure as the main critic network, and the parameter values ​​are consistent;

[0273] The target network has the same structure as the main network, and the parameter values ​​are the same, but the parameter update method is different;

[0274] The target network adopts a soft update strategy, and the update formula is as follows: As shown:

[0275] (50)

[0276] In the formula, are the parameters of the target network; are the parameters of the main network; is the update strategy coefficient, with a value of 0.1;

[0277] Step 3.4. Let each agent (K independent bus agents and 1 energy storage agent) receive the state and The action information of the time period is input into the main Actor network, and the recommended action for time period t is obtained based on the main Actor network;

[0278] Noise is then added to the proposed action, and the noisy action (original action plus noise) is sent back to the environment;

[0279] The noise calculation method is shown in formula (51):

[0280] (51)

[0281] In the formula, is the noise in period t, is the noise in period t-1, is the speed at which the noise returns to the mean, is the long-term mean of the noise, is the amplitude of the noise, is the average value of the standard normal distribution in period t A random sample drawn from

[0282] Step 3.5. Consider the model constraints to determine the final actual action; the specific process is:

[0283] Step 3.5.1 Based on the recommended action set obtained by the main actor network in 3.4, buses with different charging methods are assigned to four different sets , , , , count the number of buses in each set, respectively , , , ;

[0284] In the formula, represents the set of buses that are not charging, It means a bus that uses photovoltaics for charging. represents the set of buses charged using energy storage, represents the set of buses that use the grid for charging;

[0285] Representing a collection The number of buses in Representing a collection The number of buses in Representing a collection The number of buses in Representing a collection The number of buses in

[0286] Step 3.5.2. Determine whether the number of buses that need to be charged (charging mode value is not 0) is greater than the number of charging piles;

[0287] When the number of buses that need to be charged (charging mode value is not 0) is greater than the number of charging piles, go to step 3.5.7;

[0288] When the number of buses that need to be charged (charging mode value is not 0) is less than or equal to the number of charging piles, if Then proceed to step 3.5.3; if and Then go to step 3.5.5; if none of them are satisfied (not satisfied ,as well as and ) then the current recommended action set is defined as the final actual action set;

[0289] Step 3.5 has been adjusting the actions in the proposed action set to satisfy the actual constraints (those in Section 1.8);

[0290] Step 3.5.3. Calculate the photovoltaic power supply per minute under the recommended action set obtained by the main Actor network in step 3.4 (The calculation method is the same as the content on the left side of formula (26). This step here is to process the constraints corresponding to formula 26). If Value greater than , go to step 3.5.4; otherwise, fix this time Charging solutions for all buses in the

[0291] Fixed this time After finding all the charging schemes for buses, determine whether any vehicle is charged using the energy storage device; if so, proceed to step 3.5.5; otherwise, define the current action set as the final actual action set;

[0292] Step 3.5.4. Replace The bus with the shortest charging duration The specific charging process is as follows:

[0293] Step 3.5.4.1. Compare the power supply unit price of the energy storage device during period t The unit price of the power grid in period t Size, when When , go to step 3.5.4.3; when When , go to step 3.5.4.3; if none of them meet the requirements ( When ,),calculate After charging the energy storage device, the total power consumption of the energy storage device in time period t (Formula 2), go to step 3.4.5.2;

[0294] Step 3.5.4.2. If the total power consumption Greater than the amount of energy stored at the beginning of period t , go to step 3.5.4.3; otherwise, let the bus exist During this time period, the energy storage device is used for charging, and the process returns to step 3.5.3;

[0295] Step 3.5.4.3. Order the bus exist During this period, the grid is used for charging, and the process returns to step 3.5.3;

[0296] Step 3.5.5. Calculate the total power consumption of the energy storage device during period t ,like Greater than the amount of energy stored at the beginning of period t , go to step 3.5.6; otherwise, fix this time The charging solution for buses , and define the current action set as the final actual action set;

[0297] Step 3.5.6. Replace The bus with the shortest charging duration The specific charging process is as follows:

[0298] Step 3.5.6.1. Compare the cost of photovoltaic power generation during period t The unit price of the power grid in period t Size, if , proceed to 3.5.6.3; otherwise, calculate After using photovoltaic charging, the photovoltaic power supply per minute (The calculation method is the same as the content on the left side of formula (26). This step here is to process the constraints corresponding to formula 26);

[0299] Step 3.5.6.2. If the photovoltaic power supply per minute Greater than the photovoltaic power generation power during the period t , go to step 3.5.6.3; otherwise, let the bus During the period t, the photovoltaic power is used for charging, and the process returns to step 3.5.5;

[0300] Step 3.5.6.3. Order the bus During the period t, the battery is charged using the power grid and the process returns to step 3.5.5;

[0301] Step 3.5.7. Calculate the charging priority of all buses that need to be charged (Take bus k as an example):

[0302] (52)

[0303] In the formula, , is the calculation coefficient of charging priority;

[0304] Step 3.5.8. Arrange the buses that need to be charged in descending order of charging priority, take the top O buses in charging priority, and proceed to step 3.5.1;

[0305] Step 3.6. Convert the finalized actual action set into a charging plan, instruct each bus and energy storage device to execute the determined charging plan, obtain a new state, observe the environmental state and calculate the reward (calculate the reward in the new state based on the content in step 2.4 after executing the action);

[0306] Step 3.7. Store the state, action, reward, and new state as a sample in the experience pool for subsequent learning process;

[0307] Step 3.8. Randomly extract a batch of data samples from the experience pool (a total of The state and action of each sample are input into the main Critic network and the corresponding actual Q value is output, that is, ;

[0308] Input the new state and the action at the last moment into the main Actor network to obtain the recommended action (charging plan) for each sample in the new state, and then obtain the final actual action in the new state through step 3.5;

[0309] The new state and the final actual action under the new state are input into the target Critic network and the corresponding target Q value is output, that is, ;

[0310] Step 3.9. Update the target Q value based on formula (53) and record the target Q value as y:

[0311] (53)

[0312] In the formula, is the reward value recorded in the sample, is the discount factor;

[0313] Step 3.10. Calculate the loss function of the Critic network , apply Adam optimizer to update the main critic network parameters;

[0314] (54)

[0315] In the formula, and are the y value and actual Q value of the nth sample data in the extracted batch data respectively;

[0316] Step 3.11. Calculate the loss function of the Actor network , the loss function of the Actor network corresponding to the bus agent and the energy storage agent is the same:

[0317]

[0318] Apply the Adam optimizer to update the main Actor network parameters of each agent;

[0319] Step 3.12. Use the soft update strategy to update the parameters of the target Critic network and the target Actor network every 10 generations;

[0320] Step 3.13. Repeat steps 3.4 to 3.12 until the termination condition of learning is met (the maximum number of iterations is reached). Save the network parameters of the main Critic network, the main Actor network, the target Critic network, and the target Actor network, and proceed to step 4.

[0321] Adjust hyperparameters according to the performance during learning to optimize the learning effect and save the network parameters with the best effect;

[0322] Evaluate the trained agent network model, observe its performance in unseen conditions, ensure the stability of its output results (the obtained charging solution can cope with basic conditions and emergencies in different seasons and environments), and output and save the final parameters.

[0323] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0324] Specific implementation method 10: This implementation method is different from any one of the specific implementation methods 1 to 9 in that in step 4, the state of the multi-agent system to be tested in the time period t is and Actions during the period Input the main Actor network, and the main Actor network outputs the recommended action set for the current period; obtain the actual action set based on the recommended action set;

[0325] Convert the actual action set into a charging plan (charging start time, charging duration, charging method) and execute it;

[0326] The specific process is:

[0327] Step 4.1. At the starting moment, load the main Actor network parameters saved in step 3.13 into the main Actor network;

[0328] Step 4.2. Set the state of the multi-agent system to be tested at time t and Actions during the period ( and ) is input into the main Actor network of step 4.1, and the main Actor network outputs the recommended action set for the current period; the actual action set is obtained based on the recommended action set;

[0329] Convert the actual action set into a charging plan (charging start time, charging duration, charging method) and execute it;

[0330] Step 4.3. Repeat step 4.2 until .

[0331] The other steps and parameters are the same as those in Specific Embodiments 1 to 9.

[0332] The present invention may also have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. An optimization method for hybrid charging scheme of electric buses based on deep reinforcement learning, characterized by: The specific process of the method is: Step 1. Determine the charging scheduling objectives and constraints of the public transportation system; the specific process is as follows: Step 1.

1. Define basic variables; the specific process is: Step 1.1.

1. The bus station with photovoltaic facilities has I bus routes, I routes are equipped with K electric buses, and a total of N trips are required every day; There are O charging piles in the station, and the charging power of each charging pile is P CP , P CP The unit is kW; The rated capacity of the battery of electric bus k is recorded as B k , B k The unit is kWh, k=1,2,...,K; The battery health state of bus k is recorded as SOH k , SOH k The unit is %; The rated capacity of the battery in the energy storage device is recorded as B e , B e The unit is kWh; The battery health state of the energy storage device is recorded as SOH e , SOH e The unit is %; The photovoltaic layout area is recorded as A, and the unit of A is m 2 ; Step 1.1.

2. Divide the whole day into T time periods with Δt as the time interval. Indicates rounding down, Δt unit is min; The starting time of period t is recorded as h t,sta The end time is recorded as h t,end , t=1,2,…,T; Step 1.1.

3. Let the photovoltaic power generation in period t be P t,PV , P t,PV The unit is kW; The solar radiation in period t is denoted as G t , G t The unit is kW / m 2 ; Put bus k at h 1,sta The battery state of charge SOC at the time is recorded as The energy storage device is 1,sta The battery state of charge SOC at the time is recorded as Step 1.1.

4. Place bus k at h t,sta The set of shifts that have been completed before time is defined as If bus k is in motion during period t, the number of bus trips during period t is recorded as If bus k is not in motion during period t, then The first bus trip that bus k will take after time period t is recorded as Step 1.1.

5. In period t, there are three charging methods available for bus k: charging with photovoltaic power, charging with energy storage device and charging with power grid. Any bus can only choose one charging method in the same period. If bus k chooses to use photovoltaic charging during period t, then Otherwise, If bus k chooses to use the energy storage device to charge during period t, then Otherwise, If bus k chooses to charge from the grid during period t, then Otherwise, There are two charging methods for the energy storage device: charging by photovoltaic power generation and charging by power grid. If the energy storage device chooses to charge by photovoltaic power generation during the period t, then Otherwise, If the energy storage device chooses to use the power grid for charging during period t, then Otherwise, Step 1.

2. Determine the battery state of charge SOC of the energy storage device; Step 1.

3. Determine the battery state of charge SOC of the bus battery; Step 1.

4. Determine the charging cost of the bus system throughout the day; Step 1.

5. Determine the average photovoltaic power generation self-use rate of the public transportation system; Step 1.

6. Set the minimum total charging cost of the public transportation system and the maximum self-use rate of photovoltaic power generation as the optimization objectives of the model; Step 1.

7. Determine the number of times the bus and energy storage device are charged during period t and Step 1.

8. Set model constraints; Step 2. Convert the bus system charging scheduling problem into a Markov control problem; Step 3. Use the double-delayed deep deterministic policy gradient algorithm to solve the Markov control problem and determine the network parameters of the main critic network, the main actor network, the target critic network, and the target actor network; Step 4. Set the state S of the intelligent system to be tested in time period t t and action A in period t-1 t-1 Input the main Actor network, and the main Actor network outputs the recommended action set for the current period; obtain the actual action set based on the recommended action set; Convert the actual action set into a charging scheme and execute it; In the step 1.2, the battery state of charge SOC of the energy storage device is determined; The specific process is: Step 1.2.

1. Record the charge of the energy storage device during period t as The expression is: In the formula, is the charge amount of the energy storage device during period t, in kWh; is a 0-1 integer variable, indicating the charging state of the energy storage device; A value of 1 indicates that the energy storage device has a charging plan within the t period. A value of 0 indicates that there is no charging plan for the energy storage device during period t; Indicates the charging duration of the energy storage device in time period t, in min; The value is taken at the minimum charging time Between the time interval Δt, The unit is min; ξ is the charging efficiency of the charging pile; Step 1.2.

2. Determine the total power consumption of the energy storage device during period t The expression is: In the formula, is the total power consumption of the energy storage device in period t, in kWh; is a 0-1 integer variable, indicating the charging status of bus k; variable The value of 1 indicates that there is a charging plan for vehicle k in period t. A value of 0 means that bus k will not have any charging plan during this period; represents the charging duration of bus k in period t, in min; The value is taken at the minimum charging time Between the time interval Δt, The unit is min; Step 1.2.

3. Use formulas (3)-(4) to determine the energy storage device at h t,sta The SOC at the moment, The expression is: In the formula, is the energy storage device at h t,sta The remaining power at the moment, The unit is kWh; The energy storage device is t,sta SOC at all times, The unit is %; In the step 1.3, the battery state of charge SOC of the bus battery is determined; The specific process is: Step 1.3.

1. Calculate the charge of bus k during period t The expression is: In the formula, is the charging amount of bus k in period t, in kWh; represents the charging duration of bus k in period t, in min; The value is taken at the minimum charging time Between the time interval Δt, The unit is min; ξ is the charging efficiency of the charging pile, and its value is 0.9; Step 1.3.

2. Determine the power consumption E of bus k on shift n k,n , the expression is: lnE k,n =-8.11+0.5523lnL n +0.78lnG k +0.35lnT n +0.0077|t k,n -t * |(6) In the formula, E k,n is the power consumption of bus k in shift n, E k,n The unit is kWh; L n is the operating mileage of flight n, L n Unit: km; G k is the curb mass of bus k, in kg; T n is the travel time of flight n, T n The unit is min; τ k,n is the average ambient temperature when bus k performs shift n, τ k,n The unit is ℃; τ * is the optimal operating temperature of the bus battery, τ * The unit is °C; The total power consumption of bus k As shown in formula (7): The unit is kWh; Step 1.3.

3. Battery state of charge of bus k As shown in formula (8): The unit is %; In step 1.4, the charging cost of the public transportation system throughout the day is determined; The specific process is: Step 1.4.

1. Determine the charging cost of the energy storage device during period t The expression is: In the formula, is the charging cost of the energy storage device in period t, in yuan; c PV is the cost of photovoltaic power generation, in yuan / kWh; c t,SG is the unit price of the power grid in period t, in yuan / kWh; Step 1.4.

2. Let the unit price of power supply of the energy storage device in time period t be c t,ESS , the expression is: In the formula, c t,ESS is the power supply unit price of the energy storage device in time period t, in yuan / kWh; Step 1.4.

3. Determine the charging cost of bus k during period t The expression is: In the formula, is the charging cost of bus k in period t, in yuan; Step 1.4.

4. Let the total charging cost of the bus system in period t be C t , the expression is: In the formula, C t is the total charging cost of the public transportation system in period t, in yuan; The public transportation system includes buses, photovoltaic facilities, and energy storage devices; Step 1.4.

5. The calculation method of the charging cost Z1 of the public transportation system throughout the day is shown in formula (13): Where Z1 is the charging cost of the public transportation system throughout the day, in yuan; In step 1.5, the average photovoltaic power generation self-use rate of the public transportation system is determined; The specific process is: Step 1.5.

1. Use μ t It represents the self-use rate of photovoltaic power generation of the public transportation system in period t, and the expression is: Among them, the photovoltaic power generation power P in period t t,PV Calculated by formula (15): In the formula, ξ PV is the photovoltaic power generation efficiency; is the photovoltaic rated power, in kW; β is the temperature coefficient; τ t,c is the temperature of the photovoltaic cell in period t, in °C; is the reference temperature of the photovoltaic cell, in °C; In the formula, τ t,amb is the ambient temperature in °C; τ NOCT It is the battery temperature when the photovoltaic cell is working normally, in °C; Step 1.5.

2. Determine the average photovoltaic power generation self-use rate Z2 of the public transportation system throughout the day, expressed as: In step 1.6, the minimum total charging cost of the public transportation system and the maximum self-use rate of photovoltaic power generation are set as the optimization objectives of the model; As shown in formulas (18) and (19): min Z1(18) max Z2(19) In step 1.7, the number of times the bus and the energy storage device are charged in time period t is determined. and The specific process is: Step 1.7.

1. Record the charging state of bus k at minute t′ in period t as Indicates that bus k is in charging state at minute t′ in period t; Indicates that bus k is not in charging state at minute t′ in period t; t′=1,2,...,Δt; The charging state of the energy storage device at minute t' in period t is recorded as Indicates that the energy storage device is in a charging state at minute t' in period t; Indicates that the energy storage device is not in a charging state at the t′th minute in the t period; t′=1,2,...,Δt; Step 1.7.

2. Count the number of times the bus and energy storage device are charged during period t and In the formula, represents the number of times the bus is charged in period t; Indicates the number of times the energy storage device is charged in time period t; represents the charging status of bus k at the Δt-th minute in the t-1 period; represents the charging status of bus k at the first minute of period t; Indicates the charging state of the energy storage device at the Δt minute in the t-1 period; Indicates the charging state of the energy storage device at the first minute of period t; ∧, ∨, are logical operation symbols, representing AND operation, OR operation, and NOT operation respectively; In step 1.8, the model constraints are set; the model constraints are as follows: The model constraints also include that the energy storage device cannot perform charging and discharging activities at the same time; In the formula, is the battery state of charge of bus k at the end of period t, which is numerically equivalent to the battery state of charge of bus k at the beginning of period t+1; is the battery state of charge of the energy storage device battery at the end of period t; and They are the upper and lower limits of SOC of the electric bus battery; and They are the upper and lower limits of the SOC of the battery of the energy storage device; The bus k runs If but like The calculation method is shown in formula (6); M EB With M ESS Respectively represent the maximum number of charging times of the bus and energy storage device in a day; Formulas (22) and (23) indicate that the bus and the energy storage device can only have one charging mode at any time; Formula (25) indicates that the number of buses charging at the same time cannot exceed the number of charging piles; Formula (26) indicates that the total photovoltaic power supply power cannot exceed the photovoltaic power generation power at any time; Formula (27) indicates that the SOC of the bus battery at the end of each period cannot exceed the SOC upper limit of the bus battery; Formula (28) indicates that the bus is at the end of its trip. The SOC after the test should not be less than the lower limit of the SOC of the bus battery; Formula (29) indicates that the SOC of the energy storage device at the end of each period needs to be within the constraints of the upper and lower limits of the SOC of the energy storage device battery; Formula (30) indicates that the total number of charging times for a bus in a day cannot exceed the maximum number of charging times; Formula (31) indicates that the total number of charging times of the energy storage device in a day cannot exceed the maximum number of charging times; In step 2, the charging scheduling problem of the public transportation system is transformed into a Markov control problem; the specific process is: Step 2.

1. Set the K buses as K independent bus agents, using k as the number; Setting the energy storage device as an independent energy storage agent; An agent system consisting of K independent bus agents and 1 energy storage agent; Step 2.

2. Let the state of the agent system at time t be S t , S t The expression of is as follows: In the formula, s t,env is the basic environmental state during period t; is the state of bus agents 1, ..., k, ..., K at time period t; is the state of the energy storage agent at time period t; In the formula, Indicates the bus k-distance frequency The remaining time for departure is numerically equal to the starting time h of the t period t,sta With shift The difference in departure times, min; represents the state of bus agent k at time period t; Step 2.

3. Record the action of the agent system in time period t as A t , A t The expression of is as follows: In the formula, represents the action of bus agent k during period t, k = 1, 2, ..., K; represents the action of the energy storage agent during time period t; and As shown in formulas (37) and (38): In the formula, represents the charging mode of bus agent k during period t; Indicates that bus agent k does not charge during period t, Indicates that bus agent k uses photovoltaics to charge during period t, Indicates that bus agent k uses energy storage device to charge during period t, Indicates that bus agent k uses the power grid to charge during period t; represents the charging mode of the energy storage agent during period t; Indicates that the energy storage agent does not charge during period t. Indicates that the energy storage agent uses photovoltaics to charge during period t, Indicates that the energy storage agent uses the power grid to charge during period t; and They represent the charging start time of bus agent k and energy storage agent in period t, Step 2.

4. Let the reward of the public transportation system in period t be R t , R t The expression of is as follows: In the formula, r t,ESS It is the penalty coefficient based on the simultaneous charging and discharging of the energy storage device. The penalty function under the condition that the energy storage device cannot perform charging and discharging activities at the same time; r t,SOC is the penalty function transformed from constraints (27)-(29), and the calculation method is as shown in formula (41); r t,cn is the penalty function transformed from constraints (30) and (31), and its calculation method is as shown in formula (44); is the coefficient in the group reward; In the formula, is the SOC penalty cost of bus agent k in period t, which is calculated as shown in formula (42); is the SOC penalty cost of the energy storage agent in period t, calculated as in formula (43); In the formula, is the SOC penalty coefficient of the bus agent; In the formula, is the SOC penalty coefficient of the energy storage agent; In the formula, and are the basic charging penalty costs of the bus agent and the energy storage agent in period t; and are the additional charging penalty costs of the bus agent and the energy storage agent in period t; In the formula, ω base is the basic penalty coefficient; ω etr It is the additional penalty factor when the cumulative number of charging times approaches the limit; In step 3, the double-delayed deep deterministic policy gradient algorithm is used to solve the Markov control problem; the specific process is: Step 3.

1. Build the main network, which includes the main Actor network and the main Critic network; The specific process is: Step 3.1.

1. Build the main Actor network, which consists of an input layer, three hidden layers, a ReLU activation function layer, and an output layer; The specific process is: Step 3.1.1.

1. For the bus agent, the input of the main Actor network is a 6-dimensional vector consisting of the state of the bus agent at time period t and the action of the bus agent at time period t-1; For the energy storage agent, the input of the main Actor network is a 4-dimensional vector consisting of the state of the energy storage agent in period t and the action of the energy storage agent in period t-1; Step 3.1.1.

2. Set the number of neurons in the three hidden layers to 128; Step 3.1.1.

3. The output of the main Actor network is the recommended action for the current period; the dimension of the recommended action is 3; Step 3.1.

2. For the bus agent, the recommended action output by the main actor network Enter the Softmax function; Enter the Tanh activation function respectively; For the energy storage agent, the recommended actions output by the main Actor network Enter the Softmax function; Enter the Tanh activation function respectively; Step 3.1.

3. Build the main critic network, which consists of an input layer, three hidden layers, a ReLU activation function, and an output layer; The specific process is: Step 3.1.3.

1. Set the input of the main critic network to a 9K+13-dimensional vector consisting of the states and actions of each agent and the basic environment state; Step 3.1.3.

2. Set the number of neurons in the three hidden layers to 128; Step 3.1.3.

3. The Critic network directly outputs the Q-value estimates of all agent state-action pairs; Step 3.1.

4. Change the learning rate lr of the Actor network Act Set to 1e-4; The learning rate lr of the Critic network Cri Set to 5e-4; Set the discount factor γ to 0.99; Generate exploration noise during action exploration; the speed v at which the noise returns to the mean is set to 0.15; the amplitude m of the noise is set to 0.2; the long-term mean p of the noise is set to 0; and the initial noise value u1 is set to 0; Use the experience replay pool to store and randomly extract training samples, and the pool size is set to 1e-6; Use Adam optimizer to update the main network parameters; Step 3.

2. Use the He initialization method to initialize the weights of the main Actor network and the main Critic network in the main network; set the initial bias to 0; set the maximum number of iterations to 1000; Step 3.

3. Build the target network, which includes the target Actor network and the target Critic network; The target Actor network has the same structure as the main Actor network, and the parameter values ​​are consistent; The target critic network has the same structure as the main critic network, and the parameter values ​​are consistent; The target network adopts a soft update strategy, and the update formula is shown in formula (50): i tar =ψθ+(1-ψ)θ tar (50) In the formula, θ tar is the parameter of the target network; θ is the parameter of the main network; ψ is the update strategy coefficient, which is 0.1; Step 3.

4. Let each agent receive the state of period t and the action information of period t-1 from the environment and input them into the main Actor network, and obtain the recommended action of period t based on the main Actor network; Noise is then added to the proposed action and the noisy action is sent back to the environment; The noise calculation method is shown in formula (51): In the formula, u t is the noise in period t, u t-1 is the noise in the t-1 period, v is the speed at which the noise returns to the mean, p is the long-term mean of the noise, m is the amplitude of the noise, and ε t is a random sample drawn from a standard normal distribution N(0,1) in period t; Step 3.

5. Consider the model constraints to determine the final actual action; the specific process is: Step 3.5.1 Based on the recommended action set obtained by the main Actor network in 3.4, buses with different charging methods are assigned to four different sets K unc , K PV , K ESS , K SG , count the number of buses in each set, denoted as n unc 、n PV 、n ESS 、n SG ; In the formula, K unc represents the set of buses that are not charged, K PV represents a bus that uses photovoltaic charging, K ESS represents the set of buses charged by energy storage, K SG represents the set of buses that use the grid for charging; n unc Denotes the set K unc The number of buses in, n PV Denotes the set K PV The number of buses in, n ESS Denotes the set K ESS The number of buses in, n SG Denotes the set K SG The number of buses in Step 3.5.

2. Determine whether the number of buses that need to be charged is greater than the number of charging piles; When the number of buses that need to be charged is greater than the number of charging piles, go to step 3.5.7; When the number of buses that need to be charged is less than or equal to the number of charging piles, if n PV ≠0, proceed to step 3.5.3; if n PV =0 and n ESS ≠0, go to step 3.5.5; if none of them are satisfied, define the current recommended action set as the final actual action set; Step 3.5.

3. Calculate the photovoltaic power supply per minute under the recommended action set obtained by the main Actor network in step 3.4 like Value greater than P t,PV , go to step 3.5.4; otherwise, fix K at this time PV Charging solutions for all buses in the Fixed at this time K PV After finding all the charging schemes for buses, determine whether any vehicle is charged using the energy storage device; if so, proceed to step 3.5.5; otherwise, define the current action set as the final actual action set; Step 3.5.

4. Replace K PV The bus with the shortest charging duration The specific charging process is as follows: Step 3.5.4.

1. Compare the power supply unit price c of the energy storage device during period t t,ESS and the unit price c of the power grid in period t t,SG size, when c t,ESS >c t,SG When c t,ESS ≤c t,SG When n ESS =0, go to step 3.5.4.3; if none of them are satisfied, calculate After charging the energy storage device, the total power consumption of the energy storage device in time period t Go to step 3.5.4.2; Step 3.5.4.

2. If the total power consumption Greater than the amount of energy stored at the beginning of period t Go to step 3.5.4.3; otherwise, let the bus During the period t, the energy storage device is used to charge, and the process returns to step 3.5.3; Step 3.5.4.

3. Order the bus During the period t, the battery is charged using the power grid and the process returns to step 3.5.3; Step 3.5.

5. Calculate the total power consumption of the energy storage device during period t like Greater than the amount of energy stored at the beginning of period t Go to step 3.5.6; otherwise, fix K at this time ESS The charging solution for buses And define the current action set as the final actual action set; Step 3.5.

6. Replace K ESS The bus with the shortest charging duration The specific charging process is as follows: Step 3.5.6.

1. Compare the cost of electricity c of photovoltaic power generation during period t PV and the unit price c of the power grid in period t t,SG Size, if c PV >c t,SG , proceed to 3.5.6.3; otherwise, calculate After using photovoltaic charging, the photovoltaic power supply per minute Step 3.5.6.

2. If the photovoltaic power supply per minute Greater than the photovoltaic power generation power P during the period t t,PV , go to step 3.5.6.3; otherwise, let the bus During the period t, the photovoltaic power is used for charging, and the process returns to step 3.5.5; Step 3.5.6.

3. Order the bus During the period t, the battery is charged using the power grid and the process returns to step 3.5.5; Step 3.5.

7. Calculate the charging priority of all buses that need to be charged In the formula, is the calculation coefficient of charging priority; Step 3.5.

8. Arrange the buses that need to be charged in descending order of charging priority, take the buses with the top O charging priorities, and proceed to step 3.5.1; Step 3.

6. Convert the final determined actual action set into a charging plan, make each bus and energy storage device execute the determined charging plan, obtain a new state, and calculate the reward; Step 3.

7. Store the state, action, reward, and new state as a sample in the experience pool; Step 3.

8. Randomly extract a batch of data samples from the experience pool; input the state and action of each sample into the main critic network and output the corresponding actual Q value, i.e., Q(S,A); Input the new state and the action at the last moment into the main Actor network to obtain the recommended action of each sample in the new state, and then obtain the final actual action in the new state through step 3.5; The new state and the final actual action under the new state are input into the target Critic network and the corresponding target Q value is output, i.e., Q tar (S′,A′); Step 3.

9. Update the target Q value based on formula (53), and record the updated target Q value as y: y=r+γQ tar (S′,A′) (53) Where r is the reward value recorded in the sample, and γ is the discount factor; Step 3.

10. Calculate the loss function L of the Critic network Cri , apply Adam optimizer to update the main critic network parameters; In the formula, y n With Q n (S, A) are the y value and actual Q value of the nth sample data in the extracted batch data respectively; N sam Indicates the number of data samples randomly drawn from the experience pool; Step 3.

11. Calculate the loss function L of the Actor network Act , the loss function of the Actor network corresponding to the bus agent and the energy storage agent is the same: Apply the Adam optimizer to update the main Actor network parameters of each agent; Step 3.

12. Use the soft update strategy to update the parameters of the target Critic network and the target Actor network every 10 generations; Step 3.

13. Repeat steps 3.4 to 3.12 until the termination condition of learning is met, save the network parameters of the main Critic network, the main Actor network, the target Critic network, and the target Actor network, and go to step 4; In step 4, the state S of the intelligent system to be tested in time period t is t and action A in period t-1 t-1 Input the main Actor network, and the main Actor network outputs the recommended action set for the current period; obtain the actual action set based on the recommended action set; convert the actual action set into a charging plan and execute it; The specific process is: Step 4.

1. Using h 1,sta At the starting moment, load the main Actor network parameters saved in step 3.13 into the main Actor network; Step 4.

2. Set the state S of the intelligent system to be tested at time t t and action A in period t-1 t-1 Input the main Actor network of step 4.1, and the main Actor network outputs the recommended action set for the current period; obtain the actual action set based on the recommended action set; Convert the actual action set into a charging scheme and execute it; Step 4.

3. Repeat step 4.2 until h T,end .

Citation Information

Patent Citations

  • Distributed building facade photovoltaic integrated power generation and storage integrated system

    CN107046393A

  • Electric bus charging scheme optimization method considering influence of time-of-use electricity price

    CN112906983A

  • Cooperative regulation and control method and system for electric buses

    CN115759673A

  • Electric bus route vehicle scheduling method under'light-storage-network 'cooperative power supply

    CN116128235A

  • Methods and systems for sustainable charging of an electric vehicle

    US20240067031A1

Cited By

  • Photovoltaic energy storage and bus charging scheduling method and system based on probability game and robust check

    CN122782588A