A Reinforcement Learning-Based Optimization Method for Charging Scheduling of Hybrid Electric Bus Systems

Abstract

This invention relates to a method for optimizing the charging schedule of a hybrid-powered electric bus system based on reinforcement learning. The invention pertains to the field of urban public transportation operation and management, specifically to a method for optimizing the charging schedule of a hybrid-powered electric bus system. The purpose of this invention is to address the problems of resource redundancy and waste, high energy consumption, and high operating costs caused by unreasonable resource allocation in existing electric bus systems. The specific process of the method is as follows: defining a state space; defining an action space; defining a reward function; establishing a multi-objective Markov decision model based on the state space, action space, and reward function; constructing a reinforcement learning network model based on the multi-objective Markov decision model; obtaining a dataset; training the reinforcement learning network model based on the dataset, obtaining the trained reinforcement learning network model, and saving the parameters; and obtaining the charging scheme for the actual optimization period based on the trained reinforcement learning network model.

Description

Technical Field

[0001] This invention belongs to the field of urban public transportation operation and management, and specifically relates to a method for optimizing charging scheduling of hybrid-powered electric bus systems. Background Technology

[0002] To reduce carbon emissions and operating costs of urban public transportation systems, countries worldwide are vigorously promoting the electrification of buses. However, the electricity consumed by electric buses comes from the power grid, and under a power structure dominated by thermal power generation, the carbon emissions from electric buses due to their electricity consumption remain significant. To further reduce carbon emissions from electric bus systems, a hybrid power supply model combining photovoltaic (PV) power generation and energy storage (ESD) grid has emerged. In this model, PV power generation equipment converts solar energy into electricity, while energy storage equipment stores the generated electricity, avoiding the intermittency and volatility of PV power generation and providing a relatively stable power supply for buses. The power grid then supplements the buses' power supply when PV power generation is insufficient. This new power supply model not only reduces the dependence of electric buses on the traditional power grid but also avoids the impact of centralized charging on the grid, which is of great significance for promoting the sustainable development of urban transportation.

[0003] However, this multi-source hybrid power supply mode exacerbates the complexity of bus charging scheduling, causing the bus system to face new challenges while dealing with existing problems, such as the uncertain return charging time due to travel time uncertainty and the uneven distribution of charging demand. For example, photovoltaic power generation fluctuates significantly with time and weather conditions, making the clean energy available for vehicles highly uncertain. Although energy storage can buffer photovoltaic fluctuations to some extent, its capacity is limited, and if the charging strategy is inappropriate, it may still fail to meet the real-time charging needs of buses or lead to idle photovoltaic power resources. In this situation, charging scheduling methods based on a single power source and hybrid charging scheduling methods that ignore the uncertainty of photovoltaic power generation are difficult to fully cope with the dynamic changes in the external environment, and the resulting bus charging schemes often fail to achieve optimal results in practice. Therefore, this invention comprehensively considers the uncertainty caused by multi-energy coupling and proposes a charging scheduling method based on reinforcement learning to solve the charging scheduling problem of the "photovoltaic-storage-grid" hybrid power supply electric bus system. Summary of the Invention

[0004] The purpose of this invention is to address the problems of resource redundancy and waste, high energy consumption, and high operating costs caused by unreasonable resource allocation in existing electric bus systems, and to propose a charging scheduling optimization method for hybrid power supply electric bus systems based on reinforcement learning.

[0005] The specific process of the reinforcement learning-based hybrid power supply electric bus system charging scheduling optimization method is as follows:

[0006] Step 1: Define the state space;

[0007] Step 2: Define the action space;

[0008] Step 3: Define the reward function;

[0009] Step 4: Based on the state space, action space, and reward function, establish a multi-objective Markov decision model;

[0010] Step 5: Construct a reinforcement learning network model based on a multi-objective Markov decision model;

[0011] Step 6: Obtain the dataset;

[0012] Step 7: Train a reinforcement learning network model based on the dataset, obtain the trained reinforcement learning network model, and save the parameters;

[0013] Step 8: Obtain the charging scheme for the actual optimization period based on the trained reinforcement learning network model.

[0014] The beneficial effects of this invention are as follows:

[0015] This invention models the bus charging scheduling problem under the "photovoltaic-storage-grid" hybrid energy supply mode as a multi-objective Markov decision model. Considering the feature distribution shift and policy conflict caused by environmental differences, the charging scheduling task is divided into a regular task when there is sufficient light and a special task when there is insufficient light. A multi-agent deep reinforcement learning solution algorithm with a multi-task head architecture is designed. Based on actual bus system operation data and environmental data, charging scheduling strategies under various objective preferences are trained and obtained.

[0016] The reinforcement learning-based charging scheduling optimization method proposed in this invention can directly map multi-source observation data (such as real-time irradiance, temperature, energy storage battery state of charge, vehicle arrival status, etc.) to charging control commands under different environmental conditions according to user preferences. This minimizes operating costs and carbon emissions while meeting real-time charging scheduling requirements, solving the problems of resource redundancy and waste, high energy consumption, and high operating costs caused by unreasonable resource allocation in existing electric bus systems. Attached Figure Description

[0017] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0018] Specific Implementation Method 1: The specific process of the charging scheduling optimization method for hybrid power supply electric bus system based on reinforcement learning in this implementation method is as follows:

[0019] Step 1: Define the state space;

[0020] Step 2: Define the action space;

[0021] Step 3: Define the reward function;

[0022] Step 4: Based on the state space, action space, and reward function, establish a multi-objective Markov decision model;

[0023] Step 5: Construct a reinforcement learning network model based on a multi-objective Markov decision model;

[0024] Step 6: Obtain the dataset;

[0025] Step 7: Train a reinforcement learning network model based on the dataset, obtain the trained reinforcement learning network model, and save the parameters;

[0026] Step 8: Obtain the charging scheme for the actual optimization period based on the trained reinforcement learning network model.

[0027] Specific Implementation Method Two: This implementation method differs from Specific Implementation Method One in that: a state space is defined in step 1; the specific process is as follows:

[0028] Step 1.1: Let L represent the number of bus routes; N represent the number of buses in the public transportation system;

[0029] This represents the average travel energy consumption of the bus route n. ;

[0030] K represents the number of charging stations. Indicates the power of the charging station;

[0031] A represents the photovoltaic deployment area. Indicates the rated power of the photovoltaic module;

[0032] Indicates the battery capacity of the energy storage device; Indicates the battery capacity of the bus;

[0033] The unit is kWh; The unit is kW; the unit of A is m. 2 ; The unit is kW; The unit is kWh; The unit is kWh;

[0034] Step 1.2, with To optimize the start time of the time period, To optimize the end time of the optimization period, the optimization period is divided into T steps, each lasting 1 minute;

[0035] Step 1.3

[0036] make Indicate whether bus n is in the middle of its journey at step t; if so, let... Otherwise, the value is 0;

[0037] make Indicate whether bus n will depart from the depot with a step length t; if so, let... Otherwise, the value is 0;

[0038] make This represents the state of charge (SOC) of the battery at a step length t for bus n. The unit is %

[0039] make Let represent the charging duration of bus n from the start time of its most recent charging to the start time of step t. If the bus has finished charging before the start of step t, then let . ; The unit is min;

[0040] This represents the remaining time until bus n departs next (starting from the initial time of step t). The unit is min;

[0041] Step 1.4, using This represents the state of charge (SOC) of the energy storage device at a step size t. The unit is %

[0042] SOC indicates the state of charge of the battery;

[0043] This indicates the charging duration of the energy storage device within a step size t. The unit is min;

[0044] Step 1.5, use Represents the irradiance at step size t. , The unit is kW / m 2 ;

[0045] Indicates the ambient temperature at step t. The unit is ℃;

[0046] This represents the power generation of the photovoltaic module within a step size t. The unit is kW;

[0047] , (See formula (7)) These represent the unit prices for electricity supplied by photovoltaic, energy storage, and the power grid, respectively. , , All units are in yuan / kWh;

[0048] Step 1.6: Treat the bus and energy storage device as independent intelligent entities;

[0049] use This represents the state of bus n at step size t. The composition is shown in formula (1);

[0050] use This indicates the state of the energy storage device at step size t. The composition is shown in formula (2);

[0051] use This represents the environmental state of the public transportation system at a step size t. The composition is shown in formula (3);

[0052] (1)

[0053] (2)

[0054] (3)

[0055] Step 1.7: Use the states of all agents and the environment state to represent the state of the public transportation system at step length t. As shown in formula (4);

[0056] (4)

[0057] In the formula, This indicates vector concatenation;

[0058] This represents the state of bus 1 at step length t;

[0059] This represents the state of bus 2 at step length t;

[0060] Indicates bus At step size t;

[0061] All intelligent agents are One bus and one energy storage device;

[0062] Step 1.8: The state space is constructed from the T steps of the bus system states. As shown in formula (5);

[0063] (5)

[0064] in, This indicates the status of the public transportation system with a step size of 1. This indicates the status of the public transportation system with a step size of 2. Indicates step size The status of the public transportation system.

[0065] The other steps and parameters are the same as in Specific Implementation Method 1.

[0066] Specific Implementation Method Three: This implementation method differs from Specific Implementation Method One or Two in that: step 2 defines the action space; the specific process is as follows:

[0067] Step 2.1, use This represents the actual movement of bus n within a step length t (corresponding to the charging method). Since the electricity for buses in the public transportation system comes from three sources: photovoltaics, energy storage, and the power grid, There are four possible values: 0, 1, 2, and 3, which represent that the bus n does not charge within a step length t, charges using photovoltaic power, charges using energy storage, and charges using the power grid, respectively.

[0068] Step 2.2, using This represents the actual action of the energy storage device within a step size t (corresponding to the charging method), considering that the power for energy storage comes from both photovoltaic and grid sources. There are three possible values: 0, 1, and 2, which represent that the energy storage device does not charge at step t, charges using photovoltaic power, and charges using the power grid, respectively.

[0069] Step 2.3: Represent the set of actual actions for step length t using the actual actions of all agents. (The charging scheme for the bus system at step length t) is shown in formula (6):

[0070] (6)

[0071] in, This represents the actual movement of bus 1 within a step length t. This represents the actual movement of bus 2 within a step length t. Indicates bus In the actual action of step length t.

[0072] Other steps and parameters are the same as in specific implementation method one or two.

[0073] Specific Implementation Method Four: This implementation method differs from Specific Implementation Methods One to Three in that a reward function is defined in step 3; the specific process is as follows:

[0074] Step 3.1: Define the unit price of energy storage charging. The calculation method It is equal to the ratio of the total charging cost up to the end of step t-1 to the total charging amount, as shown in formula (7):

[0075] (7)

[0076] In the formula,

[0077] This refers to the charging efficiency of the charging station; a value of 0.95 is recommended.

[0078] It is an indicator function, which is only valid when the condition inside the parentheses is true. The value is 1 if it is not 1, and 0 otherwise; for example, when hour, The value is 1 if it is 1, otherwise it is 0.

[0079] Step length The charging power when charging an energy storage device. ,in This is the minimum charging power for charging energy storage devices; a value of 40 is recommended, and the unit is kWh.

[0080] Indicates the energy storage device in step size The actual actions;

[0081] This indicates the unit price of electricity supplied by the power grid;

[0082] Step 3.2, use This represents the total charging cost of the public transportation system over a step length t. Equal to the charging cost of all buses in the bus system with a step length t. Charging costs of energy storage devices The sum is shown in formula (8):

[0083] (8)

[0084] Represented as:

[0085] (9)

[0086] Represented as:

[0087] (10)

[0088] In the formula, The step size t represents the charging power when the energy storage device is being charged.

[0089] The unit is yuan; The unit is yuan; The unit is yuan;

[0090] Step 3.3, use This represents the CO2 emissions during the production process of the public transportation system using electricity over a step length t. Equal to the CO2 emissions from photovoltaic power generation during the production process within the public transportation system (step length t). CO2 emissions from electricity consumption during production processes The sum is shown in formula (11):

[0091] (11)

[0092] Represented as:

[0093] (12)

[0094] Represented as:

[0095] (13)

[0096] In the formula, , These are the carbon emission factors of photovoltaics and the carbon emission factors of energy storage;

[0097] , All units are kgCO2 / kWh; The unit is kgCO2; The unit is kgCO2; The unit is kgCO2;

[0098] Step 3.4: Design constraint penalty items based on the actual operational constraints of the public transportation system;

[0099] The specific process is as follows:

[0100] Step 3.4.1: Construct charging position constraint penalty terms This is to restrict buses to charging only at bus depots. The calculation method is shown in formula (14):

[0101] (14)

[0102] In the formula, This is the suggested action obtained by bus n at step length t, and the method of obtaining it is shown in step 5.5;

[0103] Step 3.4.2: Construct battery state of charge constraint penalty terms , , , These are used to limit the SOC (State of Charge) of bus batteries and energy storage batteries between the lower and upper limits of SOC, respectively. , The calculation method is shown in formulas (15) and (16):

[0104] (15)

[0105] (16)

[0106] In the formula, and These are the lower and upper limits of the State of Charge (SOC) for bus batteries, respectively, in percentages (%).

[0107] and These are the lower and upper limits of the State of Charge (SOC) for energy storage batteries, in percentages (%).

[0108] Use intermediate variables express ;

[0109] Step 3.4.3: Construct charging time constraint penalty terms This is used to limit the charging time of a bus on a single charge to no less than the minimum charging time of the bus. , The unit is min. The calculation method is shown in formula (17):

[0110] (17)

[0111] In the formula, This indicates the minimum charging time for the bus.

[0112] Step 3.4.4: Construct the constraint and penalty terms for the number of charging buses. This is to limit the number of buses charging at bus depots at the same time to no more than the number of charging stations. The calculation method is shown in formula (18):

[0113] (18)

[0114] judge After addition and with Compare;

[0115] Step 3.4.5: Construct charging power constraint penalty terms This is used to limit the amount of photovoltaic power used by buses to no more than their total power generation. The calculation method is shown in formula (19):

[0116] (19)

[0117] judge Add and then multiply Later and Compare;

[0118] Step 3.5, use This represents the overall situation of constraint violations at time step t. The calculation method is shown in formula (20):

[0119] (20)

[0120] Step 3.6: Constructing the reward function requires designing the corresponding optimization objective. The optimization objective of this invention is to minimize the daily charging cost of the public transportation system and minimize CO2 emissions; therefore, a two-dimensional reward function is constructed. As shown in formula (21), From charging cost incentives and CO2 emission incentives composition, as well as The calculation method is shown in formulas (22) and (23):

[0121] (twenty one)

[0122] (twenty two)

[0123] (twenty three).

[0124] The other steps and parameters are the same as those in one of the specific implementation methods one to three.

[0125] Specific Implementation Method Five: This implementation method differs from Specific Implementation Methods One through Four in that: a multi-objective Markov decision model is established in step 4; the specific process is as follows:

[0126] Step 4.1, use This represents the discount factor, and a value of 0.99 is recommended. Furthermore, since the reinforcement learning charging scheduling algorithm involved in this invention can directly approximate the optimal strategy through interaction with the environment, the state transition probability is not explicitly defined.

[0127] Step 4.2: Combining the above, the bus charging scheduling problem under the "photovoltaic-storage-grid" hybrid energy supply mode is described as consisting of a state space (obtained in Step 1), an action space (obtained in Step 2), a reward function (obtained in Step 3), and a discount factor. A multi-objective Markov decision model is constructed (two objectives: minimizing the daily charging cost of the public transport system and minimizing CO2 emissions).

[0128] The other steps and parameters are the same as those in specific implementation methods one through four.

[0129] Specific Implementation Method Six: This implementation method differs from Specific Implementation Methods One through Five in that: in step 5, a reinforcement learning network model is constructed based on a multi-objective Markov decision model; the specific process is as follows:

[0130] Step 5.1: Select the average daily solar irradiance as the dividing condition for charging scheduling tasks, and divide the charging scheduling tasks into regular tasks when there is sufficient sunlight and special tasks when there is insufficient sunlight.

[0131] Sufficient sunlight refers to an average daily photovoltaic power generation exceeding [a certain value]. ;

[0132] Insufficient sunlight refers to a daily average photovoltaic power generation power of less than or equal to ;

[0133] Step 5.2: Build a shared network for multiple tasks. The shared network is used to extract common features from the input information. The shared network consists of three layers: MLP, GRU, and MLP.

[0134] The input information of the shared network is the state of bus n within step t. (Agent state, see step 1) Environmental state of the public transportation system at step t (Environmental conditions, see step 1) Bus In step length actual actions (See step 2 for historical actions) Overall situation of constraint violations (See step 3), date (Date and month information corresponding to the current charging scheduling task), step size and target preference vector (A two-dimensional vector, where the values ​​of the two elements are in...) The complex vector is composed of seven types of information (interval and sum to 1); the output is a vector containing the common features of the input information; as shown in formulas (24)-(26) (taking the bus agent n as an example to illustrate the information flow in the shared network):

[0135] (twenty four)

[0136] (25)

[0137] (26)

[0138] In the formula, It is an intermediate variable in a shared network;

[0139] Indicates step size The corresponding feature vector output by the gated recurrent unit (GRU);

[0140] Indicates vector concatenation; MLP represents a multilayer perceptron network;

[0141] Indicates step size The corresponding feature vector output by the gated recurrent unit (GRU); It is a zero vector; GRU stands for Gated Recurrent Unit;

[0142] This represents the output of a shared network.

[0143] The multilayer perceptron network (MLP) consists of a three-layer network structure: an input layer, a hidden layer, and an output layer. B represents the number of neurons in the hidden layer, and a value of 128 is recommended. Formulas (27)-(29) illustrate the information flow in the MLP network.

[0144] (27)

[0145] (28)

[0146] (29)

[0147] The simplified form of the MLP network is shown in equation (30):

[0148] (30)

[0149] In the formula, It is the feature vector input to the MLP;

[0150] and It is an intermediate variable in MLP;

[0151] , , These are the weight matrices for the input layer, hidden layer, and output layer of the MLP, respectively.

[0152] , These are the bias terms of the input layer, hidden layer, and output layer of the MLP (all weight matrices and bias terms are automatically updated during network training);

[0153] This is the output of the MLP;

[0154] express Activation function;

[0155] Step 5.3: Construct parallel task head networks for specific tasks, targeting optimization objective 1 and optimization objective 2; the specific process is as follows:

[0156] The parallel task head network for specific tasks consists of two parallel task head networks with the same structure, corresponding to the two optimization objectives of the model respectively. It can generate local action function values ​​(which can be understood as long-term cumulative rewards) in the corresponding task scenario based on the extracted common features.

[0157] Two parallel task headers with identical structures are a regular task header and a special task header; the structure is a multilayer perceptron network (MLP).

[0158] Equation (31) illustrates the information flow for bus n with optimization objective 1 in two parallel task head networks with identical structures:

[0159] (31)

[0160] Equation (32) illustrates the information flow for optimizing bus n in two parallel task head networks with identical structures:

[0161] (32)

[0162] In the formula,

[0163] The output of the task head network is represented by the Q-vector of the local action of bus n at step t under optimization objective 1. It contains the local action function values ​​that bus n can obtain by taking four actions at step t.

[0164] Optimization objective 1 is the total charging cost of the public transportation system over a step length t. ;

[0165] It is the task type; It's a routine task;

[0166] The output of the task head network is represented by the Q-vector of the local action of bus n at step t under optimization objective 2, which includes the local action function values ​​that bus n can obtain by taking four actions at step t.

[0167] Optimization objective 2 is to reduce CO2 emissions during the production process of the electricity used by the public transportation system over a step length t. ;

[0168] Step 5.4: Set a weighted greedy strategy to select suggested actions based on the local action Q vector;

[0169] The specific process is as follows:

[0170] Taking the bus agent n as an example, after obtaining the local action Q vector, the agent first uses formula (33) to calculate the comprehensive local action Q vector by weighting. ; indicates as:

[0171] (33)

[0172] In the formula, and These are the target preference vectors. The two elements in;

[0173] Then bus n will choose Find the element with the largest value, and the action corresponding to the element with the largest value. Suggested actions for bus n at step length t. (Values ​​0, 1, 2, and 3 correspond to different charging modes); as shown in formula (34):

[0174] (34)

[0175] In the formula, It is the action used to retrieve the element with the maximum value. ;

[0176] Step 5.5: Calculate the total number of records of constraint violations for the suggested actions selected in Step 5.4. And by using action correction strategies, suggested actions that do not meet the constraints are corrected to obtain the actual action set;

[0177] The specific process is as follows:

[0178] Step 5.5.1: Obtain the set of buses whose step size t has a non-zero value. (Buses with non-zero action values ​​are charging buses, as determined in step 5.4);

[0179] Step 5.5.2: Calculate the set based on formula (35) Charging priority for all vehicles in the country:

[0180] (35)

[0181] In the formula, This indicates the charging priority of bus n within a step size t;

[0182] Step 5.5.3: For the charging position constraint, set the action of the bus in motion to 0 and set the charging duration of the bus in motion to 0.

[0183] Step 5.5.4: Regarding battery state of charge constraints, set the actions of buses whose battery SOC reaches the upper limit to 0; set the actions of buses whose battery SOC is less than the upper limit to 0. The bus's action is set to 1, and the battery SOC is less than 1. The charging priority of buses has been adjusted to the maximum value. A value of 5000 is recommended.

[0184] Step 5.5.5: Regarding the charging time constraint, set the charging duration to... However, the action of the bus with an action of 0 is set to 1, and all charging durations are kept in [a certain state]. The charging priority of buses has been adjusted to ;

[0185] Step 5.5.6: Regarding the constraint on the number of charging buses, when the number of buses charging at the same time is greater than the number of charging piles, charging piles are allocated to buses according to charging priority from high to low until the charging pile resources are exhausted, and the action of buses that have not been allocated charging piles is set to 0.

[0186] If there are buses with the same charging priority, a bus will be randomly selected and a charging station will be assigned to it (for example, if it is the turn of a bus with this priority, there are 5 buses with this priority. If there are enough charging stations, each bus will be assigned a charging station. If there are not enough charging stations, a bus will be randomly selected and a charging station will be assigned to it).

[0187] Step 5.5.7: Simultaneously consider the photovoltaic power generation situation and the charging power constraint of the energy storage SOC;

[0188] Step 5.5.8: Set the suggested actions of all current agents as actual actions to determine the set of actual actions;

[0189] Step 5.6: Construct a parallel hybrid network for multiple tasks. This network also has two parallel networks with identical structures to correspond to the two optimization objectives of the model; the specific process is as follows:

[0190] Hybrid networks consist of a weight parameter network and a value function hybrid network;

[0191] The weight parameter network consists of four supernetworks; the value function hybrid network consists of two linear networks.

[0192] The four networks in the weighted parameter network all take the bus system state as input with a step size t as input. The outputs correspond to the weights of the two linear networks, respectively. , ( ), and deviation , Among them, except The parameters are obtained by calculation using a single-layer linear network, while the remaining parameters are obtained by fitting a two-layer linear network with the ReLU function; Formulas (36)-(39) show the information flow in the weight parameter network;

[0193] (36)

[0194] (37)

[0195] (38)

[0196] (39)

[0197] In the formula,

[0198] , It is the weight matrix of supernetwork 1; , It is the weight matrix of HyperNet 2; It is the weight matrix of HyperNet 3; , It is the weight matrix of HyperNet 4;

[0199] , It is the bias term of supernetwork 1; , It is the bias term of SuperNetwork 2; It is the bias term of SuperNetwork 3; , It is the bias term of SuperNetwork 4;

[0200] Indicate the goal, Indicates target 1. Indicates objective 2;

[0201] Input information of value function hybrid networks It is composed of the local action function values ​​corresponding to the actual actions of each agent. Taking the bus agent 1 as an example, agent 1 will construct the action function based on the actual actions taken at step t. and Extract the corresponding values ​​and place them in and The output of the hybrid network with a median function is the Q-value of the joint action of each objective at step size t. ;

[0202] Combining the output of the weighted parameter network, the information flow in the value function hybrid network can be expressed as follows:

[0203] (40)

[0204] (41)

[0205] In the formula, It is an intermediate variable in a value function hybrid network;

[0206] It is an absolute value operation, and the non-negative weights enable the local action value function network and the joint value function network to optimize in the same direction during training;

[0207] This indicates summing the elements of a vector.

[0208] Combining shared networks, parallel task head networks, weighted greedy policies, action correction policies, and parallel hybrid networks forms the framework for multi-task head multi-agent deep reinforcement learning algorithms (M3RL).

[0209] The other steps and parameters are the same as those in specific implementation methods one through five.

[0210] Specific Implementation Method Seven: This implementation method differs from Specific Implementation Methods One through Six in that step 5.5.7 simultaneously considers both photovoltaic power generation and the charging power constraint imposed by the energy storage SOC; the specific process is as follows:

[0211] Step 5.5.7.1: Obtain the set of vehicles using photovoltaic charging with a step size t. (As determined in step 5.4), the set of vehicles using energy storage charging is defined as follows: (As determined in step 5.4), proceed to step 5.5.7.2;

[0212] Step 5.5.7.2: Based on formula (42), calculate the total charging amount of the bus using photovoltaic power within step length t. Proceed to step 5.5.7.3;

[0213] (42)

[0214] Step 5.5.7.3, if Then in Randomly select a bus and replace the suggested action of the randomly selected bus with the action of using energy storage to charge, and then reacquire energy. If not, proceed to step 5.5.7.4; otherwise, proceed to step 5.5.7.6.

[0215] Step 5.5.7.4: Based on formula (43), calculate the total charging amount of the bus using the energy storage device within the step length t. ;like If the suggested action for the randomly selected bus in step 5.5.7.3 is replaced with charging via the power grid, then proceed to step 5.5.7.5; otherwise, proceed to step 5.5.7.6.

[0216] (43)

[0217] Step 5.5.7.5: Return to step 5.6.7.1 (at this point, the data in step 5.6.7.1 is also updated);

[0218] Step 5.5.7.6: Calculate based on formula (43) ;like Then in Randomly select a bus and replace the suggested action of the randomly selected bus with charging via the power grid, then proceed to step 5.5.7.7; otherwise, proceed to step 5.5.7.8.

[0219] Step 5.5.7.7, Reacquire Return to step 5.5.7.6;

[0220] Step 5.5.7.8: Calculate the remaining photovoltaic power based on formula (44). ;like and At that time, the operation of the energy storage device is set to 1, and the charging power is... Set as Conversely, the operation of the energy storage device is set to zero.

[0221] (44)

[0222] The other steps and parameters are the same as those in specific implementation methods one through six.

[0223] Specific Implementation Method Eight: This implementation method differs from Specific Implementation Methods One through Seven in that: in step 6, the dataset is acquired; the specific process is as follows:

[0224] Step 6.1

[0225] Collect data from actual bus stations (e.g., for one consecutive year). A bus is in a state with step length t. The energy storage device in the state of step t The environmental state of the public transportation system in step t ;

[0226] Step 6.2: Use the collected data as the training set;

[0227] Step 6.3: Set up a sampling method that combines sequential sampling and random sampling; specifically:

[0228] When the number of training iterations is less than 50, data is input in the order of spring, summer, autumn, and winter to help the agent learn seasonal patterns; when the number of training iterations is greater than or equal to 50, random input is used to enhance the generalization ability of the reinforcement learning solution network model in step 5.

[0229] The other steps and parameters are the same as those in any of the specific implementation methods one to seven.

[0230] Specific Implementation Method Nine: This implementation method differs from Specific Implementation Methods One through Eight in that: in step 7, a reinforcement learning network model is trained based on the dataset to obtain the trained reinforcement learning network model and save the parameters; the specific process is as follows:

[0231] Step 7.1: Based on the number of optimization objectives (2), set up two objective preference subspaces, namely:

[0232] ;

[0233] ;

[0234] The sampling interval for the target preference subspace. A value of 0.1 is recommended.

[0235] Step 7.2: Construct two reinforcement learning network models as described in Step 5, serving as the evaluation network and the target network, respectively;

[0236] Set two lengths as The experience pools are used to store experience data from regular tasks and special tasks, respectively. ; A value of 5000 is recommended.

[0237] in, Indicates step size The action; Indicates step size The action; Indicates step size The status of the public transportation system; Indicates step size The status of the public transportation system; Indicates step size The reward;

[0238] Step 7.3

[0239] Set the maximum number of iterations. (Recommended value: 2000), Minimum number of update iterations (Recommended value: 50), Batch sample data volume (Recommended value: 32), Learning rate (Recommended value) );

[0240] Set the initial value, increment, and final value of the special experience sampling ratio for regular tasks. , , (Recommended values ​​are 0.1, 0.01, and 0.2);

[0241] Set the initial value, increment, and final value of the sampling ratio for special tasks and special experiences. , , (Recommended values ​​are 0.2, 0.05, and 0.8);

[0242] Randomly initialize the evaluation network parameters (including weights and biases), and then copy the evaluation network parameters to the target network.

[0243] Step 7.4: Sample from the training set, activate the task head network for regular or special tasks according to the sampled data type, and obtain the current task date; randomly obtain a set of preference vectors from the target preference subspace;

[0244] Step 7.5: Each agent observes the environment and returns the corresponding state and action information. Based on the shared network, the parallel task head network for specific tasks for optimization objective 1 and optimization objective 2, and the weighted greedy strategy, the agent obtains the suggested action and calculates the overall situation of constraint violation of the suggested action.

[0245] Step 7.6: Each agent uses the action correction strategy to correct the suggested actions that do not meet the constraints, obtains the actual action set, returns the actual action set to the environment for execution, the environment feeds back a reward vector and transitions to the next state;

[0246] Step 7.7: Adjust the step size Actions, stride length Actions, stride length Public transportation system status, step length The status and step length of the public transportation system The rewards are stored as experience in the experience pool of the corresponding regular or special task;

[0247] Step 7.8: When the number of experiences in the experience pool exceeds the threshold (A value of 100 is recommended) when drawing from the experience pool. The parameters of the reinforcement learning network model described in step 5 are updated using empirical data from a sample.

[0248] When the number of iterations is less than At that time, the sample experience data only came from the regular task experience pool;

[0249] When the number of iterations is greater than or equal to For routine tasks, the sample experience data will be processed according to... , The proportion is randomly obtained from the regular task experience pool and the special task pool;

[0250] When the number of iterations is greater than or equal to At that time, for specific tasks, the sample experience data will be processed according to... , The proportion is randomly obtained from the regular task experience pool and the special task pool;

[0251] Every In the next iteration, for routine tasks, the reinforcement learning network model will... The increase in the percentage of samples will increase the sampling ratio of special experiences under both routine and special tasks (sample experience data will be distributed according to...). , The proportion is randomly obtained from the regular task experience pool and the special task pool until it reaches the preset maximum value;

[0252] Every In the next iteration, for specific tasks, the reinforcement learning network model will... The increase in the percentage of samples will increase the sampling ratio of special experiences under both routine and special tasks (sample experience data will be distributed according to...). , The proportion is randomly obtained from the regular task experience pool and the special task pool until it reaches the preset maximum value;

[0253] This is a constant; a value of 10 is recommended.

[0254] Step 7.9: Based on the extracted sample experience data, calculate the loss function according to formulas (45)-(47). ; indicates as:

[0255] (45)

[0256] (46)

[0257] (47)

[0258] In the formula,

[0259] It is the first The loss value of each target under the current batch of samples;

[0260] It is sample experience In the evaluation network, the first The Q-value of the joint action under one objective The corresponding actions are the actual action sets recorded in the experience pool;

[0261] It is to evaluate network parameters;

[0262] It is sample experience In the Q-value of the joint action of targets under one objective;

[0263] These are the target network parameters;

[0264] It is sample experience Middle The reward value for each objective;

[0265] It is sample experience In the target network, the first Q-value of joint actions under multiple objectives; The corresponding action is the set of actions obtained by the target network, with the next state as input.

[0266] Step 7.10, based on The gradient is calculated using backpropagation, and the parameters in the reinforcement learning solution network model described in step 5 are updated using the Adam optimizer.

[0267] Step 7.11: Remove dominated experience from the experience pool;

[0268] Dominated means that, under the same local observation conditions, the current solution is not better than another solution on all objectives, and is strictly inferior to another solution on at least one objective function;

[0269] Step 7.12: Increase the sampling probability of the empirically sparse preference subspace (two optimization objectives, such as favoring the first objective, i.e.) There are 20 data points where the weight corresponding to optimization objective 1 is greater than the weight corresponding to optimization objective 2; the focus is more on the second objective. There are 80 data points whose weights corresponding to optimization objective 1 are less than those corresponding to optimization objective 2; therefore, we should increase the number of data points that fall under the first category to enhance the exploration of that area.

[0270] Step 7.13, repeat steps 7.4-7.12. Next, the evaluation network parameters are copied to the target network;

[0271] This is a constant; a value of 10 is recommended.

[0272] Step 7.14: Repeat steps 7.4 to 7.13 until the number of iterations reaches the preset maximum number of iterations, obtain the trained reinforcement learning solution network model, and save the parameters.

[0273] The validation set (data collected over a year is divided into two independent parts. One part is used as the training set to train model parameters. The other part is data that the model has not seen or been trained on. However, because its source is consistent and its overall distribution is consistent with the training set, this part of the data can verify the model's performance) evaluates the network performance. If the generated charging scheduling scheme can meet the real-time charging scheduling needs of electric buses while reducing operating costs and carbon emissions, the final parameters of the shared network and the parallel task head network in the evaluation network are output and saved.

[0274] The other steps and parameters are the same as those in specific implementation methods one through eight.

[0275] Specific Implementation Method Ten: This implementation method differs from Specific Implementation Methods One through Nine in that: in step 8, the charging scheme for the actual optimization period is obtained based on the trained reinforcement learning network model; the specific process is as follows:

[0276] Step 8.1: Determine the actual optimization period and the user's target preference weights, and load the final saved model parameters before starting optimization;

[0277] Step 8.2: Obtain the status information of the bus and energy storage, and input it together with the target preference weights into the shared network of the trained reinforcement learning network model. The output of the shared network is input into the task head network, and the actual action set is obtained by combining the weighted greedy strategy and the action correction strategy. The actual action set is then converted into a charging scheme for execution.

[0278] Step 8.3: Repeat step 8.2 until the end of the optimization period.

[0279] The other steps and parameters are the same as those in specific implementation methods one through nine.

[0280] This invention may have other embodiments. Without departing from the spirit and essence of this invention, those skilled in the art can make various corresponding changes and modifications according to this invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A method for optimizing charging scheduling in a hybrid-powered electric bus system based on reinforcement learning, characterized by: The specific process of the method is as follows: Step 1: Define the state space; Step 2: Define the action space; Step 3: Define the reward function; Step 4: Based on the state space, action space, and reward function, establish a multi-objective Markov decision model; Step 5: Construct a reinforcement learning network model based on a multi-objective Markov decision model; the specific process is as follows: Step 5.1: Divide the charging scheduling task into regular tasks when there is sufficient light and special tasks when there is insufficient light. Sufficient sunlight refers to an average daily photovoltaic power generation exceeding [a certain value]. ; Insufficient sunlight refers to a daily average photovoltaic power generation power of less than or equal to ; in It is the minimum charging power to charge an energy storage device; Step 5.2: Build a shared network for multiple tasks. The shared network is used to extract common features from the input information. The shared network consists of three layers: MLP, GRU, and MLP. Step 5.3: Construct parallel task head networks for specific tasks, targeting optimization objective 1 and optimization objective 2; the specific process is as follows: The parallel task head network for specific tasks consists of two parallel task head networks with identical structures; Two parallel task headers with identical structures are a regular task header and a special task header; the structure is a multilayer perceptron network (MLP). Optimization objective 1 is the total charging cost of the public transportation system over a step length t. ; Optimization objective 2 is to reduce CO2 emissions during the production process of the electricity used by the public transportation system over a step length t. ; Step 5.4: Set a weighted greedy strategy to select suggested actions based on the local action Q vector; Step 5.5: Calculate the total number of records of constraint violations for the suggested actions selected in Step 5.

4. And by using action correction strategies, suggested actions that do not meet the constraints are corrected to obtain the actual action set; Step 5.6: Build a hybrid network; the specific process is as follows: Hybrid networks consist of a weight parameter network and a value function hybrid network; The weight parameter network consists of four supernetworks; the value function hybrid network consists of two linear networks. Step 6: Obtain the dataset; Step 7: Train a reinforcement learning network model based on the dataset, obtain the trained reinforcement learning network model, and save the parameters; Step 8: Obtain the charging scheme for the actual optimization period based on the trained reinforcement learning network model.

2. The method for optimizing charging scheduling of a hybrid power supply electric bus system based on reinforcement learning according to claim 1, characterized in that: Step 1 defines the state space; the specific process is as follows: Step 1.1: Let L represent the number of bus routes; N represent the number of buses in the public transportation system; This represents the average travel energy consumption of bus route n. ; K represents the number of charging stations. Indicates the charging pile's power; A represents the area of ​​photovoltaic installation. Indicates the rated power of the photovoltaic module; Indicates the battery capacity of the energy storage device; Indicates the battery capacity of the bus; Step 1.2, with To optimize the start time of the time period, To optimize the end time of the optimization period, the optimization period is divided into T steps, each lasting 1 minute; Step 1.3 make Indicate whether bus n is in the middle of its journey at step t; if so, let... Otherwise, the value is 0; make Indicate whether bus n will depart from the depot with a step length t; if so, let... Otherwise, the value is 0; make This represents the battery state of bus n at a step size t; make Let represent the charging duration of bus n from the start time of its most recent charging to the start time of step t. If the bus has finished charging before the start of step t, then let . ; This represents the remaining time until bus n departs next. Step 1.4, using This represents the state of charge (SOC) of the energy storage device at a step size t; SOC indicates the state of charge of the battery; This indicates the charging duration of the energy storage device within a step size t; Step 1.5, use Represents the irradiance at step size t. ; Indicates the ambient temperature at step t; This represents the power generation of the photovoltaic module within a step size t; , , These represent the unit price of electricity supplied by photovoltaic, energy storage, and grid, respectively. Step 1.6: Treat the bus and energy storage device as independent intelligent entities; use This represents the state of bus n at step size t. The composition is shown in formula (1); use This indicates the state of the energy storage device at step size t. The composition is shown in formula (2); use This represents the environmental state of the public transportation system at a step size t. The composition is shown in formula (3); (1) (2) (3) Step 1.7: Use the states of all agents and the environment state to represent the state of the public transportation system at step length t. As shown in formula (4); (4) In the formula, This indicates vector concatenation; This represents the state of bus 1 at step length t; This represents the state of bus 2 at step length t; Indicates bus In the state at step length t; All intelligent agents are One bus and one energy storage device; Step 1.8: The state space is constructed from the T steps of the bus system states. As shown in formula (5); (5) in, This indicates the status of the public transportation system with a step size of 1. This indicates the status of the public transportation system with a step size of 2. Indicates step size The status of the public transportation system.

3. The method for optimizing charging scheduling of a hybrid power supply electric bus system based on reinforcement learning according to claim 2, characterized in that: Step 2 defines the action space; the specific process is as follows: Step 2.1, use This represents the actual movement of bus n within a step length t. There are four possible values: 0, 1, 2, and 3, which represent that the bus n does not charge within a step length t, charges using photovoltaic power, charges using energy storage, and charges using the power grid, respectively. Step 2.2, using This indicates the actual action of the energy storage device within a step size t. There are three possible values: 0, 1, and 2, which represent that the energy storage device does not charge at step t, charges using photovoltaic power, and charges using the power grid, respectively. Step 2.3: Represent the set of actual actions for step length t using the actual actions of all agents. As shown in formula (6): (6) in, This represents the actual movement of bus 1 within a step length t. This represents the actual movement of bus 2 within a step length t. Indicates bus In the actual action of step length t.

4. The method for optimizing charging scheduling of a hybrid power supply electric bus system based on reinforcement learning according to claim 3, characterized in that: Step 3 defines the reward function; the specific process is as follows: Step 3.1: Define the unit price of energy storage charging The calculation method It is equal to the ratio of the total charging cost up to the end of step t-1 to the total charging amount, as shown in formula (7): (7) In the formula, It refers to the charging efficiency of the charging station; It is an indicator function, which is only valid when the condition inside the parentheses is true. The value is 1 if it is set to 1, otherwise the value is 0. Step length Charging power when charging an energy storage device. ,in It is the minimum charging power to charge an energy storage device; Indicates the energy storage device in step size The actual actions; This indicates the unit price of electricity supplied by the power grid; Step 3.2, use This represents the total charging cost of the public transportation system over a step length t. Equal to the charging cost of all buses in the bus system with a step length t. Charging costs of energy storage devices The sum is shown in formula (8): (8) Represented as: (9) Represented as: (10) In the formula, The step size t represents the charging power when the energy storage device is being charged. Step 3.3, use This represents the CO2 emissions during the production process of the public transportation system using electricity over a step length t. Equal to the CO2 emissions from photovoltaic power generation during the production process within the public transportation system (step length t). CO2 emissions from electricity consumption during production processes The sum is shown in formula (11): (11) Represented as: (12) Represented as: (13) In the formula, , These are the carbon emission factors of photovoltaics and the carbon emission factors of energy storage; Step 3.4: Design constraint penalty items based on the actual operational constraints of the public transportation system; The specific process is as follows: Step 3.4.1: Construct charging position constraint penalty terms , The calculation method is shown in formula (14): (14) In the formula, This is the suggested action obtained by bus n at step length t; Step 3.4.2: Construct battery state of charge constraint penalty terms , , , The calculation method is shown in formulas (15) and (16): (15) (16) In the formula, and These are the lower and upper limits of the State of Charge (SOC) for bus batteries, respectively. and These are the lower and upper limits of the SOC of energy storage batteries, respectively. Use intermediate variables express ; Step 3.4.3: Construct charging time constraint penalty terms , The calculation method is shown in formula (17): (17) In the formula, This indicates the minimum charging time for the bus. Step 3.4.4: Construct the constraint and penalty terms for the number of charging buses. , The calculation method is shown in formula (18): (18) Step 3.4.5: Construct charging power constraint penalty terms , The calculation method is shown in formula (19): (19) Step 3.5, use This represents the overall situation of constraint violations at time step t. The calculation method is shown in formula (20): (20) Step 3.6: Construct a two-dimensional reward function As shown in formula (21), From charging cost incentives and CO2 emission incentives composition, as well as The calculation method is shown in formulas (22) and (23): (21) (22) (23)。 5. The method for optimizing charging scheduling of a hybrid power supply electric bus system based on reinforcement learning according to claim 4, characterized in that: Step 4 involves establishing a multi-objective Markov decision model; the specific process is as follows: Step 4.1, use Indicates the discount factor; Step 4.2: The state space, action space, reward function, and discount factor are defined. To form a multi-objective Markov decision model.

6. The method for optimizing charging scheduling of a hybrid power supply electric bus system based on reinforcement learning according to claim 5, characterized in that: In step 5, a reinforcement learning network model is constructed based on a multi-objective Markov decision model; the specific process is as follows: Step 5.1: Divide the charging scheduling task into regular tasks when there is sufficient light and special tasks when there is insufficient light. Sufficient sunlight refers to an average daily photovoltaic power generation exceeding [a certain value]. ; Insufficient sunlight refers to a daily average photovoltaic power generation power of less than or equal to ; Step 5.2: Build a shared network for multiple tasks. The shared network is used to extract common features from the input information. The shared network consists of three layers: MLP, GRU, and MLP. The input information of the shared network is the state of bus n within step t. The environmental state of the public transportation system in step t ,the bus In step length actual actions Overall situation of constraint violations ,date Step length and target preference vector A complex vector composed of seven types of information; the output is a vector containing common features of the input information; as shown in formulas (24)-(26): (24) (25) (26) In the formula, It is an intermediate variable in a shared network; Indicates step size The corresponding feature vector output by the gated recurrent unit (GRU); Indicates vector concatenation; MLP represents a multilayer perceptron network; Indicates step size The corresponding feature vector output by the gated recurrent unit (GRU); It is a zero vector; GRU stands for Gated Recurrent Unit; This represents the output of a shared network. The multilayer perceptron network (MLP) consists of a three-layer network structure: an input layer, a hidden layer, and an output layer. B represents the number of neurons in the hidden layer. Formulas (27)-(29) illustrate the information flow in the MLP network. (27) (28) (29) The simplified form of the MLP network is shown in equation (30): (30) In the formula, It is the feature vector input to the MLP; and It is an intermediate variable in MLP; , , These are the weight matrices for the input layer, hidden layer, and output layer of the MLP, respectively. , These are the bias terms for the input layer, hidden layer, and output layer of the MLP, respectively. This is the output of the MLP; express Activation function; Step 5.3: Construct parallel task head networks for specific tasks, targeting optimization objective 1 and optimization objective 2; the specific process is as follows: The parallel task head network for specific tasks consists of two parallel task head networks with identical structures; Two parallel task headers with the same structure are a regular task header and a special task header; The structure is a multilayer perceptron network (MLP); Equation (31) illustrates the information flow for bus n with optimization objective 1 in two parallel task head networks with identical structures: (31) Equation (32) illustrates the information flow for optimizing bus n in two parallel task head networks with identical structures: (32) In the formula, The output of the task head network is represented by the Q-vector of the local action of bus n at step t under optimization objective 1. It contains the local action function values ​​that bus n can obtain by taking four actions at step t. Optimization objective 1 is the total charging cost of the public transportation system over a step length t. ; It is the task type; It's a routine task; The output of the task head network is represented by the Q-vector of the local action of bus n at step t under optimization objective 2, which includes the local action function values ​​that bus n can obtain by taking four actions at step t. Optimization objective 2 is to reduce CO2 emissions during the production process of the electricity used by the public transportation system over a step length t. ; Step 5.4: Set a weighted greedy strategy to select suggested actions based on the local action Q vector; The specific process is as follows: First, the weighted calculation of the comprehensive local action Q vector is performed using formula (33). ; indicates as: (33) In the formula, and These are the target preference vectors. The two elements in; Then bus n will choose Find the element with the largest value, and the action corresponding to the element with the largest value. Suggested actions for bus n at step length t. As shown in formula (34): (34) In the formula, It is the action used to retrieve the element with the maximum value. ; Step 5.5: Calculate the total number of records of constraint violations for the suggested actions selected in Step 5.

4. And by using action correction strategies, suggested actions that do not meet the constraints are corrected to obtain the actual action set; The specific process is as follows: Step 5.5.1: Obtain the set of buses whose step size t has a non-zero value. ; Step 5.5.2: Calculate the set based on formula (35) Charging priority for all vehicles in the country: (35) In the formula, This indicates the charging priority of bus n within a step size t; Step 5.5.3: For the charging position constraint, set the action of the bus in motion to 0 and set the charging duration of the bus in motion to 0. Step 5.5.4: Regarding battery state of charge constraints, set the actions of buses whose battery SOC reaches the upper limit to 0; set the actions of buses whose battery SOC is less than the upper limit to 0. The bus's action is set to 1, and the battery SOC is less than 1. The charging priority of buses has been adjusted to the maximum value. ; Step 5.5.5: Regarding the charging time constraint, set the charging duration to... However, the action of the bus with an action of 0 is set to 1, and all charging durations are kept in [a certain state]. The charging priority of buses has been adjusted to ; Step 5.5.6: Regarding the constraint on the number of charging buses, when the number of buses charging at the same time is greater than the number of charging piles, charging piles are allocated to buses according to charging priority from high to low until the charging pile resources are exhausted, and the action of buses that have not been allocated charging piles is set to 0. If there are buses with the same charging priority, then a charging station will be randomly selected and assigned. Step 5.5.7: Simultaneously consider the photovoltaic power generation situation and the charging power constraint of the energy storage SOC; Step 5.5.8: Set the suggested actions of all current agents as actual actions to determine the set of actual actions; Step 5.6: Build a hybrid network; the specific process is as follows: Hybrid networks consist of a weight parameter network and a value function hybrid network; The weight parameter network consists of four supernetworks; the value function hybrid network consists of two linear networks. The four networks in the weighted parameter network all take the bus system state as input with a step size t as input. The outputs correspond to the weights of the two linear networks, respectively. , and deviation , Formulas (36)-(39) illustrate the information flow in the weighted parameter network; (36) (37) (38) (39) In the formula, , It is the weight matrix of SuperNetwork 1; , It is the weight matrix of HyperNet 2; It is the weight matrix of HyperNet 3; , It is the weight matrix of HyperNet 4; , It is the bias term of supernetwork 1; , It is the bias term of SuperNetwork 2; It is the bias term of SuperNetwork 3; , It is the bias term of SuperNetwork 4; Indicate the goal, Indicates objective 1. Indicates objective 2; Input information of value function hybrid networks The output of the value function hybrid network is composed of the local action function values ​​corresponding to the actual actions of each agent; the output of the value function hybrid network is the joint action Q value of each objective at step t. ; Combining the output of the weighted parameter network, the information flow in the value function hybrid network can be expressed as follows: (40) (41) In the formula, It is an intermediate variable in a value function hybrid network; It is an absolute value operation; This indicates summing the elements of a vector.

7. The method for optimizing charging scheduling of a hybrid power supply electric bus system based on reinforcement learning according to claim 6, characterized in that: Step 5.5.7 considers both photovoltaic power generation and the charging power constraint imposed by the energy storage SOC; the specific process is as follows: Step 5.5.7.1: Obtain the set of vehicles using photovoltaic charging with a step size t. The set of vehicles that use energy storage for charging is defined as follows: Proceed to step 5.5.7.2; Step 5.5.7.2: Based on formula (42), calculate the total charging amount of the bus using photovoltaic power within step length t. Proceed to step 5.5.7.3; (42) Step 5.5.7.3, if Then in Randomly select a bus and replace the suggested action of the randomly selected bus with the action of using energy storage to charge, and then reacquire energy. Proceed to step 5.5.7.4; Otherwise, proceed to step 5.5.7.6; Step 5.5.7.4: Based on formula (43), calculate the total charging amount of the bus using the energy storage device within the step length t. ;like If the suggested action for the randomly selected bus in step 5.5.7.3 is replaced with charging via the power grid, then proceed to step 5.5.7.5; otherwise, proceed to step 5.5.7.

6. (43) Step 5.5.7.5, return to step 5.6.7.1; Step 5.5.7.6: Calculate based on formula (43) ;like Then in Randomly select a bus and replace the suggested action of the randomly selected bus with charging via the power grid, then proceed to step 5.5.7.7; otherwise, proceed to step 5.5.7.

8. Step 5.5.7.7, Reacquire Return to step 5.5.7.6; Step 5.5.7.8: Calculate the remaining photovoltaic power based on formula (44). ;like and At that time, the operation of the energy storage device is set to 1, and the charging power is... Set as ; Conversely, the operation of the energy storage device is set to zero; (44)。 8. The method for optimizing charging scheduling of a hybrid power supply electric bus system based on reinforcement learning according to claim 7, characterized in that: Step 6 involves obtaining the dataset; the specific process is as follows: Step 6.1: Collect data from actual bus stations. A bus is in a state with step length t. The energy storage device in the state of step t The environmental state of the public transportation system in step t ; Step 6.2: Use the collected data as the training set; Step 6.3: Set up a sampling method that combines sequential sampling and random sampling; specifically: When the number of training iterations is less than 50, data is entered in the order of spring, summer, autumn, and winter; when the number of training iterations is greater than or equal to 50, random input is used.

9. The method for optimizing charging scheduling of a hybrid power supply electric bus system based on reinforcement learning according to claim 8, characterized in that: In step 7, a reinforcement learning network model is trained based on the dataset to obtain the trained reinforcement learning network model and save the parameters; the specific process is as follows: Step 7.1: Set up two target preference subspaces based on the number of optimization targets, namely: ； ； The sampling interval for the target preference subspace; Step 7.2: Construct two reinforcement learning network models as described in Step 5, serving as the evaluation network and the target network, respectively; Set two lengths as The experience pools are used to store experience data from regular tasks and special tasks, respectively. ; in, Indicates step size The action; Indicates step size The action; Indicates step size The status of the public transportation system; Indicates step size The status of the public transportation system; Indicates step size The reward; Step 7.3 Set the maximum number of iterations. Minimum number of update iterations Batch sample data volume Learning rate ; Set the initial value, increment, and final value of the special experience sampling ratio for regular tasks. , , ; Set the initial value, increment, and final value of the sampling ratio for special tasks and special experiences. , , ; Randomly initialize the evaluation network parameters and copy them to the target network; Step 7.4: Sample from the training set, activate the task head network for regular or special tasks according to the sampled data type, and obtain the current task date; randomly obtain a set of preference vectors from the target preference subspace; Step 7.5: Each agent observes the environment and returns the corresponding state and action information. Based on the shared network, the parallel task head network for specific tasks for optimization objective 1 and optimization objective 2, and the weighted greedy strategy, the agent obtains the suggested action and calculates the overall situation of constraint violation of the suggested action. Step 7.6: Each agent uses the action correction strategy to correct the suggested actions that do not meet the constraints, obtains the actual action set, returns the actual action set to the environment for execution, the environment feeds back a reward vector and transitions to the next state; Step 7.7: Adjust the step size Actions, stride length Actions, stride length Public transportation system status, step length The status and step length of the public transportation system The rewards are stored as experience in the experience pool of the corresponding regular or special task; Step 7.8: When the number of experiences in the experience pool exceeds the threshold At that time, draw from the experience pool The parameters of the reinforcement learning network model described in step 5 are updated using empirical data from a sample. When the number of iterations is less than At that time, the sample experience data only came from the regular task experience pool; When the number of iterations is greater than or equal to For routine tasks, the sample experience data will be processed according to... , The proportion is randomly obtained from the regular task experience pool and the special task pool; When the number of iterations is greater than or equal to At that time, for specific tasks, the sample experience data will be processed according to... , The proportion is randomly obtained from the regular task experience pool and the special task pool; Every In the next iteration, for routine tasks, the reinforcement learning network model will... The increase will increase the sampling ratio of special experiences under regular tasks and special tasks until it reaches the preset maximum value; Every In the next iteration, for specific tasks, the reinforcement learning network model will... The increase will increase the sampling ratio of special experiences under regular tasks and special tasks until it reaches the preset maximum value; It is a constant; Step 7.9: Based on the extracted sample experience data, calculate the loss function according to formulas (45)-(47). ; indicates as: (45) (46) (47) In the formula, It is the first The loss value of each target under the current batch of samples; It is sample experience In the evaluation network, the first Q-value of joint actions under multiple objectives; It is to evaluate network parameters; It is sample experience In the Q-value of the joint action of targets under one objective; These are the target network parameters; It is sample experience The Middle The reward value for each objective; It is sample experience In the target network, the first Q-value of joint actions under multiple objectives; Step 7.10, based on The gradient is calculated using backpropagation, and the parameters in the reinforcement learning solution network model described in step 5 are updated using the Adam optimizer. Step 7.11: Remove dominated experience from the experience pool; Step 7.12: Increase the sampling probability of the empirically sparse preference subspace; Step 7.13, repeat steps 7.4-7.

12. Next, the evaluation network parameters are copied to the target network; It is a constant; Step 7.14: Repeat steps 7.4 to 7.13 until the number of iterations reaches the preset maximum number of iterations, obtain the trained reinforcement learning solution network model, and save the parameters.

10. The method for optimizing charging scheduling of a hybrid power supply electric bus system based on reinforcement learning according to claim 9, characterized in that: In step 8, the charging scheme for the actual optimization period is obtained based on the trained reinforcement learning network model. The specific process is as follows: Step 8.1: Determine the actual optimization period and the weight of user target preferences; Step 8.2: Obtain the status information of the bus and energy storage, and input it together with the target preference weights into the shared network of the trained reinforcement learning network model. The output of the shared network is input into the task head network, and the actual action set is obtained by combining the weighted greedy strategy and the action correction strategy. The actual action set is then converted into a charging scheme for execution. Step 8.3: Repeat step 8.2 until the end of the optimization period.

Images