Rail transit energy management method based on multiple parallel reinforcement learning

Abstract

The invention discloses a rail transit energy management method based on multiple parallel reinforcement learning, and the method comprises the steps: constructing an energy management model for the cooperation of an energy storage system, a new energy power generation system and a traction power supply system for a power fusion type rail transit cooperative energy supply system, and constructing an energy management model for the cooperation of an energy storage system, a new energy power generation system and a traction power supply system based on a Markov decision model under a multi-agent interaction system; discrete training and online operation are carried out, offline training is based on an original power data set, a multiple parallel reinforcement learning algorithm is adopted, and an obtained training reinforcement learning model is applied to online operation so as to optimize an energy management process; according to the method, energy flow of the collaborative energy supply system is dynamically adjusted in real time in a multi-agent collaboration mode, self-adaptive energy exchange of subsystems in a complex scene is achieved, and meanwhile the convergence speed of an algorithm is increased through a multiple parallel reinforcement learning algorithm in an offline training link; and the operation economy and stability of the traction power supply system can be effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of electrified railways, and in particular relates to a rail transit energy management method based on multiple parallel reinforcement learning. Background Art

[0002] Electrified railway transportation has the advantages of fast speed, large capacity and low comprehensive cost in land transportation, and plays a vital role in promoting social and economic development. As the only source of electricity for electric locomotives, the traction power supply system consumes a large amount of electricity for train traction every year. Therefore, the electricity expenditure in this field has become one of the main management expenses of railway operating units. It is imperative to explore ways to reduce traction energy consumption, so as to support the railway economy and promote low-carbon development in the transportation system. The power-integrated traction power supply collaborative energy supply system integrates the energy storage system, the new energy generation system and the power-integrated equipment. The system enables the electric energy to flow across the phases through the power-integrated device. At the same time, the assembly of the energy storage system can store the unused regenerative braking energy of the electric locomotive and the power generation energy of the new energy system. Under the framework of this collaborative energy supply system, the flow direction and utilization mode of energy have undergone fundamental changes compared with the structure of the traditional traction power supply system. Therefore, it is crucial to design a set of efficient, energy-saving and intelligent energy management strategies.

[0003] At present, there have been preliminary studies on energy management methods for power-integrated traction power supply cooperative energy supply systems, but they are still in their infancy. At present, the technical routes of energy management can be mainly divided into two categories: rule-based strategies and optimization algorithm-based strategies. Rule-based strategies usually establish charging and discharging rules based on the traction network voltage or locomotive power. In the field of urban rail transit, voltage control-based strategies are usually used to stabilize the traction network voltage and use it as a criterion for charging and discharging the energy storage system. For AC electrified railways, thresholds are usually set based on the power of electric locomotives to control system operation. However, the effectiveness of rule-based energy management systems depends on the designer's experience and knowledge, and fixed rules may be affected by the system operation mode and are difficult to migrate to other systems. In order to overcome these limitations, strategies based on optimization algorithms can achieve adaptive control functions through reinforcement learning, dynamic programming, model predictive control and other methods. However, such algorithms generally have the problems of long training time, the need to manually adjust a large number of hyperparameters, and difficulty in online operation.

[0004] According to the public patent information, the current research on the energy management method of the power-integrated traction power supply collaborative energy supply system is still in its infancy. The patent "An Electrified Railway Energy Storage Traction Power Supply System and Control Method Thereof" (Publication No.: CN110829435A) discloses an energy management scheme suitable for AC electrified railways, but this method is based on threshold control and is difficult to adapt to the real-time changing traffic organization, and the system has no adaptive effect due to the fixed threshold; the patent "An Electrified Railway Energy Storage System Energy Management Method Based on Fuzzy Control" (Publication No.: CN114498692B) proposes an energy management method based on fuzzy control. Although it has a certain adaptive control capability, there are a large number of hyperparameters that need to be set manually; the patent "An Electrified Railway Energy Storage System Energy Management Method Based on Reinforcement Learning" (Publication No.: CN116316755B) proposes an energy management strategy suitable for single-phase AC traction power supply systems based on reinforcement learning algorithms. Although it has the ability of adaptive charging and discharging, it can only realize the control of energy storage systems and lacks control over new energy power generation systems. Summary of the invention

[0005] In order to overcome the defects of the prior art, the present invention proposes a rail transit energy management method based on multiple parallel reinforcement learning for the power-integrated traction power supply collaborative energy supply system. By establishing a mathematical model of the energy management process of the power-integrated traction power supply collaborative energy supply system under multiple working conditions and a Markov decision model of the energy management process under a multi-agent interaction architecture, and using a multiple parallel reinforcement learning algorithm to solve the decision model, the energy flow of the collaborative energy supply system is dynamically adjusted in real time to achieve the operation economy and stability of the collaborative energy supply system.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A rail transit energy management method based on multiple parallel reinforcement learning, for the power-integrated rail transit collaborative energy supply system, has constructed an energy management model for the coordination of energy storage system, new energy power generation system and traction power supply system. The core of the energy management method is to respectively set an intelligent agent in the energy storage system and the new energy power generation system for interaction, construct a Markov decision model under a multi-agent interaction system, and perform discrete training and online operation. The offline training is based on the original power data set, using multiple parallel reinforcement learning algorithms, and speeding up the training process by constructing multiple parallel data chains, multiple parallel data chain independent updates and fusion update chains, and applying the acquired training reinforcement learning model to the online operation to optimize the energy management process.

[0008] The offline training specifically includes the following steps:

[0009] S1, loading typical daily power data into the Markov decision model;

[0010] S2, performing Monte Carlo sampling on the power data to obtain an original power data set, including an original power data set of an electric locomotive and an original power data set of a new energy power generation system;

[0011] S3, determining the number of parallel update chains, stretching or compressing the original power data set, and obtaining multiple parallel update training data sets of the new energy power generation system and the electric locomotive, including the parallel update data chain of the new energy power generation system and the parallel update data chain of the electric locomotive;

[0012] S4. Based on the multiple parallel update data chains, calculate the state variables of the new energy power generation system and the state variables of the energy storage system, sample the actions of the new energy system and the energy storage system, and after completing the action selection, calculate the system power parameters according to the power balance equation under the operating conditions of the coordinated energy supply system, and update the energy storage system parameters after completing the energy exchange;

[0013] S5: After completing one round of updates of the multiple parallel update chains, the rewards of the new energy power generation system and the energy storage system are calculated according to the actions selected in the current state, and the state-action value function matrix of the new energy power generation system is updated according to the state-action value function matrix update equation of the new energy power generation system, and the state-action value function matrix of the energy storage system is updated according to the state-action value function matrix update equation of the energy storage system;

[0014] S6: After the update is completed, the norm of the state-action value function matrix F of the new energy power generation system and the norm of the state-action value function matrix F of the energy storage system are calculated respectively until the matrix converges, otherwise return to S2;

[0015] The online operation makes decisions based on the converged new energy power generation system state-action value function matrix and the converged energy storage system state-action value function matrix obtained in the offline training.

[0016] Furthermore, the intelligent agent is the basic decision-making unit of the Markov decision model. The energy storage system and the renewable energy power generation system each have an intelligent agent for independent decision-making. The Markov decision model includes state variables, actions, rewards, returns and strategies. The state variables include the renewable energy power generation system state and the energy storage system state variables, wherein the renewable energy power generation system state includes the electric locomotive power, the renewable energy power generation system power and the external power system exchange power, and the energy storage system state variables include the electric locomotive power, the renewable energy power generation system maximum output power, the energy storage system exchange power, the external power system exchange power and the energy storage system charge state;

[0017] The actions are the outputs of the decision-making system, including the actions of the new energy power generation system and the energy storage system, which are respectively expressed as discrete power sets of gradient descent. The decision of the energy management system is to select the most suitable action element from the discrete power set to control the energy exchange of each system;

[0018] The rewards are used to measure the value of different actions in each state, including new energy power generation system rewards and energy storage system rewards. The new energy power generation system rewards include energy feedback efficiency rewards and energy loss ratio rewards. The energy storage system rewards include energy absorption rewards, energy feedback rewards and average traction power rewards.

[0019] The return is the expected value of the discounted reward and . By setting the discount factor, the return of the new energy power generation system and the return of the energy storage system are calculated.

[0020] The strategy is the sampling probability of action in the current state. The probability of the new energy power generation system taking action is calculated based on the state-action value function matrix of the new energy power generation system, and the probability of the energy storage system taking action is calculated based on the state-action value function matrix of the energy storage system.

[0021] Furthermore, the operating conditions of the collaborative energy supply system include three basic conditions: traction condition, idle condition, and regenerative braking condition;

[0022] Under the traction condition, the power of the electric locomotive is first compensated by the new energy power generation system. If there is a power shortage, the shortage is compensated by the energy storage system and the external power system to construct an energy balance equation under the traction condition.

[0023] Under the idle condition, if the state of charge value of the energy storage system is less than the maintenance calibration value, the energy storage system consumes the energy of the new energy power generation system for charging, and constructs an energy balance equation under the idle condition;

[0024] Under the regenerative braking condition, the energy storage system absorbs the regenerative braking energy of the electric locomotive and the energy of the new energy power generation system. If the power limit is exceeded, the excess energy will be returned to the power system to construct an energy balance equation under the regenerative braking condition.

[0025] Furthermore, for a power data set with a total time window length of T, let the starting number of the data set obtained by Monte Carlo sampling be i, the sampling probability be p(i), the length of the sampled data set be T0, P lc is the power of the electric locomotive, then the sampled original power data set P of the electric locomotive lc Described as formula (1):

[0026]

[0027] According to the corresponding relationship of data points, the original power data set P of the new energy power generation system pv Described as formula (2):

[0028] P pv = {P pv (i),P pv (i),...,P pv (i+T0-1),P pv (i+T0)} (2)

[0029] In formula (2), P pv Output power for the renewable energy power generation system.

[0030] Furthermore, the original power data set is stretched or compressed, and the multiple parallel update data chains of the electric locomotive are updated with the kth parallel update data chain. and data link elements To describe, as formula (3):

[0031]

[0032] Then the multiple parallel update data chains of the new energy power generation system are updated with the kth parallel update data chain and data link elements To describe, as formula (4):

[0033]

[0034] In equations (3) and (4), ΔG is the linear transformation gradient.

[0035] Furthermore, based on the multiple parallel update data chains of the electric locomotive and the multiple parallel update data chains of the new energy power generation system, the state variable of the new energy power generation system is the state variable of the new energy power generation system under the kth parallel update data chain at time t. To describe, the set of state variables is S pv,k , as shown in formula (5):

[0036]

[0037] The energy storage system state variable is updated with the state variable of the energy storage system under the data link in parallel at time t. To describe, the set of state variables is S ESS,k , as shown in formula (6):

[0038]

[0039] In formulas (5) and (6), The power exchanged between the external power system and the traction power supply system at time t-Δt under the kth update data link, SOCk (t) is the charge state of the energy storage system at time t under the kth updated data link.

[0040] Furthermore, in the state variables of the new energy power generation system Next new energy power generation system action The sampling probability Calculate as formula (7):

[0041]

[0042] In the energy storage system state variables Energy storage system action The sampling probability Calculate as formula (8):

[0043]

[0044] In formulas (7) and (8), is the state-action value function matrix of the new energy power generation system, is the state-action value function matrix of the energy storage system.

[0045] Furthermore, the state-action value function matrix update equation of the new energy power generation system is described by the state-action value function matrix update equation of the new energy power generation system of the kth parallel update chain, as shown in formula (9):

[0046]

[0047] The energy storage system state-action value function matrix update equation is described by the energy storage system state-action value function matrix update equation of the kth parallel update chain, A pv is the action space of the new energy power generation system. As shown in formula (10):

[0048]

[0049] In formulas (9) and (10), α is the learning factor, is the corresponding reward of the new energy power generation system calculated according to the action selected in the current state, is the corresponding reward of the energy storage system calculated according to the action selected in the current state, A ess It is the action space of the energy storage system.

[0050] Furthermore, the norm of the state-action value function matrix F of the new energy power generation system is calculated as shown in formula (11):

[0051]

[0052] In formula (11), ||Q||F represents the F-norm value of the matrix Q, and x, y, and z represent the subscripts of the element q in the Q matrix.

[0053] The agent in the update chain based on the original power data set adopts a greedy strategy, that is, directly selecting the action corresponding to the maximum state-action value function matrix value, and the multiple parallel update data chains randomly sample the actions according to the probability density distribution function. At this time, the actions of the agent in the new energy power generation system Actions of agents in energy storage systems They are expressed as formula (12) and (13) respectively:

[0054]

[0055] In formula (12), Represents the state-action value function matrix of the renewable energy power generation system, represents the state-action value function matrix of the energy storage system, represents the probability density distribution function of the renewable energy power generation system, as shown in formula (14). In formula (13), Represents the probability density distribution function of the energy storage system, as shown in formula (15):

[0056]

[0057] Compared with the prior art, the present invention has the following technical effects:

[0058] (1) It can realize the coordinated power supply of traction substation-energy storage system-new energy power generation system, absorb the regenerative braking energy of electric locomotives and the energy of new energy power generation, and reduce energy loss and waste;

[0059] (2) It can realize the adaptive charging or discharging of the energy storage system and the new energy power generation system, and adjust the strategy according to the real-time power demand of the electric locomotive to reduce the power peak as much as possible and reduce the power impact of the traction network;

[0060] (3) It can improve the speed of reinforcement learning training. By constructing multiple parallel update chains, it can achieve multiple updates in one training, effectively improving the adaptability and energy utilization efficiency of the energy management system. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a structural diagram of the rail transit collaborative energy supply system of the present invention;

[0062] Figure 2 This is the flowchart of offline multiple parallel reinforcement learning;

[0063] Figure 3 It is the original electric locomotive power and the maximum demand power diagram;

[0064] Figure 4 It is the maximum output power curve of photovoltaic power generation system;

[0065] Figure 5 It is a graph showing the change of the norm of the state-action value function matrix F with the number of iterations;

[0066] Figure 6 Exchange power curve diagram for energy storage system;

[0067] Figure 7 Exchange power curve diagram for photovoltaic power generation system. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the invention, all other embodiments obtained by ordinary technicians in this field without creative work, any modifications, equivalent substitutions, improvements, etc., should be included in the protection scope of the present invention.

[0069] An energy management method for a rail transit collaborative energy supply system based on a dynamic programming algorithm, the energy management method is used in a power-smoothing rail transit collaborative energy supply system, such as Figure 1 As shown, the structure of the electrified railway energy storage system includes a traction power supply system, an external power system, a power integration device, a unidirectional step-down transformer, a new energy power generation system, an energy storage system and an energy management system. The power supply system of the system adopts a single-phase AC power frequency system. The system structure is based on the traditional traction power supply system structure. A power integration device is installed inside the traction substation to connect the two ends of the electrical phase at the outlet of the substation. A low-voltage DC bus is constructed inside the power integration device. The new energy power generation system and the energy storage system are connected to the DC bus through an electric energy conversion device. The energy management system will collect various parameters inside the system in real time and control each DC / DC converter to complete energy control.

[0070] The rail transit energy management method based on multiple parallel reinforcement learning, for the power-integrated rail transit collaborative energy supply system, constructs an energy management model for the coordination of energy storage system, new energy power generation system and traction power supply system. The core of the energy management method is to respectively set an intelligent agent in the energy storage system and the new energy power generation system for interaction, construct a Markov decision model under the multi-agent interaction system, and perform discrete training and online operation. The offline training is based on the original power data set, using multiple parallel reinforcement learning algorithms, and speeding up the training process by constructing multiple parallel data chains, multiple parallel data chain independent updates and fusion update chains, and applying the acquired training reinforcement learning model to the online operation to optimize the energy management process.

[0071] like Figure 2 As shown, the offline training specifically includes the following steps:

[0072] S1. Loading typical daily power data into the Markov decision model; the Markov decision model includes state variables, actions, rewards, returns and strategies, and the state variables, actions, rewards, returns and strategies are described by system parameters at time t.

[0073] The state variables include new energy power generation system state variables and energy storage system state variables The state of new energy power generation system Including the electric locomotive power and P at time t lc (t), the power of the new energy power generation system P at time t-Δt pv (t-Δt) and the external power system exchange power P upg (t-Δt), as shown in formula (S1):

[0074]

[0075] Energy storage system state variables Including the electric locomotive power and P at time t lc (t), the maximum output power P of the new energy power generation system at time t pv (t), the energy storage system exchange power P at time t-Δt pv (t-Δt), external power system exchange power P upg (t-Δt) and the state of charge SOC(t) of the energy storage system at time t, as shown in formula (S2):

[0076]

[0077] The actions are the outputs of the decision-making system, including the actions of the new energy generation system and energy storage system action They are respectively expressed as discrete power sets of gradient descent. The decision of the energy management system is to select the most suitable action element from the discrete power set to control the energy exchange of each system. Suppose the typical daily maximum power output of the new energy power generation system is The maximum charging power of the energy storage system is The maximum discharge power of the energy storage system is The discrete gradient of the action of the new energy power generation system is g pv , the discrete gradient of the energy storage system action is g ess , then the action of the new energy power generation system It can be expressed as a discrete power set as in formula (S3):

[0078]

[0079] Energy storage system action It can be expressed as a discrete power set as in formula (S4):

[0080]

[0081] The reward is used to measure the value of different actions in each state, including the new energy power generation system reward and the energy storage system reward. The new energy power generation system reward includes the energy feedback efficiency reward and the energy loss ratio reward. Suppose the total number of time windows is M, the number of data points in a single time window is N, m and n are the time window number and data point number respectively, and the energy feedback efficiency is The energy loss ratio is The reward coefficient is r pv , a1 and a2 are parameter weights, and the energy power transferred from the new energy power generation system to the traction power supply system is Energy feedback efficiency is η de , the power of the electric locomotive under traction conditions is The energy power transferred from the new energy power generation system to the energy storage system is The energy storage system's energy replenishment power under idle conditions is P ess,ics (t), then the reward of the new energy power generation system at time t is Described as formula (S5):

[0082]

[0083] In formula (S5), represents the energy feedback efficiency, calculated as formula (S6), Represents the energy loss ratio, calculated as formula (S7):

[0084]

[0085]

[0086] The energy storage system reward includes energy absorption reward, energy feedback reward and average traction power reward; energy storage system reward It can be described as formula (S8):

[0087]

[0088] In formula (S8), r ess is the reward parameter, a3, a4 and a5 are weight parameters, is the energy absorption efficiency of the energy storage system, calculated as formula (S9), is the energy feedback efficiency of the energy storage system, calculated as formula (S10), is the average traction power reduction percentage, calculated as formula (S11):

[0089]

[0090] The return is the expected value of the discounted reward and the return G of the new energy power generation system is calculated by setting the discount factor γ. pv and the return of the energy storage system G ess , as (S12) and (S13) respectively:

[0091]

[0092] The strategy is the sampling probability of the action in the current state, according to the state-action value function matrix of the new energy power generation system Calculate the actions taken by the renewable energy power generation system Probability As shown in formula (S14):

[0093]

[0094] According to the state-action value function matrix of the energy storage system Calculating energy storage system to take action Probability As shown in formula (S15):

[0095]

[0096] S2, performing Monte Carlo sampling on the power data to obtain an original power data set, including an original power data set of an electric locomotive and an original power data set of a new energy power generation system;

[0097] S3, determining the number of parallel update chains, stretching or compressing the original power data set, and obtaining multiple parallel update training data sets of the new energy power generation system and the electric locomotive, including the parallel update data chain of the new energy power generation system and the parallel update data chain of the electric locomotive;

[0098] For a power data set with a total time window length of T, let the starting number of the data set obtained by Monte Carlo sampling be i, the sampling probability be p(i), the length of the sampled data set be T0, P lc is the power of the electric locomotive, then the sampled original power data set P of the electric locomotive lc Described as formula (S15):

[0099]

[0100] According to the corresponding relationship of data points, the original power data set P of the new energy power generation system pv Described as formula (S16):

[0101] P pv = {P pv (i),P pv (i),...,P pv (i+T0-1),P pv (i+T0)} (S16)

[0102] In formula (S16), P pv Output power for the renewable energy power generation system.

[0103] If the original power data set is stretched or compressed, the multiple parallel update data chains of the electric locomotive are updated with the kth parallel update data chain and data link elements To describe, as shown in formula (S17):

[0104]

[0105] Then the multiple parallel update data chains of the new energy power generation system are updated with the kth parallel update data chain and data link elements To describe, as formula (S18):

[0106]

[0107] In equations (S17) and (S18), ΔG is the linear transformation gradient.

[0108] S4. Based on the multiple parallel update data chains, calculate the state variables of the new energy power generation system and the state variables of the energy storage system, sample the actions of the new energy system, and sample the actions of the energy storage system;

[0109] Based on the multiple parallel update data chain of electric locomotive and the multiple parallel update data chain of new energy power generation system, the state variable of the new energy power generation system is the state variable of the new energy power generation system under the kth parallel update data chain at time t. To describe, the set of state variables is S pv,k , as shown in formula (S19):

[0110]

[0111] The energy storage system state variable is updated with the state variable of the energy storage system under the data link in parallel at time t. To describe, the set of state variables is S ESS,k , as shown in formula (S20):

[0112]

[0113] In formulas (S19) and (S20), The power exchanged between the external power system and the traction power supply system at time t-Δt under the kth update data link, SOC k (t) is the charge state of the energy storage system at time t under the kth updated data link.

[0114] In the renewable energy power generation system state variables Next new energy power generation system action The sampling probability Calculate as formula (S21):

[0115]

[0116] In the energy storage system state variables Energy storage system action The sampling probability Calculate as formula (S22):

[0117]

[0118] In formulas (S21) and (S22), is the state-action value function matrix of the new energy power generation system, is the state-action value function matrix of the energy storage system.

[0119] After completing the action selection, the system power parameters are calculated according to the power balance equation under the three basic collaborative energy supply system operating conditions: traction condition, idle condition, and regenerative braking condition. After completing the energy exchange, the energy storage system parameters are updated;

[0120] Under the traction condition, the power of the electric locomotive is first compensated by the new energy power generation system. If there is a power shortage, the shortage is compensated by the energy storage system and the external power system to construct the energy balance equation under the traction condition, such as formula (S23):

[0121]

[0122] In formula (S23), P lc (t) is the power of the electric locomotive measured by the traction substation at time t, is the discharge power of the energy storage system at time t, P is the energy power directly transferred from the new energy power generation system to the traction power supply system at time t, upg (t) is the external power system compensation power at time t, and T is the time window length.

[0123] Under the idle condition, if the energy storage system state of charge value is less than the maintenance calibration value SOC ma, the energy storage system consumes the energy of the new energy power generation system for charging, and constructs the energy balance equation under idle conditions, such as formula (S24):

[0124]

[0125] In formula (S24), Transfer energy power from new energy power generation system to energy storage system, It is the energy replenishment power of the energy storage system under idle conditions.

[0126] Under the regenerative braking condition, the energy storage system absorbs the regenerative braking energy of the electric locomotive and the energy of the new energy power generation system. If the power limit is exceeded, the excess energy will be returned to the power system to construct the energy balance equation under the regenerative braking condition, such as formula (S25):

[0127]

[0128] In formula (S25), is the charging power of the energy storage system at time t.

[0129] S5: After completing one round of updates of the multiple parallel update chains, the new energy power generation system reward is calculated according to the action selected in the current state. And energy storage system rewards

[0130] According to the new energy power generation system state-action value function matrix update equation shown in formula (S26), the new energy power generation system state-action value function matrix is ​​updated

[0131]

[0132] According to the energy storage system state-action value function matrix update equation as shown in formula (S27), the energy storage system state-action value function matrix is ​​updated

[0133]

[0134] In formulas (S26) and (S27), α is the learning factor, is the corresponding reward of the new energy power generation system calculated according to the action selected in the current state, is the corresponding reward of the energy storage system calculated according to the action selected in the current state, A ess It is the action space of the energy storage system.

[0135] S6: After the update is completed, the norm of the state-action value function matrix F of the new energy power generation system as shown in formula (S28) and the norm of the state-action value function matrix F of the energy storage system as shown in formula (S29) are calculated respectively until the matrix converges, otherwise return to S2;

[0136]

[0137] In formulas (S28) and (S29), and Respectively represent the state-action value function matrix Q of the new energy power generation system pv The total number of state space and action space elements, Denotes the matrix Q pv The elements in x pv With y pv Represents elements The ordinal number of the elements in the state space and action space; and Respectively represent the energy storage system state-action value function matrix Q ess The total number of state space and action space elements, Denotes the matrix Q ess The elements in x ess With y ess Represents elements The ordinal number of the element in the state space and action space.

[0138] The online operation makes decisions based on the converged new energy power generation system state-action value function matrix and the converged energy storage system state-action value function matrix obtained in the offline training.

[0139] against Figure 1 The power-integrated traction power supply coordinated energy supply system shown in the figure is equipped with a battery energy storage system and a photovoltaic power generation system. The typical daily traction load parameters are shown in Table 1. The typical daily traction load-original electric locomotive power and maximum demand power curve are shown in Figure 3 shown.

[0140] Table 1 Typical daily traction load parameters

[0141]

[0142] The parameters of the energy storage system and photovoltaic power generation system are shown in Table 2. The maximum output power curve of the photovoltaic power generation system on a typical day is shown in Figure 4 shown.

[0143] Table 2 Energy storage system and photovoltaic power generation system parameters

[0144]

[0145] As shown in Table 1 and Table 2, Figure 3 and Figure 4 The data in is used for testing. The relationship between the F norm of the state-action value function matrix and the number of iterations is as follows Figure 5 As shown, it can be seen that with the increase in the number of parallel update chains, the convergence speed is significantly faster than the original training curve, and the convergence point is higher than the original curve, indicating that the method proposed in the present invention can effectively accelerate the convergence speed of reinforcement learning.

[0146] Table 3 Parameter comparison table

[0147]

[0148] When the parallel training method with 5 parallel update chains is adopted, the energy exchange situation is simulated. The exchange power between the energy storage system and the photovoltaic power generation system is as follows: Figure 6 and Figure 7 As shown. Through analysis, it can be seen that for the energy storage system, the method proposed in the present invention is used for control, and the energy storage system can realize adaptive energy exchange according to the power size of the electric locomotive, and absorb the energy of the new energy power generation system for energy replenishment under idle conditions; for the photovoltaic power generation system, the method proposed in the present invention is used for control, and the energy of the photovoltaic power generation system can be directly used for locomotive traction, and can also be used for energy storage system energy replenishment, forming a good match with the energy storage system. Under this strategy, the specific comparison parameters are shown in Table 3. It can be seen that the peak power after power compensation is significantly reduced, and the role of peak shaving and valley filling is effectively played.

Claims

1. A rail transit energy management method based on multiple parallel reinforcement learning, characterized in that: For the power-integrated rail transit collaborative energy supply system, an energy management model for the coordination of energy storage system, new energy power generation system and traction power supply system is established. The core of the energy management method is to construct a Markov decision model under a multi-agent interaction system, and conduct discrete training and online operation. The offline training is based on the original power data set, and multiple parallel reinforcement learning algorithms are used to speed up the training process. The acquired training reinforcement learning model is applied to the online operation to optimize the energy management process. The offline training specifically includes the following steps: S1, loading typical daily power data into the Markov decision model; S2, performing Monte Carlo sampling on the power data to obtain an original power data set, including an original power data set of an electric locomotive and an original power data set of a new energy power generation system; S3, determining the number of parallel update chains, stretching or compressing the original power data set, and obtaining multiple parallel update training data sets of the new energy power generation system and the electric locomotive, including the parallel update data chain of the new energy power generation system and the parallel update data chain of the electric locomotive; S4. Based on the multiple parallel update data chains, calculate the state variables of the new energy power generation system and the state variables of the energy storage system, sample the actions of the new energy system and the energy storage system, and after completing the action selection, calculate the system power parameters according to the power balance equation under the operating conditions of the coordinated energy supply system, and update the energy storage system parameters after completing the energy exchange; S5: After completing one round of updates of the multiple parallel update chains, the rewards of the new energy power generation system and the energy storage system are calculated according to the actions selected in the current state, and the state-action value function matrix of the new energy power generation system is updated according to the state-action value function matrix update equation of the new energy power generation system, and the state-action value function matrix of the energy storage system is updated according to the state-action value function matrix update equation of the energy storage system; S6: After the update is completed, the norm of the state-action value function matrix F of the new energy power generation system and the norm of the state-action value function matrix F of the energy storage system are calculated respectively until the matrix converges, otherwise return to S2; The online operation makes decisions based on the converged new energy power generation system state-action value function matrix and the converged energy storage system state-action value function matrix obtained in the offline training.

2. The rail transit energy management method based on multiple parallel reinforcement learning according to claim 1 is characterized in that: An intelligent agent is respectively set in the energy storage system and the new energy power generation system for interaction, and a Markov decision model under the multi-agent interaction system is constructed. The Markov decision model includes state variables, actions, rewards, returns and strategies.

3. The rail transit energy management method based on multiple parallel reinforcement learning according to claim 2 is characterized in that: The state variables include the state of the new energy power generation system and the state variables of the energy storage system, wherein the state of the new energy power generation system includes the power of the electric locomotive, the power of the new energy power generation system and the exchange power of the external power system, and the state variables of the energy storage system include the power of the electric locomotive, the maximum output power of the new energy power generation system, the exchange power of the energy storage system, the exchange power of the external power system and the charge state of the energy storage system; The actions include actions of the new energy power generation system and actions of the energy storage system, which are respectively expressed as discrete power sets; The rewards include new energy power generation system rewards and energy storage system rewards. The new energy power generation system rewards include energy feedback efficiency rewards and energy loss ratio rewards. The energy storage system rewards include energy absorption rewards, energy feedback rewards and average traction power rewards. The return is the expected value of the discounted reward and . By setting the discount factor, the return of the new energy power generation system and the return of the energy storage system are calculated. The strategy is the sampling probability of action in the current state. The probability of the new energy power generation system taking action is calculated based on the state-action value function matrix of the new energy power generation system, and the probability of the energy storage system taking action is calculated based on the state-action value function matrix of the energy storage system.

4. The rail transit energy management method based on multiple parallel reinforcement learning according to claim 3 is characterized in that: The operating conditions of the collaborative energy supply system include three basic conditions: traction condition, idle condition, and regenerative braking condition; Under the traction condition, the power of the electric locomotive is first compensated by the new energy power generation system. If there is a power shortage, the shortage is compensated by the energy storage system and the external power system to construct an energy balance equation under the traction condition. Under the idle condition, if the state of charge value of the energy storage system is less than the maintenance calibration value, the energy storage system consumes the energy of the new energy power generation system for charging, and constructs an energy balance equation under the idle condition; Under the regenerative braking condition, the energy storage system absorbs the regenerative braking energy of the electric locomotive and the energy of the new energy power generation system. If the power limit is exceeded, the excess energy will be returned to the power system to construct an energy balance equation under the regenerative braking condition.

5. The rail transit energy management method based on multiple parallel reinforcement learning according to claim 4 is characterized in that: For a power data set with a total time window length of T, let the starting number of the data set obtained by Monte Carlo sampling be i, the sampling probability be p(i), the length of the sampled data set be T0, P lc is the power of the electric locomotive, then the sampled original power data set P of the electric locomotive lc Described as formula (1): The original power data set P of the new energy power generation system pv Described as formula (2): P pv ={P pv (i),P pv (i),...,P pv (i+T0-1),P pv (i+T0)} (2) In formula (2), P pv Output power for the renewable energy power generation system.

6. The rail transit energy management method based on multiple parallel reinforcement learning according to claim 5 is characterized in that: The electric locomotive parallel update data link is a kth parallel update data link and data link elements To describe, as formula (3): The new energy power generation system updates the data chain in parallel with the kth parallel update data chain and data link elements To describe, as formula (4): In equations (3) and (4), ΔG is the linear transformation gradient.

7. The rail transit energy management method based on multiple parallel reinforcement learning according to claim 6 is characterized in that: The state variable of the new energy power generation system is updated by the state variable of the new energy power generation system under the kth parallel update data link at time t. To describe, the set of state variables is S pv,k , as shown in formula (5): The energy storage system state variable is updated with the state variable of the energy storage system under the data link in parallel at time t. To describe, the set of state variables is S ESS,k , as shown in formula (6): In formulas (5) and (6), The power exchanged between the external power system and the traction power supply system at time t-Δt under the kth update data link, SOC k (t) is the charge state of the energy storage system at time t under the kth updated data link.

8. The rail transit energy management method based on multiple parallel reinforcement learning according to claim 7 is characterized in that: In the renewable energy power generation system state variables Next new energy power generation system action The sampling probability Calculate as formula (7): In the energy storage system state variables Energy storage system action The sampling probability Calculate as formula (8): In formulas (7) and (8), is the state-action value function matrix of the new energy power generation system, is the state-action value function matrix of the energy storage system.

9. The rail transit energy management method based on multiple parallel reinforcement learning according to claim 8 is characterized in that: The state-action value function matrix update equation of the new energy power generation system is described by the state-action value function matrix update equation of the new energy power generation system of the kth parallel update chain, as shown in formula (9): The energy storage system state-action value function matrix update equation is described by the energy storage system state-action value function matrix update equation of the kth parallel update chain, A pv is the action space of the new energy power generation system; as shown in formula (10): In formulas (9) and (10), α is the learning factor, is the corresponding reward of the new energy power generation system calculated according to the action selected in the current state, is the corresponding reward of the energy storage system calculated according to the action selected in the current state, A ess It is the action space of the energy storage system.

10. The rail transit energy management method based on multiple parallel reinforcement learning according to claim 9, characterized in that: The norm of the state-action value function matrix F of the new energy power generation system is calculated as follows: In formula (11), ||Q|| F represents the F-norm value of the matrix Q, and x, y, and z represent the subscripts of the element q in the Q matrix.

Images