Adaptive droop control method for urban rail hybrid energy storage system based on deep reinforcement learning

Abstract

This invention discloses an adaptive droop control method for a hybrid energy storage system in urban rail transit based on deep reinforcement learning. Addressing the voltage fluctuation problem in the DC traction network caused by frequent traction and regenerative braking of urban rail transit trains, a hybrid energy storage system composed of supercapacitors and batteries is employed. An energy droop control strategy is used to achieve energy interaction between the energy storage units and the DC traction power supply network. A multi-agent deep deterministic policy gradient algorithm collaborative control architecture is established, and a multi-objective collaborative reward mechanism that considers energy saving efficiency, voltage regulation efficiency, and safe operation of energy storage components is constructed. Through reinforcement learning policy training and sequential decision execution, the charge and discharge droop coefficients of the supercapacitors and batteries are dynamically adjusted according to the system operating status. This method can improve the energy-saving and voltage regulation characteristics of the traction network and effectively prevent overcharging and over-discharging of energy storage components.

Description

Technical Field

[0001] This invention relates to the fields of urban rail transit energy storage and energy-saving technology, specifically to an adaptive droop control method for urban rail hybrid energy storage systems based on deep reinforcement learning. Background Technology

[0002] Urban rail transit trains operate on short routes with frequent starts and stops, resulting in rapid changes in traction power. If braking energy cannot be recovered, it can easily lead to waste. Single energy storage devices struggle to balance rapid response and large-capacity energy storage, making it difficult to achieve a balance between high-power traction and energy recovery. Hybrid energy storage systems, combining the characteristics of supercapacitors and batteries, can simultaneously meet power density and energy capacity requirements. Supercapacitors provide rapid response to traction power fluctuations, while batteries provide stable energy; both operate through a bidirectional DC / DC converter. A well-designed control strategy can not only reduce traction power fluctuations and improve braking energy recovery efficiency but also extend the lifespan of energy storage components. However, the complex operating conditions of trains necessitate a coordinated approach that balances voltage stabilization, energy saving, and the safe operation of energy storage components—a problem that current technologies urgently need to address. Summary of the Invention

[0003] Technical challenges: To achieve efficient recovery and recycling of regenerative braking energy between the traction power supply system and the hybrid energy storage system, how to achieve coordinated control and optimized energy allocation between supercapacitors and batteries, while ensuring stable operation of the traction power supply network and improving energy utilization efficiency, is a technical problem that needs to be addressed.

[0004] Technical Solution: To address the aforementioned problems, this invention proposes an adaptive droop control method for urban rail hybrid energy storage systems based on deep reinforcement learning. The hybrid energy storage system, composed of supercapacitors and batteries, is connected to the DC traction power supply network via a bidirectional DC / DC converter. The system state space is constructed by real-time monitoring of system state variables, and a multi-objective collaborative reward mechanism is designed to balance energy efficiency, voltage regulation efficiency, and safe operation of energy storage components. Based on the current droop control strategy, a multi-agent deep deterministic policy gradient (MADDPG) algorithm is used to adaptively optimize the droop coefficient online, achieving intelligent energy management for both the supercapacitor and battery. By combining reinforcement learning strategy training with a sequential decision execution mechanism, the control strategy is continuously optimized under different system operating states, thereby achieving the comprehensive goals of traction network stability, energy saving, and safe operation of energy storage components.

[0005] This invention proposes an adaptive droop control method for urban rail hybrid energy storage systems based on deep reinforcement learning. Its key features include: employing a cooperative control architecture based on the MADDPG algorithm; constructing a multi-objective cooperative reward mechanism that balances energy efficiency, voltage regulation efficiency, and the safe operation of energy storage components; and designing a reinforcement learning strategy training and sequential decision-making execution mechanism. Through a continuous "perception-decision-feedback" interaction process between the agent and the environment, the adaptive optimization of the droop coefficient of the urban rail hybrid energy storage system is achieved. The invention is implemented as follows:

[0006] The traction power supply system adopts a DC voltage level of 1500V. The supercapacitor energy storage unit and the battery energy storage unit together constitute a hybrid energy storage system, and are connected to the DC traction power supply network through bidirectional DC / DC converters to realize energy interaction between the energy storage unit and the traction power supply system.

[0007] The design of the current droop control for the urban rail hybrid energy storage system is as follows: Taking a supercapacitor as an example, the DC traction grid voltage... Respectively compared with the supercapacitor charging threshold and supercapacitor discharge threshold The results are compared and adjusted by the charge / discharge droop coefficients to generate a supercapacitor reference current. Then, it is combined with the feedback supercapacitor current. The comparison is performed, and the control signal is generated after adjustment by the PI controller. The drive duty cycle of the bidirectional DC / DC converter switching device is then output through the PWM module. This enables control of the supercapacitor's charging and discharging current. The initial droop coefficient is determined based on the design slope of the I-V droop curve and calculated jointly from the allowable fluctuation range of the traction grid voltage and the rated current of the energy storage system. During operation, the initial droop coefficient is dynamically corrected by the charging and discharging droop coefficient adjustment amount output by the intelligent agent. The droop coefficient is updated as follows:

[0008] (1)

[0009] , , , These are the initial charge / discharge droop coefficients of supercapacitors and batteries, respectively. , , , These are the adjustment amounts for the supercapacitor charge / discharge droop coefficient and the battery charge / discharge droop coefficient, respectively. , , , These are the adjusted charge / discharge droop coefficients for supercapacitors and batteries, respectively.

[0010] A collaborative control architecture based on the MADDPG algorithm is constructed, with Agent A (supercapacitor energy storage unit) and Agent B (battery energy storage unit) built for each. Agent A optimizes the supercapacitor's charge-discharge droop coefficient online and outputs the adjustment amount. Agent B optimizes the battery's charge-discharge droop coefficient online and outputs the adjustment amount. The two agents operate in a shared environment with the urban rail traction power supply system and the hybrid energy storage system, sharing system state information to achieve collaborative control between the supercapacitor and the battery. A centralized training and distributed execution mechanism is adopted. During the training phase, the policy network is optimized using the system's global state information. During the execution phase, each agent outputs corresponding actions based on its current operating state, achieving adaptive droop control of the urban rail hybrid energy storage system.

[0011] To realize the application of the MADDPG algorithm in urban rail hybrid energy storage systems, environmental states, agent actions, and multi-objective collaborative reward mechanisms were designed.

[0012] 1) Traction power supply environment and condition characterization design:

[0013] During the MADDPG algorithm training process, the environment provides updated state and corresponding reward signals after each action performed by the agent. Therefore, the adaptive droop control process of the urban rail hybrid energy storage system is described as a Markov decision process. The DC traction power supply system and the hybrid energy storage system constitute the environment of the agent. The agent outputs corresponding control actions by sensing environmental state information and continuously optimizes the control strategy based on the reward signals fed back from the environment to maximize the cumulative reward. Considering the comprehensive requirements for the energy-saving and voltage-stabilizing performance of the traction network and the safe operation of energy storage components under frequent train traction and braking conditions, the DC traction network voltage is selected... Battery voltage Supercapacitor voltage Battery state of charge State of charge of supercapacitors Reference power of hybrid energy storage system Train speed and train acceleration As an environmental state space The state space is represented as:

[0014] (2)

[0015] 2) Motion space design and control output:

[0016] The MADDPG algorithm selects control actions from the action space and dynamically adjusts the charge / discharge droop coefficient. Based on the control requirements of the supercapacitor energy storage unit and the battery energy storage unit, action spaces corresponding to Agent A and Agent B are constructed respectively. and action space ,Strategy Representing the state space Mapping relationship to action space: This is used to describe the control behavior of the agent under different system states. The action space is constructed as follows:

[0017] (3)

[0018] 3) Design of a multi-objective collaborative reward mechanism:

[0019] Reward signals are the feedback from the environment to the agent's actions, and the design of the reward function directly affects the agent's learning effectiveness and control performance. Energy efficiency will be improved. Voltage regulation efficiency Supercapacitor safety operation indicators Battery safety operation indicators As an evaluation metric for the reward function; among which, energy efficiency With voltage regulation efficiency As a globally shared reward item, it applies to both intelligent agents simultaneously; supercapacitor safety operation indicators and battery safety operation indicators As a local constraint, it guides each agent to independently optimize control based on the characteristics of different energy storage units, and constructs reward functions for Agent A and Agent B respectively. and The agent continuously interacts with the environment and optimizes its control strategy based on the reward signals it receives, thereby maximizing the system's long-term cumulative reward. The reward function is defined as follows:

[0020] (4)

[0021] in , , , , , These are the weighting coefficients for each indicator. , , and As shown in equations (5) to (8) respectively.

[0022] (5)

[0023] Energy efficiency Defined as the percentage change in the total output energy of a substation before and after the installation of a hybrid energy storage system, relative to the total output energy of the substation without an energy storage system. These represent the DC traction network currents with and without hybrid energy storage installed, respectively. These represent the DC traction grid voltage under conditions of installed / non-installed hybrid energy storage;

[0024] (6)

[0025] Voltage regulation efficiency The evaluation is based on the integral of the portion of the DC traction network voltage that exceeds / falls below the limit. These are the set upper and lower safety limits for the DC traction network voltage, respectively. These refer to the time during which the DC traction network voltage exceeds the upper / lower safety limit during train operation.

[0026] (7)

[0027] (8)

[0028] in, and These are the states of charge of the supercapacitor. Lower and upper limits for safe operation and These are the battery state of charge. Lower and upper limits for safe operation. When the state of charge of a supercapacitor or battery exceeds the preset safe operating range, the corresponding indicator value is calculated based on the degree of deviation from the safe boundary. When it is within the safe operating range, the indicator value is 0. and It participates in agent training as part of the reward function, and is used to constrain the operating state of supercapacitors and batteries to prevent overcharging and over-discharging of energy storage components.

[0029] Design a reinforcement learning strategy training and sequential decision-making execution mechanism.

[0030] (1) In the reinforcement learning strategy training module, a simulation platform for urban rail traction power supply and hybrid energy storage system is established and used as the environment for Agent A and Agent B. The two agents continuously interact with the environment and update their strategies based on the reward function of the environment feedback. In each training round, the train operation status is randomly initialized within the preset safe operation range. The agents learn by continuously interacting with the environment until the reward converges, and the training obtains a control strategy that can achieve energy saving, voltage stabilization and safe operation of energy storage components in the traction network under random operation environment.

[0031] (2) In the sequential decision execution module, Agent A and Agent B execute decisions online based on the real-time system status, dynamically outputting the adjustment amounts of the supercapacitor charge / discharge droop coefficient and the battery charge / discharge droop coefficient. By combining reinforcement learning strategy training with sequential decision execution, the control strategy is continuously updated.

[0032] The system operates as follows:

[0033] Step 1: Agent A and Agent B interact with the system environment for training. By collecting the system's status, actions and reward information in real time, they repeatedly interact and train N times until the accumulated reward stabilizes and converges. Then, they are used to realize online sequential decision-making in the urban rail hybrid energy storage system.

[0034] Step 2: The system outputs the adjustment amount of the supercapacitor charge and discharge droop coefficient and the battery charge and discharge droop coefficient respectively according to the current operating status. The adjustment amount is then fused with the corresponding initial droop coefficient through the droop coefficient update mechanism to obtain the charge and discharge droop coefficient applicable to the current operating status.

[0035] Step 3: Based on the real-time feedback voltage of the traction network generated by train operation The difference between the current and the charging / discharging threshold is processed by the adjusted droop coefficient to obtain the reference current of the energy storage unit. This reference current is then compared with the actual current of the energy storage unit. After adjustment by the PI controller, a PWM control signal is output to drive the bidirectional DC / DC converter and realize the charging and discharging control of the energy storage unit.

[0036] Step 4: When the train operation task ends and the train stops running, the system stops the intelligent agent decision output and ends the adaptive droop control process of the urban rail hybrid energy storage system.

[0037] Beneficial Effects: This invention employs the MADDPG algorithm for online adaptive optimization of the droop coefficient in urban rail hybrid energy storage systems. By constructing a multi-objective collaborative reward mechanism targeting voltage regulation efficiency, energy saving efficiency, and safe operation of energy storage components, the intelligent agent is guided to autonomously learn and obtain the optimized droop coefficient. The proposed method can adjust control parameters in real time according to the system's operating status, enabling the control strategy to adapt to the energy interaction needs under different operating conditions. This effectively improves the energy-saving and voltage regulation characteristics of the traction network, prevents overcharging and over-discharging of energy storage components, and comprehensively enhances the energy saving and economic benefits of urban rail transit systems. Attached Figure Description

[0038] Figure 1 The diagram shows the adaptive droop control structure of a hybrid energy storage system for urban rail transit based on the MADDPG algorithm.

[0039] Figure 2A structural diagram illustrating the training of learning strategies and the sequential decision-making execution mechanism; Detailed Implementation

[0040] To further illustrate the technical solution of the present invention, the following are specific embodiments.

[0041] The traction power supply system adopts a DC voltage level of 1500V. The supercapacitor energy storage unit and the battery energy storage unit together constitute a hybrid energy storage system, and are connected to the DC traction power supply network through bidirectional DC / DC converters to realize energy interaction between the energy storage unit and the traction power supply system.

[0042] The design of the current droop control for the urban rail hybrid energy storage system is as follows: Taking a supercapacitor as an example, the DC traction grid voltage... Respectively compared with the supercapacitor charging threshold and supercapacitor discharge threshold The results are compared and adjusted by the charge / discharge droop coefficients to generate a supercapacitor reference current. Then, it is combined with the feedback supercapacitor current. The comparison is performed, and the control signal is generated after adjustment by the PI controller. The drive duty cycle of the bidirectional DC / DC converter switching device is then output through the PWM module. This enables control of the supercapacitor's charging and discharging current, with charging and discharging thresholds of 1520V and 1480V, respectively. The initial droop coefficient is determined based on the slope of the I-V droop curve and calculated jointly from the allowable fluctuation range of the traction grid voltage and the rated current of the energy storage system. During operation, the initial droop coefficient is dynamically corrected by the charging and discharging droop coefficient adjustment amount output by the intelligent agent. The droop coefficient is updated as follows:

[0043] (1)

[0044] , , , These are the initial charge / discharge droop coefficients of supercapacitors and batteries, respectively. , , , These are the adjustment amounts for the supercapacitor charge / discharge droop coefficient and the battery charge / discharge droop coefficient, respectively. , , , These are the adjusted charge / discharge droop coefficients for supercapacitors and batteries, respectively. , , , The values ​​are 0.325, 0.45, 0.52, and 0.72, respectively.

[0045] A collaborative control architecture based on the MADDPG algorithm is constructed. Supercapacitor agent Agent A and battery agent Agent B are built based on the supercapacitor energy storage unit and the battery energy storage unit, respectively. Agent A is used to optimize the supercapacitor's charge-discharge droop coefficient online and output the adjustment amount of the supercapacitor's charge-discharge droop coefficient. Agent B is used to optimize the battery's charge-discharge droop coefficient online and output the adjustment amount of the battery's charge-discharge droop coefficient, such as... Figure 1 As shown, two intelligent agents operate in a shared environment using the urban rail traction power supply system and the hybrid energy storage system, sharing system state information to achieve coordinated control between the supercapacitor and the battery. A centralized training and distributed execution mechanism is adopted. During the training phase, the policy network is optimized using the system's global state information. During the execution phase, each agent outputs corresponding actions based on its current operating state, achieving adaptive droop control of the urban rail hybrid energy storage system.

[0046] To realize the application of the MADDPG algorithm in urban rail hybrid energy storage systems, environmental states, agent actions, and multi-objective collaborative reward mechanisms were designed.

[0047] 1) Traction power supply environment and condition characterization design:

[0048] During the MADDPG algorithm training process, the environment provides updated state and corresponding reward signals after each action performed by the agent. Therefore, the adaptive droop control process of the urban rail hybrid energy storage system is described as a Markov decision process. The DC traction power supply system and the hybrid energy storage system constitute the environment of the agent. The agent outputs corresponding control actions by sensing environmental state information and continuously optimizes the control strategy based on the reward signals fed back from the environment to maximize the cumulative reward. Considering the comprehensive requirements for the energy-saving and voltage-stabilizing performance of the traction network and the safe operation of energy storage components under frequent train traction and braking conditions, the DC traction network voltage is selected... Battery voltage Supercapacitor voltage Battery state of charge State of charge of supercapacitors Reference power of hybrid energy storage system Train speed and train acceleration As an environmental state space The state space is represented as:

[0049] (2)

[0050] 2) Motion space design and control output:

[0051] The MADDPG algorithm selects control actions from the action space and dynamically adjusts the charge / discharge droop coefficient. Based on the control requirements of the supercapacitor energy storage unit and the battery energy storage unit, action spaces corresponding to Agent A and Agent B are constructed respectively. and action space ,Strategy Representing the state space Mapping relationship to action space: This is used to describe the control behavior of the agent under different system states. The action space is constructed as follows:

[0052] (3)

[0053] 3) Design of a multi-objective collaborative reward mechanism:

[0054] Reward signals are the feedback from the environment to the agent's actions, and the design of the reward function directly affects the agent's learning effectiveness and control performance. Energy efficiency will be improved. Voltage regulation efficiency Supercapacitor safety operation indicators Battery safety operation indicators As an evaluation metric for the reward function; among which, energy efficiency With voltage regulation efficiency As a globally shared reward item, it applies to both intelligent agents simultaneously; supercapacitor safety operation indicators and battery safety operation indicators As a local constraint, it guides each agent to independently optimize control based on the characteristics of different energy storage units, and constructs reward functions for Agent A and Agent B respectively. and The agent continuously interacts with the environment and optimizes its control strategy based on the reward signals it receives, thereby maximizing the system's long-term cumulative reward. The reward function is defined as follows:

[0055] (4)

[0056] in, , , , , , The weighting coefficients for each indicator are 0.5, 0.5, 0.5, 0.5, 10, and 10, respectively. , , and As shown in equations (5) to (8) respectively;

[0057] (5)

[0058] Energy efficiency Defined as the percentage change in the total output energy of a substation before and after the installation of a hybrid energy storage system, relative to the total output energy of the substation without an energy storage system. These represent the DC traction network currents with and without hybrid energy storage installed, respectively. These represent the DC traction grid voltage under conditions of installed / non-installed hybrid energy storage;

[0059] (6)

[0060] Voltage regulation efficiency The evaluation is based on the integral of the portion of the DC traction network voltage that exceeds / falls below the limit. These are the set upper and lower safety limits for the DC traction network voltage, respectively. These refer to the time during which the DC traction network voltage exceeds the upper / lower safety limit during train operation.

[0061] (7)

[0062] (8)

[0063] in, and These are the states of charge of the supercapacitor. The lower and upper limits for safe operation are set to 0.15 and 0.85, respectively. and These are the battery state of charge. The lower and upper limits for safe operation are set to 0.2 and 0.85, respectively. When the state of charge of the supercapacitor or battery exceeds the preset safe operating range, the corresponding indicator value is calculated based on the degree of deviation from the safety boundary. When it is within the safe operating range, the indicator value is 0. and It participates in agent training as part of the reward function, and is used to constrain the operating state of supercapacitors and batteries to prevent overcharging and over-discharging of energy storage components.

[0064] Design a reinforcement learning strategy training and sequential decision-making execution mechanism, such as Figure 2 As shown.

[0065] (1) In the reinforcement learning strategy training module, a simulation platform for urban rail traction power supply and hybrid energy storage system is established and used as the environment for Agent A and Agent B. The two agents continuously interact with the environment and update their strategies based on the reward function of the environment feedback. In each training round, the train operation status is randomly initialized within the preset safe operation range. The agents learn by continuously interacting with the environment until the reward converges, and the training obtains a control strategy that can achieve energy saving, voltage stabilization and safe operation of energy storage components in the traction network under random operation environment.

[0066] (2) In the sequential decision execution module, Agent A and Agent B execute decisions online based on the real-time system status, and dynamically output the adjustment amount of the supercapacitor charging and discharging droop coefficient and the battery charging and discharging droop coefficient; the control strategy is continuously updated by combining reinforcement learning strategy training with sequential decision execution.

[0067] The system operates as follows:

[0068] Step 1: Agent A and Agent B interact with the system environment for training. By collecting the system's status, actions, and reward information in real time, they repeatedly interact and train N times (N≥100) until the accumulated reward stabilizes and converges. Then, they are used in the urban rail hybrid energy storage system. The intelligent agents make online sequential decisions based on the real-time collected system operating status.

[0069] Step 2: Based on the output actions of the intelligent agent, obtain the adjustment amount of the supercapacitor charge and discharge droop coefficient and the battery charge and discharge droop coefficient respectively, and fuse them with the corresponding initial droop coefficient through the droop coefficient update mechanism to obtain the real-time charge and discharge droop coefficient applicable to the current operating state.

[0070] Step 3: Based on the real-time feedback voltage of the traction network generated by train operation The difference between the current and the charging / discharging threshold is processed by the adjusted droop coefficient to obtain the reference current of the energy storage unit. This reference current is then compared with the actual current of the energy storage unit. After adjustment by the PI controller, a PWM control signal is output to drive the bidirectional DC / DC converter and realize the charging and discharging control of the energy storage unit.

[0071] Step 4: When the train operation task ends and the train stops running, the system stops the intelligent agent decision output and ends the adaptive droop control process of the urban rail hybrid energy storage system.

[0072] This invention employs an adaptive droop control method for urban rail hybrid energy storage systems based on deep reinforcement learning. Using a hybrid energy storage system composed of supercapacitors and batteries as the controlled object, energy interaction between the energy storage units and the DC traction power supply network is achieved through current droop control. A multi-objective collaborative reward mechanism is constructed, comprehensively considering factors such as train operating conditions, traction network operating status, and energy storage unit state of charge. A reinforcement learning strategy training and sequential decision-making execution mechanism is designed. Through a continuous "perception-decision-feedback" interaction process between the agent and the environment, adaptive optimization of the droop coefficient is achieved. The obtained control strategy can dynamically adjust the charging and discharging behavior of the supercapacitors and batteries according to the system operating status, enabling the control strategy to adapt to the energy interaction requirements under different operating conditions. This effectively improves the energy-saving and voltage-stabilizing characteristics of the traction network, prevents overcharging and over-discharging of energy storage components, and comprehensively enhances the energy-saving and economic benefits of the urban rail transit system.

[0073] The embodiments described above are merely preferred embodiments of the present invention, used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. For those skilled in the art, various modifications, substitutions, equivalent improvements, or combinations can be made to the above embodiments without departing from the concept and technical essence of the present invention, and all such modifications, substitutions, equivalent improvements, or combinations should fall within the scope of protection defined by the claims of the present invention.

Claims

1. An adaptive droop control method for urban rail hybrid energy storage systems based on deep reinforcement learning, characterized in that: A collaborative control architecture based on the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm is adopted. By constructing a multi-objective collaborative reward mechanism that considers energy efficiency, voltage regulation efficiency, and the safe operation of energy storage components, and designing a reinforcement learning policy training and sequential decision execution mechanism, the droop coefficient of the urban rail hybrid energy storage system is adaptively optimized through a continuous "perception-decision-feedback" interaction process between the agent and the environment. The method includes the following steps: The traction power supply system adopts a DC voltage level of 1500V. The supercapacitor energy storage unit and the battery energy storage unit together constitute a hybrid energy storage system, and are respectively connected to the DC traction power supply network through bidirectional DC / DC converters to realize energy interaction between the energy storage unit and the traction power supply system. The design of the current droop control for the urban rail hybrid energy storage system is as follows: The deviation between the DC traction network voltage and the charge / discharge threshold is adjusted by the charge / discharge droop coefficient to obtain the reference value of the energy storage unit's charge / discharge current. This reference current, after adjustment by the controller, generates the converter duty cycle signal to achieve charge / discharge control of the energy storage unit. The initial droop coefficient is determined based on the slope of the I-V droop curve and calculated jointly from the allowable fluctuation range of the traction network voltage and the rated current of the energy storage system. During operation, the initial droop coefficient is dynamically corrected by the charge / discharge droop coefficient adjustment amount output by the intelligent agent. The droop coefficient is updated as follows: （1） , , , These are the initial charge / discharge droop coefficients of supercapacitors and batteries, respectively. , , , These are the adjustment amounts for the supercapacitor charge / discharge droop coefficient and the battery charge / discharge droop coefficient, respectively. , , , These are the adjusted charge / discharge droop coefficients for supercapacitors and batteries, respectively. A collaborative control architecture based on the MADDPG algorithm is adopted, constructing a supercapacitor agent (AgentA) and a battery agent (AgentB) based on the supercapacitor energy storage unit and the battery energy storage unit, respectively. AgentA is used to optimize the supercapacitor's charge-discharge droop coefficient online and output the adjustment amount of the supercapacitor's charge-discharge droop coefficient; AgentB is used to optimize the battery's charge-discharge droop coefficient online and output the adjustment amount of the battery's charge-discharge droop coefficient. The two agents use the urban rail traction power supply system and the hybrid energy storage system as a common operating environment, sharing system state information to achieve collaborative control between the supercapacitor and the battery. A centralized training and distributed execution mechanism is adopted. During the training phase, the policy network is optimized using the system's global state information. During the execution phase, each agent outputs corresponding execution actions according to the current operating state, realizing adaptive droop control of the urban rail hybrid energy storage system. To realize the application of the MADDPG algorithm in urban rail hybrid energy storage system, environmental state, agent action and multi-objective cooperative reward mechanism are designed respectively; 1) Traction power supply environment and condition characterization design: During the MADDPG algorithm training process, the environment provides updated state and corresponding reward signals after each action performed by the agent. Therefore, the adaptive droop control process of the urban rail hybrid energy storage system is described as a Markov decision process. The DC traction power supply system and the hybrid energy storage system constitute the agent's environment. The agent outputs corresponding control actions by sensing environmental state information and continuously optimizes the control strategy based on the reward signals from the environment to maximize cumulative rewards. Considering the comprehensive requirements for the energy-saving and voltage-stabilizing performance of the traction network and the safe operation of energy storage components under frequent train traction and braking conditions, the DC traction network voltage is selected... Battery voltage Supercapacitor voltage Battery state of charge State of charge of supercapacitors Reference power of hybrid energy storage system Train speed and train acceleration As an environmental state space The state space is represented as: （2） 2) Motion space design and control output: The MADDPG algorithm selects control actions from the action space and dynamically adjusts the charge / discharge droop coefficient; based on the control requirements of the supercapacitor energy storage unit and the battery energy storage unit, it constructs action spaces corresponding to Agent A and Agent B respectively. and action space ,Strategy Representing the state space Mapping relationship to action space: This is used to describe the control behavior of the agent under different system states; the action space is constructed as follows: （3） 3) Design of a multi-objective collaborative reward mechanism: The reward signal is the feedback from the environment to the agent's actions; the design of the reward function directly affects the agent's learning effect and control performance; energy efficiency. Voltage regulation efficiency Supercapacitor safety operation indicators Battery safety operation indicators As an evaluation metric for the reward function; among which, energy efficiency With voltage regulation efficiency As a globally shared reward item, it applies to both intelligent agents simultaneously; supercapacitor safety operation indicators and battery safety operation indicators As a local constraint, it guides each agent to independently optimize control based on the characteristics of different energy storage units, and constructs reward functions for Agent A and Agent B respectively. and The agent continuously interacts with the environment and optimizes its control strategy based on the reward signals it receives, thereby maximizing the system's long-term cumulative reward. The reward function is defined as follows: （4） in , , , , , These are the weighting coefficients for each indicator. , , and As shown in equations (5) to (8) respectively; （5） Energy efficiency Defined as the percentage change in the total output energy of a substation before and after the installation of a hybrid energy storage system, relative to the total output energy of the substation without an energy storage system. These represent the DC traction network currents with and without hybrid energy storage installed, respectively. These represent the DC traction grid voltage under conditions of installed / non-installed hybrid energy storage; （6） Voltage regulation efficiency The evaluation is based on the integral of the portion of the DC traction network voltage that exceeds / falls below the limit. These are the set upper and lower safety limits for the DC traction network voltage, respectively. These refer to the time during which the DC traction network voltage exceeds the upper / lower safety limit during train operation. （7） （8） in, and These are the states of charge of the supercapacitor. Lower and upper limits for safe operation and These are the battery state of charge. Lower and upper limits for safe operation; Design a reinforcement learning strategy training and sequential decision-making execution mechanism; (1) In the reinforcement learning strategy training module, a simulation platform for urban rail traction power supply and hybrid energy storage system is established and used as the environment for Agent A and Agent B. The two agents continuously interact with the environment and update their strategies based on the reward function of the environment feedback. In each training round, the train operation status is randomly initialized within the preset safe operation range. The agents learn through continuous interaction with the environment until the reward converges, and the training obtains a control strategy that can achieve energy saving, voltage stabilization and safe operation of energy storage components in the traction network under random operation environment. (2) In the sequential decision execution module, Agent A and Agent B execute decisions online based on the real-time system status, and dynamically output the adjustment amount of the supercapacitor charge and discharge droop coefficient and the battery charge and discharge droop coefficient; the control strategy is continuously updated by combining reinforcement learning strategy training with sequential decision execution. The system operates as follows: Step 1: Agent A and Agent B interact with the system environment for training. By collecting the system's status, actions and reward information in real time, they repeatedly interact and train N times until the accumulated reward stabilizes and converges. Then, they are used to realize online sequential decision-making in the urban rail hybrid energy storage system. Step 2: The system outputs the adjustment amount of the supercapacitor charge and discharge droop coefficient and the battery charge and discharge droop coefficient respectively according to the current operating status. The adjustment amount is then fused with the corresponding initial droop coefficient through the droop coefficient update mechanism to obtain the charge and discharge droop coefficient applicable to the current operating status. Step 3: Based on the real-time feedback voltage of the traction network generated by train operation The difference between the current and the charging / discharging threshold is processed by the adjusted droop coefficient to obtain the reference current of the energy storage unit. This reference current is then compared with the actual current of the energy storage unit. After adjustment by the PI controller, a PWM control signal is output to drive the bidirectional DC / DC converter and realize the charging and discharging control of the energy storage unit. Step 4: When the train operation task ends and the train stops running, the system stops the intelligent agent decision output and ends the adaptive droop control process of the urban rail hybrid energy storage system.

Images