A power distribution method for a hybrid regenerative braking system of urban rail based on multi-agent reinforcement learning

Abstract

The application discloses a kind of city rail hybrid regenerative braking system power distribution methods based on multi-agent reinforcement learning, for city rail regenerative braking energy utilization rate is low, traction network voltage fluctuation and energy storage equipment safety is insufficient and so on, construct by lithium battery, super capacitor and net side energy feed system composition Hybrid Energy Management System. Establish system energy flow model, design bottom layer inverter and bidirectional DC-DC converter collaborative control strategy;Based on multi-agent reinforcement learning, construct collaborative decision architecture, combine traction network state, energy storage state and train operation state to construct multidimensional state space, and design reward function considering voltage stabilization, energy saving, energy storage safety and surplus energy consumption, realize regenerative braking energy dynamic optimization distribution.The application can improve the efficiency of regenerative braking energy utilization, reduce the traction network voltage fluctuation, protect the safe operation of energy storage unit, and can be applied to city rail and other DC traction power supply system.

Description

Technical Field

[0001] This invention relates to the field of regenerative braking energy utilization in urban rail transit, and in particular to a power allocation method for a hybrid regenerative braking system for urban rail transit based on multi-agent reinforcement learning. Background Technology

[0002] With the accelerating pace of urbanization, rail transit has become the backbone of modern urban public transportation systems. However, the large amount of regenerative braking energy generated during its operation, if not effectively recovered and utilized, will lead to energy waste. Current common methods, such as resistive dissipation or direct feedback to the grid, not only have low utilization rates but may also cause grid voltage fluctuations and harmonic pollution, affecting power supply quality and system stability. Therefore, researching how to efficiently recover regenerative braking energy and optimize its utilization is of significant theoretical and engineering value for promoting energy conservation and emission reduction in urban rail transit. By integrating energy storage systems and energy feedback technologies, not only can the rational storage and scheduling of braking energy be achieved, improving energy efficiency, but harmonics can also be suppressed, reactive power compensated, and grid compatibility enhanced. Furthermore, optimizing the coordinated operation of multiple energy storage devices helps extend equipment lifespan, reduce maintenance costs, and improve system economy and reliability. This technology is not only applicable to urban rail transit but can also be extended to electric-driven transportation systems such as high-speed rail and trams, possessing broad application prospects and significant social benefits for promoting green transportation construction and enhancing urban sustainable development capabilities. Summary of the Invention

[0003] This invention provides a power allocation method for a hybrid regenerative braking system for urban rail transit based on multi-agent reinforcement learning. The hybrid regenerative braking system includes a lithium battery, a supercapacitor, and a grid-side inverter. The specific implementation of this invention is as follows:

[0004] First, establish the energy flow model of the urban rail hybrid regenerative braking system as shown in equation (1), and collect the DC traction voltage in real time. Hybrid energy storage system given power State of charge of supercapacitors lithium battery state of charge Supercapacitor terminal voltage Lithium battery terminal voltage Train speed Train acceleration Energy feed power Based on the operating parameters, a multidimensional state space for transient energy impact is constructed as shown in equation (2).

[0005]

[0006] in Indicates traction power. Indicates the power of the supercapacitor. Indicates the lithium battery power. Indicates the power of the feed; when At that time, energy is absorbed by the traction network and discharged by the supercapacitor and lithium battery. ; At that time, energy is released to the traction network, and the supercapacitor and lithium battery are charged. .

[0007]

[0008] in , , , , , , , , These represent the DC traction grid voltage at time t, the total power demand of the hybrid energy storage system, the state of charge of the supercapacitor, the state of charge of the lithium battery, the terminal voltage of the supercapacitor, the terminal voltage of the lithium battery, the train speed, the train acceleration, and the energy feed power, respectively.

[0009] To achieve optimal dynamic allocation of regenerative braking energy under complex operating conditions, this invention adopts a heterogeneous collaborative decision-making model based on multi-agent reinforcement learning, constructs a collaborative optimization mechanism with local independent decision-making and global information interaction, and realizes joint action decision-making between energy storage agents and energy feed-in agents.

[0010] The action space of the intelligent agent includes: the supercapacitor power adjustment amount output by intelligent agent A. Power adjustment amount of the energy feed system output by agent B As shown in equation (3);

[0011]

[0012] in , Let t be the agent's action at time t. , The power adjustment of the supercapacitor and energy feed system at time t;

[0013] A multi-objective reward function based on dynamic adjustment of train operation status is constructed. Addressing the multi-objective optimization needs of urban rail regenerative braking energy management, and considering four optimization objectives—traction network voltage stabilization, system energy saving, energy storage safety, and efficient utilization of surplus energy—a multi-agent differentiated reward function is designed to achieve multi-objective collaborative optimization. Specifically, the voltage stabilization rate is defined by the integral of the traction network voltage deviation. As shown in equation (4), the energy saving rate is defined by the ratio of the change in system traction energy consumption. As shown in equation (5), the safe operating range of supercapacitor SOC is limited. As determined by formula (6), the utilization rate of residual regenerative braking energy is defined as the ratio of the energy storage absorption margin to the energy feed-in power. As shown in equation (7);

[0014]

[0015] in, These represent the DC traction grid voltages with and without a regenerative braking energy utilization system installed. These are the set upper and lower safety limits for the DC traction network voltage, respectively. The time during which the DC traction voltage exceeds the upper / lower safety limit during train operation;

[0016]

[0017] in These represent the DC traction network currents with and without a regenerative braking energy utilization system installed. These represent the feed voltage and current, respectively.

[0018]

[0019] in The optimal value for SOC variation. The penalty coefficient is... This is the offset coefficient;

[0020]

[0021] in To provide power feedback, Indicates traction power. Indicates the power of the supercapacitor. Indicates the power of the lithium battery;

[0022] Intelligent agent A focuses on the safety stabilization and energy-saving optimization of the energy storage system, and integrates the voltage stabilization rate, energy saving rate and supercapacitor SOC safety constraints to construct a reward function as shown in equation (8);

[0023]

[0024] Intelligent agent B focuses on the efficient consumption of residual braking energy, and integrates the voltage stability rate, energy saving rate and residual energy utilization rate to construct a reward function as shown in equation (9), so as to achieve the coordinated balance between energy storage regulation and energy feedback;

[0025]

[0026] in , , , , , These are the weighting coefficients;

[0027] Through the actual voltage of the traction substation Discharge voltage threshold of energy storage system Charging voltage threshold Compare, based on the output Positive and negative values ​​determine the operating status of the energy storage system. The power correction amount from the PI controller is obtained from the traction network voltage loop PI. Power correction amount With load power Feedforward to obtain the given total power of hybrid energy storage Given a total power of hybrid energy storage With the power adjustment of the supercapacitor of the energy storage smart body Output to obtain real-time power command for supercapacitor Real-time battery power The PWM drive signal is generated by adjusting the inner current loop and the outer voltage loop to control the bidirectional DC-DC converter to complete the charging and discharging of the hybrid energy storage unit.

[0028] For energy-feed system control, a reference component for the q-axis current is set. When the traction network voltage is greater than the energy feed start voltage Then, in conjunction with the power feed-in requirements The amount of energy feed power adjustment output by the energy feed agent Output energy feed real-time power command The reference components of the d-axis current in the dq0 coordinate system are calculated as shown in equation (10);

[0029]

[0030] in The real-time d-axis voltage is obtained by performing Park transformation on the phase obtained from the three-phase AC voltage measurement by the voltage sensor through the phase-locked loop.

[0031] The system is running, and the specific steps are as follows:

[0032] Step 1: Build equivalent models of lithium batteries and supercapacitors, as well as a hybrid energy storage + energy feed-in energy management model. Based on electrical parameters, establish the system energy balance relationship and construct the initial operation control benchmark of the system.

[0033] Step 2: Implement voltage regulation control of the energy feeder system and power regulation of the bidirectional DC-DC converter. Based on the load capacity of the hybrid energy storage system and the energy feeder system, the power distribution module generates the real-time hybrid energy storage power demand. and energy feed power demand Hybrid energy storage power demand The high-frequency power command of the supercapacitor is obtained through a low-pass filter. and battery low-frequency power command .

[0034] Step 3: Collect multi-dimensional state data such as DC traction network voltage, energy storage SOC, train operating conditions, and energy feed power to construct the system state space and complete the real-time identification of traction and regenerative braking conditions.

[0035] Step 4: Based on the multi-agent MATD3 algorithm and combined with a multi-objective reward function, power allocation is optimized in real time. Agent A and Agent B output the real-time power compensation amount of the supercapacitor respectively. and energy feed real-time power compensation amount Supercapacitor high-frequency power command The real-time power command of the supercapacitor is obtained by real-time compensation with the output of Agent A. Real-time battery power Power demand from hybrid energy storage and The difference value can be obtained, and the real-time power can be fed back. Through energy feed power demand With real-time power compensation amount The output is obtained.

[0036] Step 5: Through time-series experience updates and reward convergence judgment, if convergence is not achieved, policy retraining is triggered; if convergence is achieved, parameters are fine-tuned periodically to continuously adapt to power grid and train operating condition disturbances.

[0037] Step 6: The optimized power allocation strategy is fed back to the system in a closed loop, and the system control parameters are updated iteratively to achieve dynamic adaptive energy management and optimized utilization of regenerative braking energy throughout the entire process.

[0038] Beneficial Effects: This invention constructs an integrated hybrid regenerative braking system model comprising lithium batteries, supercapacitors, and grid-side inverters. It employs a hierarchical closed-loop control architecture and combines a multi-agent reinforcement learning algorithm to achieve coordinated power allocation between energy storage and the energy feed system. The multi-agent system adopts a collaborative working mode of local independent decision-making and global information interaction. Through a multi-objective hierarchical reward function, it considers multiple optimization objectives, including traction grid voltage stabilization, system energy saving, energy storage SOC safety management, and efficient recovery of remaining braking energy. It can adapt to complex dynamic operating conditions such as train start-stop and braking, and optimize the regenerative braking energy allocation ratio in real time. This invention effectively improves the utilization rate of regenerative braking energy, reduces braking resistor energy consumption, and enhances system energy-saving efficiency. Simultaneously, it constrains energy storage components to operate within a safe range, preventing overcharging and over-discharging, and extending the service life of energy storage equipment. Through the synergistic effect of energy storage voltage stabilization and energy feed, it effectively suppresses traction grid voltage fluctuations and power disturbances, improving power supply stability and system economic efficiency. It is adaptable to various DC traction transportation power supply scenarios such as urban rail, trams, and regional EMUs. Attached Figure Description

[0039] Figure 1 A multi-agent reinforcement learning collaborative energy management system architecture;

[0040] Figure 2 This is a control structure diagram of a hybrid regenerative braking system for urban rail transit based on the MATD3 algorithm. Detailed Implementation

[0041] To further illustrate the technical solution of the present invention in detail, so that those skilled in the art can clearly and completely understand and implement the present invention, the present invention will be further described in detail below with reference to specific embodiments. The described embodiments are only some embodiments of the present invention, not all embodiments, and should not be regarded as limiting the scope of protection of the present invention.

[0042] This invention discloses a power allocation method for a hybrid regenerative braking system in urban rail transit based on multi-agent reinforcement learning. The applicable hybrid regenerative braking system mainly consists of a lithium battery energy storage unit, a supercapacitor energy storage unit, and a grid-side energy feed system, and operates entirely based on a DC traction power supply system. Regenerative braking energy generated during train braking can be absorbed and stored through the hybrid energy storage system, with surplus energy fed back via a grid-side inverter. During train traction, the energy storage system releases electrical energy to support the traction grid's power demand, thereby achieving efficient recovery and dynamic scheduling of regenerative braking energy while simultaneously mitigating traction grid voltage fluctuations.

[0043] like Figure 1The invention presented here adopts a hierarchical control architecture of "bottom-level converter collaborative control + upper-level multi-agent reinforcement learning optimization". The bottom layer realizes closed-loop precise control of the three-phase grid-connected inverter and the bidirectional DC-DC converter, completing the basic execution of energy storage charging and discharging and energy feedback. The upper layer relies on multi-agent reinforcement learning algorithms to realize intelligent power allocation of the hybrid energy storage and energy feedback system, adapting to complex time-varying operating conditions such as starting, stopping, braking, and constant speed operation of urban rail trains. The specific method is as follows:

[0044] An energy flow model for a hybrid regenerative braking system for urban rail transit is established as shown in equation (1), and the DC traction voltage is collected in real time. Hybrid energy storage system given power State of charge of supercapacitors lithium battery state of charge Supercapacitor terminal voltage Lithium battery terminal voltage Train speed Train acceleration Energy feed power The operating parameters are used to construct a multi-dimensional state space for transient energy impact as shown in equation (2).

[0045]

[0046] in Indicates traction power. Indicates the power of the supercapacitor. Indicates the lithium battery power. Indicates the power of the feed; when At that time, energy is absorbed by the traction network and discharged by the supercapacitor and lithium battery. ; At that time, energy is released to the traction network, and the supercapacitor and lithium battery are charged. ;

[0047]

[0048] in , , , , , , , , These represent the DC traction grid voltage at time t, the total power demand of the hybrid energy storage system, the state of charge of the supercapacitor, the state of charge of the lithium battery, the terminal voltage of the supercapacitor, the terminal voltage of the lithium battery, the train speed, the train acceleration, and the energy feed power, respectively.

[0049] To achieve optimal dynamic allocation of regenerative braking energy under complex operating conditions, this invention adopts a heterogeneous collaborative decision-making model based on multi-agent reinforcement learning to realize joint action decision-making between the energy storage agent and the energy feed-in agent. The two agents do not work in complete isolation, nor are they completely merged into one agent; instead, they belong to a collaborative mode of local independent decision-making plus global information interaction.

[0050] The action space of the intelligent agent includes: the supercapacitor power adjustment amount output by intelligent agent A. Power adjustment amount of the energy feed system output by agent B As shown in equation (3);

[0051]

[0052] in , Let t be the agent's action at time t. , The power adjustment of the supercapacitor and energy feed system at time t;

[0053] A multi-objective reward function based on dynamic adjustment of train operation status is constructed. Addressing the multi-objective optimization needs of urban rail regenerative braking energy management, and considering four optimization objectives—traction network voltage stabilization, system energy saving, energy storage safety, and efficient utilization of surplus energy—a multi-agent differentiated reward function is designed to achieve multi-objective collaborative optimization. Specifically, the voltage stabilization rate is defined by the integral of the traction network voltage deviation. As shown in equation (4), the energy saving rate is defined by the ratio of the change in system traction energy consumption. As shown in equation (5), the safe operating range of supercapacitor SOC is limited. As determined by formula (6), the utilization rate of residual regenerative braking energy is defined as the ratio of the energy storage absorption margin to the energy feed-in power. As shown in equation (7);

[0054]

[0055] in, These represent the DC traction grid voltages with and without a regenerative braking energy utilization system installed. These are the set upper and lower safety limits for the DC traction network voltage, respectively. The time during which the DC traction voltage exceeds the upper / lower safety limit during train operation;

[0056]

[0057] in These represent the DC traction network currents with and without a regenerative braking energy utilization system installed. These represent the feed voltage and current, respectively.

[0058]

[0059] in The optimal value for SOC variation is 0.5. The penalty coefficient is set to 10. The offset coefficient is set to 10.

[0060]

[0061] in To provide power feedback, Indicates traction power. Indicates the power of the supercapacitor. Indicates the power of the lithium battery;

[0062] Intelligent agent A focuses on the safety stabilization and energy-saving optimization of the energy storage system, and integrates the voltage stabilization rate, energy saving rate and supercapacitor SOC safety constraints to construct a reward function as shown in equation (8);

[0063]

[0064] Intelligent agent B focuses on the efficient consumption of residual braking energy, and integrates the voltage stability rate, energy saving rate and residual energy utilization rate to construct a reward function as shown in equation (9), so as to achieve the coordinated balance between energy storage regulation and energy feedback;

[0065]

[0066] in , , , , , , where is the weighting coefficient, and the values ​​are 2, 2, 10, 2, 2, 2;

[0067] The coordination mechanism between agents A and B adopts a collaborative approach of centralized training and decentralized execution. When the energy storage system reaches the set start-up value, agent A completes the internal power allocation of the hybrid energy storage. Agent B determines the energy feed power adjustment amount based on the remaining regenerative braking power after deducting the absorbable power of the energy storage. When the energy storage SOC approaches the safety boundary or the energy storage reaches the power limit, the energy feed system increases energy absorption. The designed differentiated multi-objective reward function unifies the optimization direction of the two by sharing the two global evaluation indicators of voltage regulation rate and energy saving rate. At the same time, it adds a supercapacitor SOC safety constraint item and a dedicated sub-item of remaining regenerative braking energy utilization rate to divide the work: the energy storage agent is penalized when the energy storage charge state approaches the safety boundary by relying on the SOC constraint item, and actively reduces power absorption to release the remaining braking energy. The energy feed agent relies on the energy utilization rate sub-item to take over the surplus energy that the energy storage cannot absorb and obtain positive benefits. If either agent mismanages, it will worsen the voltage regulation and energy saving indicators, causing both parties to suffer negative rewards simultaneously.

[0068] A multi-agent reinforcement learning algorithm is employed for online iterative and closed-loop optimization. The multi-agent system continuously interacts with the urban rail DC traction power supply environment for trial and error training, constantly collecting system status, actions, and reward feedback data to iteratively update the strategy network parameters and gradually optimize the power allocation strategy. During training, multi-objective reward function constraints are used to continuously balance voltage stabilization effects, energy-saving benefits, and energy storage safety performance until the strategy converges and stabilizes.

[0069] like Figure 2 As shown, this invention designs a hierarchical collaborative control strategy for a hybrid energy storage system and an energy feed system, respectively realizing grid-connected voltage regulation control of the grid-side inverter and power regulation control of the energy storage bidirectional converter. The specific control steps are as follows:

[0070] Through the actual voltage of the traction substation Discharge voltage threshold of energy storage system Charging voltage threshold Compare, based on the output Positive and negative values ​​determine the operating status of the energy storage system. The power correction amount from the PI controller is obtained from the traction network voltage loop PI. Power correction amount With load power Feedforward to obtain the given total power of hybrid energy storage Given a total power of hybrid energy storage With the power adjustment of the supercapacitor of the energy storage smart body Output to obtain real-time power command for supercapacitor Real-time battery power The PWM drive signal is generated by adjusting the inner current loop and the outer voltage loop to control the bidirectional DC-DC converter to complete the charging and discharging of the hybrid energy storage unit.

[0071] For energy-feed system control, to ensure system unity-power operation, a q-axis current reference component is set. When the traction network voltage is greater than the energy feed start voltage Then, in conjunction with the power feed-in requirements The amount of energy feed power adjustment output by the energy feed agent Output energy feed real-time power command The reference components of the d-axis current in the dq0 coordinate system are calculated as shown in equation (10);

[0072]

[0073] in The real-time d-axis voltage is obtained by performing Park transformation on the phase obtained from the three-phase AC voltage measurement by the voltage sensor through the phase-locked loop.

[0074] The system is running, and the specific steps are as follows:

[0075] Step 1: Build equivalent models of lithium batteries and supercapacitors, as well as a hybrid energy storage + energy feed-in energy management model. Based on electrical parameters, establish the system energy balance relationship and construct the initial operation control benchmark of the system.

[0076] Step 2: Implement grid-connected inverter voltage regulation and bidirectional DC-DC converter power regulation. Based on the load capacity of the hybrid energy storage system and the energy feed system, the power distribution module generates the real-time hybrid energy storage power demand. and energy feed power demand Hybrid energy storage power demand The high-frequency power command of the supercapacitor is obtained through a low-pass filter. and battery low-frequency power command .

[0077] Step 3: Collect multi-dimensional state data such as DC traction network voltage, energy storage SOC, train operating conditions, and energy feed power to construct the system state space and complete the real-time identification of traction and regenerative braking conditions.

[0078] Step 4: Based on the multi-agent MATD3 algorithm and combined with a multi-objective reward function, power allocation is optimized in real time. Agent A and Agent B output the real-time power compensation amount of the supercapacitor respectively. and energy feed real-time power compensation amount Supercapacitor high-frequency power command The real-time power command of the supercapacitor is obtained by real-time compensation with the output of Agent A. Real-time battery power Power demand from hybrid energy storage and The difference value can be obtained, and the real-time power can be fed back. Through energy feed power demand With real-time power compensation amount The output is obtained.

[0079] Step 5: Through time-series experience updates and reward convergence judgment, if convergence is not achieved, policy retraining is triggered; if convergence is achieved, parameters are fine-tuned periodically to continuously adapt to power grid and train operating condition disturbances.

[0080] Step 6: The optimized power allocation strategy is fed back to the system in a closed loop, and the system control parameters are updated iteratively to achieve dynamic adaptive energy management and optimized utilization of regenerative braking energy throughout the entire process.

[0081] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Without departing from the concept, spirit, and core principles of the present invention, those skilled in the art can make various modifications, equivalent substitutions, and structural variations based on the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should fall within the protection scope of the appended claims.

Claims

1. A power allocation method for a hybrid regenerative braking system for urban rail transit based on multi-agent reinforcement learning, characterized in that: For a hybrid regenerative braking energy utilization system for urban rail transit, consisting of a lithium battery energy storage unit, a supercapacitor energy storage unit, and a grid-side energy feeder system, a multi-agent collaborative decision-making and control architecture is constructed, comprising heterogeneous energy storage agents and energy feeder agents. This architecture includes an energy storage agent, which generates power response decisions at different time scales based on the real-time operating states of the supercapacitor and lithium battery; and an energy feeder agent, which generates energy feeder power adjustment decisions based on the traction grid voltage state, remaining regenerative braking energy, and energy storage absorption capacity. Through an information interaction mechanism between agents, each agent shares global environmental information including the traction grid operating state, energy storage charge state, train operating state, and energy flow change trends, and performs independent strategy updates based on local observation information, achieving collaborative optimization control between the energy storage system and the energy feeder system. This invention is divided into design and operation parts, specifically implemented as follows: Design section: Establish the energy flow model of the urban rail hybrid regenerative braking system as shown in equation (1), and collect DC traction voltage in real time. Hybrid energy storage system given power State of charge of supercapacitors lithium battery state of charge Supercapacitor terminal voltage Lithium battery terminal voltage Train speed Train acceleration Energy feed power The operating parameters are used to construct a multi-dimensional state space for transient energy impact as shown in equation (2). in Indicates traction power. Indicates the power of the supercapacitor. Indicates the lithium battery power. Indicates the power of the feed; when At this time, energy is absorbed by the traction network and discharged by the supercapacitor and lithium battery. ;when At this time, energy is released to the traction network, and the supercapacitor and lithium battery are charged. ; in , , , , , , , , These represent the DC traction grid voltage at time t, the total power demand of the hybrid energy storage system, the state of charge of the supercapacitor, the state of charge of the lithium battery, the terminal voltage of the supercapacitor, the terminal voltage of the lithium battery, the train speed, the train acceleration, and the energy feed power, respectively. To achieve optimal dynamic allocation of regenerative braking energy under complex operating conditions, this invention adopts a heterogeneous collaborative decision-making model based on multi-agent reinforcement learning, constructs a collaborative optimization mechanism with local independent decision-making and global information interaction, and realizes joint action decision-making between energy storage agents and energy feed-in agents. The action space of the intelligent agent includes: the supercapacitor power adjustment amount output by intelligent agent A. Power adjustment amount of the energy feed system output by agent B As shown in equation (3); in , Let t be the agent's action at time t. , The power adjustment of the supercapacitor and energy feed system at time t; A multi-objective reward function based on dynamic adjustment of train operation status is constructed. Addressing the multi-objective optimization needs of urban rail regenerative braking energy management, and considering four optimization objectives—traction network voltage stabilization, system energy saving, energy storage safety, and efficient utilization of surplus energy—a multi-agent differentiated reward function is designed to achieve multi-objective collaborative optimization. Specifically, the voltage stabilization rate is defined by the integral of the traction network voltage deviation. As shown in equation (4), the energy saving rate is defined by the ratio of the change in system traction energy consumption. As shown in equation (5), the safe operating range of supercapacitor SOC is limited. As determined by formula (6), the utilization rate of residual regenerative braking energy is defined as the ratio of the energy storage absorption margin to the energy feed-in power. As shown in equation (7); in, These represent the DC traction grid voltages with and without a regenerative braking energy utilization system installed. These are the set upper and lower safety limits for the DC traction network voltage, respectively. The time during which the DC traction voltage exceeds the upper / lower safety limit during train operation; in These represent the DC traction network currents with and without a regenerative braking energy utilization system installed. These represent the feed voltage and current, respectively. in The optimal value for SOC variation. The penalty coefficient is... This is the offset coefficient; in To provide power feedback, Indicates traction power. Indicates the power of the supercapacitor. Indicates the power of the lithium battery; Intelligent agent A is a hybrid energy storage power allocation intelligent agent. The controlled object is a hybrid energy storage system composed of supercapacitors and lithium batteries. Its main output is the power adjustment of supercapacitors. The power of lithium batteries is determined by the difference between the total power demand of hybrid energy storage and the real-time power of supercapacitors. Focusing on the safe voltage regulation and energy-saving optimization of energy storage system, the reward function is constructed by integrating the voltage regulation rate, energy saving rate and supercapacitor SOC safety constraints as shown in equation (8). Intelligent agent B is a grid-side energy feed power regulation intelligent agent, and the controlled object is the grid-side energy feed system; focusing on the efficient absorption of residual braking energy, the reward function is constructed by integrating the voltage regulation rate, energy saving rate and residual energy utilization rate as shown in equation (9), so as to achieve the coordinated balance between energy storage regulation and energy feed feedback; in , , , , , These are the weighting coefficients; The coordination mechanism between agents A and B adopts a collaborative approach of centralized training and decentralized execution. When the energy storage system reaches the set start-up value, agent A completes the internal power distribution of the hybrid energy storage. Agent B determines the energy feed power adjustment amount based on the remaining regenerative braking power after deducting the absorbable power of the energy storage. When the energy storage SOC approaches the safety boundary or the energy storage reaches the power limit, the energy feed system increases energy absorption. The designed differentiated multi-objective reward function unifies the optimization direction of the two by sharing the two global evaluation indicators of voltage regulation rate and energy saving rate. At the same time, it adds a supercapacitor SOC safety constraint item and a dedicated sub-item of remaining regenerative braking energy utilization rate to divide the work: the energy storage agent is penalized when the energy storage charge state approaches the safety boundary by relying on the SOC constraint item, and actively reduces power absorption to release the remaining braking energy. The energy feed agent relies on the energy utilization rate sub-item to take over the surplus energy that the energy storage cannot absorb and obtain positive benefits. If either agent mismanages, it will worsen the voltage regulation and energy saving indicators, causing both parties to suffer negative rewards simultaneously. Through the actual voltage of the traction substation Discharge voltage threshold of energy storage system Charging voltage threshold Compare, based on the output Positive and negative values ​​determine the operating status of the energy storage system. The power correction amount from the PI controller is obtained from the traction network voltage loop PI. Power correction amount With load power Feedforward to obtain the given total power of hybrid energy storage Given a total power of hybrid energy storage With the power adjustment of the supercapacitor of the energy storage smart body Output the real-time power command of the supercapacitor Real-time battery power The PWM drive signal is generated by adjusting the inner current loop and the outer voltage loop to control the bidirectional DC-DC converter to complete the charging and discharging of the hybrid energy storage unit. For energy-feed system control, a reference component for the q-axis current is set. When the traction network voltage is greater than the energy feed start voltage Then, in conjunction with the power feed-in requirements The amount of energy feed power adjustment output by the energy feed agent Output energy feed real-time power command The reference components of the d-axis current in the dq0 coordinate system are calculated as shown in equation (10); in The real-time d-axis voltage is obtained by performing Park transformation on the phase obtained from the three-phase AC voltage measurement through a phase-locked loop using a voltage sensor. Operation section: The system employs a multi-agent reinforcement learning algorithm for online iterative and closed-loop optimization. The multi-agent continuously interacts with the urban rail DC traction power supply environment for online trial and error training, constantly collecting system status, actions, and reward feedback data, iteratively updating the strategy network parameters, and gradually optimizing the power allocation strategy. During training, the system relies on multi-objective reward function constraints to continuously balance voltage regulation effects, energy-saving benefits, and energy storage safety performance until the strategy converges and stabilizes. The optimized power allocation strategy is then distributed in real-time to the underlying inverter and bidirectional DC-DC control loop to update the system operating status and synchronously collect operating condition data for the next moment, achieving a closed-loop dynamic optimization process encompassing "state observation - strategy decision-making - hardware execution - parameter update." The system is running, and the specific steps are as follows: Step 1: Build equivalent models of lithium batteries and supercapacitors and hybrid energy storage + energy feed-in energy management models. Based on electrical parameters, establish the system energy balance relationship and construct the initial operation control benchmark of the system. Step 2: Implement grid-connected inverter voltage regulation and bidirectional DC-DC converter power regulation. Based on the load capacity of the hybrid energy storage system and the energy feed system, the power distribution module generates the real-time hybrid energy storage power demand. and energy feed power demand Hybrid energy storage power demand The high-frequency power command of the supercapacitor is obtained through a low-pass filter. and battery low-frequency power command ; Step 3: Collect multi-dimensional status data such as DC traction network voltage, energy storage SOC, train operating conditions, and energy feed power to construct the system state space and complete the real-time identification of traction and regenerative braking conditions. Step 4: Based on the multi-agent MATD3 (Multi-Agent Dual-Delay Deep Deterministic Policy Gradient Algorithm) algorithm, and combined with the multi-objective reward function, power allocation is optimized in real time. Agent A and Agent B output the real-time power compensation amount of the supercapacitor respectively. and energy feed real-time power compensation amount Supercapacitor high-frequency power command The real-time power command of the supercapacitor is obtained by real-time compensation with the output of Agent A. Real-time battery power Power demand from hybrid energy storage and The difference value can be obtained, and the real-time power can be fed back. Through energy feed power demand With real-time power compensation amount Output obtained; Step 5: Through time-series experience updates and reward convergence judgment, if convergence is not achieved, policy retraining is triggered; if convergence is achieved, parameters are fine-tuned periodically to continuously adapt to power grid and train operating condition disturbances. Step 6: The optimized power allocation strategy is fed back to the system in a closed loop, and the system control parameters are updated iteratively to achieve dynamic adaptive energy management and optimized utilization of regenerative braking energy throughout the entire process.

Images