Heavy-haul train group operation speed and energy cooperative control method and device

Abstract

This invention discloses a method and apparatus for coordinated control of speed and energy in heavy-haul train group operations, comprising: constructing speed control agents and energy management agents for each train, and constructing a shared action value hybrid network for all agents; setting a state observation set and an action set, wherein the state observation set data is obtained by Kalman filtering optimization; each agent using DDPG, outputting a joint action decision based on the current state observation set data and the action value hybrid network function; each train extracting its own action decision from the joint action decision, and controlling the locomotive acceleration and fuel cell stack power, while simultaneously acquiring the state observation set data after the train group executes the action decision; calculating the train group reward function value based on the state observation set data after the train group executes the action decision, and updating each agent and the action value hybrid network. This invention can achieve coordinated optimization of speed control and energy management, improving the energy efficiency of heavy-haul train group operations.

Description

Technical Field

[0001] This invention belongs to the field of train hybrid energy storage train group coordination technology, specifically relating to a method and device for coordinated control of operating speed and energy of heavy-haul train groups. Background Technology

[0002] Heavy-haul trains generally refer to extra-long and extra-heavy freight trains that operate on transport lines with concentrated freight volume, using large dedicated freight cars and traction by two or more locomotives. Heavy-haul trains have high load-bearing capacity and a large number of cars in a single train. Composed of numerous locomotives and rolling stock, heavy-haul trains typically cover very long tracks with varying gradients and curvatures. Therefore, the control design of heavy-haul trains faces significant challenges due to complex modeling, complex operating conditions, and inherent input constraint problems.

[0003] Furthermore, due to the energy crisis and environmental pollution, railway transportation faces increasing pressure to achieve green and energy-saving goals. For railway transportation, the use of train formations with interval tracking and the adoption of heavy-haul hydrogen fuel cell hybrid energy storage trains are effective ways to improve transportation efficiency and achieve energy conservation and emission reduction.

[0004] By coordinating the speed control of heavy-haul train groups, trains can track each other at smaller intervals, optimizing the operating speed of hybrid energy storage train groups, improving energy economy and operational comfort, and ensuring safe following distances. The load power required for the hybrid energy storage trains to operate at optimized speeds is allocated to the hydrogen fuel cell stacks and lithium-ion battery packs by energy management strategies, reducing hydrogen fuel consumption. This coordinated optimization of speed control and energy management can further improve the energy economy of heavy-haul train groups during operation.

[0005] For example, Chinese patent application No. 202011528282.4 discloses a control method for optimized driving and energy management of contactless power supply trains. With the goal of optimizing the economy of the train's operation along the entire line and ensuring on-time arrival, it improves the overall energy utilization efficiency of the train while ensuring timely arrival at stations within acceptable error ranges. Chinese patent application No. 202111108250.3 discloses a joint optimization method for speed planning and energy management of fuel cell vehicles, solving the problem of joint optimization of speed planning and energy management for fuel cell vehicles in scenarios with multiple consecutive traffic lights.

[0006] However, the above literature only considers the speed planning and energy management of a single vehicle, and cannot be applied to the coordinated optimization control of speed control and energy management of train groups. Summary of the Invention

[0007] This invention provides a method and apparatus for coordinated control of speed and energy of heavy-haul train groups, which realizes coordinated optimization of speed control and energy management, and improves the energy economy of heavy-haul train groups during operation.

[0008] To achieve the above technical objectives, the present invention adopts the following technical solution: A method for coordinated control of speed and energy of heavy-haul train groups includes: Step A: Construct a speed control agent and an energy management agent based on DDPG for each train, and build a hybrid action value network shared by all agents; set up a state observation set and an action set, and optimize the state observation set data based on multi-source sensor data using Kalman filtering; Step B: All agents output joint action decisions based on the current state observation set data; the total action value function value of all agents is calculated using an action value hybrid network; Step C: Each train extracts its own action decision from the joint action decision, controls the locomotive's acceleration and fuel cell stack power, and simultaneously acquires the state observation set data of the train group after executing the action decision; Step D: Based on the state observation set data after the train group makes action decisions, calculate the reward function value of the train group, and then calculate the target value of the total action value function of all agents; based on the total action value function value of all agents and the target value of the action value function, update each agent and the action value hybrid network, and return to step B.

[0009] Furthermore, the state observation set covers the kinematic state and powertrain state of each train in the train group; using Kalman filtering and based on the train's dynamic model, multi-source sensor data are fused and recursively calculated to obtain the optimal state estimate, which serves as the state observation set data used by DDPG.

[0010] Furthermore, the multi-source sensor data includes data collected by the following sensors: GPS, IMU, wheel speed sensor, current sensor, and voltage sensor.

[0011] Furthermore, the action value hybrid network merges the local action value functions of all agents while maintaining the consistency of action policies, as shown below: ; ; In the formula, The action value function representing the overall action value of the train group is used to fuse the action values ​​of each agent during the training phase to guide the updating of the training network in each train agent. Represents intelligent agents Action value function, The number of trains in the train group; Indicates all train groups State observation set data of each agent, Represents intelligent agents State observation data, Indicates all train groups A joint action strategy for individual agents Represents intelligent agents Action strategy.

[0012] Furthermore, the action value hybrid network employs a hypernetwork, inputting the state observation set data of the train group to calculate the weights and offsets of the hybrid network.

[0013] Furthermore, when outputting joint action decisions, based on the limited safe distance between locomotives and the limitation of the state of charge of batteries in the train hybrid energy storage system, safety thresholds for the current acceleration and fuel cell stack power of each train are set based on the observed state, and the final joint action decision is output under the input safety constraints.

[0014] Furthermore, the reward function for step D, taking into account energy efficiency, safety, and comfort, is expressed as: ; In the formula, and They represent trains and the train behind exist Location at any given moment Indicates train With the train behind Between Reference distance at any given time; and They represent trains With the train behind exist The speed of time; Indicates train exist Acceleration at all times The change Indicates train exist Hydrogen consumption at any given time. This indicates the number of trains in the train group. This indicates the duration of one training epoch. This represents the weighting coefficient.

[0015] Furthermore, given the chemical energy density of hydrogen, the hydrogen consumption of each train's fuel cell during operation is calculated: ; In the formula, Indicates any train at Hydrogen consumption at any given time. Indicates fuel cell efficiency. This indicates that hydrogen has a low calorific value. This indicates that the train's fuel cell stack is in Output power at time: ; In the formula, and The fuel cell stacks of the train are respectively Output voltage and current at any given time; It represents the thermodynamic electromotive force, also known as the Nernst voltage, which is the theoretical highest voltage; This indicates the voltage loss caused by concentration polarization. This indicates the voltage loss caused by the activation polarization effect. This indicates the voltage loss caused by the Ohmic polarization effect.

[0016] Further, step D includes: D1: Calculate the reward function for the train group; D2: Store the current and next time state observation data, the available action space for each agent at the current and next time, the action selected at the current time, and the reward for the train group into the experience replay pool; D3: When the experience replay pool size meets the set value, sample data from the experience replay pool and update the parameters of the Critic training network by minimizing the loss value. The loss function during Critic network training and updates. for: ; in, Indicates train group State observation set data at time t, Indicates train group Joint action decision-making at any moment Middle intelligence agent Action strategy , Representing intelligent agents Actor training network, Representative input agent State observation data, This represents exploratory noise in action strategies; Indicates the amount of sampled data; The objective value of the action value function in the current optimized state: ; in, Estimate the next-time optimal state in a hybrid network for target action value The Q-value function under the following conditions For each intelligent agent, the target network is... In optimization state The generated joint action strategy; For the train group in status Execute joint action strategy The instant reward received; It is a discount factor; Policy gradient for updating network parameters when training an arbitrary agent (Actor): ; In the formula, Represents intelligent agents Actor training network policy gradient, For intelligent agents The parameters of the Actor-trained network; Represents intelligent agents Critic training network Relative to action gradient, For intelligent agents Parameters of the Critic training network; Represents intelligent agents Actor training network Relative to its parameters The gradient; Update the agent through a soft update mechanism. Critic target network parameters and Actor target network parameters : ; ; In the formula, Indicates the learning rate; D4: Repeat the above steps until all network parameters in the model are updated.

[0017] A device for coordinated control of speed and energy of heavy-haul train groups includes: a sensor, a memory, a processor, and a controller; The sensor is used to sense the information that the train needs to observe and transmits the observation information to the Kalman filter state observer for optimization processing in order to obtain the state observation set data of the train group. The memory stores a computer program, which, when executed by the processor and controller, causes the processor and controller to implement the heavy-haul train group speed and energy coordinated control method described above.

[0018] The present invention provides a method and apparatus for coordinated control of speed and energy of heavy-haul train groups. By controlling the speed of the train group, trains can track each other at smaller intervals, optimizing the operating speed of hybrid energy storage train groups, improving energy efficiency and operational comfort, and ensuring safe following distances. The load power required for the hybrid energy storage trains to operate at optimized speeds is allocated to the hydrogen fuel cell stacks and lithium-ion battery stacks by an energy management strategy, reducing hydrogen fuel consumption. This achieves coordinated optimization of speed control and energy management, improving the energy efficiency of heavy-haul train group operations and alleviating the green energy-saving pressure faced by railway transportation. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of a heavy-haul train group and train powertrain in an embodiment of the present invention; Figure 2 This is a flowchart of the method in an embodiment of the present invention; Figure 3 This is a diagram of the multi-agent reinforcement learning framework in an embodiment of the present invention. Detailed Implementation

[0020] The embodiments of the present invention will be described in detail below. These embodiments are based on the technical solutions of the present invention and provide detailed implementation methods and specific operation processes to further explain the technical solutions of the present invention.

[0021] Example 1

[0022] This embodiment provides a method for coordinated control of speed and energy of heavy-haul train groups, applicable to the control of speed and energy consumption of heavy-haul train groups. Figure 1 The train group shown is configured to coordinate the operating speed of each train and the energy of the lithium-ion and hydrogen fuel hybrid energy storage system, including... Figure 2 The following steps are shown.

[0023] Step A: Construct a speed control agent and an energy management agent based on DDPG for each train, and build a hybrid network of action value shared by all agents; set up a state observation set and an action set, and the state observation set data is optimized based on multi-source sensor data using Kalman filtering.

[0024] A1: Setting the State Observation Set. The state observation set covers the kinematic and powertrain states of each train in the train group. It is obtained by collecting multi-source sensor data such as GPS, IMU, wheel speed sensors, and current / voltage sensors. The data is then input into the Kalman filter state observer, which performs recursive calculations based on the system dynamic model (train kinematic equations, energy flow model) to obtain the state observation set data used in practice. This effectively suppresses measurement noise and process interference, providing a more accurate state input for calculating the motion value function.

[0025] A2: Set the action set, including the acceleration of each train and the power of the fuel cell stack.

[0026] A3: Based on DDPG (Deep Deterministic Policy Gradient Algorithm), construct a speed control agent and an energy management agent for each train in the train group. The action output by the speed control agent is the train acceleration, and the action output by the energy management agent is the power of the train's fuel cell stack.

[0027] Each agent comprises four neural networks used to approximate the Q-value function and policy. The Critic target network is used to approximate the Q-value function of the state-action state in the next time step after Kalman filtering optimization. The next action is approximately estimated by the Actor target network in the optimized state. Therefore, we can obtain the objective value of the Q-value function under the current optimization state: ; The Critic training network outputs the state-action Q-value function of the current Kalman-filtered optimized state. This is used to evaluate the current policy. To enhance the agent's exploration of the environment, DDPG adds a Gaussian noise function to the behavioral policy. The objective of the Critic training network is defined as: ; A4: Construct a comprehensive action-value hybrid network, such as Figure 3 As shown, the local action value functions of all input agents in the Kalman filter optimized state are merged while maintaining policy consistency: ; And it satisfies the following monotonicity conditions: ; In the formula, The action value function representing the overall action value of the train group is used to fuse the action values ​​of each agent during the training phase to guide the updating of the training network in each train agent. Represents intelligent agents Action value function, The number of trains in the train group; Indicates all train groups State observation set data of each agent, Represents intelligent agents State observation data, Indicates all train groups A joint action strategy for individual agents Represents intelligent agents Action strategy.

[0028] Step B: Based on the current state observation set data, all agents output a joint action decision, and use an action value hybrid network to calculate the total action value function value of all agents.

[0029] When outputting joint action decisions, the joint action decisions are based on a greedy strategy and a limit-limited double exponential smoothing safety strategy: based on the limited safety distance between locomotives and the limitation of the state of charge of batteries in the train hybrid energy storage system, the safety thresholds for the acceleration and fuel cell power of each train are set based on the observed state, and the final joint action decisions are output within the input safety thresholds: the acceleration and fuel cell power of all trains.

[0030] The greedy strategy here refers to selecting the joint action decision that outputs the optimal value (maximum value of the overall action value function of the train group output by the action value hybrid network) while meeting the safety threshold.

[0031] Limiting double exponential smoothing safety strategy: Smooth the actions output by the intelligent agent (such as acceleration changes and power changes) to prevent drastic changes in actions, so as to ensure the smoothness of train operation and the safety of the actuator, and avoid control oscillations caused by signal jitter.

[0032] The system monitors the distance between vehicles in real time and dynamically calculates the allowable range (upper and lower limits) of acceleration and train power, and forces the final output action decision to fall within this range.

[0033] Step C: Each train extracts its own action decision from the joint action decision and controls the locomotive's acceleration and fuel cell stack power, while acquiring the state observation set data after the train group executes the action decision.

[0034] C1: Extract the actions of each train from the joint action decision obtained in step B, and determine the acceleration of the train and the power of the fuel cell stack at the current moment; C2: The speed controller controls the train's traction or braking system to adjust the train's acceleration, and the DC / DC controller outputs a control quantity (MOSFET duty cycle) to the DC / DC converter to adjust the power of the train's fuel cell stack.

[0035] Step D: Based on the state observation set data after the train group executes the action decision, calculate the reward function value of the train group and the total action value function value of all agents; based on the total action value function value of all agents before the train group executes the action decision, and the total action value function value and reward function value after the action decision is executed, update each agent and the action value hybrid network, and return to step B.

[0036] D1: Calculate the reward function, considering the balance between energy efficiency, safety, and comfort during the overall operation of the train group: ; In the formula, and They represent trains and the train behind exist Location at any given moment Indicates train With the train behind Between Reference distance at any given time; and They represent trains With the train behind exist The speed of time; Indicates train exist Acceleration at all times The change Indicates train exist Hydrogen consumption at any given time. This indicates the number of trains in the train group. This indicates the duration of one training epoch. This represents the weighting coefficient.

[0037] Given the chemical energy density of hydrogen (MJ / kg), calculate the hydrogen consumption of each train during fuel cell operation: ; In the formula, Indicates any train at Hydrogen consumption at any given time. Indicates fuel cell efficiency. This indicates that hydrogen has a low calorific value. This indicates that the train's fuel cell stack is in Output power at time: ; In the formula, and The fuel cell stacks of the train are respectively Output voltage and current at any given time; It represents the thermodynamic electromotive force, also known as the Nernst voltage, which is the theoretical highest voltage; This indicates the voltage loss caused by concentration polarization. This indicates the voltage loss caused by the activation polarization effect. This indicates the voltage loss caused by the Ohmic polarization effect.

[0038] D2: Transfer the current state observation set data. and the next time step state observation set data Each agent's available action space at the current and next time steps, the action selected at the current time step, and the reward function value calculated by D1 are stored in the experience replay pool.

[0039] D3: When the experience replay pool size meets the set value, sample data from the experience replay pool and update the parameters of the Critic training network by minimizing the loss value. The loss function for updating the Critic training network is: ; in, Indicates train group State observation set data at time t, Indicates train group Joint action decision-making at any moment Middle intelligence agent Action strategy , Representing intelligent agents Actor training network, Representative input agent State observation data, This represents exploratory noise in action strategies; Indicates the amount of sampled data; The objective value of the action value function in the current optimized state: ; in, Estimate the next-time optimal state in a hybrid network for target action value The Q-value function under the following conditions For each intelligent agent, the target network is... In optimization state The generated joint action strategy; For the train group in status Execute joint action strategy The instant reward received; It is a discount factor; Policy gradient for updating network parameters when training an arbitrary agent (Actor): ; In the formula, Represents intelligent agents Actor training network policy gradient, For intelligent agents The parameters of the Actor-trained network; Represents intelligent agents Critic training network Relative to action gradient, For intelligent agents Parameters of the Critic training network; Represents intelligent agents Actor training network Relative to its parameters The gradient; Update the agent through a soft update mechanism. Critic target network parameters and Actor target network parameters : ; ; In the formula, Indicates the learning rate; D4: Repeat the above steps until all network parameters in the model have been updated. This soft update mechanism ensures that parameters can be updated slowly, thereby improving the stability of learning.

[0040] During parameter updates, the learning rate is adjusted using state estimation uncertainty information provided by Kalman filtering, such as the covariance matrix. This accelerates convergence and avoids local optima. Furthermore, the action value hybrid network employs a hypernetwork, which, based on global state information (i.e., the state observation set), determines the total action value function output by the hybrid network. As close as possible to the score calculated by the reward function J With expected future scores The sum of the values ​​of the training action value hybrid network is used to calculate the weights and offsets of the connections between neurons in each layer, so as to make full use of global state information to improve model performance.

[0041] D4: Repeat the above steps until all network parameters in the model are updated.

[0042] The action value hybrid network and its supernetwork parameters are trained together with the Critic network parameters of each agent. For samples sampled from the experience replay pool, the local action value function is first calculated by the Critic network of each agent. Then the supernetwork uses the global state observation set The weights and biases of the hybrid network are generated, thereby obtaining the overall motion value function of the train group. Then, based on the overall action value function of the target... Construct the time difference loss function: ; By performing gradient backpropagation on the loss function, the Critic network and the action-value hybrid network of each agent are updated, thereby achieving adaptive training of the action-value hybrid network.

[0043] The above embodiments are preferred embodiments of this application. Those skilled in the art can make various changes or improvements based on them. Without departing from the overall concept of this application, these changes or improvements should fall within the scope of protection claimed in this application.

Claims

1. A method for coordinated control of speed and energy of heavy-haul train groups, characterized in that, include: Step A: Construct a speed control agent and an energy management agent based on DDPG for each train, and build a hybrid network of action value shared by all agents; A state observation set and an action set are set, and the state observation set data is obtained by optimizing the data from multiple sensor sources using Kalman filtering; Step B: All agents output joint action decisions based on the current state observation set data; the total action value function value of all agents is calculated using an action value hybrid network; Step C: Each train extracts its own action decision from the joint action decision, controls the locomotive's acceleration and fuel cell stack power, and simultaneously acquires the state observation set data of the train group after executing the action decision; Step D: Based on the state observation set data after the train group makes action decisions, calculate the reward function value of the train group, and then calculate the target value of the total action value function of all agents; based on the total action value function value of all agents and the target value of the action value function, update each agent and the action value hybrid network, and return to step B.

2. The method for coordinated control of speed and energy of heavy-haul train groups according to claim 1, characterized in that, The state observation set covers the kinematic state and powertrain state of each train in the train group; Kalman filtering is used and based on the dynamic model of the train, multi-source sensor data is fused and recursively calculated to obtain the optimal estimate of the state, which is used as the state observation set data for DDPG.

3. The method for coordinated control of speed and energy of heavy-haul train groups according to claim 1, characterized in that, The multi-source sensor data includes data collected by the following sensors: GPS, IMU, wheel speed sensor, current sensor, and voltage sensor.

4. The method for coordinated control of speed and energy of heavy-haul train groups according to claim 1, characterized in that, The action-value hybrid network merges the local action-value functions of all agents while maintaining the consistency of action policies, and is expressed as follows: ； ； In the formula, The action value function representing the overall action value of the train group is used to fuse the action values ​​of each agent during the training phase to guide the updating of the training network in each train agent. Represents intelligent agents Action value function, The number of trains in the train group; Indicates all train groups State observation set data of each agent, Represents intelligent agents State observation data, Indicates all train groups A joint action strategy for individual agents Represents intelligent agents Action strategy.

5. The method for coordinated control of speed and energy of heavy-haul train groups according to claim 4, characterized in that, The action value hybrid network uses a hypernetwork, inputting the state observation set data of the train group, and calculating the weights and offsets of the hybrid network.

6. The method for coordinated control of speed and energy of heavy-haul train groups according to claim 1, characterized in that, When outputting joint action decisions, based on the limited safe distance between locomotives and the limitation of the state of charge of batteries in the train hybrid energy storage system, the safety thresholds for the current acceleration and fuel cell power of each train are set based on the observed state, and the final joint action decision is output under the input safety constraints.

7. The method for coordinated control of speed and energy of heavy-haul train groups according to claim 1, characterized in that, Reward function in step D Taking into account energy efficiency, safety, and comfort, it is expressed as follows: ； In the formula, and They represent trains and the train behind exist Location at any given moment Indicates train With the train behind Between Reference distance at any given time; and They represent trains With the train behind exist The speed of time; Indicates train exist Acceleration at all times The change Indicates train exist Hydrogen consumption at any given time. This indicates the number of trains in the train group. This indicates the duration of one training epoch. This represents the weighting coefficient.

8. The method for coordinated control of speed and energy of heavy-haul train groups according to claim 7, characterized in that, Given the chemical energy density of hydrogen, calculate the hydrogen consumption of each train's fuel cell during operation: ； In the formula, Indicates any train at Hydrogen consumption at any given time. Indicates fuel cell efficiency. This indicates that hydrogen has a low calorific value. This indicates that the train's fuel cell stack is in Output power at time: ； In the formula, and The fuel cell stacks of the train are respectively Output voltage and current at any given time; It represents the thermodynamic electromotive force, also known as the Nernst voltage, which is the theoretical highest voltage; This indicates the voltage loss caused by concentration polarization. This indicates the voltage loss caused by the activation polarization effect. This indicates the voltage loss caused by the Ohmic polarization effect.

9. The method for coordinated control of speed and energy of heavy-haul train groups according to claim 4, characterized in that, Step D includes: D1: Calculate the reward function for the train group; D2: Store the current and next time state observation data, the available action space for each agent at the current and next time, the action selected at the current time, and the reward for the train group into the experience replay pool; D3: When the experience replay pool size meets the set value, sample data from the experience replay pool and update the parameters of the Critic training network by minimizing the loss value. The loss function during Critic network training and updates. for: ； in, Indicates train group State observation set data at time t, Indicates train group Joint action decision-making at any moment Middle intelligence agent Action strategy , Representing intelligent agents Actor training network, Representative input agent State observation data, This represents exploratory noise in action strategies; Indicates the amount of sampled data; The objective value of the action value function in the current optimized state: ； in, Estimate the next-time optimal state in a hybrid network for target action value The Q-value function under the following conditions For each intelligent agent, the target network In optimization state The generated joint action strategy; For the train group in status Execute joint action strategy The instant reward received; It is a discount factor; Policy gradient for updating network parameters when training an arbitrary agent (Actor): ； In the formula, Represents intelligent agents Actor training network policy gradient, For intelligent agents The parameters of the Actor-trained network; Represents intelligent agents Critic training network Relative to action gradient, For intelligent agents Parameters of the Critic training network; Represents intelligent agents Actor training network Relative to its parameters The gradient; Update the agent through a soft update mechanism. Critic target network parameters and Actor target network parameters : ； ； In the formula, Indicates the learning rate; D4: Repeat the above steps until all network parameters in the model are updated.

10. A device for coordinated control of speed and energy of heavy-haul train groups, characterized in that, include: Sensors, memory, processors, and controllers; The sensor is used to sense the information that the train needs to observe and transmits the observation information to the Kalman filter state observer for optimization processing in order to obtain the state observation set data of the train group. The memory stores a computer program, which, when executed by the processor and controller, causes the processor and controller to implement the method for coordinated control of the speed and energy of heavy-haul train groups as described in any one of claims 1 to 9.

Images