Train energy-saving operation control method based on security reinforcement learning and related device

Abstract

Embodiments of the present application provide a train energy-saving operation control method based on safety reinforcement learning and related devices. The method comprises: establishing a distance-discretized train energy-saving operation simulation environment; in order to meet the constraints of speed limits during train operation, in network training, unsafe actions are reset through a safety action reset mechanism; considering the constraints of planned operation time, a relaxed dynamic reward function is designed to guide the continuous evolution of the network to convergence, with the goal of minimizing energy consumption. On real train data and line information, it is verified that the operation control strategy output by the method meets the expectations and can meet the requirements of train planned operation time while achieving the goal of optimal energy consumption. Based on this, the embodiments of the present application can accelerate the training speed of the optimal strategy and ensure the control accuracy of the control strategy for train operation.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of train operation control technology, and in particular to a train energy-saving operation control method and related devices based on safety reinforcement learning. Background Technology

[0002] Urban rail transit, as an efficient, safe, comfortable, and fast mode of transportation, has achieved remarkable development over the past few decades. The core problem of train energy-saving control is to find a train speed distribution that minimizes energy consumption and maintains a consistent timetable, while satisfying constraints such as train characteristics, track gradient, curves, and speed limits. This process is also known as Train Trajectory Optimization (TTO). The TTO problem is comprehensive and complex. PMP methods and nonlinear programming methods face significant difficulties when dealing with hard constraints arising from more complex issues such as track gradients and limits, making it difficult to find the optimal strategy. Traditional RL algorithms, through soft constraints, penalize actions that do not meet the constraints, resulting in low sampling efficiency, slow learning, and even the potential to break the constraints and obtain train trajectories exceeding safety limits. Therefore, existing train control strategies suffer from long computation times and low accuracy when considering planned running time constraints. Summary of the Invention

[0003] This invention provides a train energy-saving operation control method and related device based on safety reinforcement learning, which can accelerate the training speed of the optimal control strategy and improve the control accuracy of the control strategy for train operation.

[0004] In a first aspect, embodiments of the present invention provide a train energy-saving operation control method based on safety reinforcement learning, comprising:

[0005] Based on the longitudinal forces acting on the train during operation, a dynamic model of the train operation is constructed.

[0006] Discretize the train operation route based on the distance dimension so that the optimization of train operation strategy meets the requirements of Markov decision process;

[0007] A Markov model of the train operation control process is constructed based on the train's running time and operating energy consumption.

[0008] Based on the S-A2C reinforcement learning algorithm, actions are generated through a policy network. During the execution of "state-action", the safety of the actions is evaluated by the environment. Unsafe actions are reset midway until the safety constraints of train operation are met.

[0009] According to the optimal strategy, train running time and train running energy consumption are negatively correlated. Relaxing the train planned running time constraint constructs a relaxed sparse reward function.

[0010] Based on the contributions of runtime reward and runtime energy reward to speed guidance under different states, a dynamic dense reward function is constructed.

[0011] A relaxed dynamic reward function is constructed by combining a relaxed sparse reward function and a dynamic dense reward function.

[0012] Optimize train energy-saving operation control strategies based on relaxed dynamic reward functions.

[0013] In some embodiments, the dynamic model of the train operation is represented as:

[0014]

[0015] Where x∈[0,X] is the train travel distance, the train speed is v∈[0,V], t is the train travel time, M is the train mass, and f(v), b(v), w0(v) and w i (x) represent the traction force, braking force, basic train resistance, and additional track resistance experienced by the train, respectively, α f and α b These are the traction and braking force utilization coefficients, 0≤α. f ,α b ≤1.

[0016] In some embodiments, the discretization of the train route based on the distance dimension includes:

[0017] Based on the inflection points of the train track gradient and speed limit, the line is discretized into H large segments, each segment being l in length. h (h=1,2,L,H), with the maximum discrete interval set to Δl, each large segment is discretized as follows: If there are multiple small segments, then the speed limit and gradient of each segment are constants, and the discrete spatial dimension is:

[0018]

[0019] In some embodiments, the Markov decision process is defined as in Representing the state space, Represents the action space, Represents the state transfer function. Let γ represent the reward function, and γ represent the discount factor; in step i, the Agent's state is... Through sampling action Interact with the environment to transition to a new state. i+1 and receive a reward r i The current action taken not only affects the immediate reward but also influences the subsequent state through future rewards; the state is defined as follows: Among them, T i and E i Indicates runtime and energy consumption:

[0020]

[0021] On the interval [0,X], the train's travel time T N ∈[T min ,T max The train's operating energy consumption E N ∈[E min E max The action space can be defined as... MA represents maximum traction operation, CR represents cruise operation, CO represents coasting, and MB represents maximum braking. The objective function is constructed based on the train's running time and energy consumption as follows:

[0022]

[0023] Where π is the train operation control strategy, J E =E N J ΔT =|T N -T P |,T P This indicates the planned travel time of the train within the section.

[0024] In some embodiments, the generation of actions through a policy network, during the execution of a "state-action" process, involves assessing the safety of actions through environmental evaluation, resetting unsafe actions midway, until the safety constraints for train operation are met, including:

[0025] According to the policy network π(·|s) i ;θ now )generate probability distribution Pdist A Based on this, sampling action a i Interacting with the environment to achieve state transitions, using a safety decision coefficient ξ sd Determine whether an operating condition reset is required;

[0026]

[0027] When v(x) > V max (x), then ξ sd When = 1, it means that action a continues.i This would cause the train to exceed its speed limit, which does not meet the requirements for safe train operation, necessitating a reset of the operation; the environment outputs an intermediate state s. i 'and safety judgment coefficient ξ sd To avoid actions a′ after reset i =a i Action sets need to be built. Right now Imminent action a i probability density Simultaneously, the probability distribution of each element in the action space is normalized to obtain a new probability distribution. Action a′ is obtained again based on the reset probability distribution. i , with (s′ i ,a′ i The agent will re-interact with the environment until it meets the safety constraints, causing the agent to transition to a new state. i+1 .

[0028] In some embodiments, the sparse reward function is constructed as follows:

[0029] During train operation, the deviation between the train's running time and the planned running time within a section should be less than the allowable time deviation Δt, i.e., |T N (π)-T P |≤Δt;

[0030] For any optimal train control strategy π * If T N (π * ) = T P ,T P ∈[T min ,T max ], then T P Energy consumption E of the optimal strategy N (π * If ) is negatively correlated, then for Then there is Therefore, the reward function for a train's travel time is defined as:

[0031]

[0032] Through the energy reward function R E Measuring the energy consumption level of train operation:

[0033]

[0034] Based on the train's operating energy consumption, a sparse reward function is constructed:

[0035]

[0036] in, and These are the weighting coefficients for running time and running energy consumption, respectively.

[0037] In some embodiments, the construction process of the dynamic reward function is as follows:

[0038] Based on the train's speed level under different conditions, a dynamic dense reward function is designed as follows:

[0039]

[0040] r t and r e These are concentrated time rewards and energy rewards:

[0041]

[0042]

[0043] The planned average operating speed is This indicates that the train accelerates to MT. Then it's converted to CR, and finally, the energy consumption for MB to complete all tasks is λ. e and λ t These are the dynamic balance coefficients for energy consumption and operating time, respectively.

[0044]

[0045] The weight values ​​are balanced using a linear balancing function Fbal(x).

[0046]

[0047] Therefore, the relaxed dynamic reward function can be defined as:

[0048]

[0049] Secondly, embodiments of the present invention also provide a train energy-saving operation control device based on safety reinforcement learning, the device comprising:

[0050] The analysis module is used to construct a dynamic model of train operation based on the longitudinal forces experienced during train operation.

[0051] The discrete module is used to discretize the train operation route based on the distance dimension, so that the optimization of train operation strategy meets the requirements of Markov decision process;

[0052] The building block is used to construct a Markov model of the train operation control process based on the train's running time and operating energy consumption.

[0053] The action module is used to generate actions through a policy network based on the S-A2C reinforcement learning algorithm. During the execution of the "state-action" process, the safety of the action is evaluated by the environment, and unsafe actions are reset midway until the safety constraints of train operation are met.

[0054] The constraint module is used to relax the train running time constraint and the train running energy consumption according to the optimal strategy to construct a relaxed sparse reward function.

[0055] The guidance module is used to construct a dynamic dense reward function based on the contribution of runtime reward and runtime energy reward to speed guidance under different states;

[0056] The module is used to construct a relaxed dynamic reward function by combining a relaxed sparse reward function and a dynamic dense reward function.

[0057] The optimization module is used to optimize the train energy-saving operation control strategy based on the relaxed dynamic reward function.

[0058] Thirdly, embodiments of the present invention also provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the train energy-saving operation control method based on safety reinforcement learning as described in the first aspect.

[0059] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions for executing the train energy-saving operation control method based on safety reinforcement learning as described in the first aspect.

[0060] According to embodiments of the present invention, a train energy-saving operation control method and related apparatus based on safety reinforcement learning are provided. The train energy-saving operation control method based on safety reinforcement learning includes: constructing a dynamic model of train operation based on the longitudinal force conditions during train operation; discretizing the train operation line based on the distance dimension to ensure that the train operation strategy optimization conforms to the requirements of a Markov decision process; constructing a Markov model of the train operation control process based on the train's running time and energy consumption; generating actions through a policy network based on the S-A2C reinforcement learning algorithm, and in the execution of "state-action," evaluating the safety of actions through the environment, resetting unsafe actions midway until the safety constraints of train operation are met; constructing a relaxed sparse reward function by relaxing the train planned running time constraint according to the loosely negatively correlated attribute of train running time and train running energy consumption under the optimal strategy; constructing a dynamic dense reward function based on the contribution of running time reward and running energy consumption reward to speed guidance under different states; constructing a relaxed dynamic reward function by combining the relaxed sparse reward function and the dynamic dense reward function; and optimizing the train energy-saving operation control strategy based on the relaxed dynamic reward function. Based on this, this invention establishes a distance-discrete train energy-saving operation simulation environment. To meet the speed limit constraints during train operation, a safe action reset mechanism is used during network training to reset unsafe actions. Considering the constraint of planned running time, a relaxed dynamic reward function is designed with the goal of minimizing energy consumption, guiding the network's continuous evolution to convergence. On real train data and track information, the operation control strategy output by the method is verified to meet expectations, satisfying the train's planned running time requirements while achieving the goal of optimal energy consumption. Therefore, this invention can accelerate the training speed of the optimal control strategy and improve the control accuracy of the control strategy for train operation. Attached Figure Description

[0061] Figure 1A This is a flowchart of a train energy-saving operation control method based on safety reinforcement learning provided in one embodiment of the present invention;

[0062] Figure 1B This is a schematic diagram of line discretization provided in one embodiment of the present invention;

[0063] Figure 2 This is a diagram illustrating a secure action reset mechanism provided in one embodiment of the present invention;

[0064] Figure 3 This is a verification case of the line gradient provided in one embodiment of the present invention;

[0065] Figure 4 This is a verification case of the S-A2C algorithm reward curve without temporary rate limits provided in one embodiment of the present invention;

[0066] Figure 5 This is a verification case provided by one embodiment of the present invention, showing the train trajectory without temporary speed limits;

[0067] Figure 6 This is a verification case of a temporary speed limit (1) provided in one embodiment of the present invention;

[0068] Figure 7 This is a verification case of temporary speed limit (2) provided in one embodiment of the present invention;

[0069] Figure 8 This is a verification case of a temporary speed limit (3) provided in one embodiment of the present invention;

[0070] Figure 9 This is a schematic diagram of a train energy-saving operation control device based on safety reinforcement learning provided in one embodiment of the present invention;

[0071] Figure 10 This is a schematic diagram of an electronic device provided in one embodiment of the present invention. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0073] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the following drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0074] In this embodiment of the invention, the terms "furthermore," "exemplarily," or "optionally" are used as examples, illustrations, or descriptions and should not be construed as being more preferred or advantageous than other embodiments or designs. The use of the terms "furthermore," "exemplarily," or "optionally" is intended to present the relevant concepts in a specific manner.

[0075] To facilitate a more convenient description of the working principle of the embodiments of the present invention, the following introduction of relevant technical scenarios is given first.

[0076] Urban rail transit, as an efficient, safe, comfortable, and fast mode of transportation, has achieved remarkable development over the past few decades. The core problem of train energy-saving control is to find a train speed distribution that minimizes energy consumption and maintains a consistent timetable, while satisfying constraints such as train characteristics, track gradient, curves, and speed limits. This process is also known as Train Trajectory Optimization (TTO). The TTO problem is comprehensive and complex. PMP methods and nonlinear programming methods face significant difficulties when dealing with hard constraints arising from more complex issues such as track gradients and limits, making it difficult to find the optimal strategy. Traditional RL algorithms, through soft constraints, penalize actions that do not meet the constraints, resulting in low sampling efficiency, slow learning, and even the potential to break the constraints and obtain train trajectories exceeding safety limits. Therefore, existing train control strategies suffer from long computation times and low accuracy when considering planned running time constraints.

[0077] Based on this, the present invention provides a train energy-saving operation control method and related apparatus based on safety reinforcement learning. The train energy-saving operation control method based on safety reinforcement learning includes: constructing a dynamic model of train operation based on the longitudinal force conditions during train operation; discretizing the train operation line based on the distance dimension to ensure that the train operation strategy optimization conforms to the requirements of a Markov decision process; constructing a Markov model of the train operation control process based on the train's running time and energy consumption; generating actions through a policy network based on the S-A2C reinforcement learning algorithm, and in the execution of "state-action," evaluating the safety of actions through the environment, resetting unsafe actions midway until the safety constraints of train operation are met; constructing a relaxed sparse reward function by relaxing the train planned running time constraint according to the loosely negatively correlated attribute of train running time and train running energy consumption under the optimal strategy; constructing a dynamic dense reward function based on the contribution of running time reward and running energy consumption reward to speed guidance under different states; constructing a relaxed dynamic reward function by combining the relaxed sparse reward function and the dynamic dense reward function; and optimizing the train energy-saving operation control strategy based on the relaxed dynamic reward function. Based on this, this invention establishes a distance-discrete train energy-saving operation simulation environment. To meet the speed limit constraints during train operation, a safe action reset mechanism is used during network training to reset unsafe actions. Considering the constraint of planned running time, a relaxed dynamic reward function is designed with the goal of minimizing energy consumption, guiding the network's continuous evolution to convergence. On real train data and track information, the operation control strategy output by the method is verified to meet expectations, satisfying the train's planned running time requirements while achieving the goal of optimal energy consumption. Therefore, this invention can accelerate the training speed of the optimal control strategy and improve the control accuracy of the control strategy for train operation.

[0078] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0079] like Figure 1A As shown, Figure 1A This is a flowchart of a train energy-saving operation control method based on safety reinforcement learning provided in an embodiment of the present invention. The train energy-saving operation control method based on safety reinforcement learning may include, but is not limited to, steps S101 to S108.

[0080] Step S101: Construct a dynamic model of the train operation based on the longitudinal force conditions during train operation;

[0081] Step S102: Discretize the train operation route based on the distance dimension to make the train operation strategy optimization conform to the requirements of the Markov decision process;

[0082] Step S103: Construct a Markov model of the train operation control process based on the train's running time and operating energy consumption;

[0083] Step S104: Based on the S-A2C reinforcement learning algorithm, actions are generated through the policy network. During the execution of "state-action", the safety of the actions is evaluated by the environment. Unsafe actions are reset midway until the safety constraints of train operation are met.

[0084] Step S105: According to the optimal strategy, the train running time and train running energy consumption are not negatively correlated. Relax the train planned running time constraint to construct a relaxed sparse reward function.

[0085] Step S106: Based on the contributions of runtime reward and runtime energy reward to speed guidance under different states, construct a dynamic dense reward function;

[0086] Step S107: Construct a relaxed dynamic reward function by combining a relaxed sparse reward function and a dynamic dense reward function;

[0087] Step S108: Optimize the train energy-saving operation control strategy based on the relaxed dynamic reward function.

[0088] It is understandable that establishing a system model includes the following:

[0089] 1.1 Train Dynamics Model: Based on the longitudinal force conditions during train operation, a dynamics model of train operation is constructed to optimize the train operation control strategy.

[0090] 1.2 Track Discretization: The track is discretized from the distance dimension to ensure that train operation strategy optimization conforms to the requirements of the Markov decision process. Based on the inflection points of track gradient and speed limit, the speed limit and gradient of each discretized section are kept constant.

[0091] 1.3 Establish a Markov model for the train operation control process.

[0092] It is understandable that secure reinforcement learning models include the following:

[0093] 2.1 Safe Action Reset Mechanism: Based on the A2C (advance actor-critic) algorithm, a safe reinforcement learning algorithm S-A2C (safe advance actor-critic) is proposed. Actions are generated through a policy network. During the execution of "state-action", the safety of the action is evaluated by the environment. Unsafe actions are reset midway until the safety constraints of train operation are met. This ensures that the "state-action" is always within the safe boundary of the environment during the algorithm iteration process, which can better adapt to the safety requirements of the train operation process and improve the efficiency of algorithm sampling.

[0094] 2.2 Relaxed Dynamic Reward Function: The relaxed dynamic reward mechanism combines relaxed sparse rewards and dynamic dense rewards. Based on the fact that the train running time and train running energy consumption of the optimal strategy are loosely negatively correlated, a relaxed sparse reward function is constructed by relaxing the train plan running time constraint. According to the contribution of running time reward and running energy consumption reward to speed guidance under different states, a dynamic dense reward is constructed to improve the adoption efficiency and convergence speed of the algorithm.

[0095] 2.3 Update process of S-A2C algorithm: The S-A2C algorithm updates the Agent's state and reward according to 2.1 and 2.2, and then updates the value network parameter ω and policy network parameter θ according to the A2C update process.

[0096] This invention establishes a distance-discrete train energy-saving operation simulation environment. To meet the speed limit constraints during train operation, a safety action reset mechanism is used during network training to reset unsafe actions. Considering the constraint of planned running time, a relaxed dynamic reward function is designed with the goal of minimizing energy consumption, guiding the network's continuous evolution to convergence. On real train data and track information, the operation control strategy output by the method is verified to meet expectations, satisfying the train's planned running time requirements while achieving the goal of optimal energy consumption. Based on this, the embodiments of this invention can accelerate the training speed of the optimal control strategy and improve the control accuracy of the control strategy for train operation.

[0097] Compared with existing technologies, the present invention has at least the following beneficial effects:

[0098] (1) Based on the traditional reinforcement learning algorithm, this invention proposes a safe reinforcement learning method and adopts a safe action reset mechanism. It can reset unsafe actions by evaluating the safety of the "state-action" pair throughout the state transition process, so as to ensure that the train state always meets the safety boundary constraints of the environment during and at the end of the learning process, thus ensuring the safety of the strategy and improving the sampling efficiency of the algorithm.

[0099] (2) This invention proposes a relaxed dynamic reward mechanism that combines relaxed sparse rewards and dynamic dense rewards to balance the potential convergence stability problems that may arise from safe action reset mechanisms. The relaxed sparse reward is obtained through an extended train schedule running time constraint, making it easier for the learning system to obtain samples that satisfy this constraint and greatly reducing the risk of the algorithm getting trapped in local optima. The dynamic dense reward is a dynamic balancing coefficient based on the initial velocity of the state, used to balance the contribution of running time and energy consumption to obtaining rewards under different states.

[0100] The following description, in conjunction with the accompanying drawings and specific embodiments, further illustrates the train energy-saving operation control method based on safety reinforcement learning of the present invention:

[0101] I. System Model

[0102] 1.1 Train Dynamics Model

[0103] In train operation control optimization, the first step is to define the dynamic model of train operation, which can be expressed as:

[0104]

[0105] Where x∈[0,X] is the train travel distance; the train speed is v∈[0,V]; t is the train travel time; M is the train mass; f(v), b(v), w0(v) and w i (x) represent the traction force, braking force, basic train resistance, and additional track resistance experienced by the train, respectively. α f and α b These are the traction and braking force utilization coefficients, 0≤α. f ,α b ≤1.

[0106] The energy consumption of a train's operation can be described as follows:

[0107]

[0108] 1.2 Line Discretization

[0109] To meet the requirements of Markov processes, the train operation control process is discretized from the distance dimension, such as... Figure 1B As shown, based on the inflection points of the track gradient and speed limit, the line is discretized into H large segments, each segment being l in length. h To ensure accuracy, each large segment (h = 1, 2, L, H) needs further discretization. Setting the maximum discretization interval to Δl, each large segment can then be discretized into... If there are multiple small segments, then the speed limit and gradient of each segment are constants. The discrete spatial dimension is:

[0110]

[0111] 1.3 Markov Model for Train Operation Control

[0112] After discretizing the track according to the unequal distance discretization rule, the train state is defined as being related to its position. During train operation, the position, speed, energy consumption, etc. of the train in a certain state are only related to the previous state and the actions taken, and are independent of other states. Therefore, the train operation control process can be represented as a multi-stage Markov decision process (MDP). The more stages there are, the closer this process is to a continuous train operation control process.

[0113] Markov processes are classic sequential decision processes and can be defined as follows: in Representing the state space; Represents the action space; Represents the state transfer function; Let represent the reward function; γ represents the discount factor. In step i, the Agent's state is... Through sampling action Interact with the environment to transition to a new state. i+1 and receive a reward r i The current action not only affects the immediate reward but also influences the subsequent state through future rewards. This state can be defined as... Among them, T i and E i Indicates runtime and energy consumption:

[0114]

[0115] On the interval [0,X], the train's travel time T N ∈[T min ,T max The train's operating energy consumption E N ∈[E min E max The action space can be defined as... MA represents maximum traction operation, CR represents cruise operation, CO represents coasting, and MB represents maximum braking. The objective function is constructed based on the train's running time and energy consumption as follows:

[0116]

[0117] Where π is the train operation control strategy, J E =E N J ΔT =|T N -T P |,TP This indicates the planned travel time of the train within the section.

[0118] II. Secure Reinforcement Learning Model

[0119] 2.1 A2C Algorithm Flow

[0120] The A2C algorithm consists of two networks: a value network (commentator) and a policy network (actor), which can be defined by parameters ω. now and θ now Definition. The strategy function generates a deterministic train operation control strategy, i.e., a. i :π(·|s i ;θ now To stabilize the training process and avoid overfitting, the parameter ω is used. new and θ new Define and update the target network.

[0121] When updating the value network with the goal of minimizing loss, the loss function and its gradient are defined as follows:

[0122]

[0123]

[0124] Among them, y i =r i +γ·Q(s i+1 ;ω), the TD error used in the network learning process is defined as:

[0125] δ i =r i +γ·Q(s i+1 ;ω)-Q(s i ;ω) (8)

[0126] Update the value network and policy network:

[0127]

[0128]

[0129] 2.2 Secure Action Reset Mechanism

[0130] In traditional reinforcement learning, the agent is in state s i Use action a i Interact with the environment and transition to a new state. i+1 And through reward r i The process of evaluating this state transition is denoted as... However, the environment is often bounded. To ensure that each state transition satisfies the environmental boundaries, a penalty is imposed on state-action pairs that exceed the environmental boundaries, while the state remains unchanged. This approach, known as the soft-constraint state preservation mechanism, is used to correct unsafe states. This trial-and-error method of collecting data to obtain the optimal strategy in practice incurs significant time costs and may even harm the learning system.

[0131] Under normal circumstances, to ensure the train is in section [x] i ,x i+1 To ensure safe operation, it must always meet the following security constraints:

[0132] v(x) <V max (x)x∈[x i ,x i+1 (11)

[0133] To address this issue, we propose an S-A2C (safe advance actor critic) mechanism. Its core idea is to construct a safe action reset mechanism that resets actions when unsafe actions occur, ensuring the effectiveness of each exploration. Figure 2 As shown. Specifically as follows:

[0134] According to the policy network π(·|s) i ;θ now )generate probability distribution Pdist A Based on this, sampling action a i Interacting with the environment to achieve state transitions, using a safety decision coefficient ξ sd Determine whether an operating condition reset is required.

[0135]

[0136] When v(x) > V max (x), then ξ sd When = 1, it means that action a continues. i This would cause the train to exceed its speed limit, which does not meet the requirements for safe train operation, necessitating a reset of the action; the environment outputs an intermediate state s′. i and safety judgment coefficient ξ sd To avoid actions a′ after reset i =a i Action sets need to be built. Right now Imminent action a i probability density Simultaneously, the probability distribution of each element in the action space is normalized to obtain a new probability distribution. Action a′ is obtained again based on the reset probability distribution. i , with (s′ i ,a′ i The agent will re-interact with the environment until it meets the safety constraints, causing the agent to transition to a new state. i+1 .

[0137] 2.3 Relaxed Dynamic Reward Mechanism

[0138] During train operation, the deviation between the train's running time and the planned running time within a section should be less than the allowable time deviation Δt, i.e., |T N (π)-T P |≤Δt.

[0139] For any optimal train control strategy π * If T N (π * ) = T P ,T P ∈[T min ,T max ], then T P Energy consumption E of the optimal strategy N (π * ) is negatively correlated. Therefore, for Then there is Therefore, the reward function for a train's travel time can be defined as:

[0140]

[0141] Through the energy reward function R E Measuring the energy consumption level of train operation:

[0142]

[0143] Based on the train's operating energy consumption, a sparse reward function is constructed:

[0144]

[0145] in and These are the weighting coefficients for running time and running energy consumption, respectively.

[0146] Sparse rewards are inefficient in guiding algorithm evolution, while dense rewards can effectively accelerate the evolution process. Therefore, we designed a dynamic dense reward function based on the speed level of the train in different states:

[0147]

[0148] rt and r e These are concentrated time rewards and energy rewards:

[0149]

[0150]

[0151] The planned average operating speed is This indicates that the train accelerates to MT. Then it's converted to CR, and finally MB completes the entire task based on energy consumption. λ e and λ t These are the dynamic balance coefficients for energy consumption and operating time, respectively.

[0152]

[0153] Use a linear balancing function Fbal(x) to balance its weight values.

[0154]

[0155] Therefore, the relaxed dynamic reward function can be defined as:

[0156]

[0157] III. Results and Analysis

[0158] To verify the effectiveness of the train operation control strategy optimization based on the S-A2C algorithm, this invention uses the Yizhuang Line of the Beijing Subway, from Jiugong Station to Yizhuangqiao Station, as an example, and conducts simulation experiments using a Type B train. The line gradient is as follows: Figure 3 As shown in Table 1, the fixed speed limit data for the line is shown in Table 2, and the relevant parameters of the algorithm are shown in Table 2.

[0159] Table 1 Fixed Speed ​​Limits for the Line

[0160]

[0161] Table 2 Algorithm-related parameter settings

[0162]

[0163] On sections without temporary speed limits, the algorithm's reward curve is as follows: Figure 4 As shown, the S-A2C calculation's reward convergence curve is smooth, effectively avoiding interference from unsafe actions. The convergence process is stable with minimal fluctuations. The train's trajectory is as follows: Figure 5As shown, the S-A2C algorithm exhibits good continuity and smoothness in its actions, without the additional energy loss caused by frequent transitions. In the early stages, it rapidly increases train speed using the MT (Medium-Temperature) mode, then adjusts the speed using MT, CR (Crew-Return), and CO (Cool-Return) modes to maintain it at a reasonable level. Finally, when transitioning to the MB (Medium-Traction) mode, the speed is less than 53 km / h, meaning that the braking process consumes less energy, with traction energy primarily used to overcome basic resistance and additional track resistance. Its running time is 129.99 (s), running energy consumption is 75501 (J), and running time deviation is 0.01 (s).

[0164] Figures 6 to 8 The train's running trajectories on three lines with different temporary speed limits show that they exhibit similar characteristics. Before the start of the temporary speed limit, the train first uses manual (MT) deceleration, then automatic (CO) deceleration. As it approaches the temporary speed limit, the speed decreases to near the limit, minimizing energy loss due to steering and braking, thus meeting the temporary speed limit constraints and ensuring train safety. The corresponding train running times for these three trajectories are 129.97 s, 129.09 s, and 129.71 s, with running time deviations of 0.03 s, 0.91 s, and 0.29 s, respectively. Therefore, the S-A2C can achieve satisfactory train running trajectories under different speed limit conditions, demonstrating strong adaptability.

[0165] Compared with existing technologies, the present invention has at least the following beneficial effects:

[0166] (1) Based on the traditional reinforcement learning algorithm, this invention proposes a safe reinforcement learning method and adopts a safe action reset mechanism. It can reset unsafe actions by evaluating the safety of the "state-action" pair throughout the state transition process, so as to ensure that the train state always meets the safety boundary constraints of the environment during and at the end of the learning process, thus ensuring the safety of the strategy and improving the sampling efficiency of the algorithm.

[0167] (2) This invention proposes a relaxed dynamic reward mechanism that combines relaxed sparse rewards and dynamic dense rewards to balance the potential convergence stability problems that may arise from safe action reset mechanisms. The relaxed sparse reward is obtained through an extended train schedule running time constraint, making it easier for the learning system to obtain samples that satisfy this constraint and greatly reducing the risk of the algorithm getting trapped in local optima. The dynamic dense reward is a dynamic balancing coefficient based on the initial velocity of the state, used to balance the contribution of running time and energy consumption to obtaining rewards under different states.

[0168] In addition, such as Figure 9 As shown, one embodiment of the present invention also discloses a train energy-saving operation control device based on safety reinforcement learning, the device comprising:

[0169] Analysis module 110 is used to construct a dynamic model of train operation based on the longitudinal force conditions during train operation.

[0170] Discrete module 120 is used to discretize the train operation line based on the distance dimension, so that the optimization of train operation strategy meets the requirements of Markov decision process;

[0171] Module 130 is used to construct a Markov model of the train operation control process based on the train's running time and operating energy consumption.

[0172] Action module 140 is used to generate actions through a policy network based on the S-A2C reinforcement learning algorithm. During the execution of "state-action", the safety of the action is evaluated by the environment. Unsafe actions are reset midway until the safety constraints of train operation are met.

[0173] Constraint module 150 is used to relax the train running time constraint and construct a relaxed sparse reward function according to the attribute that the train running time and train running energy consumption are negatively correlated according to the optimal strategy.

[0174] The guidance module 160 is used to construct a dynamic dense reward function based on the contribution of runtime reward and runtime energy reward to speed guidance under different states.

[0175] Module 170 is used to construct a relaxed dynamic reward function by combining a relaxed sparse reward function and a dynamic dense reward function.

[0176] Optimization module 180 is used to optimize train energy-saving operation control strategies based on a relaxed dynamic reward function.

[0177] The train energy-saving operation control device based on safety reinforcement learning in this embodiment of the invention is used to execute the train energy-saving operation control method based on safety reinforcement learning in the above embodiment. Its specific processing procedure is the same as that of the train energy-saving operation control method based on safety reinforcement learning in the above embodiment, and will not be described in detail here.

[0178] In addition, such as Figure 10 As shown, one embodiment of the present invention also discloses an electronic device, including: at least one processor 210; at least one memory 220 for storing at least one program; when the at least one program is executed by the at least one processor 210, it implements the train energy-saving operation control method based on safety reinforcement learning as in any of the preceding embodiments.

[0179] In addition, one embodiment of the present invention discloses a computer-readable storage medium storing computer-executable instructions for executing the train energy-saving operation control method based on safety reinforcement learning as described in any of the preceding embodiments.

[0180] The system architecture and application scenarios described in the embodiments of this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of system architecture and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0181] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0182] In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0183] The terms “component,” “module,” “system,” etc., used in this specification are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process or execution thread, and components may be located on a single computer or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, or a network, such as the Internet interacting with other systems via signals).

Claims

1. A train energy-saving operation control method based on safety reinforcement learning, comprising: Based on the longitudinal forces acting on the train during operation, a dynamic model of the train operation is constructed. Discretize the train operation route based on the distance dimension so that the optimization of train operation strategy meets the requirements of Markov decision process; A Markov model of the train operation control process is constructed based on the train's running time and energy consumption. Based on the S-A2C reinforcement learning algorithm, actions are generated through a policy network. During the execution of "state-action", the safety of the actions is evaluated by the environment. Unsafe actions are reset midway until the safety constraints of train operation are met. According to the optimal strategy, train running time and train running energy consumption are negatively correlated. Relaxing the train planning running time constraint constructs a relaxed sparse reward function. Based on the contributions of runtime reward and runtime energy reward to speed guidance under different states, a dynamic dense reward function is constructed. A relaxed dynamic reward function is constructed by combining a relaxed sparse reward function and a dynamic dense reward function. Optimize train energy-saving operation control strategy based on relaxed dynamic reward function; The construction process of the sparse reward function is as follows: During train operation, the deviation between the train's travel time within a section and the planned travel time should be less than the allowable time deviation. ,Right now , It is a train operation control strategy; For any optimal train control strategy ,if ,but Energy consumption of the optimal strategy It is a negative correlation, then for Then there is Therefore, the reward function for a train's running time is defined as: Through energy consumption reward function Measuring the energy consumption level of train operation: Based on the train's operating energy consumption, a sparse reward function is constructed: in, and These are the running time and the running energy consumption weighting coefficient, respectively, and the train's running energy consumption. ; The construction process of the dynamic reward function is as follows: Based on the train's speed level under different conditions, a dynamic dense reward function is designed as follows: and These are concentrated time rewards and energy rewards: The planned average operating speed is , This indicates that the train accelerates to MA. Then it is converted to CR, and finally the energy consumption of MB to complete all tasks. and These are the dynamic balance coefficients for energy consumption and operating time, respectively. MA represents maximum traction operation, CR represents cruise operation, and MB represents maximum braking. Using a linear equilibrium function To balance their weight values, Therefore, the relaxed dynamic reward function can be defined as: 。 2. The method according to claim 1, characterized in that, The dynamic model of the train operation is represented as follows: in, The train travels a certain distance, and the train's speed is [missing information]. t represents the train travel time, and M represents the train mass. , , and These represent traction force, braking force, basic train resistance, and additional track resistance experienced by the train, respectively. and These are the traction and braking force utilization coefficients, respectively. .

3. The method according to claim 2, characterized in that, The discretization of train operation lines based on the distance dimension includes: Based on the inflection points of the train track gradient and speed limit, the line is discretized into... There are 1 major sections, each section being 1000 long. Set the maximum discrete interval to Then each large segment is discretized into If there are multiple small segments, then the speed limit and gradient of each segment are constants, and the discrete spatial dimension is: 。 4. The method according to claim 3, characterized in that, The Markov decision process is defined as follows: ,in Representing the state space, Represents the action space. Represents the state transfer function. Represents the reward function, Represents the discount factor; in step i, the Agent's state is Through sampling action Interact with the environment to transition to a new state and receive a reward The current action taken not only affects the immediate reward but also influences the subsequent state through future rewards; the state is defined as follows: ,in, and Indicates runtime and energy consumption: In the interval Above, the train's travel time Train operating energy consumption The action space can be defined as MA represents maximum traction operation, CR represents cruise operation, CO represents coasting, and MB represents maximum braking. The objective function is constructed based on the train's running time and energy consumption as follows: in, It is a train operation control strategy. , , This indicates the planned travel time of the train within the section.

5. The method according to claim 4, characterized in that, The process of generating actions through a policy network, and in the execution of "state-action," assessing the safety of actions through environmental evaluation, resetting unsafe actions midway until the safety constraints of train operation are met, includes: According to the policy network generate probability distribution Based on this, sampling action Interacting with the environment to achieve state transitions, and using safety decision coefficients. Determine whether an operating condition reset is required; when ,but When, it indicates that the action will continue. This would cause the train to exceed its speed limit, which does not meet the requirements for safe train operation, necessitating a reset of the operation; the environment outputs an intermediate state. and safety judgment coefficient To avoid actions after reset Action sets need to be built. ,Right now Imminent action probability density At the same time, the probability distribution of each element in the action space is normalized to obtain a new probability distribution. The action is obtained by re-applying the probability distribution after the reset. ,by The agent will re-interact with the environment until it meets security constraints, causing it to transition to a new state. .

6. A train energy-saving operation control device based on safety reinforcement learning, characterized in that, The device executes the train energy-saving operation control method based on safety reinforcement learning as described in any one of claims 1 to 5, and the device comprises: The analysis module is used to construct a dynamic model of train operation based on the longitudinal forces experienced during train operation. The discrete module is used to discretize the train operation route based on the distance dimension, so that the optimization of train operation strategy meets the requirements of Markov decision process; The building block is used to construct a Markov model of the train operation control process based on the train's running time and operating energy consumption. The action module is used to generate actions through a policy network based on the S-A2C reinforcement learning algorithm. During the execution of "state-action", the safety of the action is evaluated by the environment. Unsafe actions are reset midway until the safety constraints of train operation are met. The constraint module is used to relax the train running time constraint and the train running energy consumption according to the optimal strategy to construct a relaxed sparse reward function. The guidance module is used to construct a dynamic dense reward function based on the contribution of runtime reward and runtime energy reward to speed guidance under different states; The module is used to construct a relaxed dynamic reward function by combining a relaxed sparse reward function and a dynamic dense reward function. The optimization module is used to optimize the train energy-saving operation control strategy based on the relaxed dynamic reward function.

7. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the train energy-saving operation control method based on safety reinforcement learning as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing computer-executable instructions for performing the train energy-saving operation control method based on safety reinforcement learning as described in any one of claims 1 to 5.

Images