Train tracking operation cooperative optimization control method and device

Abstract

The application discloses a train tracking operation cooperative optimization control method and device based on safety reinforcement learning, and the method comprises the following steps: obtaining adjacent train operation state data of adjacent front and rear vehicles and train operation state data of the current vehicle; calculating safety braking distances of the adjacent trains under the most unfavorable working conditions, and obtaining minimum dynamic safety tracking intervals of the current adjacent train pair; forming a target tracking interval, calculating actual net intervals of the adjacent front and rear vehicles, calculating interval errors, and obtaining speed cooperation errors; constructing a multi-agent constraint Markov decision model; constructing a dynamic constraint multi-agent joint strategy optimization, and updating a shared strategy network during centralized training; and deploying the trained dynamic constraint multi-agent joint strategy optimization in vehicle-mounted control equipment of each tracking train, so as to realize train tracking operation cooperative optimization control. In the embodiment of the application, the cooperative tracking control of multiple trains under the conditions of safety, tightness and stability is realized.

Description

Technical Field

[0001] This invention relates to the field of big data technology, and in particular to a train tracking operation collaborative optimization control method and device based on security reinforcement learning. Background Technology

[0002] Existing train tracking operation control is usually based on moving block, virtual train formation (VCTS) and automatic train control system. It obtains information such as train position, speed and track speed limit through vehicle-to-ground communication or vehicle-to-vehicle communication, and generates traction, coasting and braking control commands based on safe braking distance or target tracking interval. Such schemes can reduce train tracking interval to a certain extent, but most methods still mainly rely on fixed safety margin, preset running curve or local closed-loop control law.

[0003] The first type of technical solution in the prior art is a train tracking control method based on rules, PID / fuzzy PID or model predictive control. This type of method usually calculates the current tracking error based on the position of the preceding train, the position of the following train, the train length and the safety distance, and then the controller outputs acceleration or traction / braking control quantities based on the speed error and the spacing error. If model predictive control is used, the speed, acceleration, speed limit and safety distance are written into the constraints in the rolling optimization window, and the control input is obtained by solving the optimization problem online.

[0004] The second type of technical solution is a single-train or multi-train tracking control method based on reinforcement learning. This type of method treats the train's operating state as the agent's observation, traction / braking commands as actions, and punctuality, energy saving, speed tracking, or safe distance penalties as reward functions. The control strategy is trained through DQN, DDPG, SAC, PPO, or independent multi-agent learning methods. This type of solution has the ability to handle nonlinear control problems, but safety constraints are often indirectly reflected in the form of rewards and penalties or fixed thresholds.

[0005] The third type of technical solution is the multi-agent cooperative control method, such as using independent learning, centralized training and distributed execution, or multi-agent proximal policy optimization framework to handle the multi-train queue control problem. This type of solution can enhance the cooperative learning ability among multiple agents, but conventional MAPPO or IPPO usually do not uniformly incorporate the minimum dynamic safe tracking interval of trains, system-level safety cost, dynamic cost threshold and Lagrange constraint adjustment into the policy update process.

[0006] Therefore, this technical solution also needs to address how to unify the modeling of train longitudinal dynamics, minimum dynamic safety tracking interval, and multi-vehicle collaborative decision-making under vehicle-to-vehicle communication conditions; how to incorporate dynamic safety constraints into multi-agent policy optimization in the form of system-level safety costs; how to improve training stability and safety constraint satisfaction through dynamic cost thresholds and Lagrange multiplier adjustment mechanisms; and how to utilize global information to improve collaborative learning effects during the training phase while maintaining distributed online control during the execution phase. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art. This invention provides a train tracking operation cooperative optimization control method and device based on safety reinforcement learning, which realizes cooperative tracking control of multiple trains under safe, close and stable conditions.

[0008] To address the aforementioned technical problems, embodiments of the present invention provide a train tracking operation cooperative optimization control method based on safety reinforcement learning, the method comprising: During each control cycle, each tracking train obtains the operating status data of adjacent trains (both preceding and following) and its own operating status data based on vehicle-to-vehicle communication. The adjacent train operating status data includes train configuration information, speed information, and acceleration information, while the own train operating status data includes its own position information, speed information, and acceleration information. Based on the operating status data of adjacent trains and the operating status data of this train, the safe braking distance of adjacent trains under the most unfavorable operating conditions is calculated, and the minimum dynamic safe tracking interval of the current adjacent train pair is obtained by combining the basic safety margin. The target tracking interval is formed by superimposing the minimum dynamic safety tracking interval with an additional tracking margin. The actual net distance between the front and rear trains is calculated based on the position information in the adjacent train operation status data and the speed difference between the tracking train and the preceding train. The distance error between the actual net distance and the target tracking interval is calculated, and the speed coordination error is obtained based on the speed difference between the tracking train and the preceding train. A multi-agent constrained Markov decision model is constructed based on the train's operating status data, adjacent train operating status data, minimum dynamic safe tracking interval, target tracking interval, spacing error, and speed coordination error. Based on the multi-agent constrained Markov decision model, a shared policy network is used to construct a dynamic constrained multi-agent joint policy optimization, and during the centralized training of the dynamic constrained multi-agent joint policy optimization, a proximal policy optimization is used to prune the target and update the shared policy network. The trained dynamic constraint multi-agent joint strategy is optimized and deployed in the onboard control equipment of each tracking train to achieve collaborative optimization control of train tracking operation.

[0009] Optionally, the process by which each tracking train obtains the adjacent train operation status data of the adjacent preceding and following trains and its own operation status data based on vehicle-to-vehicle communication includes: Let the first The train was at The position, velocity, and acceleration information for each control cycle are respectively... , and The control cycle is The discrete state update relation of the train is then expressed as: ; ; In the formula, For the first The train was at The actual longitudinal acceleration within each control cycle; and this actual longitudinal acceleration is obtained after constraining the train's traction capacity, braking capacity, and track speed limit. During train operation, the following speed constraints and control input constraints are satisfied: ; ; In the formula, Speed ​​limits were imposed on the line. For the first The train was at Longitudinal control commands within a control cycle and To control the lower and upper limits of the input.

[0010] Optionally, the step of calculating the safe braking distance of adjacent trains under the most unfavorable operating conditions based on the operating status data of adjacent trains and the operating status data of the current train, and calculating the minimum dynamic safe tracking interval of the current adjacent train pair in combination with the basic safety margin, includes: Assuming the train continues to accelerate at maximum traction during the response and traction cut-off phases, the duration of the traction cut-off phase is defined as follows: The corresponding running distance Represented as: ; In the formula, The initial velocity at the moment of braking triggering; The maximum traction acceleration maintained during the response phase; Speed ​​at the end of the traction shearing phase for: ; After the traction force is cut off, the train enters a coasting transition phase until the braking system establishes effective braking force. The train then naturally decelerates due to running resistance. Let the time of the coasting transition phase be . The running distance corresponding to the coasting transition phase is... Represented as: ; In the formula, Let be the equivalent natural deceleration of the train under the action of running resistance; then the speed at the end of the coasting transition phase. for: ; After the braking force is fully established, the train enters full braking mode and decelerates at maximum emergency braking speed. The braking distance during the deceleration process until the vehicle comes to a complete stop is... Represented as: ; No. Tracking the safe braking distance of the train under the most unfavorable operating conditions for: ; At any moment , No. The train suddenly applied maximum emergency braking, for the first The train, at its time Immediately enter maximum emergency braking state, therefore the safe braking distance of the vehicle in front can be expressed as: ; In the formula, The current speed of the vehicle in front; This represents the maximum emergency braking deceleration value of the vehicle in front. For the A following train may maintain its original traction during the delay phase before entering maximum emergency braking; therefore, the safe braking distance of the following train is crucial. , further expressed as: ; In the formula, For the first The system tracks the current speed of the train. Then the first The train's locomotive and the first Basic safety margin between the rear ends of the train Then the first Train tracking at time The corresponding minimum dynamic safety tracking interval is expressed as: .

[0011] Optionally, the step of forming the target tracking interval by superimposing an additional tracking margin on the minimum dynamic safety tracking interval includes: Definition of the first The train was being tracked. The actual distance at any given time always satisfies the following: ; To prevent the tracking train from running close to the dynamic safety boundary for extended periods, an additional tracking margin is introduced on top of the minimum dynamic safety tracking interval. , define the first Train tracking at time The target tracking interval is: ; in, For the first Train tracking at time Target tracking interval; For the first Train tracking at time Corresponding minimum dynamic safety tracking interval; additional tracking margin .

[0012] Optionally, the step of calculating the actual net distance between the trains based on the position information in the adjacent train operation status data and the speed difference between the tracking train and the preceding train includes: For any i A train is being tracked, and the train directly ahead of it is the 1st train. The train will be the first The actual clearance between the following train and the train in front is: ; In the formula, For the first Within the first control cycle, the first The actual net distance between the tracking train and the train in front; The length of the vehicle in front; and These are the location information for the vehicle in front and the vehicle itself.

[0013] Optionally, the step involves calculating the spacing error between the actual net distance and the target tracking interval, and obtaining the speed coordination error based on the speed difference between the tracking train and the preceding train. For any adjacent train pairs The spacing error is defined as: ; In the formula, For any adjacent train pairs exist The time interval error; For the first The train was being tracked. The actual distance at any given moment remains constant; For the first The train was being tracked. The target tracking interval at any given time; No. Train tracking speed coordination error relative to the preceding train for: ; In the formula, For the first The train was being tracked. Speed ​​at any given moment; For the car in front Speed ​​at any given moment; The objective of multi-train cooperative tracking control can be described as: under the premise of satisfying dynamic safety constraints, gradually reducing the distance error and speed coordination error between adjacent trains, i.e.: .

[0014] Optionally, the construction of a multi-agent constrained Markov decision model based on the train's operating status data, adjacent train operating status data, minimum dynamic safe tracking interval, target tracking interval, spacing error, and speed coordination error includes: Based on the train's operating status data, adjacent train operating status data, minimum dynamic safe tracking interval, target tracking interval, spacing error, and speed coordination error, the multi-train cooperative tracking control problem is modeled as a multi-agent constrained Markov decision model as follows: ; in, For each intelligent agent's local observation space; For the centralized training phase, the global joint state space is defined. For joint action space; This represents a state transition relationship; For collaborative reward functions; For safety cost function; Discount factor; During the online execution phase, each tracking train makes distributed decisions based solely on its own operational status data and the operational status data of the vector train; for the train in the middle of the queue... The partial observation of the train tracking is represented as follows: ; In the formula, and The first The train tracks the actual distance between itself and the trains in front and behind it; , and These are the speeds of the vehicle in front, the vehicle itself, and the vehicle behind; and This refers to the longitudinal acceleration information of adjacent trains obtained through the vehicle-to-vehicle communication system. During the centralized training phase, all local observations of the tracked trains are combined into a global joint state, denoted as: ; In the formula, Centralized reward value assessment and cost value assessment used only during the training phase; No. The action of each agent is defined as the longitudinal control command of the vehicle within the current control cycle, which will be the first... The agent in the th... The control action output within each control cycle Recorded as: ; In the formula, and The actions of all following trains constitute a combined action, determined by both the train's braking and traction capabilities. for: ; With joint actions Correspondingly, the state transition relationships in multi-agent constrained Markov decision models Represents the given global joint state and joint actions Under the given conditions, the state transition relationship for the evolution of the global joint state of the multi-vehicle queue to the next control cycle is as follows: ; For the For a train tracking system, its single-step local reward function is defined as: ; In the formula, For spacing tracking rewards; For speed-coordinated reward items; This is a stability penalty item; For the The single-step safety cost of a train tracking system is defined as: ; In the formula, For the first The actual tracking distance between the tracking train and the train in front; For the minimum dynamic safety tracking interval; when When it is within the safe and feasible region, the corresponding safety cost is 0; when At that time, the safety cost is taken as the depth of the safety boundary violation; The immediate safety cost is defined as the maximum safety cost of all adjacent train pairs as follows: ; Based on local observation, global joint state, action space, state transition relationships, cooperative reward function, and safety cost function, the objective of multi-vehicle cooperative tracking control is expressed as: maximizing the long-term cumulative cooperative reward of the entire fleet while satisfying the cumulative safety cost constraint, i.e.: ; ; In the formula, For multi-agent joint strategies; The length of a single round of interaction; The allowed cumulative security cost threshold.

[0015] Optionally, the spacing tracking reward term is defined as follows: ; In the formula, As a weight for spacing rewards; This is the spacing error scale parameter; For the first The spacing error of the trains being tracked; The speed-coordination reward item is defined as follows: ; In the formula, Weighting for speed-related rewards; For velocity error scale parameters; For the first The speed error between the tracking train and the train ahead of it; To balance the smoothness of control input with the comfort of train operation, a stability penalty term is introduced, which is defined as follows: ; In the formula, The weight of the stability penalty term; As a comprehensive stability index, it is defined as follows: ; ; In the formula, To control the input normalization constant; To control the rate of change of input or equivalent jerk; This is the normalization constant for jerk. To control the cycle.

[0016] Optionally, the step of constructing a dynamic constrained multi-agent joint policy optimization based on the multi-agent constrained Markov decision model using a shared policy network, and updating the shared policy network by pruning the target using proximal policy optimization during centralized training of the dynamic constrained multi-agent joint policy optimization, includes: Let the first The local strategy for train tracking is The optimization of the dynamic constraint multi-agent joint policy using a shared policy network is as follows: ; In the formula, For sharing policy network parameters; For the first Partial observation of the tracking train; For the first The agent in the th... The control actions output within each control cycle; All the actions of the tracking train constitute a joint action; Combine the local observations of all tracked trains into a global joint state; During the centralized training phase, a reward-value network is dynamically constrained to optimize the proximal policy of the multi-agent system. and cost-value network , and Both take the global joint state as input and estimate the expected cumulative reward and expected cumulative cost of the discount, respectively. Based on cumulative safety costs, during centralized training of multi-agent proximal policy optimization with dynamic constraints, in the first... During round training, the following conditions must be met: ; In the formula, For the first The dynamic cost threshold corresponding to each round of training; the scheduling method for the dynamic cost threshold is defined as follows: ; In the formula, This is the initial cost threshold; For the ultimate target cost, The attenuation coefficient; when This indicates that the security risks of the current strategy exceed the requirements of the current stage's constraints. As the cost increases, subsequent policy updates will place greater emphasis on security costs; when This indicates that the strategy has basically met the security constraints of the current stage. With reduced speed, strategy updates can focus more on spacing tracking, speed coordination, and operational stability. Let the reward advantage and cost advantage be respectively and Then the combined Lagrange dominance function can be written as: ; In the formula, This is the overall advantage function used for policy updates; Let the first An intelligent agent at time The probability ratio between the old and new strategies is: ; In the formula, This is the current strategy; The old strategy used when sampling trajectories; combined with the Lagrange dominance function, the optimization objective of the shared policy network for state-constrained multi-agent joint policy optimization is expressed as: ; In the formula, The pruning factor optimized for near-end strategies; Reward Value Network and cost-value network Each update is performed by minimizing the estimation error, and its loss function is expressed as follows: ; ; In the formula, and These are the parameters for the reward value network and the cost value network, respectively. and These are the estimated rewards and costs, respectively. The Lagrange multipliers are updated based on the cumulative safety cost estimate of the current training batch: ; In the formula, The learning rate of the Lagrange multipliers; This is the estimated cumulative cost of the current training batch; if Then increase This makes subsequent policy updates place greater emphasis on security constraints; if Then reduce or maintain a low level. This allows the strategy to further optimize collaborative tracking performance while meeting security constraints.

[0017] In addition, embodiments of the present invention also provide a train tracking operation cooperative optimization control device based on safety reinforcement learning, the device comprising: Data acquisition module: Used to obtain the operating status data of adjacent trains and adjacent following trains and the operating status data of the train itself based on vehicle-to-vehicle communication in each control cycle. The operating status data of adjacent trains includes train arrangement information, speed information and acceleration information. The operating status data of the train itself includes the position information, speed information and acceleration information of the train itself. The first calculation module is used to calculate the safe braking distance of adjacent trains under the most unfavorable operating conditions based on the operating status data of adjacent trains and the operating status data of the current train, and to calculate the minimum dynamic safe tracking interval of the current adjacent train pair in combination with the basic safety margin. The second calculation module is used to form a target tracking interval by superimposing an additional tracking margin on the minimum dynamic safety tracking interval, and to calculate the actual net distance between the front and rear trains based on the position information in the adjacent train operation status data and the speed difference between the tracking train and the preceding train, to calculate the spacing error between the actual net distance and the target tracking interval, and to obtain the speed coordination error based on the speed difference between the tracking train and the preceding train. Model building module: used to build a multi-agent constrained Markov decision model based on the train's operating status data, adjacent train operating status data, minimum dynamic safe tracking interval, target tracking interval, spacing error, and speed coordination error; Target update module: used to construct a dynamic constrained multi-agent joint policy optimization based on the multi-agent constrained Markov decision model using a shared policy network, and to update the shared policy network by pruning the target using proximal policy optimization during the centralized training of the dynamic constrained multi-agent joint policy optimization. Model deployment module: used to deploy the trained dynamic constraint multi-agent joint strategy optimization in the on-board control equipment of each tracking train to achieve collaborative optimization control of train tracking operation.

[0018] In this embodiment of the invention, by unifying dynamic safety tracking interval, multi-agent collaborative reinforcement learning, system-level safety cost, dynamic cost threshold, and Lagrange constraint adjustment mechanism into the same control framework, it is possible to improve the collaborative tracking capability, operational tightness, and control stability of multi-vehicle platoons while ensuring dynamic safety constraints; it is possible to more effectively meet the dynamic safety constraints in the process of close multi-vehicle tracking; it is possible to enhance the system-level safety constraint capability of multi-vehicle platoons; it is possible to balance training exploration capability and final safety; it is possible to uniformly optimize collaborative tracking performance and safety constraints; it is possible to more fully characterize the coupling relationship of multi-vehicle platoons; it has lower online deployment complexity and better engineering applicability; and it is possible to improve tracking tightness and platoon consistency while ensuring safety. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the train tracking and operation collaborative optimization control method based on safety reinforcement learning in an embodiment of the present invention. Figure 2 This is a schematic diagram of the train safety braking process in an embodiment of the present invention; Figure 3 This is a schematic diagram of the close tracking and relative braking relationship of the trains in an embodiment of the present invention; Figure 4 This is a schematic diagram of the control framework of the train tracking operation system in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structural composition of the train tracking and operation collaborative optimization control device based on safety reinforcement learning in an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Example 1, please refer to Figure 1 , Figure 1 This is a flowchart illustrating the train tracking and operation collaborative optimization control method based on safety reinforcement learning in an embodiment of the present invention.

[0023] like Figure 1 This paper presents a cooperative optimization control method for train tracking operation based on safety reinforcement learning. The method includes: S101: In each control cycle, each tracking train obtains the adjacent train operation status data and its own operation status data based on vehicle-to-vehicle communication. The adjacent train operation status data includes train arrangement information, speed information and acceleration information. The own train operation status data includes its own position information, speed information and acceleration information. In the specific implementation of this invention, a maglev train can be used as the specific implementation object, but the modeling concept can also be extended to rail transit vehicles with similar longitudinal motion constraints. To facilitate the construction of a multi-vehicle cooperative tracking control model, each train is equivalent to a single-mass system running along the track direction, and only the longitudinal motion process of the train is considered. The longitudinal motion state of the train is described by position, velocity and acceleration. The longitudinal acceleration is determined by traction force, braking force and total running resistance. The total running resistance may include magnetic resistance, current collector resistance, air resistance, gradient additional resistance and curve additional resistance, etc.

[0024] In multi-vehicle collaborative tracking scenarios, multiple vehicle queues are set up as follows: The train consists of a lead train and the following trains. The lead train runs according to a preset operating curve. Each following train obtains the position, speed and acceleration of the adjacent trains through car-to-car communication, and generates longitudinal traction / braking control inputs based on its own current status, thereby realizing the coordinated tracking operation of multiple train platoons.

[0025] Let the first The train was at The position, velocity, and acceleration information for each control cycle are respectively... , and The control cycle is The discrete state update relation of the train is then expressed as: ; ; In the formula, For the first The train was at The actual longitudinal acceleration within each control cycle; and this actual longitudinal acceleration is obtained after constraining the train's traction capacity, braking capacity, and track speed limit. During train operation, the following speed constraints and control input constraints are satisfied: ; ; In the formula, Speed ​​limits were imposed on the line. For the first The train was at Longitudinal control commands within a control cycle and To control the lower and upper limits of the input.

[0026] S102: Calculate the safe braking distance of adjacent trains under the most unfavorable operating conditions based on the operating status data of adjacent trains and the operating status data of this train, and calculate the minimum dynamic safe tracking interval of the current adjacent train pair in combination with the basic safety margin. In the specific implementation of this invention, under vehicle-to-vehicle communication conditions, the tracking train can obtain the running status of the preceding train in real time and dynamically calculate the minimum safe tracking interval based on the relative braking relationship. This safe interval is not a fixed constant, but is determined by the relative braking process of the preceding and following trains under the most unfavorable braking conditions. Therefore, it is necessary to establish a train safe braking distance model that considers response lag and braking establishment process.

[0027] A complete braking process can be divided into the response and traction cut-off phase, the coasting transition phase, and the full braking phase, such as Figure 2 As shown; after the train receives the braking command, it needs to go through communication transmission, equipment processing, and actuator action before the traction force is gradually cut off; to reflect the safety redundancy under the most unfavorable operating condition, it is assumed that the train is still running at maximum traction acceleration during the response and traction cut-off phase; let the duration of this phase be... The corresponding running distance It can be represented as: ; In the formula, The initial velocity at the moment of braking triggering; The maximum traction acceleration maintained during the response phase; Speed ​​at the end of the traction shearing phase for: ; After the traction force is cut off, the train enters a coasting transition phase until the braking system establishes effective braking force; during this process, the train decelerates naturally mainly due to running resistance; let the time of the coasting transition phase be... The running distance corresponding to the coasting transition phase is... Represented as: ; In the formula, Let be the equivalent natural deceleration of the train under the action of running resistance; then the speed at the end of the coasting transition phase. for: ; After the braking force is fully established, the train enters full braking mode and decelerates at maximum emergency braking speed. The braking distance during the deceleration process until the vehicle comes to a complete stop is... Represented as: ; In the formula, For the first Maximum emergency braking deceleration amplitude of the train; In summary, the first Tracking the safe braking distance of the train under the most unfavorable operating conditions for: ; In close-tracking scenarios, simply calculating the safe braking distance of the following train is insufficient to determine the minimum safe interval; the key to close-tracking lies in determining the minimum dynamic safe tracking interval under short-interval operating conditions based on the relative braking relationship between the preceding and following trains; such as Figure 3 As shown, let time be... , No. The train suddenly applied maximum emergency braking, and the... Due to factors such as communication transmission, information processing, and pressure buildup by the actuators, the train needs a certain delay before it can establish full braking force; for the first The train, at its time Immediately enter maximum emergency braking state, therefore the safe braking distance of the vehicle in front can be expressed as: ; In the formula, The current speed of the vehicle in front; This represents the maximum emergency braking deceleration value of the vehicle in front.

[0028] For the A following train may maintain its original traction during the delay phase before entering maximum emergency braking; therefore, the safe braking distance of the following train is crucial. , further expressed as: ; In the formula, For the first The system tracks the current speed of the train. To ensure that two adjacent trains come to a complete stop under the most unfavorable operating conditions, the first... The train locomotive and the first A basic safety margin is still maintained between the rear of the train cars. Then the first Train tracking at time The corresponding minimum dynamic safety tracking interval can be expressed as: ; In the formula, For the first The minimum dynamic safety tracking interval that a train must meet to track another train. To ensure a safe braking distance for following vehicles, To ensure a safe braking distance from the vehicle in front, Based on a safety margin.

[0029] S103: Based on the minimum dynamic safety tracking interval, an additional tracking margin is superimposed to form a target tracking interval. The actual net distance between the front and rear trains is calculated based on the position information in the adjacent train operation status data and the speed difference between the tracking train and the preceding train. The spacing error between the actual net distance and the target tracking interval is calculated, and the speed coordination error is obtained based on the speed difference between the tracking train and the preceding train. In the specific implementation of this invention, the minimum dynamic safety tracking interval is jointly determined by the relative braking distance between the preceding and following trains and the basic safety margin, and changes dynamically with the speed state, response time delay, and braking capacity of adjacent trains, thus reflecting the dynamic safety boundary under close tracking scenarios; to ensure the safety of close tracking operation, the first... The train was being tracked. The actual distance at any given time always satisfies the following: ; To prevent the tracking train from running close to the dynamic safety boundary for extended periods, an additional tracking margin is introduced on top of the minimum dynamic safety tracking interval. , define the first Train tracking at time The target tracking interval is: ; in, For the first Train tracking at time Target tracking interval; For the first Train tracking at time Corresponding minimum dynamic safety tracking interval; additional tracking margin It is used to absorb measurement errors, model uncertainties and control fluctuations, and improve the robustness of close tracking operation.

[0030] For any i A train is being tracked, and the train directly ahead of it is the 1st train. Trains; to describe the spatial relationship between adjacent trains, the first... The actual tracking distance between a train and the train in front is defined as: ; In the formula, For the first Within the first control cycle, the first The actual net distance between the tracking train and the train in front; The length of the vehicle in front; and These are the location information for the vehicle in front and the vehicle itself.

[0031] Any adjacent train pairs The spacing error is defined as: ; In the formula, For any adjacent train pairs exist The time interval error; For the first The train was being tracked. The actual distance at any given moment remains constant; For the first The train was being tracked. The target tracking interval at any given time; when When the actual distance is greater than the target distance, it indicates that the actual distance is greater than the target distance; when When the actual distance is less than the target tracking interval, it means that the tracking train is closer to the safety boundary.

[0032] To characterize the speed coordination relationship between adjacent trains, the first... The speed coordination error of the tracking train relative to the preceding train is: ; In the formula, For the first The train was being tracked. Speed ​​at any given moment; For the car in front Speed ​​at a given moment; when When the following train is traveling at a speed higher than the train in front, it may increase the risk of a rear-end collision if the actual following distance is insufficient; when When the speed of the following train is lower than that of the train in front, the queuing distance may gradually increase, which is not conducive to close tracking. Therefore, the speed coordination error reflects not only the consistency of motion between adjacent trains, but also the tightness and stability of the queuing operation.

[0033] Combined speed deviation The objective of multi-train cooperative tracking control can be described as: under the premise of satisfying dynamic safety constraints, gradually reducing the spacing error and speed deviation between adjacent trains, i.e.: .

[0034] This enables safe, close, and stable coordinated tracking operation of multiple trains.

[0035] S104: Construct a multi-agent constrained Markov decision model based on the train's operating status data, adjacent train operating status data, minimum dynamic safe tracking interval, target tracking interval, spacing error, and speed coordination error; In the specific implementation of this invention, such as Figure 3The multi-car cooperative tracking control architecture shown depicts each tracking train as a cooperative agent. Each agent generates longitudinal control actions based on its own train status and the status of neighboring trains obtained through train-to-train communication. The combined actions of all tracking trains, after being processed by the train dynamics environment, collectively influence the multi-car platoon's operational state at the next moment. Simultaneously, the control process needs to satisfy the minimum dynamic safe tracking interval constraint. Therefore, based on the train's operational state data, neighboring train operational state data, minimum dynamic safe tracking interval, target tracking interval, spacing error, and speed coordination error, the multi-car cooperative tracking control problem is modeled as a multi-agent constrained Markov decision model as follows: ; in, For each intelligent agent's local observation space; For the centralized training phase, the global joint state space is defined. For joint action space; This represents a state transition relationship; For collaborative reward functions; For safety cost function; Discount factor; During the online execution phase, each tracking train makes distributed decisions based solely on its own status and the status of adjacent trains; for the train in the middle of the queue... The local observation of a train being tracked can be represented as follows: ; In the formula, and The first The train tracks the actual distance between itself and the trains in front and behind it; , and These are the speeds of the vehicle in front, the vehicle itself, and the vehicle behind; and This refers to the longitudinal acceleration information of adjacent trains obtained through the vehicle-to-vehicle communication system. During the centralized training phase, all local observations of the tracked trains are combined into a global joint state, denoted as: ; In the formula, This is used only for centralized reward and cost-value assessment during the training phase. "Centralized training" here refers to evaluating the overall collaborative benefits and system safety costs of the train fleet using the global joint state of multiple train queues during strategy training; it does not mean that the central controller uniformly outputs all train actions during the online execution phase. In the distributed execution phase, each tracking train still relies solely on its own local observations. It can independently output control actions to meet the needs of distributed control in multi-vehicle collaborative tracking.

[0036] No. The action of each agent is defined as the longitudinal control command of the vehicle within the current control cycle; to avoid confusion with the actual acceleration sign of the train, this embodiment defines the control action output by the agent. Recorded as: ; In the formula, and It is determined by both the train's braking capacity and traction capacity; It can represent longitudinal acceleration commands, and can also be further mapped to equivalent traction / braking control commands; all tracking train actions constitute a combined action. for: ; With joint actions Correspondingly, the state transition relationships in multi-agent constrained Markov decision models Represents the given global joint state and joint actions Under the given conditions, the state transition relationship for the evolution of the global joint state of the multi-vehicle queue to the next control cycle is as follows: ; Specifically, the simulation environment receives joint actions Then, based on the train's traction / braking capacity constraints and the line speed limit constraints, the agent outputs actions. Mapped to actual longitudinal acceleration Subsequently, the position, speed, and acceleration of each tracking train are updated based on the discrete state update relationship of the trains. After the train state is updated, the actual distance, speed coordination error, minimum dynamic safe tracking interval, and distance error for the next moment are calculated based on the tracking relationship between adjacent trains and the dynamic safety interval model, and local observations for the next moment are generated accordingly. and the global joint state during the centralized training phase The pilot train's state is determined by a preset operating curve and is not controlled by a reinforcement learning agent.

[0037] To ensure close, coordinated, and stable tracking of multiple train platoons under safe conditions, this application constructs a cooperative reward function and a safety cost function. The reward function guides the tracking trains to maintain the target tracking interval, improves speed coordination, and suppresses control fluctuations; the cost function characterizes the degree to which the actual tracking interval violates the minimum dynamic safe tracking interval.

[0038] For the For a train tracking system, its single-step local reward function is defined as: ; In the formula, For spacing tracking rewards; For speed-coordinated reward items; This is a stability penalty item.

[0039] Spacing tracking reward characterizes the tracking effect of the actual tracking spacing on the target tracking interval. The closer the actual tracking spacing is to the target tracking interval, the larger the reward; the closer the actual tracking spacing is to the target tracking interval, the larger the reward. Spacing tracking reward is defined as: ; In the formula, As a weight for spacing rewards; This is the spacing error scale parameter; For the first The spacing error of the train tracking.

[0040] Speed ​​matching reward is used to encourage the following train to maintain the same speed as the train ahead. The reward increases as the speed error between the following and preceding train decreases; conversely, the reward gradually decreases as the speed error increases. The speed matching reward is defined as follows: ; In the formula, Weighting for speed-related rewards; For velocity error scale parameters; For the first The speed error between the tracking train and the train in front of it.

[0041] To balance the smoothness of control input with the comfort of train operation, a stability penalty term is introduced, which is defined as follows: ; In the formula, The weight of the stability penalty term; As a comprehensive stability index, it is defined as follows: ; ; In the formula, To control the input normalization constant; To control the rate of change of input or equivalent jerk; This is the normalization constant for jerk. To control the cycle.

[0042] Considering that multi-train cooperative control needs to take into account the overall operational efficiency of the entire queue, the system's real-time cooperative reward is defined as the average of the local rewards of all tracking trains: ; Train operation safety is characterized by a cost function; safety cost is used to measure the degree to which the actual tracking interval violates the minimum dynamic safe tracking interval.

[0043] For the The single-step safety cost of a train tracking system is defined as: ; In the formula, For the first The actual tracking distance between the tracking train and the train in front; For the minimum dynamic safety tracking interval; when When it is within the safe and feasible region, the corresponding safety cost is 0; when In this case, the safety cost is taken as the depth of the safety boundary violation.

[0044] To enhance safety and conservatism in multi-train platooning operations, the system's instantaneous safety cost is defined as the maximum value of the safety costs of all adjacent train pairs: ; Based on local observation, global joint state, action space, state transition relationships, cooperative reward function, and safety cost function, the objective of multi-vehicle cooperative tracking control is expressed as: maximizing the long-term cumulative cooperative reward of the entire fleet while satisfying the cumulative safety cost constraint, i.e.: ; ; In the formula, For multi-agent joint strategies; The length of a single round of interaction; The allowed cumulative security cost threshold.

[0045] Multi-agent CMDP (Constrained Markov Decision Process) modeling clarifies the relationships between observation, action, state transition, reward evaluation, and safety constraints in multi-vehicle cooperative tracking control, providing a foundation for the subsequent design of DC-MAPPO (Dynamically Constrained Multi-Agent Proximal Policy Optimization) control methods.

[0046] S105: Based on the multi-agent constrained Markov decision model, a shared policy network is used to construct a dynamic constrained multi-agent joint policy optimization, and during the centralized training of the dynamic constrained multi-agent joint policy optimization, a proximal policy optimization is used to prune the target and update the shared policy network. In the specific implementation of this invention, based on the multi-agent proximal policy optimization framework, a dynamic cost threshold and a Lagrange constraint adjustment mechanism are introduced to construct a multi-agent proximal policy optimization method that integrates dynamic constraint mechanisms, namely DC-MAPPO. A centralized training-distributed execution architecture is adopted. In the training phase, the global joint state is used for centralized value evaluation. In the execution phase, control actions are generated only by relying on the local observations of each tracking train, thus taking into account both the multi-train collaborative learning capability and the need for online distributed control. The update process of the shared policy network, reward value network, cost value network, and dynamic constraint mechanism is explained from the perspective of DC-MAPPO algorithm implementation.

[0047] In multi-vehicle cooperative tracking scenarios, each tracking train has similar dynamic characteristics and a consistent cooperative tracking target. To improve training efficiency and enhance policy generalization ability, this embodiment adopts a parameter-sharing policy network, meaning all tracking trains share the same set of policy network parameters; let the... The local strategy for train tracking is The optimization of the dynamic constraint multi-agent joint policy using a shared policy network is as follows: ; In the formula, For sharing policy network parameters; For the first Partial observation of the tracking train; For the first The agent in the th... The control actions output within each control cycle; All the actions of the tracking train constitute a joint action; The local observations of all tracked trains are combined into a global joint state.

[0048] Shared policy networks use local observations of a single tracked train. The system takes input as input, outputs motion distribution parameters, and generates longitudinal control actions for the train within the current control cycle. During the training phase, a Tanh-Gaussian strategy is used for continuous motion sampling to ensure the strategy has sufficient exploration capability in the continuous motion space. During the verification and online execution phases, the motion mean can be used as a deterministic control command to improve the stability of the control output.

[0049] During the intensive training phase, DC-MAPPO sets up a reward-value network. and cost-value network Both are in a globally united state. As input, the system's expected cumulative reward and expected cumulative cost are estimated respectively. The reward value network is used to evaluate the overall cooperative tracking performance of the fleet, and the cost value network is used to characterize the dynamic safety constraint satisfaction. Since the reward and cost rewards differ in numerical distribution and statistical characteristics, this embodiment uses two independent value networks for estimation to improve the fitting accuracy of the value function and the training stability.

[0050] The dynamic constraint threshold mechanism is a key safety constraint design in the DC-MAPPO method. Unlike the fixed cost threshold, this mechanism gradually tightens the upper limit of the cumulative safety cost of the system during the training process, so that the policy retains the necessary exploration space in the early stage of training and gradually approaches the target safety constraint in the later stage of training.

[0051] In multi-train cooperative tracking scenarios, the safety constraint object is no longer the safety cost of tracking a single train, but rather the fleet-level safety cost jointly determined by all adjacent train pairs. Therefore, this application uses a dynamic cost threshold to adjust the upper limit of the safety risk allowed for the entire multi-train queue during the training phase, enabling the policy training process to achieve a balance between exploration capability and safety constraint satisfaction.

[0052] Based on the cumulative security cost, during the centralized training of DC-MAPPO, in the first... During round training, the following conditions must be met: ; In the formula, For the first The dynamic cost threshold corresponding to each training round; unlike the fixed cost threshold, the dynamic cost threshold does not remain constant throughout the training process, but gradually transitions from a relaxed state to a target safety constraint state with each training iteration. The scheduling method of the dynamic cost threshold is defined as follows: ; In the formula, As the initial cost threshold, For the ultimate target cost threshold, The cost threshold is the decay coefficient. In the early stages of training, a relatively lenient cost threshold can provide necessary space for policy exploration and prevent the policy from becoming overly conservative due to excessive safety penalties. As training progresses, the cost threshold gradually approaches the target safety constraints, guiding the policy to gradually enter the safe and feasible domain.

[0053] To unify the maximization of collaborative rewards with the constraints of safety costs within the same optimization framework, this application introduces Lagrange multipliers. Dynamic cost threshold Given the maximum allowed cumulative security cost of the system during the current training phase, the Lagrange multiplier... The intensity of the safety penalty is adaptively adjusted based on the deviation between the actual cumulative cost and the dynamic threshold; when... This indicates that the security risks of the current strategy exceed the requirements of the current stage's constraints. As the cost increases, subsequent policy updates will place greater emphasis on security costs; when This indicates that the strategy has basically met the security constraints of the current stage. With reduced speed, strategy updates can focus more on spacing tracking, speed coordination, and operational stability.

[0054] Let the reward advantage and cost advantage be respectively and Then the combined Lagrange dominance function can be written as: ; In the formula, This is the overall advantage function used for policy updates.

[0055] The comprehensive advantage function of policy updates shows that policy updates are not only affected by the cooperative reward advantage, but also constrained by the safety cost advantage. When the safety cost is high, the Lagrange term weakens the update advantage of high-risk actions, thereby inhibiting the policy from optimizing in the direction of violating the dynamic safety boundary.

[0056] DC-MAPPO inherits the near-end update concept from PPO-Clip during policy update, limiting the range of probability ratio changes between the old and new policies to avoid training instability caused by excessively large single policy update magnitudes; let the th... An intelligent agent at time The probability ratio between the old and new strategies is: ; In the formula, This is the current strategy; The old strategy used when sampling trajectories; combined with the Lagrange dominance function, the optimization objective of the shared policy network for state-constrained multi-agent joint policy optimization is expressed as: ; In the formula, The objective function is the pruning coefficient for near-end policy optimization. On the one hand, it limits the policy update range through pruning operations to ensure training stability. On the other hand, it introduces dynamic safety cost constraints into the policy optimization process by integrating the Lagrange multiplier, thereby achieving joint optimization between collaborative tracking performance and safety constraint satisfaction.

[0057] The reward value network and cost value network are updated by minimizing the value estimation error, and their loss functions can be expressed as: ; ; In the formula, and These are the parameters for the reward value network and the cost value network, respectively. and These are the estimated values ​​for reward returns and cost returns, respectively.

[0058] The Lagrange multipliers are updated based on the cumulative safety cost estimate of the current training batch: ; In the formula, The learning rate of the Lagrange multipliers; This is the estimated cumulative cost of the current training batch; if Then increase This makes subsequent policy updates place greater emphasis on security constraints; if Then reduce or maintain a low level. This allows the strategy to further optimize collaborative tracking performance while meeting security constraints.

[0059] The DC-MAPPO control method achieves policy coordination among multiple tracking trains through a shared policy network. It characterizes the overall operational effectiveness and system safety risks of the multi-train platoon using a centralized reward value network and cost value network. Safety constraints are progressively strengthened during training through dynamic cost thresholds and Lagrange multiplier adjustment mechanisms. During the execution phase, only the trained shared policy network is retained, and each tracking train independently generates longitudinal control commands based on local observations. Therefore, this method can achieve safe, close, and stable collaborative tracking operation of multiple train platoons under dynamic safety constraints.

[0060] S106 optimizes and deploys the trained dynamic constraint multi-agent joint strategy in the onboard control equipment of each tracking train to achieve collaborative optimization control of train tracking operation.

[0061] Based on the aforementioned multi-train cooperative tracking operation model, multi-agent constrained Markov decision process model, and DC-MAPPO control method integrating dynamic constraint mechanisms, a centralized training-distributed execution architecture will be adopted to complete control strategy optimization and online deployment. During the training phase, the overall cooperative reward and system safety cost of the multi-train queue are centrally evaluated using the global joint state, and the policy is updated by constraining the dynamic cost threshold and Lagrange multiplier adjustment mechanism. During the execution phase, only the trained shared policy network is retained, and each tracking train independently generates longitudinal control commands based on local observations.

[0062] After training is complete, the policy network parameters will be shared. Deployed to the onboard control equipment of each tracking train; during the online execution phase, the reward value network, cost value network, and Lagrange multipliers no longer participate in control decisions; The train tracking train in the Within each control cycle, the operating status of adjacent trains is obtained through vehicle-to-vehicle communication, and local observation is constructed by combining the status of the current train. And input the shared policy network. Output longitudinal control commands The control command, after being constrained by traction / braking capacity and track speed limits, is applied to the longitudinal movement of the train, thereby updating the train's operating status and entering the next control cycle.

[0063] Therefore, the training phase can fully learn the coupling relationship of the multi-vehicle queue and the system-level safety constraints by utilizing the global joint state, while the execution phase relies only on local observations to achieve distributed control. This reduces the burden of online communication and computation, and ensures that the multi-vehicle queue can achieve safe, close and stable collaborative tracking operation under the dynamic safety tracking interval constraint.

[0064] Compared with existing train tracking control methods, this application unifies dynamic safety tracking intervals, multi-agent cooperative reinforcement learning, system-level safety costs, dynamic cost thresholds, and Lagrange constraint adjustment mechanisms into the same control framework. This can improve the cooperative tracking capability, operational tightness, and control stability of multi-train platoons while ensuring dynamic safety constraints. The specific technical advantages are as follows.

[0065] 1. This method can more effectively satisfy dynamic safety constraints in multi-vehicle close tracking processes. Existing methods typically use fixed safety distances or indirectly constrain safety behavior through rewards and penalties, which are difficult to fully adapt to multi-vehicle tracking scenarios where the speed, braking capacity, and response delay of the vehicles in front and behind change in real time. This application transforms the minimum dynamic safe tracking interval, which considers the braking response delay, into a safety cost, enabling the dynamic safety boundary to directly participate in the optimization of reinforcement learning policies, thereby improving the policy's ability to satisfy safety constraints.

[0066] 2. This application enhances the system-level safety constraints of multi-train platoons. Existing methods, which use average safety costs or single-train safety evaluations, may weaken local high-risk states. This application uses the maximum value of the safety costs of all adjacent train pairs as the system's instantaneous safety cost, allowing the training process to focus on adjacent train pairs with the smallest safety margin and the highest risk in the platoon, thereby improving the overall safety in multi-train close-tracking scenarios.

[0067] 3. It can balance training exploration capabilities with ultimate security; existing fixed security threshold methods are prone to problems such as limited exploration in the early stages of training or insufficient security constraints in the later stages of training. This application sets a dynamic cost threshold that gradually tightens with each training round, allowing policy training to gradually transition from a relatively relaxed exploration phase to a strict security constraint phase, thereby improving the ability to satisfy ultimate security constraints while ensuring the policy exploration space.

[0068] 4. This application enables unified optimization of collaborative tracking performance and safety constraints. Existing reinforcement learning methods based on fixed penalty terms struggle to adaptively adjust constraint strength according to changes in safety risks during training. This application introduces Lagrange multipliers to adjust the safety penalty strength based on the deviation between the cumulative safety cost and the dynamic cost threshold, and combines reward advantage and cost advantage into a comprehensive advantage function, enabling policy updates to simultaneously consider gap tracking, velocity collaboration, control stability, and system safety constraints.

[0069] 5. It can more fully characterize the coupling relationship of multi-vehicle queues; existing independent learning or single-vehicle control methods are difficult to handle the coupling propagation problem between speed error, spacing change and control input in multi-vehicle queues. This application adopts a centralized training-distributed execution architecture, and uses the global joint state to evaluate reward value and cost value during the training phase, enabling the policy to learn the overall operating characteristics of the multi-vehicle queue, thereby improving the consistency and stability of multi-vehicle cooperative tracking.

[0070] 6. The online deployment complexity is lower, and its engineering applicability is better. During the execution phase, this application only retains the trained shared policy network. Each tracking train only needs to construct local observations based on its own and adjacent train's operating status to independently generate longitudinal traction / braking control commands, eliminating the need for continuous centralized optimization or global value assessment during the online phase. Therefore, this method reduces the online communication and computational burden, making it more suitable for onboard control equipment deployment.

[0071] 7. This method improves tracking tightness and queue consistency while ensuring safety. Simulation results show that the proposed method can reduce average speed coordination error and queue consistency error while maintaining a queue safety boundary violation rate of 0% and a tight tracking achievement rate of 100.00%. Compared to the independent learning-based dynamic constraint method, this method enables multi-vehicle queues to achieve more compact, consistent, and stable cooperative tracking operation while ensuring that the dynamic safety boundary is not breached.

[0072] Example 2, please refer to Figure 5 , Figure 5 This is a schematic diagram of the structural composition of the train tracking and operation collaborative optimization control device based on safety reinforcement learning in an embodiment of the present invention.

[0073] like Figure 5 As shown, a train tracking operation cooperative optimization control device based on safety reinforcement learning is disclosed, the device comprising: Data acquisition module 501: Used to obtain, within each control cycle, the operating status data of adjacent trains (both preceding and following trains) and the operating status data of the train itself based on vehicle-to-vehicle communication. The adjacent train operating status data includes train configuration information, speed information, and acceleration information, and the train operating status data includes the train's position information, speed information, and acceleration information. First calculation module 502: used to calculate the safe braking distance of adjacent trains under the most unfavorable operating conditions based on the operating status data of adjacent trains and the operating status data of the current train, and to calculate the minimum dynamic safe tracking interval of the current adjacent train pair in combination with the basic safety margin. The second calculation module 503 is used to form a target tracking interval by superimposing an additional tracking margin on the minimum dynamic safety tracking interval, and to calculate the actual net distance between the front and rear trains based on the position information in the adjacent train operation status data and the speed difference between the tracking train and the preceding train, to calculate the distance error between the actual net distance and the target tracking interval, and to obtain the speed coordination error based on the speed difference between the tracking train and the preceding train. Model building module 504: used to build a multi-agent constrained Markov decision model based on the train's operating status data, adjacent train operating status data, minimum dynamic safe tracking interval, target tracking interval, spacing error, and speed coordination error; Target update module 505: is used to construct a dynamic constrained multi-agent joint policy optimization based on the multi-agent constrained Markov decision model using a shared policy network, and to prune the target and update the shared policy network using proximal policy optimization during the centralized training of the dynamic constrained multi-agent joint policy optimization. Model deployment module 506: used to deploy the trained dynamic constraint multi-agent joint strategy optimization in the on-board control equipment of each tracking train to achieve collaborative optimization control of train tracking operation.

[0074] For a detailed description of the implementation of Example 2, please refer to Example 1, which will not be repeated here.

[0075] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0076] Furthermore, the above provides a detailed description of the train tracking operation collaborative optimization control method and device based on safety reinforcement learning provided by the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A train tracking operation collaborative optimization control method based on security reinforcement learning, characterized in that, The method includes: During each control cycle, each tracking train obtains the operating status data of adjacent trains (both preceding and following) and its own operating status data based on vehicle-to-vehicle communication. The adjacent train operating status data includes train configuration information, speed information, and acceleration information, while the own train operating status data includes its own position information, speed information, and acceleration information. Based on the operating status data of adjacent trains and the operating status data of this train, the safe braking distance of adjacent trains under the most unfavorable operating conditions is calculated, and the minimum dynamic safe tracking interval of the current adjacent train pair is obtained by combining the basic safety margin. The target tracking interval is formed by superimposing the minimum dynamic safety tracking interval with an additional tracking margin. The actual net distance between the front and rear trains is calculated based on the position information in the adjacent train operation status data and the speed difference between the tracking train and the preceding train. The distance error between the actual net distance and the target tracking interval is calculated, and the speed coordination error is obtained based on the speed difference between the tracking train and the preceding train. A multi-agent constrained Markov decision model is constructed based on the train's operating status data, adjacent train operating status data, minimum dynamic safe tracking interval, target tracking interval, spacing error, and speed coordination error. Based on the multi-agent constrained Markov decision model, a shared policy network is used to construct a dynamic constrained multi-agent joint policy optimization, and during the centralized training of the dynamic constrained multi-agent joint policy optimization, a proximal policy optimization is used to prune the target and update the shared policy network. The trained dynamic constraint multi-agent joint strategy is optimized and deployed in the on-board control equipment of each tracking train to achieve collaborative optimization control of train tracking operation; The calculation of the safe braking distance of adjacent trains under the most unfavorable operating conditions based on the operating status data of adjacent trains and the operating status data of the current train, and the calculation of the minimum dynamic safe tracking interval of the current adjacent train pair in combination with the basic safety margin, includes: If the train is still accelerating with maximum traction force in response to the traction cut-off phase, the duration of the traction cut-off phase is defined as The corresponding distance of travel is then is represented as: ； wherein is the initial speed at the braking trigger time; is the maximum traction acceleration maintained during the response phase; Speed at end of traction resection phase Is: ； After the traction force is cut off, the train enters the coasting transition phase until the braking system establishes effective braking force, the train is naturally decelerated by the running resistance, and the time of the coasting transition phase is set as The running distance corresponding to the coasting transition phase is The running distance corresponding to the coasting transition phase is ； In the formula, is the equivalent natural deceleration of the train when it is subjected to the action of the operating resistance; is the speed of the train at the end of the coasting transition phase is: ； After the braking force is fully established, the train enters full braking mode and decelerates at maximum emergency braking speed. The braking distance during the deceleration process until the vehicle comes to a complete stop is... Represented as: ； No. Tracking the safe braking distance of the train under the most unfavorable operating conditions for: ； At any moment , No. The train suddenly applied maximum emergency braking, for the first The train, at its time Immediately enter maximum emergency braking state, therefore the safe braking distance of the vehicle in front can be expressed as: ； In the formula, The current speed of the vehicle in front; This represents the maximum emergency braking deceleration value of the vehicle in front. For the A following train may maintain its original traction during the delay phase before entering maximum emergency braking; therefore, the safe braking distance of the following train is crucial. , further expressed as: ； In the formula, For the first The system tracks the current speed of the train. Then the first The train's locomotive and the first Basic safety margin between the rear ends of the train Then the first Train tracking at time The corresponding minimum dynamic safety tracking interval is expressed as: ； The construction of a multi-agent constrained Markov decision model based on the train's operating status data, adjacent train operating status data, minimum dynamic safe tracking interval, target tracking interval, spacing error, and speed coordination error includes: Based on the train's operating status data, adjacent train operating status data, minimum dynamic safe tracking interval, target tracking interval, spacing error, and speed coordination error, the multi-train cooperative tracking control problem is modeled as a multi-agent constrained Markov decision model as follows: ； in, For each intelligent agent's local observation space; For the centralized training phase, the global joint state space is defined. For joint action space; This represents a state transition relationship; For collaborative reward functions; For safety cost function; Discount factor; During the online execution phase, each tracking train makes distributed decisions based solely on its own operational status data and the operational status data of the vector train; for the train in the middle of the queue... The partial observation of the train tracking is represented as follows: ； In the formula, and The first The train tracks the actual distance between itself and the trains in front and behind it; , and These are the speeds of the vehicle in front, the vehicle itself, and the vehicle behind; and This refers to the longitudinal acceleration information of adjacent trains obtained through the vehicle-to-vehicle communication system. During the centralized training phase, all local observations of the tracked trains are combined into a global joint state, denoted as: ； In the formula, Centralized reward value assessment and cost value assessment used only during the training phase; No. The action of each agent is defined as the longitudinal control command of the vehicle within the current control cycle, which will be the first... The agent in the th... The control action output within each control cycle Recorded as: ； In the formula, and The actions of all following trains constitute a combined action, determined by both the train's braking and traction capabilities. for: ； With joint actions Correspondingly, the state transition relationships in multi-agent constrained Markov decision models Represents the given global joint state and joint actions Under the given conditions, the state transition relationship for the evolution of the global joint state of the multi-vehicle queue to the next control cycle is as follows: ； For the For a train tracking system, its single-step local reward function is defined as: ； In the formula, For spacing tracking rewards; For speed-coordinated reward items; This is a stability penalty item; For the The single-step safety cost of a train tracking system is defined as: ； In the formula, For the first The actual tracking distance between the tracking train and the train in front; For the minimum dynamic safety tracking interval; when When it is within the safe and feasible region, the corresponding safety cost is 0; when At that time, the safety cost is taken as the depth of the safety boundary violation; The immediate safety cost is defined as the maximum safety cost of all adjacent train pairs as follows: ； Based on local observation, global joint state, action space, state transition relationships, cooperative reward function, and safety cost function, the objective of multi-vehicle cooperative tracking control is expressed as: maximizing the long-term cumulative cooperative reward of the entire fleet while satisfying the cumulative safety cost constraint, i.e.: ； ； In the formula, For multi-agent joint strategies; The length of a single round of interaction; The allowed cumulative security cost threshold; The method of constructing a dynamic constrained multi-agent joint policy optimization based on the multi-agent constrained Markov decision model using a shared policy network, and updating the shared policy network by pruning the target using proximal policy optimization during centralized training of the dynamic constrained multi-agent joint policy optimization, includes: Let the first The local strategy for train tracking is The optimization of the dynamic constraint multi-agent joint policy using a shared policy network is as follows: ； In the formula, For sharing policy network parameters; For the first Partial observation of the tracking train; For the first The agent in the th... The control actions output within each control cycle; All the actions of the tracking train constitute a joint action; Combine the local observations of all tracked trains into a global joint state; During the centralized training phase, a reward-value network is dynamically constrained to optimize the proximal policy of the multi-agent system. and cost-value network , and Both take the global joint state as input and estimate the expected cumulative reward and expected cumulative cost of the discount, respectively. Based on cumulative safety costs, during centralized training of multi-agent proximal policy optimization with dynamic constraints, in the first... During round training, the following conditions must be met: ； In the formula, For the first The dynamic cost threshold corresponding to each round of training; the scheduling method for the dynamic cost threshold is defined as follows: ； In the formula, This is the initial cost threshold; For the ultimate target cost, The attenuation coefficient; when This indicates that the security risks of the current strategy exceed the requirements of the current stage's constraints. As the cost increases, subsequent policy updates will place greater emphasis on security costs; when This indicates that the strategy has basically met the security constraints of the current stage. With reduced speed, strategy updates can focus more on spacing tracking, speed coordination, and operational stability. Let the reward advantage and cost advantage be respectively and Then the combined Lagrange dominance function can be written as: ； In the formula, This is the overall advantage function used for policy updates; Let the first An intelligent agent at time The probability ratio between the old and new strategies is: ； In the formula, This is the current strategy; The old strategy used when sampling trajectories; combined with the Lagrange dominance function, the optimization objective of the shared policy network for state-constrained multi-agent joint policy optimization is expressed as: ； In the formula, The pruning factor optimized for near-end strategies; Reward Value Network and cost-value network Each update is performed by minimizing the estimation error, and its loss function is expressed as follows: ； ； In the formula, and These are the parameters for the reward value network and the cost value network, respectively. and These are the estimated rewards and costs, respectively. The Lagrange multipliers are updated based on the cumulative safety cost estimate of the current training batch: ； In the formula, The learning rate of the Lagrange multipliers; This is the estimated cumulative cost of the current training batch; if Then increase This makes subsequent policy updates place greater emphasis on security constraints; if Then reduce or maintain a low level. This allows the strategy to further optimize the performance of collaborative tracking control while meeting safety constraints.

2. The train tracking operation cooperative optimization control method according to claim 1, characterized in that, The process by which each tracking train obtains the operating status data of adjacent trains (both preceding and following trains) and its own operating status data based on vehicle-to-vehicle communication includes: Let the first The train was at The position, velocity, and acceleration information for each control cycle are respectively... , and The control cycle is The discrete state update relation of the train is then expressed as: ； ； In the formula, For the first The train was at The actual longitudinal acceleration within each control cycle; and this actual longitudinal acceleration is obtained after constraining the train's traction capacity, braking capacity, and track speed limit. During train operation, the following speed constraints and control input constraints are satisfied: ； ; In the formula, Speed ​​limits were imposed on the line. For the first The train was at Longitudinal control commands within a control cycle and To control the lower and upper limits of the input.

3. The train tracking operation cooperative optimization control method according to claim 1, characterized in that, The process of forming the target tracking interval by superimposing an additional tracking margin on the minimum dynamic safety tracking interval includes: Definition of the first The train was being tracked. The actual distance at any given time always satisfies the following: ； To prevent the tracking train from running close to the dynamic safety boundary for extended periods, an additional tracking margin is introduced on top of the minimum dynamic safety tracking interval. , define the first Train tracking at time The target tracking interval is: ； in, For the first Train tracking at time Target tracking interval; For the first Train tracking at time Corresponding minimum dynamic safety tracking interval; additional tracking margin .

4. The train tracking operation cooperative optimization control method according to claim 1, characterized in that, The step of calculating the actual net distance between trains based on the position information in the adjacent train operation status data and the speed difference between the tracking train and the preceding train includes: For any i A train is being tracked, and the train directly ahead of it is the 1st train. The train will be the first The actual clearance between the following train and the train in front is: ； In the formula, For the first Within the first control cycle, the first The actual net distance between the tracking train and the train in front; The length of the vehicle in front; and These are the location information for the vehicle in front and the vehicle itself.

5. The train tracking operation cooperative optimization control method according to claim 1, characterized in that, The calculation of the spacing error between the actual net distance and the target tracking interval, and the obtaining of the speed coordination error based on the speed difference between the tracking train and the preceding train: For any adjacent train pairs The spacing error is defined as: ； In the formula, For any adjacent train pairs exist The time interval error; For the first The train was being tracked. The actual distance at any given moment remains constant; For the first The train was being tracked. The target tracking interval at any given time; No. Train tracking speed coordination error relative to the preceding train for: ； In the formula, For the first The train was being tracked. Speed ​​at any given moment; For the car in front Speed ​​at any given moment; The objective of multi-train cooperative tracking control can be described as: under the premise of satisfying dynamic safety constraints, gradually reducing the distance error and speed coordination error between adjacent trains, i.e.: 。 6. The train tracking operation cooperative optimization control method according to claim 1, characterized in that, The spacing tracking reward is defined as follows: ； In the formula, As a weight for spacing rewards; This is the spacing error scale parameter; For the first The spacing error of the trains being tracked; The speed-coordination reward item is defined as follows: ； In the formula, Weighting for speed-related rewards; For velocity error scale parameters; For the first The speed error between the tracking train and the train ahead of it; To balance the smoothness of control input with the comfort of train operation, a stability penalty term is introduced, which is defined as follows: ； In the formula, The weight of the stability penalty term; As a comprehensive stability index, it is defined as follows: ； ； In the formula, To control the input normalization constant; To control the rate of change of input or equivalent jerk; This is the normalization constant for jerk. To control the cycle.

7. A train tracking and operation cooperative optimization control device based on safety reinforcement learning, characterized in that, The apparatus for executing the train tracking operation cooperative optimization control method according to any one of claims 1-6, the apparatus comprising: Data acquisition module: Used to obtain the operating status data of adjacent trains and adjacent following trains and the operating status data of the train itself based on vehicle-to-vehicle communication in each control cycle. The operating status data of adjacent trains includes train arrangement information, speed information and acceleration information. The operating status data of the train itself includes the position information, speed information and acceleration information of the train itself. The first calculation module is used to calculate the safe braking distance of adjacent trains under the most unfavorable operating conditions based on the operating status data of adjacent trains and the operating status data of the current train, and to calculate the minimum dynamic safe tracking interval of the current adjacent train pair in combination with the basic safety margin. The second calculation module is used to form a target tracking interval by superimposing an additional tracking margin on the minimum dynamic safety tracking interval, and to calculate the actual net distance between the front and rear trains based on the position information in the adjacent train operation status data and the speed difference between the tracking train and the preceding train, to calculate the spacing error between the actual net distance and the target tracking interval, and to obtain the speed coordination error based on the speed difference between the tracking train and the preceding train. Model building module: used to build a multi-agent constrained Markov decision model based on the train's operating status data, adjacent train operating status data, minimum dynamic safe tracking interval, target tracking interval, spacing error, and speed coordination error; Target update module: used to construct a dynamic constrained multi-agent joint policy optimization based on the multi-agent constrained Markov decision model using a shared policy network, and to update the shared policy network by pruning the target using proximal policy optimization during the centralized training of the dynamic constrained multi-agent joint policy optimization. Model deployment module: used to deploy the trained dynamic constraint multi-agent joint strategy optimization in the on-board control equipment of each tracking train to achieve collaborative optimization control of train tracking operation.

Images