Virtual marshalling train reference curve calculation method based on improved reinforcement learning algorithm

Abstract

The application discloses a virtual marshalling train reference curve calculation method based on an improved reinforcement learning algorithm, applied to the technical field of rail transit operation control, and comprises the following steps: constructing an operation optimization model of a virtual marshalling train based on the mathematical form of virtual marshalling train operation indexes and constraints, and determining decision variables; converting the operation optimization model to a reinforcement learning framework; applying an improved reinforcement learning DQN algorithm to solve the operation optimization model, and obtaining a virtual marshalling train reference curve. The application integrates independent distribution planning of each train unit, generates a respective cooperative reference curve for each vehicle in the vehicle fleet on the basis of considering the operation indexes of the whole train formation, and solves the reference curve through the improved reinforcement learning DQN algorithm, so that the existing independent distribution planning is replaced, the preceding vehicle no longer runs completely independently but considers the operation state of the following vehicle, and the operation indexes of the virtual marshalling train, such as synchronous parking, punctual arrival and accurate parking, are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of rail transit operation control, in particular to a virtual marshalling train reference curve calculation method based on an improved reinforcement learning algorithm. BACKGROUND

[0002] A virtual marshalling train is composed of multiple (including two) train units with independent traction / braking, communication and operation control capabilities, and each train unit can keep a small safe tracking distance. The virtual marshalling train can quickly and flexibly adjust the number of train units in the formation, thereby better meeting the dynamic changes in passenger flow demand, but how to realize the control of the virtual marshalling train to achieve safe operation, on-time arrival, precise parking, synchronous parking, synchronous departure and other train operation goals is still a difficult problem.

[0003] In the existing virtual marshalling train operation control scheme design, an independent step-by-step planning method is usually used, that is, the front train first plans a reference curve independently and controls the train operation, and the rear train receives the position, speed and acceleration of the front train in real time as the tracking target. Under the constraint condition of meeting the safe distance, the reference curve of the rear train is planned, and the rear train is controlled to run. Since the state and control ability of the rear train are not considered when the front train is planned and controlled, and the rear train needs to plan the reference curve in real time based on the control result of the front train, there are problems such as asynchronous arrival at the station, difficulty in real-time tracking of the rear train, and difficulty in secondary planning adjustment of the rear train under actual operation conditions. Specifically:

[0004] First, asynchronous arrival at the station. From the outside, the virtual marshalling train is equivalent to a physically connected train, and all train units in the formation are required to start and stop in unison to better meet the needs of passengers getting on and off the train. However, in actual operation, the distance between trains must always be greater than the safe distance to ensure safety, and the size of the safe distance is determined by the actual running state of the two trains. When running at high speed between stations, the distance between train units will increase due to the influence of the safe distance, and when parking at the station, the distance needs to be shortened to meet the length limit of the platform and the needs of passengers, etc. Therefore, if the front train does not consider the mutual relationship with the rear train when planning its reference curve, it will be difficult for the rear train to park synchronously with the front train while ensuring the safe distance, which will result in the rear train continuing to run for a period of time after the front train independently parks at the station to meet the distance requirement between the two trains parked at the station, thereby causing a large time difference between the parking times of the two trains, which does not meet the needs of urban rail transit passenger organization.

[0005] Second, the rear car is difficult to track in real time. In the current virtual marshalling train operation mode, the rear car can only calculate its control acceleration after receiving the latest state information of the front car, so as to track the operation of the front car. However, in the actual operation scene, there are many factors affecting the car-to-car communication, such as communication delay, signal equipment failure, clock beat error, etc. Once the rear car does not receive or delays to receive the information of the front car, the rear car will not be able to control the train or can only control the train according to the information of the last period, so as to cause the tracking problem. In addition, the calculation of the safety distance also needs the information of the front car, and the safety distance has the characteristics of high-order nonlinearity due to its own characteristics, and the calculation time needs to be considered, and the wrong safety distance will cause more dangerous situations.

[0006] Third, the rear car is difficult to adjust the secondary planning. The planning generates a curve for tracking, but the existence of control error and deviation will cause the train to not run completely according to the curve. In order to improve the curve tracking effect as much as possible, real-time secondary online planning is often added. However, in the operation mode of the front car completely ignoring the operation of the rear car, even if the rear car makes the most beneficial action to track the front car, it cannot guarantee the consistent operation of the whole virtual marshalling train. And in the harsh car-to-car communication environment, real-time secondary optimization may fail due to information delay and loss.

[0007] Therefore, how to provide a virtual marshalling train reference curve calculation method based on an improved reinforcement learning algorithm to overcome the problem that the front car planning and control in independent step-by-step planning do not consider the state and control ability of the rear car, resulting in the problems of asynchronous arrival at the station, difficulty in real-time tracking of the rear car, and difficulty in secondary planning adjustment of the rear car in actual operation conditions is a problem that those skilled in the art need to solve. SUMMARY

[0008] Therefore, the present application provides a virtual marshalling train reference curve calculation method based on an improved reinforcement learning algorithm. By integrating the independent distributed planning of each train unit, a respective cooperative reference curve is generated for each car in the train fleet based on considering the overall operation index of the train formation, and the improved reinforcement learning DQN algorithm is used for solving, replacing the existing independent distributed planning, so that the front car is no longer completely independent and considers the operation state of the rear car, and the operation index of synchronous parking, on-time arrival and accurate parking of the virtual marshalling train is realized.

[0009] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0010] The virtual marshalling train reference curve calculation method based on the improved reinforcement learning algorithm comprises:

[0011] Step (1): based on the mathematical form of the virtual marshalling train operation index and constraint, an operation optimization model of the virtual marshalling train is constructed, and the decision variable is determined;

[0012] Step (2): Transform the running optimization model into a reinforcement learning framework;

[0013] Step (3): Apply the improved reinforcement learning DQN algorithm to solve the running optimization model and obtain the reference curve of the virtual train formation.

[0014] Optional, virtual train formation operation indicators include:

[0015] Simultaneous parking, as follows:

[0016]

[0017] Among them, e s The final time difference between when the car in front and the car behind come to a stop; The final time the car in front stopped; The final time the car behind came to a stop;

[0018] Arrive on time, as shown in the following formula:

[0019]

[0020] in, The final time when the vehicle in front stopped and the time difference with the target; The target time for the vehicle in front to stop;

[0021] Precise parking, as shown below:

[0022]

[0023]

[0024] in, and These are the differences between the final and target positions of the vehicles in front and behind, respectively. and These are the final stopping positions of the car in front and the car behind; and These are the target locations where the car in front and the car behind stopped, respectively.

[0025] Optional, virtual train formation operation constraints include:

[0026] Train dynamics constraints are as follows:

[0027]

[0028]

[0029]

[0030]

[0031] where, and are the position and velocity of the front and rear vehicles at time k, respectively; 1 and m 2 are the mass of the front and rear vehicles, respectively; and are the traction and braking forces of the front and rear vehicles, respectively; and are the base resistance, curve additional resistance, and slope additional resistance of the front and rear vehicles, respectively;

[0032] Driving force constraints are as follows:

[0033]

[0034]

[0035] where, F t and F b are the traction and braking forces, respectively; are the velocity of the front or rear vehicle at time k; and are the maximum traction and braking forces that the front or rear vehicle can provide at velocity

[0036] Road speed limit constraints are as follows:

[0037] The front and rear vehicle velocities satisfy the constraint:

[0038]

[0039]

[0040] where, and are the warning curve limit speeds of the front and rear vehicles, respectively;

[0041] Safety distance constraints are as follows:

[0042]

[0043] sm = min(d(t));

[0044]

[0045] d e + L + k1≤ d(t)≤ d e + L + k2;

[0046] where d(t) is the vehicle distance at any time during the emergency braking process.​ are the headway position, speed and acceleration of the front train before emergency braking; sm is the minimum distance between trains; d e is the safety distance constraint; L is the train length; k1 and k2 are control margins.

[0047] Optionally, the decision variable is the train control acceleration u in each calculation step, as follows:

[0048] u1, u2, u3, …, u n .

[0049] Optionally, the running optimization model is converted into a reinforcement learning framework through a Markov decision process, as follows:

[0050]

[0051] where s0, s1, …, s n are the virtual marshalling train states at each step; u1, u2, …, u n are the control acceleration instructions at each step.

[0052] Optionally, the improved reinforcement learning DQN algorithm is as follows:

[0053] Initialization stage: set the parameters of the improved reinforcement learning DQN algorithm, and initialize a deep neural network as the value function;

[0054] Observation stage: randomly select actions from the initial state to obtain the next state, store the data in the experience space, and continue until the training times reach the preset value;

[0055] Training stage: select high-value actions from the initial state according to the value function and the policy function for iteration, and extract samples from the experience space to train the value function at a preset period, until the training result meets the preset success rate requirement, and the mature value function is obtained;

[0056] Reference curve generation stage: a series of complete start-to-end data is obtained from the initial state using the mature value function and the policy function, and the virtual marshalling train reference curve is obtained.

[0057] Optionally, the parameters of the improved reinforcement learning DQN algorithm are set, and a deep neural network is initialized as the value function, specifically as follows:

[0058] State:

[0059] The position, speed, and time of the virtual marshalling train are taken as state variables, as follows:

[0060]

[0061] where s k is the state of the virtual marshaling train at the kth step; are the position, speed, and time of the front and rear trains, respectively, before and after the kth step;

[0062] The initial state, final state, and target state are respectively:

[0063]

[0064]

[0065]

[0066] where, are the initial position, speed, and time of the front and rear trains, respectively; are the final position, speed, and time of the front and rear trains when they stop, respectively; are the target position, speed, and time of the front and rear trains when they stop, respectively;

[0067] Action:

[0068] The discrete control acceleration encoding value of the front and rear trains is taken as the action, and the control acceleration value of the two trains is obtained by decoding the action. The encoding function is as follows:

[0069]

[0070] where p is the gear number of the front train control acceleration; q is the gear number of the rear train control acceleration; are the control accelerations of the front and rear trains at the kth step; p1 and q2 are the gear order of the current control acceleration; A k is the action value at the kth step;

[0071] Environment:

[0072] It includes train dynamics and track parameters.

[0073] For the front train, the interaction formula between the agent and the environment is as follows:

[0074]

[0075]

[0076]

[0077] where Δt is the time calculation step;

[0078] For the front train, the acceleration contained in the track resistance is calculated according to the track parameters, as follows:

[0079]

[0080] wherein, is the acceleration of the preceding vehicle at the kth step; F t (v) and F b (v) are the traction force and braking force of the preceding vehicle at speed v, respectively; f b , f c , f g are the basic resistance, curve additional resistance, and slope additional resistance, respectively, experienced by the preceding vehicle; m 1 is the mass of the preceding vehicle;

[0081] Value function:

[0082] Q = Q net (s k , A k ), 0≤k≤n;

[0083] wherein, Q net is the value function, is a deep neural network, the input is the state and action, and the output is the Q value of the current state combined with the action, and is the future income generated by the current combination;

[0084] Policy function:

[0085]

[0086] wherein, is the ith action at the kth step;

[0087] Reward function:

[0088]

[0089] e = [e x , e v , e t , e x , e v , e t ];

[0090] wherein, e is the allowable error matrix; R1 is the positive reward given when the virtual marshalling train stops within the required position, speed, and time error; P1 is the negative reward given when the two trains do not stop in the stopping area; P2 is the negative reward given when the two trains do not stop according to the schedule; P3 is the negative reward given when the preceding vehicle or the following vehicle exceeds the road speed limit; P4 is the negative reward given when the distance between the two vehicles is less than the safe distance;

[0091] Experience replay: the data extracted from the experience space cannot be directly used for training because it does not meet the input and output requirements of the neural network. The current estimated future expected value needs to be calculated according to the data, and the processed data can be used to train the neural network according to the learning rate a, as follows:

[0092]

[0093] wherein U k is the future expected value estimated according to the k-th step of data extraction; R k is the reward value in the k-th step of data extraction; and γ is the decay rate.

[0094] Optionally, the improved reinforcement learning DQN algorithm parameters are set, and further include:

[0095] Experience reward: in the algorithm initialization stage, a reference curve data obtained through past training is imported from outside as a basis to add a reward, and a negative reward is given once the exploration curve deviates from the experience data too far, as follows:

[0096]

[0097] wherein B1 and B2 are integers; x w and t w are the position and time in the experience data respectively; g1 and g2 are the set guide ranges;

[0098] Curve evaluation mechanism: if there is no external experience data to be imported, the first curve explored is used as experience data, and a curve evaluation mechanism is adopted to replace a curve with a low score with a curve with a high score, and the score function is as follows:

[0099]

[0100] wherein a, b, and c are weight coefficients for each target of the virtual marshalling train operation.

[0101] Optionally, the improved reinforcement learning DQN algorithm parameters are set, and further include:

[0102] Local traversal mechanism: when the train triggers the boundary, a local traversal selection action is performed within a certain range, and a table recording the illegal action is added to the algorithm to perform local traversal.

[0103] Optionally, the improved reinforcement learning DQN algorithm parameters are set, and further include:

[0104] Action space contraction mechanism:

[0105] When it is determined that the train is in a traction or braking working condition, only the corresponding acceleration or braking instruction is considered;

[0106] When a train stops, the action can only be directed to another train that is still running, and the control acceleration of the stopped train is always zero.

[0107] Compared with the prior art, the application provides a virtual marshalling train reference curve calculation method based on an improved reinforcement learning algorithm. The independent distribution planning of each train unit is integrated, a respective cooperative reference curve is generated for each vehicle in the vehicle fleet based on the overall operation index of the formation, and the improved reinforcement learning DQN algorithm is used for solving, replacing the existing independent distribution planning, so that the front vehicle is no longer completely independently operated but considers the operation state of the rear vehicle, and the operation index of the virtual marshalling train synchronous stopping, on-time arrival and precise stopping is realized. The above idea is realized through the following technical means, including: the operation optimization model of the virtual marshalling train is constructed based on the mathematical form of the virtual marshalling train operation index, including: synchronous stopping, on-time arrival, precise stopping and related constraints, including: train dynamics constraints, driving force constraints, road speed limit constraints and safety distance constraints; the train control acceleration in each calculation step is determined as a decision variable; the constructed virtual marshalling train operation optimization model is converted to the reinforcement learning framework through the Markov decision process, so that the model can be solved using the reinforcement learning algorithm; the improved reinforcement learning DQN algorithm is used for solving, which is roughly divided into an initialization stage, an observation stage, a training stage and a reference curve generation stage, and a cooperative reference curve is obtained; wherein the initialization stage sets the reinforcement learning DQN algorithm parameters, including: state, action, environment, value function, policy function, reward function, experience replay, experience reward, curve evaluation mechanism, local traversal mechanism and action space contraction mechanism. The virtual marshalling train running scene is converted to the reinforcement learning algorithm framework, the improved DQN algorithm is used to explore and optimize, and the cooperative reference curve is finally obtained, which has the following effects: the problem of difficult synchronous stopping caused by the following vehicle tracking the leading vehicle is solved. Through the cooperative curve planning of the leading vehicle and the following vehicle, the running target of the virtual marshalling train is met; the pressure of real-time tracking control of the following train is relieved. The running mode of the following vehicle which depends on real-time tracking control is changed, and the reference curve is added to the following vehicle, which controls the vehicle based on the reference curve, thereby reducing the real-time calculation pressure of the train automatic driving system. At the same time, the operation of the front vehicle can also consider the situation of the rear vehicle due to the reference curve, and the overall formation is more synchronous; the improved reinforcement learning algorithm is used to process the complex multi-vehicle cooperative reference curve optimization problem, which can better solve the problem of high problem dimension and large real-time calculation amount. Through the improvement of the algorithm considering the characteristics of the rail transit running scene, the cooperative reference speed curve meeting the requirements can be found relatively quickly under the premise of ensuring the solving quality. BRIEF DESCRIPTION OF DRAWINGS

[0108] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings required by the embodiments or the prior art description will be briefly introduced as follows. Obviously, the accompanying drawings in the following description only represent some embodiments of the present application, and all other drawings obtained by those of ordinary skill in the art without creative effort based on the provided drawings also belong to the protection scope of the present application.

[0109] Figure 1 The method flowchart of the present application.

[0110] Figure 2 The road speed limit warning curve schematic diagram of the present application.

[0111] Figure 3 The basic logical structure schematic diagram of the improved reinforcement learning DQN algorithm of the present application.

[0112] Figure 4 The description mode structure schematic diagram of the improved reinforcement learning DQN algorithm of the present application for solving the virtual marshalling train cooperative reference curve problem.

[0113] Figure 5 The stage structure schematic diagram of the improved reinforcement learning DQN algorithm of the present application for solving the virtual marshalling train cooperative reference curve.

[0114] Figure 6 The local traversal mechanism principle schematic diagram of the present application.

[0115] Figure 7 The overall logical structure schematic diagram of the improved reinforcement learning DQN algorithm of the present application for solving the virtual marshalling train cooperative reference curve. DETAILED DESCRIPTION

[0116] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments only represent some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort belong to the protection scope of the present application.

[0117] Embodiment 1

[0118] Embodiment 1 of the present application discloses a virtual marshalling train reference curve calculation method based on an improved reinforcement learning algorithm, as shown in Figure 1 , which comprises:

[0119] Step (1): based on the mathematical form of the virtual marshalling train operation index and constraint, an operation optimization model of the virtual marshalling train is constructed, and the decision variable is determined;

[0120] The virtual marshalling train operation index comprises:

[0121] Meanwhile parking, if the parking time difference of two vehicles is too large, it will affect the passenger's getting on and off and other passenger flow scheduling operations during the station entry parking phase, so as to minimize the difference between the parking time of the two vehicles as the goal, as follows:

[0122]

[0123] Wherein, e s is the final time difference between the front and rear vehicles; is the final time of the front vehicle; is the final time of the rear vehicle;

[0124] On-time arrival, the subway timetable is designed according to passenger flow and operating cost and many other factors, and the train runs according to the timetable to maximize passenger demand and save cost, so on the basis of synchronous parking of two vehicles, only the final time and target time difference of the front vehicle parking are taken as the optimization goal, as follows:

[0125]

[0126] Wherein, is the final time and target time difference of the front vehicle; is the target time of the front vehicle;

[0127] Precise parking, only the train door and the platform screen door are aligned to ensure the passenger's getting on and off, which puts forward higher requirements for the parking accuracy of the virtual formation train, so the difference between the final position and the target position of the front and rear vehicles is taken as the minimum goal, as follows:

[0128]

[0129]

[0130] Wherein, and are the difference between the final position and the target position of the front and rear vehicles; and are the final positions of the front and rear vehicles; and are the target positions of the front and rear vehicles.

[0131] The virtual formation train operation constraints include:

[0132] Train dynamics constraints, the state information of the train unit shall satisfy the train dynamics equation based on Newton's second law, as follows:

[0133]

[0134]

[0135]

[0136]

[0137] where, and are the position and velocity of the front and rear vehicle at time k; m 1 and m 2 are the mass of the front and rear vehicle; and are the traction and braking force of the front and rear vehicle; and are the basic resistance, curve additional resistance, and slope additional resistance of the front and rear vehicle;

[0138] Driving force constraint, due to the working characteristics of the motor, the traction force and the commonly used braking force that the train unit can provide at different speeds are limited, which depends on the traction / braking characteristic curve of the vehicle, provided by the vehicle manufacturer, as follows:

[0139]

[0140]

[0141] where, F t and F b are the traction and braking force; are the velocity of the front or rear vehicle at time k; and are the maximum traction and maximum braking force that the front or rear vehicle can provide at velocity

[0142] Road speed limit constraint, the train needs to meet the road speed limit when running on the line, and danger may occur once the train speed exceeds the road speed limit. Especially when transitioning from a high speed limit section to a low speed limit section, entering the braking phase too late will inevitably touch the speed limit. In addition, when transitioning from a low speed limit section to a high speed limit section, in order to prevent the front vehicle train from accelerating directly after leaving the low speed limit section, causing the train spacing to widen. We make the low speed limit section of the front vehicle train. Therefore, on the basis of the road speed limit, a warning curve is generated to ensure that the train can enter the speed reduction section in time to prevent triggering the speed limit, as shown in Figure 2 .

[0143] The front and rear vehicle speed satisfies the constraint:

[0144]

[0145] ​

[0146] wherein, and are the front and rear car warning curve speed limits, respectively;

[0147] Safety distance constraint. In order to ensure that the virtual marshaled train does not become dangerous due to too close distance during operation, the distance between the front and rear cars needs to be constrained. The basic calculation principle of the safety distance constraint is as follows:

[0148] The minimum distance between trains is sm, and once the distance is less than this distance, it is considered to be dangerous. At each time of the formation operation, it is assumed that the two cars trigger emergency braking at the same time, and then the distance between the two cars at any time during the emergency braking process can be obtained, as follows:

[0149]

[0150] As long as the distance between the front and rear cars during the emergency braking process is greater than the minimum distance sm, it can be considered that the current state of the train meets the safety distance constraint. Considering the critical case when the minimum distance during the emergency braking process is exactly equal to sm, the initial distance between the two cars at this time is the safety distance constraint value, as follows:

[0151] Then, when

[0152] sm = min(d(t));

[0153] Safety distance constraint, as follows:

[0154]

[0155] Considering the error in subsequent tracking control, in order to ensure safety, a control margin k1, k2 is added to the safety distance constraint. Then the constraint that the train distance satisfies is as follows:

[0156] d e + L + k1≤ d(t)≤ d e + L + k2;

[0157] wherein d(t) is the distance between the two cars at any time during the emergency braking process; are the front and rear car head positions, speeds, and accelerations before emergency braking, respectively; sm is the minimum distance between trains; d e is the safety distance constraint; L is the car length; and k1 and k2 are control margins.

[0158] Since the virtual marshalling train considers more refined safety constraints and targets during operation, the traditional operating condition switching point optimization method can no longer meet the requirements, and therefore the train control acceleration u in each calculation step is selected as the decision variable. Through a series of control acceleration sequences, the entire information of the virtual marshalling train reference curve can be obtained, as follows:

[0159] u1, u2, u3……u n .

[0160] wherein it is assumed that the entire virtual marshalling train operation is divided into n calculation steps.

[0161] Step (2): The virtual marshalling train operation can be regarded as a discrete process separated according to time calculation steps. In each step, the train operation is controlled by selecting a suitable control instruction until the train meets the target and stops at the platform. This process can be described as a Markov decision process (a multi-step decision process), and the operation optimization model is converted to a reinforcement learning framework through the Markov decision process, as follows:

[0162]

[0163] wherein s0, s1……s n are the virtual marshalling train states at each step, respectively; u1, u2……u n are the control acceleration instructions at each step, respectively.

[0164] Step (3): The improved reinforcement learning DQN algorithm is applied to solve the operation optimization model to obtain the virtual marshalling train reference curve.

[0165] The DQN algorithm is a value-based algorithm in reinforcement learning, and the basic logic architecture is shown in Figure 3 The agent starts from the initial state, selects an action, and interacts with the environment to enter the next state. At the same time, a reward is obtained according to the current state through a reward function, which represents whether the current state meets the human given expectation requirement. The “state-action-state-reward” of one iteration will be stored in the experience space, and data will be extracted from the experience space regularly to train the value function. Through training, the value function learns the knowledge of “what action to choose in what state to obtain a larger reward”, so that in the next iteration, the action with higher value will be selected to make the state change more in line with expectations. The algorithm is repeatedly looped until the state can be stabilized to reach the target state.

[0166] The description of the improved reinforcement learning DQN algorithm for solving the virtual marshalling train collaborative reference curve problem is shown in Figure 4 .

[0167] By setting the reward function artificially to guide the agent to move towards the target state constantly, and converting the process constraints into the environment, the optimization problem under the constraint condition can be solved.

[0168] The improved reinforcement learning DQN algorithm for solving the virtual marshaling train cooperative reference curve is divided into four stages: initialization stage, observation stage, training stage and reference curve generation stage, as shown in Figure 5 The specific process is as follows:

[0169] Initialization stage: set the parameters of the improved reinforcement learning DQN algorithm, and initialize a deep neural network as the value function;

[0170] Observation stage: randomly select actions from the initial state to obtain the next state, store data in the experience space, and repeat until the training times reach the preset value;

[0171] Training stage: select high-value actions from the initial state according to the value function and policy function for iteration, and extract samples from the experience space to train the value function at a preset period, until the training result meets the preset success rate requirement, and the mature value function is obtained;

[0172] Reference curve generation stage: starting from the initial state, a series of complete start-to-end data is obtained by using the mature value function and policy function for iteration, and the virtual marshaling train reference curve is obtained.

[0173] Set the parameters of the improved reinforcement learning DQN algorithm, and initialize a deep neural network as the value function, specifically:

[0174] State:

[0175] Take the position, speed and time of the virtual marshaling train as state variables, as follows:

[0176]

[0177] Where, s k is the state of the k-th step virtual marshaling train; are the positions, speeds and times of the trains before and after k time respectively;

[0178] The initial state, final state and target state are respectively:

[0179]

[0180]

[0181]

[0182] Where, The initial position, speed, and time of the front and rear vehicles, respectively; The final position, speed, and time when the front and rear vehicles stop, respectively; The target position, speed, and time when the front and rear vehicles stop, respectively;

[0183] Action:

[0184] The discrete control acceleration encoding values of the front and rear vehicles are taken as actions, and the control acceleration values of the two vehicles are obtained by decoding the actions. The encoding function is as follows:

[0185]

[0186] Where p is the gear number of the front vehicle control acceleration; q is the gear number of the rear vehicle control acceleration; The control acceleration of the kth step of the front and rear vehicles, respectively; p1 and q2 are the gear order of the current control acceleration; A k The action value of the kth step;

[0187] The specific encoding rules are as follows:

[0188] The control acceleration range of the front vehicle is discretized into m gears, and the control acceleration range of the rear vehicle is discretized into n gears. It is specified that the total number of encoding values is m*n.

[0189] For example: The control acceleration is discretized into 20 gears from -1 to 1, so there are a total of 20*20=400 encoding values. When the front vehicle selects the 1st gear control acceleration and the rear vehicle selects the 2nd gear control acceleration, the encoding value is 1*2=2.

[0190] Environment: Considering that the state is obtained through interaction with the environment to get the next state, analogizing train operation, the environment is equivalent to train dynamics and line conditions. The boundary of the environment is the road speed limit, virtual marshalling safety distance, and train reversing hard constraint conditions.

[0191] Including: train dynamics and line parameters;

[0192] For the front vehicle, the interaction formula between the agent and the environment is as follows:

[0193]

[0194]

[0195]

[0196] Where Δt is the time calculation step;

[0197] For the front vehicle, the line resistance contained acceleration is calculated according to the line parameters, as follows:

[0198]

[0199] wherein, is the acceleration of the preceding vehicle at the kth step; F t (v) and F b (v) are the traction force and braking force of the preceding vehicle at the speed v, respectively; f b , f c , f g are the basic resistance, curve additional resistance, and slope additional resistance, respectively, that the preceding vehicle is subjected to; m 1 is the mass of the preceding vehicle;

[0200] Value function: the value function of the DQN algorithm is a deep neural network, and the state-action value function is used in the present application. The input of the neural network is the state and action, and the output is the Q value of the current state combined with the action. The greater the Q value, the higher the future income generated by the current combination. As follows:

[0201] Q = Q net (s k , A k ), 0≤k≤n;

[0202] wherein, Q net is the value function, which is a deep neural network, the input of which is the state and action, and the output is the Q value of the current state combined with the action, and the future income generated by the current combination;

[0203] The neural network needs to be constructed according to the complexity of the problem. Generally, it is 3-6 layers with 20-100 neurons per layer. It needs to be adjusted according to the actual neural network fitting effect.

[0204] Policy function: the DQN algorithm is a value-based algorithm, and the policy function is relatively fixed. The present application uses a deterministic policy function, the input of which is the current state, and the output is the optimal action value. The rule for selecting the action is to input all the actions combined with the current state into the value function, and select the action in the value maximum combination as the optimal strategy. The state input each time will be combined with all possible actions and input into the value function to calculate the Q value. The action in the "state-action" combination with the maximum Q value is selected as the output of the function, as follows:

[0205]

[0206] wherein, is the kth action of the i th kind at the kth step;

[0207] Reward function: the reward function is the most core part of the DQN algorithm. The reward represents the expected change direction of the state. A reasonable reward function can greatly speed up the algorithm and quickly guide the agent to achieve the goal. In combination with the goal of virtual marshalling train operation and related constraints, the principles for designing the reward are as follows:

[0208] A positive reward is given when the train state reaches the target state within the allowable range.

[0209] A negative reward is given when the train state reaches the target state outside the allowable range.

[0210] A negative reward is given when the train state violates the constraint condition.

[0211] The specific functions are as follows:

[0212]

[0213] e = [e x ,e v ,e t ,e x ,e v ,e t ];

[0214] where e is the allowable error matrix; R1 is a positive reward given when the virtual marshalling train stops at the required position, speed, and time error; P1 is a negative reward given when the two trains do not stop in the stopping area; P2 is a negative reward given when the two trains do not stop according to the schedule; P3 is a negative reward given when the front or rear train exceeds the road speed limit; P4 is a negative reward given when the distance between the two trains is less than the safe distance;

[0215] Experience replay: "state-action-state-reward" data in the algorithm process is constantly stored in the experience space, and a part of the data is extracted from it every period of time to let the value function (deep neural network) learn. The judgment ability of the value function is constantly strengthened, so that it can accurately judge the best action that should be taken in the current state, as follows:

[0216] The data extracted from the experience space does not meet the input and output requirements of the neural network and cannot be directly used for training. The current estimated future expected value needs to be calculated according to the data, and the processed data can be used to train the neural network according to the learning rate α, as follows:

[0217]

[0218] where U k is the estimated future expected value according to the kth step of data extraction; R k is the reward value in the kth step of data extraction; γ is the decay rate.

[0219] By describing the algorithm logic, it can be found that there is a problem that the process reward of the algorithm is too small, which will lead to the exploration of the agent being blind. Because the number of steps from the starting point to the end point is too many and the action space dimension is also high, the agent is difficult to reach the target state. In view of this, the following improved DQN algorithm meets the use requirements of the virtual marshalling train scene, as follows:

[0220] Experience reward: In the algorithm initialization stage, a reference curve data obtained by past training is imported from outside as the basis to add reward. Once the exploration curve deviates from the experience data too far, a negative reward will be given to ensure that the agent can be guided immediately throughout the whole process, as follows:

[0221]

[0222] Wherein, B1 and B2 are integers; x w and t w are the position and time in the experience data respectively; g1 and g2 are the set guiding range;

[0223] Curve evaluation mechanism: If there is no external experience data to import, the first curve explored will be used as experience data. In order to ensure the iteration optimization, a curve evaluation mechanism is also designed. The obtained multiple successful curves are scored, and the curve with a high score will replace the curve with a low score to ensure that the exploration result is constantly optimized. The scoring function is as follows:

[0224]

[0225] Wherein, a, b, c are weight coefficients for each target of the virtual marshalling train running.

[0226] Local traversal mechanism: In the process of exploration of the agent, the environment boundary will be inevitably touched. At this time, the traditional processing method is to obtain a negative reward and return to the starting point directly. The second is to retreat one step and reselect the action. However, in the field of rail, the train strictly follows the train dynamics, and the driving force is also limited. Retreating one step may still not be able to avoid triggering the boundary. Directly returning to the starting point will reduce the efficiency of the algorithm, so a local traversal mechanism is designed. When the train triggers the boundary, the action will be selected within a certain range, as shown in Figure 6 . A table recording the illegal action is added in the algorithm to perform local traversal, ensuring that the train can run to the stop in one training and will not return to the starting point.

[0227] Action space contraction mechanism: In some specific running scenarios, some actions in the action space of the virtual marshalling train are obviously "unreasonable" and can be artificially reduced to speed up the search process of the curve, as follows:

[0228] When it is determined that the train is in traction or braking condition, only the corresponding acceleration or braking instruction is considered;

[0229] When a train stops, the action can only be directed to another train that is still running, and the acceleration of the stopped train is always zero.

[0230] After setting the key elements of the DQN algorithm, the training and learning according to the algorithm program can be started, and the improved reinforcement learning DQN algorithm is constantly optimized to solve the overall logical architecture of the virtual marshalling train cooperative reference curve, as shown in Figure 7 , specifically as follows:

[0231] Set the parameters of the DQN algorithm itself: initialize the neural network, set the number of steps for each training, the length of the observation period, the length of the training period, set the data normalization principle, the size of the experience space, the experience playback period, the number of experience playback samples, the target network update period, the decay rate, the learning rate. Build an action violation record table. Note: Parameters are manually adjusted according to specific training feedback to achieve better training results.

[0232] Set the parameters related to the operation of the virtual marshalling train: set the target state according to the inter-station operation requirements, import the line parameters, set the parking error, punctuality error and synchronization parking tolerance error parameters. If there is successful curve experience data, import it.

[0233] Observation period: randomly select actions from the initial state to obtain the next state, store data in the experience space. Exit when the number of training times reaches the set value.

[0234] Training period: From the initial state, select actions according to the value function and policy function, and extract samples from the experience space according to the set period to train the value function, so that the value judgment of the value function meets the target of virtual marshalling train operation. Stop training when the training success rate reaches more than 95%, and take out the current value function.

[0235] Reference curve generation: From the initial state, use the trained value function to select actions with the policy function until the virtual marshalling train reaches the target state, i.e. the cooperative reference speed curve is obtained.

[0236] At this point, the generation of the virtual marshalling train cooperative reference speed curve for the actual inter-station scene is complete, and the reference curve will be used as the input of the train unit automatic driving system for subsequent use.

[0237] The embodiment of the application discloses a virtual marshalling train reference curve calculation method based on an improved reinforcement learning algorithm. The independent distribution planning of each train unit is integrated, a respective cooperative reference curve is generated for each vehicle in the vehicle fleet based on the consideration of the overall operation index of the train, and the improved reinforcement learning DQN algorithm is used for solving, so that the existing independent distribution planning is replaced, the operation of the front vehicle is no longer completely independent, but the operation state of the rear vehicle is considered, and the operation index of the virtual marshalling train synchronous parking, punctual arrival and accurate parking is realized. And the above idea is realized through the following technical means, including: the operation index of the virtual marshalling train, including: synchronous parking, punctual arrival, accurate parking and related constraints, including: train dynamics constraint, driving force constraint, road speed limit constraint, safety distance constraint are transformed into mathematical form to build the operation optimization model of the virtual marshalling train, and the train control acceleration in each calculation step is determined as a decision variable; the built operation optimization model of the virtual marshalling train is transformed into the reinforcement learning framework through the Markov decision process, so that the model can be solved by using the reinforcement learning algorithm; the improved reinforcement learning DQN algorithm is used for solving, and the cooperative reference curve is obtained, which includes: initialization stage, observation stage, training stage, reference curve generation stage; wherein, the initialization stage sets the reinforcement learning DQN algorithm parameters, including: state, action, environment, value function, policy function, reward function, experience replay, experience reward, curve evaluation mechanism, local traversal mechanism and action space contraction mechanism.

[0238] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0239] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the application. Therefore, the application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for calculating a virtual marshalling train reference curve based on an improved reinforcement learning algorithm, characterized in that, The application relates to a method for generating a virtual marshalling train reference curve based on a reinforcement learning algorithm. The method comprises the following steps: Step (1): constructing a virtual marshalling train operation optimization model based on a virtual marshalling train operation index and a constraint mathematical form, and determining a decision variable; Step (2): converting the operation optimization model into a reinforcement learning framework; Step (3): applying an improved reinforcement learning DQN algorithm to solve the operation optimization model, and obtaining a virtual marshalling train reference curve. The improved reinforcement learning DQN algorithm comprises the following steps: An initialization stage: setting improved reinforcement learning DQN algorithm parameters, and initializing a deep neural network as a value function; An observation stage: starting from an initial state, randomly selecting an action to obtain a next state, storing data into an experience space, and stopping until the number of training times reaches a preset value; A training stage: selecting a high-value action according to a value function and a policy function for iteration, extracting samples from the experience space according to a preset period to train the value function, and stopping until the training result meets a preset success rate requirement, and obtaining a mature value function; A reference curve generation stage: starting from an initial state, using the mature value function and the policy function to obtain a series of complete start-to-end data, and obtaining a virtual marshalling train reference curve. The setting of the improved reinforcement learning DQN algorithm parameters and the initialization of the deep neural network as the value function comprises the following steps: A state: wherein, is the the state of the virtual marshaled train; respectively, the position, speed, and time of the train before and after the time Taking the position, speed and time of the virtual marshalling train as state variables, the state variables are as follows: wherein, respectively the initial position, velocity, time of the front and rear vehicles; respectively the final position, velocity, time when the front and rear vehicles stop; respectively the target position, velocity, time when the front and rear vehicles stop; An initial state, a final state and a target state are as follows: An action: wherein, is the number of gears for the front vehicle to control acceleration; is the number of gears for the rear vehicle to control acceleration; is the control acceleration of the first step of the front and rear vehicles, respectively; is the control acceleration of the first step of the front and rear vehicles, respectively; is the gear order for the current control acceleration; is the action value of the first step; Taking the discrete control acceleration coding value of the front and rear trains as the action, the control acceleration values of the two trains are obtained by decoding the action, and the coding function is as follows: An environment: Including train dynamics and line parameters; wherein is the time step size; For the front train, the interaction formula between the agent and the environment is as follows: wherein, is the first is the acceleration of the preceding vehicle; and are the tractive force and braking force of the preceding vehicle, respectively, at a speed ; , , are the basic resistance, the curve additional resistance, the slope additional resistance, respectively, to which the preceding vehicle is subjected; is the mass of the preceding vehicle; For the front train, the acceleration contained in the line resistance is calculated according to the line parameters, and the formula is as follows: wherein, is a value function, is a deep neural network with inputs being the state and action, and output being the value, is the future return generated by the current combination; A value function: wherein is a first step is a first action; A policy function: wherein, is the error matrix; is a positive reward given when the virtual consist train stops within the required position, speed, and time error; is a negative reward given when neither car stops in the stopping area; is a negative reward given when neither car stops according to the schedule; is a negative reward given when either the leading car or the trailing car exceeds the road speed limit; is a negative reward given when the distance between the two cars is less than the safe distance. Experience replay: the data drawn from the experience space does not meet the input and output requirements of the neural network and cannot be directly used for training. The current estimated future expected value needs to be calculated according to the data, and the processed data can be used for training according to the learning rate The neural network is trained as follows: wherein, is according to the first the future expected value estimated from the data at step is the first the reward value in the data at step is the decay rate. 2.The virtual marshalling train reference curve calculation method based on the improved reinforcement learning algorithm according to claim 1, characterized in that, A reward function: The virtual marshalling train operation index comprises the following: wherein, is the final time difference for the front and rear vehicles to stop; is the final time for the front vehicle to stop; is the final time for the rear vehicle to stop; Simultaneous stopping, as follows: wherein, is the final time of the preceding vehicle and the target time difference; is the target time of the preceding vehicle; On-time arrival, as follows: wherein, and respectively are the difference between the final position and the target position where the preceding and following vehicles are parked; and respectively are the final positions where the preceding and following vehicles are parked; and respectively are the target positions where the preceding and following vehicles are parked. 3.The virtual marshalling train reference curve calculation method based on the improved reinforcement learning algorithm according to claim 1, characterized in that, Precise stopping, as follows: The virtual marshalling train operation constraint comprises the following: wherein, , and , are the positions and velocities of the front and rear vehicles at the time t; and are the masses of the front and rear vehicles; , and , are the tractive and braking forces of the front and rear vehicles; , , and , , are the basic resistances, curve additional resistances, and slope additional resistances of the front and rear vehicles;​ Train dynamics constraint, as follows: in, and These are traction force and braking force, respectively. For the car in front or behind The speed of time; and The speed of the vehicle in front or behind is respectively The maximum traction and braking force that can be provided at that time; Driving force constraint, as follows: Road speed limit constraint, as follows: wherein, and are the front and rear vehicle warning curve speed limits, respectively. Front and rear train speed constraint: wherein, is the vehicle headway at any time during the emergency braking process; are the vehicle headway, speed, and acceleration, respectively, of the front and rear vehicles before emergency braking; is the minimum headway between trains; is the safety headway constraint; is the vehicle length; and is the control margin. 4.The virtual marshalling train reference curve calculation method based on the improved reinforcement learning algorithm according to claim 1, wherein, The decision variable is the train control acceleration within each calculation step As follows: 。 5.The virtual marshalling train reference curve calculation method based on the improved reinforcement learning algorithm according to claim 1, wherein, Safety distance constraint, as follows: wherein, are the virtual consist states for each step, respectively; are the control acceleration commands for each step, respectively. 6.The virtual marshalling train reference curve calculation method based on improved reinforcement learning algorithm according to claim 1, wherein, The operation optimization model is converted into the reinforcement learning framework through a Markov decision process, and the formula is as follows: The setting of the improved reinforcement learning DQN algorithm parameters further comprises the following steps: wherein, and are integers; and are the position and time in empirical data, respectively; and are the set guidance range; Experience reward: in the algorithm initialization stage, a reference curve data obtained through previous training is imported as a basis to add a reward, and a negative reward is given once the exploration curve deviates from the experience data too far, and the formula is as follows: wherein, 、 、 is the weight coefficient for each target of virtual marshalling train operation.

7. The virtual marshalling train reference curve calculation method based on the improved reinforcement learning algorithm according to claim 1, characterized in that, Curve evaluation mechanism: if there is no external experience data to be imported, the first curve explored is used as experience data, and a curve evaluation mechanism is adopted, so that a curve with a high score replaces a curve with a low score, and the scoring function is as follows: The setting of the improved reinforcement learning DQN algorithm parameters further comprises the following steps: Local traversal mechanism: when the train triggers the boundary, local traversal is performed within a certain range to select an action, and a table recording the illegal action is added in the algorithm to perform local traversal. 8.The virtual marshalling train reference curve calculation method based on the improved reinforcement learning algorithm according to claim 1, wherein, Setting the improved reinforcement learning DQN algorithm parameters, further comprising: Action space contraction mechanism: When it is determined that the train is in traction or braking condition, only the corresponding acceleration or braking instruction is considered; When a train stops, the action can only be directed to another train that is still running, and the control acceleration of the stopped train is always zero.

Images