A train precise parking control method based on deep reinforcement learning
By employing deep reinforcement learning methods and utilizing offline imitation learning and online optimization based on historical ATO data, the problem of inaccurate stopping accuracy of trains at low speeds using air brakes was solved, achieving high-precision stopping control and reducing debugging time.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI ELECTRIC THALES TRANSPORTATION AUTOMATION SYST CO LTD
- Filing Date
- 2022-09-06
- Publication Date
- 2026-05-29
Smart Images
Figure CN115496201B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of rail transit technology, and in particular to a method, device and control system for precise train stopping control based on deep reinforcement learning. Background Technology
[0002] Subways play a vital role in urban public transportation, and their level of intelligence is increasing, with a growing proportion of fully automated driverless lines. On driverless lines, the lack of a driver places even higher demands on the precision of ATO (Automatic Train Control) at station stops. Therefore, optimizing train control algorithms has always been a key research focus for ATO software to achieve precise train stops.
[0003] Train control algorithms are typically tested on a train simulation model first, and only after achieving the expected results are they tested on a real train. A train is a nonlinear dynamic system composed of complex technical equipment, operating in a complex environment, and exhibiting complex spatiotemporal distribution characteristics. During the final stop phase, the train brakes switch from electric braking to air braking, which is a system with significant lag, nonlinearity, and a degree of randomness.
[0004] Typically, a target curve is designed using classical control methods, along with a tracking controller to achieve automatic driving. The tracking controller often employs the classic PI control algorithm to track the target curve. PI control performs well during traction or electric braking; however, in the low-speed air braking phase, due to the large delay, nonlinearity, and randomness of air braking, the actual train speed cannot closely match the target curve as expected, resulting in a large range of train stopping accuracy.
[0005] The conventional approach to train simulation is to simplify the train into a linear dynamic system, described by dynamic equations to meet the requirements of control algorithm research. However, due to these significant simplifications, the simulation model struggles to accurately simulate the braking system's performance during the final stop at a station, hindering the effective simulation of stop control algorithms. Even after testing on the simulation model, the stop control algorithm still requires extensive real-vehicle debugging to improve the algorithm and refine its parameters. Real-vehicle debugging involves multi-disciplinary collaboration, is labor-intensive, and is not conducive to algorithm improvement. Furthermore, the effectiveness of existing stop control algorithms is correlated with vehicle characteristics and track conditions; migrating algorithm parameters from one existing line to a new line typically requires re-tuning.
[0006] The train control process typically consists of four stages: acceleration, cruising, coasting, and braking. The accuracy of a train's stop is only related to braking; how the train is controlled before braking has virtually no impact on stopping accuracy. During high-speed braking, the electric braking system is active, while the air brakes are inactive; during low-speed braking, the electric braking is inactive, and the air brakes are active. The electric braking system has a fast response and good consistency, and can follow the control commands of the PI controller very well. The air braking system, however, has a slow response and high randomness, and the PI controller cannot control it well, leading to significant deviations in stopping accuracy. Summary of the Invention
[0007] To address the aforementioned issues, this application proposes a method, device, and control system for precise train stopping control based on deep reinforcement learning.
[0008] This application proposes a method for precise train stopping control based on deep reinforcement learning, comprising the following steps:
[0009] Collect ATO (Automatic Train Operation) stop data and preprocess it to obtain expert dataset X;
[0010] Based on the expert dataset X, imitation learning of behavior cloning is performed to initialize the train's policy network;
[0011] The policy network is trained online using a deep policy gradient determination method to obtain a deep optimized network.
[0012] The deep optimization network is output and saved, and then used for precise train stopping control.
[0013] As an optional implementation of this application, optionally, ATO parking data is collected and preprocessed to obtain an expert dataset X, including:
[0014] Collect ATO stop data for all existing lines;
[0015] According to the preset data preprocessing rules, the ATO stop data is cleaned and segmented to obtain ATO stop preprocessed data;
[0016] From the ATO stop preprocessing data, the data of the final braking stage of each ATO stop is extracted, and the data whose stopping trajectory meets the expectations are selected to form an expert dataset X.
[0017] As an optional implementation of this application, optionally, imitation learning of behavior cloning is performed based on the expert dataset X to initialize the train's policy network, including:
[0018] Define a loss function, and use the loss function to calculate the error between the policy network output and the actual expert actions in the expert dataset X; wherein the loss function is:
[0019]
[0020] Wherein, the loss function L(s, a; θ) is the single-sample error, used to describe the error between the output value of the policy network with parameter θ in the input state s and the output a in the expert database in the same state s.
[0021] symbol This means "defined as", that is, the expression on the left is defined by the expression on the right;
[0022] [π(s;θ)-a] 2 It is the square of the error between the output of the policy network π(s;θ) and the actual expert action a. Multiplying it by 1 / 2 is to eliminate the constant term 2 when taking the derivative of gradient descent.
[0023] Define a cost function, and use this cost function to calculate the sum of the loss functions of all sample points along an entire trajectory in the expert database X. Then, sum the losses of all trajectories to obtain the total cost function J(θ); where the cost function is:
[0024]
[0025] Update the neural network parameters θ using the following gradient descent formula until the algorithm converges:
[0026]
[0027] in:
[0028] θ now : Current neural network parameters;
[0029] θ new : Updated neural network parameters;
[0030] β: Learning rate, a hyperparameter that can be set;
[0031] The cost function J(θ) at θ now The gradient value at that point;
[0032] The policy network is initialized using the neural network parameters θ.
[0033] As an optional implementation of this application, the policy network can be trained online using a deep policy gradient determination method to obtain a deeply optimized network, including:
[0034] Preset depth determination strategy gradient method;
[0035] Based on the aforementioned deep policy gradient determination method, a deep optimization architecture for the policy network and value network is constructed, resulting in a deep optimization network; wherein, in the deep optimization network:
[0036] The state s of the train reinforcement learning framework is used as the input to the policy network, and the policy network outputs action a.
[0037] The state s of the train reinforcement learning framework and the action a output by the policy network are used as inputs to the value network, and the value network outputs the value q(s, a; w).
[0038] The deep optimization network is optimized and updated so that its policy network and value network learn in accordance with the characteristics of the train.
[0039] As an optional implementation of this application, the depth-determined policy gradient method may be a depth-determined policy gradient algorithm based on the Actor-Critic architecture.
[0040] As an optional implementation of this application, the policy network may be updated using Monte Carlo round-based updates or temporal differential updates.
[0041] As an optional implementation of this application, the method for updating the policy network of the deep optimization network may be as follows:
[0042] In the final stage of the train's braking at the station, a policy network π(s; θ) with fixed parameters controls the train's stopping. The complete trajectory controlled by the policy network has T steps, and the immediate reward for each step before the train stops is 0, i.e.: r1 = r2 = ... = r T-1 =0;
[0043] Determine whether the train's stopping accuracy D is within the specified range:
[0044] If the train stopping accuracy D is within the specified range, the final step of the trajectory receives a reward of 100 - |D|, i.e.: r T =100-|D|;
[0045] If the train stopping accuracy D is not within the specified range, the reward for the last step of the trajectory is -(100+|D|), that is: r T =-(100+|D|);
[0046] If the train stopping accuracy D is within the specified range, the cumulative reward for each step to the end of the trajectory is: u1 = u2 = ... = u T=100-|D|;
[0047] If the train stopping accuracy D is not within the specified range, the cumulative reward for each step to the end of the trajectory is: u1 = u2 = ... = u T =-(100+|D|);
[0048] Calculate the cost function:
[0049]
[0050] Update parameters:
[0051]
[0052] in:
[0053] w now : Current neural network parameters;
[0054] w new : Updated neural network parameters;
[0055] α: Learning rate, is a hyperparameter that can be set;
[0056] The cost function J(w) in w now The gradient value at that point.
[0057] As an optional implementation of this application, the method for updating the value network of the deeply optimized network may be as follows:
[0058] With fixed value network parameters w, a policy network is used to control train stops at stations.
[0059] Calculate the policy gradient through backpropagation:
[0060]
[0061] Gradient ascent to update parameters:
[0062]
[0063] in:
[0064] θ now : Current neural network parameters;
[0065] θ new : Updated neural network parameters;
[0066] β: Learning rate, is a hyperparameter that can be set;
[0067] Operator g(θ) at θ now The gradient value at that point.
[0068] In another aspect, this application proposes an apparatus for implementing the aforementioned deep reinforcement learning-based train precise stopping control method, comprising:
[0069] The ATO data acquisition module is used to collect ATO stop data and preprocess it to obtain expert dataset X;
[0070] An initialization module is used to perform imitation learning of behavior cloning based on the expert dataset X, and to initialize the train's policy network;
[0071] The network optimization module is used to train the policy network online using a deep policy gradient determination method to obtain a deep optimized network.
[0072] The train control module is used to output and save the deep optimization network, and to use the deep optimization network for precise train stopping control.
[0073] In another aspect, this application also proposes a control system, comprising:
[0074] processor;
[0075] Memory used to store processor-executable instructions;
[0076] The processor is configured to implement the deep reinforcement learning-based train precise stopping control method when executing the executable instructions.
[0077] Technical effects of the present invention:
[0078] This application collects and preprocesses ATO (Automatic Train Operation) stopping data to obtain an expert dataset X; based on the expert dataset X, it performs imitation learning through behavior cloning to initialize the train's policy network; it trains the policy network online using a deep policy gradient determination method to obtain a deep optimized network; it outputs and saves the deep optimized network and uses it for precise train stopping control. This achieves the following technical effects:
[0079] Make full use of the existing line's ATO historical data and conduct offline imitation learning; rely on the generalization ability of neural networks to decouple the coupling relationship between PI control parameters and vehicle characteristics.
[0080] This technology can reduce the online debugging time of ATO software on real vehicles and improve the accuracy of stopping at stations; the policy network and value network of reinforcement learning will adjust as the train characteristics change, achieving the effect of lifelong learning.
[0081] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description
[0082] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.
[0083] Figure 1 The diagram illustrates the implementation process of the train precise stopping control method based on deep reinforcement learning according to the present invention.
[0084] Figure 2 The diagram shown is a control schematic of the vehicle controller and the train control interface of the present invention;
[0085] Figure 3 The diagram shown is a structural schematic of the reinforcement learning framework of the present invention;
[0086] Figure 4 The diagram shows the architecture of the strategy network for the train of the present invention.
[0087] Figure 5 The diagram shown is a schematic representation of the deeply optimized architecture of this invention. Detailed Implementation
[0088] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.
[0089] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.
[0090] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.
[0091] Example 1
[0092] First, it is necessary to understand the train's control principles. For example... Figure 2The diagram shows the control interface between the onboard controller and the train. The onboard controller outputs traction and braking commands to the train, requesting traction or braking. It also outputs an 8-bit digital signal to a D / A converter, which converts the 8-bit digital signal into a 4-20mA current signal and outputs it to the train. The onboard controller fuses information from speed sensors, acceleration sensors, and positioning antennas to calculate the train's positioning in real time. Based on this positioning, it retrieves gradient and speed limit information from the track database. The speed, acceleration, and other real-time calculated status information obtained by the onboard controller from sensors are used as input information for the ATO module, which is then used for real-time train control.
[0093] This invention proposes a deep reinforcement learning algorithm to improve the robustness of the control model, reduce real-vehicle debugging time, and enhance ATO (Automatic Train Operation) stopping accuracy. Based on this, using an expert dataset X, an offline imitation learning method is also proposed. This method initializes the behavior policy network by cloning and imitating historical ATO arrival data from successful ATO processes; then, reinforcement learning is used to optimize the policy network online. Specifically:
[0094] like Figure 1 As shown, this application proposes a train precise stopping control method based on deep reinforcement learning, including the following steps:
[0095] S1. Collect ATO parking data and preprocess it to obtain expert dataset X;
[0096] Based on a large amount of good historical ATO (Automatic Train Operation) data, a behavior cloning method is used to initialize the policy network, and a deep deterministic policy gradient algorithm model is further used to optimize the behavior policy network. As an optional implementation of this application, optionally, ATO parking data is collected and preprocessed to obtain an expert dataset X, including:
[0097] Collect ATO stop data for all existing lines;
[0098] According to the preset data preprocessing rules, the ATO stop data is cleaned and segmented to obtain ATO stop preprocessed data;
[0099] From the ATO stop preprocessing data, the data of the final braking stage of each ATO stop is extracted, and the data whose stopping trajectory meets the expectations are selected to form an expert dataset X.
[0100] Currently, existing lines control trains using PI control algorithms, and the onboard controllers record a large amount of logs. From these existing logs, a robust ATO (Automatic Train Operation) stop database can be established using the following methods:
[0101] The first method involves using automated scripts to segment the vehicle logs according to station-to-station data, with each data entry containing a complete process from departure to arrival at the station.
[0102] The second method utilizes proximity disc information from scripts and logs to accumulate the distance the train travels from the moment the proximity disc is detected until the train stops. This distance is used to determine the train's stopping accuracy. If the stopping accuracy is within a certain range (e.g., ±20cm), the stop is considered good and included in the good ATO stopping database. (The proximity disc's detection range is ±0.5m).
[0103] In this embodiment, ATO (Automatic Train Operation) stop data from all existing lines is collected first, and then the data is cleaned and segmented to select well-performing trajectories to form an expert dataset. Specifically, the data from the final braking phase of each ATO stop is extracted, and well-performing trajectories are selected to form an expert dataset X. In the expert dataset X, the tuple (s, a) means that a traditional good PI (Physical Induction) algorithm performs action a in state s. In this embodiment, the data cleaning and segmentation methods are not limited and can be handled by the user.
[0104] S2. Based on the expert dataset X, perform imitation learning of behavior cloning to initialize the train's policy network;
[0105] Imitation learning of behavior cloning is performed on the expert dataset X to initialize the policy network for reinforcement learning.
[0106] like Figure 3 As shown, the reinforcement learning framework for controlling the train is illustrated, and the meaning of each framework unit is explained as follows:
[0107] Agent: The vehicle controller (VOBC) that executes the policy network;
[0108] Action: The action information consists of traction commands, braking commands, and position levels;
[0109] Environment: Trains, tracks, and other subsystems that affect train operation;
[0110] State: Inputs such as train speed, acceleration, target distance, gradient, and approach disk signal are used as state information;
[0111] Reward: The reward is directly related to the stopping accuracy. If the train stopping accuracy D is within the specified range (e.g., ±20cm), the entire trajectory receives a reward of 100-|D|; if the train stopping accuracy D is not within the specified range, the entire trajectory receives a reward of –(100+|D|).
[0112] Trajectory: A trajectory is a continuous sequence of states and actions. Before reaching the platform stopping point, a traditional PI algorithm is used; from 5 meters away, a deep reinforcement learning algorithm is applied. The trajectory begins when the reinforcement learning algorithm executes and ends after the train stops and the ATO (Automatic Train Control) relinquishes control.
[0113] By mapping each part to the ATO deep reinforcement learning-based stop control algorithm, we obtain the following: Figure 4 The policy network of the reinforcement learning algorithm is shown.
[0114] like Figure 4 As shown, this network utilizes the architectural information of the aforementioned reinforcement learning framework: the train's speed, acceleration, gradient, target distance, approach disk signal, and positioning error are used as inputs as State information. This State information is used as the input layer of a fully connected neural network, and the tanh activation function is chosen for the output layer, limiting the output range to [-1, 1]. Then, scaling is applied, mapping it to the Action space range [-255, 255] (D / A converter precision). An output action of 0 indicates a coasting command, a positive output indicates the corresponding traction level, and a negative output indicates the corresponding braking level.
[0115] The train's policy network is initialized using an imitation learning method based on behavior cloning. The "state s - expert action a" pairs from the expert dataset X are extracted separately and used as supervised training data. A deterministic policy network can then be trained using classic supervised learning methods. The training process only requires using the expert data to iteratively minimize the loss function; no interaction with the environment is needed. Specifically,
[0116] As an optional implementation of this application, optionally, imitation learning of behavior cloning is performed based on the expert dataset X to initialize the train's policy network, including:
[0117] Define a loss function, and use the loss function to calculate the error between the policy network output and the actual expert actions in the expert dataset X; wherein the loss function is:
[0118]
[0119] Wherein, the loss function L(s, a; θ) is the single-sample error, used to describe the error between the output value of the policy network with parameter θ in the input state s and the output a in the expert database in the same state s.
[0120] symbol This means "defined as", that is, the expression on the left is defined by the expression on the right;
[0121] [π(s;θ)-a] 2 It is the square of the error between the output of the policy network π(s;θ) and the actual expert action a. Multiplying it by 1 / 2 is to eliminate the constant term 2 when taking the derivative of gradient descent.
[0122] Define a cost function, and use this cost function to calculate the sum of the loss functions of all sample points along an entire trajectory in the expert database X. Then, sum the losses of all trajectories to obtain the total cost function J(θ); where the cost function is:
[0123]
[0124] Update the neural network parameters θ using the following gradient descent formula until the algorithm converges:
[0125]
[0126] in:
[0127] θ now : Current neural network parameters;
[0128] θ new : Updated neural network parameters;
[0129] β: Learning rate, a hyperparameter that can be set;
[0130] The cost function J(θ) at θ now The gradient value at that point;
[0131] The policy network is initialized using the neural network parameters θ.
[0132] In this section, behavior cloning optimizes the policy network π(s; θ) using 'a' as the label, and trains the policy network using a regression method in supervised learning. The policy network is initialized with the parameters θ obtained from the behavior cloning method, and it already possesses the ability to control train stopping at stations.
[0133] In this embodiment, to obtain better stopping accuracy, a deep deterministic policy gradient method is further used to train the policy network online.
[0134] S3. Train the policy network online using the deep policy gradient determination method to obtain a deep optimization network;
[0135] As an optional implementation of this application, the depth-determining policy gradient method may be a depth-determining policy gradient algorithm based on the Actor-Critic architecture. Figure 5As shown, the deep policy gradient method is an Actor-Critic method. Through the above step S2, the action a output by the policy network initialized by the neural network parameters θ is combined with the state s to further optimize the value network.
[0136] As an optional implementation of this application, the policy network can be trained online using a deep policy gradient determination method to obtain a deeply optimized network, including:
[0137] Preset depth determination strategy gradient method;
[0138] Based on the aforementioned deep policy gradient determination method, a deep optimization architecture for the policy network and value network is constructed, resulting in a deep optimization network; wherein, in the deep optimization network:
[0139] The state s of the train reinforcement learning framework is used as the input to the policy network, and the policy network outputs action a; as shown. Figure 5 As shown, the policy network (actor): aπ(s; θ), has been initialized using behavior clones;
[0140] The state s of the train reinforcement learning framework and the action a output by the policy network are used as inputs to the value network, and the value network outputs a value q(s, a; w); as shown. Figure 5 As shown, where: the value network (critic): q(s,a;w), the value network outputs a scalar used to evaluate the quality of the action; the input of the value network is the state s and the action output a of the policy network;
[0141] The deep optimization network is optimized and updated so that its policy network and value network learn in accordance with the characteristics of the train.
[0142] The optimization and updating of deep optimization networks involves updating the policy network and value network within the deep optimization network, allowing their respective neural network parameters to adjust as the train characteristics change, thus achieving the effect of lifelong learning.
[0143] In this embodiment, as an optional implementation of this application, the policy network is optionally updated using Monte Carlo round-based updates or temporal differential updates.
[0144] Specifically, the value network uses Monte Carlo round updates, and the value network is updated once per round.
[0145] As an optional implementation of this application, the method for updating the policy network of the deep optimization network may be as follows:
[0146] In the final stage of the train's braking at the station (e.g., 10m), a policy network π(s; θ) with fixed parameters controls the train's stopping. The complete trajectory controlled by the policy network has T steps, and the immediate reward for each step before the train stops is 0, i.e.: r1 = r2 = ... = r T-1 =0;
[0147] Determine whether the train's stopping accuracy D is within the specified range:
[0148] If the train stopping accuracy D is within the specified range (e.g., ±20cm), the trajectory receives a reward of 100-|D| in the final step, i.e.: r T =100-|D|;
[0149] If the train stopping accuracy D is not within the specified range (e.g., ±20cm), the reward for the last step of the trajectory is -(100+|D|), that is: r T =-(100+|D|);
[0150] If the train stopping accuracy D is within the specified range (e.g., ±20cm), the cumulative reward for each step to the end of the trajectory is: u1=u2=…=u T =100-|D|;
[0151] If the train's stopping accuracy D is not within the specified range (e.g., ±20cm), the cumulative reward for each step to the end of the trajectory is: u1 = u2 = ... = u T =-(100+|D|);
[0152] Calculate the cost function:
[0153]
[0154] Update parameters:
[0155]
[0156] in:
[0157] w now : Current neural network parameters;
[0158] w new : Updated neural network parameters;
[0159] α: Learning rate, is a hyperparameter that can be set;
[0160] The cost function J(w) in w now The gradient value at that point.
[0161] The neural network parameters w of the policy network obtained above new The system will be updated and iterated in rounds to update the policy network and adjust it to changes in train characteristics, thus achieving the effect of lifelong learning.
[0162] The policy network uses temporal difference updates and a deterministic policy gradient algorithm for updates. The goal of the policy network is to increase the value of the value network.
[0163] As an optional implementation of this application, the method for updating the value network of the deeply optimized network may be as follows:
[0164] With fixed value network parameters w, a policy network is used to control train stops at stations.
[0165] Calculate the policy gradient through backpropagation:
[0166]
[0167] Gradient ascent to update parameters:
[0168]
[0169] in:
[0170] θ now : Current neural network parameters;
[0171] θ new : Updated neural network parameters;
[0172] β: Learning rate, is a hyperparameter that can be set;
[0173] Operator g(θ) at θ now The gradient value at that point.
[0174] The neural network parameters θ of the value network obtained above are... new The value network will be updated iteratively using a single-step update method to adapt to changes in train characteristics, thus achieving the effect of lifelong learning.
[0175] S4. Output and save the deep optimization network, and use the deep optimization network for precise train stopping control.
[0176] Both the policy network and value network of the deep optimization network are updated and used in real time to control the stopping accuracy of the train. In the low-speed air braking stage, a deep reinforcement learning algorithm is used to replace the original PI control algorithm, reducing the online debugging time of the ATO software on the actual vehicle and improving the stopping accuracy. This solves the technical problem that "the air braking system has slow response and high randomness, and the PI controller cannot control the air braking system well, resulting in a large deviation in stopping accuracy."
[0177] Therefore, this application can make full use of the historical ATO data of existing lines for offline imitation learning; relying on the generalization ability of neural networks, it decouples the coupling relationship between PI control parameters and vehicle characteristics; it can reduce the online debugging time of ATO software on real vehicles and improve the stopping accuracy; the policy network and value network of reinforcement learning will adjust with the changes in train characteristics, achieving the effect of lifelong learning.
[0178] It should be noted that although the above reinforcement learning architecture design and network update method are introduced using Actor-Critic as an example, those skilled in the art will understand that this disclosure is not limited to this. In fact, users can flexibly set the optimization algorithm of the policy network according to the actual application scenario, as long as the technical functions of this application can be implemented according to the above technical methods.
[0179] Example 2
[0180] Based on the implementation principle of Embodiment 1, this application, in another aspect, proposes an apparatus for implementing the aforementioned deep reinforcement learning-based train precise stopping control method, comprising:
[0181] The ATO data acquisition module is used to collect ATO stop data and preprocess it to obtain expert dataset X;
[0182] An initialization module is used to perform imitation learning of behavior cloning based on the expert dataset X, and to initialize the train's policy network;
[0183] The network optimization module is used to train the policy network online using a deep policy gradient determination method to obtain a deep optimized network.
[0184] The train control module is used to output and save the deep optimization network, and to use the deep optimization network for precise train stopping control.
[0185] For details on the application principles and information interaction principles of the above modules, please refer to the description of Example 1. This example will not repeat them.
[0186] Obviously, those skilled in the art should understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the control methods described above. The modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps into a single integrated circuit module. Thus, the present invention is not limited to any specific hardware and software combination.
[0187] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the control methods described above. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.
[0188] Example 3
[0189] Furthermore, this application also proposes a control system, comprising:
[0190] processor;
[0191] Memory used to store processor-executable instructions;
[0192] The processor is configured to implement the deep reinforcement learning-based train precise stopping control method when executing the executable instructions.
[0193] This disclosure discloses an embodiment of a system including a processor and a memory for storing processor-executable instructions. The processor is configured to implement any of the aforementioned deep reinforcement learning-based train precise stopping control methods when executing the executable instructions.
[0194] It should be noted here that the number of processors can be one or more. Furthermore, the control system in this embodiment may also include input devices and output devices. The processors, memory, input devices, and output devices can be connected via a bus or other means, without specific limitations herein.
[0195] As a computer-readable storage medium, memory can be used to store software programs, computer-executable programs, and various modules, such as the program or module corresponding to the train precise stopping control method based on deep reinforcement learning in this disclosure. The processor executes various functional applications and data processing of the control system by running the software program or module stored in the memory.
[0196] Input devices can be used to receive input digital numbers or signals. These signals can be key signals related to user settings and function control of the device / terminal / server. Output devices can include display devices such as screens.
[0197] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.
Claims
1. A train precise stopping control method based on deep reinforcement learning, characterized in that, Includes the following steps: Collect ATO (Automatic Train Operation) stop data and preprocess it to obtain expert dataset X; Based on the expert dataset X, imitation learning of behavior cloning is performed to initialize the train's policy network; The policy network is trained online using a deep policy gradient determination method to obtain a deep optimized network, including: Preset depth determination strategy gradient method; Based on the aforementioned deep policy gradient determination method, a deep optimization architecture for the policy network and value network is constructed, resulting in a deep optimization network; wherein, in the deep optimization network: The state s of the train reinforcement learning framework is used as the input to the policy network, and the policy network outputs action a. The state s of the train reinforcement learning framework and the action a output by the policy network are used as inputs to the value network, and the value network outputs the value. ; Optimize and update the deep optimization network so that the policy network and value network of the deep optimization network learn according to the characteristics of the train; The update method for the policy network of the deep optimization network is as follows: During the final stage of the train's braking at the station, a policy network with fixed parameters is used. Controlling train stops at stations; where the complete trajectory of train movement controlled by the policy network has T steps, and the immediate reward for each step before the train stops is 0, that is: ; Determine whether the train's stopping accuracy D is within the specified range: If the train stopping accuracy D is within the specified range, the final step of the trajectory receives a reward of 100 - |D|, that is: ; If the train stopping accuracy D is outside the specified range, the final step of the trajectory will receive a reward of [missing value]. ,Right now: ; If the train stopping accuracy D is within the specified range, the cumulative reward for each step to the end of the trajectory is: ; If the train stopping accuracy D is not within the specified range, the cumulative reward for each step to the end of the trajectory is: ; Calculate the cost function : ； Update parameters: ； in: : Current neural network parameters; : Updated neural network parameters; The learning rate is a hyperparameter that can be set. Cost function exist The gradient value at that point; The deep optimization network is output and saved, and then used for precise train stopping control.
2. The train precise stopping control method based on deep reinforcement learning according to claim 1, characterized in that, Collect ATO (Automatic Train Operation) stop data and preprocess it to obtain expert dataset X, which includes: Collect ATO stop data for all existing lines; According to the preset data preprocessing rules, the ATO stop data is cleaned and segmented to obtain ATO stop preprocessed data; From the ATO stop preprocessing data, the data of the final braking stage of each ATO stop is extracted, and the data whose stopping trajectory meets the expectations are selected to form an expert dataset X.
3. The train precise stopping control method based on deep reinforcement learning according to claim 1, characterized in that, Based on the expert dataset X, imitation learning for behavior cloning is performed, and the train's policy network is initialized, including: Define a loss function, and use the loss function to calculate the error between the policy network output and the actual expert actions in the expert dataset X; wherein the loss function is: Among them, the loss function It is the single-sample error, used to describe the parameter. The error between the output value generated by the policy network under input state s and the output a under the same state s in the expert database; symbol This means "defined as", that is, the expression on the left is defined by the expression on the right. It is a policy network The square of the error between the output and the actual expert action a is multiplied by ½ to eliminate the constant term 2 when taking the derivative of gradient descent; Define a cost function, and use this cost function to calculate the sum of the loss functions of all sample points along an entire trajectory in the expert database X. Then, sum the losses of all trajectories to obtain the total cost function. : ； Update neural network parameters using the following gradient descent formula until the algorithm converges: in: : Current neural network parameters; : Updated neural network parameters; The learning rate is a hyperparameter that can be set. Cost function exist The gradient value at that point; Using the neural network parameters Initialize the policy network.
4. The train precise stopping control method based on deep reinforcement learning according to claim 1, characterized in that, The depth-determined policy gradient method is a depth-determined policy gradient algorithm based on the Actor-Critic architecture.
5. The train precise stopping control method based on deep reinforcement learning according to claim 1, characterized in that, The method for updating the value network of the deep optimization network is as follows: Fixed value network parameters Use policy networks to control train stops at stations; Calculate the policy gradient through backpropagation: ； Gradient ascent to update parameters: in: : Current neural network parameters; : Updated neural network parameters; The learning rate is a hyperparameter that can be set. Operator exist The gradient value at that point.
6. An apparatus for implementing the train precise stopping control method based on deep reinforcement learning as described in any one of claims 1-5, characterized in that, include: The ATO data acquisition module is used to collect ATO stop data and preprocess it to obtain expert dataset X; An initialization module is used to perform imitation learning of behavior cloning based on the expert dataset X, and to initialize the train's policy network; The network optimization module is used to train the policy network online using a deep policy gradient determination method to obtain a deep optimized network. The train control module is used to output and save the deep optimization network, and to use the deep optimization network for precise train stopping control.
7. A control system, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement, when executing the executable instructions, a train precise stopping control method based on deep reinforcement learning as described in any one of claims 1 to 5.