Train recommended speed curve generation method based on state and action value network
By using the DQN method based on state and action value networks, the generation of recommended train speed curves is optimized, which solves the problems of large human factors and algorithm complexity in existing technologies, and achieves more accurate energy consumption assessment and intelligent improvement of ATO controller.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING JIAOTONG UNIV
- Filing Date
- 2022-08-25
- Publication Date
- 2026-04-10
AI Technical Summary
Existing methods for generating recommended train speed curves rely on ATP curve constraints and human experience, making them highly susceptible to human factors. Furthermore, existing algorithms are computationally complex under complex track conditions, prone to getting trapped in local optima, and the convergence speed of neural networks is difficult to achieve the expected level.
The DQN method based on state and action value networks is adopted. By dividing the action and state spaces, designing a reward function, splitting the neural network into state value and action value networks, and using an ε-greedy exploration strategy for training, the energy consumption assessment of train operation is optimized.
It improves the accuracy of train operation energy consumption calculation, enhances the learning effect of neural networks, improves the intelligence level of ATO controller, and generates speed curves that better meet the requirements of energy saving, safety, and efficiency.
Smart Images
Figure CN115392122B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of urban rail transit technology, specifically relating to a method for generating train recommended speed curves based on state and action value networks. Background Technology
[0002] In recent years, with the continuous increase in the operating mileage and passenger volume of urban rail transit, the pressure of energy conservation and emission reduction has also increased. Train recommended speed curve optimization is a major method for reducing traction energy consumption, and it also serves as the tracking target of the ATO system controller in actual line operation, guiding train operation. Therefore, how to generate an energy-saving, safe, efficient, and comfortable curve has become a focus of attention for many scholars. In engineering, the generation of curves mainly relies on ATP curve constraints and human experience judgment. From existing research results, a large amount of theoretical and technical research has been carried out in train speed curve optimization and operation control, mainly including analytical methods using the maximum principle for mathematical derivation, evolutionary algorithms such as genetic algorithms and ant colony algorithms that use the optimization objective function as the fitness function for population iteration, and numerical methods such as dynamic programming and artificial intelligence combined with computer science.
[0003] In engineering, curve generation mainly relies on ATP curve constraints and human experience judgment, making it highly susceptible to human factors. Existing analytical methods based on the maximum principle often require simplified model calculations when dealing with complex circuit conditions, and the algorithm is difficult to solve. Evolutionary algorithms are prone to getting stuck in local optima. Artificial intelligence neural network methods often fail to achieve the expected convergence speed in practical applications, and the design of a suitable reward function based on the performance indicators of the optimization objective directly affects the quality and convergence of the neural network. Summary of the Invention
[0004] This invention aims to provide a method for generating train recommended speed curves based on state and action value networks to solve the above problems.
[0005] The technical solution of this invention is:
[0006] A method for generating train recommendation speed curves based on state and action value networks includes the following steps:
[0007] Step 1: Develop DQN in a JavaScript environment, design the components, set the required environment parameters and performance indicators, divide the action and state spaces, set the relevant parameters of the neural network and enable GPU usage, determine the exploration strategy, input the variables of the experience pool, and display the results of the DQN loss function and auxiliary modules.
[0008] Step 2: Analyze the performance indicators of the train operation curve and design a reward function. Transform the traction energy consumption model in the reward function into the problem of exploring the difference between the speed area and the average speed area and the proportion of coasting conditions in the entire line, and jointly evaluate energy consumption to evaluate the action.
[0009] Step 3: Train the neural network by splitting it into a state value (sv) network and an action value (qv) network. The actual action value is set as a weighted average of the "state value" and the "action value correction value". The output action is selected according to the ε-greedy exploration strategy.
[0010] Step 4: As the neural network trains and iterates, the model gradually converges towards the optimal direction. Based on the convergence of the model in the later stages of training, a probability relationship between the exploration results and subsequent training is established, and the optimal recommendation speed curve is output after training is completed.
[0011] Preferably, the specific steps of step 1 are as follows:
[0012] Step 101: Develop DQN in a JavaScript environment and build functional modules: environment module, reward module, neural network module, exploration module, experience pool module, training module, and auxiliary module;
[0013] Step 102: Configure the environment module, which is divided into two parts: static data of the operating environment and fixed parameters of train performance. The static data of the operating environment includes the line length, planned running time, line speed limit, line gradient and curve radius. The fixed parameters of train performance include the train mass, maximum speed, basic resistance parameters and traction and braking characteristics.
[0014] Step 103: Construct a reward module. The reward value adopts a percentage system. The total reward value is the sum of the reward basis of each indicator. The indicators include traction energy consumption, deviation between actual running time and planned running time, and precise stopping.
[0015] Step 104: Construct the neural network module; First, divide the action and state space, set 21 actions, where 0-9 are braking conditions corresponding to different braking force percentages, 10 is the coasting condition, and 11-20 are traction conditions corresponding to different traction force percentages. Each frame of the state includes 6 attributes, namely the current time and remaining time, the current position and remaining position, the current speed and the remaining overspeed range. The state data input to the neural network is normalized using a fixed ratio method. Five frames of state data are input at a time and flattened into a 30-dimensional feature.
[0016] Step 105: Next, split the neural network into a state value network and an action value network, and set the relevant parameters of the neural network: enable GPU, use tanh as the activation function for the hidden layer, use the linear function to activate the output layer, and use adam as the optimization function; the neural network adopts a fully connected approach, with one Flatten layer, three hidden layers, 30 units in the Flatten layer, 40 units in the first hidden layer, 160 units in the second hidden layer, 120 units in the third hidden layer, and 22 units in the output layer;
[0017] Step 106: Set up the exploration module and adopt the ε-greedy exploration strategy; extract the state input from the environment module into the neural network, randomly select actions with a probability of ε, and execute the action with the highest value with a probability of 1-ε. The better the model convergence, the smaller the exploration probability ε.
[0018] Step 107: Set up the experience pool module to store the current state, reward, action, and next state into the experience pool;
[0019] Step 108: Construct a training module, randomly select mini-batch data to train the neural network; input the current state and action into the current value network to obtain Q(s,a,θ), and input the reward and next state into the target value network to obtain... The loss function of DQN is expressed by the following formula:
[0020]
[0021] Step 109: Construct auxiliary modules, including the optimal train recommended speed curve and the display of the score trend graph as the number of explorations changes.
[0022] Preferably, the specific steps of step 2 are as follows:
[0023] Step 201: Calculate the average speed using the ratio of line length to planned running time, obtain the average speed area, and then calculate the actual running speed area and the area difference between the two.
[0024] Step 202: Record the scores for each operating condition throughout the entire process. The coasting condition scores the highest score of 1 point. The cruise condition scores based on the current acceleration: the smaller the acceleration, the higher the score, with scores ranging from 0 to 1. Other operating conditions do not score.
[0025] Step 203: Combine the difference between the actual operating speed area and the average speed area with the total score of the entire process to evaluate the train traction energy consumption level in the reward function.
[0026] Preferably, the operation process of step 3 is as follows:
[0027] Step 301: Split the neural network into an SV network and a QV network. The output of the SV network is the state value, and the output of the QV network is the action value correction value. The two share the feature dimension of the state.
[0028] Step 302: The SV network corresponds to 1 output, and the QV network corresponds to 3 sets of outputs. The 3 sets of outputs divide the 21 actions into 3 groups, each containing 7 actions.
[0029] Step 303: The two networks are merged into a one-input multiple-output structure. The final output action value is a weighted sum of one state value and three sets of action value correction values. Based on the final action value, there is a probability of ε to randomly select an action, and a probability of 1-ε to select the action with the highest action value.
[0030] Preferably, the specific steps of step 4 are as follows:
[0031] Step 401: Determine if the model has converged. If the model has converged sufficiently, low-scoring exploration results will not participate in training. If the model has not converged, low-scoring exploration results may participate in the training process.
[0032] Step 402: After the model converges and is trained, the auxiliary module outputs the optimal energy-saving recommendation speed curve and score trend graph.
[0033] The beneficial effects of this invention are as follows:
[0034] 1. To address the challenge of measuring and evaluating train operation energy consumption indicators in a complex manner, an energy consumption model was constructed that explores the difference between the speed area and the average speed area, as well as the proportion of coasting conditions on the entire line. The energy consumption level is then comprehensively evaluated based on these two factors, which improves the accuracy of energy consumption calculation and solves the design problem of the neural network reward function.
[0035] 2. The method proposed in this invention, which splits a neural network into a dual network of state value and action value, can effectively enhance the learning effect of the neural network and effectively distinguish whether the cause of the current high Q value is the state itself or the quality of the action. Attached Figure Description
[0036] Figure 1 A flowchart of a train recommended speed curve generation method based on state and action value network provided in an embodiment of the present invention;
[0037] Figure 2 A schematic diagram illustrating the construction of the operating energy consumption model for the train recommended speed curve generation method based on state and action value network provided in this embodiment of the invention;
[0038] Figure 3 A neural network structure diagram of the train recommendation speed curve generation method based on state and action value network provided in an embodiment of the present invention;
[0039] Figure 4 The convergence curve of the training of the train recommendation speed curve generation method based on state and action value network provided in the embodiment of the present invention;
[0040] Figure 5 The recommended speed curve diagram is provided for the train recommended speed curve generation method based on state and action value network in the embodiments of the present invention. Detailed Implementation
[0041] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. The embodiments of the present invention are not limited thereto.
[0042] Example 1
[0043] like Figure 1 As shown, the train recommendation speed curve generation method based on state and action value networks includes the following steps:
[0044] Step 1: Develop DQN in a JavaScript environment, design the components, set the required environment parameters and performance indicators, divide the action and state spaces, set the relevant parameters of the neural network and enable GPU usage, determine the exploration strategy, input the variables of the experience pool, and display the results of the DQN loss function and auxiliary modules.
[0045] The specific steps for step 1 are as follows:
[0046] Step 101: Develop DQN in a JavaScript environment and build functional modules: environment module, reward module, neural network module, exploration module, experience pool module, training module, and auxiliary module;
[0047] Step 102: Configure the environment module, which is divided into two parts: static data of the operating environment and fixed parameters of train performance. The static data of the operating environment includes the line length, planned running time, line speed limit, line gradient and curve radius. The fixed parameters of train performance include the train mass, maximum speed, basic resistance parameters and traction and braking characteristics.
[0048] Step 103: Construct a reward module. The reward value adopts a percentage system. The total reward value is the sum of the reward basis of each indicator. The indicators include traction energy consumption, deviation between actual running time and planned running time, and precise stopping.
[0049] Step 104: Construct the neural network module; First, divide the action and state space, set 21 actions, where 0-9 are braking conditions corresponding to different braking force percentages, 10 is the coasting condition, and 11-20 are traction conditions corresponding to different traction force percentages. Each frame of the state includes 6 attributes, namely the current time and remaining time, the current position and remaining position, the current speed and the remaining overspeed range. The state data input to the neural network is normalized using a fixed ratio method. Five frames of state data are input at a time and flattened into a 30-dimensional feature.
[0050] Step 105: Next, split the neural network into a state value network and an action value network, and set the relevant parameters of the neural network: enable GPU, use tanh as the activation function for the hidden layer, use the linear function to activate the output layer, and use adam as the optimization function; the neural network adopts a fully connected approach, with one Flatten layer, three hidden layers, 30 units in the Flatten layer, 40 units in the first hidden layer, 160 units in the second hidden layer, 120 units in the third hidden layer, and 22 units in the output layer;
[0051] Step 106: Set up the exploration module and adopt the ε-greedy exploration strategy; extract the state input from the environment module into the neural network, randomly select actions with a probability of ε, and execute the action with the highest value with a probability of 1-ε. The better the model convergence, the smaller the exploration probability ε.
[0052] Step 107: Set up the experience pool module to store the current state, reward, action, and next state into the experience pool;
[0053] Step 108: Construct a training module, randomly select mini-batch data to train the neural network; input the current state and action into the current value network to obtain Q(s,a,θ), and input the reward and next state into the target value network to obtain... The loss function of DQN is expressed by the following formula:
[0054]
[0055] Step 109: Construct auxiliary modules, including the optimal train recommended speed curve and the display of the score trend graph as the number of explorations changes.
[0056] Step 2: Analyze the train operation curve performance indicators and design a reward function. Transform the traction energy consumption model within the reward function into a problem involving the difference between the exploration speed area and the average speed area, and the proportion of coasting conditions across the entire line. Efficiently assess energy consumption to evaluate the action. A schematic diagram of the model construction is shown below. Figure 2 As shown.
[0057] The specific steps for step 2 are as follows:
[0058] Step 201: Calculate the average speed using the ratio of line length to planned running time, obtain the average speed area, and then calculate the actual running speed area and the area difference between the two.
[0059] Step 202: Record the scores for each operating condition throughout the entire process. The coasting condition scores the highest score of 1 point. The cruise condition scores based on the current acceleration: the smaller the acceleration, the higher the score, with scores ranging from 0 to 1. Other operating conditions do not score.
[0060] Step 203: Combine the difference between the actual operating speed area and the average speed area with the total score of the entire process to evaluate the train traction energy consumption level in the reward function.
[0061] Step 3: Train the neural network by splitting it into a state value (sv) network and an action value (qv) network. The actual action value is set as a weighted average of the "state value" and the "action value correction value". The output action is selected according to the ε-greedy exploration strategy.
[0062] The operation process for step 3 is as follows:
[0063] Step 301: As Figure 3 As shown, the neural network is split into an SV network and a QV network. The output of the SV network is the state value, and the output of the QV network is the action value correction value. The two share the feature dimension of the state.
[0064] Step 302: The SV network corresponds to 1 output, and the QV network corresponds to 3 sets of outputs. The 3 sets of outputs divide the 21 actions into 3 groups, each containing 7 actions.
[0065] Step 303: The two networks are merged into a one-input multiple-output structure. The final output action value is a weighted sum of one state value and three sets of action value correction values. Based on the final action value, there is a probability of ε to randomly select an action, and a probability of 1-ε to select the action with the highest action value.
[0066] Step 4: As the neural network trains and iterates, the model gradually converges towards the optimal direction. Based on the convergence of the model in the later stages of training, a probability relationship between the exploration results and subsequent training is established, and a recommended speed curve is output after training is completed.
[0067] The specific steps for step 4 are as follows:
[0068] Step 401: As Figure 4As shown, the calibration value is set to a percentage system of 0 to 100. Initially, since the default input-output range of the neural network is between 0 and 1, the calibration value is temporarily lower than 60 and remains unchanged. In the later stages of training, as the number of explorations increases, after more than 3000 explorations, the model's calibration value gradually converges from the initial constant value towards a higher score, i.e., the optimal direction of the model. After more than 4000 explorations, the model basically converges, and exploration results below the initial calibration value are no longer used in the training of the neural network.
[0069] Step 402: After the model has converged and been trained, output the results through the auxiliary module. Figure 4 and Figure 5 . Figure 5 The recommended speed curve for energy saving is shown, where the horizontal axis represents train travel time (s), the left vertical axis represents train position (m), and the right vertical axis represents train speed (m / s). The light-colored curve in the figure represents the train speed-time curve. After reaching the expected speed of 27.8 m / s, the train selects coasting mode, which lasts from 25s to 49s, and finally switches to braking mode. The operation process meets our energy-saving goals. The dark-colored curve represents the train position-time curve corresponding to the recommended speed.
[0070] The final value function is presented in percentage form, and the results are shown in Table I:
[0071] Table I. Value function scores for different methods
[0072]
[0073] The method of splitting the neural network improves upon the basic DQN network by 15%, and the composite energy consumption evaluation further improves the score by 3.2%.
[0074] In summary, this invention can effectively solve the problem of measuring and evaluating complex energy consumption indicators, enhance the learning effect of neural networks, and improve the intelligence level of ATO controllers by applying the autonomous learning capability of the DQN algorithm.
[0075] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the processes depicted in the drawings are not necessarily essential for implementing the present invention.
Claims
1. A method for generating train recommended speed profile based on state and action value network, characterized in that, Comprise the following steps: Step 1: develop DQN in JavaScript environment, design each part of the module, set the required environment parameters and performance indicators, divide the action, state space, set the related parameters of neural network and open the GPU use, determine the exploration strategy, input the variables of experience pool, the loss function of DQN and the result display of auxiliary module; Step 2: analyze the train operation curve performance index to design the reward function, convert the traction energy consumption model in the reward function into the difference between the exploration speed area and the average speed area and the proportion of the idling working condition in the whole line, and evaluate the energy consumption together to evaluate the action; Step 3: train the neural network, split the neural network into state value sv network and action value qv network, set the actual action value as the weighted value of "state value" and "action value correction value", and select the output action according to the exploration strategy of ε-greedy; Step 4: with the iteration of neural network training, the model gradually converges to the optimal direction, the probability relationship between exploration results and subsequent training is established according to the convergence of the model after training, and the optimal recommended speed curve is output after training is completed; The specific steps of step 2 are as follows: Step 201: calculate the average speed by the ratio of the line length to the planned running time, obtain the average speed area, and then calculate the actual running speed area and the area difference between the two; Step 202: record the score of each running condition in the whole process, wherein the idling working condition gets the highest score of 1 point, the cruise working condition gets a score between 0 and 1 according to the current acceleration, the smaller the acceleration, the higher the score, and other working conditions get no score; Step 203: combine the difference between the actual running speed area and the average speed area and the total score of the whole process to evaluate the train traction energy consumption level in the reward function; The operation process of step 3 is as follows: Step 301: split the neural network into sv network and qv network, the output of sv network is state value, the output of qv network is action value correction value, and they share the feature dimension of state; Step 302: sv network corresponds to 1 output, qv network corresponds to 3 groups of outputs, and 3 groups of outputs are divided into 3 groups, each group containing 7 actions; Step 303: two networks are combined into one input and multiple output structure, the final output action value is the weighted value of 1 state value and 3 groups of action value correction values, and based on the obtained final action value, there is a probability of ε to randomly select action, and there is a probability of 1-ε to select the action with the maximum action value. 2.The state and action value network based train recommended speed curve generation method according to claim 1, wherein, The specific steps of step 1 are as follows: Step 101: develop DQN in JavaScript environment, build functional modules: environment module, reward module, neural network module, exploration module, experience pool module, training module, and auxiliary module; Step 102: configure the environment module, which is divided into two parts: running environment static data and train performance fixed parameters; wherein the running environment static data includes line length, planned running time, line speed limit, line slope and curve radius, and the train performance fixed parameters include train mass, maximum speed, basic resistance parameters and traction and braking characteristics; Step 103: Construct the reward module, the reward value takes the percentage system, and the total reward value is the sum of the rewards of various indicators, including traction energy consumption, deviation of actual running time and planned running time, and accurate parking; Step 104: Construct the neural network module; first, divide the action and state space, set 21 actions, of which 0~9 are different brake force percentage corresponding brake working conditions, 10 is the coasting working condition, and 11~20 are different traction force percentage corresponding traction working conditions; the state includes 6 attributes, respectively, the current time and the remaining time, the current position and the remaining position, the current speed and the remaining overspeed range; the state data input to the neural network is normalized by a fixed ratio, and 5 frames of state data are input at a time, which are compressed into a 30-dimensional feature; Step 105: Secondly, the neural network is divided into state value network and action value network, and the related parameters of the neural network are set: open GPU, the hidden layer activation function is tanh, the output layer uses linear function activation, and the optimization function is adam; the neural network adopts full connection, one Flatten layer, three hidden layers, the unit number of the Flatten layer is 30, the unit number of the first layer hidden layer is 40, the unit number of the second layer hidden layer is 160, the unit number of the second layer hidden layer is 120, and the output layer is 22; Step 106: Set the exploration module, and adopt the exploration strategy of ε-greedy; input the state to the neural network from the environment module, randomly select the action with the probability of ε, and execute the action with the maximum action value with the probability of 1-ε; the better the model converges, the smaller the exploration probability ε is; Step 107: Set the experience pool module, and store the current state, reward, action, and next state into the experience pool; Step 108: Constructing the training module, randomly selecting small batches of data to train the neural network; the current state and action input the current value network to obtain , the reward and the next state input the target value network to obtain , the loss function of DQN is expressed by the following formula: ； Step 109: Construct the auxiliary module, including the display of the optimal train recommended speed curve and the score trend graph changing with the exploration number. 3.The state and action value network based train recommended speed curve generation method of claim 1, wherein, The specific steps of step 4 are as follows: Step 401: Determine whether the model converges, if the model converges enough, the low-score exploration result is not involved in the training, and if the model does not converge, the low-score exploration result has a certain probability to participate in the training process; Step 402: After the model converges and the training is completed, output the optimal energy-saving recommended speed curve and the score trend graph through the auxiliary module.