A high-speed train speed curve optimization method and system based on reinforcement learning

Abstract

The present invention discloses a high-speed train speed curve optimization method and system based on reinforcement learning, including S1: obtaining road data, train running time data, train speed, train acceleration data, and preprocessing them to obtain the preprocessed road data, train running time data, train speed, and train acceleration data; S2: defining the state space and action space of the train speed curve optimization network; S3: constructing the value function of the train speed curve optimization network, and updating the value function according to the iteration step; S4: setting the reward function of the train speed curve optimization network, the reward function includes the passenger comfort function, the total energy consumption function, and the punctuality function; S5: reinforcement learning training, updating the Q table and outputting the high-speed train speed curve. This solution solves the problem that the traditional train speed curve does not take passenger comfort into consideration.

Description

Technical Field

[0001] The present invention belongs to the technical field of rail transit, and in particular relates to a high-speed train speed curve optimization method and system based on reinforcement learning. Background Art

[0002] High-speed train speed curve optimization refers to designing a reasonable train speed curve based on the operating environment, technical characteristics, and operational objectives of the high-speed train, so that the train can save operating energy as much as possible and improve operating efficiency and quality while meeting safety and comfort requirements. At present, the methods for optimizing the speed curve of high-speed trains are mainly based on mathematical programming and intelligent algorithms, including nonlinear programming, particle swarm optimization, etc.

[0003] Mathematical programming methods are based on rigorous mathematical theories, and genetic algorithms and particle swarm algorithms have global search capabilities and can find the global optimal solution. However, mathematical programming and intelligent algorithms require a lot of computing resources and time, especially for complex problems; on the other hand, traditional methods usually only consider a few goals and it is difficult to balance multiple performance indicators during train operation, such as energy saving, punctuality and comfort.

[0004] At the same time, these methods do not fully consider the impact of changes in the environment inside and outside the tunnel on passenger comfort, and it is difficult to balance various performance indicators such as energy saving, punctuality, and comfort during train operation. Summary of the invention

[0005] In view of this, the present invention proposes a high-speed train speed curve optimization method and system based on reinforcement learning, aiming to solve the problem of suboptimal speed curve caused by insufficient utilization of train status data in the current field of rail transit technology. The present invention uses reinforcement learning to model train operation indicators, comprehensively considers the balance of roads, energy consumption and passenger comfort, and improves the safety of high-speed train operation and passenger comfort.

[0006] A high-speed train speed curve optimization method based on reinforcement learning comprises the following steps:

[0007] S1: Acquire road data, train running time data, train speed, and train acceleration data, and preprocess them to obtain preprocessed road data, train running time data, train speed, and train acceleration data;

[0008] S2: Define the state space and action space of the train speed curve optimization network;

[0009] S3: construct the value function of the train speed curve optimization network and update the value function according to the iteration step;

[0010] S4: setting a reward function of the train speed curve optimization network, wherein the reward function includes a passenger comfort function, a total energy consumption function, and a punctuality function;

[0011] S5: Reinforcement learning training, update Q table and output high-speed train speed curve.

[0012] Furthermore, the step S1 acquires road data, train running time data, train speed, and acceleration data, and performs preprocessing to obtain preprocessed road data, train speed, and train acceleration data, including:

[0013] S11: Acquire road data, train running time data, train speed, and train acceleration data; wherein the road data includes the starting position, ending position, tunnel length, and tunnel diameter of each tunnel along the entire high-speed train line, as well as the starting position and ending position of bridges and speed-limited sections;

[0014] S12: preprocessing the road data, train running time data, train speed, and train acceleration data, including using the average value method to fill in missing values ​​and using the Z-score method to normalize the data, to obtain the preprocessed road data, train running time data, train speed, and train acceleration data, wherein the calculation method of the Z-score method is:

[0015]

[0016] Among them, x is the data to be processed, μ is the mean of the data to be processed, σ is the standard deviation of the data to be processed, and socre is the normalized standard score.

[0017] Furthermore, the step S2 defines the state space and action space of the train speed curve optimization network, including:

[0018] S21: define the state space, including: train position, train speed, train acceleration, train running time, road type, road speed limit;

[0019] S22: Define the action space, including: train traction force and train braking force.

[0020] Furthermore, the step S3 constructs a value function of the train speed curve optimization network, and updates the value function according to the iteration step, including:

[0021] S31: Define the value function Q(s,a), which is the expected return of taking action a in state s, calculated as:

[0022]

[0023] Where E is the expectation, γ is the discount factor, and rt is the reward of the tth iteration step, s0 is the initial state, and a0 is the initial action;

[0024] S32: Adjust the value function according to the iteration step. The calculation method of the value function update formula is:

[0025]

[0026] Among them, β is the learning rate, s′ is the next state, a′ is the next action, The next action corresponding to the next step when the Q value of the next step is maximized.

[0027] Furthermore, the step S4 sets the reward function of the train speed curve optimization network, including the passenger comfort function, the total energy consumption function, and the punctuality function, and the calculation method is:

[0028] r t =w1r t1 +w2r t2 +w3r t3

[0029] Among them, r t1 、r t2 、r t3 They are passenger comfort function, total energy consumption function and punctuality function respectively, w1, w2 and w3 are weight coefficients of the three functions respectively;

[0030] The passenger comfort function is calculated as:

[0031]

[0032] Among them, c t is the train acceleration at the t-th iteration step, T c is the total time of train acceleration and deceleration, v t is the train speed at step t, v max is the upper limit of the train speed at step t, I t is the tunnel entry and exit indicator at the t-th iteration step, k1 is the influence coefficient of train acceleration, k2 is the influence coefficient of the total time of train acceleration and deceleration, k3 is the influence coefficient of overspeed in the tunnel, and k4 is the influence coefficient of tunnel entry and exit on comfort;

[0033] Item 1: Indicates the effect of train acceleration on comfort. The smaller the acceleration, the higher the comfort.

[0034] Item 2: -k2T c , represents the impact of the total time of train acceleration and deceleration on comfort. The shorter the acceleration and deceleration time, the higher the comfort.

[0035] The third item: ―k3|v t ―v max ∣, represents the impact of speeding in the tunnel on comfort. The smaller the speeding in the tunnel, the higher the comfort. At the same time, in order to avoid the agent (train) running at too low a speed in the tunnel and avoid punishment, the absolute value is increased so that the train will be punished if it runs too fast or too slow. max Determine the speed limit based on the length and diameter of the tunnel and the road section to avoid excessive speed in the tunnel causing negative pressure in the ear;

[0036] Item 4: -k4I t c t , which indicates the effect of train acceleration or deceleration on comfort when entering or exiting a tunnel; I t is the tunnel entry and exit indicator of the t-th iteration step. When the t-th step is the tunnel entry and exit, I t is 1, otherwise it is 0; if the train accelerates or decelerates when entering or leaving the tunnel, then c t If it is not 0, the agent will be penalized to speed up or slow down when it is away from the tunnel entrance or exit, rather than when entering or leaving the tunnel;

[0037] Regarding how to determine whether it is the time to enter or exit the tunnel, the present invention uses the distance and the position of the tunnel in the entire line as the determination conditions, which are as follows:

[0038] If at the tth iteration, the distance between the train head and the tunnel entrance is less than the threshold d1, or the distance between the train head and the tunnel exit is less than the threshold d2, it is considered to be the moment of entering or exiting the tunnel, I t is 1, otherwise it is 0;

[0039] The thresholds d1 and d2 are determined according to the length of the tunnel and the speed of the train. The longer the tunnel or the faster the train speed, the larger the threshold. The calculation method is:

[0040]

[0041] Where L represents the length of the tunnel;

[0042] The passenger comfort function is one of the sub-goals of the reinforcement learning multi-objective optimization, which reflects the impact of passenger comfort on train operation, thereby guiding the agent (train) to make more reasonable decisions. The passenger comfort function takes into account a variety of factors, including the acceleration of the train, the time of acceleration and deceleration, speeding in the tunnel, and the timing of acceleration and deceleration. The larger the value of the passenger comfort function, the higher the comfort of the passengers, and the agent will receive a greater reward, which will motivate it to adopt a more comfortable operation mode.

[0043] The passenger comfort function, the total energy consumption function and the punctuality function together constitute the reward function of the train speed curve optimization network. There is a coordination relationship between them. The present invention balances their importance and influence through weight coefficients.

[0044] The calculation method of the total energy consumption function is:

[0045]

[0046] Where n is the number of sections where the train runs, F i is the average traction force in the ith interval; v (i) is the average speed in the ith interval, η i is the traction efficiency in the i-th interval, ΔT i is the running time in the ith interval, G1 is the energy consumption of train auxiliary equipment, and G2 is the train braking energy consumption;

[0047] The calculation method of the punctuality function is:

[0048]

[0049] Where time1 is the actual arrival time of the train, time2 is the scheduled arrival time of the train, Time is the maximum tolerable late arrival time of the train, min() is the minimum value, and max() is the maximum value;

[0050] If the train arrives on time or early, the reward is positive and decreases as the time of early arrival increases, with a maximum of 1;

[0051] If the train is late, the reward is negative and decreases as the delay increases, with a minimum of -1;

[0052] If the train is late for more than Time seconds, the reward is -1 and will not decrease any further;

[0053] The design of the punctuality function takes into account the difference between the actual arrival time and the scheduled arrival time of the train, as well as the maximum tolerable lateness of the train. The larger the value of the punctuality function, the higher the punctuality of the train, and the greater the reward the intelligent agent will receive, thereby motivating it to adopt a more punctual operation mode.

[0054] Traditional mathematical programming methods usually require the establishment of complex mathematical models, involving a large number of constraints and variables, and the solution process is time-consuming and difficult to adapt to environments with more constraints. Although intelligent algorithms can avoid complex mathematical models, they require a large amount of training data and parameter adjustments, and it is difficult to ensure global optimization and real-time performance.

[0055] Reinforcement learning continuously updates value functions and strategies through interaction with the environment to find the optimal action. This method is suitable for solving the problem of optimizing train speed curves, because trains encounter various uncertain factors during operation, such as tunnel speed limits, and the speed curve needs to be adjusted in real time to achieve the goals of energy saving, comfort, and punctuality. Reinforcement learning can quickly generate optimized speed curves through online learning and exploration without the need for training data and parameter adjustment, and has good global optimality.

[0056] Furthermore, the step S5 strengthens the learning training, updates the Q table and outputs the high-speed train speed curve, including:

[0057] S51: Initialize the Q table and assign a random value to each state-action pair. The full name of the Q table is Quintuple table, i.e., a four-element learning table.

[0058] S52: Set the initial state s0 to: the train speed, acceleration, and running time are 0, and start moving from the departure station;

[0059] S53: Repeat the following steps until the train arrives at the last station, including:

[0060] Starting from the current state s, randomly select an action with probability ε, and select the action with the largest Q value with probability 1-ε;

[0061] Execute action a, observe the next state s′ and current reward r t ;

[0062] Update Q(s,a) in the Q table according to the value function update formula;

[0063] Update s to s′;

[0064] Output the strategy with the highest score in the Q table;

[0065] S54: Output the high-speed train speed curve result from the trained reinforcement learning model.

[0066] The present invention also provides a high-speed train speed curve optimization system based on reinforcement learning, comprising:

[0067] Data acquisition and preprocessing module: acquires road data, train running time data, train speed, and train acceleration data, and preprocesses them to obtain preprocessed road data, train running time data, train speed, and train acceleration data;

[0068] State-action space building module: defines the state space and action space of the train speed curve optimization network;

[0069] Value function construction module: constructs the value function of the train speed curve optimization network and updates the value function according to the iteration step;

[0070] Reward function construction module: setting the reward function of the train speed curve optimization network, wherein the reward function includes a passenger comfort function, a total energy consumption function, and a punctuality function;

[0071] Reinforcement learning training module: Reinforcement learning training, updating the Q table and outputting the high-speed train speed curve.

[0072] Compared with the prior art, the advantages of the present invention are:

[0073] (1) In terms of the overall solution, the present invention takes into account the problem of insufficient passenger comfort in optimizing the speed curve of high-speed trains. In combination with tunnel conditions, reinforcement learning is adopted in the present invention, which uses data such as the real-time speed and acceleration of the train to achieve automatic optimization of the train speed curve, thereby improving the comfort and energy saving of the train operation.

[0074] (2) In terms of algorithm improvement, the present invention adopts reinforcement learning, considers the speed limit in the tunnel, the timing of acceleration and deceleration, etc., proposes a new reward function, and outputs the optimization result of the high-speed train speed curve. Compared with traditional mathematical programming and intelligent algorithms, it has higher efficiency, effect and adaptability, and prompts trains to avoid acceleration and deceleration when entering and exiting tunnels, and avoid speeding in tunnels.

[0075] (3) In terms of data acquisition, the data required by the present invention, such as the real-time speed of the train, real-time acceleration, road type, etc., are easy to obtain, and the processing method is simple, which improves the usability of the present solution.

[0076] (4) In addition, the present invention calculates the threshold of the tunnel indicator in combination with the train speed and the tunnel length, thereby avoiding ignoring the location of the tunnel entrance and exit when optimizing the train speed curve, and effectively avoiding the situation where the train accelerates or decelerates at the tunnel entrance and exit, thereby reducing passenger comfort. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 A schematic diagram of a process flow of a high-speed train speed curve optimization method based on reinforcement learning provided by the present invention;

[0078] Figure 2 A schematic diagram of the training process of a high-speed train speed curve optimization method based on reinforcement learning provided by the present invention. DETAILED DESCRIPTION

[0079] The present invention is further described below in conjunction with the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention belong to the protection scope of the present invention.

[0080] Embodiment 1: A high-speed train speed curve optimization method based on reinforcement learning, such as Figure 1 As shown, the following steps are included:

[0081] S1: Acquire road data, train running time data, train speed, and train acceleration data, and preprocess them to obtain preprocessed road data, train running time data, train speed, and train acceleration data;

[0082] S11: Acquire road data, train running time data, train speed, and train acceleration data; wherein the road data includes the starting position, ending position, tunnel length, and tunnel diameter of each tunnel along the entire high-speed train line, as well as the starting position and ending position of bridges and speed-limited sections;

[0083] S12: preprocessing the road data, train running time data, train speed, and train acceleration data, including using the average value method to fill in missing values ​​and using the Z-score method to normalize the data, to obtain the preprocessed road data, train running time data, train speed, and train acceleration data, wherein the calculation method of the Z-score method is:

[0084]

[0085] Among them, x is the data to be processed, μ is the mean of the data to be processed, σ is the standard deviation of the data to be processed, and socre is the normalized standard score.

[0086] S2: Define the state space and action space of the train speed curve optimization network;

[0087] S21: define the state space, including: train position, train speed, train acceleration, train running time, road type, road speed limit;

[0088] S22: Define the action space, including: train traction force and train braking force.

[0089] S3: construct the value function of the train speed curve optimization network and update the value function according to the iteration step;

[0090] S31: Define the value function Q(s,a), which is the expected return of taking action a in state s, calculated as:

[0091]

[0092] Where E is the expectation, γ is the discount factor, and r t is the reward of the tth iteration step, s0 is the initial state, and a0 is the initial action;

[0093] S32: Adjust the value function according to the iteration step. The calculation method of the value function update formula is:

[0094]

[0095] Among them, β is the learning rate, s′ is the next state, a′ is the next action, The next action corresponding to the next step when the Q value of the next step is maximized.

[0096] S4: Setting a reward function for the train speed curve optimization network, wherein the reward function includes a passenger comfort function, a total energy consumption function, and a punctuality function, and the calculation method is:

[0097] r t =w1r t1 +w2r t2 +w3r t3

[0098] Among them, r t1 、r t2 、r t3 They are passenger comfort function, total energy consumption function and punctuality function respectively, w1, w2 and w3 are weight coefficients of the three functions respectively;

[0099] The passenger comfort function is calculated as:

[0100]

[0101] Among them, c t is the train acceleration at the t-th iteration step, T c is the total time of train acceleration and deceleration, v t is the train speed at step t, v max is the upper limit of the train speed at step t, I t is the tunnel entry and exit indicator at the t-th iteration step, k1 is the influence coefficient of train acceleration, k2 is the influence coefficient of the total time of train acceleration and deceleration, k3 is the influence coefficient of overspeed in the tunnel, and k4 is the influence coefficient of tunnel entry and exit on comfort;

[0102] Item 1: Indicates the effect of train acceleration on comfort. The smaller the acceleration, the higher the comfort.

[0103] Item 2: -k2T c , represents the impact of the total time of train acceleration and deceleration on comfort. The shorter the acceleration and deceleration time, the higher the comfort.

[0104] The third item: ―k3|v t ―v max∣, represents the impact of speeding in the tunnel on comfort. The smaller the speeding in the tunnel, the higher the comfort. At the same time, in order to avoid the agent (train) running at too low a speed in the tunnel and avoid punishment, the absolute value is increased so that the train will be punished if it runs too fast or too slow. max It is determined based on the length and diameter of the tunnel and the speed limit of the section to avoid negative pressure in the ear caused by excessive speed in the tunnel; if the train travels too fast in the tunnel, when the speed exceeds a certain threshold, the eardrum may be under pressure, causing ear pressure injury.

[0105] Item 4: -k4I t c t , which indicates the effect of train acceleration or deceleration on comfort when entering or exiting a tunnel; I t is the tunnel entry and exit indicator of the t-th iteration step. When the t-th step is the tunnel entry and exit, I t is 1, otherwise it is 0; if the train accelerates or decelerates when entering or leaving the tunnel, then c t If it is not 0, the agent will be penalized to speed up or slow down when it is away from the tunnel entrance or exit, rather than when entering or leaving the tunnel;

[0106] When the train accelerates or decelerates in the tunnel, the air pressure in the ear cavity changes. The greater the acceleration, the more obvious the pressure change. For example, when the train acceleration reaches about 0.5m / s 2 Passengers begin to feel changes in ear pressure; when the train acceleration reaches about 1.5m / s 2 When the car starts to move, passengers begin to feel a push on their backs.

[0107] Regarding how to determine whether it is the time to enter or exit the tunnel, the present invention uses the distance and the position of the tunnel in the entire line as the determination conditions, which are as follows:

[0108] If at the tth iteration, the distance between the train head and the tunnel entrance is less than the threshold d1, or the distance between the train head and the tunnel exit is less than the threshold d2, it is considered to be the moment of entering or exiting the tunnel, I t is 1, otherwise it is 0;

[0109] The thresholds d1 and d2 are determined according to the length of the tunnel and the speed of the train. The longer the tunnel or the faster the train speed, the larger the threshold. The calculation method is:

[0110]

[0111] Where L represents the length of the tunnel;

[0112] The passenger comfort function is one of the sub-goals of the reinforcement learning multi-objective optimization, which reflects the impact of passenger comfort on train operation, thereby guiding the agent (train) to make more reasonable decisions. The passenger comfort function takes into account a variety of factors, including the acceleration of the train, the time of acceleration and deceleration, speeding in the tunnel, and the timing of acceleration and deceleration. The larger the value of the passenger comfort function, the higher the comfort of the passengers, and the agent will receive a greater reward, which will motivate it to adopt a more comfortable operation mode.

[0113] The passenger comfort function, the total energy consumption function and the punctuality function together constitute the reward function of the train speed curve optimization network. There is a coordination relationship between them. The present invention balances their importance and influence through weight coefficients.

[0114]

[0115] For example, assuming k1 = 0.01, k2 = 0.1, k3 = 0.5, k4 = 1, v max =100; T c = 10. By adjusting c t 、v t ,I t , observe the punishment received by the agent (train).

[0116] When the train is outside the tunnel, I t =0, the reward function becomes:

[0117]

[0118] Substituting the values ​​into the equation, we get:

[0119]

[0120] When other parameters remain unchanged, t From 1m / s 2 becomes 2m / s 2 hour, Decrease, r t1 The greater the change in train acceleration, the lower the passenger comfort.

[0121] When other parameters remain unchanged, c When it changes from 10s to 15s, -k2T c Decrease, r t1 As the acceleration and deceleration time of the train increases, the passenger comfort level decreases.

[0122] When the train enters and exits the tunnel and is inside the tunnel, that is, I t =1, the reward function is:

[0123]

[0124] At this time, the acceleration change will not only affect In addition, it will also affect t c t , which leads to a further increase in the penalty imposed on the agent (train);

[0125] When the train does not follow the prescribed speed in the tunnel max When driving, overspeed or underspeed occurs, resulting in ∣v t ―v max | is not 0, which leads to ―k3|v t ―v max ∣ decreases, and the penalty on the agent (train) increases;

[0126] The calculation method of the total energy consumption function is:

[0127]

[0128] Where n is the number of sections where the train runs, F i is the average traction force in the ith interval; v (i) is the average speed in the ith interval, η i is the traction efficiency in the i-th interval, ΔT i is the running time in the ith interval, G1 is the energy consumption of train auxiliary equipment, and G2 is the train braking energy consumption;

[0129] The calculation method of the punctuality function is:

[0130]

[0131] Where time1 is the actual arrival time of the train, time2 is the scheduled arrival time of the train, Time is the maximum tolerable late arrival time of the train, min() is the minimum value, and max() is the maximum value;

[0132] If the train arrives on time or early, the reward is positive and decreases as the time of early arrival increases, with a maximum of 1;

[0133] If the train is late, the reward is negative and decreases as the delay increases, with a minimum of -1;

[0134] If the train is late for more than Time seconds, the reward is -1 and will not decrease any further;

[0135] For example, assume that the scheduled arrival time of the train is 10:00, ie, time2=10:00, and the maximum tolerable late arrival time of the train is 20 minutes, ie, Time=20.

[0136] Assume that the actual arrival time of the train is between 9:30 and 10:30, that is, time1∈[9:30,10:30],

[0137] Actual arrival time Expected arrival time Maximum tolerable lateness <![CDATA[rt3]]> 9:30 10:00 20 -0.5 9:50 10:00 20 0.5 9:55 10:00 20 0.75 10:00 10:00 20 1 10:05 10:00 20 -0.25 10:10 10:00 20 -0.5 10:30 10:00 20 -1

[0138] It can be seen that the closer the agent (train) is to arriving on time within the range of the maximum tolerable lateness, the greater the reward it receives.

[0139] The design of the punctuality function takes into account the difference between the actual arrival time and the scheduled arrival time of the train, as well as the maximum tolerable lateness of the train. The larger the value of the punctuality function, the higher the punctuality of the train, and the greater the reward the intelligent agent will receive, thereby motivating it to adopt a more punctual operation mode.

[0140] Traditional mathematical programming methods usually require the establishment of complex mathematical models, involving a large number of constraints and variables, and the solution process is time-consuming and difficult to adapt to environments with more constraints. Although intelligent algorithms can avoid complex mathematical models, they require a large amount of training data and parameter adjustments, and it is difficult to ensure global optimization and real-time performance.

[0141] Reinforcement learning continuously updates value functions and strategies through interaction with the environment to find the optimal action. This method is suitable for solving the problem of optimizing train speed curves, because trains encounter various uncertain factors during operation, such as tunnel speed limits, and the speed curve needs to be adjusted in real time to achieve the goals of energy saving, comfort, and punctuality. Reinforcement learning can quickly generate optimized speed curves through online learning and exploration without the need for training data and parameter adjustment, and has good global optimality.

[0142] S5: Reinforcement learning training, update Q table and output high-speed train speed curve, such as Figure 2 As shown;

[0143] S51: Initialize the Q table and assign a random value to each state-action pair. The full name of the Q table is Quintuple table, i.e., a four-element learning table.

[0144] S52: Set the initial state s0 to: the train speed, acceleration, and running time are 0, and start moving from the departure station;

[0145] S53: Repeat the following steps until the train arrives at the last station, including:

[0146] Starting from the current state s, randomly select an action with probability ε, and select the action with the largest Q value with probability 1-ε;

[0147] Execute action a, observe the next state s′ and current reward rt ;

[0148] Update Q(s,a) in the Q table according to the value function update formula;

[0149] Update s to s′;

[0150] Output the strategy with the highest score in the Q table;

[0151] S54: Output the high-speed train speed curve result from the trained reinforcement learning model.

[0152] Embodiment 2: A high-speed train speed curve optimization system based on reinforcement learning, used to implement the above-mentioned high-speed train speed curve optimization method based on reinforcement learning, comprising:

[0153] Data acquisition and preprocessing module: acquires road data, train running time data, train speed, and train acceleration data, and preprocesses them to obtain preprocessed road data, train running time data, train speed, and train acceleration data;

[0154] State-action space building module: defines the state space and action space of the train speed curve optimization network;

[0155] Value function construction module: constructs the value function of the train speed curve optimization network and updates the value function according to the iteration step;

[0156] Reward function construction module: setting the reward function of the train speed curve optimization network, wherein the reward function includes a passenger comfort function, a total energy consumption function, and a punctuality function;

[0157] Reinforcement learning training module: Reinforcement learning training, updating the Q table and outputting the high-speed train speed curve.

[0158] It should be noted that the serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. And the terms "including", "comprising" or any other variants thereof in this article are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "including a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0159] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0160] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A high-speed train speed curve optimization method based on reinforcement learning, characterized in that: The following steps are involved: S1: Acquire road data, train running time data, train speed, and train acceleration data, and preprocess them to obtain preprocessed road data, train running time data, train speed, and train acceleration data; S2: Define the state space and action space of the train speed curve optimization network; S3: construct the value function of the train speed curve optimization network and update the value function according to the iteration step; S4: setting a reward function of the train speed curve optimization network, wherein the reward function includes a passenger comfort function, a total energy consumption function, and a punctuality function; S5: Reinforcement learning training, updating the Q table and outputting the high-speed train speed curve; The step S1 comprises the following steps: S11: Acquire road data, train running time data, train speed, and train acceleration data; wherein the road data includes the starting position, ending position, tunnel length, and tunnel diameter of each tunnel along the entire high-speed train line, as well as the starting position and ending position of bridges and speed-limited sections; S12: preprocessing the road data, train running time data, train speed, and train acceleration data, including using the average value method to fill in missing values ​​and using the Z-score method to normalize the data, to obtain the preprocessed road data, train running time data, train speed, and train acceleration data, wherein the calculation method of the Z-score method is: ； Among them, x is the data to be processed, μ is the mean of the data to be processed, σ is the standard deviation of the data to be processed, and socre is the normalized standard score; The step S2 comprises the following steps: S21: define the state space, including: train position, train speed, train acceleration, train running time, road type, road speed limit; S22: define the action space, including: train traction force, train braking force; The step S3 comprises the following steps: S31: Define value function , is the expected return of taking action a in state s, calculated as: ； Where E is the expectation, γ is the discount factor, is the reward of the t-th iteration, is the initial state, is the initial action; S32: Adjust the value function according to the iteration step. The calculation method of the value function update formula is: ； in, is the learning rate, For the next state, For the next action, To determine the next action when the next Q value is the largest; The reward function calculation method of step S4 is: ； in, They are passenger comfort function, total energy consumption function, and punctuality function. are the weight coefficients of the three functions respectively; The passenger comfort function is calculated as: ； in, is the train acceleration at the t-th iteration step, is the total time of train acceleration and deceleration, is the train speed at step t, is the upper limit of the train speed at step t, is the tunnel entry and exit indicator at the t-th iteration step, is the influence coefficient of train acceleration, is the influence coefficient of the total time of train acceleration and deceleration, is the influence coefficient of overspeed in the tunnel, is the influence coefficient of entering and exiting the tunnel on the comfort; The calculation method of the total energy consumption function is: ； Where n is the number of sections where the train runs. is the average traction force in the i-th interval; is the average speed in the ith interval, is the traction efficiency in the i-th interval, is the running time in the ith interval, is the energy consumption of train auxiliary equipment, is the train braking energy consumption; The calculation method of the punctuality function is: ； in, is the actual arrival time of the train, is the scheduled arrival time of the train, Time is the maximum tolerable late arrival time of the train, To obtain the minimum value, To get the maximum value.

2. The high-speed train speed curve optimization method based on reinforcement learning according to claim 1 is characterized in that: The step S5 comprises the following steps: S51: Initialize the Q table and assign a random value to each state-action pair; S52: Set initial state The train speed, acceleration and running time are 0, and it starts moving from the departure station; S53: Repeat the following steps until the train arrives at the last station, including: Starting from the current state s, with probability Randomly choose an action with probability Choose the action with the largest Q value; Execute action a and observe the next state and current rewards ; According to the value function update formula, update the Q table ; Update s to ; Output the strategy with the highest score in the Q table; S54: Output the high-speed train speed curve result from the trained reinforcement learning model.

3. A high-speed train speed curve optimization system based on reinforcement learning, characterized in that: include: Data acquisition and preprocessing module: acquires road data, train running time data, train speed, and train acceleration data, and preprocesses them to obtain preprocessed road data, train running time data, train speed, and train acceleration data; State-action space building module: defines the state space and action space of the train speed curve optimization network; Value function construction module: constructs the value function of the train speed curve optimization network and updates the value function according to the iteration step; Reward function construction module: setting the reward function of the train speed curve optimization network, wherein the reward function includes a passenger comfort function, a total energy consumption function, and a punctuality function; Reinforcement learning training module: reinforcement learning training, updating Q table and outputting high-speed train speed curve; To realize the high-speed train speed curve optimization method based on reinforcement learning as described in any one of claims 1-2.

Images