Urban rail suburban line fast and slow vehicle stop scheme customization method and system based on reinforcement learning

Abstract

The invention relates to an urban rail train stop scheme customization method based on reinforcement learning. The method comprises the following steps: establishing a decision process model of an urban rail suburban line fast and slow train stop scheme; and solving the decision process model of the urban suburban line fast and slow vehicle stop scheme by adopting a double-depth Q network reinforcement learning algorithm, and customizing the urban suburban line fast and slow vehicle stop scheme. By simulating the operation conditions of the urban rail train under different stop schemes, the operation efficiency of the train and the waiting time of passengers can be effectively predicted and optimized. The method not only considers the running time of the train and the getting-on and getting-off time of passengers, but also integrates the transfer demands of the passengers and the passenger carrying capacity of the train, so that an accurate model is constructed, and the model is solved through the double-depth Q network. By means of the method, the fast and slow vehicle parking scheme with the minimum waiting time of the passengers can be accurately customized, and the travel experience of the passengers is improved.

Description

Technical Field

[0001] The invention relates to the field of urban rail transit systems, and in particular to a method for customizing a stop plan for fast and slow trains on an urban rail suburban line based on reinforcement learning. Background Art

[0002] The urban rail transit system has become an important mode of transportation for urban travel due to its advantages of high efficiency, punctuality, greenness and safety, and it also bears increasing passenger pressure. In 2023, Beijing's road network passenger volume will reach 3.454 billion, with an average daily passenger intensity of 11,700 passengers per kilometer. In order to meet the growing transportation demand, the company's capacity investment is also increasing year by year. In 2023, the Beijing road network train mileage will be 745.4333 million vehicle-kilometers, with 6 lines with a minimum driving interval of less than 2' and 11 lines with a minimum driving interval of less than 2'30", but there are still 6 lines with a peak hour maximum section load rate of more than 100%, and all of them are suburban lines heading into the city during the morning peak period. Under the condition of certain capacity resources, in order to alleviate the huge passenger flow pressure during peak hours, operating companies have taken a number of measures to make full use of limited capacity resources to solve the problem of uneven temporal and spatial distribution of passengers, including a multi-route mode with mixed large and small routes, a multi-marshaling mode with different marshaling lengths, an express train skipping stops or speeding up in the suburbs, and different combinations of the above modes, among which the operation of fast and slow trains is the main means of operation organization optimization discussed in this application. Many cities at home and abroad have successfully applied fast and slow trains. The operation is to meet the changing passenger flow demand. For example, Tokyo Tsukuba Line operates four types of trains. The fast train with the shortest travel time saves 12 minutes compared with ordinary trains that stop at every station by skipping 12 stations and passing 2 slow trains. The New York Subway realizes the co-line operation of fast and slow trains by building 3 or 4 tracks. The operation organization work under this mode is relatively simple. Shanghai Metro Line 16 has direct, express and slow trains, which take 34, 46 and 57 minutes respectively. During the morning and evening rush hours on weekdays, two direct trains run to Dishui Lake and the opposite direction respectively, and one express train runs every hour for the rest. Guangzhou Metro Line 14 has fast and slow trains on the main line. The express train stops at 4 stations, about one train every half an hour. The number of express trains from Monday to Thursday is about 10 more than that on Friday. Beijing Metro Line 6 runs a total of 5 express trains to Lucheng during the morning rush hour on weekdays. The express trains are about 9 minutes apart and each train skips 8 stations in a row, which saves about 7 minutes compared with slow trains.

[0003] Whether in actual operation or theoretical research, fast and slow train operation has become a widely adopted combined operation mode, and has achieved relatively rich research results. However, there are still the following deficiencies that need to be further improved: in actual operation, relevant staff often select some stations to adopt a continuous skipping strategy based on operating experience, and the specific calculation and analysis of passenger flow is still insufficient; some theoretical studies pre-set the skipping mode, which to a certain extent limits the diversity of stop plans; in addition, existing studies mostly use evolutionary algorithms such as genetic algorithms to solve the problem, but this type of algorithm requires a high time and computing power cost when dealing with real-time dynamically changing passenger flow demands, and the search process may fall into local optimality. The reinforcement learning algorithm has a strong multi-stage decision-making ability, but there is still little research on solving this problem.

[0004] The prior art patent application with application number 202411010743.7 discloses a deep reinforcement learning solution method based on a Markov decision process model of large and small intersections and fast and slow vehicle passing conditions, and the method includes the following steps:

[0005] Step 1: Establish a Markov decision process model for comprehensive large and small crossings and fast and slow train passing conditions, where the train arrival time, departure time and number of passengers on the platform are inputs, and the departure type, departure interval, express train stop position, crossing station position and passing station position are outputs. The state variables and action variables jointly describe the operation of the train, and the state transition function describes the process of the train changing from one state to another.

[0006] Step 2: Build and solve the corresponding deep reinforcement learning algorithm of the model, store the states, actions and rewards generated in the Markov decision process model as training data, learn the optimal Q value of the state-action pair through the DQN algorithm, and finally generate the optimal scheduling strategy; define the reward function to minimize the total cost, and in each step of training, update the Q value function according to the reward obtained for the current state and action and the maximum Q value of the next state. Through continuous iterative optimization, the trained strategy can achieve efficient and robust decision-making in a complex subway scheduling environment.

[0007] However, the above-mentioned existing technologies have some limitations. For example, in the process of model building, the dynamic changes of passenger demand are not sufficiently considered, and fewer factors are considered. In practical applications, it may be difficult to accurately predict and adapt to the complex and changeable passenger flow distribution. In addition, when solving the constructed model, this method uses the DQN algorithm for training. Although the DQN algorithm performs well in dealing with static problems, its convergence speed and stability still need to be improved when facing dynamically changing passenger flow demands. In addition, the algorithm framework is relatively simple in design and cannot fully reflect the complexity and diversity in actual operations, which may affect the real-time nature of decision-making. Summary of the invention

[0008] In order to solve the above problems, the present application proposes a method for customizing the stop plan of fast and slow trains on urban rail suburban lines based on deep reinforcement learning, the method comprising:

[0009] Establish a decision-making process model for the stop plan of express and slow trains on suburban rail lines;

[0010] A dual-depth Q-network reinforcement learning algorithm is used to solve the decision-making process model of the fast and slow train stop plan for the suburban line, and a customized fast and slow train stop plan for the suburban line is obtained.

[0011] Furthermore, the decision-making process model for establishing the stop plan for express and slow trains on the urban rail suburban line includes: passenger arrival follows Poisson distribution, passenger entry and transfer travel time follows normal distribution; running express and slow trains does not change the train running order; the platform can accommodate passengers entering the station.

[0012] Furthermore, the decision-making process model for establishing the stop plan for express and slow trains on the urban rail suburban line also includes: defining the number of passengers boarding train k at station i in the tth time period as:

[0013]

[0014] in represents the number of passengers boarding train k at station i during the tth period, x i,k,t Indicates that train K stops at station i during the tth period, x i,k,t =1; otherwise, x i,k,t =0, C max

[0015] Indicates the maximum passenger capacity of the train. represents the number of passengers (people) carried by train k arriving at station i during the tth period, The number of passengers getting off train k at station i during time period t (persons), represents the number of passengers waiting for train k at station i in time period t (persons);

[0016] The passengers who board the train get off at subsequent stations in order according to the relative proportion of the total passenger flow attracted, that is,

[0017]

[0018] in represents the number of passengers getting off train k at station i during the tth period, represents the passenger volume (people) of train k arriving at and leaving station m during the tth period, and qm,i,t represents the passenger flow demand (people) from station i to station m during the tth period;

[0019] The passenger capacity of train K is

[0020]

[0021] in represents the number of passengers (people) carried by train k leaving station i during the tth period, represents the number of passengers (people) that train k carries when it arrives at station i+1 during time period t, represents the number of passengers (people) carried by train k arriving at station i during the tth period, represents the number of passengers getting off train k at station i during the tth period, It represents the number of passengers (people) boarding train k at station i during the tth time period.

[0022] Furthermore, the decision-making process model for establishing the stop plan for express and slow trains on the urban rail suburban line also includes defining the number of passengers on the platform: when the passenger demand is greater than the remaining capacity of the train, the passengers are stranded on the platform.

[0023]

[0024] in represents the number of passengers (people) at station i who want to take train k but stay in the tth time period, represents the number of passengers waiting for train k at station i during the tth period (persons), It represents the number of passengers (people) boarding and alighting on train k at station i during the tth period.

[0025] Waiting passengers include passengers who are staying on the previous train and passengers entering / transferring between two trains.

[0026]

[0027] in represents the number of passengers waiting for train k at station i during the tth period (persons), represents the number of passengers (people) at station i who wanted to take train k-1 but stayed in the tth period, It represents the number of passengers (people) waiting for train k to enter or transfer at station i during the tth time period.

[0028] Furthermore, the decision-making process model for establishing the stop plan for the express and slow trains on the urban rail suburban line also includes defining the number of arriving passengers, and the passenger arrival compliance parameter is R i,j,t The Poisson distribution is obtained from the historical data of the automatic ticket vending and checking system. The average number of passengers arriving from the departure of train k-1 to the departure of train k in time period t is

[0029]

[0030] in, represents the departure time of train k at station i in the tth period, represents the departure time of train k-1 at station i in time period t.

[0031] Furthermore, the decision-making process model for establishing the stop plan for the express and slow trains on the suburban rail line also includes defining the time relationship, the time for passengers to enter the station and arrive at the platform is

[0032]

[0033] in, represents the time when passenger m enters the station and arrives at platform i during the tth period, represents the waiting time (s) of passenger m at station i during the tth period, It represents the average travel time of passengers to the platform after entering the station (s);

[0034] The time to arrive at the platform after transfer is

[0035]

[0036] in, represents the time when passenger m transfers and arrives at platform i in the tth period, represents the time when passenger m gets off at station i during the tth period, Average travel time for passengers to reach the platform after transferring (s);

[0037] Passenger waiting time is

[0038]

[0039] in represents the boarding time of passenger m at station i during the tth period, represents the time when passenger m arrives at station i in time period t;

[0040] The average waiting time for all passengers is

[0041]

[0042] Furthermore, the objective function of the decision-making process model of the stop plan of the urban rail suburban line fast and slow trains is to minimize the average waiting time of passengers, that is,

[0043]

[0044] Furthermore, the number of passengers arriving in time period t is input into the learning network as the current state s; secondly, the action a corresponding to the maximum Q value is found in the current network. *, calculate the corresponding target Q value in the target network; then, choose whether the train stops at the station as an action based on the Q value, input the state and action into the environment, complete the interaction between passengers and trains, get the state s′, calculate the average waiting time of passengers as the reward r and s′ and input it into the network again, so as to continuously iterate and update.

[0045] Furthermore, the dual-depth Q network reinforcement learning algorithm is used to solve the decision process model of the suburban line fast and slow train stop plan, which also includes

[0046] Step 1: Initialize the action set A, the number of iterations N, the current network Q and the target network Q′ with the same structure, the network parameters ω and ω′, the learning rate α, the discount factor γ, the update frequency υ, and the number of samples l; let the loop index t = 1, and the experience replay set M = {};

[0047] Step 2: In period t, according to state s t , in network Q, the ò-greedy strategy is used to select action a t ,

[0048]

[0049] That is, an action is uniformly selected in A with a probability of ò, and the optimal action is taken with a probability of 1-ò.

[0050] Step 3: Execute a t , observe the reward r t and the next state s t+1 , and (s t ,a t ,r t ,s t+1 ) is stored in the experience playback array M,

[0051] Step 4: Randomly select l individuals X from M, X = = (s y ,a y ,r y ,s y+1 ), X:Unif(M), y∈{1,L,l}. Calculate in the network Q such that the state s y+1 The largest Q value * , a * = argmax a Q(s y+1 ,a,ω); calculate a in network Q′ * The corresponding target Q value, q y =r y +γ·Q′(s y+1 ,a * ,ω′);

[0052] Step 5: Calculate the mean square error σ between the current Q value and the target Q value, Update the parameters ω of the network Q according to the gradient, ω←ω-α·▽ ω σ;

[0053] Step 6: If N≡1(modυ), update the network ω′, ω′=ω, and go to step 7; otherwise, go directly to step 7;

[0054] Step 7: If t<N, set t=t+1 and go to step 2; otherwise go to step 8;

[0055] Step 8: The loop ends.

[0056] On the other hand, the present application also provides a reinforcement learning-based urban rail train stop plan customization system including a memory and a processor, wherein the memory stores a computer program, and is characterized in that when the processor executes the computer program, the steps of the method described in any one of claims 1 to 9 are implemented.

[0057] Compared with the prior art, the beneficial technical effects of this application are:

[0058] On the one hand, this application constructs an optimization model for the peak period of suburban lines where the temporal and spatial distribution of passenger flow is extremely uneven, with the goal of minimizing passenger travel time. The model fully considers various factors and sets detailed parameters, which can truly simulate the scenario of passengers taking suburban lines, thereby ensuring the practicality and accuracy of the optimization model.

[0059] On the other hand, the present application solves the model constructed by the dual-depth Q network, inputs the number of passengers arriving in time period t as the current state s into the learning network; finds the action a corresponding to the maximum Q value in the current network * , calculate the corresponding target Q value in the target network; then, choose whether the train stops at the station as an action based on the Q value, input the state and action into the environment, complete the interaction between the passenger and the train, get the state s′, calculate the average waiting time of the passenger as the reward r and s′ and input it into the network again, so as to continuously iterate and update. This can make the calculation result closer to the optimal solution, thereby effectively reducing the average waiting time of passengers. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 : A schematic diagram of a method for customizing a stop plan for fast and slow trains on a suburban rail line based on reinforcement learning in this application;

[0061] Figure 2 : Schematic diagram of modeling of the method for customizing the stop plan of fast and slow trains on the urban rail suburban line based on reinforcement learning in this application;

[0062] Figure 3: Schematic diagram of the model solution algorithm of the method for customizing the stop plan of fast and slow trains on the suburban line of urban rail based on reinforcement learning in this application;

[0063] Figure 4 : Schematic diagram of the reinforcement learning-based urban rail suburban line fast and slow train stop plan customization system of the present application; DETAILED DESCRIPTION

[0064] Explanation of terms in this application:

[0065] Reinforcement learning: A machine learning method in which an agent learns how to perform tasks in a given environment by interacting with the environment. The agent receives state information of the environment at each time step, selects actions based on the current strategy, and then receives feedback from the environment, including rewards and new states. In this way, the agent gradually improves its strategy to maximize the long-term cumulative reward. In this application, reinforcement learning is applied to the customization of the stop plan for fast and slow trains on the urban rail suburban line to optimize the waiting time for passengers.

[0066] Suburban express and slow trains: In the urban rail transit system, a train operation mode combining express and slow trains is designed according to passenger flow and line characteristics. Express trains mainly serve passengers traveling long distances and stop at major stations, while slow trains increase the number of stops on the basis of express trains to serve short-distance passengers and provide more frequent services.

[0067] Poisson distribution: A statistical probability distribution that is often used to describe the probability of a random event occurring in a fixed time or space. In this application, Poisson distribution is used to simulate the random process of passenger arrivals to predict the number of passengers arriving at the platform in different time periods. By estimating the parameters of Poisson distribution, the arrival rate of passengers in a specific time period can be calculated more accurately.

[0068] Dual Deep Q Network: A deep reinforcement learning algorithm that improves learning stability and convergence speed by introducing two deep neural networks to estimate the action value function. In this application, the dual deep Q network is used to optimize the stop plan of fast and slow trains on the urban rail suburban line. By simulating passenger boarding and alighting behaviors and predicting passenger flow changes, the train stop strategy is intelligently adjusted to reduce the average waiting time of passengers and improve transportation efficiency.

[0069] Maximum Q value: In the framework of reinforcement learning, the maximum value refers to the maximum expected reward that the agent can obtain according to the current strategy under a given state. In this application, the maximum value is associated with minimizing the average waiting time of passengers, that is, when the agent chooses an action at each time step, it will take into account the long-term impact of the action on the waiting time of passengers and choose the action that will bring the minimum waiting time. In this way, the dual-depth Q network can learn how to adjust the train stop strategy under different passenger flow and time conditions to achieve the goal of optimizing passenger waiting time.

[0070] Action set A: In the reinforcement learning framework, an action set refers to the set of all possible actions that an agent can perform in a given state. In this application, the action set includes the train's stop decisions at different stations, such as choosing to stop or skipping a stop. The agent selects the best action by evaluating the impact of each action on the waiting time of passengers.

[0071] ò-Greedy Strategy: A commonly used exploration strategy in reinforcement learning, which allows the agent to choose the optimal action most of the time, but randomly chooses other actions with a smaller probability to explore new possible strategies. This strategy helps the agent avoid falling into local optimal solutions and find better solutions in the long run. In this application, the ò-Greedy Strategy is applied to the optimization process of the stop plan for fast and slow trains on the urban rail suburban line, ensuring that the agent can explore and learn a wider range of stop strategies while pursuing the minimization of the average waiting time of passengers, so as to find better solutions in the long run to reduce and minimize the average waiting time of passengers.

[0072] Experience replay array: A method for storing the experience gained by an agent during its interaction with the environment. In reinforcement learning, the agent gains experience through interaction with the environment, and these experiences usually include information such as state, action, reward, and new state. The experience replay array allows the agent to store these experiences and randomly extract these experiences in the subsequent learning process to break the dependency of the time series and improve learning efficiency. In this application, the experience replay array is used to store the experience in the process of optimizing the stop plan of fast and slow trains on the urban rail suburban line, so that the agent can more effectively learn how to adjust the train stop strategy to reduce the average waiting time of passengers.

[0073] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] like Figure 1As shown, the present application provides a method for customizing a stop plan for fast and slow trains on a suburban rail line based on reinforcement learning, the method comprising the following steps:

[0075] S1 Establish a decision-making process model for the stop plan of fast and slow trains on the suburban line of urban rail;

[0076] S2 uses a dual-depth Q-network reinforcement learning algorithm to solve the decision-making process model of the fast and slow train stop plan for the suburban line, and customizes the fast and slow train stop plan for the suburban line.

[0077] In some embodiments, the specific process of model construction in step S1 is:

[0078] like Figure 2 As shown, for the convenience of problem description, this application is based on the following assumptions: 1) During the study period, passenger arrivals follow a Poisson distribution, and passenger entry and transfer travel time follow a normal distribution; 2) The operation of fast and slow trains does not change the train operation sequence; 3) The platform can accommodate the passengers entering the station. The main symbols, parameters and variables involved in the paper are shown in Table 1.

[0079]

[0080]

[0081]

[0082] Table 1 Main symbols, parameters and variables

[0083] In this application, the parameters are defined as follows:

[0084] 2.1 Parameter calculation

[0085] (1) Number of passengers getting on and off the bus

[0086] Whether passengers can get on the train depends on whether the train stops at the station or not.

[0087] The relationship between the remaining capacity of the vehicle and the passenger flow demand. Therefore, the number of passengers on the vehicle is

[0088]

[0089] In the above formula (1) represents the number of passengers boarding train k at station i during the tth period, x i,k,t Indicates that train K stops at station i during the tth period, x i,k,t =1; otherwise, x i,k,t =0, C max Indicates the maximum passenger capacity of the train. represents the number of passengers (people) carried by train k arriving at station i during the tth period, The number of passengers getting off train k at station i during time period t (persons), represents the number of passengers waiting for train k at station i in the tth time period; N is the total number of stations, and the passenger capacity of the departure station when the train arrives is 0, that is,

[0090] In this application, the passengers in the car get off at the subsequent stations in order according to the relative proportion of the total passenger flow attracted, that is,

[0091]

[0092] In the above formula (2), represents the number of passengers getting off train k at station i during the tth period, represents the number of passengers (people) of train k arriving at and leaving station m during time period t, q m,i,t represents the passenger flow demand (people) from station i to station m during the tth period;

[0093] (2) Train passenger capacity

[0094]

[0095] In the above formula (3), represents the number of passengers (people) carried by train k leaving station i during the tth period, represents the number of passengers (people) that train k carries when it arrives at station i+1 during time period t, represents the number of passengers (people) carried by train k arriving at station i during the tth period, represents the number of passengers getting off train k at station i during the tth period, It represents the number of passengers (people) boarding train k at station i during the tth period, and the number of passengers leaving the final station is 0, that is,

[0096] (3) Number of passengers on the platform

[0097] When passenger demand exceeds the remaining capacity of the train, passengers are stranded on the platform.

[0098]

[0099] In the above formula (4) represents the number of passengers (people) at station i who want to take train k but stay in the tth time period, represents the number of passengers waiting for train k at station i during the tth period (persons), It represents the number of passengers (people) boarding and alighting on train k at station i during the tth period.

[0100] Waiting passengers include passengers who are staying on the previous train and passengers entering / transferring between two trains.

[0101]

[0102] In the above formula (5) represents the number of passengers waiting for train k at station i during the tth period (persons), represents the number of passengers (people) at station i who wanted to take train k-1 but stayed in the tth period, It represents the number of passengers (people) waiting for train k to enter or transfer at station i during the tth time period.

[0103] (4) Number of arriving passengers

[0104] According to the assumptions, the passenger arrival is subject to the parameter R i,j,t The Poisson distribution can be obtained from the historical data of the automatic ticket vending and checking system. Then the average number of passengers arriving from the departure of train k-1 to the departure of train k in time period t is

[0105]

[0106] In the above formula (6) represents the departure time of train k at station i in the tth period, represents the departure time of train k-1 at station i in time period t.

[0107] by The number of passengers entering or transferring within a certain time period is a random variable X, then

[0108] According to the properties of Poisson distribution, we can get the time period The probability distribution of the number of passengers arriving in is:

[0109]

[0110] In the above formula, e is the base of natural logarithm.

[0111] In practical applications, in order to more accurately predict passenger arrivals, more factors may need to be considered, such as holidays, special events, weather conditions, etc., all of which may have an impact on passenger arrival patterns.

[0112] (5) Time relationship

[0113] Assuming that the platform is large enough to accommodate incoming passengers, the time it takes for passengers to enter the station and walk to the platform is

[0114]

[0115] In the above formula (8), represents the time when passenger m enters the station and arrives at platform i during the tth period, represents the waiting time (s) of passenger m at station i during the tth period, It represents the average travel time of passengers to the platform after entering the station (s);

[0116] The time to arrive at the platform after transfer is

[0117]

[0118] In the above formula (9) represents the time when passenger m transfers and arrives at platform i in the tth period, represents the time when passenger m gets off at station i during the tth period, Average travel time for passengers to reach the platform after transferring (s);

[0119] Passenger waiting time is

[0120]

[0121] In the above formula, represents the boarding time of passenger m at station i during the tth period, represents the time when passenger m arrives at station i in time period t;

[0122] In actual operation, it usually takes 3 to 5 seconds from the time the train stops at the station to the time the doors open. This time can be ignored, that is, the time when the train arrives at the station and the passengers get on the train is the same.

[0123]

[0124] The average waiting time for passengers is

[0125]

[0126] According to the assumption that the passenger's entry and exit time follows the expected t in,μ , standard deviation t in,σ If the normal distribution of

[0127]

[0128] In the formula, It represents the average travel time of passengers to the platform after entering the station (s), t in,μ represents the expected time of passengers entering the station and traveling, t in,σ represents the standard deviation of the time it takes for passengers to enter the station and travel. e is the base of the natural logarithm. The probability density function represents the passenger's entry and travel time.

[0129] According to the assumption that the passenger transfer travel time follows the expected t trans,μ , standard deviation t trans,σ If the normal distribution of

[0130]

[0131] In the formula, represents the average travel time (s) of passengers to the platform after transfer, t trans,μ represents the expected transfer travel time of passengers, t trans,σ represents the standard deviation of the passenger transfer travel time, e is the base of the natural logarithm, The probability density function represents the passenger transfer travel time.

[0132] 2.2 Objective Function

[0133] The goal is to minimize the average waiting time of passengers.

[0134]

[0135] Where: Z is the optimization target.

[0136] 2.3 Constraints

[0137] (1) Train continuous cross-station constraints ensure that the train will not skip directly from the first station to the last station and ensure that it stops at the last station.

[0138]

[0139] In the above formula, x i+m-1,k,t Indicates that train k stops at station i+m-1 during the tth period, then x i,k,t =1; otherwise, x i,k,t =0,M max,t It is the upper limit of the train's continuous crossing of stations.

[0140] (2) Constraints on trains that continuously skip stops at stations to ensure that the number of trains that continuously skip stops at stations does not exceed the upper limit:

[0141]

[0142] Where: M max,s It is the upper limit of trains continuously stopping at a station.

[0143] (3) Station stop restrictions

[0144]

[0145] (4) Tracking interval constraints

[0146] Due to the speed difference between fast and slow trains, in order to ensure that adjacent trains do not interfere with each other,

[0147]

[0148] in,

[0149]

[0150]

[0151] Through the above steps, a complete optimization model for the stop plan of the express and slow trains on the urban rail suburban line was constructed. The model fully considers the various factors affecting passengers' riding on the urban rail suburban line, the randomness of passengers arriving at the station and the constraints of the operation of the urban rail express and slow trains, and aims to accurately reduce the average waiting time of passengers and improve the efficiency of train operation.

[0152] In some embodiments, Figure 3 As shown in FIG. 1 , in step S2, the specific process of using the dual-depth Q network reinforcement learning algorithm to solve the decision-making process model of the suburban line fast and slow train stop plan is as follows:

[0153] The process of considering the train stop plan is a multi-stage dynamic decision-making process, and the reinforcement learning algorithm has obvious advantages in solving multi-stage dynamic combinatorial optimization problems. At the same time, the state dimension disaster caused by continuous tasks and the problem of over-estimation of a single network, this paper adopts the Double Deep Q Network (DDQN) to solve the constructed model.

[0154] The algorithm framework mainly includes elements such as state, action, environment and reward. First, the number of passengers arriving in time period t is input into the learning network as the current state s; second, the action a corresponding to the maximum Q value is found in the current network. * , calculate the corresponding target Q value in the target network; then, choose whether the train stops at the station as an action based on the Q value, input the state and action into the environment, complete the interaction between passengers and trains, get the state s′, calculate the average waiting time of passengers as the reward r and s′ and input it into the network again, so as to continuously iterate and update. The specific calculation process is as follows:

[0155] Step 1: Initialize the action set A, the number of iterations N, the current network Q and the target network Q′ with the same structure, the network parameters ω and ω′, the learning rate α, the discount factor γ, the update frequency υ, and the number of samples l; let the loop index t = 1, and the experience replay set M = {};

[0156] Step 2: In period t, according to state s t , in network Q, the ò-greedy strategy is used to select action a t ,

[0157]

[0158] An action is uniformly selected from A with probability ò, and the best action is taken with probability 1-ò.

[0159] Step 3: Execute a t , observe the reward r t and the next state s t+1 , and (s t ,a t ,r t ,s t+1 ) is stored in the experience playback array M,

[0160] Step 4: Randomly select l individuals X from M, X = = (s y ,a y ,r y ,s y+1 ), X:Unif(M), y∈{1,L,l}. Calculate in the network Q such that the state s y+1 The largest Q value * , a * = argmax a Q(s y+1 ,a,ω); calculate a in network Q′ * The corresponding target Q value, q y =r y +γ·Q′(s y+1 ,a * ,ω′);

[0161] Step 5: Calculate the mean square error σ between the current Q value and the target Q value, Update the parameters ω of the network Q according to the gradient; ω←ω-α·▽ ω σ;

[0162] Step 6: If N≡1(modυ), update the network ω′, ω′=ω, and go to step 7; otherwise, go directly to step 7;

[0163] Step 7: If t<N, set t=t+1 and go to step 2; otherwise go to step 8;

[0164] Step 8: The loop ends.

[0165] In summary, after completing the above steps, the algorithm will gradually optimize the strategy through continuous iteration in order to achieve the goal of minimizing the average waiting time of passengers. In practical applications, the algorithm can effectively cope with the dynamic changes of passenger flow and improve train operation efficiency and passenger satisfaction. In addition, by adjusting hyperparameters such as learning rate and discount factor, the performance of the model can be further improved to better adapt to different operating scenarios.

[0166] In some embodiments, Figure 4As shown, the present application also provides a system 10 for triggering conditions of an operation activity method for issuing customized rewards to users based on different trigger conditions. The system 10 includes at least one processor 101, a memory 102, an input device 103 and a display device 104. The input device 103 is used to obtain input from the outside world. The memory 102 stores instructions. When the instructions are executed by at least one processor 101, the steps of the method described in the method embodiment are implemented, and the running results are displayed on the display device 104, and the steps of the method described in the method embodiment are implemented.

[0167] In one embodiment of the present invention, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for operating an activity for issuing customized rewards to users based on different trigger conditions as described in any of the above embodiments are executed.

[0168] The embodiments of this document also provide a computer-readable instruction, wherein when a processor executes the instruction, the program therein causes the processor to execute the steps of the operation activity method of issuing customized rewards to users based on different trigger conditions as described in any of the above embodiments.

[0169] The embodiments and functional operations of the subject matter described in this specification may be implemented in: digital electronic circuitry, tangibly implemented computer software or firmware, computer hardware, including the structures disclosed in this specification and their structural equivalents, or a combination of more than one of the above. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on one or more tangible non-transitory program carriers, for execution by a data processing device or to control the operation of a data processing device.

[0170] Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to an appropriate receiver device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of the above devices.

[0171] The processes and logic flows described in this specification may be performed by one or more programmable computers that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits), and the apparatus may also be implemented as special purpose logic circuits.

[0172] To transmit interactions with a user, embodiments of the subject matter described in this specification may be implemented on a computer having: a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user; and a keyboard and a pointing device, such as a mouse or a trackball, with which the user can transmit input to the computer. Other kinds of devices may also be used to transmit interactions with a user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic input, voice input, or tactile input. In addition, a computer may interact with a user by sending documents to a device used by the user and receiving documents from the device; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0173] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or the scope of what can be claimed, but rather as an illustration of features that can concretize a particular embodiment of a particular invention. The specific features described in this specification in the context of independent embodiments can also be implemented in combination with a single embodiment. Conversely, the various features described in the context of a single embodiment can also be implemented independently in multiple embodiments, or in any suitable sub-combination. In addition, although the features can be described above as a combination effect and even initially required, one or more features from the required combination can be removed from the combination in some cases, and the required combination can be turned to a sub-combination or a variation of the sub-combination.

[0174] Similarly, although operations are described in the accompanying drawings in a particular order, it should not be understood that: in order to achieve the desired results, such operations are required to be performed in the particular order shown or in a sequential order, or that all illustrated operations are performed. In certain cases, multitasking and parallel processing can be advantageous. In addition, the separation of various system modules and components in the above-mentioned embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0175] Specific implementations of the subject matter have been described. Other implementations are within the scope of the following claims. For example, the activities recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes described in the accompanying drawings do not necessarily require the specific order or sequential order shown in order to achieve the desired results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

1. A method for customizing a stop plan for urban rail trains based on reinforcement learning, characterized in that: The method comprises the following steps: Establish a decision-making process model for the stop plan of express and slow trains on suburban rail lines; A dual-depth Q-network reinforcement learning algorithm is used to solve the decision-making process model of the fast and slow train stop plan for the suburban line, and a customized fast and slow train stop plan for the suburban line is obtained.

2. The method for customizing a stop plan for an urban rail train based on reinforcement learning as claimed in claim 1, characterized in that: The decision-making process model for establishing a stop plan for fast and slow trains on suburban rail lines includes: passenger arrival follows Poisson distribution, passenger entry and transfer travel time follows normal distribution; running fast and slow trains does not change the train running order; the platform can accommodate passengers entering the station.

3. The method for customizing a stop plan for an urban rail train based on reinforcement learning as claimed in claim 1, characterized in that: The decision-making process model for establishing the stop plan for express and slow trains on the urban rail suburban line also includes: defining the number of passengers boarding train k at station i in the tth time period as: in represents the number of passengers boarding train k at station i during time period t, x i,k,t Indicates that train K stops at station i during the tth period, x i,k,t =1; otherwise, x i,k,t =0, C max Indicates the maximum passenger capacity of the train. represents the number of passengers (people) carried by train k arriving at station i during the tth period, The number of passengers getting off train k at station i during time period t (persons), represents the number of passengers waiting for train k at station i in time period t (persons); The passengers who board the train get off at subsequent stations in order according to the relative proportion of the total passenger flow attracted, that is, in represents the number of passengers getting off train k at station i during the tth period, represents the number of passengers (people) of train k arriving at and leaving station m during time period t, q m,i,t represents the passenger flow demand (people) from station i to station m during the tth period; The passenger capacity of train K is in represents the number of passengers (people) carried by train k leaving station i during the tth period, represents the number of passengers (people) that train k carries when it arrives at station i+1 during time period t, represents the number of passengers (people) carried by train k arriving at station i during the tth period, represents the number of passengers getting off train k at station i during the tth period, It represents the number of passengers (people) boarding train k at station i during the tth time period.

4. The method for customizing a stop plan for an urban rail train based on reinforcement learning as claimed in claim 3, characterized in that: The decision-making process model for establishing the stop plan for fast and slow trains on the urban rail suburban line also includes defining the number of passengers on the platform: when the passenger demand is greater than the remaining capacity of the train, the passengers are stranded on the platform. in represents the number of passengers (people) at station i who want to take train k but stay in the tth time period, represents the number of passengers waiting for train k at station i during the tth period (persons), It represents the number of passengers (people) boarding and alighting on train k at station i during the tth period. Waiting passengers include passengers who are staying on the previous train and passengers entering / transferring between two trains. in represents the number of passengers waiting for train k at station i during the tth period (persons), represents the number of passengers (people) at station i who wanted to take train k-1 but stayed in the tth period, It represents the number of passengers (people) waiting for train k to enter or transfer at station i during the tth time period.

5. The method for customizing a stop plan for an urban rail train based on reinforcement learning as claimed in claim 3, characterized in that: The decision-making process model for establishing the stop plan for the suburban line of the urban rail line also includes defining the number of arriving passengers, and the passenger arrival obedience parameter is R i,j,t The Poisson distribution is obtained from the historical data of the automatic ticket vending and checking system. The average number of passengers arriving from the departure of train k-1 to the departure of train k in time period t is in, represents the departure time of train k at station i in the tth period, represents the departure time of train k-1 at station i in time period t.

6. The method for customizing a stop plan for an urban rail train based on reinforcement learning as claimed in claim 3, characterized in that: The decision-making process model for establishing the stop plan for the suburban rail line express and slow trains also includes defining the time relationship, the time for passengers to enter the station and arrive at the platform is in, represents the time when passenger m enters the station and arrives at platform i during the tth period, represents the waiting time (s) of passenger m at station i during the tth period, It represents the average travel time for passengers to reach the platform after entering the station (s); The time to arrive at the platform after transfer is in, represents the time when passenger m transfers and arrives at platform i in the tth period, represents the time when passenger m gets off at station i during the tth period, Average travel time for passengers to reach the platform after transferring (s); Passenger waiting time is in represents the boarding time of passenger m at station i during the tth period, represents the time when passenger m arrives at station i in time period t; The average waiting time for all passengers is In the above formula, t i,k,t represents the average waiting time (s) of passengers before train k arrives at station i in the tth period, represents the waiting time (s) of passenger m at station i during the tth period, It represents the number of passengers (people) waiting for train k to enter or transfer at station i during the tth time period.

7. The method for customizing a stop plan for an urban rail train based on reinforcement learning as claimed in claim 1, characterized in that: The decision process model for establishing the stop plan of the express and slow trains on the urban rail suburban line also includes: the objective function of the decision process model for the stop plan of the express and slow trains on the urban rail suburban line is to minimize the average waiting time of passengers, that is, In the above formula, t i,k,t represents the average waiting time of passengers before train k arrives at station i in the tth period (s) 8. The method for customizing a stop plan for an urban rail train based on reinforcement learning as claimed in claim 1, characterized in that ,The number of passengers arriving in period t is input into the learning network as the current state s; Secondly, find the action corresponding to the maximum Q value in the current network Calculate the corresponding target Q value in the target network; Then, whether the train stops at the station is selected as an action based on the Q value, the state and action are input into the environment, the interaction between the passenger and the train is completed, the state s′ is obtained, and the average waiting time of the passengers is calculated as the reward r and s′ and input into the network again, so as to continuously iterate and update.

9. The method for customizing a stop plan for an urban rail train based on reinforcement learning as claimed in claim 8, characterized in that: The decision-making process model of the fast and slow train stop plan of the suburban line is solved by using the dual deep Q network reinforcement learning algorithm, including Step 1: Initialize the action set A, the number of iterations N, the current network Q and the target network Q′ with the same structure, the network parameters ω and ω′, the learning rate α, the discount factor γ, the update frequency υ, and the number of samples l; let the loop index t = 1, and the experience replay set M = {}; Step 2: In period t, according to state s t , in network Q, the ò-greedy strategy is used to select action a t , An action is uniformly selected from A with probability ò, and the best action is taken with probability 1-ò. Step 3: Execute a t , observe the reward r t and the next state s t+1 , and (s t ,a t ,r t ,s t+1 ) M=[M|(s t ,a t ,r t ,s t+1 )] is stored in the experience playback array M, M = [M|(s t ,a t ,r t ,s t+1 )]. Step 4: Randomly select l individuals X from M, X = (s y ,a y ,r y ,s y+1 ), X:Unif(M), y∈{1,L,l}. Calculate in the network Q such that the state s y+1 The largest Q value * , a * = argmax a Q(s y+1 ,a,ω); calculate a in network Q′ * The corresponding target Q value, q y =r y +γ·Q′(s y+1 ,a * ,ω′); Step 5: Calculate the mean square error σ between the current Q value and the target Q value, Update the parameters ω of the network Q according to the gradient, Step 6: If N≡1(modυ), update the network ω′, ω′=ω, and go to step 7; otherwise, go directly to step 7; Step 7: If t<N, set t=t+1 and go to step 2; otherwise go to step 8; Step 8: The loop ends.

10. A system for customizing a stop plan for an urban rail train based on reinforcement learning comprises a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

Images