Real-time adjustment method of urban rail transit train operation diagram based on reinforcement learning
By proposing a reinforcement learning-based method for real-time adjustment of urban rail transit train schedules, and improving the dual deep Q-network using a competitive deep Q-network, the train timetable is optimized. This solves the problems of high computational resource consumption and poor real-time performance in existing technologies, and enables real-time adjustment of train schedules and reduction of passenger waiting time.
Patent Information
- Application Number
- CN202411152875.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-21
AI Technical Summary
Existing methods for handling train delays in urban rail transit consume large amounts of computational resources, have poor real-time performance, are prone to getting stuck in local optima, and are difficult to adapt to diverse tasks. Furthermore, traditional reinforcement learning methods have high computational overhead and cannot meet the requirements for real-time response.
A real-time adjustment method for urban rail transit train timetables based on reinforcement learning is proposed. This method improves the dual deep Q-network by using a competitive deep Q-network to construct an adjustment model. By combining constraints such as headway, running time, station dwell time, passenger flow, and turnaround time, the reinforcement learning algorithm is used to optimize the train timetable and achieve real-time adjustment.
It has improved the real-time adjustment capability of train timetables, optimized train intervals, reduced passenger waiting time, and met the high efficiency and high quality requirements of urban rail transit operation management.
Smart Images

Figure CN118674235B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of urban rail transit operation organization scheme optimization, and more specifically, to a real-time adjustment method for an urban rail transit train operation diagram based on reinforcement learning. Background Art
[0002] The rail transit systems in most cities in my country have entered a new stage of ultra-large-scale network operation. However, as the years of operation increase, some lines and facilities are aging, resulting in frequent train delays. Especially during peak hours, once a delay occurs, the impact is wide and long, including passenger delays, reduced train capacity, reduced service quality, and even endangering passenger safety. At present, the emergency response to urban rail train delays mainly relies on the dispatcher's experience decision-making and manual intervention.
[0003] At the theoretical research level, heuristic algorithms are currently widely used to solve rail transit scheduling problems. However, considering that heuristic algorithms are prone to fall into local optimal solutions and have poor adaptability, reinforcement learning algorithms are currently used to solve the problem of train operation adjustment. The existing methods for urban rail train adjustment have the following problems: First, the existing manual or traditional methods require a lot of computing resources and time when dealing with complex problems, resulting in a very slow solution process, which limits their feasibility in practical applications. Second: Operational optimization algorithms are prone to fall into local optimal solutions during the search process and cannot find the global optimal solution. This limits their effectiveness in high-dimensional complex problems. Third: Some reinforcement learning methods usually require manual adjustment of parameters or design of specific heuristic rules to adapt to different types of problems. This rigidity makes it difficult for them to adapt to diverse tasks. Fourth: In applications that require real-time response or processing of large-scale data, existing reinforcement learning methods often cannot provide sufficient real-time performance because of their high computational overhead. Summary of the invention
[0004] In order to solve the above technical problems, the present invention proposes a real-time adjustment method for urban rail transit train operation diagram based on reinforcement learning, which improves the scheduling efficiency and meets the high standards of urban rail transit operation management personnel for scheduling quality and efficiency.
[0005] The present invention provides a method for real-time adjustment of an urban rail transit train operation diagram based on reinforcement learning, comprising:
[0006] S1: Based on the basic operating conditions of the line, an adjustment model is constructed with the objective function of minimizing the sum of the deviation between the actual train operation schedule and the planned train operation schedule and the total waiting time of passengers;
[0007] S2: Determine the constraint conditions of the adjustment model;
[0008] S3: Introduce reinforcement learning algorithm based on the adjustment model and initialize the adjustment optimization model;
[0009] S4: Use the competitive deep Q network to improve the dual deep Q network and obtain the final adjusted optimization model;
[0010] S5: Using the adjustment optimization model to repeatedly train different delay situations of different lines to obtain different matrix lookup tables, importing them into the urban rail transit train mapping system, and obtaining an improved train operation schedule in real time based on the demand information and the delay situation.
[0011] In this scheme, an adjustment model is constructed, and the modeling goal is to minimize the sum of the deviation between the adjusted train operation schedule and the planned train operation schedule and the total waiting time of passengers;
[0012] The objective function of the adjustment model is expressed as: in, It represents the sum of the deviation between the actual train operation schedule after modification and the planned train operation schedule and the total waiting time of passengers. represents the train label, Indicates the total number of trains; Indicates the station index of the intersection, Indicates the set of platforms going up and down the route; and Respectively represent the time when the actual train arrives at the platform and the time when the planned train arrives at the platform; and The actual train departure time and the planned train departure time are respectively The conversion coefficient representing the magnitude of the train operation time deviation value, represents the conversion factor representing the magnitude of the total waiting time for passengers, Represents the total waiting time of passengers.
[0013] In this scheme, the constraint conditions of the adjustment model are constructed based on the train interval constraint, running time constraint, station stop time constraint, passenger flow constraint and return time constraint.
[0014] In this solution, a reinforcement learning algorithm is introduced on the basis of the adjustment model to initialize the adjustment optimization model. The state representation of the adjustment optimization model is specifically:
[0015] Initial time step Corresponding to the master clock time referenced by the events occurring on each service line, when the train departs according to the planned schedule, under the condition of delay, the interval time between the front and rear vehicles is calculated according to the interval time between the trains running under the currently adjusted schedule; define the state feature set, expressed as: , among which Represents each time step The local observation value on is the train status of the platform, using the operator Determine the nearest train, represents the train label, Indicates the actual train arrival time at the platform. Indicates the station index of the intersection, Indicates the set of platforms going up and down the route;
[0016] Local state vector The calculation process is expressed as: ,in Indicates the delay value, Indicates the distance between the nearest train on the platform and the preceding train. Indicates the distance between the nearest train on the platform and the following train.
[0017] In this solution, a reinforcement learning algorithm is introduced on the basis of the adjustment model to initialize the adjustment optimization model. The action representation of the adjustment optimization model is specifically:
[0018] Three action strategies are set according to the actual operation situation for the train to choose to use during the movement:
[0019] Strategy A represents the normal state, where the train does not take any action and maintains the current state;
[0020] Strategy B represents the movement of trains, where the train chooses a time level for a section when entering a block. ;symbol Indicates taking the largest Value function maximize;
[0021] Strategy C designs train detention, and the train determines the detention time at the station before departure ;
[0022] According to the given state feature set, the train will execute the action strategy at each time step. The calculation of the action strategy is expressed as: ;
[0023] Local Motion Vector The calculation process is expressed as: .
[0024] In this solution, a reinforcement learning algorithm is introduced based on the adjustment model to initialize the adjustment optimization model. The reward function definition of the adjustment optimization model is specifically as follows:
[0025] The reward function is set according to whether the interval between trains meets the requirements of safe operation and whether the normal and punctual train operation schedule can be quickly restored;
[0026] The train stop delay is divided into two situations. The first situation is that the train needs to stop at the station. In this case, the train does not depart at the scheduled time of the planned timetable or the delay is not resolved. Rewards are given based on the difference between the current inspection time and the scheduled departure time of the train and the difference between the actual train operation timetable and the planned train operation timetable. calculate;
[0027] The second situation is that when the delay problem has been resolved and the delayed train has departed or the unaffected train has departed normally, the reward will be given according to the difference between the actual departure time of the train and the scheduled departure time. calculate;
[0028] The reward calculation related to the reward combination delay in the two cases is expressed as: The train interval constraints, running time constraints, station stop time constraints, passenger flow constraints and return time constraints in the adjustment model are converted into reward expressions.
[0029] In this scheme, the competitive deep Q network is used to improve the dual deep Q network to obtain the final adjustment optimization model, which is as follows:
[0030] The dual deep Q network DDQN is introduced as the basic model for adjusting the optimization model, setting two independent Q functions, the main network For selecting actions, target networks For evaluating actions, the Q-value calculation of the dual deep Q network DDQN is expressed as:
[0031] in, represents the parameters of the main network, represents the local state vector, represents the local action vector, represents the learning rate, represents the action selected by the dual deep Q network, represents the train label, Indicates the station index of the intersection;
[0032] The competitive deep Q network is used to improve the dual deep Q network to generate a competitive dual deep Q network framework. By introducing two branch networks, one for evaluating the value of the state and the other for evaluating the pros and cons of each action, the improved network framework learns the relationship between the state and the action more effectively, so that the adjustment optimization model can learn the value of the state and the relative advantages of each action separately.
[0033] In this scheme, the action value corresponding to the competitive dual-depth Q network is The calculation is as follows: in Status The value of is directly output through a fully connected layer; For Action Relative to the state The advantage is output through a fully connected layer with the same size as the action space, and each action corresponds to an advantage value; is the size of the action space, represents the value trade-off parameter in the competitive dual deep Q network, Represents the dominance parameter in the balanced competitive dual deep Q-network.
[0034] Compared with the prior art, the present invention has the following beneficial effects:
[0035] The method proposed by the present invention has the function of real-time adjustment; it supports the pre-training of models so that the model can be quickly retrieved in the event of a delay or other emergency, and the corresponding train detention or operation level change strategies can be selected in real time and dynamically, which has strong real-time performance. At the same time, the present invention has good expansion and migration capabilities.
[0036] The train operation diagram optimized by the present invention has a more uniform running interval; according to the actual situation of on-site delays, with the goal of minimizing the total running delay time of the running trains, the subsequent trains are detained and the train interval running level is modified in a coordinated manner, so that the subsequent trains run as evenly as possible. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the technical solutions in the embodiments or exemplary embodiments of the present invention, the drawings required for use in the embodiments or exemplary descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained according to the drawings without paying creative work.
[0038] Figure 1 The present invention is a flow chart of a method for real-time adjustment of an urban rail transit train operation diagram based on reinforcement learning.
[0039] Figure 2 This is the subway circulation route map studied in this invention.
[0040] Figure 3 The figure is a schematic diagram of train interaction under the moving block signal system of the present invention.
[0041] Figure 4 A schematic diagram of the queuing process and passenger waiting time at a particular station according to the present invention.
[0042] Figure 5 It is a schematic diagram of the state definition of a train delay according to the present invention.
[0043] Figure 6 It is a schematic diagram of the competitive deep Q network structure of the present invention.
[0044] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION
[0045] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.
[0046] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.
[0047] Figure 1 A flow chart of a method for real-time adjustment of an urban rail transit train operation diagram based on reinforcement learning according to the present invention is shown;
[0048] like Figure 1 As shown, the present invention provides a method for real-time adjustment of an urban rail transit train operation diagram based on reinforcement learning, comprising:
[0049] S1: According to the basic operating conditions of the line, an adjustment model is constructed with the objective function of minimizing the sum of the deviation between the actual train operation schedule and the planned train operation schedule and the total waiting time of passengers.
[0050] In this step, the conventional bidirectional circulation operation mode of the subway line is considered, which usually consists of stations, storage lines and depots. Figure 2 As shown, the collection of sites is Station and Station Stations and stations between and Station The stations in between are indicated as turnaround stations.
[0051] At present, subway trains generally adopt communication-based train control system (CBTC) and operate in moving block mode. The core advantage of moving block mode is to use real-time communication technology to dynamically adjust the train running interval to ensure the safe and efficient operation of the train. In this mode, the running interval between trains is not statically fixed, but dynamically adjusted according to the real-time distance between trains and the braking performance of the trains. Figure 3 As shown in the figure, the line segment after train k-1 represents the moving block boundary, which is a dynamic boundary of the train operation area and changes as the train moves. This design allows the train to follow the preceding vehicle more closely while maintaining a safe distance, thereby improving the transportation efficiency and capacity of the track. The red dot after train k-1 represents the scheduled stopping position or target point of train k.
[0052] The core of the train schedule adjustment problem is to quickly restore the normal operation of the train and run it according to the planned schedule as soon as possible, while also considering the passenger experience. The adjustment model is constructed, and the modeling goal is to minimize the sum of the deviation between the adjusted train operation schedule and the planned train operation schedule and the total waiting time of passengers;
[0053] The objective function of the adjustment model is expressed as: in, It represents the sum of the deviation between the actual train operation schedule after modification and the planned train operation schedule and the total waiting time of passengers. represents the train label, Indicates the total number of trains; Indicates the station index of the intersection, Indicates the set of platforms going up and down the route; and Respectively represent the time when the actual train arrives at the platform and the time when the planned train arrives at the platform; and The actual train departure time and the planned train departure time are respectively The conversion coefficient representing the magnitude of the train operation time deviation value, represents the conversion factor representing the magnitude of the total waiting time for passengers, Represents the total waiting time of passengers.
[0054] S2: Determine the constraint conditions of the adjustment model.
[0055] The constraints of the adjustment model are constructed based on the train interval constraints, running time constraints, station stop time constraints, passenger flow constraints and return time constraints.
[0056] Train interval constraint: Subway trains are operated in sections, so a certain distance needs to be maintained between subway trains to ensure that the trains can brake safely and meet the rules of section operation, and also meet the conditions for responding to various emergency situations. Therefore, the minimum tracking interval constraint of subway trains is expressed as: in Indicates the minimum tracking interval of the positive line.
[0057] Running time constraint: The running time of a train in a section can be divided into several levels, called section running time levels, denoted by In the adjustment model, the Boolean variable To represent the running time level of the currently selected section. The running time constraint of the train in each section is expressed as: ,The above formula means that only one segment's segment operation time level can be selected; Indicates that the train running time within a section is limited to the specified time corresponding to the specific section running time level.
[0058] When a train is delayed, a moving block system is used. If the distance between the two trains is less than the sum of the minimum braking distance and the safety distance, the rear train can only stop. If it is greater than the minimum braking distance Safe distance If the sum of the speed levels is , different speed levels can be selected, and the distance between the two trains must be greater than the sum of the distance and the safety distance under the current operating level selection. The constraint representation of the train running time is expanded and expressed as: Station stop time constraint: The minimum stop time of a train passing through a station is related to the train door opening and closing time, the passenger boarding and alighting time, among which the train door opening and closing time is only related to the nature of the train, which can be expressed as: ,in Indicates the minimum station dwell time.
[0059] Passenger flow constraints: When formulating passenger flow constraints, the needs of passengers and the boarding and alighting process are taken into consideration. represents the passenger arrival rate at the station node. Since the subway line under consideration is bidirectional, the train services in the up and down directions are independent. Therefore, the station node and The passenger arrival rate is 0, which is expressed as:
[0060] Introducing a new variable represents the arrival time of the latest train service before the arrival of train service k (including the arrival), which is used to calculate the waiting passengers on the platform. If train service k is serving on platform i, then .otherwise is equal to The value of is expressed as: The number of passengers waiting on the platform includes rejected passengers who were unable to board the previous bus. and time window Therefore, the number of passengers waiting on the platform It is expressed as: in , represents the number of passengers waiting for the train when it arrives at the station. Indicates the number of passengers who cannot get on the train when it arrives at the station.
[0061] Next, we analyze the limiting factors for the number of passengers getting on the train. First, consider the capacity of the train service. When the train arrives, the remaining space is related to the number of passengers getting off. Since it is difficult to obtain information about the destination of the passengers, the number of passengers getting off is approximated by proportion. When the train service arrives at the terminal, all passengers should get off (i.e. the proportion of passengers getting off is 1), and the number of passengers getting off is It is expressed as: in, Indicates the number of passengers who got off the train when it arrived at the station.
[0062] The stop time of the train service is the second limiting factor, and the third limiting factor is the number of passengers that can board the train, which is also limited by the actual demand (i.e. the number of passengers waiting on the platform). The minimum of these three limiting factors determines the number of passengers that can board train service k. , expressed as: in , Indicates the number of passengers boarding the train when it arrives at the station. Indicates the allowed passenger boarding speed, Indicates the allowed passenger alighting speed, Indicates the train capacity.
[0063] The change in the number of passengers boarding the bus is calculated as: in, Indicates the number of passengers on board the train when it arrives at the station.
[0064] When train k leaves platform i, the number of stranded passengers who cannot get on the train is calculated as: Figure 4 The figure shows the process of passengers boarding and queuing at the station. The dashed line represents the cumulative number of passengers arriving at the station within a period of time, and the dotted line represents the cumulative number of passengers served within a period of time. Therefore, the vertical difference between the two lines is the number of rejected passengers who cannot board the previous bus. , and the vertical difference between the two horizontal dotted lines is the number of passengers boarding The total waiting time for passengers during the train service is the size of the area enclosed by the actual and dotted lines.
[0065] Turnaround time constraints: ,in For the turnaround station The minimum return time.
[0066] S3: Introduce reinforcement learning algorithm based on the adjustment model and initialize the adjustment optimization model.
[0067] The state representation of the adjustment optimization model is specifically:
[0068] Figure 5 First, the train schedule is displayed at time step The initial time step is The master clock time to which events occurring on each service line are referenced. Figure 5 In the present invention, the interval between each vertical dotted line is set to 5 seconds, and each vertical dotted line represents the time when the train service scheduling occurs. The status obtained at this time step is the delay of the nearest train at each platform and the interval time between the train and the previous and next trains.
[0069] When the train departs according to the scheduled timetable, Figure 5 However, if a delay occurs, the situation will be similar to Figure 5 The interval between the front and rear vehicles is calculated based on the interval between trains running under the timetable being adjusted at the current time, such as Figure 5 As shown in Platforms 3 and 4. If there is no train before or after the train, the interval time is assigned a positive infinity value, as shown in Platform 4.
[0070] Define the state feature set, expressed as: , among which Represents each time step The local observation value on is the train status of the platform, using the operator Determine the nearest train, represents the train label, Indicates the actual train arrival time at the platform. Indicates the station index of the intersection. Indicates the set of platforms going up and down the route;
[0071] Local state vector The calculation process is expressed as: ,in Indicates the delay value, Indicates the distance between the nearest train on the platform and the preceding train. Indicates the distance between the nearest train on the platform and the following train;
[0072] For the delay value included in the local state vector , expressed as: The distance between the nearest train and the preceding train at the platform is expressed as: The distance between the nearest train and the following train at the platform is expressed as: .
[0073] The action representation of adjusting the optimization model is specifically:
[0074] Three action strategies are set according to the actual operation situation for the train to choose to use during the movement:
[0075] Strategy A represents the normal state, where the train does not take any action and maintains the current state;
[0076] Strategy B represents the movement of trains, where the train chooses a time level for a section when entering a block. ;symbol Indicates taking the largest Value function maximize;
[0077] Strategy C designs train detention, and the train determines the detention time at the station before departure ;
[0078] According to the given state feature set, the train will execute the action strategy at each time step. The calculation of the action strategy is expressed as: ;
[0079] Local Motion Vector The calculation process is expressed as: ;
[0080] The calculation of strategy C is expressed as: .
[0081] The reward function definition of the adjustment optimization model is specifically:
[0082] The reward function is set according to whether the interval between trains meets the requirements of safe operation and whether the normal and punctual train operation schedule can be quickly restored;
[0083] The train stop delay is divided into two situations. The first situation is that the train needs to stop at the station. In this case, the train does not depart at the scheduled time of the timetable or the delay is not resolved, such as Figure 5 The reward is based on the difference between the current inspection time and the scheduled departure time of the train and the difference between the actual train operation schedule and the planned train operation schedule. Calculation, expressed as: The second situation is when the delay problem has been resolved and the delayed train has departed or the unaffected train has departed normally, such as Figure 5 The reward is based on the difference between the actual departure time of the train and the scheduled departure time. Calculation, expressed as: The reward calculation related to the reward combination delay in the two cases is expressed as: And the driving interval constraint in the adjustment model is converted into the expression of reward, which is expressed as: The runtime constraint is converted into an expression of reward, which is determined by the interval between the two trains and is expressed as: in, is a constant used to convert the runtime constraint reward into a numerical value.
[0084] The station dwell time constraint is converted into a reward expression, which is expressed as: in, is a constant used to convert the runtime constraint reward into a numerical value.
[0085] The station dwell time constraint is converted into a reward expression, which is expressed as: The passenger flow constraint is converted into a reward expression, and the passenger satisfaction is maximized by effectively managing the passengers' boarding, waiting and train operation, which is expressed as: in, represents the penalty factor for waiting passengers, Ensure that only the number of passengers who can actually board the train is taken into account without exceeding the train capacity, The ratio of the number of passengers getting off the bus affects the available boarding space. A penalty that reflects the number of waiting passengers who were unable to board the bus.
[0086] Considering the total waiting time of all passengers, when the total waiting time of passengers increases, the reward value obtained by the agent will decrease, thereby encouraging the agent to find a train scheduling strategy that can reduce the waiting time of passengers, expressed as ,in represents the total passenger waiting time penalty factor; combining the passenger flow constraint and the passenger waiting time constraint to obtain the passenger comfort reward, expressed as: The constraints of the turnaround station are converted into the expression of reward, which is expressed as: The reward of each station is calculated and summed in the above manner to obtain the reward of the present invention, which is expressed as: in, represents the coefficient for converting the magnitude of the driving interval reward, A coefficient representing the magnitude of the conversion runtime bonus, The coefficient representing the magnitude of the station dwell time bonus, A coefficient representing the magnitude of the reward for the transition reentry constraint.
[0087] S4: Use the competitive deep Q network to improve the dual deep Q network and obtain the final adjusted optimization model;
[0088] In this step, in order to solve the problem of train timetable adjustment caused by short-term train delays, the present invention designs a competitive dual deep Q network (DDDQN) algorithm based on reinforcement learning that conforms to actual operating conditions to solve the train timetable adjustment problem with Markov chain properties. The dual deep Q network DDQN is introduced as the basic model of the adjustment optimization model, and two independent Q functions are set. The main network For selecting actions, target networks For evaluating actions, the Q-value calculation of the dual deep Q network DDQN is expressed as: in, represents the parameters of the main network, represents the local state vector, represents the local action vector, represents the learning rate, represents the action selected by the dual deep Q network, represents the train label, Indicates the station index of the intersection;
[0089] in, represents the parameters of the target network, represents the learning rate of the next state, The local state vector representing the next state, represents a learnable parameter.
[0090] In order to update the parameters, we need to update the loss function relative to the parameters The gradient of the main network parameters Use gradient descent to update the parameters of the target network Updated by smoothing moving average, expressed as: in, represents the learning rate that controls parameter updates, Represents the smoothing factor that controls the update speed of the target network parameters;
[0091] In order to solve the problem of inaccurate estimation of state values and action advantages in dual deep Q networks, a competitive deep Q network (DuelingDQN) is introduced to improve the algorithm based on DDQN.
[0092] like Figure 6As shown in the figure, the competitive deep Q network is used to improve the dual deep Q network to generate a competitive dual deep Q network framework. By introducing two branch networks, one for evaluating the value of the state and the other for evaluating the pros and cons of each action, the improved network framework learns the relationship between the state and the action more effectively, so that the adjustment optimization model can learn the value of the state and the relative advantages of each action separately. DDQN aims to alleviate the problem of overestimating Q value in the DQN algorithm. By separating the action selection and action evaluation of the target network, the overestimation of Q value is avoided. Because accurate state value estimation helps to understand the operating conditions at a specific station and time, and accurate action advantage estimation helps to make better decisions in multiple trains and complex scheduling constraints. Therefore, combining the advantages of dual deep Q network and competitive deep Q network, the algorithm can estimate Q value more accurately and improve learning efficiency. The design of competitive deep Q network DuelingDQN enables the network to better adapt to large-scale state and action space, improves learning efficiency, and overcomes the limitations of traditional DQN in dealing with complex problems with a large number of possible states and actions.
[0093] The action value corresponding to the competitive dual deep Q network The calculation is as follows: in Status The value of is directly output through a fully connected layer; For Action Relative to the state The advantage is output through a fully connected layer with the same size as the action space, and each action corresponds to an advantage value; is the size of the action space, represents the value trade-off parameter in the competitive dual deep Q network, Represents the dominance parameter in the balanced competitive dual deep Q-network.
[0094] S5: Using the adjustment optimization model to repeatedly train different delay situations of different lines to obtain different matrix lookup tables, importing them into the urban rail transit train mapping system, and obtaining an improved train operation schedule in real time based on the demand information and the delay situation.
[0095] Repeatedly train different delays of different lines to obtain different matrix lookup tables and import them into the urban rail transit train map compilation system, and obtain improved train operation schedules in real time according to the delays according to the needs of users. These lookup tables are embedded in the urban rail transit train map compilation system to meet the personalized needs of users. In actual operation, according to the delay situation, the system can optimize the train operation schedule in real time to provide a more flexible and efficient train operation plan. The trained model is integrated into the train operation diagram compilation system, which can capture the system status in real time and output corresponding adjustment actions according to the model strategy to achieve dynamic adjustment and optimization of the operation diagram. In the actual application process, the system will continue to collect data and feed this data back to the model for iterative learning to adapt to the dynamic changes in the train operation diagram compilation system. This not only improves the efficiency of scheduling, but also meets the high standards of urban rail transit operation managers for scheduling quality and efficiency.
[0096] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0097] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0098] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0099] Those skilled in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, and other media that can store program codes.
[0100] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.
Claims
1. A real-time adjustment method for urban rail transit train operation diagram based on reinforcement learning, characterized in that: The following steps are involved: S1: Based on the basic operating conditions of the line, an adjustment model is constructed with the objective function of minimizing the sum of the deviation between the actual train operation schedule and the planned train operation schedule and the total waiting time of passengers; S2: Determine the constraint conditions of the adjustment model; S3: Introduce reinforcement learning algorithm based on the adjustment model and initialize the adjustment optimization model; S4: Use the competitive deep Q network to improve the dual deep Q network and obtain the final adjusted optimization model; S5: using the adjustment optimization model to repeatedly train different delay situations of different lines to obtain different matrix lookup tables, importing them into the urban rail transit train mapping system, and obtaining an improved train operation schedule in real time based on the demand information and the delay situation; An adjustment model is constructed, the modeling objective of which is to minimize the sum of the deviation between the adjusted train operation schedule and the planned train operation schedule and the total waiting time of passengers; The objective function of the adjustment model is expressed as: in, It represents the sum of the deviation between the actual train operation schedule after modification and the planned train operation schedule and the total waiting time of passengers. represents the train label, Indicates the total number of trains; Indicates the station index of the intersection, Indicates the set of platforms going up and down the route; and Respectively represent the time when the actual train arrives at the platform and the time when the planned train arrives at the platform; and The actual train departure time and the planned train departure time are respectively The conversion coefficient representing the magnitude of the train operation time deviation value, The conversion factor representing the magnitude of the total waiting time for passengers, Indicates the total waiting time of passengers; The constraints of the adjustment model are constructed based on the train interval constraints, running time constraints, station stop time constraints, passenger flow constraints and return time constraints; A reinforcement learning algorithm is introduced on the basis of the adjustment model to initialize the adjustment optimization model. The state representation of the adjustment optimization model is specifically: Initial time step The master clock time corresponding to the events occurring on each service line is referenced. When a train departs according to the planned timetable, the interval between the preceding and following vehicles is calculated based on the interval between trains running under the current adjusted timetable under the delay condition; Define the state feature set, expressed as: , among which Represents each time step The local observation value on is the train status of the platform, using the operator Determine the nearest train, represents the train label, Indicates the actual train arrival time at the platform. Indicates the station index of the intersection, Indicates the set of platforms going up and down the route; Local state vector The calculation process is expressed as: ,in Indicates the delay value, Indicates the distance between the nearest train on the platform and the preceding train. Indicates the distance between the nearest train on the platform and the following train; A reinforcement learning algorithm is introduced on the basis of the adjustment model to initialize the adjustment optimization model. The action representation of the adjustment optimization model is specifically: Three action strategies are set according to the actual operation situation for the train to choose to use during the movement: Strategy A represents the normal state, where the train does not take any action and maintains the current state; Strategy B represents the movement of trains, where the train chooses a time level for a section when entering a block. ;symbol Indicates taking the largest Value function maximize; Strategy C designs train detention, and the train determines the detention time at the station before departure ; According to the given state feature set, the train will execute the action strategy at each time step. The calculation of the action strategy is expressed as: ; Local Motion Vector The calculation process is expressed as: ; A reinforcement learning algorithm is introduced based on the adjustment model to initialize the adjustment optimization model. The reward function definition of the adjustment optimization model is specifically as follows: The reward function is set according to whether the interval between trains meets the requirements of safe operation and whether the normal and punctual train operation schedule can be quickly restored; The train stop delay is divided into two situations. The first situation is that the train needs to stop at the station. In this case, the train does not depart at the scheduled time of the planned timetable or the delay is not resolved. Rewards are given based on the difference between the current inspection time and the scheduled departure time of the train and the difference between the actual train operation timetable and the planned train operation timetable. calculate; The second situation is that when the delay problem has been resolved and the delayed train has departed or the unaffected train has departed normally, the reward will be given according to the difference between the actual departure time of the train and the scheduled departure time. calculate; The reward calculation related to the reward combination delay in the two cases is expressed as: The train interval constraints, running time constraints, station stop time constraints, passenger flow constraints and return time constraints in the adjustment model are converted into reward expressions.
2. According to the method of real-time adjustment of urban rail transit train operation diagram based on reinforcement learning in claim 1, it is characterized in that: The competitive deep Q network is used to improve the dual deep Q network to obtain the final adjustment optimization model, which is as follows: The dual deep Q network DDQN is introduced as the basic model for adjusting the optimization model, setting two independent Q functions, the main network For selecting actions, target networks For evaluating actions, the Q-value calculation of the dual deep Q network DDQN is expressed as: in, represents the parameters of the main network, represents the local state vector, represents the local action vector, represents the learning rate, represents the action selected by the dual deep Q network, represents the train label, Indicates the station index of the intersection; The competitive deep Q network is used to improve the dual deep Q network to generate a competitive dual deep Q network framework. By introducing two branch networks, one for evaluating the value of the state and the other for evaluating the pros and cons of each action, the improved network framework learns the relationship between the state and the action more effectively, so that the adjustment optimization model can learn the value of the state and the relative advantages of each action separately.
3. The method for real-time adjustment of urban rail transit train operation diagram based on reinforcement learning according to claim 2 is characterized in that: The action value corresponding to the competitive dual deep Q network The calculation is as follows: in Status The value of is directly output through a fully connected layer; For Action Relative to the state The advantage is output through a fully connected layer with the same size as the action space, and each action corresponds to an advantage value; is the size of the action space, represents the value trade-off parameter in the competitive dual deep Q network, Represents the dominance parameter in the balanced competitive dual deep Q-network.
