A train flow allocation optimization method suitable for railway freight transport bottleneck nodes

Abstract

The application provides a train flow allocation optimization method suitable for railway freight transport bottleneck nodes, which solves an optimal empty train allocation scheme by using a DQN algorithm through empty train demand information of a loading station, access empty train information of a boundary portal and parameter information of a loading end line and a technical station: when one empty train is accessed from the boundary portal, it is arranged to meet a loading demand of a certain loading station; when many empty trains are accessed from the boundary portal within a solution period, they are accessed in sequence in time, and there is no two or more trains accessed from the boundary portal at the same time; there are also many loading demands of the loading station within the solution period; the DQN algorithm arranges all the accessed empty trains within the solution period, and the objective function and the constraint condition evaluate the advantages and disadvantages of the empty train allocation strategy, and finally the DQN algorithm finds an optimal empty train allocation strategy. The solution process of the DQN algorithm is more intuitive and practical, and can be better migrated to other examples.

Description

Technical Field

[0001] This invention provides a method for optimizing train flow allocation to adapt to bottleneck nodes in railway freight transport, belonging to the field of empty car allocation technology at the railway loading end. Background Technology

[0002] In railway freight transport, loaded cars are unloaded at the unloading end, and the resulting empty cars are transported to the loading end for loading. In my country, the allocation of empty cars at the loading end mainly relies on experience, allocating them evenly to loading stations based on their loading needs, prioritizing those furthest away. While this method effectively meets the loading needs of stations farther from the loading demarcation point, it can also lead to some stations not receiving their required empty cars at the appropriate time, thus affecting the implementation of loading plans. Rational allocation of empty car resources can improve the fulfillment rate of loading plans and enhance the transport efficiency of railway companies.

[0003] In terms of methods for solving the empty vehicle dispatch optimization problem, there are simplex method, tabular operation method, heuristic algorithm, etc. Most of the above literature draws on the modeling process of transportation problem (TP) and transforms the empty vehicle dispatch problem into an optimization problem of minimizing (maximizing) the objective function under constraints. Since the decision space is finite, it is intuitive to obtain the optimal solution of the problem by exhaustive search, but the number of feasible solutions usually grows exponentially with the scale of the road network, and it is impossible to find a feasible solution in polynomial time. Heuristic algorithms improve the efficiency of searching the solution space by using set heuristic rules, such as simulated annealing algorithm, particle swarm algorithm [5] and genetic algorithm, etc., but once the problem changes, such as the constraints and objective function change or the problem scale increases, the heuristic rules need to be modified. Reinforcement learning methods can automatically discover heuristic rules through the interaction between the agent and the environment, and the algorithm needs less modification.

[0004] Background information on empty car allocation at the railway loading end:

[0005] In railway transportation, train operation modes are relatively simple, and the rolling stock is generally transported in a circular manner: that is, unloading is completed at the unloading end, the train returns to the loading end for loading, and then the train is transported back to the unloading end from the loading end. A schematic diagram of this operation is shown below. Figure 1 .

[0006] There are two types of empty cars that can be accessed at the loading end: ① 5,000-ton unit empty car trains, and ② 10,000-ton combined empty car trains. A combined empty car train is composed of two unit empty car trains. Due to limitations in loading equipment and track capacity at the loading end, combined trains are generally broken down into two unit empty car trains at the loading end's technical station before being dispatched to the loading station. After the empty car trains complete loading operations, to reduce locomotive travel costs within the section, the unit empty car trains need to be combined at the preceding technical station as much as possible. Furthermore, the types of empty car models accessed are divided into three categories: C64, C70, and C80, with different models required for different types of goods. The problem of empty car allocation at the railway loading end can be described as a combinatorial optimization problem: under the constraints of line capacity, technical station decomposition capacity, technical station combination capacity, matching of empty car demand with train type, and matching of empty car demand with train model, minimizing the delay time for meeting empty car demand, maximizing the number of loaded car trains, and minimizing the waiting time for loaded car trains to be combined at the technical station.

[0007] Existing technologies, such as the non-dominated sorting genetic algorithm (NSGZ-II) solution method (Xu Heying, Lü Hongxia, Lü Miaomiao, et al. Research on optimization of empty car allocation at the loading end of heavy-haul railway [J]. Transportation Systems Engineering and Information, 2023, 23(01):141-151.), model the empty car allocation scenario at the loading end of the railway, set the objective function, constraints and the dominance relationship of the solution, and use the NSGZ-II algorithm to solve it.

[0008] Specifically as follows:

[0009] 1. Model the railway loading end empty car allocation problem using mathematical notation, including defining constraints and objective functions. The railway loading end empty car allocation problem is modeled as a combinatorial optimization problem.

[0010] 2. Define the encoding form of the NSGZ-II algorithm solution. The NSGZ-II algorithm solution represents a railway loading end empty car allocation scheme, and the optimal solution of the NSGZ-II algorithm represents the solution that maximizes the objective function. If there are multiple objective functions, it is necessary to define the importance of each objective function and define the dominance relationship of the solutions based on the importance.

[0011] 3. Define the crossover and mutation operations of the NSGZ-II algorithm. The crossover operation aims to swap data at different or the same positions in two solutions, while the mutation operation aims to change data at random positions in a solution.

[0012] Finally, the NSGZ-II algorithm process is as follows:

[0013] Step 1: Set algorithm parameters: Population size P size Number of iterations n max The crossover probability control parameter α and the mutation probability parameter β are initialized, and the number of iterations n is 0.

[0014] Step 2: Randomly generate P size An initial feasible solution.

[0015] Step 3 involves performing non-dominated sorting, selection, crossover, and mutation operations on the initial population to generate the first generation subpopulation, n = n + 1.

[0016] Step 4: Merge the parent and offspring populations, perform a fast non-dominated sort, and calculate the crowding of individuals.

[0017] Step 5 generates a new parent population through a binary tournament.

[0018] Step 6 involves selection, crossover, and mutation operations on the newly generated parent population to generate a new offspring population.

[0019] Step 7 n = n + 1

[0020] Step 8: Determine if n > n max Check if the condition is met. If it is, output the result. If not, go to Step 4.

[0021] The drawbacks of this technology are:

[0022] 1. Existing technologies that use genetic algorithms to solve problems are prone to generating solutions that do not meet the constraints during crossover and mutation operations, resulting in low efficiency in optimizing solutions through crossover and mutation operations.

[0023] 2. When it is necessary to modify or add constraints and objective functions, the algorithm requires many changes, and in actual empty vehicle dispatching scenarios, some constraints are cumbersome to represent with mathematical symbols. Summary of the Invention

[0024] This invention proposes a railway loading terminal empty car allocation method based on DQN, which completes the empty car allocation function based on known empty car demand information and access to empty car information of each loading station at a certain time period.

[0025] The technical problem to be solved by this invention:

[0026] 1. Purpose of the invention: Taking the empty car allocation scenario at the railway loading end as the research object, the empty car allocation process is modeled as a Markov decision process, and the DQN algorithm in reinforcement learning is used to solve it. Compared with the traditional heuristic method, it is found that the solution process of the DQN algorithm is more intuitive and realistic, and can be better transferred to other examples.

[0027] 2. By setting a reward function, the problem of cumbersome expression constraints and objective functions in existing technologies is solved. The DQN algorithm can set a reward function to penalize empty car dispatching strategies that violate constraints and reward empty car dispatching strategies that help maximize the objective function. The setting of the reward function is consistent with human intuition and experience and is relatively simple.

[0028] 3. It solves the problem that traditional heuristic methods require significant algorithm modifications when changing the objective function or constraints. The DQN algorithm only requires simple modifications to the reward function, making the modification process simpler and more convenient.

[0029] The specific technical solution of this invention:

[0030] A method for optimizing train flow allocation to adapt to bottleneck nodes in railway freight transport includes the following processes:

[0031] By analyzing empty train demand information at loading stations, empty train access information at boundary points, and parameter information from loading lines and technical stations, the DQN algorithm is used to find the optimal empty train allocation scheme: when an empty train accesses the boundary point, it is assigned to fulfill a specific loading demand at a particular loading station. During the solution period, many empty trains access the boundary point sequentially; no two or more trains access the boundary point simultaneously. There are also many loading demands at various loading stations during the solution period. The DQN algorithm schedules all accessing empty trains within the solution period, evaluates the merits of the empty train allocation strategies using the objective function and constraints, and finally identifies the optimal empty train allocation strategy.

[0032] in:

[0033] The station's empty car demand information includes: the time window for the station to request empty cars, the type of empty cars, and the model of the empty cars;

[0034] The information on empty vehicles entering the boundary in the future will include: the time when the empty vehicle enters the boundary, the type of the empty vehicle, and the vehicle model of the empty vehicle;

[0035] The parameter information for the loading end line and technical stations includes: the train throughput capacity of each section within the loading end network during the solution period; the train combination and decomposition capacity of each technical station at the loading end; the loading time at the loading station; the decomposition operation time at the technical station; and the train running time within the section.

[0036] The objective function and constraints include: the number of trains passing through a section cannot exceed the train capacity of the section, and the number of train combinations and decompositions at a technical station cannot exceed the combination and decomposition capacity of that technical station.

[0037] The first objective function is to minimize the empty train delay time G1:

[0038]

[0039]

[0040]

[0041] In the formula This indicates that the i-th empty car arriving at the boundary has reached z. j The moment when the loading station meets the m-th empty car demand; Indicates the type of train that connects to the i-th empty car at the dividing point; The type is The empty train from the boundary station P to Z j runtime; The type is The empty train from the boundary station P to Z k runtime; The type is The empty train from z k To z j runtime; t represents the decomposition technical operation time of technical station k (the time to decompose a combined empty train into a unit empty train); j,m Indicates loading station z j The delay time for the m-th loading request; Indicate z j The number of loading requests at the station; n Z This indicates the number of loading stations.

[0042] The second objective function is to maximize the number of train combinations G2.

[0043]

[0044] In the formula n Q Indicates the number of technical stations; Representing a 0-1 variable: z j Does the loaded train of the m-th empty car demand at station q meet the requirements? k For station combinations, the value of the combination is 1 if the combination is valid, otherwise it is 0.

[0045] The third objective function is to minimize the train waiting combination time G3.

[0046]

[0047]

[0048] In the formula, Indicates loading station z j The loaded vehicle for the m-th loading requirement arrives at the technical station q. k The moment; The type is The heavy train from z j The running time to technical station k; z represents a 0-1 variable. j Does the loaded train of the m-th empty car demand at the station match the demand for empty cars? The station's m1st empty car demand was loaded onto a loaded car train in q. k The stations are combined; if a combination is formed, the value is 1; otherwise, the value is 0.

[0049] The DQN algorithm finds an optimal empty vehicle dispatching strategy. Specifically, the DQN algorithm is designed to solve for an empty vehicle dispatching strategy that satisfies the constraints and optimizes the above three objective functions.

[0050] The problem of empty car allocation at the railway loading end is modeled as a sequential decision problem and solved using a reinforcement learning algorithm.

[0051] Reinforcement learning refers to a method in which an agent interacts with its environment through a series of decisions and continuously tries and fails, ultimately finding the optimal policy. When designing reinforcement learning algorithms:

[0052] First, a reinforcement learning environment and agent for empty vehicle dispatching need to be built. In addition to updating the environment after the agent makes a decision, the current state of the environment also needs to be provided to the agent so that the agent can make new decisions based on the state of the environment.

[0053] Secondly, a reward function needs to be constructed to guide the agent to iterate the strategy quickly. The reward function will evaluate the quality of the agent's decisions. The agent can increase the reward value by continuously improving the empty vehicle dispatch strategy.

[0054] Ultimately, the agent finds the optimal strategy during the algorithm iteration process.

[0055] Specifically, the environmental state definition, agent action definition, and reward function settings for reinforcement learning-based empty vehicle dispatching are as follows:

[0056] (1) Definition of the state of the reinforcement learning environment for empty vehicle allocation

[0057] State s t Defined at decision time t:

[0058] The vector formed, i.e.

[0059]

[0060] In the formula l j,t Indicates loading station z j The number of remaining unmet empty vehicle demands at decision time t; Indicates technical station q k The number of trains to be combined at decision time t; Indicates technical station q k The number of combined trains at decision time t; Technical station q k The number of decomposed trains at decision time t; η j,j+1,t Indicates the interval (z) at decision time t. j ,z j+1 The number of trains that have passed through.

[0061] Set s1 as the initial state, s f If it is a terminated state, then s1 and s f It can be represented as follows:

[0062]

[0063]

[0064] (2) Definition of agent action

[0065] For a single train, determine its final destination and connecting stations; for a combined train, first decide whether to decompose it. If decomposed, determine its decomposed stations, destinations, and connecting stations; if not decomposed, determine its destination. (Using a) 不在技术站组合 The decision not to combine the units after loading is represented by the agent's action set as follows:

[0066] a 单元列车 ={(z j ,h w )|z j ∈Z,h w ∈{H j a 不在技术站组合}} (10)

[0067]

[0068] a 组合列车,直达 ={z j |z j ∈Z} (12)

[0069] a t ∈{a 单元列车 ,a 组合列车,分解 ,a 组合列车,直达} (13)

[0070] In the formula, Z represents the set of loading stations. There are n Z There are several loading stations, with index j, i.e., Z = {z1, ..., z2}. j ,…z nZ};H j Indicates the unit train in z jAfter loading is completed at the station, all technical stations that could potentially be combined with it are assembled. F j This indicates that if the combined train is disassembled and reassigned to z j A site is a collection of all technical sites that can be broken down into its components; q k The term "technical station" refers to a technical station, and the set of technical stations is Q. There are n Q There is one technical station, indexed as k, i.e.

[0071] (3) Reward function setting

[0072] The importance of the objective functions is ranked as follows: G1 > G2 > G3. The reward function assigns appropriate reward values ​​to different objectives to evaluate the merits of the agent's policies.

[0073] The reward function is divided into two parts: immediate reward and final reward. The immediate reward is used to evaluate the agent's strategy at each step, while the final reward is used to check whether the loading requirements of each station are met and whether there are any remaining trains to be combined at the technical station after the agent has allocated all empty cars, and to impose corresponding penalties.

[0074]

[0075] r t 即时奖励 =r t 时间窗奖励 +r t 组合奖励 +r t 违反技术站分解能力约束的惩罚

[0076] +r t 违反技术站组合能力约束的惩罚 +r t 违反区间通过能力的惩罚 (15)

[0077]

[0078]

[0079]

[0080]

[0081]

[0082] This paper presents a DQN-based algorithm for empty car dispatching at the railway loading end, based on the state of the reinforcement learning environment, agent actions, and reward function defined above.

[0083] In the DQN algorithm, the neural network acts as an agent, trained using data from an experience replay pool. The input to the neural network is the current state s of the environment. t The output is the next empty vehicle dispatch strategy, i.e., state s. t The probability of each action is determined, and an ε-greedy strategy is used to select a specific action for output.

[0084] Parameter settings for the DQN-based empty car dispatching algorithm at the railway loading end:

[0085] 1. Neural network structure: A multi-layer fully connected neural network is selected;

[0086] 2. Experience replay pool capacity;

[0087] 3. The number of data points q sampled from the experience replay pool in a single run;

[0088] 4. Target network Q w-(s,a) Fixed update step count ti;

[0089] 5. The ε value of the ε-greedy strategy;

[0090] 6. Neural network learning rate l;

[0091] 7. Reward discount rate γ;

[0092] 8. Training rounds (ei). Attached Figure Description

[0093] Figure 1 A schematic diagram of the operation of unit trains and combined trains on existing railways;

[0094] Figure 2 This is a schematic diagram of the road network for the present invention;

[0095] Figure 3 This is the reinforcement learning process for empty vehicle dispatching at the loading end of the present invention;

[0096] Figure 4 This is a flowchart of the railway loading end empty car dispatching algorithm based on DQN of the present invention;

[0097] Figure 5 This is a schematic diagram of the DQN neural network of the present invention;

[0098] Figure 6 The reward curve of the DQN-based empty car dispatching algorithm at the railway loading end during agent training is shown in the example. Detailed Implementation

[0099] The specific technical solutions of the present invention will be described with reference to the embodiments.

[0100] I. Equipment required for the technical solution of this invention: A personal computer. The computer includes a Python 3.8 programming environment with the torch-2.0.0-cu118 extension package, an NVIDIA-RTX3050 GPU, an AMD-Ryzen5-5600H CPU, and 16GB of RAM.

[0101] II. Information required for the technical solution of this invention: empty car demand information of each loading station during the time period to be solved, information on empty cars to be connected at the future dividing point, and parameter information of the loading end line and technical station.

[0102] The empty car demand information includes: the time window in which the loading station needs empty cars, the type of empty cars, and the model of the empty cars. Specifically, it means that a certain loading station needs one empty train of units or combinations (empty car type) of C70, C64, or C80 (empty car model) between certain times (time window).

[0103] The information on empty trains entering the boundary at future demarcation points will include: the time when the empty train enters the boundary, the type of the empty train, and the model of the empty train. Specifically, it means: at what time (hour and minute) the empty trains of a certain unit or combination (empty train type) of C70, C64, or C80 (empty train model) arrive at the boundary station.

[0104] The parameter information for the loading-end lines and technical stations includes: the train throughput capacity of each section (the lines between adjacent loading stations are called sections) within the loading-end network during the solution period (the maximum number of trains that can pass through a section during the solution period; the empty car dispatching scheme cannot arrange for the number of trains passing through a section to exceed the throughput capacity of that section); and the train combination and decomposition capabilities of each technical station at the loading end (the maximum number of combined trains that a technical station can decompose into unit trains (decomposition capability) during the solution period, and the maximum number of combined trains that a technical station can merge into pairs of combined trains). Combined capacity); loading time at loading stations (the time taken for empty trains to be loaded at loading stations; generally, the loading time for a single empty train is shorter, while the loading time for a combined empty train is longer, so the loading time at each loading station is divided into the loading time for a single train and the loading time for a combined train); technical station breakdown operation time (the time spent by a combined train being split into two single trains at a certain technical station); train running time in the section (generally, the running time of combined trains and single trains in the section is different, and the running time of empty cars and loaded cars is also different).

[0105] III. Technical solution of the railway loading end empty car dispatching method based on DQN of the present invention:

[0106] Based on the aforementioned information on empty train demand at loading stations, empty train access at boundary points, and parameters of loading lines and technical stations, the DQN algorithm is used to find a better empty train allocation scheme. Specifically, when an empty train accesses a boundary point, it is allocated to fulfill a specific loading demand at a particular loading station. During the solution period, many empty trains access the boundary point sequentially; no two or more trains access the boundary point simultaneously. There are also many loading demands at various loading stations during the solution period. The DQN algorithm can allocate all accessing empty trains within the solution period because an objective function can evaluate the merits of the empty train allocation strategy. Finally, the DQN algorithm can find a better empty train allocation strategy.

[0107] 1. To more intuitively understand the empty car demand information at loading stations, the empty car access information at boundary points, and the parameter information of loading-end routes and technical stations, the following provides a road network and its loading demand information and empty car access information for a certain period of time.

[0108] Figure 2 In the road network, z1 is the dividing station, and z2 to z 10 As loading stations, and with Z3, Z4, Z5, and Z7 also serving as technical stations, trains can be decomposed (combined empty trains are split into single empty trains) or combined (single loaded trains are merged into combined loaded trains) at technical stations. From the railway network, the running time of single empty trains / combined empty trains / single loaded trains / combined loaded trains in each section can be obtained, and the running time of single empty trains / combined empty trains / single loaded trains / combined loaded trains between any two loading stations can also be calculated.

[0109] Figure 2 The table below shows the information on empty vehicles entering the boundary at the corresponding time period of 7 minutes to 449 minutes on the road network.

[0110] Table 1. Information on Empty Vehicles Accessing the Boundary Point

[0111]

[0112] t in the figure i v represents the access time of the i-th empty car at the boundary. i Indicates the train model that connects to the empty i-th train. This indicates the type of train that connects to the empty train in the i-th column, where 2 indicates a combined train and 1 indicates a unit train.

[0113] Figure 2 The table below shows the empty car demand information for each loading station within the corresponding time period of 7min-449min on the road network.

[0114] Table 2 Information on Empty Vehicle Demand

[0115]

[0116]

[0117] V in the diagram j,m The diagram shows the train type required for the m-th loading demand at loading station j. This indicates the type of train required for the m-th loading demand at loading station j. This represents the time window requirement for the m-th loading demand at loading station j. The vehicle needs to indicate the lower limit of the time window. This indicates the upper limit of the time window.

[0118] Figure 2 The parameter information of the loading end lines and technical stations within the corresponding time period of 7min-449min is shown in the table below.

[0119] Table 3. Intersection Passage Capacity

[0120]

[0121] Table 4 Station Operation Parameters

[0122]

[0123] The train's travel time within a certain section can be determined by... Figure 1 It is concluded that the parameter information of other loading-end lines and technical stations is mainly divided into section throughput capacity and station operation parameters. η j,j+1 Indicates the throughput of a certain interval. This indicates the decomposition capability of a certain technical station. This indicates the combined capabilities of a particular technology station. This represents the time spent at a certain technical station to break down a combined train into unit trains. This indicates the loading time of a train unit at a specific loading station. This indicates the loading time of a combined train at a specific loading station.

[0124] 2. Given the empty car demand information of each loading station during the time period to be solved, the information of empty cars to be connected at the future dividing point, and the parameter information of the loading end line and technical station, the following describes how to use the DQN algorithm to solve the empty car dispatching strategy.

[0125] To solve the empty car dispatching strategy using the DQN algorithm, it is first necessary to define the objective function and constraints. The objective function is to evaluate which empty car dispatching strategies are good and which are bad (the objective function can evaluate the empty car dispatching strategies). The constraints are the rules that the empty car dispatching strategy cannot violate. This is already reflected in the parameter information of the loading end line and technical station. Specifically, these include: the number of trains passing through a section cannot exceed the train capacity of the section, and the number of train combinations and decompositions at a technical station cannot exceed the combination and decomposition capacity of that technical station.

[0126] To further understand the objective function, we will now represent it using mathematical notation. Since a train arriving at the loading station too early will occupy the track and incur empty car storage costs, and a train arriving too late will not guarantee that the loading operation will be completed on time, we need to meet the time window requirements of the loading demand as much as possible, while ensuring that the loading demand model and train type match. That is, we need to minimize the empty car delay.

[0127] The first objective function is expressed as:

[0128]

[0129]

[0130]

[0131] In the formula This indicates that the i-th empty car arriving at the boundary has reached z. j The moment when the loading station meets the m-th empty car demand; Indicates the type of train that connects to the i-th empty car at the dividing point; The type is The empty train from the boundary station P to Z j runtime; The type is The empty train from the boundary station P to Z k runtime; The type is The empty train from z k To z j runtime; t represents the decomposition technical operation time of technical station k (the time to decompose a combined empty train into a unit empty train); j,m Indicates loading station z j The delay time for the m-th loading request; Indicate z j The number of loading requests at the station; n Z This indicates the number of loading stations.

[0132] To reduce the locomotive's travel cost within the section, empty trains should be combined as many times as possible at the technical station after loading. Therefore, the second objective function is to maximize the number of train combinations.

[0133]

[0134] In the formula n Q Indicates the number of technical stations; Representing a 0-1 variable: z j Does the loaded train of the m-th empty car demand at station q meet the requirements? k For station combinations, the value of the combination is 1 if the combination is valid, otherwise it is 0.

[0135] When trains of the same type are combined at a train combination station, their waiting time for combination should be minimized. Therefore, the third objective function is to minimize the train waiting time for combination.

[0136]

[0137]

[0138] In the formula, Indicates loading station z j The loaded vehicle for the m-th loading requirement arrives at the technical station q. k The moment; The type is The heavy train from z j The running time to technical station k; z represents a 0-1 variable. j Does the loaded train of the m-th empty car demand at the station match the demand for empty cars? The station's m1st empty car demand was loaded onto a loaded car train in q. k The stations are combined; if a combination is formed, the value is 1; otherwise, the value is 0.

[0139] 3. After constructing the objective function for empty car allocation at the railway loading end, the DQN algorithm can be designed to solve for the empty car allocation strategy that satisfies the constraints and optimizes the above three objective functions.

[0140] Specifically, the design concept and process of the railway loading end empty car dispatching method based on DQN are as follows:

[0141] First, it's important to clarify that the DQN algorithm is a type of reinforcement learning method. Below, we'll use the scenario of empty wagon allocation at the railway loading end to illustrate the approach to empty wagon allocation using reinforcement learning:

[0142] like Figure 3As shown, from the dispatcher's perspective, when the estimated arrival time, train type, and class of an empty train at the boundary station are obtained, decisions need to be made regarding the loading station, decomposition location, and combination location of the train. When all empty train allocation decisions are made, a solution to the aforementioned combinatorial optimization problem under multiple objectives and constraints is obtained. Therefore, the railway loading-end empty train allocation problem can be modeled as a sequential decision problem and solved using reinforcement learning algorithms.

[0143] Reinforcement learning refers to a method where an agent interacts with its environment through a series of decisions and trial and error to find the optimal policy. When designing a reinforcement learning algorithm: First, an empty vehicle dispatching reinforcement learning environment and an agent must be constructed. The environment needs to be updated after each decision by the agent, and the current state representation of the environment must be provided to the agent. The agent then makes new decisions based on the state of the environment. Second, a reward function must be constructed to guide the agent to rapidly iterate its policy. The reward function evaluates the quality of the agent's decisions, and the agent increases its reward value by continuously improving its empty vehicle dispatching policy. Finally, the agent finds the optimal policy during the algorithm's iterative process.

[0144] Specifically, the environmental state definition, agent action definition, and reward function settings for reinforcement learning-based empty vehicle dispatching are as follows:

[0145] (1) Definition of the state of the reinforcement learning environment for empty vehicle allocation

[0146] The empty car dispatching process exhibits the Markov property, meaning that future empty car dispatching decisions depend only on the current state of the empty car dispatching environment and are independent of historical information about past empty car dispatching environments. Therefore, the state of the empty car dispatching reinforcement learning environment needs to accurately characterize the impact of different policy behaviors on the empty car dispatching environment to ensure the Markov property of the empty car dispatching process, thereby enabling the agent to iterate its policies more effectively. After completing an empty car dispatching decision, the empty car demand information at the corresponding loading station, the train decomposition and combination information at the technical station, and the section passage information will change. Therefore, the state s... t Defined as at decision time t

[0147] The vector formed, that is:

[0148]

[0149] In the formula l j,t Indicates loading station z j The number of remaining unmet empty vehicle demands at decision time t; Indicates technical station q k The number of trains to be combined at decision time t; Indicates technical station q k The number of combined trains at decision time t; Technical station q k The number of decomposed trains at decision time t; η j,j+1,t Indicates the interval (z) at decision time t. j ,z j+1 The number of trains that have passed through.

[0150] Set s1 as the initial state, s f If it is a terminated state, then s1 and s f It can be represented as follows:

[0151]

[0152]

[0153] (2) Definition of agent action

[0154] When an intelligent agent receives an empty train at a boundary, it makes decisions based on the train type and model, specifically as follows: For a single train, it determines its final destination and connecting stations; for a combined train, it first decides whether to decompose it. If decomposed, it determines its decomposition station, destination, and connecting station; if not decomposed, it determines its destination. (Using a...) 不在技术站组合 The decision not to combine the units after loading can be represented by the agent's action set as follows:

[0155] a 单元列车 ={(z j ,h w )|z j ∈Z,h w ∈{H j a 不在技术站组合}} (10)

[0156]

[0157] a 组合列车,直达 ={z j |z j ∈Z} (12)

[0158] a t ∈{a 单元列车 ,a 组合列车,分解 ,a 组合列车,直达} (13)

[0159] In the formula, Z represents the set of loading stations. There are n Z There is a loading station with index j, i.e. H j Indicates the unit train in z j After loading is completed at the station, all technical stations that could potentially be combined with it are assembled. F jThis indicates that if the combined train is disassembled and reassigned to z j A site is a collection of all technical sites that can be broken down into its components; q k The term "technical station" refers to a technical station, and the set of technical stations is Q. There are n Q There is one technical station, indexed as k, i.e.

[0160] (3) Reward function setting

[0161] The primary objective of empty car dispatching is to minimize the delay time of empty car demand at loading stations, followed by maximizing the number of train combinations that can be loaded, and finally, minimizing the waiting time of combined trains. The importance of the objective functions is ranked as follows: G1 > G2 > G3. Therefore, the reward function needs to be assigned appropriate reward values ​​for different objectives to evaluate the merits of the agent's strategy.

[0162] The reward function is divided into two parts: immediate reward and final reward. The immediate reward is used to evaluate the agent's strategy at each step, while the final reward is used to check whether the loading requirements of each station are met and whether there are any remaining trains to be combined at the technical station after the agent has allocated all empty cars, and to impose corresponding penalties.

[0163]

[0164] r t 即时奖励 =r t 时间窗奖励 +r t 组合奖励 +r t 违反技术站分解能力约束的惩罚

[0165] +r t 违反技术站组合能力约束的惩罚 +r t 违反区间通过能力的惩罚 (15)

[0166]

[0167]

[0168]

[0169]

[0170]

[0171] Then, we introduce a DQN-based empty car dispatching algorithm for railway loading end, which is based on the state, agent actions, and reward function of the empty car dispatching reinforcement learning environment as defined above.

[0172] The Deep Q Network (DQN) algorithm uses a neural network instead of the tables used in traditional Q-learning algorithms to approximate the state-action value function, solving the problem of excessively large state spaces that tables cannot store. Its algorithm flow is as follows: Figure 4 As shown. The neural network in the DQN algorithm acts as an agent, trained using data from the experience replay pool. Figure 5 As shown, the input to the neural network is the current state s of the environment. t The output is the next empty vehicle dispatch strategy, i.e., state s. t The probability of each action is determined, and an ε-greedy strategy is used to select a specific action for output.

[0173] Parameter settings for the DQN-based empty car dispatching algorithm at the railway loading end:

[0174] 1. Neural network structure: Multi-layer fully connected neural networks or other types can be selected.

[0175] 2. Experience replay pool capacity

[0176] 3. The number of data points q sampled from the experience replay pool in a single run.

[0177] 4. Target network Q w-(s,a) Fixed update steps ti

[0178] 5. The ε value of the ε-greedy strategy

[0179] 6. Neural network learning rate l

[0180] 7. Reward Discount Rate γ

[0181] 8. Training rounds (ei)

[0182] The beneficial effects of the technical solution of this invention are as follows:

[0183] (1) The railway empty car dispatching scenario is described as a Markov decision process and solved using the DQN algorithm. The DQN algorithm can find a better solution that satisfies the constraints in dispatching scenarios with large action space and state space. Furthermore, compared with solution 4 in the solution results of related literature (Xu Heying, Lü Hongxia, Lü Miaomiao, et al. Research on optimization of empty car dispatching at loading end of heavy-haul railway [J]. Transportation Systems Engineering and Information, 2023, 23(01):141-151.), it is found that the DQN algorithm reduces the train waiting time by 18 minutes.

[0184] Empty car dispatching results based on the DQN-based empty car dispatching algorithm at the railway loading end:

[0185] Table 5 [Related Literature] In the example, the DQN algorithm found 28 combined trains with a train combination waiting time of 88 minutes.

[0186]

[0187] Table 6 shows the results of the formulation in reference [7] 4.

[0188]

[0189]

[0190] According to relevant examples, the agent also found a large number of solutions that meet the time window requirements of empty train demand during the training process. Among them, when the number of combined trains is 28, the optimal train combination waiting time is 88 minutes, which saves 18 minutes compared to solution 4 in another literature (Wang Long, Ma Jianjun, Lin Boliang, et al. Dynamic dispatching of empty trains and calculation method of empty flow at the boundary of the railway network [J]. Journal of Railway Engineering, 2015, 37(06):1-9).

[0191] Figure 6 The reward curve for the DQN-based railway loading end empty car dispatching algorithm during agent training.

[0192] (2) A reinforcement learning environment for the railway empty car dispatch scenario was constructed. It has good reusability among similar cases. When the constraints and objective function need to be modified, it is convenient to modify the algorithm.

[0193] (3) The process of solving the optimal strategy for empty car dispatching using the DQN algorithm is more in line with the work habits of dispatchers and is expected to be extended to "online and real-time" empty car dispatching scenarios.

Claims

1. A method for optimizing train flow allocation to adapt to bottleneck nodes in railway freight transport, characterized in that, Includes the following processes: By utilizing empty car demand information at loading stations, future empty car access information at boundary points, and parameter information from loading lines and technical stations, the DQN algorithm is used to find the optimal empty car allocation scheme: when an empty car accesses a boundary point, it is assigned to satisfy a specific loading demand at a loading station; during the solution period, many empty car trains access the boundary point sequentially, with no two or more trains accessing the boundary point at the same time; and during the solution period, there are also loading demands from many loading stations; the DQN algorithm allocates all accessing empty car trains during the solution period, and the objective function and constraints evaluate the merits of the empty car allocation strategy. Finally, the DQN algorithm finds the optimal empty car allocation strategy. The environmental state definition, agent action definition, and reward function settings for reinforcement learning-based empty vehicle dispatching are as follows: (1) Definition of the state of the empty vehicle dispatch reinforcement learning environment State Defined as in Decision-making moment: The vector formed, i.e. ， In the formula Indicates loading station exist The number of remaining unmet empty vehicle demands at the time of decision; Indicates technical station exist The number of trains to be combined at the decision-making moment; Indicates technical station exist The number of combined trains at the decision-making moment; Technical Station exist The number of decomposed trains at the decision moment; Indicates in Decision time interval Number of trains that have passed; set up This is the initial state. If it is a terminated state, then and It can be represented as follows: ， ， (2) Definition of agent action For a single train, determine its final destination and connecting stations; for a combined train, first decide whether to decompose it. If decomposed, determine its decomposed stations, destinations, and connecting stations; if not decomposed, determine its destination. The decision not to combine the units after loading is represented by the agent's action set as follows: ， ， ， ， In the formula Indicates the assembly point at the loading station. There are a total of Each loading station, indexed as follows: ,Right now ; Indicates that the unit train is in After loading is completed at the station, all technical stations that can be combined with it are assembled. ; This indicates that if the combined train is disassembled and reassigned... A site is a collection of all technical sites that can be broken down into its components. This refers to a technical station; a collection of technical stations is... , There are a total of One technical site, indexed as ,Right now ; (3) Setting the reward function The importance ranking of the objective functions is as follows: The reward function assigns appropriate reward values ​​to different objectives to evaluate the merits of the agent's strategy; The reward function is divided into two parts: immediate reward and end reward. The immediate reward is used to evaluate the agent's strategy at each step. The end reward is used to check whether the loading requirements of each station are met and whether the technical station has any remaining trains to be combined after the agent has allocated all empty cars, and to impose corresponding penalties. ， ， ， ， ， ， 。 2. The method for optimizing train flow allocation to adapt to bottleneck nodes in railway freight transport according to claim 1, characterized in that: The loading station's empty car demand information includes: the time window for the loading station to request empty cars, the type of empty cars, and the model of the empty cars; The information on empty vehicles entering the future boundary includes: the time when the empty vehicle enters the boundary, the type of the empty vehicle, and the vehicle model of the empty vehicle; The parameter information of the loading end line and technical station includes: the train throughput capacity of each section in the loading end road network during the solution period; the train combination capacity and decomposition capacity of each technical station at the loading end; the loading time of the loading station; the decomposition operation time of the technical station; and the train running time in the section.

3. The method for optimizing train flow allocation to adapt to bottleneck nodes in railway freight transport according to claim 1, characterized in that: The objective function and constraints specifically include: the number of trains passing through a section cannot exceed the train capacity of the section, and the number of train combinations and decompositions at a technical station cannot exceed the combination and decomposition capacity of that technical station. The first objective function is to minimize the empty train delay time G1: ， ， ， In the formula Indicates the first access point of the boundary port Arrival of empty car Loading station meets the requirements of the first The moment when there is a demand for empty cars; Indicates the boundary port access number The type of train with empty cars; The type is The empty train from the boundary station P to runtime; The type is The empty train from the boundary station P to runtime; The type is empty trains from to runtime; This represents the decomposition technical operation time of technical station k, that is, the time to decompose the combined empty train into the unit empty train; Indicates loading station The The delay time for loading the vehicle; express The number of loading requests at the station; Indicates the number of loading stations; The second objective function is to maximize the number of train combinations G2. ， In the formula Indicates the number of technical stations; Representing 0-1 variables: The station's first Whether the empty car demand for loaded car trains is met For station combinations, the value of the combination is 1 if the combination is valid, otherwise it is 0. The third objective function is to minimize the train waiting combination time G3; ， ， In the formula, Indicates loading station No. The loaded trucks required for the project arrived at the technical station. The moment; The type is Heavy train from The running time to technical station k; Represents 0-1 variables, The station's first Does the demand for empty cars be matched by the demand for loaded car trains? The station's first The demand for empty cars was met by loading heavy cars into the train. The stations are combined; if a combination is formed, the value is 1; otherwise, the value is 0.

4. The method for optimizing train flow allocation to adapt to bottleneck nodes in railway freight transport according to claim 3, characterized in that: The DQN algorithm finds an optimal empty vehicle dispatching strategy. Specifically, the DQN algorithm is designed to solve for an empty vehicle dispatching strategy that satisfies the constraints and optimizes the above three objective functions. The problem of empty car allocation at the railway loading end is modeled as a sequential decision problem and solved using a reinforcement learning algorithm. Reinforcement learning refers to a method in which an agent interacts with its environment through a series of decisions and continuously tries and fails, ultimately finding the optimal policy. When designing reinforcement learning algorithms: First, a reinforcement learning environment and agent for empty vehicle dispatching need to be built. In addition to updating the environment after the agent makes a decision, the current state of the environment also needs to be provided to the agent so that the agent can make new decisions based on the state of the environment. Secondly, a reward function needs to be constructed to guide the agent to quickly iterate its strategy. The reward function will evaluate the quality of the agent's decisions, and the agent will increase the reward value by continuously improving the empty vehicle allocation strategy.

5. The method for optimizing train flow allocation to adapt to bottleneck nodes in railway freight transport according to claim 1, characterized in that: In the DQN algorithm, the neural network acts as an agent, trained using data from an experience replay pool; the input to the neural network is the current state of the environment. The output is the next empty vehicle allocation strategy, i.e., the state. The probability of taking each action is given below, and the following is used: The strategy selects a specific action to output. Parameter settings for the DQN-based empty car dispatching algorithm at the railway loading end: (1) Neural network structure: A multi-layer fully connected neural network is selected; (2) Experience playback pool capacity; (3) Number of data samples taken from the experience playback pool in a single session ; (4) Target network Fixed update steps ; (5) Strategy value; (6) Neural network learning rate ; (7) Reward Discount Rate ; (8) Training rounds ei.

Images