High-speed railway train working diagram conflict intelligent resolution method based on reinforcement learning

Abstract

The invention provides an intelligent resolution method for high-speed railway train working diagram conflicts based on reinforcement learning, and belongs to the technical field of railway transportation and operation planning optimization. The method comprises the following steps: acquiring railway working diagram data and railway conflict events; constructing a high-speed railway train working diagram conflict resolution model based on the railway working diagram data; training the high-speed railway train working diagram conflict resolution model by using the railway working diagram data and the railway conflict event to obtain a trained high-speed railway train working diagram conflict resolution model; and importing a to-be-processed railway conflict event into the trained high-speed railway train working diagram conflict resolution model to obtain a railway conflict resolution scheme for the railway conflict event. A railway working diagram basic model capable of interacting with reinforcement learning is constructed, a simple calculation subset and a complex calculation subset are designed for different types of conflicts possibly encountered by a railway, a heuristic algorithm is used as a baseline algorithm, and intelligent resolution of high-speed railway train working diagram conflicts is completed.

Description

Technical Field

[0001] The present invention relates to the technical field of railway transportation and operations optimization, and in particular to a method for intelligently resolving conflicts in a high-speed railway train diagram based on reinforcement learning. Background Art

[0002] Railway timetables are the foundation of railway transportation and directly impact its efficiency. Typically, compiling a high-quality, conflict-free railway timetable requires approximately two to three hundred people for a month, a tedious and time-consuming task. Because obtaining an optimal railway timetable is an NP-hard problem, existing methods rely solely on heuristic algorithms to approach the optimal solution. Currently, domestic research on railway timetable compilation is insufficient, and commonly used methods include mathematical programming, heuristic algorithms, or a combination of both to find feasible solutions. Specifically, for example, the railway timetable compilation problem can be modeled as a multi-objective programming problem, using the ε-constraint method to determine the Pareto frontier. Alternatively, a co-evolutionary ant colony algorithm with a hierarchical optimization strategy can be used to solve the railway timetable compilation problem. Other methods include time-loop iteration, dual algorithms, minimum translation time rules, and translation operation time methods to resolve conflicts and optimize the diagram. However, these algorithms remain insufficient for complex and busy trunk corridors like the Beijing-Shanghai High-Speed ​​Railway. Therefore, current optimization algorithms suffer from the following problems: 1. Traditional optimization algorithms struggle to handle large-scale, nonlinear combinatorial optimization problems and cannot meet the real-time requirements of railway scheduling. 2. The vast variable space of railway timetables creates a combinatorial explosion, making it difficult to find a satisfactory solution quickly. 3. These heuristic algorithms have certain drawbacks and may become trapped in local optima. Furthermore, reinforcement learning methods have recently begun to gain prominence in train timetable adjustments, but most are based solely on Q-Learning algorithms. Even when using the PPO algorithm, the action operators used are relatively simple and have a low level of abstraction. This makes them less targeted at timetable conflicts, hindering efficient timetable adjustments. Summary of the Invention

[0003] The present invention provides a method for intelligently resolving conflicts in high-speed railway train diagrams based on reinforcement learning, which is used to solve the problems in the existing technology that large-scale nonlinear combinatorial optimization problems are difficult to handle and cannot meet the real-time requirements of railway scheduling.

[0004] The present invention provides a method for intelligently resolving conflicts in a high-speed railway train diagram based on reinforcement learning, comprising the following steps.

[0005] Obtain railway operation diagram data and railway conflict events; Constructing a high-speed railway train operation diagram conflict resolution model based on the railway operation diagram data; Using the railway operation diagram data and the railway conflict events to train the high-speed railway train operation diagram conflict resolution model to obtain a trained high-speed railway train operation diagram conflict resolution model; The railway conflict event to be processed is introduced into the trained high-speed railway train operation diagram conflict resolution model to obtain a railway conflict resolution plan for the railway conflict event.

[0006] According to the reinforcement learning-based intelligent conflict resolution method for high-speed railway train diagrams provided by the present invention, the high-speed railway train diagram conflict resolution model is constructed based on the railway diagram data, comprising: Three basic train actions are preset, operator sets and complex conflict operator sets are derived based on the three basic actions, encapsulation processing is performed on the basic model, mathematical modeling is performed, and the high-speed railway train operation diagram conflict resolution model is obtained.

[0007] According to the method for intelligently resolving conflicts in a high-speed railway train diagram based on reinforcement learning provided by the present invention, before constructing a high-speed railway train diagram conflict resolution model based on the railway diagram data, the method includes: Preprocessing the railway operation diagram data and the railway conflict events to obtain preprocessed data; Based on the pre-processed data, simple operator sets and complex operator sets are constructed for different types of railway conflicts.

[0008] According to the reinforcement learning-based intelligent conflict resolution method for high-speed railway train diagrams provided by the present invention, the high-speed railway train diagram conflict resolution model is trained using the railway diagram data and the railway conflict events, including: Based on the three types of train events in the railway timetable, obtain the standard running time, maximum slow travel time unit and maximum slow travel time; The standard running time, the maximum slow travel time unit and the maximum slow travel time are first limited by using a preset relationship; A second limitation is imposed on the time intervals of different operations based on a preset condition; Based on a reinforcement learning algorithm, the high-speed railway train operation diagram conflict resolution model is trained using the first restriction, the second restriction, the simple operator set and the complex operator set to obtain a trained high-speed railway train operation diagram conflict resolution model.

[0009] According to the high-speed railway train diagram conflict intelligent resolution method based on reinforcement learning provided by the present invention, the first limiting formula is:

[0010] in, For trains from Stand The standard operating hours of the station, For trains The maximum slow time unit within the standard running time, For trains Actual running time; The second limiting formula is:

[0011] in, and is a preset variable, where For trains With the station The time interval constraint between For trains and trains Different operation time intervals.

[0012] The method for intelligently resolving conflicts in a high-speed railway train diagram based on reinforcement learning provided by the present invention further includes: Count the number of stop conflicts, running time conflicts, interval conflicts, and overtaking conflicts to obtain the number of conflicts on the entire railway operation diagram; Based on NP-hard combinatorial optimization, the number of conflicts in the entire graph is used as the objective function of the high-speed railway train operation graph conflict resolution model.

[0013] The method for intelligently resolving conflicts in a high-speed railway train diagram based on reinforcement learning provided by the present invention further includes: A heuristic algorithm is used to collect samples, and the high-speed railway train timetable conflict resolution model is pre-trained. The high-speed railway train timetable conflict resolution model is self-exploratory trained using reinforcement learning, and entropy is added to the action probability during sampling.

[0014] The present invention also provides a high-speed railway train diagram conflict intelligent resolution device based on reinforcement learning, comprising the following modules: Data acquisition module, which acquires railway operation diagram data and railway conflict events; A model building module, which builds a high-speed railway train operation diagram conflict resolution model based on the railway operation diagram data; a model training module, which trains the high-speed railway train operation diagram conflict resolution model using the railway operation diagram data and the railway conflict events to obtain a trained high-speed railway train operation diagram conflict resolution model; The prediction module imports the railway conflict event to be processed into the trained high-speed railway train operation diagram conflict resolution model to obtain a railway conflict resolution plan for the railway conflict event.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above methods when executing the computer program.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which implements any of the above methods when executed by a processor.

[0017] The present invention provides a reinforcement learning-based intelligent conflict resolution method for high-speed railway train diagrams. This method constructs a mathematical optimization problem for railway diagram compilation, builds a railway diagram environment based on this optimization problem, and designs a corresponding reinforcement learning algorithm. This algorithm is trained within this environment to solve the dynamic optimization problem of railway diagram conflicts. Data is preprocessed to construct a basic railway diagram model that can interact with reinforcement learning. Simple and complex operator sets are designed for different types of conflicts that may be encountered on the railway. These sets serve as actions for the subsequent reinforcement learning algorithm, allowing targeted selection of operators to resolve the corresponding conflicts. Based on this operator set, a reinforcement learning algorithm is proposed to resolve conflicts, implementing a heuristic algorithm as a baseline algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 It is a flow chart of the method for intelligently resolving conflicts in high-speed railway train operation diagrams based on reinforcement learning provided by the present invention.

[0020] Figure 2 It is a schematic diagram of the reinforcement learning framework provided by the present invention.

[0021] Figure 3 Schematic diagram of the stop time out-of-range operator set provided by the present invention.

[0022] Figure 4 This is a schematic diagram of the first operator set with running time out of range provided by the present invention.

[0023] Figure 5 Schematic diagram of the second operator set with out-of-range running time provided by the present invention.

[0024] Figure 6Schematic diagram of the interval time deficiency operator set provided by the present invention.

[0025] Figure 7 This is a schematic diagram of the complex conflict operator set provided by the present invention.

[0026] Figure 8 This is a schematic diagram comparing the performance of reinforcement learning and heuristic algorithms; Figure 9 This is a diagram comparing the probability of reinforcement learning and heuristic algorithms getting into trouble; Figure 10 This is a comparison diagram of the probability histograms of reinforcement learning and heuristic algorithms successfully resolving all conflicts; Figure 11 It is a structural schematic diagram of the high-speed railway train operation diagram conflict intelligent resolution device based on reinforcement learning provided by the present invention.

[0027] Figure 12 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0028] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0029] The core of the present invention is to provide a method for intelligently resolving conflicts in high-speed railway train timetables based on reinforcement learning. By constructing a mathematical optimization problem for railway timetable compilation, building an environment for railway timetables based on this optimization problem, and designing a corresponding reinforcement learning algorithm, intelligent resolution of conflicts in high-speed railway train timetables is achieved.

[0030] Therefore, there is an urgent need for an intelligent conflict resolution method for high-speed railway train timetables based on reinforcement learning to solve the above problems.

[0031] The following combination Figures 1-12 The present invention describes a method, device, equipment and medium for intelligently resolving conflicts in a high-speed railway train diagram based on reinforcement learning.

[0032] Figure 1 This is one of the flow charts of the high-speed railway train diagram conflict intelligent resolution method based on reinforcement learning provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 101: Obtain railway operation diagram data and railway conflict events.

[0033] In one embodiment, railway operation diagram data and railway conflict events are preprocessed to obtain preprocessed data; and simple operator sets and complex operator sets are constructed for different types of railway conflicts based on the preprocessed data.

[0034] Step 102: Construct a high-speed railway train operation diagram conflict resolution model based on the railway operation diagram data.

[0035] During the specific implementation, three basic train actions are preset, and operator sets and complex conflict operator sets are derived based on the three basic actions. They are encapsulated and processed on the basic model, and mathematical modeling is performed to obtain a conflict resolution model for the high-speed railway train operation diagram.

[0036] Step 103: Train the high-speed railway train operation diagram conflict resolution model using the railway operation diagram data and railway conflict events to obtain a trained high-speed railway train operation diagram conflict resolution model.

[0037] In specific implementation, based on the three types of events of trains in the railway timetable, the standard running time, maximum slow travel time unit and maximum slow travel time are obtained; The standard running time, the maximum slow-moving time unit and the maximum slow-moving time are first limited by using a preset relationship; A second limitation is imposed on the time interval between different operations based on a preset condition; Based on the reinforcement learning algorithm, the high-speed railway train timetable conflict resolution model is trained using the first constraint, the second constraint, the simple operator set and the complex operator set to obtain a trained high-speed railway train timetable conflict resolution model.

[0038] The first limiting formula is:

[0039] in, For trains from Stand The standard operating hours of the station, For trains The maximum slow time unit within the standard running time, For trains Actual running time; The second limiting formula is:

[0040] in, and is a preset variable, where For trains With the station The time interval constraint between For trains and trains Different operation time intervals.

[0041] In one embodiment, the number of stop conflicts, the number of running time conflicts, the number of interval conflicts, and the number of overtaking conflicts are counted to obtain the number of conflicts in the entire railway operation diagram; Based on NP-hard combinatorial optimization, the number of conflicts in the entire graph is used as the objective function of the conflict resolution model for high-speed railway train operation graph.

[0042] A heuristic algorithm is used to collect samples and pre-train the high-speed railway train timetable conflict resolution model. Reinforcement learning is used to conduct mutual self-exploration training on the high-speed railway train timetable conflict resolution model, and entropy is added to the action probability during sampling.

[0043] Step 104: import the railway conflict event to be processed into the trained high-speed railway train diagram conflict resolution model to obtain a railway conflict resolution solution for the railway conflict event.

[0044] This embodiment provides a method for intelligently resolving conflicts in high-speed railway train timetables based on reinforcement learning. A railway timetable environment is constructed based on a mathematical optimization problem, and a corresponding reinforcement learning algorithm is designed. This algorithm is trained within this environment to solve the dynamic optimization problem of railway timetable conflicts. Data is preprocessed to construct a basic railway timetable model that can interact with reinforcement learning. Simple and complex operator sets are designed for different types of conflicts that may be encountered on the railway. These sets serve as actions for the subsequent reinforcement learning algorithm, allowing targeted selection of operators to resolve the corresponding conflicts. Based on this operator set, a reinforcement learning algorithm is proposed to resolve conflicts, implementing a heuristic algorithm as a baseline algorithm to achieve intelligent resolution of high-speed railway train timetable conflicts.

[0045] Based on the above embodiment, this embodiment describes a method for intelligently resolving conflicts in a high-speed railway train diagram based on reinforcement learning, as follows: In this example, data is preprocessed to construct a basic railway diagram model that can interact with reinforcement learning. Furthermore, corresponding sets of simple and complex operators are designed for different types of conflicts that may be encountered on the railway. These operators serve as actions for the subsequent reinforcement learning algorithm, allowing targeted selection of operators to resolve the corresponding conflicts. Based on this operator set, a reinforcement learning algorithm is proposed to resolve conflicts, and a heuristic algorithm is implemented as a baseline algorithm.

[0046] First, we conduct mathematical modeling for the railway operation diagram problem. The railway operation diagram model can be expressed as:

[0047] in, Srepresents a collection of railway platforms, T Represents a set of trains. There are two different types of trains (denoted by 𝐿), This is a railway operation diagram model.

[0048] In the railway operation diagram, there are three types of events for trains: Depart, Arrive, and Pass, which mean departure (from a certain station), arrival (at a certain station), and passing (a certain station). By default, Depart and Arrive events indicate that the train has stopped at the station. Indicates that train 𝑡 is at the station Is there a stop ( = 1 means there is a stop), if train 𝑡 is at the station There are stops, remember For train 𝑡 at the station The stop time is Must be greater than or equal to station Minimum stop time , and is less than or equal to its longest stop time ,Right now

[0049] At the station ( , ), if train 𝑡 is between If there is a stop, there will be additional time when the train leaves, which is recorded as ; Similarly, if the train 𝑡 is If there is a stop, there will be additional additional time, recorded as ; At the same time, train 𝑡 from arrive Requires a pure runtime ; Thus, the train 𝑡 from arrive The standard running time can be expressed as:

[0050] is the standard running time, and there is also a maximum slow running time , which means that at the station ( ), train 𝑡 is allowed to run within the standard time Maximum slow travel based on Time unit, we record is the actual running time of train 𝑡, which must satisfy the following relationship:

[0051] We use Denote the departure time of train 𝑡 at station 𝑘, using Denote the time when train 𝑡 arrives at station 𝑘, using It represents the time when train 𝑡 passes through station 𝑘. The time period for compiling the train operation diagram is (Generally one day is the cycle, i.e. = 1440 𝑚𝑖𝑛), and we define the operator 𝛩 to prevent negative time differences: 𝑚𝑜dT = 𝑥− 𝑦+ 𝑚𝑜d

[0052] in, 𝑚𝑜dT is a defined operator that represents the time difference. , any two trains (𝑖, 𝑗) must satisfy a series of interval time constraints, which are related to the type of train and the type of event at the station. We use Indicates the type of train 𝑖 (benchmark car or ordinary car), and uses Indicates type The event type of an event of a train includes (departure), (arrival) and (through), finally we use the interval time function:

[0053] in, is the defined tau function, As time, the event type pairs at station 𝑆 are mapped to the constrained interval time. At this time, all the interval time constraints that the train pair (𝑖, 𝑗) needs to satisfy are as follows (stipulating that train 𝑗 operates after train 𝑖): For variables , , , ,and hour , the interval between different operations is , then

[0054] To constrain multiple events; On the other hand, at the station Any two trains (𝑖, 𝑗) must also satisfy the requirement that there is no intersection on the graph, that is:

[0055] in, For departure time, To reach the arrival time, between two adjacent stations, the train that leaves station k first must arrive at station k+1 first. The above is the constraint condition; All the inequalities listed above are all the constraints that need to be satisfied. Even on a busy railway section, all the above constraints must be satisfied between any two trains. The huge number of constraints is the fundamental reason for the difficulty of the problem.

[0056] The original railway operation diagram is a loose railway operation diagram, which contains many conflicts. Conflicts can be divided into intra-train conflicts and inter-train conflicts. Intra-train conflicts can be divided into stop time exceeding the range and running time exceeding the range; while inter-train conflicts can be divided into insufficient interval time and overtaking. use , , , They represent the number of stop conflicts, travel time conflicts, interval conflicts, and overtaking conflicts on the diagram, respectively. There are many criteria for evaluating the quality of a train timetable, such as the average total train travel time, the overall OD accessibility of the diagram, and the balance of stops across the entire diagram. However, the most fundamental factor is the number of conflicts across the entire diagram. Therefore, the objective function of this embodiment at this stage is defined as the number of conflicts across the entire railway timetable, with the goal of minimizing this number:

[0057] in, To minimize the number of conflicts, the entire railway diagram compilation model optimization problem has been fully presented. It is essentially an NP-hard combinatorial optimization problem.

[0058] The reinforcement learning algorithm is designed based on the above combinatorial optimization problem. Figure 2 As shown in Figure 2, the reinforcement learning paradigm is that the agent interacts with the environment and finally maximizes the cumulative reward:

[0059] in, is the reinforcement learning expression, is the state distribution sampling, Sampling the policy function; This embodiment provides a method for intelligently resolving conflicts in high-speed railway train timetables based on reinforcement learning. A railway timetable environment is constructed based on a mathematical optimization problem, and a corresponding reinforcement learning algorithm is designed. This algorithm is trained within this environment to solve the dynamic optimization problem of railway timetable conflicts. Data is preprocessed to construct a basic railway timetable model that can interact with reinforcement learning. Simple and complex operator sets are designed for different types of conflicts that may be encountered on the railway. These sets serve as actions for the subsequent reinforcement learning algorithm, allowing targeted selection of operators to resolve the corresponding conflicts. Based on this operator set, a reinforcement learning algorithm is proposed to resolve conflicts, implementing a heuristic algorithm as a baseline algorithm to achieve intelligent resolution of high-speed railway train timetable conflicts.

[0060] Based on the above embodiment, this embodiment describes the constraints in the intelligent conflict resolution method for high-speed railway train diagrams based on reinforcement learning, as follows: At the beginning of each training round, we get a new relaxed railway graph. The relaxed railway graph is mainly generated by some simple random actions such as adding trains, canceling trains, adding stops, canceling stops, randomly perturbing the running time, and random translation.

[0061] Because single-train conflicts (stop time exceeding the range, running time exceeding the range) are easily mediated, and overtaking conflicts and insufficient headroom conflicts do not generate new single-train conflicts (that is, inter-train conflicts are adjusted without causing intra-train conflicts), the reinforcement learning method pre-processes and eliminates intra-train conflicts (stop time exceeding the range and running time exceeding the range) from the initial relaxed railway timetable. This leaves only overtaking conflicts and insufficient headroom conflicts in the entire adjustment process. After this processing, we obtain the reinforcement learning training environment used for network training.

[0062] We define three basic actions in the model: adding / canceling trains, adding / canceling stops, and shifting all events before or after a specified event at a station forward / backward. Building upon these basic actions, we derive operator sets for four conflict types and a general set of complex conflict operators (for difficult-to-resolve overtaking and insufficient headroom conflicts). We chose to further abstract and encapsulate this basic model primarily because the basic action set's operations are non-directional, and each has relatively large degrees of freedom. Directly utilizing these basic action sets to optimize the model would be extremely challenging.

[0063] The four types of conflicts mentioned above are: exceeding the stop time limit, exceeding the running time limit, overtaking, and insufficient interval time. The first two conflicts are intra-train conflicts (involving only one train), and the last two conflicts are inter-train conflicts (involving two trains).

[0064] Stop time out of range: Two operators are designed for the conflict “stop time out of range”, such as when train 𝑡 is at a station There are stops ( ) But the stop time When the train stops at the station, operator 1 in Figure 6.1 is used to extend the train's stop time until it is equal to the lower bound of the stop time. .

[0065] When train 𝑡 arrives at a station There are stops ( ) But the stop time When using Figure 3 Operator 2 shortens the train's stop time until it equals the upper bound of the stop time. .

[0066] Run time out of range: Train 𝑡 from arrive The operating time range is determined by the standard operating time and maximum slow travel time Define, use Denotes that the train 𝑡 arrive The actual running time, then

[0067] when When the above inequality rules are violated, a "running time out of range" conflict will occur, and operators are used at this time. The operator set is as follows: Figure 4 shown.

[0068] When train 𝑡 arrive Actual running time , we call operator 1 in Figure 5 to extend the running time of the train until it is equal to the minimum lower bound of the running time .

[0069] When train 𝑡 arrive Actual running time When we call Figure 6 Operator 2 shortens the train's running time until it equals the upper bound of the running time. .

[0070] overtake: Overtaking refers to the time at the station ( , ), the departure order of trains (𝑖, 𝑗) ( ) and arrival order( ) in reverse order, i.e.

[0071] Since overtaking conflicts involve the relationship between any two trains at any two stations, they are particularly complex and are a difficult problem in the compilation of railway timetables. In addition, they are safety hazards that are absolutely not allowed to exist in actual diagram compilation.

[0072] Considering the complexity of overtaking conflicts, we set 13 conflict operators for it, such as Figure 5 It is worth mentioning that the adjustment time of these operators is the minimum adjustment time that can resolve the current conflict.

[0073] Insufficient interval time: At the same station , any two trains ( i, j ) A series of interval time constraints to be satisfied. When any one or more of them are not satisfied, an insufficient interval time conflict occurs. This embodiment designs 10 conflict operators to resolve insufficient interval time conflicts, such as Figure 6 These operators are the same as the overtaking operators, and the adjustment time when resolving conflicts is the minimum adjustment time.

[0074] Complex conflict operator set: This embodiment designs five common complex conflict operators, such as Figure 7 As shown. Note that the above diagram is only an example of each operator in the complex strategy and does not fully represent the complex conflict operator. The complex strategy designed in this embodiment is highly versatile and can be used for both overtaking conflicts and insufficient headway conflicts, simply by specifying the required parameters. Furthermore, the complex operator in this embodiment can be called by each conflicting train, regardless of whether it is the leading or trailing train, though this is not reflected in the diagram.

[0075] Network Settings: This embodiment uses the PPO algorithm as the optimization algorithm, and the network structure adopts the Actor-Critic structure. The state inputs of the Actor network and the Critic network are a global information list and a local train schedule matrix. The Actor network is used to predict the probability of each operator that can be adopted to obtain the highest return in this round, and the Critic network is used to predict the state value function of the current state. .

[0076] Reward function settings: The reinforcement learning method's reward function is based on the number of conflicts reduced after the operator is applied. To minimize conflicts, the reinforcement learning method uses a weighted approach to calculate the final single-step reward based on the number of conflicts reduced. Conflicts at the station immediately preceding and preceding the conflict are given greater weight, while conflicts at the station immediately following and following the conflict are given less weight. This approach helps to minimize conflicts and congregate them at the lowest station possible.

[0077] This embodiment calculates the cumulative reward value using a discount factor based on the trajectory sampled from the replay buffer:

[0078] in is the value function, which is the expected return obtained when starting from a certain state-action pair and continuing according to a certain strategy.

[0079] Among them, the discount factor = 0.99.

[0080] Performance improvement strategies: To aid model training, this example uses a heuristic algorithm to collect samples for supervised learning as preliminary training. This is then followed by reinforcement learning self-exploratory training. Furthermore, because the network is relatively small, entropy is added to the action probabilities during sampling to prevent premature model convergence, thereby increasing the model's exploration power.

[0081] During testing, because the initial image of each new environment is significantly different, to help the model adapt to the new environment, the trained model is first subjected to k rounds of reinforcement learning self-exploration training in the new environment to fine-tune the model parameters. This is called model warm-up. This fine-tuning method allows the model to adapt to the environment in advance, thereby performing better in the new environment.

[0082] To ensure the model performs well in environments of varying difficulty, a multi-strategy decision-making approach was used during testing. Eight strategies were set in the experiment. Different strategies were tested simultaneously in the same environment in parallel, and the highest-scoring strategy from the eight strategies was selected as the adjustment solution for that environment.

[0083] Based on the above embodiment, this embodiment uses specific data to briefly illustrate this method, as follows: This example randomly generates initial relaxed railway operation diagrams of varying complexity and difficulty, and sets the maximum number of steps in each debugging round to 512. The reinforcement learning method of this application is compared with the heuristic algorithm in terms of the number of steps required to resolve the initial relaxed railway operation diagram, the probability of getting stuck (such as local optimality and blind exploration), and the probability of successfully resolving conflicts.

[0084] This example sets the environment difficulty to 16 levels, corresponding to the number of random generator perturbations from 50 to 800 (50, 100, 150, 200, ...). At each level, we test the number of steps required by the two algorithms to resolve 100 different initial conflict run graphs. Figure 8 It can be seen that the reinforcement learning method of this application is superior to the heuristic algorithm in terms of efficiency and performance. The heuristic algorithm adopts a one-step greedy strategy. In the current state, the heuristic algorithm will use all available operators and then select the operator that works best in the current state. However, the current optimality is not necessarily the future optimality. In contrast, the reinforcement learning method of this application gradually makes the model proficient in the process of adjusting the map through training, allowing the intelligent agent of this application to comprehensively consider the environment and solve problems more quickly.

[0085] Since the maximum number of adjustment steps is 512 and far exceeds the normal number of steps required for each difficulty level, if the problem is not solved after 512 steps in an environment, it is considered that the method is stuck in a dilemma (local optimum, blind exploration, etc.) in this environment. The difficulty of the environment is set to 8 levels, corresponding to the number of random generator perturbations from 100 to 800 (100, 200, ...). This embodiment counts the number of times the reinforcement learning method and the heuristic algorithm are stuck in a dilemma at each difficulty level, as well as the number of successes and the corresponding probabilities. Figure 9 、 10 It can be seen that the reinforcement learning method of this embodiment is far more effective than the heuristic algorithm. Regardless of the difficulty level, the probability of the reinforcement learning method of this application getting into trouble is much lower than that of the heuristic algorithm, and the probability of successfully resolving all conflicts is much higher than that of the heuristic algorithm.

[0086] The following describes the high-speed railway train timetable conflict intelligent resolution device based on reinforcement learning provided by the present invention. The high-speed railway train timetable conflict intelligent resolution device based on reinforcement learning described below and the high-speed railway train timetable conflict intelligent resolution method based on reinforcement learning described above can be referenced to each other.

[0087] like Figure 11 As shown, the high-speed railway train operation diagram conflict intelligent resolution device 1100 based on reinforcement learning provided by the present invention includes the following modules: Data acquisition module 1101, acquires railway operation diagram data and railway conflict events; A model building module 1102 builds a high-speed railway train operation diagram conflict resolution model based on the railway operation diagram data; The model training module 1103 trains the high-speed railway train diagram conflict resolution model using the railway diagram data and the railway conflict events to obtain a trained high-speed railway train diagram conflict resolution model; The prediction module 1104 imports the railway conflict event to be processed into the trained high-speed railway train diagram conflict resolution model to obtain a railway conflict resolution solution for the railway conflict event.

[0088] Figure 12 An example of a physical structure diagram of an electronic device is shown below. Figure 12 As shown, the electronic device may include: a processor 1210, a communication interface 1220, a memory 1230, and a communication bus 1240, wherein the processor 1210, the communication interface 1220, and the memory 1230 communicate with each other via the communication bus 1240. The processor 1210 may call the logic instructions in the memory 1230 to execute a high-speed railway train diagram conflict intelligent resolution method based on reinforcement learning, which includes: Obtain railway operation diagram data and railway conflict events; Constructing a high-speed railway train operation diagram conflict resolution model based on the railway operation diagram; Using the railway operation diagram data and the railway conflict events to train the high-speed railway train operation diagram conflict resolution model to obtain a trained high-speed railway train operation diagram conflict resolution model; The railway conflict event to be processed is introduced into the trained high-speed railway train operation diagram conflict resolution model to obtain a railway conflict resolution plan for the railway conflict event.

[0089] Furthermore, the logic instructions in the aforementioned memory 1230 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0090] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for intelligently resolving conflicts in a high-speed railway train diagram based on reinforcement learning provided by the above methods is implemented. The method comprises: Obtain railway operation diagram data and railway conflict events; Constructing a high-speed railway train operation diagram conflict resolution model based on the railway operation diagram data; Using the railway operation diagram data and the railway conflict events to train the high-speed railway train operation diagram conflict resolution model to obtain a trained high-speed railway train operation diagram conflict resolution model; The railway conflict event to be processed is introduced into the trained high-speed railway train operation diagram conflict resolution model to obtain a railway conflict resolution plan for the railway conflict event.

[0091] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0092] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for intelligently resolving conflicts in high-speed railway train operation diagrams based on reinforcement learning, characterized in that: The method comprises: Obtain railway operation diagram data and railway conflict events; Constructing a high-speed railway train operation diagram conflict resolution model based on the railway operation diagram data; Using the railway operation diagram data and the railway conflict events to train the high-speed railway train operation diagram conflict resolution model, to obtain a trained high-speed railway train operation diagram conflict resolution model; The railway conflict event to be processed is introduced into the trained high-speed railway train operation diagram conflict resolution model to obtain a railway conflict resolution plan for the railway conflict event.

2. The high-speed railway train operation diagram conflict intelligent resolution method based on reinforcement learning according to claim 1 is characterized in that: The method of constructing a high-speed railway train operation diagram conflict resolution model based on the railway operation diagram data comprises: Three basic train actions are preset, operator sets and complex conflict operator sets are derived based on the three basic actions, encapsulation processing is performed on the basic model, mathematical modeling is performed, and the high-speed railway train operation diagram conflict resolution model is obtained.

3. The high-speed railway train operation diagram conflict intelligent resolution method based on reinforcement learning according to claim 1 is characterized in that: Before building the high-speed railway train operation diagram conflict resolution model based on the railway operation diagram data, the method includes: Preprocessing the railway operation diagram data and the railway conflict events to obtain preprocessed data; Based on the preprocessed data, simple operator sets and complex operator sets are constructed for different types of railway conflicts.

4. The method for intelligently resolving conflicts in high-speed railway train operation diagrams based on reinforcement learning according to claim 3 is characterized in that: The using the railway operation diagram data and the railway conflict events to train the high-speed railway train operation diagram conflict resolution model comprises: Based on the three types of events of trains in the railway operation diagram, the standard running time, the maximum slow-moving unit time and the maximum slow-moving time are obtained; The standard running time, the maximum slow-moving time unit and the maximum slow-moving time are first limited by using a preset relationship; A second limitation is imposed on the time intervals of different operations based on preset conditions; Based on a reinforcement learning algorithm, the high-speed railway train operation diagram conflict resolution model is trained using the first restriction, the second restriction, the simple operator set and the complex operator set to obtain a trained high-speed railway train operation diagram conflict resolution model.

5. The method for intelligently resolving conflicts in high-speed railway train operation diagrams based on reinforcement learning according to claim 4 is characterized in that: The first limiting formula is: in, For trains from Stand The standard operating hours of the station, For trains The maximum slow time unit within the standard running time, For trains Actual running time; The second limiting formula is: in, and is the preset variable, where For trains With the station The time interval constraint between For trains and train Different operation time intervals.

6. The high-speed railway train operation diagram conflict intelligent resolution method based on reinforcement learning according to claim 4 is characterized in that: Also includes: Count the number of stop conflicts, running time conflicts, interval conflicts and overtaking conflicts, and obtain the number of conflicts in the entire railway operation diagram; Based on NP-hard combinatorial optimization, the number of conflicts in the entire graph is used as the objective function of the conflict resolution model of the high-speed railway train operation graph.

7. The method for intelligently resolving conflicts in high-speed railway train operation diagrams based on reinforcement learning according to claim 4 is characterized in that: Also includes: A heuristic algorithm is used to collect samples, the high-speed railway train operation diagram conflict resolution model is pre-trained, reinforcement learning is used to perform mutual self-exploration training on the high-speed railway train operation diagram conflict resolution model, and entropy is added to the action probability during sampling.

8. A high-speed railway train operation diagram conflict intelligent resolution device based on reinforcement learning, characterized in that: include: Data acquisition module, which acquires railway operation diagram data and railway conflict events; A model building module, which builds a high-speed railway train operation diagram conflict resolution model based on the railway operation diagram data; A model training module, using the railway operation diagram data and the railway conflict events to train the high-speed railway train operation diagram conflict resolution model to obtain a trained high-speed railway train operation diagram conflict resolution model; The prediction module imports the railway conflict event to be processed into the trained high-speed railway train operation diagram conflict resolution model to obtain a railway conflict resolution plan for the railway conflict event.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images