Train scheduling optimization method, device and equipment and storage medium
By deploying a decision network at a train station and adjusting the train strategy using DDPG algorithm and reward function, the problem that the existing technology cannot adjust train scheduling in real time is solved, and the optimal scheduling and energy-saving effect in the case of delay disturbance is achieved.
Patent Information
- Application Number
- CN202310640063.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-31
- Publication Date
- 2024-12-06
AI Technical Summary
The existing technology cannot adjust the train scheduling plan in real time. Faced with delay disturbances in train operation, resulting in error accumulation and energy consumption increase. Heuristic algorithms and mixed integer planning calculations take a long time and cannot meet the real-time requirements.
Through the decision network deployed at the current site, interactive training is performed using DDPG algorithm and preset train operation simulation environment, and a reward function is constructed to calculate the overlapping time between the braking time and acceleration time of adjacent trains, and obtain the policy information of the target train. And sent to the train to adjust the operation.
It realizes the optimal scheduling and energy-saving solution for real-time and rapid determination of trains in the event of delay disturbance, solves the problem of real-time adjustment of train scheduling and reduces energy consumption.
Smart Images

Figure CN119090695A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of rail transit technology, and in particular to a train scheduling optimization method, device, equipment and storage medium. Background Art
[0002] As a vital component of rail transit, subways have become the preferred mode of transportation for large and medium-sized cities developing urban rail transit. However, unexpected delays during train stops, such as when doors remain locked due to passengers, can disrupt the optimal scheduling plan, rendering it ineffective. If the optimal scheduling plan is still followed, errors caused by these delays will accumulate, further increasing energy consumption.
[0003] However, the current heuristic algorithms and mixed integer programming cannot meet the real-time requirements. A single calculation consumes a lot of time and computing power, and in the face of disturbances in train operation, it is impossible to adjust the train scheduling plan in real time. Summary of the Invention
[0004] The present invention provides a train dispatch optimization method, device, equipment and storage medium, which can quickly determine the optimal dispatch energy-saving plan for the train in real time when a delay disturbance occurs.
[0005] According to one aspect of the present invention, a train scheduling optimization method is provided, comprising:
[0006] When it is determined that the target train parked at the current station meets the train rescheduling conditions, the current state information of all trains on the target line corresponding to the target train is obtained through the decision network deployed at the current station;
[0007] The decision network is obtained through at least one round of interactive training between an initial decision network based on the DDPG algorithm and a preset train operation simulation environment; a reward function is constructed within the preset train operation simulation environment; the reward function is implemented by calculating the overlap time between the braking time information and the acceleration time information of adjacent trains;
[0008] Acquire, based on the current state information, strategy information for the target train to leave the current station and arrive at the next station;
[0009] The strategy information is sent to the target train so that the target train operates according to the strategy information.
[0010] According to another aspect of the present invention, a train scheduling optimization device is provided, comprising:
[0011] A train information acquisition module is used to determine that a target train parked at a current station meets the train rescheduling conditions, and to obtain the current status information of all trains on the target line corresponding to the target train through a decision network deployed at the current station;
[0012] The decision network is obtained through at least one round of interactive training between an initial decision network based on the DDPG algorithm and a preset train operation simulation environment; a reward function is constructed within the preset train operation simulation environment; the reward function is implemented by calculating the overlap time between the braking time information and the acceleration time information of adjacent trains;
[0013] a strategy information acquisition module, configured to acquire strategy information for the target train to leave the current station and arrive at the next station according to the current state information;
[0014] The strategy information sending module is used to send the strategy information to the target train so that the target train runs according to the strategy information.
[0015] According to another aspect of the present invention, an electronic device is provided, comprising:
[0016] at least one processor; and
[0017] a memory communicatively connected to the at least one processor; wherein,
[0018] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the train scheduling optimization method described in any embodiment of the present invention.
[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the train scheduling optimization method described in any embodiment of the present invention when executed.
[0020] The technical solution of the present invention determines that the target train stopped at the current station meets the train rescheduling conditions, and then obtains the current state information of all trains on the target line corresponding to the target train through a decision network deployed at the current station; wherein, the decision network is obtained through at least one round of interactive training between an initial decision network based on the deep deterministic policy gradient (DDPG) algorithm and a preset train operation simulation environment; a reward function is constructed in the preset train operation simulation environment; the reward function is implemented by calculating the overlap time between the braking time information and the acceleration time information of adjacent trains; based on the current state information, the strategy information of the target train leaving the current station and arriving at the next station is obtained; the strategy information is sent to the target train so that the target train operates according to the strategy information. This solves the problem that the existing technology cannot adjust the train scheduling strategy in real time in the face of delay disturbances in train operation, and achieves the effect of quickly and in real time determining the optimal scheduling energy-saving plan for the train when delay disturbances occur.
[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the technical solutions in the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0023] Figure 1a A flow chart of a train scheduling optimization method provided in the first embodiment;
[0024] Figure 1b A schematic diagram of the structure of a decision maker provided in the first embodiment;
[0025] Figure 1c A schematic diagram of a train dispatch optimization method implemented using an agent-based decision network provided in the first embodiment;
[0026] Figure 2a A schematic diagram of calculating the overlap time between train braking time information and acceleration time information provided in the first embodiment;
[0027] Figure 2b A flowchart of a decision network acquisition method provided in the first embodiment;
[0028] Figure 3 A schematic diagram of the structure of a train dispatching optimization device provided in the second embodiment;
[0029] Figure 4 3 is a schematic diagram of the structure of an electronic device for implementing the train scheduling optimization method of this embodiment. DETAILED DESCRIPTION
[0030] In order to enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.
[0031] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0032] Example 1
[0033] Figure 1a This is a flow chart of a train scheduling optimization method provided in the first embodiment. This embodiment is applicable to the case of rescheduling when a train is delayed. The method can be executed by a train scheduling optimization device. The train scheduling optimization device can be implemented in the form of hardware and / or software. The train scheduling optimization device can be configured in an intelligent agent deployed at a station. Figure 1a As shown, the method includes:
[0034] S110. When it is determined that the target train stopped at the current station meets the train rescheduling condition, the current status information of all trains on the target line corresponding to the target train is obtained through the decision network deployed at the current station.
[0035] Optionally, determining whether the target train stopped at the current station meets the train rescheduling conditions may include: obtaining the current planned stop duration of the target train at the current station from the currently stored driving strategy of the target train, and obtaining the current actual stop duration of the target train at the current station; when the current actual stop duration exceeds the current planned stop duration, determining that the target train meets the train rescheduling conditions.
[0036] Current state information can include current speed, current position, and the current stored driving strategy. For example, Train 5, currently parked at Station 1, failed to close its doors on time according to the original driving strategy (i.e., the target train's current stored driving strategy). The original driving strategy had Train 5's current planned stop duration at Station 1 of 3 minutes. However, Train 5's current actual stop duration at Station 1 is 4 minutes, exceeding the current planned stop duration. Therefore, it can be determined that Train 5 has experienced a delay disturbance. In other words, the decision network deployed at Station 1 determines that Train 5 meets the train rescheduling conditions. The decision network can obtain the current speed, current position, and current stored driving strategy of all trains on the corresponding line of Train 5 through sensors.
[0037] S120. Obtain strategy information for the target train to leave the current station and arrive at the next station according to the current state information.
[0038] The strategy information may include the cruising speed and the duration of the stop at the next station.
[0039] Specifically, current status information such as the current speed, current position, and current stored driving strategy data of all trains on the target line can be input into the corresponding decision network, which outputs the cruising speed of the target train from the current station to the next station, as well as the length of time the target train will stop at the next station.
[0040] In this embodiment, the intelligent agent can be deployed at the site, and the intelligent agent can be provided with a decision maker, which is implemented through a decision network. Figure 1b A schematic diagram of the decision maker structure is provided for this embodiment. The decision maker includes an input layer, a selector, and a voter. The input layer inputs the current status information of all trains (including current speed, current position, and currently stored driving strategy). The selector selects the target decision maker from multiple decision makers in the decision network based on which train is running at which station. Finally, the voter outputs the cruising speed to the next station and the length of the stop at the next station. The number of decision makers in the decision network can be determined according to the number of stations and trains in the line. For example, assuming that a subway line has M stations and a total of N trains, then M*N decision makers need to be deployed to provide intelligent decision guidance for each train at each station.
[0041] S130: Send the strategy information to the target train so that the target train runs according to the strategy information.
[0042] In this example, the decision network's decisions are limited to the train's departure time, involving only determining the train's cruising speed at the next station and the duration of the stop at the next station. This design ensures that the dimension of the policy output is always 2, regardless of the number of trains or stations in the test segment. The decision network outputs policies in the range [-1, 1] and then maps these policies to values within the range of stop duration and cruising speed. This mapping ensures that the decision network's decisions meet the practical constraints of the system.
[0043] The intelligent agent can send the strategy information obtained based on the decision network to the target train so that the target train can run according to the strategy information, that is, reconstruct the driving schedule for the target train at the current moment, thereby achieving the purpose of optimizing train scheduling.
[0044] In order to enable those skilled in the art to better understand the train dispatch optimization method of the present invention, Figure 1c This embodiment provides a schematic diagram of an agent-based decision network for implementing a train scheduling optimization method. The agent outputs a decision action (i.e., strategy information) for the train currently leaving station m. Based on the train's status information (such as train identification, planned and actual stop durations) and the status information of other trains on the line (such as speed, displacement, and train control strategy), the agent determines the cruising speed and next stop duration for the train leaving station m.
[0045] In this technical solution, since the network is pre-trained offline, the network weights obtained through training can effectively cope with various situations. The DDPG algorithm is used to optimize train scheduling. When faced with different states, the output decision ensures that the train can maximize energy recovery and utilization by taking this decision at the current moment.
[0046] The technical solution of this embodiment is to obtain the current state information of all trains on the target line corresponding to the target train through a decision network deployed at the current station when determining that the target train stopped at the current station meets the train rescheduling conditions. The decision network is obtained through at least one round of interactive training between an initial decision network based on the DDPG algorithm and a preset train operation simulation environment. The preset train operation simulation environment includes a reward function. The reward function is implemented by calculating the overlap time between the braking time information and the acceleration time information of adjacent trains. Based on the current state information, the strategy information for the target train to leave the current station and arrive at the next station is obtained. The strategy information is sent to the target train so that the target train operates according to the strategy information. This solves the problem that the existing technology cannot adjust the train scheduling strategy in real time when delay disturbances occur in train operation. It achieves the effect of quickly and in real time determining the optimal scheduling and energy-saving plan for the train when delay disturbances occur.
[0047] In this embodiment, the preset train operation simulation environment is a pre-built virtual environment that simulates the train operation environment. This environment can be used to construct an environment suitable for reinforcement learning. The necessary functions include the reset function and the step function. The simpy library can be used to construct the environment deduction during the step. It can also construct a state space and action space suitable for reinforcement learning. It can also construct a reward function and a function to determine whether the destination has been reached.
[0048] The reward function can be used to provide feedback during the interaction between the initial decision network and the pre-set train operation simulation environment. This can be achieved by calculating the overlap between the braking and acceleration time information of adjacent trains. The reward function can take into account the actions selected by the agent and compare them with the overlap time calculated in the previous step. By doing so, the utilization of the regenerative braking energy that has been utilized since the previous step can be accurately determined, which becomes the step reward. This reward function incentivizes the agent to optimize the use of regenerative braking energy and encourages efficient and energy-efficient operation of the train system.
[0049] For example, Figure 2a As shown, trains i-1, i, and i+1 are connected to the overhead contact line. Train i-1 is accelerating to depart from station n, train i is braking to arrive at station n, and train i+1 is accelerating to depart from station n-1 and arrive at station n. The energy saved by braking train i can be reused by trains i-1 and i+1 via the overhead contact line.
[0050] Depend on Figure 2a The speed-time relationship diagram on the left side shows that train i is at b i(n-1) Start braking at time a in At time point, braking ends and the train arrives at station n; train i+1 is at d (i+1)(n-1) The time point starts to accelerate, at c (i+1)(n-1) At the time point, the acceleration ends and the train leaves station n-1. There is a gap between train i and train i+1 from d (i+1)(n-1) to a in overlap time. Figure 2a In the speed-time relationship diagram on the right side, the train i and train i-1 are similar. There is a relationship between them from d (i-1)n to a in The reward function can determine the size of the reward value by calculating the overlap time.
[0051] The preset train operation simulation environment may include basic station information, basic train information, train time model, train energy consumption model and train operation model.
[0052] Basic site information may include site identification information, site quantity, site distribution information, and distance information between adjacent sites;
[0053] The basic train information may include train identification information and train number;
[0054] The train time model can be used to display the time information of each train running to each station;
[0055] The train energy consumption model can be used to display the energy consumption information of the train from the time it arrives at the station to the time it leaves the station;
[0056] The train operation model can be used to display the operation information of three adjacent trains at the same station, such as Figure 2a shown.
[0057] The decision network is essentially a result of the interaction between an initial decision network based on the reinforcement learning algorithm (DDPG) and a pre-set train operation simulation environment. The network parameters are updated through repeated training to gradually reach the point where it can output optimal decisions. The initial decision network based on the DDPG algorithm can include an actor network, an actor target network, a critic network, and a critic target network.
[0058] Based on the above preset train operation simulation environment and initial decision network, Figure 2b This is a flow chart of a decision network acquisition method provided in this embodiment. Figure 2b As shown, the method may include:
[0059] S210: Initialize the initial decision network.
[0060] S220. Utilize the current state of the train to be decided and scheduled to depart at the current time in the preset train operation simulation environment and the Actor network to obtain the current first driving strategy that matches the current state, and add preset noise to the current first driving strategy to obtain the current second driving strategy.
[0061] For example, given a current state S0, the action a'0 is obtained through the Actor network, and then noise N is added to obtain the action a0=a'0+N (the noise is to ensure a certain amount of exploration, for example, the noise can be Gaussian noise). Here a'0 is the current first driving strategy, N is the preset noise, and a0 is the current second driving strategy.
[0062] S230: Utilize the train to be decided to execute the current second driving strategy to obtain the next state and reward value.
[0063] Continuing with the example of S220, a0 is input into the preset train operation simulation environment, that is, the train to be decided is used to execute the current second driving strategy to obtain S1 and r1, where S1 is the next state and r1 is the reward value.
[0064] S240: Store the current state, the current second driving strategy, the next state, and the reward value as a set of experience data in the experience replay buffer.
[0065] Continuing with the example of S230 , a set of experience data ( S0 , a0 , S1 , r1 ) is obtained, and then the experience data is put into the experience playback buffer.
[0066] S250: Using the stored experience data in the experience replay buffer, the initial decision network parameters are updated to obtain a decision network.
[0067] Specifically, in an optional implementation, updating the initial decision network parameters using the experience data stored in the experience replay buffer may include the following steps:
[0068] Step 1: Sampling the stored experience data to obtain at least one set of target experience data;
[0069] Step 2: For each set of target experience data, perform the following operations: update the Actor network according to the current state in the currently processed target experience data to obtain the current updated Actor network; update the Critic network according to the next state and reward value in the currently processed target experience data to obtain the current updated Critic network;
[0070] In an optional embodiment, updating the Actor network according to the current state in the currently processed target experience data to obtain the current updated Actor network may include: using the Actor network to obtain a first predicted driving strategy based on the current state in the currently processed target experience data; using the Critic network to obtain a first Q value based on the current state and the first predicted driving strategy; using the first Q value to obtain a first loss function of the Actor network, and using the first loss function to update the Actor network to obtain the current updated Actor network.
[0071] For example, (S0, a0, S1, r1) is used as the current processing target experience data. Input S0 into the Actor network and obtain the predicted action (i.e., the first predicted driving strategy) a 0_预测 , S0 and a 0_预测 Input the Critic network and get the first Q value Q(s0,a 0_预测 ), thereby using the first Q value to obtain the first loss function of the Actor network, and using the first loss function to update the Actor network to obtain the current updated Actor network.
[0072] In another optional embodiment, updating the Critic network according to the next state and reward value in the currently processed target experience data to obtain the currently updated Critic network can include: using the Actor target network to obtain a second predicted driving strategy based on the next state in the currently processed target experience data; using the Critic target network to obtain a second Q value based on the next state and the second predicted driving strategy; using the second Q value and the reward value to obtain a second loss function of the Critic network, and using the second loss function to update the Critic network to obtain the currently updated Critic network.
[0073] Exemplarily, S1 is input into the Actor target network to obtain the second predicted driving strategy a1, S1 and a1 are input into the Critic target network to obtain the second Q value Q(s1,a1), the difference between Q(s0,a0) and r1+γQ(s1,a1) is determined as the second loss function, and the second loss function is used to update the Critic network to obtain the current updated Critic network.
[0074] Step 3: At every first preset interval, the parameters of the currently updated Actor network are processed by a first ratio, and then the Actor target network is updated;
[0075] For example, assume that the parameters of the Actor network after the current update are θ Q' , we can use θ Q' After the first proportional processing, it is τθ Q +(1-τ)θ Q' , according to τθ Q +(1-τ)θ Q' Updates the Actor target network.
[0076] Step 4: Every second preset interval, the parameters of the currently updated critic network are processed by the second ratio, and then the critic target network is updated.
[0077] The operation of updating the Critic target network can be done in the same way as the operation of updating the Actor target network in step 3.
[0078] Example 2
[0079] Figure 3 This is a structural diagram of a train dispatch optimization device provided in the second embodiment. Figure 3 As shown, the device includes: a train information acquisition module 310, a strategy information acquisition module 320 and a strategy information sending module 330.
[0080] The train information acquisition module 310 is configured to, upon determining that a target train stopping at a current station meets train rescheduling conditions, acquire the current state information of all trains on the target line corresponding to the target train through a decision network deployed at the current station; wherein the decision network is obtained through at least one round of interactive training between an initial decision network based on the DDPG algorithm and a preset train operation simulation environment;
[0081] A strategy information acquisition module 320 is configured to acquire strategy information for the target train to leave the current station and arrive at the next station based on the current state information;
[0082] The strategy information sending module 330 is configured to send the strategy information to the target train so that the target train runs according to the strategy information.
[0083] The technical solution of this embodiment is to obtain the current state information of all trains on the target line corresponding to the target train through a decision network deployed at the current station when determining that the target train stopped at the current station meets the train rescheduling conditions. The decision network is obtained through at least one round of interactive training between an initial decision network based on the DDPG algorithm and a preset train operation simulation environment. The preset train operation simulation environment includes a reward function. The reward function is implemented by calculating the overlap time between the braking time information and the acceleration time information of adjacent trains. Based on the current state information, the strategy information for the target train to leave the current station and arrive at the next station is obtained. The strategy information is sent to the target train so that the target train operates according to the strategy information. This solves the problem that the existing technology cannot adjust the train scheduling strategy in real time when delay disturbances occur in train operation. It achieves the effect of quickly and in real time determining the optimal scheduling and energy-saving plan for the train when delay disturbances occur.
[0084] Optionally, the initial decision network based on the DDPG algorithm includes an Actor network, an Actor target network, a Critic network, and a Critic target network;
[0085] The preset train operation simulation environment includes basic station information, basic train information, train time model, train energy consumption model and train operation model;
[0086] The basic site information includes site identification information, site quantity, site distribution information and distance information between adjacent sites;
[0087] The basic train information includes train identification information and train quantity;
[0088] The train time model is used to display the time information of each train running to each station;
[0089] The train energy consumption model is used to display the energy consumption information of the train from arriving at the station to leaving the station;
[0090] The train operation model is used to display the operation information of three adjacent trains at the same station.
[0091] Optionally, the preset train operation simulation environment further includes a reward function; the reward function is implemented by calculating the overlap time between the braking time and the acceleration time of adjacent trains.
[0092] Optionally, the train dispatch optimization device further includes a decision network acquisition module, which may specifically include:
[0093] A network initialization submodule, used to initialize the initial decision network;
[0094] The current second driving strategy acquisition submodule is used to use the current state of the train to be decided and scheduled to depart at the current time in the preset train operation simulation environment and the Actor network to obtain the current first driving strategy that matches the current state, and add preset noise to the current first driving strategy to obtain the current second driving strategy;
[0095] A next state and reward value acquisition submodule is used to use the train to be decided to execute the current second driving strategy to obtain the next state and reward value;
[0096] An experience data storage submodule, configured to store the current state, the current second driving strategy, the next state, and the reward value as a set of experience data in an experience playback buffer;
[0097] The decision network training submodule is used to update the initial decision network parameters using the experience data stored in the experience replay buffer to obtain the decision network.
[0098] Optionally, the decision network training submodule may include:
[0099] a target experience data acquisition unit, configured to sample the stored experience data to acquire at least one set of target experience data;
[0100] An actor network and critic network update unit is configured to perform the following operations for each set of target experience data: updating the actor network according to the current state in the currently processed target experience data to obtain a currently updated actor network; and updating the critic network according to the next state and reward value in the currently processed target experience data to obtain a currently updated critic network;
[0101] An Actor target network updating unit, configured to update the Actor target network after performing a first proportional processing on the parameters of the currently updated Actor network at a first preset interval;
[0102] The critic target network updating unit is configured to update the critic target network after performing a second proportional processing on the parameters of the currently updated critic network at every second preset interval.
[0103] Optional, Actor network and Critic network update units, specifically used for:
[0104] Obtaining a first predicted driving strategy based on a current state in the currently processed target experience data using the Actor network;
[0105] Obtaining a first Q value using the critic network based on the current state and the first predicted driving strategy;
[0106] A first loss function of the Actor network is obtained using the first Q value, and the Actor network is updated using the first loss function to obtain a current updated Actor network.
[0107] Optionally, the Actor Network and Critic Network update units can also be used to:
[0108] Obtaining a second predicted driving strategy based on a next state in the currently processed target experience data using the Actor target network;
[0109] Obtaining a second Q value using the critic target network based on the next state and the second predicted driving strategy;
[0110] The second Q value and the reward value are used to obtain a second loss function of the critic network, and the second loss function is used to update the critic network to obtain a current updated critic network.
[0111] Optionally, the current state information includes the current speed, the current position and the currently stored driving strategy; the strategy information includes the cruising speed and the duration of the stop at the next station.
[0112] Optionally, the train information acquisition module 310 may be used to:
[0113] Obtaining a current planned stop duration of the target train at the current station from the currently stored driving strategy of the target train, and obtaining a current actual stop duration of the target train at the current station;
[0114] When the current actual stop duration exceeds the current planned stop duration, it is determined that the target train meets the train rescheduling condition.
[0115] The train scheduling optimization device provided in this embodiment can execute the train scheduling optimization method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0116] Example 3
[0117] Figure 4 Schematic diagram of an electronic device 400 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers and may also represent various forms of mobile devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.
[0118] like Figure 4 As shown, the electronic device 400 includes at least one processor 401, and a memory connected to the at least one processor 401, such as a read-only memory (ROM) 402, a random access memory (RAM) 403, etc., wherein the memory stores a computer program that can be executed by the at least one processor, and the processor 401 can execute various appropriate driving strategies and processes according to the computer program stored in the read-only memory (ROM) 402 or the computer program loaded from the storage unit 408 to the random access memory (RAM) 403. Various programs and data required for the operation of the electronic device 400 can also be stored in the RAM 403. The processor 401, ROM 402 and RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.
[0119] Multiple components in the electronic device 400 are connected to the I / O interface 405, including an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a magnetic disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the electronic device 400 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.
[0120] Processor 401 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of processor 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processor, controller, microcontroller, etc. Processor 401 executes the various methods and processes described above, such as the train scheduling optimization method.
[0121] In some embodiments, the train scheduling optimization method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 400 via the ROM 402 and / or the communication unit 409. When the computer program is loaded into the RAM 403 and executed by the processor 401, one or more steps of the train scheduling optimization method described above can be performed. Alternatively, in other embodiments, the processor 401 can be configured to execute the train scheduling optimization method by any other appropriate means (e.g., by means of firmware).
[0122] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0123] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.
[0124] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0125] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0126] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0127] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.
[0128] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.
[0129] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.
Claims
1. A train dispatch optimization method, characterized in that: include: When it is determined that the target train parked at the current station meets the train rescheduling conditions, the current state information of all trains on the target line corresponding to the target train is obtained through the decision network deployed at the current station; The decision network is obtained through at least one round of interactive training between an initial decision network based on a deep deterministic policy gradient (DDPG) algorithm and a preset train operation simulation environment; a reward function is constructed within the preset train operation simulation environment; the reward function is implemented by calculating the overlap time between the braking time information and the acceleration time information of adjacent trains; Acquire, based on the current state information, strategy information for the target train to leave the current station and arrive at the next station; The strategy information is sent to the target train so that the target train operates according to the strategy information.
2. The method according to claim 1, characterized in that The initial decision network based on the DDPG algorithm includes an Actor network, an Actor target network, a Critic network, and a Critic target network; The preset train operation simulation environment includes basic station information, basic train information, train time model, train energy consumption model and train operation model; The basic site information includes site identification information, site quantity, site distribution information and distance information between adjacent sites; The basic train information includes train identification information and train quantity; The train time model is used to display the time information of each train running to each station; The train energy consumption model is used to display the energy consumption information of the train from arriving at the station to leaving the station; The train operation model is used to display the operation information of three adjacent trains at the same station.
3. The method according to claim 2, characterized in that The decision network is obtained through at least one round of interactive training between an initial decision network based on a deep deterministic policy gradient (DDPG) algorithm and a preset train operation simulation environment, including: Initializing the initial decision network; Using the current state of the train to be decided and scheduled to depart at the current moment in a preset train operation simulation environment and the Actor network, a current first driving strategy matching the current state is obtained, and preset noise is added to the current first driving strategy to obtain a current second driving strategy; Utilizing the train to be decided to execute the current second driving strategy to obtain a next state and reward value; storing the current state, the current second driving strategy, the next state, and the reward value as a set of experience data in an experience replay buffer; The initial decision network parameters are updated using the experience data stored in the experience replay buffer to obtain the decision network.
4. The method according to claim 3, characterized in that Updating the initial decision network parameters using the stored experience data in the experience replay buffer includes: Sampling the stored experience data to obtain at least one set of target experience data; For each set of target experience data, the following operations are performed: the actor network is updated according to the current state in the currently processed target experience data to obtain the currently updated actor network; the critic network is updated according to the next state and reward value in the currently processed target experience data to obtain the currently updated critic network; At every first preset interval, performing a first proportional processing on the parameters of the currently updated Actor network, and then updating the Actor target network; At every second preset interval, the parameters of the currently updated critic network are subjected to a second proportional processing, and then the critic target network is updated.
5. The method according to claim 4, characterized in that Updating the Actor network according to the current state in the currently processed target experience data to obtain a current updated Actor network, including: Obtaining a first predicted driving strategy based on a current state in the currently processed target experience data using the Actor network; Obtaining a first Q value using the critic network based on the current state and the first predicted driving strategy; A first loss function of the Actor network is obtained using the first Q value, and the Actor network is updated using the first loss function to obtain a current updated Actor network.
6. The method according to claim 4, characterized in that The critic network is updated according to the next state and reward value in the currently processed target experience data to obtain the current updated critic network, including: Obtaining a second predicted driving strategy based on a next state in the currently processed target experience data using the Actor target network; Obtaining a second Q value using the critic target network based on the next state and the second predicted driving strategy; The second Q value and the reward value are used to obtain a second loss function of the critic network, and the second loss function is used to update the critic network to obtain a current updated critic network.
7. The method according to claim 1, characterized in that The current state information includes the current speed, the current position and the currently stored driving strategy; the strategy information includes the cruising speed and the duration of the stop at the next station.
8. The method according to claim 7, characterized in that Determine whether the target train stopping at the current station meets the train rescheduling conditions, including: Obtaining a current planned stop duration of the target train at the current station from the currently stored driving strategy of the target train, and obtaining a current actual stop duration of the target train at the current station; When the current actual stop duration exceeds the current planned stop duration, it is determined that the target train meets the train rescheduling condition.
9. A train dispatch optimization device, characterized in that: include: A train information acquisition module is used to determine that a target train parked at a current station meets the train rescheduling conditions, and to obtain the current status information of all trains on the target line corresponding to the target train through a decision network deployed at the current station; The decision network is obtained through at least one round of interactive training between an initial decision network based on a deep deterministic policy gradient (DDPG) algorithm and a preset train operation simulation environment; a reward function is constructed within the preset train operation simulation environment; the reward function is implemented by calculating the overlap time between the braking time information and the acceleration time information of adjacent trains; a strategy information acquisition module, configured to acquire strategy information for the target train to leave the current station and arrive at the next station according to the current state information; The strategy information sending module is used to send the strategy information to the target train so that the target train runs according to the strategy information.
10. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the train scheduling optimization method according to any one of claims 1 to 8.
11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the train scheduling optimization method according to any one of claims 1 to 8 when executed.
