A driving scheduling method, system, device and medium based on deep reinforcement learning

Abstract

The application provides a driving scheduling method and system based on deep reinforcement learning, a device and a medium, comprising: establishing a first driving scheduling deep reinforcement learning model based on historical state information, and providing action decisions using the first driving scheduling deep reinforcement learning model; establishing a driving scheduling simulation model, and taking the historical state information and the action decisions as driving strategies of the driving scheduling simulation model, so that the driving scheduling simulation model outputs parameter values for optimizing the first driving scheduling deep reinforcement learning model according to the driving strategies; and performing interactive offline training on the first driving scheduling deep reinforcement learning model and the driving scheduling simulation model to obtain a second driving scheduling deep reinforcement learning model, and deploying the second driving scheduling deep reinforcement learning model to a steel actual production system for real-time driving scheduling. The application realizes fast training of a long-time period environment by using simulation to simulate an actual production cycle, reduces a model development period, and accelerates the online speed of the model.

Claims

1. A traffic scheduling method based on deep reinforcement learning, characterized in that, The method includes the following steps: Obtain historical status information of the actual steel production system; A first deep reinforcement learning model for train scheduling is established based on the historical state information, and action decisions are provided using the first deep reinforcement learning model for train scheduling. A train dispatching simulation model is established based on data information from the steel production site. The historical state information and the action decisions are used as the driving strategy for the train dispatching simulation model, so that the train dispatching simulation model outputs parameter values ​​for optimizing the first train dispatching deep reinforcement learning model according to the driving strategy. The first train scheduling deep reinforcement learning model and the train scheduling simulation model are interactively trained offline to obtain the second train scheduling deep reinforcement learning model. The second deep reinforcement learning model for train scheduling is deployed to the actual steel production system for real-time train scheduling. Before deploying the second deep reinforcement learning model for train scheduling to the actual steel production system, the method further includes: giving the second deep reinforcement learning model for train scheduling a scheduling action strategy, and performing feedforward verification on the train scheduling simulation model to determine the feasibility of the second deep reinforcement learning model for train scheduling.

2. The traffic scheduling method based on deep reinforcement learning according to claim 1, characterized in that, The process of interactively training the first deep reinforcement learning model for train scheduling and the train scheduling simulation model offline to obtain the second deep reinforcement learning model for train scheduling includes: Data is transmitted between the first train scheduling deep reinforcement learning model and the train scheduling simulation model using a pre-set data interface; The train dispatching simulation model operates based on the received status information of the actual steel production system and real-time or pre-input action decision information, and returns reward and penalty values ​​to the first train dispatching deep reinforcement learning model for iterative learning and training to obtain the second train dispatching deep reinforcement learning model.

3. The traffic scheduling method based on deep reinforcement learning according to claim 2, characterized in that, When the reward and penalty values ​​are returned to the first deep reinforcement learning model for iterative learning and training, the reward function of the first deep reinforcement learning model for train scheduling is calculated as follows: in, This indicates the return value of the reward function; This represents the return value of the reward function for each crane; This indicates the current number of overhead cranes in the middle of the tunnel.

4. The traffic scheduling method based on deep reinforcement learning according to claim 3, characterized in that, The formula for calculating the return value of the reward function for each crane is as follows: Among them, case (1) indicates that the first train scheduling deep reinforcement learning model has no task for the corresponding train during the execution period at the current time, and the reward / penalty value returned is 0. Case (2) indicates that at the current moment, the train corresponding to the first train dispatching deep reinforcement learning model is performing a task, and the returned reward / penalty value is... In the formula, This is the moment when the current driving task ends. It is the earliest time of the recommended time period for the current driving task. It is the first weight value; Case (3) indicates that the corresponding train has completed its task during the execution of the first train dispatching deep reinforcement learning model at the current time, and the returned reward / penalty value is... .

5. The traffic scheduling method based on deep reinforcement learning according to claim 4, characterized in that, When the returned reward / penalty value hour, The calculation formula is as follows: in, This indicates the latest recommended time for the task currently in progress. This represents the second weight value; This represents a constant value indicating the penalty for completing the task outside the recommended time period; This represents a constant value for the reward.

6. A traffic dispatching system based on deep reinforcement learning, characterized in that, The system includes: The information acquisition module is used to obtain historical status information of the actual steel production system; The action decision module is used to establish a first deep reinforcement learning model for train scheduling based on the historical state information, and to provide action decisions using the first deep reinforcement learning model for train scheduling. An optimization module is used to establish a train dispatching simulation model based on data information from the steel production site, and to use the historical state information and the action decisions as the driving strategy for the train dispatching simulation model, so that the train dispatching simulation model outputs parameter values ​​for optimizing the first train dispatching deep reinforcement learning model according to the driving strategy. The training module is used to perform interactive offline training on the first train scheduling deep reinforcement learning model and the train scheduling simulation model to obtain the second train scheduling deep reinforcement learning model. The train dispatching module is used to deploy the second train dispatching deep reinforcement learning model to the actual steel production system for real-time train dispatching. Before deploying the second train dispatching deep reinforcement learning model to the actual steel production system, the module further includes: giving the second train dispatching deep reinforcement learning model a dispatching action strategy, and performing feedforward verification on the train dispatching simulation model to determine the feasibility of the second train dispatching deep reinforcement learning model.

7. The traffic dispatching system based on deep reinforcement learning according to claim 6, characterized in that, The process by which the training module performs interactive offline training on the first deep reinforcement learning model for train scheduling and the train scheduling simulation model to obtain the second deep reinforcement learning model for train scheduling includes: Data is transmitted between the first train scheduling deep reinforcement learning model and the train scheduling simulation model using a pre-set data interface; The train dispatching simulation model operates based on the received status information of the actual steel production system and real-time or pre-input action decision information, and returns reward and penalty values ​​to the first train dispatching deep reinforcement learning model for iterative learning and training to obtain the second train dispatching deep reinforcement learning model.

8. A train dispatching device based on deep reinforcement learning, characterized in that, include: processor; and, A computer-readable medium storing instructions that, when executed by the processor, cause the device to perform the deep reinforcement learning-based traffic scheduling method as described in any one of claims 1 to 5.

9. A computer-readable medium, characterized in that, It stores instructions that are loaded by a processor and executed as described in any one of claims 1 to 5, which are based on the deep reinforcement learning-based vehicle scheduling method.

Images