Inland river port electric transfer vehicle charging scheduling method

Abstract

The invention discloses an inland river port electric transfer vehicle charging scheduling method. The method comprises the steps that a transfer vehicle task matching model is constructed with maximization of the transfer vehicle utilization rate and completion of all transfer tasks as targets; constructing a transfer vehicle charging scheduling model by taking minimization of delay, maximization of transfer vehicle efficiency and minimization of charging cost as targets; constructing an energy-traffic cooperative scheduling model through the transfer vehicle task matching model and the transfer vehicle charging scheduling model; taking the initial states of the transfer task and all the transfer vehicles as the input of an energy-traffic cooperative scheduling model, solving the energy-traffic cooperative scheduling model in which the initial states of the transfer task and all the transfer vehicles are input through a deep Q network algorithm, and generating an optimal energy-traffic cooperative scheduling scheme; and scheduling a transfer vehicle through the optimal energy-traffic cooperative scheduling scheme to complete a port transfer task. The transportation efficiency of the whole port can be improved, and the transportation cost can be reduced.

Description

Technical Field

[0001] The present invention relates to the fields of Internet big data and port dispatching, and in particular to a charging dispatching method for electric transfer vehicles in inland ports. Background Art

[0002] As energy consumption and greenhouse gas emissions continue to increase, ports are facing an increasingly urgent task of reducing carbon emissions. Among them, diesel-powered transshipment vehicles account for nearly half of port emissions. To meet this challenge, electric transshipment vehicles (EVs) have become the key to green transformation.

[0003] However, the 24-hour operation requirements of electric transfer vehicles are in stark contrast to the fragility of the port's power infrastructure. In order to balance the green transformation of ports and economic efficiency, it is particularly important to develop an optimized transfer vehicle scheduling strategy. This is not only related to the port's environmental image, but also directly related to its operational efficiency and cost. Therefore, we need to consider a variety of factors, such as the charging needs of transfer vehicles, port operation time, cargo types, etc., to develop a scientific and reasonable scheduling plan to achieve green, efficient and sustainable port operations.

[0004] Since the smooth and timely operation of transfer vehicles has a great impact on the overall operating efficiency of inland ports, the arrival delay of transfer vehicles will have an adverse impact on the operation of inland ports. At the same time, battery charging is an effective way for transfer vehicles to refuel in inland ports. Especially for transfer vehicles equipped with this function, it saves a lot of charging time and has advantages over plug-in transfer vehicles. However, transfer vehicles face energy and transportation-related constraints. In order to effectively complete the cargo transfer task, they need to visit charging stations regularly to maintain sufficient state of charge (SoC). Therefore, how to design a scheduling method that considers the dual constraints of transfer vehicles and energy fields is a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the technical problem to be solved by the present invention is: how to provide a charging scheduling method for electric transshipment vehicles in inland ports, which ensures the utilization rate of transshipment vehicles and the completion of transshipment tasks by constructing an accurate transshipment vehicle task matching model, and at the same time constructs a dynamic transshipment vehicle charging scheduling model to maximize the efficiency of transshipment vehicles, minimize delays and charging costs, and optimize energy efficiency. In addition, the method should also be able to manage the battery status, ensure that the battery is reasonably used and maintained, and support integration with the existing management system of the inland port.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] The charging scheduling method for electric transfer vehicles in inland ports includes:

[0008] S1: Obtain the port's transshipment tasks and the initial status of all transshipment vehicles;

[0009] S2: Construct a transfer vehicle task matching model with the goal of maximizing the utilization rate of transfer vehicles and completing all transfer tasks;

[0010] S3: Construct a charging scheduling model for transfer vehicles with the goal of minimizing delays, maximizing transfer vehicle efficiency, and minimizing charging costs;

[0011] S4: The transfer vehicle task matching model is used as the first-stage model, and the transfer vehicle charging scheduling model is used as the second-stage model. The energy-traffic coordinated scheduling model is constructed through the first-stage model and the second-stage model.

[0012] S5: Taking the transfer task and the initial state of all transfer vehicles as the input of the energy-traffic coordinated scheduling model, solving the energy-traffic coordinated scheduling model with the transfer task and the initial state of all transfer vehicles through the deep Q network algorithm, and generating the optimal energy-traffic coordinated scheduling scheme;

[0013] S6: Dispatch the transfer vehicles through the optimal energy-traffic coordinated scheduling plan to complete the port's transfer tasks.

[0014] Preferably, in step S2, the objective function of the transport vehicle task matching model is expressed as:

[0015]

[0016] Where: X i,v An indicator that indicates the matching of the transfer task i with the transfer vehicle v. When X i,v = 1, indicating that the transfer task i is included in the task list of the transfer vehicle v, X i,v = 0, it means that the transfer task i is not included in the task list of the transfer vehicle v; each transfer task i consists of a tuple Description, where p i is the pickup node, d i To deliver the node, is the expected task completion time; R s1.veh Represents the reward function for transporting vehicles in the first stage model.

[0017] Preferably, in step S2, the constraints of the transfer vehicle task matching model include task quantity constraints;

[0018] The task quantity constraint is expressed as:

[0019]

[0020]

[0021]

[0022] Where: X max Indicates the upper limit of the number of tasks.

[0023] Preferably, in step S3, the objective function of the transfer vehicle charging scheduling model is expressed as:

[0024]

[0025]

[0026]

[0027] Where: m represents the charging station; a represents the path between nodes; represents the cost function of the charging station; represents the no-load operation cost of the transfer vehicle; represents the cost of purchasing electricity from the grid; Indicates the unit battery level; C idle Indicates the no-load running cost per unit time.

[0028] Preferably, in step S3, the constraints of the transfer vehicle charging scheduling model include a transfer vehicle power constraint;

[0029] The power constraint of the transfer vehicle is expressed as:

[0030]

[0031]

[0032]

[0033] Where: Y i,v,c,t Indicates whether the transfer vehicle v actively completes the transfer task i at time t under the power state c; X i,v An indicator that indicates the matching of the transfer task i with the transfer vehicle v. When X i,v = 1, indicating that the transfer task i is included in the task list of the transfer vehicle v, X i,v = 0, it means that the transfer task i is not included in the task list of the transfer vehicle v; each transfer task i consists of a tuple Description, where p i is the pickup node, d i To deliver the node, The expected task completion time.

[0034] Preferably, in step S3, the constraints of the transfer vehicle charging scheduling model include transfer vehicle state constraints;

[0035] The state constraint of the transfer vehicle is expressed as:

[0036]

[0037] Where: and Indicates the number of transfer vehicles in loading, charging and idle states; The number of transfer vehicles with a battery state of c waiting at charging station n at time t; Battery n represents the total number of batteries at charging station n; represents the number of batteries with a charge state of c in charging station n at time t; Indicates the number of transfer vehicles in loading status; Indicates the number of transfer vehicles in charging status; represents the number of transfer vehicles in idle state; R represents the re-dispatching node; C represents the battery status.

[0038] Preferably, in step S3, the constraints of the transfer vehicle charging scheduling model include charging power constraints;

[0039] The charging power constraint is expressed as:

[0040]

[0041] Where: CtoP represents the energy equivalent to unit SoC.

[0042] Preferably, in step S5, the processing steps of the deep Q network algorithm are as follows:

[0043] S501: converting the energy-traffic coordinated scheduling model with the transfer task and the initial state of all transfer vehicles into a Markov model;

[0044] S502: Generate the optimal energy-traffic coordinated scheduling solution through a two-stage deep Q network algorithm combined with a Markov model.

[0045] Preferably, in step S501, the converted Markov model includes:

[0046] 1) State space, action space and reward function in the first stage

[0047]

[0048]

[0049]

[0050] Where: represents the state space, the first target action and the reward function of the first stage; i represents the transport task; d represents the date; Indicates the initial position and initial power state of the transport vehicle; EV V represents the Vth transfer vehicle; represents the reward function of the second stage; X i,v An indicator that indicates the matching of the transfer task i with the transfer vehicle v. When X i,v = 1, indicating that the transfer task i is included in the task list of the transfer vehicle v, X i,v =0, it means that the transfer task i is not included in the task list of the transfer vehicle v;

[0051] 2) State space, action space and reward function in the second stage

[0052]

[0053]

[0054]

[0055]

[0056]

[0057] Where: represents the first state space and the second state space of the second stage; represents the second target action and the third target action in the second stage; t represents time; n v,t represents the position of the transfer vehicle at time t; c v,t represents the power status of the transport vehicle at time t; i represents the transport task at time t; Indicates the charging price; represents the power status of the transfer vehicle in charging station n at time t; stationM represents the Mth charging station; SoCC represents the power status of the Cth transfer vehicle; represents the charging power of charging station n at time t; C idle Indicates the no-load operation cost per unit time; Indicates the number of transfer vehicles in idle, charging and waiting states; Y i,v,c,t Indicates whether the transfer vehicle v actively completes the transfer task i at time t under the power state c.

[0058] Preferably, in step S502, the processing steps of the deep Q network method are as follows:

[0059] S5021: Design a DQN to approximate the Q-value function and randomly initialize its weights. The DQN is used to predict the Q-value of each action based on the current state.

[0060] S5022: Create an experience replay cache to store the states, actions, rewards, and new states generated by the interaction of the matching agent, the transfer vehicle agent, and the charging station agent with the Markov model;

[0061] S5023: At each time step, it includes the following two stages:

[0062] Phase 1: The matching agent selects the action space from the first phase according to the current state of the transport vehicle. Select the first target action and in the state space of the first stage And the reward function Execute the first target action in , and observe the reward and new state of the first stage;

[0063] Phase 2: The transfer vehicle agent moves from the action space of phase 2 according to the current state of the charging station. Select the second target action and in the state space of the second stage And the reward function The second target action is performed in the first stage, and the first reward and the first new state of the first stage are observed; then the charging station agent selects from the action space of the second stage according to the current state of the transfer vehicle's power state Select the third target action and in the state space of the second stage And the reward function Execute the third target action in the first stage and observe the second reward and the second new state in the first stage;

[0064] S5024: storing the current state, first target action, reward and new state of the transport vehicle in the first stage, the current state, second target action, first reward and first new state of the charging station in the second stage, and the current state, third target action, second reward and second new state of the transport vehicle's power state as a set of experience in the experience playback cache;

[0065] S5025: randomly extract a batch of experiences from the experience replay cache to train the DQN, and update the weight of the DQN by minimizing the error between the predicted Q value and the target Q value;

[0066] The loss function when training DQN is as follows:

[0067] Loss=(r+γmaxQ target (s′,a)-Q current (s,a)) 2 ;

[0068] Where: r represents the current reward; γ represents the decay coefficient; maxQ target (s′, a) represents the target maximum Q value under a given state s and action a; Q current (s,a) represents the Q value currently predicted by the model given state s and action a;

[0069] S5026: Repeat steps S5023 to S5025. Through continuous iteration and optimization, DQN will gradually learn how to make the best decision in the energy-traffic coordinated scheduling model to maximize the long-term cumulative reward.

[0070] S5027: Input the current state of the transfer vehicle into the trained DQN and output the optimal energy-traffic coordinated scheduling solution.

[0071] The present invention also discloses a charging dispatching system for electric transport vehicles in inland ports, which is implemented based on the charging dispatching method for electric transport vehicles in inland ports of the present invention; the system comprises:

[0072] The acquisition module is used to obtain the port's transshipment tasks and the initial status of all transshipment vehicles;

[0073] The first model building module is used to build a transfer vehicle task matching model with the goal of maximizing the utilization rate of transfer vehicles and completing all transfer tasks;

[0074] The second model building module is used to build a transfer vehicle charging scheduling model with the goals of minimizing delays, maximizing transfer vehicle efficiency and minimizing charging costs;

[0075] The third model building module is used to use the transfer vehicle task matching model as the first stage model and the transfer vehicle charging scheduling model as the second stage model, and to build an energy-traffic coordinated scheduling model through the first stage model and the second stage model;

[0076] A calculation module is used to take the transfer task and the initial state of all transfer vehicles as the input of the energy-traffic coordinated scheduling model, solve the energy-traffic coordinated scheduling model with the transfer task and the initial state of all transfer vehicles as the input through the deep Q network algorithm, and generate the optimal energy-traffic coordinated scheduling plan;

[0077] The execution module is used to dispatch the transfer vehicles through the optimal energy-traffic coordinated scheduling plan to complete the port's transfer tasks.

[0078] Preferably, in the first model building module, the objective function of the transfer vehicle task matching model is expressed as:

[0079]

[0080] Where: Xi,v An indicator that indicates the matching of the transfer task i with the transfer vehicle v. When X i,v = 1, indicating that the transfer task i is included in the task list of the transfer vehicle v, X i,v = 0, it means that the transfer task i is not included in the task list of the transfer vehicle v; each transfer task i consists of a tuple Description, where p i is the pickup node, d i To deliver the node, is the expected task completion time; R s1.veh Represents the reward function for transporting vehicles in the first stage model.

[0081] Preferably, in the first model building module, the constraints of the transfer vehicle task matching model include task quantity constraints;

[0082] The task quantity constraint is expressed as:

[0083]

[0084]

[0085]

[0086] Where: X max Indicates the upper limit of the number of tasks.

[0087] Preferably, in the second model building module, the objective function of the transfer vehicle charging scheduling model is expressed as:

[0088]

[0089]

[0090]

[0091] Where: m represents the charging station; a represents the path between nodes; represents the cost function of the charging station; represents the no-load operation cost of the transfer vehicle; represents the cost of purchasing electricity from the grid; Indicates the unit battery level; C idle Indicates the no-load running cost per unit time.

[0092] Preferably, in the second model building module, the constraints of the transfer vehicle charging scheduling model include a transfer vehicle power constraint;

[0093] The power constraint of the transfer vehicle is expressed as:

[0094]

[0095]

[0096]

[0097] Where: Y i,v,c,t Indicates whether the transfer vehicle v actively completes the transfer task i at time t under the power state c; X i,v An indicator that indicates the matching of the transfer task i with the transfer vehicle v. When X i,v = 1, indicating that the transfer task i is included in the task list of the transfer vehicle v, X i,v = 0, it means that the transfer task i is not included in the task list of the transfer vehicle v; each transfer task i consists of a tuple Description, where p i is the pickup node, d i To deliver the node, The expected task completion time.

[0098] Preferably, in the second model building module, the constraints of the transfer vehicle charging scheduling model include transfer vehicle state constraints;

[0099] The state constraint of the transfer vehicle is expressed as:

[0100]

[0101]

[0102]

[0103] Where: and Indicates the number of transfer vehicles in loading, charging and idle states; The number of transfer vehicles with a battery state of c waiting at charging station n at time t; Battery n represents the total number of batteries at charging station n; represents the number of batteries with a charge state of c in charging station n at time t; Indicates the number of transfer vehicles in loading status; Indicates the number of transfer vehicles in charging status; represents the number of transfer vehicles in idle state; R represents the re-dispatching node; C represents the battery status.

[0104] Preferably, in the second model building module, the constraints of the transfer vehicle charging scheduling model include transfer vehicle state constraints;

[0105] The state constraint of the transfer vehicle is expressed as:

[0106]

[0107]

[0108]

[0109] Where: and Indicates the number of transfer vehicles in loading, charging and idle states; The number of transfer vehicles with a battery state of c waiting at charging station n at time t; Battery n represents the total number of batteries at charging station n; represents the number of batteries with a charge state of c in charging station n at time t; Indicates the number of transfer vehicles in loading status; Indicates the number of transfer vehicles in charging status; represents the number of transfer vehicles in idle state; R represents the re-dispatching node; C represents the battery status.

[0110] Preferably, in the second model building module, the constraints of the transfer vehicle charging scheduling model include charging power constraints;

[0111] The charging power constraint is expressed as:

[0112]

[0113] Where: CtoP represents the energy equivalent to unit SoC.

[0114] Preferably, in the calculation module, the processing steps of the deep Q network algorithm are as follows:

[0115] Step 1: Convert the energy-traffic coordinated scheduling model with the initial state of the transfer task and all transfer vehicles into a Markov model;

[0116] Step 2: Generate the optimal energy-traffic coordinated scheduling solution through a two-stage deep Q-network algorithm combined with the Markov model.

[0117] Preferably, in step 1 of the calculation module, the converted Markov model includes:

[0118] 1) State space, action space and reward function in the first stage

[0119]

[0120]

[0121]

[0122] Where: represents the state space, the first target action and the reward function of the first stage; i represents the transport task; d represents the date; Indicates the initial position and initial power state of the transport vehicle; EV V represents the Vth transfer vehicle; R s1.veh represents the reward function for transporting vehicles in the first stage model; represents the reward function of the second stage; X i,v An indicator that indicates the matching of the transfer task i with the transfer vehicle v. When X i,v = 1, indicating that the transfer task i is included in the task list of the transfer vehicle v, X i,v =0, it means that the transfer task i is not included in the task list of the transfer vehicle v;

[0123] 2) State space, action space and reward function in the second stage

[0124]

[0125]

[0126]

[0127]

[0128]

[0129] Where: represents the first state space and the second state space of the second stage; represents the second target action and the third target action in the second stage; t represents time; n v,t represents the position of the transfer vehicle at time t; c v,t represents the power status of the transport vehicle at time t; i represents the transport task; represents the cost of purchasing electricity from the grid; represents the power status of the transfer vehicle in charging station n at time t; stationM represents the Mth charging station; SoCC represents the power status of the Cth transfer vehicle; Indicates the unit battery level; C idle Indicates the no-load operation cost per unit time; Indicates the number of transfer vehicles in idle, charging and waiting states; Y i,v,c,t Indicates whether the transfer vehicle v actively completes the transfer task i at time t under the power state c.

[0130] Preferably, in step 2 of the calculation module, the processing steps of the deep Q network method are as follows:

[0131] 1) Design a DQN to approximate the Q-value function and randomly initialize its weights. The DQN is used to predict the Q-value of each action based on the current state;

[0132] 2) Create an experience replay cache to store the states, actions, rewards, and new states generated by the matching agent, transfer vehicle agent, and charging station agent interacting with the Markov model;

[0133] 3) At each time step, it includes the following two stages:

[0134] Phase 1: The matching agent selects the action space from the first phase according to the current state of the transport vehicle. Select the first target action and in the state space of the first stage And the reward function Execute the first target action in , and observe the reward and new state of the first stage;

[0135] Phase 2: The transfer vehicle agent moves from the action space of phase 2 according to the current state of the charging station. Select the second target action and in the state space of the second stage And the reward function Execute the second target action in the second stage, observe the first reward and the first new state of the second stage; the charging station agent selects the action space of the second stage according to the current state of the transfer vehicle's power state Select the third target action and in the state space of the second stage And the reward function Execute the third target action in the process, and observe the second reward and the second new state in the second stage;

[0136] 4) The current state, first target action, reward and new state of the transport vehicle in the first stage, the current state, second target action, first reward and first new state of the charging station in the second stage, and the current state, third target action, second reward and second new state of the transport vehicle's power state are stored as a set of experience in the experience playback cache;

[0137] 5) Randomly extract a batch of experiences from the experience replay cache to train the DQN, and update the weights of the DQN by minimizing the error between the predicted Q value and the target Q value;

[0138] The loss function when training DQN is as follows:

[0139] Loss=(r+γmaxQ target (s′,a)-Q current (s,a)) 2 ;

[0140] Where: r represents the current reward; γ represents the decay coefficient; maxQ target (s′, a) represents the target maximum Q value under a given state s and action a; Q current (s,a) represents the Q value currently predicted by the model given state s and action a;

[0141] 6) Repeat steps 3) to 5). Through continuous iteration and optimization, DQN will gradually learn how to make the best decisions in the energy-traffic collaborative scheduling model to maximize the long-term cumulative rewards.

[0142] 7) Input the current state of the transfer vehicle into the trained DQN and output the optimal energy-traffic coordinated scheduling plan.

[0143] The present invention also discloses a computer device, comprising: one or more processors;

[0144] The processor is used to store one or more programs;

[0145] When the one or more programs are executed by the one or more processors, the charging scheduling method for inland port electric transport vehicles of the present invention is implemented.

[0146] The present invention also discloses a computer-readable storage medium, characterized in that a computer program is stored thereon, and when the computer program is executed, the charging scheduling method for electric transfer vehicles in inland ports of the present invention is implemented. Compared with the prior art, the charging scheduling method for electric transfer vehicles in inland ports of the present invention has the following beneficial effects:

[0147] The present invention constructs a transfer vehicle task matching model with the goal of maximizing the utilization rate of transfer vehicles and completing all transfer tasks. First, through reasonable task allocation, it is ensured that each electric transfer vehicle can be fully utilized, reducing the idle or waiting time of transfer vehicles, thereby improving the overall transportation efficiency of the port. Secondly, optimizing the task matching model can ensure that all transfer tasks can be completed within the specified time, avoiding task delays caused by improper allocation of transfer vehicles, thereby ensuring the completion of port transfer tasks. Finally, by improving the utilization rate of transfer vehicles and reducing idle time, the energy consumption and labor costs in the transportation process can be reduced, thereby reducing the overall operating costs of the port.

[0148] The present invention constructs a charging scheduling model for transfer vehicles with the goal of minimizing delays, maximizing transfer vehicle efficiency, and minimizing charging costs. First, through reasonable scheduling arrangements, it is ensured that electric transfer vehicles can obtain battery charging in time when needed, avoiding transportation delays due to insufficient battery power, thereby ensuring the completion of port transfer tasks. Secondly, optimizing the scheduling model can ensure that transfer vehicles complete transfer tasks in the shortest time, improve transportation efficiency, reduce waiting and idle time, and further improve the overall transportation efficiency of the port. Finally, by reasonably arranging the charging time and the use of charging stations, the charging cost can be reduced, and the increase in operating costs due to excessively high charging costs can be avoided, thereby reducing the overall transportation cost of the port.

[0149] The present invention constructs an energy-transportation collaborative scheduling model using the transfer vehicle task matching model as the first-stage model and the transfer vehicle charging scheduling model as the second-stage model. First, the two models are constructed in stages and integrated into a complete energy-transportation collaborative scheduling model, which can achieve global optimization from task allocation to transportation scheduling, ensuring the efficiency and coordination of the entire transportation process. Secondly, constructing the model in stages can make the decision-making at each stage clearer and more efficient, reduce redundancy and repetition in the decision-making process, and improve the overall decision-making efficiency. Finally, through global optimization and efficient decision-making, the stability of the entire energy-transportation collaborative system can be enhanced, and the risk of system collapse caused by problems in a single link can be reduced.

[0150] The present invention solves the energy-traffic collaborative scheduling model through a deep Q-network algorithm. First, the deep Q-network algorithm can handle problems in high-dimensional state space and action space, and has efficient solving capabilities for complex energy-traffic collaborative scheduling models. Secondly, through the combination of deep learning and reinforcement learning, the deep Q-network algorithm can learn the optimal scheduling strategy and provide the optimal scheduling solution for inland port electric transfer vehicles and battery charging. Finally, the deep Q-network algorithm has the ability to learn and adapt to environmental changes in real time, and can dynamically adjust the scheduling solution according to actual conditions to ensure the real-time and robustness of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0151] In order to make the purpose, technical solution and advantages of the invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0152] Figure 1 Logical block diagram of the charging scheduling method for electric transshipment vehicles in inland ports. DETAILED DESCRIPTION

[0153] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0154] It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. In the description of the present invention, it should be noted that the orientation or position relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, or the orientation or position relationship in which the invention product is usually placed when used, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance. In addition, the terms "horizontal", "vertical", etc. do not mean that the components are required to be absolutely horizontal or suspended, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted. In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0155] The following is a further detailed description through specific implementation methods:

[0156] Embodiment 1:

[0157] This embodiment discloses a charging scheduling method for electric transfer vehicles in an inland port.

[0158] like Figure 1As shown, the charging scheduling method for electric transfer vehicles in inland ports includes:

[0159] S1: Obtain the port's transshipment tasks and the initial status of all transshipment vehicles;

[0160] S2: Construct a transfer vehicle task matching model with the goal of maximizing the utilization rate of transfer vehicles and completing all transfer tasks;

[0161] S3: Construct a charging scheduling model for transfer vehicles with the goal of minimizing delays, maximizing transfer vehicle efficiency, and minimizing charging costs;

[0162] S4: The transfer vehicle task matching model is used as the first-stage model, and the transfer vehicle charging scheduling model is used as the second-stage model. The energy-traffic coordinated scheduling model is constructed through the first-stage model and the second-stage model.

[0163] S5: Taking the transfer task and the initial state of all transfer vehicles as the input of the energy-traffic coordinated scheduling model, solving the energy-traffic coordinated scheduling model with the transfer task and the initial state of all transfer vehicles through the deep Q network algorithm, and generating the optimal energy-traffic coordinated scheduling scheme;

[0164] S6: Dispatch the transfer vehicles through the optimal energy-traffic coordinated scheduling plan to complete the port's transfer tasks.

[0165] The present invention constructs a transfer vehicle task matching model with the goal of maximizing the utilization rate of transfer vehicles and completing all transfer tasks. First, through reasonable task allocation, it is ensured that each electric transfer vehicle can be fully utilized, reducing the idle or waiting time of transfer vehicles, thereby improving the overall transportation efficiency of the port. Secondly, optimizing the task matching model can ensure that all transfer tasks can be completed within the specified time, avoiding task delays caused by improper allocation of transfer vehicles, thereby ensuring the completion of port transfer tasks. Finally, by improving the utilization rate of transfer vehicles and reducing idle time, the energy consumption and labor costs in the transportation process can be reduced, thereby reducing the overall operating costs of the port.

[0166] The present invention constructs a charging scheduling model for transfer vehicles with the goal of minimizing delays, maximizing transfer vehicle efficiency, and minimizing charging costs. First, through reasonable scheduling arrangements, it is ensured that electric transfer vehicles can obtain battery charging in time when needed, avoiding transportation delays due to insufficient battery power, thereby ensuring the completion of port transfer tasks. Secondly, optimizing the scheduling model can ensure that transfer vehicles complete transfer tasks in the shortest time, improve transportation efficiency, reduce waiting and idle time, and further improve the overall transportation efficiency of the port. Finally, by reasonably arranging the charging time and the use of charging stations, the charging cost can be reduced, and the increase in operating costs due to excessively high charging costs can be avoided, thereby reducing the overall transportation cost of the port.

[0167] The present invention constructs an energy-transportation collaborative scheduling model using the transfer vehicle task matching model as the first-stage model and the transfer vehicle charging scheduling model as the second-stage model. First, the two models are constructed in stages and integrated into a complete energy-transportation collaborative scheduling model, which can achieve global optimization from task allocation to transportation scheduling, ensuring the efficiency and coordination of the entire transportation process. Secondly, constructing the model in stages can make the decision-making at each stage clearer and more efficient, reduce redundancy and repetition in the decision-making process, and improve the overall decision-making efficiency. Finally, through global optimization and efficient decision-making, the stability of the entire energy-transportation collaborative system can be enhanced, and the risk of system collapse caused by problems in a single link can be reduced.

[0168] The present invention solves the energy-traffic collaborative scheduling model through a deep Q-network algorithm. First, the deep Q-network algorithm can handle problems in high-dimensional state space and action space, and has efficient solving capabilities for complex energy-traffic collaborative scheduling models. Secondly, through the combination of deep learning and reinforcement learning, the deep Q-network algorithm can learn the optimal scheduling strategy and provide the optimal scheduling solution for inland port electric transfer vehicles and battery charging. Finally, the deep Q-network algorithm has the ability to learn and adapt to environmental changes in real time, and can dynamically adjust the scheduling solution according to actual conditions to ensure the real-time and robustness of the system.

[0169] In the specific implementation process, the objective function of the transfer vehicle task matching model is expressed as:

[0170]

[0171] Where: X i,v An indicator that indicates the matching of the transfer task i with the transfer vehicle v. When X i,v = 1, indicating that the transfer task i is included in the task list of the transfer vehicle v, X i,v = 0, it means that the transfer task i is not included in the task list of the transfer vehicle v; each transfer task i consists of a tuple Description, where p i is the pickup node, d i To deliver the node, is the expected task completion time; R s1.veh Represents the reward function for transporting vehicles in the first stage model.

[0172] The constraints of the transfer vehicle task matching model include task quantity constraints;

[0173] The formula for task quantity constraint is expressed as:

[0174]

[0175]

[0176]

[0177] Where: X max Indicates the upper limit of the number of tasks.

[0178] It is required that each task can only be assigned to one transfer vehicle.

[0179] Constraints An upper limit is set for the total number of tasks, denoted as X max .

[0180] Constraints Defining decision variables as binary values ​​reflects the discrete nature of their assignments.

[0181] In the specific implementation process, the objective function of the transfer vehicle charging scheduling model is expressed as:

[0182]

[0183]

[0184]

[0185]

[0186] Where: m represents the charging station; a represents the path between nodes; represents the cost function of the charging station; represents the no-load operation cost of the transfer vehicle; represents the cost of purchasing electricity from the grid; Indicates the unit battery level; C idle Indicates the no-load running cost per unit time.

[0187] Objective Function represents minimizing the energy-transportation system cost of the inland port.

[0188] Calculate electricity purchase costs.

[0189] Designed to reduce idle time to optimize the utilization of transfer vehicles.

[0190] Constraints of the charging scheduling model for transfer vehicles

[0191] 1. In order to ensure the balance of transfer vehicles between terminal cranes and yard nodes, a transfer vehicle power constraint is constructed.

[0192] In this example, the process of generating a list of transfer vehicle tasks relies on the best match between transfer vehicles and tasks. Once these task lists are determined, the second phase involves creating a transfer vehicle scheduling and energy management model. In this phase, key decisions include when to visit a specific charging station or proceed to the next task assignment. To formalize this process, we use a binary variable Y i,v,c,t , which indicates whether the transfer vehicle v is actively engaged in task i at time t in the state of charge c. To facilitate this formulation, the state of charge (SoC) in the model is considered as a discrete value. It should be mentioned that this assumption has little impact on the results. Therefore, X i,v and Y i,v,c,t The relationship between them is shown as follows, summarizing their basic connections in the model.

[0193] The constraints of the transfer vehicle charging scheduling model include the transfer vehicle power constraints;

[0194] The power constraint of the transfer vehicle is expressed as:

[0195]

[0196]

[0197]

[0198] Where: Y i,v,c,t Indicates whether the transfer vehicle v actively completes the transfer task i at time t under the power state c; X i,v An indicator that indicates the matching of the transfer task i with the transfer vehicle v. When X i,v = 1, indicating that the transfer task i is included in the task list of the transfer vehicle v, X i,v = 0, it means that the transfer task i is not included in the task list of the transfer vehicle v; each transfer task i consists of a tuple Description, where p i is the pickup node, d i To deliver the node, The expected task completion time.

[0199] It indicates whether the transfer vehicle v with the state of charge (SoC) c at time t is assigned to task j, forming the core decision point for each transfer vehicle to participate in the task.

[0200] The number of transport vehicles with a specific Socc traveling on road a at time t is defined, revealing the spatial distribution of transport vehicles with different energy levels. This information is crucial for optimizing path planning and resource allocation in an operational environment.

[0201] By requiring tasks to be completed within a predetermined time frame The equations together support the fundamental aspects of the model, addressing the issues of task allocation, transfer vehicle energy management, and time-sensitive task completion in a dynamic operational environment.

[0202] Operational harmony of cargo loading, unloading and storage is maintained by limiting the power consumption of transfer vehicles.

[0203] 2. In order to emphasize the balance of transfer vehicles at the charging station node, the transfer vehicle state constraints are constructed.

[0204] In this example, constraints on transfer vehicles on the transport road are proposed. Three variables ( and ) represents the number of transfer vehicles in loading, charging, and idle states. Since the battery supply at the battery charging station is limited, the charging transfer vehicle may need to wait for the battery to become available. To solve this problem, we introduce the variable Represents the number of transfer vehicles with a charge state of c waiting at charging station n at time t. n represents the total number of batteries in charging station n, covering the overall energy resources. Representing the number of batteries with state of charge c in charging station n at time t helps to achieve accurate energy management and allocation within the operational framework.

[0205] Therefore, the constraints of the transfer vehicle charging scheduling model include the transfer vehicle state constraints;

[0206] The state constraint of the transfer vehicle is expressed as:

[0207]

[0208]

[0209]

[0210] Where: and Indicates the number of transfer vehicles in loading, charging and idle states; The number of transfer vehicles with a battery state of c waiting at charging station n at time t; Battery n represents the total number of batteries at charging station n, covering the overall energy resources; Represents the number of batteries with a charge state of c in charging station n at time t, which helps to achieve accurate energy management and allocation within the operational framework; Indicates the number of transfer vehicles in charging status; represents the number of transfer vehicles in idle state; R represents the re-dispatching node; C represents the battery status.

[0211] The state constraints of the transport vehicles ensure the fairness of resource allocation and simplify the operation process. 3. In order to maintain the balance of batteries with different power states, the charging power constraints are constructed.

[0212] In this embodiment, the constraints of the transfer vehicle charging scheduling model include charging power constraints;

[0213] The charging power constraint is expressed as:

[0214]

[0215] This constraint defines the charging power of charging station n at time t. The state of charge (SoC) is calculated by subtracting the cumulative required SoC of the outbound transfer vehicle (expressed as all a belongs to the set Ωn C.head of The sum of all c) and the cumulative required SoC of the waiting transfer vehicles (expressed as the sum of all c The difference is further determined by subtracting the cumulative total SoC of the arriving transfer vehicles (expressed as all a belonging to the set Ωn C.tail of The sum of the vehicles waiting for transfer ) is adjusted, where CtoP represents the energy equivalent to unit SoC.

[0216] The charging power constraint aligns the total number of batteries at charging station n with their availability, thus optimizing energy management and overall system efficiency.

[0217] The above three constraints together promote the smooth operation and effectiveness of the system.

[0218] In the specific implementation process, the processing steps of the deep Q network algorithm are as follows:

[0219] S501: converting the (mixed integer nonlinear) energy-traffic coordinated scheduling model with the transfer task and the initial states of all transfer vehicles into a Markov model;

[0220] S502: Generate the optimal energy-traffic coordinated scheduling solution through a two-stage deep Q network algorithm combined with a Markov model.

[0221] 1. The converted Markov model includes:

[0222] 1) State space, action space and reward function in the first stage

[0223]

[0224]

[0225]

[0226] Where: represents the state space, the first target action and the reward function of the first stage; i represents the transport task; d represents the date; Indicates the initial position and initial power state of the transport vehicle; EV V represents the Vth transfer vehicle; R s1.veh represents the reward function for transporting vehicles in the first stage model; represents the reward function of the second stage; X i,v An indicator that indicates the matching of the transfer task i with the transfer vehicle v. When X i,v = 1, indicating that the transfer task i is included in the task list of the transfer vehicle v, X i,v =0, it means that the transfer task i is not included in the task list of the transfer vehicle v;

[0227] 2) State space, action space and reward function in the second stage

[0228]

[0229]

[0230]

[0231]

[0232]

[0233] Where: represents the first state space and the second state space of the second stage; represents the second target action and the third target action in the second stage; t represents time; n v,t represents the position of the transfer vehicle at time t; c v,t represents the power status of the transport vehicle at time t; i represents the transport task; C t grid represents the cost of purchasing electricity from the grid; represents the power status of the transfer vehicle in charging station n at time t; stationM represents the Mth charging station; SoCC represents the power status of the Cth transfer vehicle; Indicates the unit battery level; C idle Indicates the no-load operation cost per unit time; Indicates the number of transfer vehicles in idle, charging and waiting states; Yi,v,c,t Indicates whether the transfer vehicle v actively completes the transfer task i at time t under the power state c.

[0234] In the first phase, the matching agent selects a transfer vehicle based on the current state, making an anticipatory choice from the set of possible actions for each task. The state contains information such as the specific date d, task characteristics, Initial position of the transfer vehicle and initial And other information.

[0235] In the second phase, the transfer vehicle agent makes decisions from the action space based on its observable environment state, which includes factors such as date, current time, transfer vehicle location, SoC, task list, and electricity price.

[0236] After the transfer vehicle agent makes a decision, the charging station agent determines the best electricity purchase from the action space based on the environmental state it perceives. The state includes information such as date d, time t, and the location n of the transfer vehicle at that time. v,t 、SoCc of the instant transport vehicle v,t , the battery status of charging station n (expressed as ) and current prices The agent will transfer states through the state transfer equation through a sequential decision-making process.

[0237] In the first stage, the matching agent’s reward includes the transfer vehicle utilization reward.

[0238] In the second stage, the reward functions for the transfer vehicle and charging station agents include three key elements: electricity procurement cost, penalty for idle travel time, and penalty for task delay. Together, these components shape the behavior of the agents, promote efficient energy management, reduce inefficient transfer vehicle movements, and emphasize the importance of timely task completion within the operational framework.

[0239] 2. By integrating Q-learning and deep learning, the DQN method has shown remarkable effectiveness in solving scheduling problems based on discrete actions. The output of the DQN model provides Q values ​​for all feasible discrete actions based on the current state as input. The two-stage DQN system shows the interaction of three agents in a port environment. Each agent's actions have the potential to affect the state information of the environment, which can also serve as input data for other agents.

[0240] In the first phase, the matching agent can select |V| optimal actions, each associated with a Q-value embedded in its neural network. Similarly, in the second phase, the transfer vehicle agent and the charging station agent work in their own decision spaces. The decision set of the transfer vehicle agent includes |M|+1 optimal actions, each associated with its Q-value, while the charging station agent uses |C| optimal actions and their corresponding Q-values. This flexible action selection system enables the agents to make intelligent decisions and adapt to the changing conditions and demands of the operating environment.

[0241] Unlike supervised learning methods, DQN does not provide labeled samples for neural network learning. Therefore, the present invention creates two neural networks specifically for the DQN method: a target network and a current network. The target network is used as a reference to guide algorithm updates, similar to the role of ground truth in supervised learning. The current network and the target network share the same parameters and structure because they are initially initialized with the same values. As training progresses, the current network is updated faster than the target network. The mean square error between the target Q value and the predicted Q value is called the loss function. The parameters of the current network will be modified according to the calculated gradient of the loss function. When the outputs of the target network and the current network are close enough, the learning process has converged.

[0242] Specifically, the processing steps of the Q network method are as follows:

[0243] S5021: Design a DQN (deep Q network, which combines a deep neural network (DNN) and a reinforcement learning algorithm called Q-learning) to approximate the Q-value function and randomly initialize its weights. DQN is used to predict the Q value of each action based on the current state.

[0244] S5022: Create an experience replay cache to store the states, actions, rewards, and new states generated by the interaction of the matching agent, the transfer vehicle agent, and the charging station agent with the Markov model;

[0245] S5023: At each time step, it includes the following two stages:

[0246] Phase 1: The matching agent selects the action space from the first phase according to the current state of the transport vehicle. Select the first target action and in the state space of the first stage And the reward function Execute the first target action in , and observe the reward and new state of the first stage;

[0247] Phase 2: The transfer vehicle agent moves from the action space of phase 2 according to the current state of the charging station. Select the second target action and in the state space of the second stage And the reward function The second target action is performed in the first stage, and the first reward and the first new state of the first stage are observed; then the charging station agent selects from the action space of the second stage according to the current state of the transfer vehicle's power state Select the third target action and in the state space of the second stage And the reward function Execute the third target action in the first stage and observe the second reward and the second new state in the first stage;

[0248] S5024: storing the current state, first target action, reward and new state of the transport vehicle in the first stage, the current state, second target action, first reward and first new state of the charging station in the second stage, and the current state, third target action, second reward and second new state of the transport vehicle's power state as a set of experience in the experience playback cache;

[0249] S5025: randomly extract a batch of experiences from the experience replay cache to train the DQN, and update the weight of the DQN by minimizing the error between the predicted Q value and the target Q value;

[0250] The loss function when training DQN is as follows:

[0251] Loss=(r+γmaxQ target (s′,a)-Q current (s,a)) 2 ;

[0252] Where: r represents the current reward; γ represents the decay coefficient; maxQ target (s′, a) represents the target maximum Q value under a given state s and action a; Q current (s,a) represents the Q value currently predicted by the model given state s and action a;

[0253] S5026: Repeat steps S5023 to S5025. Through continuous iteration and optimization, DQN will gradually learn how to make the best decision in the energy-traffic coordinated scheduling model to maximize the long-term cumulative reward.

[0254] S5027: Input the current state of the transfer vehicle into the trained DQN and output the optimal energy-traffic coordinated scheduling solution.

[0255] In this embodiment, after DQN is trained, the current state of the environment, including the transport task and the initial state of all vehicles, is input, and then a set of action values ​​(Q values) for the current state are output through DQN. Each action value corresponds to a possible action (i.e., energy-traffic coordinated scheduling scheme), which represents the expected cumulative reward obtained by executing the action in this state.

[0256] The following is a specific example of evaluation using the method of the present invention.

[0257] The method of the present invention is verified by an actual calculation example of a port, and is compared with the traditional mathematical programming method as follows:

[0258] Table 1 Comparison of computational performance between DQN and mathematical programming

[0259]

[0260] After training, the DQN reinforcement learning agent uses one week of real data to improve its (electric transshipment) EV scheduling strategy at the port. Notably, as shown in Table 1, the DQN strategy outperforms the mathematical programming in terms of idle driving rate and electricity procurement cost. This highlights the effectiveness of DQN in reducing idle driving and minimizing energy procurement expenses, making it a valuable choice for system optimization. The mathematical programming faces the problem of matching binary variables X. i,v [300*20] creates a high-dimensional search space, which makes it difficult to find the optimal solution with limited computing resources. To address this issue, we use a preprocessing method to reduce the search space, taking into account the details of the job, such as the starting and ending nodes. However, the computational results show the superior performance of DQN. In addition, the excellent speed of the DQN strategy, only 0.06 minutes, is attributed to its efficient neural network, which is able to quickly find the optimal solution, in contrast to the inherent complexity and nonlinear nature of mathematical programming.

[0261] Embodiment 2:

[0262] This embodiment discloses a charging scheduling system for electric transfer vehicles in an inland port, which is implemented based on the charging scheduling method for electric transfer vehicles in an inland port in the first embodiment.

[0263] The charging dispatching system for electric transfer vehicles in inland ports includes:

[0264] The acquisition module is used to obtain the port's transshipment tasks and the initial status of all transshipment vehicles;

[0265] The first model building module is used to build a transfer vehicle task matching model with the goal of maximizing the utilization rate of transfer vehicles and completing all transfer tasks;

[0266] The second model building module is used to build a transfer vehicle charging scheduling model with the goals of minimizing delays, maximizing transfer vehicle efficiency and minimizing charging costs;

[0267] The third model building module is used to use the transfer vehicle task matching model as the first stage model and the transfer vehicle charging scheduling model as the second stage model, and to build an energy-traffic coordinated scheduling model through the first stage model and the second stage model;

[0268] A calculation module is used to take the transfer task and the initial state of all transfer vehicles as the input of the energy-traffic coordinated scheduling model, solve the energy-traffic coordinated scheduling model with the transfer task and the initial state of all transfer vehicles as the input through the deep Q network algorithm, and generate the optimal energy-traffic coordinated scheduling plan;

[0269] The execution module is used to dispatch the transfer vehicles through the optimal energy-traffic coordinated scheduling plan to complete the port's transfer tasks.

[0270] In this embodiment, in the first model building module, the objective function of the transfer vehicle task matching model is expressed as:

[0271]

[0272] Where: X i,v An indicator that indicates the matching of the transfer task i with the transfer vehicle v. When X i,v = 1, indicating that the transfer task i is included in the task list of the transfer vehicle v, X i,v = 0, it means that the transfer task i is not included in the task list of the transfer vehicle v; each transfer task i consists of a tuple Description, where p i is the pickup node, d i To deliver the node, is the expected task completion time; R s1.veh Represents the reward function for transporting vehicles in the first stage model.

[0273] In this embodiment, in the first model building module, the constraints of the transfer vehicle task matching model include task quantity constraints;

[0274] The task quantity constraint is expressed as:

[0275]

[0276]

[0277]

[0278] Where: X max Indicates the upper limit of the number of tasks.

[0279] In this embodiment, in the second model building module, the objective function of the transfer vehicle charging scheduling model is expressed as:

[0280]

[0281]

[0282]

[0283] Where: m represents the charging station; a represents the path between nodes; represents the cost function of the charging station; represents the no-load operation cost of the transfer vehicle; represents the cost of purchasing electricity from the grid; Indicates the unit battery level; C idle Indicates the no-load running cost per unit time.

[0284] In this embodiment, in the second model building module, the constraints of the transfer vehicle charging scheduling model include the transfer vehicle power constraints;

[0285] The power constraint of the transfer vehicle is expressed as:

[0286]

[0287]

[0288]

[0289] Where: Y i,v,c,t Indicates whether the transfer vehicle v actively completes the transfer task i at time t under the power state c; X i,v An indicator that indicates the matching of the transfer task i with the transfer vehicle v. When X i,v = 1, indicating that the transfer task i is included in the task list of the transfer vehicle v, X i,v = 0, it means that the transfer task i is not included in the task list of the transfer vehicle v; each transfer task i consists of a tuple Description, where p i is the pickup node, d i To deliver the node, The expected task completion time.

[0290] In this embodiment, in the second model building module, the constraints of the transfer vehicle charging scheduling model include transfer vehicle state constraints;

[0291] The state constraint of the transfer vehicle is expressed as:

[0292]

[0293]

[0294]

[0295] Where: and Indicates the number of transfer vehicles in loading, charging and idle states; The number of transfer vehicles with a battery state of c waiting at charging station n at time t; Battery n represents the total number of batteries at charging station n; represents the number of batteries with a charge state of c in charging station n at time t; Indicates the number of transfer vehicles in loading status; Indicates the number of transfer vehicles in charging status; represents the number of transfer vehicles in idle state; R represents the re-dispatching node; C represents the battery status.

[0296] In this embodiment, in the second model building module, the constraints of the transfer vehicle charging scheduling model include transfer vehicle state constraints;

[0297] The state constraint of the transfer vehicle is expressed as:

[0298]

[0299]

[0300]

[0301] Where: and Indicates the number of transfer vehicles in loading, charging and idle states; The number of transfer vehicles with a battery state of c waiting at charging station n at time t; Battery n represents the total number of batteries at charging station n; represents the number of batteries with a charge state of c in charging station n at time t; Indicates the number of transfer vehicles in loading status; Indicates the number of transfer vehicles in charging status; represents the number of transfer vehicles in idle state; R represents the re-dispatching node; C represents the battery status.

[0302] In this embodiment, in the second model building module, the constraints of the transfer vehicle charging scheduling model include charging power constraints;

[0303] The charging power constraint is expressed as:

[0304]

[0305] Where: CtoP represents the energy equivalent to unit SoC.

[0306] In this embodiment, in the calculation module, the processing steps of the deep Q network algorithm are as follows:

[0307] Step 1: Convert the energy-traffic coordinated scheduling model with the initial state of the transfer task and all transfer vehicles into a Markov model;

[0308] Step 2: Generate the optimal energy-traffic coordinated scheduling solution through a two-stage deep Q-network algorithm combined with the Markov model.

[0309] In this embodiment, in step 1 of the calculation module, the converted Markov model includes:

[0310] 1) State space, action space and reward function in the first stage

[0311]

[0312]

[0313]

[0314] Where: represents the state space, the first target action and the reward function of the first stage; i represents the transport task; d represents the date; Indicates the initial position and initial power state of the transport vehicle; EV V represents the Vth transfer vehicle; R s1.veh represents the reward function for transporting vehicles in the first stage model; represents the reward function of the second stage; X i,v An indicator that indicates the matching of the transfer task i with the transfer vehicle v. When X i,v = 1, indicating that the transfer task i is included in the task list of the transfer vehicle v, X i,v =0, it means that the transfer task i is not included in the task list of the transfer vehicle v;

[0315] 2) State space, action space and reward function in the second stage

[0316]

[0317]

[0318]

[0319]

[0320]

[0321] Where: represents the first state space and the second state space of the second stage; represents the second target action and the third target action in the second stage; t represents time; n v,t represents the position of the transfer vehicle at time t; c v,t represents the power status of the transport vehicle at time t; i represents the transport task; represents the cost of purchasing electricity from the grid; represents the power status of the transfer vehicle in charging station n at time t; stationM represents the Mth charging station; SoCC represents the power status of the Cth transfer vehicle; Indicates the unit battery level; C idle Indicates the no-load operation cost per unit time; Indicates the number of transfer vehicles in idle, charging and waiting states; Y i,v,c,t Indicates whether the transfer vehicle v actively completes the transfer task i at time t under the power state c.

[0322] In this embodiment, in step 2 of the calculation module, the processing steps of the deep Q network method are as follows:

[0323] 1) Design a DQN to approximate the Q-value function and randomly initialize its weights. The DQN is used to predict the Q-value of each action based on the current state;

[0324] 2) Create an experience replay cache to store the states, actions, rewards, and new states generated by the matching agent, transfer vehicle agent, and charging station agent interacting with the Markov model;

[0325] 3) At each time step, it includes the following two stages:

[0326] Phase 1: The matching agent selects the action space from the first phase according to the current state of the transport vehicle. Select the first target action and in the state space of the first stage And the reward function Execute the first target action in , and observe the reward and new state of the first stage;

[0327] Phase 2: The transfer vehicle agent moves from the action space of phase 2 according to the current state of the charging station. Select the second target action and in the state space of the second stage And the reward function Execute the second target action in the second stage, observe the first reward and the first new state of the second stage; the charging station agent selects the action space of the second stage according to the current state of the transfer vehicle's power state Select the third target action and in the state space of the second stage And the reward function Execute the third target action in the process, and observe the second reward and the second new state in the second stage;

[0328] 4) The current state, first target action, reward and new state of the transport vehicle in the first stage, the current state, second target action, first reward and first new state of the charging station in the second stage, and the current state, third target action, second reward and second new state of the transport vehicle's power state are stored as a set of experience in the experience playback cache;

[0329] 5) Randomly extract a batch of experiences from the experience replay cache to train the DQN, and update the weights of the DQN by minimizing the error between the predicted Q value and the target Q value;

[0330] The loss function when training DQN is as follows:

[0331] Loss=(r+γmaxQ target (s′,a)-Q current (s,a)) 2 ;

[0332] Where: r represents the current reward; γ represents the decay coefficient; maxQ target (s′, a) represents the target maximum Q value under a given state s and action a; Q current (s,a) represents the Q value currently predicted by the model given state s and action a;

[0333] 6) Repeat steps 3) to 5). Through continuous iteration and optimization, DQN will gradually learn how to make the best decisions in the energy-traffic collaborative scheduling model to maximize the long-term cumulative rewards.

[0334] 7) Input the current state of the transfer vehicle into the trained DQN and output the optimal energy-traffic coordinated scheduling plan.

[0335] Embodiment three:

[0336] This embodiment discloses a computer device.

[0337] The computer device includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned embodiments of the method for charging and scheduling electric transfer vehicles in inland ports are implemented. Alternatively, when the processor executes the computer program, the functions of each module / unit in the above-mentioned system embodiments are implemented.

[0338] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the power system dynamic reactive power compensation device.

[0339] The computer device may be a desktop computer, a notebook computer, a palm computer, a cloud server, etc. The computer device may include, but is not limited to, a processor and a memory.

[0340] Embodiment 4:

[0341] This embodiment discloses a computer-readable storage medium.

[0342] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed, the steps in each of the above-mentioned inland port electric transfer vehicle charging scheduling method embodiments are performed.

[0343] The computer program may be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of each of the above-mentioned method embodiments may be implemented. The computer program includes a computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal and a software distribution medium, etc. Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the technical solution. Those skilled in the art should understand that those modifications or equivalent replacements of the technical solution of the present invention, without departing from the purpose and scope of the present technical solution, should be included in the scope of the claims of the present invention.

Claims

1. A method for charging and dispatching electric transport vehicles in inland ports, characterized in that: include: S1: Obtain the port's transshipment tasks and the initial status of all transshipment vehicles; S2: Construct a transfer vehicle task matching model with the goal of maximizing the utilization rate of transfer vehicles and completing all transfer tasks; S3: Construct a charging scheduling model for transfer vehicles with the goal of minimizing delays, maximizing transfer vehicle efficiency, and minimizing charging costs; S4: The transfer vehicle task matching model is used as the first-stage model, and the transfer vehicle charging scheduling model is used as the second-stage model. The energy-traffic coordinated scheduling model is constructed through the first-stage model and the second-stage model. S5: Taking the transfer task and the initial state of all transfer vehicles as the input of the energy-traffic coordinated scheduling model, solving the energy-traffic coordinated scheduling model with the transfer task and the initial state of all transfer vehicles through the deep Q network algorithm, and generating the optimal energy-traffic coordinated scheduling scheme; S6: Dispatch the transfer vehicles through the optimal energy-traffic coordinated scheduling plan to complete the port's transfer tasks.

2. The method for charging and dispatching electric transport vehicles in inland ports according to claim 1, characterized in that: In step S2, the objective function of the transfer vehicle task matching model is expressed as: Where: X i,v An indicator that indicates the matching of the transfer task i with the transfer vehicle v. When X i,v = 1, indicating that the transfer task i is included in the task list of the transfer vehicle v, X i,v = 0, it means that the transfer task i is not included in the task list of the transfer vehicle v; each transfer task i consists of a tuple Description, where p i is the pickup node, d i To deliver the node, is the expected task completion time; R s1.veh Represents the reward function for transporting vehicles in the first stage model.

3. The method for charging and dispatching electric transport vehicles in inland ports according to claim 2, characterized in that: In step S2, the constraints of the transfer vehicle task matching model include task quantity constraints; The task quantity constraint is expressed as: Where: X max Indicates the upper limit of the number of tasks.

4. The method for charging and dispatching electric transport vehicles in inland ports according to claim 1, characterized in that: In step S3, the objective function of the transfer vehicle charging scheduling model is expressed as: Where: m represents the charging station; a represents the path between nodes; represents the cost function of the charging station; represents the no-load operation cost of the transfer vehicle; represents the cost of purchasing electricity from the grid; Indicates the unit battery level; C idle Indicates the no-load running cost per unit time.

5. The method for charging and dispatching electric transport vehicles in inland ports according to claim 4, characterized in that: In step S3, the constraints of the transfer vehicle charging scheduling model include the transfer vehicle power constraints; The power constraint of the transfer vehicle is expressed as: Where: Y i,v,c,t Indicates whether the transfer vehicle v actively completes the transfer task i at time t under the power state c; X i,v An indicator that indicates the matching of the transfer task i with the transfer vehicle v. When X i,v = 1, indicating that the transfer task i is included in the task list of the transfer vehicle v, X i,v = 0, it means that the transfer task i is not included in the task list of the transfer vehicle v; each transfer task i consists of a tuple Description, where p i is the pickup node, d i To deliver the node, The expected task completion time.

6. The method for charging and dispatching electric transport vehicles in inland ports according to claim 4, characterized in that: In step S3, the constraints of the transfer vehicle charging scheduling model include transfer vehicle state constraints; The state constraint of the transfer vehicle is expressed as: Where: and Indicates the number of transfer vehicles in loading, charging and idle states; The number of transfer vehicles with a battery state of c waiting at charging station n at time t; Battery n represents the total number of batteries at charging station n; represents the number of batteries with a charge state of c in charging station n at time t; Indicates the number of transfer vehicles in loading status; Indicates the number of transfer vehicles in charging status; represents the number of transfer vehicles in idle state; R represents the re-dispatching node; C represents the battery status.

7. The method for charging and dispatching electric transport vehicles in inland ports according to claim 4, characterized in that: In step S3, the constraints of the transfer vehicle charging scheduling model include charging power constraints; The charging power constraint is expressed as: Where: CtoP represents the energy equivalent to unit SoC.

8. The method for charging and dispatching electric transport vehicles in inland ports according to claim 1, characterized in that: In step S5, the processing steps of the deep Q network algorithm are as follows: S501: converting the energy-traffic coordinated scheduling model with the transfer task and the initial state of all transfer vehicles into a Markov model; S502: Generate the optimal energy-traffic coordinated scheduling solution through a two-stage deep Q network algorithm combined with a Markov model.

9. The method for charging and dispatching electric transport vehicles in inland ports according to claim 8, characterized in that: In step S501, the converted Markov model includes: 1) State space, action space and reward function in the first stage Where: represents the state space, the first target action and the reward function of the first stage; i represents the transport task; d represents the date; Indicates the initial position and initial power state of the transport vehicle; EV V represents the Vth transfer vehicle; R s1.veh represents the reward function for transporting vehicles in the first stage model; represents the reward function of the second stage; X i,v An indicator that indicates the matching of the transfer task i with the transfer vehicle v. When X i,v = 1, indicating that the transfer task i is included in the task list of the transfer vehicle v, X i,v =0, it means that the transfer task i is not included in the task list of the transfer vehicle v; 2) State space, action space and reward function in the second stage Where: represents the first state space and the second state space of the second stage; represents the second target action and the third target action in the second stage; t represents time; n v,t represents the position of the transfer vehicle at time t; c v,t represents the power status of the transport vehicle at time t; i represents the transport task; represents the cost of purchasing electricity from the grid; represents the power status of the transfer vehicle in charging station n at time t; stationM represents the Mth charging station; SoCC represents the power status of the Cth transfer vehicle; Indicates the unit battery level; C idle Indicates the no-load operation cost per unit time; Indicates the number of transfer vehicles in idle, charging and waiting states; Y i,v,c,t Indicates whether the transfer vehicle v actively completes the transfer task i at time t under the power state c.

10. The method for charging and dispatching electric transport vehicles in inland ports according to claim 9, characterized in that: In step S502, the processing steps of the deep Q network method are as follows: S5021: Design a DQN to approximate the Q-value function and randomly initialize its weights. The DQN is used to predict the Q-value of each action based on the current state. S5022: Create an experience replay cache to store the states, actions, rewards, and new states generated by the interaction of the matching agent, the transfer vehicle agent, and the charging station agent with the Markov model; S5023: At each time step, it includes the following two stages: Phase 1: The matching agent selects the action space from the first phase according to the current state of the transport vehicle. Select the first target action and in the state space of the first stage And the reward function Execute the first target action in , and observe the reward and new state of the first stage; Phase 2: The transfer vehicle agent moves from the action space of phase 2 according to the current state of the charging station. Select the second target action and in the state space of the second stage And the reward function Execute the second target action in the second stage and observe the first reward and the first new state in the second stage; The charging station agent selects an action space from the second stage according to the current state of the transfer vehicle's power state. Select the third target action and in the state space of the second stage And the reward function Execute the third target action in the process, and observe the second reward and the second new state in the second stage; S5024: storing the current state, first target action, reward and new state of the transport vehicle in the first stage, the current state, second target action, first reward and first new state of the charging station in the second stage, and the current state, third target action, second reward and second new state of the transport vehicle's power state as a set of experience in the experience playback cache; S5025: randomly extract a batch of experiences from the experience replay cache to train the DQN, and update the weight of the DQN by minimizing the error between the predicted Q value and the target Q value; The loss function when training DQN is as follows: Loss=(r+γmaxQ target (s′,a)-Q current (s,a)) 2 ; Where: r represents the current reward; γ represents the decay coefficient; maxQ target (s′, a) represents the target maximum Q value under a given state s and action a; Q current (s,a) represents the Q value currently predicted by the model given state s and action a; S5026: Repeat steps S5023 to S5025. Through continuous iteration and optimization, DQN will gradually learn how to make the best decision in the energy-traffic coordinated scheduling model to maximize the long-term cumulative reward. S5027: Input the current state of the transfer vehicle into the trained DQN and output the optimal energy-traffic coordinated scheduling solution.

11. Charging and dispatching system for electric transport vehicles in inland ports, characterized by: Based on the inland port electric transfer vehicle charging scheduling method in claim 1, the system comprises: The acquisition module is used to obtain the port's transshipment tasks and the initial status of all transshipment vehicles; The first model building module is used to build a transfer vehicle task matching model with the goal of maximizing the utilization rate of transfer vehicles and completing all transfer tasks; The second model building module is used to build a transfer vehicle charging scheduling model with the goals of minimizing delays, maximizing transfer vehicle efficiency and minimizing charging costs; The third model building module is used to use the transfer vehicle task matching model as the first stage model and the transfer vehicle charging scheduling model as the second stage model, and to build an energy-traffic coordinated scheduling model through the first stage model and the second stage model; A calculation module is used to use the transfer task and the initial state of all transfer vehicles as the input of the energy-traffic coordinated scheduling model, solve the energy-traffic coordinated scheduling model with the transfer task and the initial state of all transfer vehicles as the input through the deep Q network algorithm, and generate the optimal energy-traffic coordinated scheduling plan; The execution module is used to dispatch the transfer vehicles through the optimal energy-traffic coordinated scheduling plan to complete the port's transfer tasks.

12. The inland port electric transport vehicle charging dispatching system according to claim 11, characterized in that: In the first model building module, the objective function of the transfer vehicle task matching model is expressed as: Where: X i,v An indicator that indicates the matching of the transfer task i with the transfer vehicle v. When X i,v = 1, indicating that the transfer task i is included in the task list of the transfer vehicle v, X i,v = 0, it means that the transfer task i is not included in the task list of the transfer vehicle v; each transfer task i consists of a tuple Description, where p i is the pickup node, d i To deliver the node, is the expected task completion time; R s1.veh Represents the reward function for transporting vehicles in the first stage model.

13. The inland port electric transport vehicle charging dispatching system as claimed in claim 12, characterized in that: In the first model building module, the constraints of the transfer vehicle task matching model include task quantity constraints; The task quantity constraint is expressed as: Where: X max Indicates the upper limit of the number of tasks.

14. The inland port electric transport vehicle charging dispatching system according to claim 11, characterized in that: In the second model building module, the objective function of the transfer vehicle charging scheduling model is expressed as: Where: m represents the charging station; a represents the path between nodes; represents the cost function of the charging station; represents the no-load operation cost of the transfer vehicle; represents the cost of purchasing electricity from the grid; Indicates the unit battery level; C idle Indicates the no-load running cost per unit time.

15. The inland port electric transport vehicle charging dispatching system as claimed in claim 14, characterized in that: In the second model building module, the constraints of the transfer vehicle charging scheduling model include the transfer vehicle power constraints; The power constraint of the transfer vehicle is expressed as: Where: Y i,v,c,t Indicates whether the transfer vehicle v actively completes the transfer task i at time t under the power state c; X i,v An indicator that indicates the matching of the transfer task i with the transfer vehicle v. When X i,v = 1, indicating that the transfer task i is included in the task list of the transfer vehicle v, X i,v = 0, it means that the transfer task i is not included in the task list of the transfer vehicle v; each transfer task i consists of a tuple Description, where p i is the pickup node, d i To deliver the node, The expected task completion time.

16. The inland port electric transport vehicle charging dispatching system according to claim 14, characterized in that: In the second model building module, the constraints of the transfer vehicle charging scheduling model include transfer vehicle state constraints; The state constraint of the transfer vehicle is expressed as: Where: and Indicates the number of transfer vehicles in loading, charging and idle states; The number of transfer vehicles with a battery state of c waiting at charging station n at time t; Battery n represents the total number of batteries at charging station n; represents the number of batteries with a charge state of c in charging station n at time t; Indicates the number of transfer vehicles in loading status; Indicates the number of transfer vehicles in charging status; represents the number of transfer vehicles in idle state; R represents the re-dispatching node; C represents the battery status.

17. The inland port electric transport vehicle charging dispatching system according to claim 14, characterized in that: In the second model building module, the constraints of the transfer vehicle charging scheduling model include transfer vehicle state constraints; The state constraint of the transfer vehicle is expressed as: Where: and Indicates the number of transfer vehicles in loading, charging and idle states; The number of transfer vehicles with a battery state of c waiting at charging station n at time t; Battery n represents the total number of batteries at charging station n; represents the number of batteries with a charge state of c in charging station n at time t; Indicates the number of transfer vehicles in loading status; Indicates the number of transfer vehicles in charging status; represents the number of transfer vehicles in idle state; R represents the re-dispatching node; C represents the battery status.

18. The inland port electric transport vehicle charging dispatching system according to claim 14, characterized in that: In the second model building module, the constraints of the transfer vehicle charging scheduling model include charging power constraints; The charging power constraint is expressed as: Where: CtoP represents the energy equivalent to unit SoC.

19. The inland port electric transport vehicle charging dispatching system according to claim 11, characterized in that: In the calculation module, the processing steps of the deep Q network algorithm are as follows: Step 1: Convert the energy-traffic coordinated scheduling model with the initial state of the transfer task and all transfer vehicles into a Markov model; Step 2: Generate the optimal energy-traffic coordinated scheduling solution through a two-stage deep Q-network algorithm combined with the Markov model.

20. The inland port electric transport vehicle charging dispatching system as claimed in claim 19, characterized in that: In step 1 of the calculation module, the converted Markov model includes: 1) State space, action space and reward function in the first stage Where: represents the state space, the first target action and the reward function of the first stage; i represents the transport task; d represents the date; Indicates the initial position and initial power state of the transport vehicle; EV V represents the Vth transfer vehicle; R s1.veh represents the reward function for transporting vehicles in the first stage model; represents the reward function of the second stage; X i,v An indicator that indicates the matching of the transfer task i with the transfer vehicle v. When X i,v = 1, indicating that the transfer task i is included in the task list of the transfer vehicle v, X i,v =0, it means that the transfer task i is not included in the task list of the transfer vehicle v; 2) State space, action space and reward function in the second stage Where: represents the first state space and the second state space of the second stage; represents the second target action and the third target action in the second stage; t represents time; n v,t represents the position of the transfer vehicle at time t; c v,t represents the power status of the transport vehicle at time t; i represents the transport task; represents the cost of purchasing electricity from the grid; represents the power status of the transfer vehicle in charging station n at time t; stationM represents the Mth charging station; SoCC represents the power status of the Cth transfer vehicle; Indicates the unit battery level; C idle Indicates the no-load operation cost per unit time; Indicates the number of transfer vehicles in idle, charging and waiting states; Y i,v,c,t Indicates whether the transfer vehicle v actively completes the transfer task i at time t under the power state c.

21. The inland port electric transport vehicle charging dispatching system as claimed in claim 19, characterized in that: In step 2 of the calculation module, the processing steps of the deep Q network method are as follows: 1) Design a DQN to approximate the Q-value function and randomly initialize its weights. The DQN is used to predict the Q-value of each action based on the current state; 2) Create an experience replay cache to store the states, actions, rewards, and new states generated by the matching agent, transfer vehicle agent, and charging station agent interacting with the Markov model; 3) At each time step, it includes the following two stages: Phase 1: The matching agent selects the action space from the first phase according to the current state of the transport vehicle. Select the first target action and in the state space of the first stage And the reward function Execute the first target action in , and observe the reward and new state of the first stage; Phase 2: The transfer vehicle agent moves from the action space of phase 2 according to the current state of the charging station. Select the second target action and in the state space of the second stage And the reward function Execute the second target action in the second stage and observe the first reward and the first new state in the second stage; The charging station agent selects an action space from the second stage according to the current state of the transfer vehicle's power state. Select the third target action and in the state space of the second stage And the reward function Execute the third target action in the process, and observe the second reward and the second new state in the second stage; 4) The current state, first target action, reward and new state of the transport vehicle in the first stage, the current state, second target action, first reward and first new state of the charging station in the second stage, and the current state, third target action, second reward and second new state of the transport vehicle's power state are stored as a set of experience in the experience playback cache; 5) Randomly extract a batch of experiences from the experience replay cache to train the DQN, and update the weights of the DQN by minimizing the error between the predicted Q value and the target Q value; The loss function when training DQN is as follows: Loss=(r+γmaxQ target (s′,a)-Q current (s,a)) 2 ; Where: r represents the current reward; γ represents the decay coefficient; maxQ target (s′, a) represents the target maximum Q value under a given state s and action a; Q current (s,a) represents the Q value currently predicted by the model given state s and action a; 6) Repeat steps 3) to 5). Through continuous iteration and optimization, DQN will gradually learn how to make the best decisions in the energy-traffic collaborative scheduling model to maximize the long-term cumulative rewards. 7) Input the current state of the transfer vehicle into the trained DQN and output the optimal energy-traffic coordinated scheduling plan.

22. A computer device, characterized in that: include: one or more processors; The processor is used to store one or more programs; When the one or more programs are executed by the one or more processors, the charging scheduling method for inland port electric transfer vehicles as described in any one of claims 1 to 10 is implemented.

23. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed, the charging scheduling method for inland port electric transfer vehicles as described in any one of claims 1 to 10 is implemented.

Images