A steel logistics intelligent scheduling method and system based on markov decision

Abstract

This invention discloses a method and system for intelligent scheduling of steel logistics based on Markov decision-making. The method includes: acquiring information on steel orders to be assigned and information on available vehicles; establishing a steel logistics scheduling decision model based on Markov decision-making based on the information on steel orders to be assigned and available vehicles; the steel logistics scheduling decision model includes an action value function and a state value function; calculating the action value function and state value function based on dynamic programming to obtain the reward after matching each steel order to be assigned with each available vehicle; determining the optimal matching result using a maximum weight matching algorithm based on the reward; and performing steel logistics scheduling based on the optimal matching result. This invention can improve the transportation efficiency of steel logistics, reduce transportation costs, and provide theoretical guarantees and technical support for improving the intelligent level of steel logistics scheduling management while minimizing the occupation of road and time resources.

Description

Technical Field

[0001] This invention relates to the field of logistics scheduling technology, and in particular to an intelligent scheduling method and system for steel logistics based on Markov decision-making. Background Technology

[0002] In the steel logistics sector, every link in the steel logistics process requires high-performance analysis and calculation to connect the needs and information of both manufacturers and transporters in a timely manner. Taking into account the special characteristics of steel logistics in terms of transportation costs and modes, the system provides intelligent scheduling solutions with lower costs and lower energy consumption in multiple links such as logistics order release, matching of transport orders and vehicles, and entry into the factory and assembly scheduling during the transportation process.

[0003] Currently, the steel industry lacks intelligent logistics management solutions, and few use intelligent systems for order allocation and scheduling. Given the numerous constraints on steel logistics order allocation and the various scheduling requirements specific to steel logistics during transportation, it is difficult to achieve cost-priority, efficient, and accurate intelligent scheduling of steel logistics. Summary of the Invention

[0004] The purpose of this invention is to provide a steel logistics intelligent scheduling method and system based on Markov decision-making, so as to realize low-cost, high-efficiency steel logistics order dispatch and transportation scheduling.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] A method for intelligent scheduling of steel logistics based on Markov decision-making includes:

[0007] Obtain information on pending steel orders and available vehicles;

[0008] A steel logistics scheduling decision model based on Markov decision-making is established based on the information of the steel orders to be allocated and the information of the available vehicles; the steel logistics scheduling decision model includes an action value function and a state value function;

[0009] The action value function and the state value function are calculated based on dynamic programming to obtain the reward after matching each steel order to be allocated and each available vehicle.

[0010] Based on the aforementioned reward, the optimal matching result is determined using the maximum weight matching algorithm;

[0011] Steel logistics scheduling is carried out based on the optimal matching result.

[0012] Optionally, the information of the steel order to be allocated includes: pick-up location, delivery location, delivery time requirements, and the type and quantity of steel.

[0013] Optionally, the information on available vehicles includes: the carrier to which the vehicle belongs, the parking location, the vehicle model, and the load limit.

[0014] Optionally, after obtaining the information on the steel orders to be allocated and the information on available vehicles, the method further includes: filtering the available vehicles based on the information on the steel orders to be allocated and the information on available vehicles.

[0015] Optionally, the step of filtering the available vehicles based on the information of the steel orders to be allocated and the information of the available vehicles specifically includes:

[0016] The available vehicles are filtered based on the latest delivery time, the pickup location, the delivery location, and the parking location:

[0017] Based on the type of steel, available vehicles that do not meet the delivery conditions for the corresponding type of steel are filtered out.

[0018] Optionally, the action value function is: The state value function is: in, For the optimal expected return, G t Let s be the parking area available for vehicles at time t. t Let a be the state of available vehicles at time t. t Let t represent the available vehicle actions.

[0019] This invention also provides a steel logistics intelligent scheduling system based on Markov decision-making, comprising:

[0020] The information acquisition module is used to acquire information on steel orders to be assigned and information on available vehicles;

[0021] The model building module is used to build a steel logistics scheduling decision model based on Markov decision-making, based on the information of the steel orders to be allocated and the information of the available vehicles; the steel logistics scheduling decision model includes an action value function and a state value function;

[0022] The reward determination module is used to calculate the action value function and the state value function based on dynamic programming to obtain the reward after matching each steel order to be allocated with each available vehicle;

[0023] The optimal matching determination module is used to determine the optimal matching result based on the reward using the maximum weight matching algorithm;

[0024] The logistics scheduling module is used to schedule steel logistics based on the optimal matching result.

[0025] Optionally, the system further includes a filtering module for filtering the available vehicles based on the information of the steel orders to be allocated and the information of the available vehicles.

[0026] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0027] This invention addresses the characteristics and limitations of steel logistics transportation regarding vehicles and transported steel. It establishes a matching relationship between orders and vehicles through a Markov decision-making model for steel logistics scheduling. The invention performs real-time analysis of the rewards a vehicle receives for selecting different orders, with the optimal matching result maximizing expected revenue. This invention improves the transportation efficiency of steel logistics, reduces transportation costs, and provides theoretical and technical support for enhancing the intelligence level of steel logistics scheduling management while minimizing road and time resource usage. This method is highly automated, easy to deploy and implement, and offers good real-time performance, making it particularly suitable for order allocation in steel logistics scenarios. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A flowchart of a steel logistics intelligent scheduling method based on Markov decision-making provided in an embodiment of the present invention;

[0030] Figure 2 This is a diagram illustrating the architecture of a steel logistics intelligent scheduling method based on Markov decision-making, provided in an embodiment of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] The purpose of this invention is to provide a steel logistics intelligent scheduling method and system based on Markov decision-making, so as to realize low-cost, high-efficiency steel logistics order dispatch and transportation scheduling.

[0033] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] like Figure 1-2 As shown, the intelligent scheduling method for steel logistics based on Markov decision-making provided by this invention includes the following steps:

[0035] Step 101: Obtain information on steel orders to be assigned and information on available vehicles.

[0036] The data management backend updates steel order information and available vehicle information in real time.

[0037] Information on steel orders to be allocated includes: pick-up location, delivery location, delivery time requirements, and the type and quantity of steel.

[0038] Information on available vehicles includes: the carrier to which the vehicle belongs, the parking location, the vehicle model, and the load limit.

[0039] The transport vehicles are screened based on the latest delivery time, vehicle type, and cargo location:

[0040] The order pickup and delivery locations, along with the vehicle parking location, jointly determine the transportation distance and time. The final matching result must meet the requirement that the delivery completion time is earlier than the latest delivery time required by the order. All vehicles at parking locations where the shortest transportation time from the vehicle parking location to the pickup point and then to the delivery destination is greater than the latest delivery time will not be included in the corresponding agent under the current order conditions.

[0041] The type and quantity of steel ordered will be used to select the transport vehicles.

[0042] Based on the restrictions on steel logistics transportation, vehicles that do not meet the delivery conditions for the corresponding type of steel are filtered out according to the type of steel ordered.

[0043] After two rounds of screening, vehicles that meet the criteria are retained.

[0044] Step 102: Establish a steel logistics scheduling decision model based on Markov decision-making, based on information on steel orders to be allocated and information on available vehicles; the steel logistics scheduling decision model includes an action value function and a state value function.

[0045] Based on the Markov decision framework design, a steel logistics scheduling decision model based on Markov decision is established. Specific methods include:

[0046] The agent in the steel logistics scheduling decision model based on Markov decision-making is for each vehicle.

[0047] The state of the steel logistics scheduling decision model based on Markov decision-making is represented by a two-dimensional vector representing the spatiotemporal state, which is quantified as a Cartesian product of a fixed number of time periods and regions:

[0048] s=(t,g)∈S

[0049] Where t∈T is the time index, g∈G is the index of the parking area where the vehicle is located, and |S|=|T|×|G|;

[0050] For the intelligent agent, there are two actions in the steel logistics scheduling decision model based on Markov decision: action1 and action2.

[0051] Action 1: Assign a vehicle to serve a specific order. In this state, the vehicle will pick up and deliver steel goods at the designated location and receive the order's reward. Action 2: Idle state. In this state, the reward received is always 0 until the state changes.

[0052] The agent's objective is to maximize its gain, which is the expected cumulative future reward starting from time t. The gain is expressed by the following formula:

[0053]

[0054] Define the state value function and action value function for the steel logistics scheduling decision model based on Markov decision-making:

[0055] State value function:

[0056] Action value function:

[0057] This leads to a Markov decision-based steel logistics scheduling decision model applicable to this method, which formally describes the interdependencies between the orders and vehicles of interest.

[0058] Step 103: Calculate the action value function and state value function based on dynamic programming to obtain the reward after matching each steel order to be assigned and each available vehicle.

[0059] The reward is defined to determine the optimization goal of the entire system. The reward is essentially the revenue from order fulfillment. The goal is to maximize the total gross merchandise volume (GMV) of steel products. At the time an order is placed, the system cannot obtain accurate future information, including the revenue of the order and the specific time of completion. Therefore, the decision of the strategy depends on the estimated completion time and revenue.

[0060] In order to use order delivery revenue as the reward for determining the optimization goal of the entire system, the data is decomposed into a set of transaction pairs (s,a,s',r) based on the steel logistics scheduling decision model based on Markov decision.

[0061] When an order is about to be assigned to a vehicle, if the vehicle's status changes, the reward will not be received immediately if the vehicle is idle. The specific update rules for the time difference between the transition between idle status and order assignment status are as follows:

[0062] V(s)←V(s)+α[0+γV(s')-V(s)]

[0063] Where s = (t, g) is the current state of the vehicle, and s' = (t+1, g) is the next state after s;

[0064] Furthermore, after the agent receives a reward, a next state transition will occur. The formula for calculating the time difference of this state transition is:

[0065] V(s)←V(s)+α[R γ +γ Δt V(s”)-V(s)]

[0066] Where s = (t, g) still represents the current state of the vehicle, and s” = (t + Δt, g) dest ) indicates the estimated completion status of the current order, i.e., the order's completion status. Δt represents the sum of the estimated times for the vehicle to complete the steel goods delivery process from the origin, including pickup, delivery, and return. When the status is idle, Δt = 1. g dest Record the destination of the order.

[0067] The value function is calculated using dynamic programming to obtain the reward that the agent will receive for receiving an order.

[0068] Step 104: Based on the reward, use the maximum weight matching algorithm to determine the optimal matching result.

[0069] Goal setting can be interpreted as finding the optimal state for each agent to optimize future global gains in a coordinated manner;

[0070] The objective function of the centralized scheduling algorithm is as follows:

[0071]

[0072]

[0073]

[0074] Among them, a ij =1 indicates that when order j is assigned to vehicle i, a ij=0 indicates that order j has not been assigned to vehicle i, where i∈[1,…,m] corresponds to all available vehicles at the current timestamp, Q π (i,j) is the action value function when vehicle i performs action j, where i=0 and j=0 correspond to the default operation of not providing any action at the current timestamp.

[0075] The centralized scheduling algorithm is transformed into a two-map matching problem, which is then computed using the KM algorithm.

[0076] Executing a dominant function eliminates all default operations, and the objective function transforms as follows:

[0077]

[0078]

[0079]

[0080] In the objective function,

[0081] The integration of reinforcement learning and optimization decision-making forms the best match, and its globally optimal expected return is:

[0082]

[0083] Decompose the global objective into the product of individual vehicles:

[0084]

[0085] Since maximizing the expected gain equals maximizing the action value function, the formula for maximizing the target turning is:

[0086]

[0087] Finally, all agents are set to be homogeneous, according to the formula

[0088]

[0089] Obtain the optimal order and vehicle matching results.

[0090] Step 105: Perform steel logistics scheduling based on the optimal matching result.

[0091] The order matching results are returned to the backend, which records the corresponding vehicle information and sends the order information to the driver of the delivery vehicle.

[0092] This invention addresses the complex constraints between goods and vehicles in steel logistics scheduling. By analyzing the benefits that different transportation orders bring to different intelligent agents (vehicles), it can improve steel logistics transportation efficiency and reduce transportation costs, providing theoretical guarantees and technical support for improving the intelligence level of steel logistics scheduling and management. This method is highly automated, easy to deploy and implement, has good real-time performance, and is more practical for order allocation in steel logistics scenarios.

[0093] This invention also provides a steel logistics intelligent scheduling system based on Markov decision-making, comprising:

[0094] The information acquisition module is used to acquire information on steel orders to be assigned and information on available vehicles;

[0095] The model building module is used to build a steel logistics scheduling decision model based on Markov decision-making, based on information about steel orders to be allocated and information about available vehicles. The steel logistics scheduling decision model includes an action value function and a state value function.

[0096] The reward determination module is used to calculate the action value function and state value function based on dynamic programming to obtain the reward after matching each steel order to be assigned with each available vehicle;

[0097] The optimal matching determination module is used to determine the optimal matching result based on the reward using the maximum weight matching algorithm;

[0098] The logistics scheduling module is used to schedule steel logistics based on the optimal matching results.

[0099] The system also includes a filtering module, which filters available vehicles based on information about steel orders to be assigned and information about available vehicles.

[0100] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0101] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for intelligent scheduling of steel logistics based on Markov decision-making, characterized in that, include: Obtain information on pending steel orders and available vehicles; The information for the steel orders to be allocated includes: pick-up location, delivery location, delivery time requirements, and the type and quantity of steel; the information for the available vehicles includes: the carrier to which the vehicle belongs, parking location, vehicle model, and load limit; After obtaining the information on steel orders to be allocated and available vehicles, the process further includes: filtering available vehicles based on the information on steel orders to be allocated and available vehicles; specifically, filtering available vehicles based on the information on steel orders to be allocated and available vehicles includes: filtering available vehicles based on the latest delivery time, the pickup location, the delivery location, and the parking location: the order pickup location and delivery location, along with the vehicle parking location, jointly determine the transportation distance and transportation time. The final matching result must satisfy that the delivery completion time is earlier than the latest delivery time required by the order. All vehicles parked at locations where the shortest transportation time from the vehicle parking location to the pickup point and then to the delivery destination is greater than the latest delivery time are not included in the corresponding agent under the current order conditions; filtering available vehicles that do not meet the delivery conditions for the corresponding type of steel based on the type of steel: based on steel logistics transportation restrictions, vehicles that do not meet the delivery conditions for the corresponding type of steel are filtered; after two filtering processes, vehicles that meet the conditions are retained. Based on the information of the steel orders to be allocated and the information of the available vehicles, a steel logistics scheduling decision model based on Markov decision-making is established. The steel logistics scheduling decision model includes an action value function and a state value function. According to the Markov decision architecture design, a steel logistics scheduling decision model based on Markov decision-making is established. Specifically, the agent of the steel logistics scheduling decision model based on Markov decision-making is each vehicle. The state of the steel logistics scheduling decision model based on Markov decision-making is represented by a two-dimensional vector representing the spatiotemporal state, which is quantified as a Cartesian product of a fixed number of time periods and regions: s = (t, g) ∈ S; where t ∈ T is the time interval. The index, g∈G, is the index of the parking area where the vehicle is located, |S|=|T|×|G|; For the agent, in the steel logistics scheduling decision model based on Markov decision, there are two actions: action1 and action2; action1: Assigning the vehicle to serve a specific order. In this state, the vehicle will go to the designated location to pick up and deliver steel goods and receive the order's reward; action2: Idle state. In this state, the reward obtained is always 0 until the state changes; The agent's goal is to maximize its gain, that is, the expected cumulative future reward starting from time t, where the gain is expressed by the following formula: To define the state value function and action value function for the steel logistics scheduling decision model based on Markov decision-making, the action value function is as follows: The state value function is: in, For the optimal expected return, G t Let s be the parking area available for vehicles at time t. t Let a be the state of available vehicles at time t. t Let t be the available vehicle actions; then, a steel logistics scheduling decision model based on Markov decision is obtained, that is, the interdependence between the orders and vehicles of interest is formally described. The action value function and the state value function are calculated based on dynamic programming to obtain the reward after matching each steel order to be allocated with each available vehicle; the reward is defined to determine the optimization objective of the entire system. The reward is essentially the revenue from order delivery. The objective is to maximize the total transaction volume of steel products. At the time of order release, the system cannot obtain accurate future information, including the revenue of the order and the specific time of completion. Therefore, the decision of the strategy depends on the estimated completion time and revenue. Based on the aforementioned reward, the optimal matching result is determined using the maximum weight matching algorithm; Steel logistics scheduling is carried out based on the optimal matching result.

2. A steel logistics intelligent scheduling system based on Markov decision-making, characterized in that, include: The information acquisition module is used to acquire information on steel orders to be assigned and information on available vehicles; The information for the steel orders to be allocated includes: pick-up location, delivery location, delivery time requirements, and the type and quantity of steel; the information for the available vehicles includes: the carrier to which the vehicle belongs, parking location, vehicle model, and load limit; The model building module is used to establish a steel logistics scheduling decision model based on Markov decision-making, based on the information of the steel orders to be allocated and the information of the available vehicles. The steel logistics scheduling decision model includes an action value function and a state value function. According to the Markov decision architecture design, the steel logistics scheduling decision model based on Markov decision-making is established. Specifically, the agent of the steel logistics scheduling decision model based on Markov decision-making is each vehicle. The state of the steel logistics scheduling decision model based on Markov decision-making is represented by a two-dimensional vector representing the spatiotemporal state, which is quantized into a Cartesian product of a fixed number of time periods and regions: s = (t, g) ∈ S; where... t∈T is the time index, g∈G is the index of the parking area where the vehicle is located, |S|=|T|×|G|; For the agent, in the steel logistics scheduling decision model based on Markov decision, there are two actions: action1 and action2; action1: Assigning the vehicle to serve a specific order. In this state, the vehicle will pick up and deliver steel goods at the designated location and receive the order's reward; action2: Idle state. In this state, the reward obtained is always 0 until the state changes; The agent's goal is to maximize its gain, that is, the expected cumulative future reward starting from time t. The gain is expressed by the following formula: To define the state value function and action value function for the steel logistics scheduling decision model based on Markov decision-making, the action value function is as follows: The state value function is: in, For the optimal expected return, G t Let s be the parking area available for vehicles at time t. t Let a be the state of available vehicles at time t. t Let t be the available vehicle actions; then, a steel logistics scheduling decision model based on Markov decision is obtained, that is, the interdependence between the orders and vehicles of interest is formally described. The reward determination module is used to calculate the action value function and the state value function based on dynamic programming to obtain the reward after matching each steel order to be allocated with each available vehicle; the reward is defined to determine the optimization objective of the entire system. The reward is essentially the revenue from order delivery. The objective is to maximize the total transaction volume of steel products. At the time of order release, the system cannot obtain accurate future information, including the revenue of the order and the specific completion time. Therefore, the strategy decision depends on the estimated completion time and revenue. The optimal matching determination module is used to determine the optimal matching result based on the reward using the maximum weight matching algorithm; The logistics scheduling module is used to schedule steel logistics based on the optimal matching result; Also includes: The filtering module is used to filter available vehicles based on the information of the steel orders to be allocated and the information of available vehicles. Specifically, this filtering includes: filtering available vehicles based on the latest delivery time, the pickup location, the delivery location, and the parking location; the pickup and delivery locations, along with the vehicle parking location, jointly determine the transportation distance and time. The final matching result must satisfy the condition that the delivery completion time is earlier than the latest delivery time required by the order. Vehicles parked at locations where the shortest transportation time from the vehicle parking location to the pickup point and then to the delivery destination is greater than the latest delivery time are not included in the corresponding agent under the current order conditions; filtering available vehicles that do not meet the delivery conditions for the corresponding type of steel based on the type of steel: based on steel logistics transportation restrictions, vehicles that do not meet the delivery conditions for the corresponding type of steel are filtered; after two filtering processes, vehicles that meet the conditions are retained.

Images