Logistics scheduling optimization method based on deep reinforcement learning and related equipment

Abstract

The invention discloses a logistics scheduling optimization method based on deep reinforcement learning and related equipment, and the scheme provided by the invention is based on a deep reinforcement learning algorithm, and through constructing a unified deep reinforcement learning optimization model, three key elements of facility site selection, personnel scheduling and path planning are integrated, so that the logistics scheduling optimization method based on deep reinforcement learning is realized. Reinforcement learning and a multi-objective optimization algorithm are utilized to reinforce learning, through self-learning of an intelligent agent in a dynamic environment, a scheduling strategy is continuously optimized, through multi-objective optimization, collaborative optimization of all parts of logistics scheduling is achieved through balance of cost and efficiency, and through continuous learning and strategy adjustment in the dynamic environment, the logistics scheduling efficiency is improved. The efficient scheduling of the logistics network is realized, and the accuracy of the scheduling decision is ensured.

Description

Technical Field

[0001] The present application relates to the technical field of logistics scheduling systems, and in particular to a logistics scheduling optimization method and related equipment based on deep reinforcement learning. Background Art

[0002] Logistics and supply chain management play a vital role in the modern economy. With the rapid development of e-commerce and the acceleration of globalization, the complexity and scale of logistics systems are increasing. Traditional logistics scheduling methods often face problems of inefficiency and high costs when dealing with large-scale, complex and changing logistics environments. Therefore, using advanced technical means to optimize the logistics distribution network, improve operational efficiency and reduce costs has become the key to the development of the industry.

[0003] In the logistics and distribution industry, several key factors that affect logistics scheduling include: warehouse site selection, personnel scheduling and distribution route planning. Warehouse site selection refers to the selection of shipping warehouses, and personnel scheduling refers to the allocation of distribution personnel / vehicles. The existing comprehensive scheduling optimization methods for these three key factors usually adopt a step-by-step optimization method, which handles facility site selection, personnel scheduling and route planning separately, which is easy to fall into local optimality and difficult to achieve overall optimality. Summary of the invention

[0004] The present application provides a logistics scheduling optimization method and related equipment based on deep reinforcement learning, which are used to solve the technical problem that the existing logistics scheduling optimization technology is prone to fall into local optimality.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a logistics scheduling optimization method based on deep reinforcement learning, comprising:

[0006] According to the received logistics order information, determine the customer location and order requirements corresponding to the customer node from the logistics order information;

[0007] According to the order requirements, the inventory information of each warehouse in the logistics system is matched to determine several candidate warehouse nodes;

[0008] The node information of the candidate warehouse node and the customer node and the vehicle status information are input as an input sequence to a preset deep reinforcement learning model, so that the deep reinforcement learning model uses the input sequence as a state quantity of deep reinforcement learning, and based on the state quantity, combined with the logistics scheduling optimization logic built into the deep reinforcement learning model, generates an action quantity, and the action quantity includes: warehouse site selection results and distribution path planning;

[0009] Based on the warehouse site selection results and the distribution route planning, a logistics scheduling instruction is generated and issued to execute the logistics distribution task according to the logistics scheduling instruction.

[0010] Preferably, it also includes:

[0011] The scheduling statistical data within a preset time period before and after the issuance of the logistics scheduling instruction is counted, and then the deviation of the statistical data before and after the instruction is issued is used as the reward amount of deep reinforcement learning, so as to determine the model adjustment parameters of the deep reinforcement learning model according to the reward amount combined with a preset reward function.

[0012] Preferably, the deep reinforcement learning model is specifically:

[0013]

[0014] In the formula, is a collection of warehouse nodes, is the set of client nodes, is a collection of delivery routes. is the customer subset, s i 、s j are the two-dimensional coordinates of node i and node j respectively, P is the route cost per unit distance, F is the fixed cost of each vehicle, and D j is the demand of the jth customer, C i is the capacity of the i-th warehouse, O i is the fixed cost of the i-th warehouse, L is the fixed load of each vehicle, x i is an indicator variable for whether warehouse i is used, e ijl is the indicator variable of whether node i and node j are connected in route l, u ij is an indicator variable indicating whether customer j is associated with warehouse i.

[0015] Preferably, the dispatch statistics include: total transportation cost, number of completed orders, average delivery time and vehicle utilization efficiency.

[0016] At the same time, the second aspect of the present application provides a logistics scheduling optimization device based on deep reinforcement learning, including:

[0017] An order response processing unit, configured to determine a customer location and order requirements corresponding to a customer node from the logistics order information according to the received logistics order information;

[0018] A warehouse node determination unit, used to match the order demand with the inventory information of each warehouse in the logistics system to determine a number of candidate warehouse nodes;

[0019] A model operation unit, used to input the node information of the candidate warehouse node, the customer node and the vehicle status information as an input sequence into a preset deep reinforcement learning model, so that the deep reinforcement learning model uses the input sequence as a state quantity of deep reinforcement learning, and based on the state quantity, combined with the logistics scheduling optimization logic built into the deep reinforcement learning model, generates an action quantity, and the action quantity includes: warehouse site selection results and distribution path planning;

[0020] The scheduling execution and issuing unit is used to generate and issue logistics scheduling instructions according to the warehouse site selection results and the distribution route planning, so as to execute the logistics distribution tasks according to the logistics scheduling instructions.

[0021] Preferably, the deep reinforcement learning model comprises: an encoder and a plurality of decoders;

[0022] The encoding unit includes: an embedding layer, a multi-head attention layer and a forward layer;

[0023] The decoding unit includes: a simple recurrent unit, a linear layer, a multi-head attention layer and a single-head attention layer.

[0024] Preferably, the deep reinforcement learning model is specifically:

[0025]

[0026] In the formula, is a collection of warehouse nodes, is the set of client nodes, is a collection of delivery routes. is the customer subset, s i 、s j are the two-dimensional coordinates of node i and node j respectively, P is the route cost per unit distance, F is the fixed cost of each vehicle, and D j is the demand of the jth customer, C i is the capacity of the i-th warehouse, O i is the fixed cost of the i-th warehouse, L is the fixed load of each vehicle, x i is an indicator variable for whether warehouse i is used, e ijl is the indicator variable of whether node i and node j are connected in route l, u ij is an indicator variable indicating whether customer j is associated with warehouse i.

[0027] A third aspect of the present application provides a central scheduling server, including: a memory and a processor;

[0028] The memory is used to store program code, and the program code corresponds to a logistics scheduling optimization method based on deep reinforcement learning as described in the first aspect of the present application;

[0029] The processor is used for reading and executing the program code.

[0030] The fourth aspect of the present application provides a logistics scheduling optimization system based on deep reinforcement learning, including: a deliveryman terminal and a central scheduling server as described in the third aspect of the present application, wherein the central scheduling server is communicatively connected to each deliveryman terminal through a built-in communication network module.

[0031] Preferably, the communication network module specifically includes: a data transmission submodule, a network management and control submodule, a security and encryption submodule, and a fault monitoring and recovery submodule.

[0032] It can be seen from the above technical solutions that this application has the following advantages:

[0033] The solution provided in this application is based on a deep reinforcement learning algorithm. By constructing a unified deep reinforcement learning optimization model, it integrates the three key elements of facility site selection, personnel scheduling and path planning, and uses reinforcement learning and multi-objective optimization algorithms. Reinforcement learning continuously optimizes scheduling strategies through self-learning of intelligent agents in a dynamic environment. Multi-objective optimization achieves coordinated optimization of various parts of logistics scheduling by balancing cost and efficiency. It can achieve efficient scheduling of logistics networks and ensure the accuracy of scheduling decisions by continuously learning and adjusting strategies in a dynamic environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0035] Figure 1 A structural diagram of an embodiment of a logistics scheduling optimization system based on deep reinforcement learning provided in this application.

[0036] Figure 2 A flow chart of an embodiment of a logistics scheduling optimization method based on deep reinforcement learning provided in this application.

[0037] Figure 3 Schematic diagram of the deep reinforcement learning model architecture in the logistics scheduling optimization method provided in this application.

[0038] Figure 4 A schematic diagram of the encoder structure of the deep reinforcement learning model in the logistics scheduling optimization method provided in this application.

[0039] Figure 5A schematic diagram of the decoder structure of the deep reinforcement learning model in the logistics scheduling optimization method provided in this application.

[0040] Figure 6 This is a diagram showing the output sequence of feasible solutions for the logistics scheduling optimization method provided in this application.

[0041] Figure 7 A visualization diagram of the feasible solution output sequence of the logistics scheduling optimization method provided in this application.

[0042] Figure 8 A structural schematic diagram of an embodiment of a logistics scheduling optimization device based on deep reinforcement learning provided in this application. DETAILED DESCRIPTION

[0043] The embodiments of the present application provide a logistics scheduling optimization method and related equipment based on deep reinforcement learning, which are used to solve the technical problem that the existing logistics scheduling optimization technology is prone to fall into local optimality.

[0044] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0045] First, a detailed description of an embodiment of a logistics scheduling optimization system based on deep reinforcement learning provided by this application is as follows:

[0046] like Figure 1 As shown, the present application provides a logistics scheduling optimization system based on deep reinforcement learning, which is characterized by including: a deliveryman terminal PDA and a central scheduling server, the central scheduling server is communicatively connected with each deliveryman terminal through a built-in communication network module, the memory is used to store program code, the program code corresponds to a logistics scheduling optimization method based on deep reinforcement learning as provided in the present application, and the processor is used to read and execute the program code in the memory.

[0047] More specifically, the communication network module specifically includes: a data transmission submodule, a network management and control submodule, a security and encryption submodule, and a fault monitoring and recovery submodule;

[0048] The data transmission submodule is responsible for data transmission in various parts of the system, including data transmission between the courier's handheld PDA and the central dispatch server, and between the central dispatch server and the big data analysis module and optimization algorithm module. It ensures the consistency and synchronization of data between various parts of the system to avoid data redundancy and conflict.

[0049] Network management and control submodule: responsible for the management and control of the communication network to ensure the efficient operation and stability of the network. Manage the network structure and connection relationship, optimize the network topology to improve transmission efficiency. : Allocate and manage network bandwidth resources to ensure that the bandwidth requirements of key tasks are met.

[0050] Security and encryption submodule: Ensure the security and privacy of data transmission to prevent data leakage and illegal access. Use advanced encryption technology (such as AES, TLS, etc.) to encrypt the transmitted data to ensure the security of data during transmission. Develop and implement network security protocols to protect the integrity and confidentiality of data transmission.

[0051] Fault monitoring and recovery submodule: Real-time monitoring of network operation status, timely detection and handling of network faults, and ensuring system stability and reliability. Use monitoring tools and technologies (SNMP) to monitor the status of network devices and connections in real time and detect faults in a timely manner. Formulate fault recovery strategies and measures (such as redundant backup, automatic switching, etc.) to quickly restore network operation and reduce the impact of faults on the system.

[0052] The system uses PDA to collect and process data in real time, and combines big data analysis and optimization algorithms to provide intelligent decision-making support for logistics scheduling, so as to improve distribution efficiency and reduce operating costs.

[0053] The system mainly includes the courier's handheld PDA and the central dispatch server. The courier's handheld PDA is used to collect data in the delivery process in real time, such as order information, location information and operation records, and execute the dispatch instructions issued by the central dispatch server. The central dispatch server is the core of the system, responsible for processing and coordinating all logistics dispatch tasks. It receives data from the PDA and performs efficient data processing and real-time analysis through the big data analysis module. The data analysis module uses a distributed database system to store massive data, and uses machine learning and data mining techniques to mine patterns and laws in the data to provide support for the optimization algorithm. The optimization algorithm module uses a deep reinforcement learning algorithm to perform collaborative optimization for facility site selection, personnel scheduling and delivery route planning. This module uses a reinforcement learning framework to find the optimal scheduling strategy in a complex and dynamic logistics environment.

[0054] Then, a detailed description of the embodiment of the logistics scheduling optimization method based on deep reinforcement learning mentioned in the above-mentioned embodiment of the logistics scheduling optimization system based on deep reinforcement learning is given.

[0055] See also Figure 2 , this embodiment provides a logistics scheduling optimization method based on deep reinforcement learning, including:

[0056] Step 101: According to the received logistics order information, determine the customer location and order requirements corresponding to the customer node from the logistics order information;

[0057] Step 102: According to the order requirements, the inventory information of each warehouse in the logistics system is matched to determine a number of candidate warehouse nodes;

[0058] Step 103: input the node information of the candidate warehouse node and the customer node and the vehicle status information as an input sequence to a preset deep reinforcement learning model, so that the deep reinforcement learning model uses the input sequence as the state quantity of deep reinforcement learning, and generates an action quantity based on the state quantity and the logistics scheduling optimization logic built into the deep reinforcement learning model;

[0059] Among them, the action volume includes: warehouse location selection results and distribution route planning;

[0060] Step 104: Generate and issue logistics dispatch instructions based on the warehouse site selection results and distribution route planning, so as to execute logistics distribution tasks according to the logistics dispatch instructions.

[0061] It should be noted that the adaptive logistics scheduling system can dynamically adjust the delivery routes and vehicle scheduling strategies according to real-time order requirements and transportation conditions to reduce the operating costs of the logistics network and improve the distribution efficiency. Reinforcement learning is an efficient optimization control strategy. The intelligent agent collects logistics network information by observing the environment. This effective information can be used as a basis for the intelligent agent to make decisions that are more in line with the actual operating environment. Through continuous learning and optimization, the system can achieve the best scheduling and distribution plan under changing demands and conditions. The setting of the logistics scheduling system based on reinforcement learning generally includes three elements: state, action and reward.

[0062] State: It refers to the information observed by the agent in the environment. In the logistics scheduling system, the abstraction of state includes two parts: facilities (warehouses and customers) and vehicles. The locations of warehouses and customers are static, and the nodes in the graph can be used to represent the locations of potential warehouses and customers. Vehicles and order demands are dynamic, and the system can observe information such as vehicle location, load, current order demand, order delivery distance, and transportation time in units of time.

[0063] Action: The design of actions is mainly divided into warehouse site selection and vehicle path planning. Each action will determine which warehouse to start from and the specific choice of delivery route, and each path planning will at least guarantee a certain transportation efficiency. For example, the system can select a warehouse and assign a courier and vehicle to meet the needs of a batch of orders, while optimizing the delivery path of the vehicle to minimize transportation costs and time.

[0064] Return: It is an evaluation criterion for the quality of the decisions made by the logistics scheduling system within a decision cycle. When evaluating the scheduling strategy, it is mainly based on the changes in the operating conditions of the logistics network before and after the decision. For example, the total transportation cost reduced by this scheduling decision, the number of completed orders, the average delivery time, the efficiency of vehicle use, and customer satisfaction are evaluated. These indicators help the system evaluate the effectiveness of each scheduling decision and guide future optimization directions.

[0065] Through this adaptive model based on reinforcement learning, the logistics scheduling system can dynamically adjust the scheduling strategy under changing demands and conditions, achieve optimal resource allocation and path planning, and thus improve the operational efficiency and service level of the entire logistics network. The model architecture is as follows: Figure 3 shown.

[0066] According to the optimization method provided in this embodiment, the optimization system can extract relevant node information of the customer node, such as the customer location and order requirements, from the logistics order information based on the received logistics order information; then, based on the order requirements obtained in the previous step, combined with the inventory information of each warehouse in the logistics system, a match is performed to determine several warehouses that can undertake these order shipments as alternative warehouse nodes.

[0067] like Figure 3 The deep reinforcement learning model shown is based on the above-mentioned deep reinforcement learning model, and takes the node information of the candidate warehouse nodes, customer nodes and the vehicle status information as the model input and the state quantity in the deep reinforcement learning framework, and combines the logistics scheduling optimization logic built into the deep reinforcement learning model to generate the action quantity in the deep reinforcement learning framework. The action quantity mainly includes: warehouse site selection results and distribution path planning. Then, according to the warehouse site selection results and distribution path planning, the logistics scheduling instructions are generated and issued to realize the generated action quantity acting on the environment. At the same time, according to the deep reinforcement learning framework, after the action quantity acts on the environment, it can be re-acted on the intelligent body according to the state quantity and reward function of the next step length generated, that is, the deep reinforcement learning model acts on the environment after generating the next step length action quantity, thereby realizing the cyclic collaborative optimization of facility site selection, personnel scheduling and distribution route planning, balancing the relationship between various goals, and improving the adaptability of logistics scheduling to complex and dynamic logistics environments.

[0068] More specifically, after the action is applied to the environment, the implementation methods of deep reinforcement learning model reward optimization include:

[0069] Step 105: Count the dispatch statistics data within a preset time period before and after the issuance of the logistics dispatch instruction, and then use the deviation of the statistical data before and after the issuance of the instruction as the reward amount of deep reinforcement learning, so as to determine the model adjustment parameters of the deep reinforcement learning model according to the reward amount combined with the preset reward function;

[0070] In this way, the model parameters of the deep reinforcement learning model can be adjusted and optimized, and the coordinated optimization of facility site selection, personnel scheduling and distribution route planning can be achieved. The relationship between various objectives can be balanced through the optimization framework to find the optimal compromise solution.

[0071] More specifically, the above dispatch statistics mainly include: total transportation cost, number of completed orders, average delivery time and vehicle utilization efficiency.

[0072] Take the following logistics scheduling scenario as an example: given a set of customers with known needs and their locations and a set of potential warehouses, the locations of the warehouses and the vehicle routes from the warehouses to the customers are determined, and each customer is assigned to a warehouse, which will dispatch a certain number of couriers to drive vehicles for delivery, with the goal of minimizing the total cost, including the operating costs of the facilities, the cost of using the vehicles, and the transportation costs. Next, we will introduce how to solve the site selection-path planning problem using the method of this application. First, the site selection-path planning problem is modeled as a Markov decision process; secondly, the structure of the model used, that is, the encoder-decoder structure, is introduced; then the multiple optimal strategy optimization algorithm used to train the model will be introduced.

[0073] The deep reinforcement learning model of this embodiment: The deep reinforcement learning model of this embodiment specialized for the logistics scheduling system is shown in formula (1); constraints (2) and (3) ensure that each selected distribution center is connected to at most one route from the warehouse; constraint (4) represents flow protection; constraint (5) ensures that the total demand of distribution centers on each route is less than the vehicle load; constraint (6) ensures that the capacity of each warehouse is greater than the total demand of distribution centers assigned to the warehouse; constraint (7) prevents connection between warehouses; constraint (8) assigns distribution centers to warehouses; constraint (9) prevents the formation of sub-loops.

[0074]

[0075]

[0076] In the formula, is a collection of warehouse nodes, is the set of client nodes, is a collection of delivery routes. is the customer subset, s i 、s j are the two-dimensional coordinates of node i and node j respectively, P is the route cost per unit distance, F is the fixed cost of each vehicle, and D j is the demand of the jth customer, C i is the capacity of the i-th warehouse, O i is the fixed cost of the i-th warehouse, L is the fixed load of each vehicle, x i is an indicator variable for whether warehouse i is used. If warehouse i is used (open), the value is 1, otherwise it is 0; e ijl u is an indicator variable indicating whether node i and node j are connected on route l. If node i and node j are connected on route l, the value is 1; otherwise, it is 0. ij is an indicator variable for whether customer j is associated with warehouse i. If customer j is assigned to warehouse i, the value is 1; otherwise, it is 0.

[0077] like Figure 3 The deep reinforcement learning model shown in the figure, in which the encoder is designed to obtain the embedding of the node set in the input instance. After that, the decoder receives the node embedding of the encoder, iteratively calculates the probability of the node to be selected (i.e., the random strategy pθ(π|G(N)), selects the node from it in each iteration, and generates a node sequence.

[0078] Specifically, firstly, the location and capacity of the candidate warehouses, the location and demand of customers, the vehicle capacity and other information are obtained through the data analysis module, and then this information is converted into a node sequence as the model input. The node sequence is defined as: X = {x i =(s i ,d i ,c i ,o i ,P)|i∈{1,...,D,...,n}}, where s i represents the two-dimensional coordinate, d i Indicates customer demand, c i represents the warehouse capacity, o i represents the cost of opening a warehouse, and P represents the vehicle capacity. For example, warehouse x0 can be represented as {(23,13),0,134,2134,22}, and customer x9 can be represented as {(11,22),12,0,0,0}. In this way, the input data information is abstracted into a node sequence.

[0079] Then, by Figure 4 The encoder shown embeds the nodes. First, different embedding layers are used to embed the warehouse and customer nodes. The embedding formulas are shown in formulas (10) and (11). In this embodiment, the dimension of the hidden state is preferably set to d h=128.

[0080]

[0081] Among them, W d and b d represents the network parameters for embedding warehouse nodes, W c and b c represents the network parameters for embedding client nodes, The warehouse and customer node information is projected into a high-dimensional space. In this way, the model can obtain more information. The multi-head attention sublayer and the feedforward sublayer together constitute the multi-head attention module. In order to prevent the gradient vanishing problem, residual connections are added to the multi-head attention sublayer and the feedforward sublayer. In order to improve the training stability of the neural network and speed up the convergence of the model, the output data is normalized after the multi-head attention sublayer and the feedforward sublayer, that is, the BN layer is added. The specific calculation formula is shown in formula (12-13):

[0082]

[0083] in, represents the multi-head attention layer, represents the BN layer, represents the feedforward sublayer. In this way, the connections between nodes are obtained, and through these connections, the model can make correct decisions. Finally, the embeddings of all nodes are aggregated to obtain the graph embedding To obtain information fusion, the calculation result is shown in formula (14):

[0084]

[0085] Next, after the encoder performs feature abstraction and information fusion on the input data, this embodiment uses Figure 5 The decoder shown in the figure decodes to obtain the solution of the model. Using a step-by-step decoding method, each time a node π is selected i , and select the next node π based on the selected node i+1 , until the complete solution sequence π={π1,π2,...,π n Specifically, the graph embedding obtained by the encoder, the first node embedding, the previous node embedding, and the global vector form the context as the input of the encoder, and then pass through a simple recurrent unit, a linear layer, a multi-head attention layer, and a single-head attention layer as a strategy function for node selection.

[0086] First, the decoder context Will be embedded by the graph Node embeddings at the first time step Node embeddings of the output nodes at the previous time step and the two-dimensional global vector g t =(c f,t ,L t ) T Composition, where c f,t is the remaining capacity of the current facility (warehouse or transfer station), L t is the remaining load of the vehicle on the current path at decoding step t. Its calculation method is shown in formula (15-17):

[0087]

[0088] Next, a simple recurrent unit is used to map the context, where the gate u is updated t and reset gate f t Deciding how much past information should be updated or forgotten allows the network to selectively incorporate relevant information from earlier time steps.

[0089]

[0090]

[0091] Then, two linear layers specialized for different stages are applied to extract different information. The linear layers allow capturing different features specific to each stage.

[0092]

[0093] Finally, a multi-head attention (MHA) layer is used to map the context to d h The context is operated by the query, key and value in the MHA layer, and the probability distribution p of the output result is θ Obtained through a single-head attention layer. The main update equation is as follows, where Φ t is the set of mask nodes during decoding:

[0094]

[0095] After obtaining the probability distribution p of the output node θ Finally, sampling is used to select each node based on the probability distribution. For example, the final generated value is [0.6, 0.1, 0.1, 0, 0, 0.2], then the probability of node 1 being selected and added to the partial solution is 0.6, and node 2 is added to the partial solution with a probability of 0.1, and so on.

[0096] After passing through the decoder, a complete output sequence can be obtained such as Figure 6 As shown in Figure 1, where squares represent warehouses and circles represent customers. You can also visualize the solution as Figure 7As shown in the figure, squares represent warehouses and dots represent customer nodes. This model can be used to generate scheduling solutions.

[0097] The above is a detailed description of an embodiment of a logistics scheduling optimization method based on deep reinforcement learning provided by the present application. The following is a detailed description of an embodiment of a logistics scheduling optimization device based on deep reinforcement learning provided by the present application.

[0098] See also Figure 8 , this embodiment provides a logistics scheduling optimization device based on deep reinforcement learning, including:

[0099] The order response processing unit 201 is used to determine the customer location and order requirements corresponding to the customer node from the logistics order information according to the received logistics order information;

[0100] The warehouse node determination unit 202 is used to match the order requirements with the inventory information of each warehouse in the logistics system to determine a number of candidate warehouse nodes;

[0101] The model operation unit 203 is used to input the node information of the candidate warehouse node and the customer node and the vehicle status information as an input sequence into a preset deep reinforcement learning model, so that the deep reinforcement learning model uses the input sequence as the state quantity of deep reinforcement learning, and based on the state quantity, combined with the logistics scheduling optimization logic built into the deep reinforcement learning model, generates an action quantity, and the action quantity includes: warehouse site selection results and distribution path planning;

[0102] The scheduling execution and issuing unit 204 is used to generate and issue logistics scheduling instructions according to the warehouse site selection results and the distribution route planning, so as to execute the logistics distribution tasks according to the logistics scheduling instructions.

[0103] Furthermore, it also includes:

[0104] The reward optimization unit 205 is used to count the scheduling statistics within a preset time period before and after the issuance of the logistics scheduling instructions, and then use the statistical data deviation before and after the instruction is issued as the reward amount of deep reinforcement learning, so as to determine the model adjustment parameters of the deep reinforcement learning model based on the reward amount combined with the preset reward function.

[0105] More specifically, the deep reinforcement learning model includes: an encoder and multiple decoders;

[0106] The encoding unit includes: embedding layer, multi-head attention layer and forward layer;

[0107] The decoding unit includes: simple recurrent unit, linear layer, multi-head attention layer and single-head attention layer.

[0108] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0109] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0110] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0111] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0112] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0113] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0114] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A logistics scheduling optimization method based on deep reinforcement learning, characterized in that: include: According to the received logistics order information, determine the customer location and order requirements corresponding to the customer node from the logistics order information; According to the order requirements, the inventory information of each warehouse in the logistics system is matched to determine several candidate warehouse nodes; The node information of the candidate warehouse node and the customer node and the vehicle status information are input as an input sequence to a preset deep reinforcement learning model, so that the deep reinforcement learning model uses the input sequence as a state quantity of deep reinforcement learning, and based on the state quantity, combined with the logistics scheduling optimization logic built into the deep reinforcement learning model, generates an action quantity, and the action quantity includes: warehouse site selection results and distribution path planning; Based on the warehouse site selection results and the distribution route planning, a logistics scheduling instruction is generated and issued to execute the logistics distribution task according to the logistics scheduling instruction.

2. The logistics scheduling optimization method based on deep reinforcement learning according to claim 1 is characterized in that: Also includes: The scheduling statistical data within a preset time period before and after the issuance of the logistics scheduling instruction is counted, and then the deviation of the statistical data before and after the instruction is issued is used as the reward amount of deep reinforcement learning, so as to determine the model adjustment parameters of the deep reinforcement learning model according to the reward amount combined with a preset reward function.

3. The logistics scheduling optimization method based on deep reinforcement learning according to claim 1 is characterized in that: The deep reinforcement learning model is specifically: In the formula, is a collection of warehouse nodes, is the set of client nodes, is a collection of delivery routes. is the customer subset, s i 、s j are the two-dimensional coordinates of node i and node j respectively, P is the route cost per unit distance, F is the fixed cost of each vehicle, and D j is the demand of the jth customer, C i is the capacity of the i-th warehouse, O i is the fixed cost of the i-th warehouse, L is the fixed load of each vehicle, x i is an indicator variable of whether warehouse i is used, e ijl is the indicator variable of whether node i and node j are connected in route l, u ij is an indicator variable indicating whether customer j is associated with warehouse i.

4. The logistics scheduling optimization method based on deep reinforcement learning according to claim 2 is characterized in that: The dispatch statistics include: total transportation cost, number of completed orders, average delivery time and vehicle utilization efficiency.

5. A logistics scheduling optimization device based on deep reinforcement learning, characterized in that: include: An order response processing unit, configured to determine a customer location and order requirements corresponding to a customer node from the logistics order information according to the received logistics order information; A warehouse node determination unit, used to match the order demand with the inventory information of each warehouse in the logistics system to determine a number of candidate warehouse nodes; A model operation unit, used to input the node information of the candidate warehouse node, the customer node and the vehicle status information as an input sequence into a preset deep reinforcement learning model, so that the deep reinforcement learning model uses the input sequence as a state quantity of deep reinforcement learning, and based on the state quantity, combined with the logistics scheduling optimization logic built into the deep reinforcement learning model, generates an action quantity, and the action quantity includes: warehouse site selection results and distribution path planning; The scheduling execution and issuing unit is used to generate and issue logistics scheduling instructions according to the warehouse site selection results and the distribution route planning, so as to execute the logistics distribution tasks according to the logistics scheduling instructions.

6. A logistics scheduling optimization device based on deep reinforcement learning according to claim 5, characterized in that: The deep reinforcement learning model includes: an encoder and a plurality of decoders; The encoding unit includes: an embedding layer, a multi-head attention layer and a forward layer; The decoding unit includes: a simple recurrent unit, a linear layer, a multi-head attention layer and a single-head attention layer.

7. The logistics scheduling optimization device based on deep reinforcement learning according to claim 5 is characterized in that: The deep reinforcement learning model is specifically: In the formula, is a collection of warehouse nodes, is the set of client nodes, is a collection of delivery routes. is the customer subset, s i 、s j are the two-dimensional coordinates of node i and node j respectively, P is the route cost per unit distance, F is the fixed cost of each vehicle, and D j is the demand of the jth customer, C i is the capacity of the i-th warehouse, O i is the fixed cost of the i-th warehouse, L is the fixed load of each vehicle, x i is an indicator variable of whether warehouse i is used, e ijl is the indicator variable of whether node i and node j are connected in route l, u ij is an indicator variable indicating whether customer j is associated with warehouse i.

8. A central dispatch server, characterized in that: include: Memory and processor; The memory is used to store program code, and the program code corresponds to a logistics scheduling optimization method based on deep reinforcement learning according to any one of claims 1 to 4; The processor is used for reading and executing the program code.

9. A logistics scheduling optimization system based on deep reinforcement learning, characterized in that: include: The delivery person terminal and the central dispatch server as claimed in claim 8, wherein the central dispatch server is communicatively connected with each delivery person terminal via a built-in communication network module.

10. A logistics scheduling optimization system based on deep reinforcement learning according to claim 9, characterized in that: The communication network module specifically includes: a data transmission submodule, a network management and control submodule, a security and encryption submodule, and a fault monitoring and recovery submodule.

Images