A warehouse logistics scheduling method based on deep reinforcement learning

Abstract

This invention discloses a warehouse logistics scheduling method based on deep reinforcement learning, including the construction of a scheduling dataset, the construction and training of a deep neural network model, and the generation of transportation routes. The deep neural network includes a pointer network and a graph neural network, used to output transportation point probability vectors and heatmap vectors, respectively. A first reward / penalty value is constructed based on the distance between the transportation point probability vector and the two-dimensional coordinates, and a second reward / penalty value is constructed based on the cross-entropy between the heatmap vector and the target two-dimensional heatmap vector. The two reward / penalty values ​​are applied to a loss function for backpropagation. The transportation route generation utilizes the local optimality of the traveling salesman problem, effectively solving the problem that adjacent transportation points in the optimal transportation route may be far apart in transportation routes obtained by existing neural network methods by exchanging transportation point positions.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of warehouse logistics scheduling, and in particular to a warehouse logistics scheduling method based on deep reinforcement learning. BACKGROUND

[0002] Warehouse logistics path scheduling problems exist widely in various industries, such as scheduling of activities such as warehousing, warehousing and transportation of raw materials, semi-finished products and finished products in the production and manufacturing industry; planning of transportation routes and schedules of vehicles, ships and aircrafts in the logistics distribution industry; and scheduling of transportation and transfer of goods in logistics hubs such as airports and ports. Therefore, an efficient and accurate path scheduling method can achieve more reasonable resource allocation and bring great convenience to related industries and fields.

[0003] Most of the existing logistics scheduling methods are to solve the logistics scheduling problem by abstracting it into a traveling salesman model (TSP model), and the solution schemes are mainly divided into two categories: artificial solution method and heuristic solution method. Artificial solution method includes branch and bound method, cut plane method, integer programming and dynamic programming algorithm, etc. This kind of algorithm usually needs to manually design features for the problem, and makes decisions according to the manually designed features. Although a better result can be obtained, the calculation complexity is high, and it is difficult to meet the real-time scheduling demand; the heuristic solution method includes A* algorithm, simulated annealing algorithm, genetic algorithm, ant colony algorithm and neural network algorithm, etc. This kind of algorithm usually uses heuristic function and iterative optimization method, and uses heuristic information to guide the search direction in the search process, in order to find a satisfactory solution in a short time. These methods are more universal and real-time than the artificial solution method. However, the heuristic solution method may not guarantee the quality of the solution, especially in large-scale problems, the decision space grows exponentially, and it is difficult to make better decisions in the vast decision space. SUMMARY

[0004] The present application aims to solve the problems of the current logistics path scheduling method, and proposes a warehouse logistics path scheduling method based on deep reinforcement learning under large-scale conditions.

[0005] The present application mainly solves the warehouse logistics path scheduling problem under large-scale conditions in the following ways: abstracting the warehouse logistics path scheduling problem into a traveling salesman model, constructing and training a neural network model, and using the neural network model to gradually optimize the solution from the local based on local optimality, thereby effectively improving the quality of the solution.

[0006] The purpose of the present application is achieved by the following technical scheme:

[0007] A warehouse logistics scheduling method based on deep reinforcement learning, the method comprising:

[0008] S1, scheduling dataset construction: using the traveling salesman problem solver to construct a scheduling dataset including a target two-dimensional coordinate vector and a target two-dimensional heat map vector;

[0009] S2, deep neural network model construction, including:

[0010] S2.1, constructing a first reward and punishment value according to the transportation point probability vector and the two-dimensional coordinate distance length;

[0011] S2.2, constructing a pointer network with the input of the two-dimensional coordinate vector and the transportation point probability vector, and the output of the transportation point probability vector, taking the output of the pointer network as the input of the next round, repeating N times, combining the first reward and punishment value to construct a loss function to train the pointer network, and obtaining a decision model;

[0012] S2.3, constructing a second reward and punishment value according to the cross entropy of the heat map vector and the target two-dimensional heat map vector;

[0013] S2.4, constructing a graph neural network with the input of the two-dimensional coordinate vector and the output of the two-dimensional heat map vector, combining the second reward and punishment value to construct a loss function to train the pointer network, and obtaining a correlation model;

[0014] S3, constructing a two-dimensional coordinate vector according to the transportation point position in the warehouse transportation task, and inputting it into the trained decision model and correlation model to obtain an initial path vector and a heat map vector,

[0015] According to the transportation point set in the heat map vector and the convex hull vector of the two-dimensional coordinate vector, the initial path vector is adjusted to obtain a to-be-corrected transportation path vector, which is input into the decision model to obtain a corrected transportation path vector, which is compared with the initial path vector, and the shortest transportation path is saved;

[0016] According to the shortest transportation path, the to-be-corrected transportation path vector is reconstructed, and after a predetermined number of cycles, the shortest path is retained as the transportation path.

[0017] Further, the scheduling dataset construction is specifically: randomly generating a two-dimensional coordinate vector of Nx2, inputting the two-dimensional coordinate vector into the traveling salesman problem solver concorde to obtain a target transportation path vector of Nx1, and generating a two-dimensional heat map vector of NxN according to the transportation point connection order of the target transportation path vector.

[0018] Further, the construction of the first reward and punishment value specifically includes:

[0019]

[0020] Where a i The transportation point with the maximum probability value in the transportation point probability vector output by the neural network, P1(a ia0...a i-1 ) denotes the probability of the i-1th transportation point to the i th transportation point output by the decision model, Dist(a i-1 , a i ) denotes the Euclidean distance of the i-1th transportation point to the i th transportation point.

[0021] Further, the constructing the second reward value specifically includes:

[0022]

[0023] wherein the heatmap target two-dimensional heat map vector, p2 ij is a two-dimensional heat map vector output by the association model.

[0024] Further, the pointer network is an attention model, and the training of the attention model specifically includes: combining the two-dimensional coordinate vector, taking the first reward value as a loss function, and using the adam algorithm to update the parameters through back propagation.

[0025] Further, the graph neural network is a graph attention neural network, and the training of the graph attention neural network specifically includes: combining the target heat map vector in the scheduling data set, taking the second reward value as a loss function, and using the adam algorithm to update the parameters through back propagation.

[0026] Further, the obtaining of the initial path vector specifically includes: inputting the constructed two-dimensional coordinate vector into the decision model to obtain N transportation point probability vectors, and taking the transportation point with the maximum probability value in each vector as the initial transportation path.

[0027] Further, the obtaining of the to-be-corrected transportation path vector specifically includes: first calculating a convex hull vector of the two-dimensional coordinate vector, and then screening N largest elements p2 ij , merging the i th transportation point and the j th transportation point corresponding to the elements into the same transportation point set, deleting the transportation points in the transportation point set from the initial transportation path, and adding the transportation point set to any transportation point in the convex hull vector at the position corresponding to the initial transportation path; finally, the number of to-be-corrected transportation paths is the same as the number of transportation points in the convex hull vector.

[0028] According to a second aspect of the present specification, a warehouse logistics scheduling device based on deep reinforcement learning is provided, which includes a memory and one or more processors, the memory stores executable code, and the processor executes the executable code to implement the warehouse logistics scheduling method based on deep reinforcement learning.

[0029] According to a third aspect of the present specification, a computer readable storage medium is provided, and a program is stored on the computer readable storage medium, and the program is executed by a processor to implement the warehouse logistics scheduling method based on deep reinforcement learning.

[0030] The present application has the following advantages:

[0031] 1. The present application provides a framework for solving large-scale traveling salesman problems by dividing them into small-scale traveling salesman problems, effectively improving logistics transportation efficiency.

[0032] 2. According to the characteristics of high correlation accuracy of graph neural network and high decision accuracy of pointer network, the pointer network is used to sort the high correlation transportation point set under the high correlation transportation point set aggregated by the graph neural network, and the advantages of the two types of neural networks are effectively utilized.

[0033] 3. By exchanging the positions of transportation points, the problem that adjacent transportation points in the optimal transportation path may be far apart in the transportation path obtained by the existing neural network method is effectively solved. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 A deep reinforcement learning warehouse logistics scheduling method flowchart is provided for the present application example;

[0035] Figure 2 An implementation block diagram is provided for the present application example;

[0036] Figure 3 A transportation path generation implementation block diagram is provided for the present application example;

[0037] Figure 4 A deep reinforcement learning warehouse logistics scheduling device schematic diagram is provided for the present application example. DETAILED DESCRIPTION

[0038] The specific embodiments of the present application are further described in detail below with reference to the accompanying drawings.

[0039] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the scope of the present application, therefore the present application is not limited to the specific embodiments disclosed below.

[0040] As Figure 1 shown, the present application provides a specific embodiment of the present application to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand.

[0041] This embodiment provides a warehouse logistics route scheduling method, the steps of which are as follows:

[0042] Step 1: Constructing the scheduling dataset. Use Python to randomly generate an Nx2 two-dimensional coordinate vector. Input the two-dimensional coordinate vector into Concorde to obtain an Nx1 target transportation path vector. Generate an NxN two-dimensional heatmap vector based on the order of the transportation points in the target transportation path vector.

[0043] For example: a two-dimensional coordinate vector is {{x0, y0}, {x1, y1}, ..., {x...} N-1 y N-1 The target transportation path vector is {a0, a1, ..., a...}. N-1}, heatmap in two-dimensional heatmap vector 0，1 heatmap 1，2 , ..., heatmap N-2，N-1 The element is 1, and the rest are 0.

[0044] Step 2: Deep Neural Network Model Construction and Training, including the following sub-steps

[0045] 2.1 Establishing a pointer network model and reward / penalty mechanism: First, constructing the reward / penalty value R1 based on the probability vector of the transportation point and the distance between the two-dimensional coordinates.

[0046] 2.2 Take the two-dimensional coordinate vector from step one and the transportation point probability vector output by the pointer network in step 2.1 as inputs and the transportation point probability vector as outputs, loop N times, construct a loss function using the reward and punishment mechanism in step 2.1, train the pointer network, and obtain the decision model.

[0047] 2.3 Establishing a Graph Neural Network and Reward / Punishment Mechanism II: Constructing the reward / punishment value R2 based on the cross-entropy between the output heatmap vector and the target heatmap vector.

[0048] 2.4 Using the two-dimensional coordinate vector from step one as input and the heatmap vector as output, a loss function is constructed using the reward and punishment mechanism two from step 2.3. The graph neural network described in step 2.3 is then trained to obtain the association model.

[0049] The specific implementation details are as follows:

[0050] a) Construct a pointer network, preferably an attention model. Construct reward / penalty values.

[0051] Where a i P1(a) refers to the transportation point with the highest probability value in the transportation point probability vector output by the pointer network in step 2.1. i \a0...ai-1 ) the probability of the i-1th transportation point to the ith transportation point output by the decision model, Dist(a i-1 , a i ) the Euclidean distance of the i-1th transportation point to the ith transportation point.

[0052] For example:

[0053] The N transportation point probability vectors are

[0054] {{P1(a0\a0), P1(a1\a0),..., P1(a N-1 \a0)},

[0055] {P1(a0\a0a1), P1(a1\a0a1),..., P1(a N-1 \a0a1)},

[0056] ...,

[0057] {P1(a0\a0...a N-1 ), P1(a1\a0...a N-1 ),..., P1(a N-1 \a0...a N-1 )}

[0058] For each transportation point probability vector

[0059] {P1(a0\a0...a K-1 ), P1(a1\a0...a K-1 ),..., P1(a N-1 \a0...a K-1 )}

[0060] Multiply the maximum P1(a i \a0...a K-1 ) with Dist(a i-1 , a i ) and accumulate to get the reward and punishment value R1.

[0061] b) input the two-dimensional coordinate vector, the transportation point probability vector output by the pointer network last time (the first time is a full zero vector), get the Nx1 transportation point probability vector, loop N times to get N transportation point probability vectors. Combine the two-dimensional coordinate vector, get the reward and punishment value according to step 2.1 as the loss function, and use the adam algorithm to update the parameters by back propagation.

[0062] c) construct a graph neural network, preferably a graph attention networks. Construct the reward and punishment value

[0063]

[0064] wherein heatmap is the target two-dimensional heat map vector in step one, p2 ij is the two-dimensional heat map vector output by the association model in step 2.3.

[0065] d) input the two-dimensional coordinate vector to obtain the two-dimensional heat map vector. Combine the target heat map vector in the scheduling data set, and obtain the reward and punishment value R2 as the loss function according to step 2.3, and use the adam algorithm to update the parameters by back propagation.

[0066] Step three: transportation path generation, including the following sub-steps

[0067] 3.1 Construct the corresponding two-dimensional coordinate vector according to the position of the transportation point in the warehouse transportation task

[0068] 3.2 Input the two-dimensional coordinate vector in step 3.1 into the decision model obtained in step 2.2 to obtain an initial transportation path vector

[0069] 3.3 Based on the scipy algorithm library, calculate the convex hull vector of the two-dimensional coordinate vector in step 3.1;

[0070] 3.4 Input the two-dimensional coordinate vector in step 3.1 into the association model obtained in step 2.4 to obtain a heat map vector

[0071] 3.5 Based on the heat map vector obtained in step 3.4, merge the transportation points into a transportation point set.

[0072] 3.6 Based on the convex hull vector obtained in step 3.3 and the transportation point set obtained in step 3.5, obtain a to-be-corrected transportation path vector

[0073] 3.7 Input the to-be-corrected transportation path vector obtained in step 3.6 into the decision model obtained in step 2.2 to obtain a corrected transportation path vector, and compare it with the original transportation path vector. If there is an improvement, replace the original transportation path vector with the corrected transportation path vector

[0074] 3.8 Repeat steps 3.6 and 3.7 T times to output the final transportation path vector, as shown in Figure 3 .

[0075] The specific implementation details are as follows:

[0076] a) Obtain the warehouse logistics plan, establish a two-dimensional rectangular coordinate system with the lower left corner of the obtained warehouse logistics plan as the origin, and obtain the coordinate positions of the warehouse and each transportation point in the two-dimensional rectangular coordinate system, and scale the coordinates so that the coordinate values are in [0, 1].

[0077] For example, the original coordinate vector is obtained according to the warehouse logistics plan

[0078] {{x0, y0}, {x1, y1},..., {x N-1 , y N-1}}. N-1 N-1}}.

[0079] Calculate x max = max(x0, x1,..., x N -1), y max = max(y0, y1,..., y N-1 )

[0080] Calculate the scaled coordinate vector as {{x0 / x max , y0 / y max}, {x1 / x max , y1 / y max},..., {x N-1 / x max , y N-1 / y max}}.

[0081] b) For the constructed two-dimensional coordinate vector, input the decision model to obtain N transport point probability vectors, and take the transport point with the maximum probability value in each vector as the initial transport path.

[0082] For example, the initial transport path is {a0, a1,..., a N-1}

[0083] c) Use the scipy, spatial.ConvexHull function in the scipy algorithm library to input the constructed two-dimensional coordinate vector to obtain a convex hull.

[0084] For example, the two-dimensional coordinate vector is {{x0, y0}, {x1, y1},..., {x N-1 , y N-1}}

[0085] The convex hull vector is {a0, a 30 , a 37 , a 50 , a 78 , a0}

[0086] d) For the constructed two-dimensional coordinate vector, input the association model to obtain a two-dimensional heat map vector.

[0087] e) Use the constructed heat map vector to find the N largest elements p2 ij , and merge the corresponding i-th transport point and j-th transport point into the same transport point set.

[0088] f) using the obtained set of transportation points, deleting the transportation points in the set of transportation points from the original transportation path, and adding them to the K arbitrary transportation points of the convex hull in step 3.3, to obtain K modified transportation paths.

[0089] For example, a set of transportation points is {a3, a 38 , a 39 , a 45}, and the set of transportation points is deleted from the initial transportation path {a0, a1,..., a N-1}, and after adding the set of transportation points to a0 in the convex hull vector of the initial transportation path, the modified path {a0, a3, a 38 , a 39 , a 45 , a1,..., a N-1} is obtained. After adding them to a 30 in the convex hull vector, the modified path {a0, a1,..., a 30 , a3, a 38 , a 39 , a 45 , a 31 , a 32 ,..., a N-1} is obtained.

[0090] g) using the different parts of the K modified transportation paths obtained, inputting the decision model, obtaining K modified transportation paths, calculating the transportation point distance of the K modified transportation paths from the original path, and keeping the path with the shortest distance.

[0091] For example, the modified transportation path {a0, a3, a 38 , a 39 , a 45 , a1,..., a N-1} is different from the modified path {a0, a1,..., a 30 , a3, a 38 , a 39 , a 45 , a 31 , a 32 ,..., a N-1}, a0,..., a 30 and a 30 ,..., a 37 part of the transportation path, and the transportation point coordinates corresponding to the two parts of the transportation path are input into the decision model to obtain the modified transportation path, the Euclidean distance of the modified transportation path is calculated, and the transportation path with shorter distance is saved.

[0092] Based on the shortest path retained, K paths to be corrected are reconstructed as the initial transportation path. This process is repeated T times until the final path is retained as the transportation path.

[0093] Corresponding to the aforementioned embodiment of a warehouse logistics scheduling method based on deep reinforcement learning, the present invention also provides an embodiment of a warehouse logistics scheduling device based on deep reinforcement learning.

[0094] See Figure 4 The present invention provides a warehouse logistics scheduling device based on deep reinforcement learning, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements a warehouse logistics scheduling method based on deep reinforcement learning in the above embodiment.

[0095] The embodiment of a warehouse logistics scheduling device based on deep reinforcement learning provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 4 The diagram shown is a hardware structure diagram of any data processing-capable device, including the deep reinforcement learning-based warehouse logistics scheduling device provided by this invention. (Except for...) Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0096] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0097] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0098] The embodiment of the present application also provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the warehouse logistics scheduling method based on deep reinforcement learning in the above embodiment.

[0099] The computer readable storage medium can be an internal storage unit of any data processing device in the above embodiments, for example, a hard disk or a memory. The computer readable storage medium can also be an external storage device of any data processing device, for example, a plug-in hard disk, a smart media card (SMC), an SD card, a flash card and the like. Further, the computer readable storage medium can include both the internal storage unit and the external storage device of any data processing device. The computer readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0100] The present application also provides a computer program product, comprising computer programs / instructions, which are executed by a processor to realize the warehouse logistics scheduling method based on deep reinforcement learning.

[0101] The above embodiments are used to explain and illustrate the present application, but not to limit the present application, and any modifications and changes made to the present application within the spirit and protection scope of the claims fall within the protection scope of the present application.

Claims

1. A warehouse logistics scheduling method based on deep reinforcement learning, characterized in that, The method comprises: S1, scheduling data set construction: using a traveling salesman problem solver to construct a scheduling data set comprising a target two-dimensional coordinate vector and a target two-dimensional heat map vector; S2, deep neural network model construction, comprising: S2.1, constructing a first reward and punishment value according to the transportation point probability vector and the two-dimensional coordinate distance length; S2.2, constructing a pointer network with the input being a two-dimensional coordinate vector and a transportation point probability vector, and the output being a transportation point probability vector, taking the output of the pointer network as the input of the next round, repeating N times, training the pointer network in combination with the first reward and punishment value to obtain a decision model; S2.3, constructing a second reward and punishment value according to the cross entropy of the heat map vector and the target two-dimensional heat map vector; S2.4, constructing a graph neural network with the input being a two-dimensional coordinate vector and the output being a two-dimensional heat map vector, and training the graph neural network in combination with the second reward and punishment value to obtain a correlation model; S3, constructing a two-dimensional coordinate vector according to the transportation point positions in the warehouse transportation task, inputting the two-dimensional coordinate vector into the trained decision model and correlation model respectively to obtain an initial path vector and a heat map vector, adjusting the initial path vector according to the transportation point set in the heat map vector and the convex hull vector of the two-dimensional coordinate vector to obtain a to-be-corrected transportation path vector, inputting the to-be-corrected transportation path vector into the decision model to obtain a corrected transportation path vector, comparing the corrected transportation path vector with the initial path vector, and saving the shortest transportation path; reconstructing the to-be-corrected transportation path vector according to the shortest transportation path, and retaining the shortest path as the transportation path after a predetermined number of cycles. 2.The warehouse logistics scheduling method based on deep reinforcement learning according to claim 1, wherein, The scheduling data set construction is specifically: randomly generating a two-dimensional coordinate vector of Nx2, inputting the two-dimensional coordinate vector into the traveling salesman problem solver concorde to obtain a target transportation path vector of Nx1, and generating a two-dimensional heat map vector of NxN according to the connection order of the transportation points of the target transportation path vector. 3.The warehouse logistics scheduling method based on deep reinforcement learning of claim 1, wherein, The construction of the first reward and punishment value specifically includes: ； wherein denotes the transport point with the largest probability value in the probability vector of the transport point output by the neural network, denotes the probability of the i-1th transport point to the ith transport point output by the decision model, denotes the Euclidean distance of the i-1th transport point to the ith transport point. 4.The warehouse logistics scheduling method based on deep reinforcement learning of claim 1, wherein, The construction of the second reward and punishment value specifically includes: wherein the heatmap target two-dimensional heat map vector, is a two-dimensional heat map vector output by the association model. 5.The warehouse logistics scheduling method based on deep reinforcement learning of claim 3, wherein, The pointer network is an attention model, and the training of the attention model is specifically: combining the two-dimensional coordinate vector, taking the first reward and punishment value as a loss function, and using the adam algorithm to update the parameters by back propagation. 6.The warehouse logistics scheduling method based on deep reinforcement learning according to claim 4, wherein, The graph neural network is a graph attention neural network, and the training of the graph attention neural network is specifically: combining the target heat map vector in the scheduling data set, taking the second reward and punishment value as a loss function, and using the adam algorithm to update the parameters by back propagation.

7. The warehouse logistics scheduling method based on deep reinforcement learning according to claim 1, characterized in that, The initial path vector is obtained by inputting the constructed two-dimensional coordinate vector into the decision model to obtain N transportation point probability vectors, and taking the transportation point with the maximum probability value in each vector as the initial transportation path. 8.The warehouse logistics scheduling method based on deep reinforcement learning of claim 1, wherein, The obtaining of the to-be-corrected transportation path vector specifically comprises: firstly calculating a convex hull vector of a two-dimensional coordinate vector, and then screening N largest elements in the heat map vector The corresponding i th transportation point and j th transportation point are merged into one transportation point set, the transportation points in the transportation point set are deleted in the initial transportation path, and the transportation point set is added to any transportation point in the convex hull vector corresponding to the position of the initial transportation path; finally, the number of obtained to-be-corrected transportation paths is the same as the number of transportation points in the convex hull vector. 9.A warehouse logistics scheduling apparatus based on deep reinforcement learning, comprising a memory and one or more processors, wherein the memory stores executable codes, and the executable codes comprise instructions for causing the one or more processors to perform the steps of, When the processor executes the executable code, the method for scheduling warehouse logistics based on deep reinforcement learning is implemented.

10. A computer-readable storage medium having stored thereon a program, characterized in that, When the processor executes the program, the method for scheduling warehouse logistics based on deep reinforcement learning is implemented.

Images