Method for designing distribution of picking shelves based on reinforcement learning simulated annealing

Abstract

The application discloses a picking shelf distribution design method based on reinforcement learning simulated annealing, relates to the technical field of intelligent algorithms, analyzes massive order data through an FP-Growth algorithm, and mines the correlation strength between materials. Subsequently, a greedy algorithm combining SKU heat and a gravity model is proposed to quickly generate a high-quality initial layout solution, laying a good foundation for subsequent optimization. Finally, aiming at the complex combinatorial optimization problem of layout optimization, a simulated annealing framework is introduced, and reinforcement learning technology is innovatively integrated to construct an adaptive large neighborhood search strategy, enabling the algorithm to intelligently select the most effective neighborhood disturbance operator according to the optimization process, thereby efficiently finding a globally better SKU layout solution and minimizing the total picking cost. The algorithm has high execution efficiency and can be applied to the restocking strategy module of warehouse management systems, warehouse execution systems and other logistics management systems.

Description

Technical Field

[0001] This invention relates to the field of intelligent algorithm technology, and in particular to a method for designing picking shelf distribution based on reinforcement learning simulated annealing. Background Technology

[0002] Picking is a core component of logistics warehousing, especially e-commerce warehousing, and its efficiency profoundly impacts the operational efficiency and cost of the entire logistics warehousing system. Currently, most mainstream picking methods involve retrieving goods from picking racks. Whether manual picking, pick-to-light electronic tag picking, or "goods-to-person" picking based on automated guided vehicles (AGVs) is used, the layout strategy of various stock keeping units (SKUs) within the racks is a key factor affecting picking efficiency. Because different rack slots differ in ergonomics and accessibility, their picking times and operational difficulties vary. Therefore, different material layout schemes will significantly affect the picking path length or equivalent picking time for a single order. Considering that a single rack typically handles hundreds or even thousands of order picking tasks per hour, refined optimization of rack SKU layout has significant engineering and economic value for improving overall logistics warehousing picking efficiency.

[0003] The shelf SKU layout optimization problem is essentially a combinatorial optimization problem. Its goal is to find the optimal layout scheme from all possible "SKU-slot" mapping combinations that satisfies a certain objective function, such as total picking distance. Combinatorial optimization problems are concise in description but difficult to solve, often facing the challenge of "combinatorial explosion." In the field of warehousing and logistics, a classic related problem is the picker path problem, which involves planning the shortest path for pickers to access all picking locations in a warehouse. SPRP can be considered a variant of the Traveling Salesman Problem. For NP-hard problems like this, traditional research methods include greedy algorithms based on bipartite networks, branch and bound algorithms, and variable neighborhood search algorithms. These methods have the advantage of fast running speed, but the solutions obtained usually still have room for optimization. Metaheuristic algorithms can usually search globally to find approximate solutions to the optimal solution. For example, genetic algorithms, ant colony algorithms, whale algorithms, and vassal algorithms have been used to solve SPRP or similar problems. However, existing solutions suffer from high expected total order picking costs and insufficient solution efficiency. Therefore, a picking rack distribution design method based on reinforcement learning simulated annealing is proposed. Summary of the Invention

[0004] The purpose of this invention is to provide a picking shelf distribution design method based on reinforcement learning simulated annealing to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a picking shelf distribution design method based on reinforcement learning simulated annealing, comprising the following steps;

[0006] S1. Import and initialize sales orders, analyze order list data, and generate initial layout;

[0007] S2. Extract SKU interrelation information from order history, use FP-Growth algorithm to mine frequent itemsets, and generate a symmetric association strength matrix to characterize the co-occurrence probability between SKUs. The specific process is as follows: S21. Data preprocessing: By preprocessing the input sales order data, the popularity H(si) of each SKU is calculated.

[0008] S22. Constructing an FP-Tree: Modeling association rules. Each OK order contains a set of SKUs. We construct a frequent pattern tree using the FP-Growth algorithm.

[0009] S23. Generate the association matrix: By mining frequent itemsets, calculate the association strength between any two SKUsi and sj.

[0010]

[0011] Where σ(∙) represents the frequency of occurrence of the corresponding SKU combination in the order set, This represents the total number of orders.

[0012] S3. Based on the popularity of SKUs and the correlation strength matrix, the initial layout of each slot on the shelf is generated using a gravity model and a greedy strategy. The SKU with the highest popularity is assigned to the storage location closest to the shelf origin. Then, the sum of the correlation strengths of the remaining SKUs and all SKUs in the assigned area is calculated iteratively as the comprehensive gravity. Each time, the SKU with the largest comprehensive gravity is selected and assigned to the current empty storage location closest to the origin.

[0013] In the calculation of the location closest to the shelf origin, the shelf origin coordinates are set as C(xo,yo)=(Column / 2,Row / 2), where Row and Column are the number of shelf layers and columns, respectively; the picking efficiency dloc of each location loc is determined by the weighted Manhattan distance d_loc=D(loc,origin) from that point to the location origin, and all locations are sorted in ascending order according to d_loc, where the weighted Manhattan distance is defined as follows:

[0014]

[0015] Where x is the horizontal column direction coordinate of the shelf and y is the vertical shelf layer direction coordinate. Since the y-axis direction usually has a greater impact on picking efficiency than the x-axis direction, let α=0.3 to increase the weight of the layer direction. When the number of SKUs to be assigned is less than the number of shelf slots, after the initial assignment of all SKUs is completed, all SKUs are added back to the set to be assigned, and the remaining empty slots are filled according to the principle of maximizing the overall attraction.

[0016] S4. Starting from the initial layout, perform global search using heuristic optimization based on simulated annealing, and integrate Q-Learning reinforcement learning algorithm during simulated annealing. The current temperature and improvement rate of simulated annealing constitute the state space. The layout is iteratively optimized by adaptively selecting neighborhood operators, and the final layout is output.

[0017] Preferably, the objective function of simulated annealing is:

[0018]

[0019] in, K represents the picking cost for an order that is OK, where K is the total number of orders. It consists of three parts: starting cost, intermediate path cost, and ending cost, and is expressed as follows;

[0020]

[0021]

[0022]

[0023]

[0024] in, The coordinates of the shelf origin are: This represents the coordinates of the storage location of the r-th visited SKU in the order "ok", where R is the total number of SKUs that need to be visited in this order; the basic neighborhood action adopts a 2-opt random exchange strategy, and the acceptance criterion adopts the Metropolis criterion: when When to accept the new layout, Time with probability Accept the new layout; adopt a geometric cooling function. ,in This is the cooling coefficient.

[0025] Preferably, the improvement rate is: recording the frequency at which the objective function is improved in the most recent N iterations to determine whether the search has stalled; the state space consists of two dimensions: temperature range and improvement rate.

[0026] Preferably, the neighborhood operators in step S4 include: a cluster reconstruction operator that uses the association strength matrix to compactly adjust a subset of strongly associated SKUs, a hotspot clustering operator that moves hotspot SKUs to high-priority storage locations, and a random perturbation operator that includes random swapping or random insertion.

[0027] Preferably, the reward function used in the Q-Learning reinforcement learning algorithm is:

[0028]

[0029] in, The picking loss for the current iteration step. Let be the new picking loss after performing neighborhood actions, be the amplification factor, λ be the additional incentive for finding the global optimum, and ξ be the penalty term to suppress ineffective searches. This is an indicator function that takes the value 1 when the global optimum is found and 0 otherwise. This is an indicator function that takes 1 if there is no improvement and 0 otherwise.

[0030] Preferably, in the Q-Learning reinforcement learning algorithm, the expected reward of taking action a in state s is evaluated by maintaining a Q-value table Q(s,a), and the Q-value is updated according to the following formula:

[0031] Q(s,a) ← Q(s,a)+η[R+γmaxQ(s',a')-Q(s,a)]

[0032] Where n is the learning rate, y is the discount factor, s" is the new state entered after executing action a, and a' is the optional action in the new state; the neighborhood operator is selected based on the Q value according to the e-greedy policy.

[0033] S5. Generate the final optimized layout result.

[0034] The technical effects and advantages of this invention are as follows:

[0035] This reinforcement learning-based simulated annealing picking shelf distribution design method analyzes massive order data using the FP-Growth algorithm to uncover the correlation strength between materials. Subsequently, a greedy algorithm combining SKU heat and gravity models is proposed to quickly generate high-quality initial layout solutions, laying a solid foundation for subsequent optimization. Finally, for the complex combinatorial optimization problem of layout optimization, a simulated annealing framework is introduced, and reinforcement learning technology is innovatively integrated to construct an adaptive large neighborhood search strategy. This allows the algorithm to intelligently select the most effective neighborhood perturbation operator based on the optimization process, thereby efficiently finding a globally better SKU layout scheme and minimizing the total picking cost. This algorithm has high execution efficiency and can be applied to the putaway and replenishment strategy modules of logistics management systems such as warehouse management systems and warehouse execution systems. Attached Figure Description

[0036] Figure 1 This is a flowchart of the overall method of the present invention;

[0037] Figure 2 Here is a flowchart of the greedy strategy allocation algorithm of this invention;

[0038] Figure 3 This is a flowchart of the simulated annealing algorithm based on reinforcement learning in this invention;

[0039] Figure 4 This is an example of a SKU for this invention.

[0040] Figure 5 This is an example of an order illustration for the present invention;

[0041] Figure 6 This is a schematic diagram of the shelf location representation in this invention;

[0042] Figure 7 This is a schematic diagram comparing the total final picking costs of various methods in this invention;

[0043] Figure 8 This is a schematic diagram comparing the running time of various methods of the present invention;

[0044] Figure 9 This is a schematic diagram comparing the weights of enhanced simulated annealing and iterative optimization using a genetic algorithm, as presented in this invention.

[0045] Figure 10 This is a schematic diagram comparing the iterative optimization weights of enhanced simulated annealing and ordinary simulated annealing algorithms according to the present invention;

[0046] Figure 11 This is a comparison diagram of shelf SKU layouts generated by various algorithms of this invention. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] Example 1: This embodiment of the invention provides, as follows Figures 1 to 3 The picking rack distribution design method based on reinforcement learning simulated annealing is shown. In this method, a picking rack with N preset slots, M materials to be assigned, and K picking orders or order predictions are first preset. The goal of the problem is to assign a corresponding material to each slot so that the total time used to pick all orders is minimized.

[0049] enter:

[0050] The set of all materials is S = {s1, s2, ..., sm};

[0051] The set of coordinates of all slots on the shelf is L={l1,l2,...,ln}, where ln=(xn,yn), representing the column number and row number of the shelf corresponding to the slot, respectively;

[0052] Select the origin coordinates C(xo, yo);

[0053] The picking order set O={o1,o2,...,ok}, where k is the order quantity, and each order o is composed of a material set o={son1,son2,...,sont}, where sont represents the picking material s corresponding to the order line t of the picking order. In cases where real picking orders cannot be obtained, historical orders with periodicity can be used, or the results of order prediction by machine learning algorithms can be used instead. Specific algorithms can include convolutional neural networks, LSTM, XGBoost, etc.

[19]

[0054] Mapping function: f:L→S, which means assigning the corresponding material to each slot;

[0055] Optimize the objective function:

[0056] Find the mapping function f that minimizes the total picking cost for all orders, i.e.:

[0057]

[0058] Estimated total picking cost for all orders:

[0059] Here, Ek is the picking cost of order ok under mapping f. Its calculation is based on the path formed by the slots containing all SKUs required for picking in that order. The picking quantity of the same SKU does not affect the number of picking operations; that is, multiple picking quantities of the same SKU in the same order line are counted as only one picking operation.

[0060] We employ a simplified path cost model, defined as the path cost of visiting each storage location sequentially from the origin in ascending order of distance from the origin for each SKU, and then returning to the origin. Ek specifically consists of three parts: the starting point weight Eks, the intermediate weights Eke, and the ending point weight Ekt.

[0061]

[0062] in, , is the picking distance of the order from the shelf origin to the first accessed slot.

[0063] , which is the total picking distance from the first picking slot to the last picking slot in this order.

[0064] , is the picking distance for this order from the last picking slot back to the original point on the shelf.

[0065] in, The weighted Manhattan distance from storage location i to storage location j is defined as:

[0066]

[0067] Where x is the horizontal coordinate of the shelf column and y is the vertical coordinate of the shelf layer. Since the y-axis direction usually has a greater impact on picking efficiency than the x-axis direction, we choose α=0.3 to increase the weight of the layer direction.

[0068] After inputting data, this method goes through two key optimization steps: initial layout generation based on a greedy algorithm and global optimization based on simulated annealing using reinforcement learning. The final optimal SKU layout result is obtained, achieving global optimization. The specific process of the method is as follows:

[0069] S1. By analyzing the order list data, an initial layout is obtained, which allows subsequent optimization to be based on a better initial solution, thereby accelerating the convergence speed of the subsequent optimization algorithm and reducing the probability of getting trapped in a local minimum interval.

[0070] S2. Extract SKU interrelation information from order history and place SKUs with high correlation in close proximity to improve picking efficiency. We choose to use the FP-Growth algorithm to mine frequent itemsets and obtain the SKU picking association matrix. The specific operation process is as follows:

[0071] S21. Data Preprocessing: By preprocessing the input sales order data, the popularity H(si) of each SKU is calculated, which is the total number of picking times for each SKU in all orders. Since each order line corresponds to only one SKU in the picking operation, even if the line contains multiple items, the actual picking operation only needs to be completed once. Therefore, the total number of order lines is used as the total number of picking times.

[0072] S22. Constructing an FP-Tree: Modeling association rules. Each order OK contains a set of SKUs. We construct a frequent pattern tree using the FP-Growth algorithm.

[0073] S23. Generate the association matrix: By mining frequent itemsets, calculate the association strength between any two SKUsi and sj.

[0074]

[0075] Where σ(∙) represents the frequency of occurrence of the corresponding SKU combination in the order set, This represents the total number of orders.

[0076] Finally, a symmetric association strength matrix AM×M is obtained, which represents the co-occurrence probability among SKUs. This matrix will serve as the basis for the "gravity" of the initial layout and will be used in subsequent reinforcement learning optimization steps.

[0077] To ensure the effectiveness and stability of the association rule mining results, we used a minimum support threshold λsupport = 0.01. It should be noted that, due to the symmetry of the physical distance between picking locations, this method introduces confidence as a filtering constraint to ensure the symmetry of the association matrix. The above parameters were obtained through pre-experiment optimization, ensuring coverage of frequent itemsets while avoiding noise interference.

[0078] S3. Greedy layout generation based on gravity model: Based on the popularity and relevance of SKUs, the initial layout of SKUs assigned to each slot in the shelf is quickly calculated. In order to avoid the slow convergence of the simulated annealing algorithm due to random initial layout, a greedy layout generation algorithm based on material popularity and gravity is proposed here. The specific logic is as follows.

[0079] S31. For priority sorting of storage locations, we first need to define the coordinates of the origin of the shelf: C(xo, yo). For picking shelves, since pickers usually stand at the horizontal center of the shelf and the middle shelf height is usually the most picking-friendly, we choose to set the origin as the center point of the shelf. Let Row and Column be the number of shelves and columns, respectively. Then the coordinates of the shelf origin are set as: C(xo, yo) = (Column / 2, Row / 2). For other application requirements, the origin coordinates can also be set to other positions relative to the shelf, such as the lower left corner of the shelf, the leftmost end of the middle shelf, etc.

[0080] Secondly, since the origin C(xc,yc) is located at the center of the shelf, the picking efficiency dloc of each location loc can be determined by its weighted Manhattan distance to the origin dloc=D(loc,origin), and all locations are sequentially arranged according to dloc and placed into the set {binsorted}, which will be used for the initial layout allocation.

[0081] S32, Allocation Strategy, which provides a specific allocation algorithm for the initial layout allocation. The specific process is as follows:

[0082] S321. Place all SKUs into the unassigned set {sunased}, which represents the SKUs that have not yet been assigned slots in this round of allocation. For the case of N>M, this set is not completely equivalent to the globally unassigned SKUs.

[0083] S322. Place the SU with the highest heat H(si) in the storage location with the smallest dlgc; and repeat the following steps:

[0084] For all remaining unassigned SKUs in {sunased}, calculate their combined gravitational pull G on the assigned regions:

[0085]

[0086] Calculate the sum of the association strengths between the SKU and the assigned SKUs, which is used as the combined attraction of the assigned SKU layout to the currently unassigned SKU;

[0087] Each time, select the SKU with the highest G value and place it in the current remaining storage location that is closest to the shelf origin. This location also corresponds to the closest location of the already assigned SKU. Then, remove this SKU from the set to be assigned until all shelf slots are filled.

[0088] This strategy ensures that highly relevant SKUs are clustered in physical space. When the number of SKUs is greater than the number of slots, the algorithm selects the SKUs with the highest picking efficiency. When the number of SKUs is less than the number of slots, after all SKUs are filled in a loop, all SKUs are added back to the set to be allocated. This allows the remaining empty slots to be filled again with SKUs with high picking activity, thereby improving the overall picking efficiency.

[0089] S4. Simulated Annealing Optimization Algorithm Based on Reinforcement Learning: Although the initial layout solution obtained by the greedy algorithm is of higher quality than that of the random layout, it is still far from the optimal solution. In this stage, a heuristic optimization algorithm based on simulated annealing is introduced to perform a global search to optimize the layout effect. To address the blindness of neighborhood search in the optimization process of simulated annealing algorithm, reinforcement learning based on Q-Learning algorithm is integrated to achieve adaptive operator selection, so as to further improve the optimization effect. The specific process is as follows:

[0090] S41. Basic Simulated Annealing and Objective Function

[0091] We start with the initial layout obtained by the greedy algorithm and adopt a heuristic algorithm framework based on simulated annealing.

[0092] The objective function for simulated annealing is:

[0093]

[0094] in, K represents the picking cost for an order that is OK, where K is the total number of orders. It consists of three parts: starting cost, intermediate path cost, and ending cost, and is expressed as follows;

[0095]

[0096]

[0097]

[0098]

[0099] in, The coordinates of the shelf origin are: This indicates the location coordinates of the r-th SKU accessed in the order OK, where R is the total number of SKUs that need to be accessed in this order;

[0100] The objective function for optimization is Etotal. The basic neighborhood action adopts a 2-opt random exchange strategy, which randomly selects two storage locations, Li and Lj, and exchanges the SKUs stored therein.

[0101] The acceptance criterion used is the Metropolis criterion: this criterion, by introducing a temperature control variable T, selects whether to accept the new layout generated in the iterative step based on the range of the energy difference ΔE before and after the optimization step.

[0102] like This indicates that the optimization effect is better and the new layout is accepted unconditionally.

[0103] like According to probability Accepting new layouts gives the algorithm the ability to escape local minima;

[0104] Finally, a geometric cooling function is used. The temperature is updated, and the process proceeds to the next iteration, where... The temperature decrease coefficient is used. The temperature control variable starts from Tinit and continues until the termination temperature threshold Tend is reached, or the number of iterations reaches the threshold θ, at which point the optimization stops and the corresponding global optimal layout is output.

[0105] S42, Reinforcement Learning Adaptive Large Neighborhood Search

[0106] Although simulated annealing possesses global search capabilities, its search efficiency is highly dependent on the selection of neighborhood operators. Using a 2-opt random swapping strategy to select neighborhood operators often leads to a large number of ineffective iterations in the later stages of the search. To overcome this limitation, a reinforcement learning mechanism is introduced to construct an adaptive large neighborhood search algorithm. Through the learning ability of the agent, the weights of various operators are dynamically adjusted according to different stages of the search process, thereby achieving a dual improvement in search efficiency and solution quality. The specific process is as follows;

[0107] S421, Reinforcement Learning Model Construction

[0108] Reinforcement learning models the selection process of neighborhood operators as a discrete-time Markov decision process, the core elements of which are defined as follows;

[0109] State space;

[0110] In order for the agent to perceive the current search environment, the state space consists of two dimensions:

[0111] Temperature range: The current simulated annealing temperature T is logarithmically discretized to reflect the stage of the search (high temperature exploration period or low temperature convergence period).

[0112] Improvement rate: Record the frequency with which the objective function is improved (accepting a better solution or a worse solution according to probability) in the most recent N iterations, in order to determine whether the search has stalled.

[0113] Action space:

[0114] Design a set of heterogeneous neighborhood perturbation operators as optional actions for the agent:

[0115] Cluster Reconstruction Operator: Using the association strength matrix A generated by FP-Growth, a strongly associated subset of SKUs is identified, and the members of this cluster are adjusted to more compact adjacent areas in the layout.

[0116] Hotspot clustering operator: For frequently accessed hot SKUs, attempt to move them towards the origin or a higher priority storage location;

[0117] Random perturbation operators: include basic perturbations such as random swapping and random insertion, maintaining the randomness of the exploration;

[0118] S422, Reward Function

[0119] To address the issue of sparse feedback in combinatorial optimization of reinforcement learning, this paper further designs a continuous reward function based on an improved ratio:

[0120] The reward function used in the Q-Learning reinforcement learning algorithm is:

[0121]

[0122] in, The picking loss for the current iteration step. Let be the new picking loss after performing neighborhood actions, be the amplification factor, λ be the additional incentive for finding the global optimum, and ξ be the penalty term to suppress ineffective searches. This is an indicator function that takes the value 1 when the global optimum is found and 0 otherwise. This is an indicator function that takes 1 if there is no improvement and 0 otherwise. =50 is the amplification factor, λ=5 is the additional incentive to find the global optimum, and ξ=0.05 is a slight penalty term to suppress invalid search.

[0123] S423, Adaptive learning mechanism based on Q-Learning

[0124] The Q-Learning algorithm is used to train the operator selection strategy. The agent evaluates the expected reward of taking action a in a specific state s by maintaining a Q-value table Q(s,a).

[0125] First, initialize the Q-value table and, starting from the greedy initial solution, perform the following steps during each iteration of the simulated annealing algorithm:

[0126] Sensing State: Determine state s based on current temperature and improvement rate;

[0127] Action selection: Based on the ε-greedy policy, select the neighborhood operator a according to Q(s,a);

[0128] Execution and Evaluation: Apply operator a to generate a new solution, and calculate ΔE and reward value R;

[0129] Acceptance criterion: Decide whether to accept the new solution based on the Metropolis criterion;

[0130] Update strategy: Based on the formula;

[0131]

[0132] Update the Q-value, where η is the learning rate and γ is the discount factor. Iterative optimization: Cool down and proceed to the next iteration until the termination condition is met. Through this mechanism, the agent can learn which operators are conducive to broad exploration in the high-temperature stage, and tend to choose operators that can perform local fine-tuning in the low-temperature stage, thereby significantly improving the targeting and efficiency of the search.

[0133] The FP-Growth algorithm is used to analyze massive order data to uncover the correlation strength between materials. A greedy algorithm combining SKU heat and a gravity model is then proposed to quickly generate high-quality initial layout solutions, laying a solid foundation for subsequent optimization. Finally, for the complex combinatorial optimization problem of layout optimization, a simulated annealing framework is introduced, and reinforcement learning technology is innovatively integrated to construct an adaptive large neighborhood search strategy. This allows the algorithm to intelligently select the most effective neighborhood perturbation operator based on the optimization process, thereby efficiently finding a globally better SKU layout scheme and minimizing the total picking cost. This algorithm has high execution efficiency and can be applied to the putaway and replenishment strategy modules of logistics management systems such as warehouse management systems and warehouse execution systems.

[0134] Example 2, as follows Figure 4-10 As shown, this embodiment uses both simulation data and real production data to verify the effectiveness of the method. To verify the versatility of the test method, we use simulation data to generate two types of data: small-scale and large-scale, to simulate different usage scenarios.

[0135] Small-scale simulation data:

[0136] SKU data: Includes 51 ordinary popular SKUs, namely: Category A, Category B, Category C, each with 17 SKUs, and 4 high-popularity SKUs;

[0137] Order data: Construct a set containing 3,000 historical orders, in which several types of common SKUs appear relatively evenly, while increasing the frequency of high-volume SKUs, and including several strongly correlated clusters;

[0138] Shelves: The shelving is 5 rows and 11 columns in length, with the origin set at the center of the shelving.

[0139] Large-scale simulation data:

[0140] SKU data: 200 SKUs, divided into: 20% category A, 30% category B, and 50% category C;

[0141] Order data: Construct a collection containing 5000 historical orders;

[0142] Shelves: The shelving is 10 rows and 20 columns in length, with the origin set at the center of the shelving.

[0143] The real data we use comes from a client's actual production data, including:

[0144] SKU data: 209 SKUs;

[0145] Order data: One week's worth of data, comprising a collection of 6,743 historical orders;

[0146] Shelves: The shelving has 11 rows and 19 columns, with the origin set at the center of the shelving.

[0147] Because real order SKUs exhibit a long-tail distribution, with a very small number of SKUs being abnormally high-frequency and a large number of SKUs being abnormally low-frequency, this paper performs the following preprocessing on the original SKU and order data to ensure rigor:

[0148] Remove abnormal orders, such as orders with a single order line SKU picking quantity less than or equal to 0 or more than 100. The former is usually a return order, and the latter is usually picked using the full case picking method.

[0149] Remove SKUs and order lines that are unsuitable for manual picking, such as SKUs that are too large or too heavy;

[0150] Remove extremely low-frequency SKUs and order lines with fewer than 10 orders;

[0151] Extremely high-frequency SKUs and their corresponding order lines are removed. In real-world scenarios, these SKUs are typically managed in dedicated picking areas and are not suitable for general location optimization. This paper uses interquartile range and median anomaly detection rules to filter extremely high-frequency orders. If an SKU's popularity simultaneously meets the following conditions: H(si) > Q3 + 2.5 × IQR and H(si) > 10 × median(H), it is identified as an extremely high-frequency abnormal SKU and removed.

[0152] We compared the following methods:

[0153] Random: Uses random SKU layout, that is, uses the standard random shuffling method for SKU layout;

[0154] Greedy: Only uses the greedy layout algorithm based on the gravity model in the method, without any subsequent optimization steps;

[0155] GA: Uses a greedy algorithm to obtain the initial layout and a genetic algorithm for subsequent optimization;

[0156] Rand-SA: Uses random SKUs to obtain the initial layout and uses traditional simulated annealing for subsequent optimization;

[0157] SA: Use a greedy algorithm to obtain the initial layout, and use traditional simulated annealing for subsequent optimization;

[0158] RL-SA: The algorithm presented in this paper uses a greedy algorithm to obtain the initial layout and a reinforcement learning-based simulated annealing algorithm for subsequent optimization.

[0159] To ensure the fairness of the experimental comparison, the parameters of each comparison algorithm are uniformly set as follows:

[0160] GA: Population size Ps=100, crossover probability Pc=0.8, mutation probability Pm=0.1, elite retention probability Pe=0.05, maximum number of iterations Gmax=1000. Experiments show that since the genetic algorithm is slow to compute and the results have already converged by the time iterates to 1000 steps, further increasing Gmax has no significant benefit to optimization.

[0161] Rand-SA and SA: Initial temperature Tinit=5000, cooling coefficient β=0.98, termination temperature Tend=0.001, iteration threshold θ=4000;

[0162] RL-SA: learning rate η=0.2, discount factor γ=0.8.

[0163] All parameters were initialized with reference to relevant literature [7]

[12] and optimized through preliminary experiments.

[0164] Appendix Figure 4 The example shown is a list of SKU data in the data. Rows 1 to 3 represent SKUs of three common popularity types: A, B, and C, respectively, while row 4 represents high popularity SKUs.

[0165] Appendix Figure 5 This is an example of an order data list in the data. Each order has a maximum of 5 order lines, and each order line corresponds to a picking task for one SKU. Popular SKUs appear more frequently.

[0166] Appendix Figure 6 This is an example of a list of shelf location data in the data. Each location has x and y coordinates, which represent the position number of the location in the column and row, respectively.

[0167] Appendix Figure 7 This table shows a comparison of the total picking cost E-values ​​obtained after optimizing the picking process using various methods.

[0168] Data analysis shows that random placement performs poorly, while the greedy algorithm improves placement performance to some extent. Compared to random placement, the picking cost is reduced, validating the effectiveness of initialization using heat and association information. Simulated annealing further improves performance compared to the greedy algorithm. The proposed RL-SA method achieves the best results, with a further reduction in picking cost compared to the basic SA. This demonstrates that diverse neighborhood search strategies guided by reinforcement learning can more effectively escape local optima and find higher-quality global placement solutions.

[0169] In comparison, the optimization effect of genetic algorithms on this problem is not ideal, with its performance only slightly better than that of greedy algorithms. This is because the crossover operator of genetic algorithms, while recombining superior parental fragments, is prone to disrupting the already formed "highly correlated SKU clusters." This "genetic drift" makes the GA algorithm inefficient in the local search phase, making it difficult to perform fine-tuning like SA.

[0170] Appendix Figure 8 The table shows a comparison of the execution time for different algorithms. As can be seen, the randomized and greedy algorithms are direct solutions and are very fast. Compared to the genetic algorithm, the simulated annealing algorithm has a significant speed advantage. Furthermore, although reinforcement learning was introduced, it did not result in a significant loss in speed optimization.

[0171] Appendix Figure 9 The graph shows the optimization process of the simulated annealing algorithm based on data 3 using genetic algorithm and reinforcement learning, and the selection cost reduction curve (the results were aligned according to Gmax and θ because the total number of iterations of the two algorithms are different). As can be seen from the graph, compared with the genetic algorithm, the simulated annealing algorithm can avoid getting trapped in the local optimum too early.

[0172] Appendix Figure 10 The graph shows the cost reduction curve of the simulated annealing algorithm optimization process based on whether reinforcement learning is used in Data 3. The analysis shows that by using reinforcement learning, it is possible to escape local minima and obtain better optimization results. Furthermore, when reaching the same optimization level, the number of iterations required for convergence is less than that of the traditional simulated annealing algorithm, showing higher search efficiency.

[0173] Appendix Figure 11The display shows a comparison of shelf SKU layouts generated using various methods. Each square corresponds to a shelf location, and the darker the color, the more popular the SKU is assigned to that location. As can be seen from the figure, the SKU layout based on the reinforcement learning simulated annealing algorithm causes the more popular SKUs to concentrate in the center of the shelf, which is very helpful in reducing picking time.

[0174] In summary, preliminary experiments show that the algorithm presented in this paper has the advantages of better optimization effect and faster convergence speed compared with similar optimization algorithms. Furthermore, due to its high execution speed, the algorithm is very suitable for use in real production environments.

[0175] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A picking shelf distribution design method based on reinforcement learning simulated annealing, characterized in that, Includes the following steps: S1. Import and initialize sales orders, analyze order list data, and generate initial layout; S2. Extract SKU interrelation information from order history, use FP-Growth algorithm to mine frequent itemsets, and generate a symmetric association strength matrix to characterize the co-occurrence probability between SKUs; S3. Based on the popularity of SKUs and the correlation strength matrix, the initial layout of each slot on the shelf is generated using a gravity model and a greedy strategy. The SKU with the highest popularity is assigned to the storage location closest to the shelf origin. Then, the sum of the correlation strengths of the remaining SKUs and all SKUs in the assigned area is calculated iteratively as the comprehensive gravity. Each time, the SKU with the largest comprehensive gravity is selected and assigned to the current empty storage location closest to the origin. S4. Starting from the initial layout, perform global search using heuristic optimization based on simulated annealing, and integrate Q-Learning reinforcement learning algorithm during simulated annealing. The current temperature and improvement rate of simulated annealing constitute the state space. The layout is iteratively optimized by adaptively selecting neighborhood operators, and the final layout is output. S5. Generate the final optimized layout result.

2. The picking shelf distribution design method based on reinforcement learning simulated annealing according to claim 1, characterized in that, The specific operation process of step S2 is as follows; S21. Data preprocessing: By preprocessing the input sales order data, the popularity H(si) of each SKU is calculated separately. S22. Constructing an FP-Tree: Modeling association rules. Each OK order contains a set of SKUs. We construct a frequent pattern tree using the FP-Growth algorithm. S23. Generate the association matrix: By mining frequent itemsets, calculate the association strength between any two SKUsi and sj. Where σ(∙) represents the frequency of occurrence of the corresponding SKU combination in the order set, This represents the total number of orders.

3. The picking shelf distribution design method based on reinforcement learning simulated annealing according to claim 2, characterized in that, In step S3, for the calculation of the storage location closest to the shelf origin, the coordinates of the shelf origin are set as C(xo,yo)=(Column / 2,Row / 2), where Row and Column are the number of layers and columns of the shelf, respectively; the picking efficiency dloc of each storage location loc is determined by the weighted Manhattan distance d_loc=D(loc,origin) from that point to the storage location origin, and all storage locations are sorted in ascending order according to d_loc.

4. The picking shelf distribution design method based on reinforcement learning simulated annealing according to claim 3, characterized in that, The weighted Manhattan distance is defined as follows: Where x is the horizontal coordinate of the shelf column and y is the vertical coordinate of the shelf layer. Since the y-axis direction usually has a greater impact on picking efficiency than the x-axis direction, we set α=0.3 to increase the weight of the layer direction.

5. The picking shelf distribution design method based on reinforcement learning simulated annealing according to claim 4, characterized in that, In step S3, when the number of SKUs to be allocated is less than the number of shelf slots, after the initial allocation of all SKUs is completed, all SKUs are added back to the set to be allocated, and the remaining empty slots are filled according to the principle of maximizing overall attraction.

6. The picking shelf distribution design method based on reinforcement learning simulated annealing according to claim 1, characterized in that, In step S4, the objective function of simulated annealing is: in, K represents the picking cost for an order that is OK, where K is the total number of orders. It consists of three parts: starting cost, intermediate path cost, and ending cost, and is expressed as follows; in, The coordinates of the shelf origin are: This represents the coordinates of the storage location of the r-th visited SKU in the order "ok", where R is the total number of SKUs that need to be visited in this order; the basic neighborhood action adopts a 2-opt random exchange strategy, and the acceptance criterion adopts the Metropolis criterion: when When to accept the new layout, Time with probability Accept the new layout; adopt a geometric cooling function. ,in This is the cooling coefficient.

7. The picking shelf distribution design method based on reinforcement learning simulated annealing according to claim 1, characterized in that, The improvement rate in step S4 is: recording the frequency at which the objective function is improved in the most recent N iterations to determine whether the search has stalled; the state space consists of two dimensions: temperature range and improvement rate.

8. The picking shelf distribution design method based on reinforcement learning simulated annealing according to claim 1, characterized in that, The neighborhood operators in step S4 include: a cluster reconstruction operator that uses the correlation strength matrix to compactly adjust a subset of strongly correlated SKUs; a hotspot clustering operator that moves hotspot SKUs to high-priority storage locations; and a random perturbation operator that includes random swapping or random insertion.

9. The picking shelf distribution design method based on reinforcement learning simulated annealing according to claim 6, characterized in that, The reward function used in the Q-Learning reinforcement learning algorithm is: in, The picking loss for the current iteration step. Let be the new picking loss after performing neighborhood actions, be the amplification factor, λ be the additional incentive for finding the global optimum, and ξ be the penalty term to suppress ineffective searches. This is an indicator function that takes the value 1 when the global optimum is found and 0 otherwise. This is an indicator function that takes 1 if there is no improvement and 0 otherwise.

10. The picking shelf distribution design method based on reinforcement learning simulated annealing according to claim 6, characterized in that, In the Q-Learning reinforcement learning algorithm, the expected reward of taking action a in state s is evaluated by maintaining a Q-value table Q(s,a), and the Q-value is updated according to the following formula: Q(s,a) ← Q(s,a)+η[R+γmaxQ(s',a')-Q(s,a)] Where n is the learning rate, y is the discount factor, s" is the new state entered after executing action a, and a' is the optional action in the new state; the neighborhood operator is selected based on the Q value according to the e-greedy policy.

Images