Simulation optimization method, device and medium for in-warehouse sorting operation system

Abstract

The application discloses a warehouse sorting operation system simulation optimization method, equipment and medium, and steps include: step S01. Constructing a simulation model of the warehouse sorting operation system; step S02. Based on the double-target network deep reinforcement learning model, the sorting task simulation training is carried out in the constructed simulation model, and the state parameters of each sorting station in the simulation training process are obtained; step S03. According to the state parameters of each sorting station obtained in step S02, the required optimization node in the production line structure of the simulation model is identified, and the shunt path is constructed according to the identified required optimization node to obtain the optimization scheme of the production line structure; step S04. According to the obtained optimization scheme, the topology structure of the simulation model is optimized and reconstructed, and the optimized simulation model is generated. The application can systematically improve the sorting performance, balance the sorting accuracy and efficiency, and ensure the continuous and stable optimization effect.

Description

Technical Field

[0001] This invention relates to the field of automated control technology for warehouse sorting, and in particular to a simulation optimization method, equipment and medium for warehouse sorting operation system. Background Technology

[0002] With the rapid development of e-commerce and the personalization and diversification of consumer demands, the logistics and warehousing industry faces the challenge of insufficient sorting efficiency for multi-category, small-batch orders. As a core link in logistics and warehousing, in-warehouse sorting operations directly impact the operational efficiency of the entire supply chain. Traditional sorting methods rely primarily on manual experience and fixed sorting processes, lacking scientific optimization methods. This makes it difficult to cope with dynamic and complex order environments, easily leading to problems such as low sorting efficiency, redundant paths, order loops and delays, and low sorting accuracy.

[0003] To achieve intelligent optimization of sorting systems, existing technologies typically utilize simulation modeling software (such as PlantSimulation) for static analysis and optimization of the sorting process, or combine it with heuristic algorithms (such as genetic algorithms) for scheduling. However, these methods are mostly static or offline optimizations, making it difficult to cope with dynamic changes and randomness in the sorting process. Although reinforcement learning has been introduced into the path planning problem, traditional Q-learning or basic deep Q-network (DQN) algorithms suffer from problems such as Q-value overestimation, low exploration efficiency, and slow convergence speed, resulting in insufficient policy robustness and generalization ability in complex warehouse sorting environments. Furthermore, existing optimization schemes often focus on a single level of algorithm or production line structure, failing to achieve synergistic optimization combining the algorithm with structural optimization of the simulation environment, thus making it difficult to fundamentally and systematically improve sorting efficiency.

[0004] Specifically, existing technologies for optimizing in-warehouse sorting have the following problems:

[0005] (1) Insufficient dynamic decision-making ability: Traditional static scheduling models cannot respond to changes in orders and equipment status in real time, resulting in suboptimal path planning and easy to cause loop delays and path redundancy.

[0006] (2) Limitations of algorithm performance: Basic reinforcement learning algorithms are prone to problems such as overestimation of Q value, unstable training, and slow convergence in sorting scenarios, which affect the quality of the strategy.

[0007] (3) The contradiction between sorting accuracy and efficiency: the front sorting station is overloaded, the identification error rate is high, and the error will have a cumulative effect, affecting the subsequent stations. Meanwhile, the sorting path of the back stations is too long, affecting the overall efficiency.

[0008] (4) Single optimization dimension: Existing technical solutions are mostly limited to algorithm improvement or process fine-tuning, lacking a collaborative optimization method that can fundamentally eliminate bottlenecks by structurally reconstructing from the system topology level. Summary of the Invention

[0009] The technical problem to be solved by this invention is: in view of the above-mentioned problems existing in the prior art, this invention provides a simulation optimization method, equipment and medium for in-warehouse sorting operation system that is simple to implement, has high sorting efficiency and accuracy and continuous and stable optimization effect, and can systematically improve sorting performance and balance sorting accuracy and efficiency.

[0010] To solve the above-mentioned technical problems, the technical solution proposed by this invention is as follows:

[0011] A simulation optimization method for an in-warehouse sorting operation system includes the following steps:

[0012] Step S01. Construct a simulation model of the warehouse sorting operation system. The model includes a conveyor network consisting of multiple conveyor belts, multiple sorting stations, and an order processing module for simulating the sorting order processing flow.

[0013] Step S02. In the constructed simulation model, a deep reinforcement learning model based on a dual-objective network is used to perform sorting task simulation training, and the state parameters of each sorting station are obtained during the simulation training process. The dual-objective network includes a local network and a target network. DQN approximates the Q-value function through a neural network. The local network is used to select the current optimal action, and the target network is used to calculate the target Q-value and periodically synchronize parameters from the local network.

[0014] Step S03. Based on the status parameters of each sorting station obtained in step S02, identify the nodes that need to be optimized in the production line structure of the simulation model, and construct a diversion path based on the identified nodes to obtain the optimization scheme of the production line structure.

[0015] Step S04. Optimize and reconstruct the topology of the simulation model according to the optimization scheme obtained in step S03 to generate the optimized simulation model.

[0016] Further, in step S01, the conveyor network is a dual-path loop network, the sorting station includes a packing station, a labeling station, and multiple loading stations. The packing station is used to trigger the packing action, the labeling station is used to trigger the labeling action, and the loading station is the destination of the order that arrives at and completes the sorting through the conveyor network. The order processing module is configured to simulate the entire process of an order from receiving, packing, labeling, sorting to loading, including: simulating sorting operations in the warehouse through simulation modeling; obtaining order information through the order receiving terminal and taking out the corresponding product from the warehouse and sending it onto the conveyor belt; after arriving at the packing station, packing the product into boxes; after packing, proceeding to the labeling station to affix the express delivery label containing the order delivery information to the surface of the package; then entering the loop conveyor belt; performing express sorting before arriving at the loading station; and finally arriving at the loading station after sorting.

[0017] Further, in step S02, the target network is calculated according to the following formula. Q value:

[0018]

[0019] in, Indicates the target network. Indicates the local network. Indicates the discount factor. Indicates an immediate reward. Indicates local network parameters, Indicates the target network parameters. and These represent the next state and the action to be updated, respectively.

[0020] During experience replay, the deep reinforcement learning model employs a priority sampling strategy based on temporal difference error for sampling. The expression for calculating the sampling probability of a sample is as follows:

[0021]

[0022]

[0023] in, For the first i TD error for each sample To control the degree of sampling bias, This is a preset constant;

[0024] Based on sampling probability Adjust the importance sampling weights for the samples:

[0025]

[0026] in, For the first iThe importance sampling weights for each sample, where β controls the strength of the bias correction. This represents the total number of samples.

[0027] Furthermore, in step S02, the deep reinforcement learning model employs an adaptive learning rate mechanism based on training error to adjust the learning rate:

[0028]

[0029] in, The initial learning rate, The current step number at time t. To decay step size, The attenuation coefficient is... Let t be the learning rate at time t.

[0030] Furthermore, step S02 also includes setting environmental constraints, which include any one or more of the following: randomly generated order type during initialization, movement time between adjacent location nodes, each workstation can only operate on one order at a time, the number of orders operated on by each sorting workstation at a time, the maximum number of location nodes traversed in a single sorting path, and order operation sequence compliance rules. The order operation sequence compliance rules stipulate that the workstation set follows a sequence from the packaging workstation. Conveyor belt To the labeling station The order of operations is as follows; it also includes defining the model's state space as including location codes, target workstation codes, and order type codes. The location codes are used to represent the information of the current location node, which includes the order receiving system, conveyor network, packaging workstation, labeling workstation, and loading workstation. The target workstation codes are used to represent the information of the target location node for sorting. The action space is defined as including actions of moving along the conveyor path and actions of stopping at the workstation and performing sorting operations. The sorting operations include packaging, labeling, and sorting. The reward function of the model is defined with the objectives of maximizing sorting accuracy, optimizing efficiency, and guiding key operations. The key operation guidance is to guide and reward key operations by giving positive incentives to specified key operations.

[0031] Furthermore, the calculation expression for the reward function is as follows:

[0032]

[0033]

[0034]

[0035]

[0036] in, The displacement penalty term corresponds to the efficiency objective. As a key operation guide item, To terminate the reward item in accordance with the corresponding accuracy target, For indicator functions, Indicates the order type. g Indicates the target workstation. To preset the displacement penalty amount, if the order type With the target workstation g The matching indicates that the sorting was correct, that is... ,but To provide rewards, if there is no match, i.e. ,but To be penalized, T represents the last displacement of the order along the sorting path. This represents the maximum number of location nodes traversed in a single sorting path. This is the displacement penalty coefficient. This is the packaging workstation. To attach a single workstation, if the agent's current state is... At the packing station Or label a single workstation This indicates that the current operation is a critical operation. In order to provide rewards, The reward amount is preset to guide key operations.

[0037] Further, step S03 includes:

[0038] Step S301. Identify key problem workstations, including redundant workstations with path redundancy and repeated access, overloaded workstations with load exceeding a specified proportion of the predicted order volume, and workstations that need optimization with a sorting accuracy rate lower than a preset threshold.

[0039] Step S302. Quantify path redundancy: Construct a quick diversion path based on the identified key problem workstations. The first end connects to the assembly workstation, the starting point of the new conveyor belt is selected from the assembly workstation with the highest order generation frequency, and the end branches in multiple directions and connects the endpoint to the loading workstation of the current path and the loading workstation of the next path.

[0040] Step S303. Path topology reconstruction: Orders concentrated in the front part of the workstations are diverted to the back part of the workstations through the addition of a new conveyor belt to realize order diversion. The load of the diverted workstations is monitored in real time. If the load rate of the target workstation still exceeds the threshold, a second diversion is triggered.

[0041] Further, in step S301, by statistically analyzing the average number of path nodes for each order type, workstations with path lengths exceeding a preset proportion of the global average are identified, and the proportion of repeatedly accessed nodes in the path is calculated to obtain redundant workstations with a risk of cyclical delays. By obtaining the order processing volume per unit time for each sorting workstation and dynamically evaluating the matching degree between the load of each sorting workstation and the predicted order volume according to the order volume prediction model, overloaded workstations with loads exceeding a preset proportion of the predicted order volume are identified.

[0042] A computer device includes a processor and a memory, the memory being used to store a computer program, and the processor being used to execute the computer program to perform the method described above.

[0043] A computer-readable storage medium storing a computer program that, when executed, implements the method described above.

[0044] Compared with the prior art, the advantages of the present invention are as follows:

[0045] 1. This invention combines deep reinforcement learning algorithms with structural optimization of the simulation environment to form a collaborative optimization framework for the warehouse sorting operation system structure with positive feedback between the algorithm and the environment. This framework can systematically improve sorting performance, increase sorting accuracy, reduce the number of nodes along the optimal sorting path, improve sorting efficiency, and reduce the number of orders that fail due to cyclical delays.

[0046] 2. This invention, by combining simulation and data analysis during the training process of deep reinforcement learning algorithm models, accurately identifies system structural bottlenecks, enabling adjustments to the production line structure at the topology level. This results in precise system structure optimization with continuous and stable effects, and can also be easily transferred to other complex logistics sorting and production scheduling scenarios. Attached Figure Description

[0047] Figure 1 This is a schematic diagram illustrating the implementation process of the simulation optimization method for the in-warehouse sorting operation system in this embodiment.

[0048] Figure 2 This is a schematic diagram of an in-warehouse sorting operation system model constructed in a specific application embodiment of the present invention.

[0049] Figure 3 This invention provides a schematic diagram illustrating the implementation process of simulating sorting order processing in a specific application embodiment.

[0050] Figure 4 This is a schematic diagram illustrating the structural principle of the deep reinforcement learning model based on a dual-target network used in this embodiment.

[0051] Figure 5 This is a schematic diagram of the priority experience playback implementation process in this embodiment.

[0052] Figure 6 This is a schematic diagram illustrating the statistical results of various order timeout failures in the Double DQN model obtained in a specific application embodiment.

[0053] Figure 7 This is a schematic diagram of the sorting accuracy statistics for target workstations 0-3 of the Double DQN model obtained in a specific application embodiment.

[0054] Figure 8 This is a schematic diagram showing the statistical results of the access frequency of nodes at each location in the Double DQN model obtained in a specific application embodiment.

[0055] Figure 9 This is a schematic diagram illustrating the principle of structural adjustment in the simulation modeling of the in-warehouse sorting operation system in a specific application embodiment.

[0056] Figure 10 This is a schematic diagram illustrating the complete implementation process of the simulation optimization of in-warehouse sorting operations in a specific application embodiment of the present invention.

[0057] Figure 11 This is a graph showing the change in the moving average reward value of the model before and after production line adjustment, obtained in a specific application embodiment of the present invention.

[0058] Figure 12 This is a schematic diagram showing the moving average of the number of nodes traversed by the model sorting path before and after production line adjustment, obtained in a specific application embodiment of the present invention.

[0059] Figure 13 This is a schematic diagram showing the statistical results of the sorting accuracy of target workstations 0-3 before and after production line adjustment in a specific application embodiment of the present invention.

[0060] Figure 14 This is a schematic diagram of the statistical results of order timeout failures of various types before and after production line adjustment in a specific application embodiment of the present invention.

[0061] Figure 15 This is a schematic diagram showing the statistical results of the access frequency of each node in the production line model before and after adjustment, obtained in a specific application embodiment of the present invention. Detailed Implementation

[0062] The present invention will be further described below with reference to the accompanying drawings and specific preferred embodiments, but this does not limit the scope of protection of the present invention.

[0063] like Figure 1 As shown, the steps of the simulation optimization method for the in-warehouse sorting operation system in this embodiment include:

[0064] Step S01. Construct a simulation model of the warehouse sorting operation system. The model includes a conveyor network consisting of multiple conveyor belts, multiple sorting stations, and an order processing module for simulating the sorting order processing flow.

[0065] In this embodiment, the transport network is a dual-path loop network. The sorting station includes a packing station, a labeling station, and multiple loading stations. The packing station is used to trigger the packing action, the labeling station is used to trigger the labeling action, and the loading station is the destination of the order that arrives and completes the sorting through the transport network.

[0066] In specific application embodiments, the digital simulation model of the in-warehouse sorting operation system built based on discrete event simulation platforms such as Plant Simulation is as follows: Figure 2 As shown in Table 1, a dual-path circulating conveyor network system consisting of 10 conveyor belts (conveyor belts 1-10) serves target workstations 0-3. Sorting workstations include packing workstations, labeling workstations, and multiple loading workstations. The packing workstation is located between conveyors 1 and 2 and is used to trigger the packing action. The labeling workstation is used to trigger the labeling action. The loading workstation is the final destination of the order, requiring the order to be reached via the conveyor network and correctly sorted. The distance between each workstation and the conveyor belt is equal, and the conveyor belt speed is fixed. Therefore, the time taken to travel from one location node to the next during order delivery is the same. The number of each workstation is shown in Table 1.

[0067] Table 1: Equipment Statistics Table

[0068]

[0069] In this embodiment, the order processing module is configured to simulate the entire process of an order from receiving, packing, labeling, sorting to loading. This includes: simulating sorting operations in the warehouse through simulation modeling; obtaining order information through the order receiving terminal and retrieving the corresponding product from the warehouse and sending it onto the conveyor belt; packing the product at the packing station; after packing, proceeding to the labeling station to affix the courier label containing the order delivery information to the surface of the package; then entering the circulating conveyor belt; sorting the courier before reaching the loading station; and finally, arriving at the loading station after sorting.

[0070] Specifically, the simulation covers the entire process of an order, from receiving, packing, labeling, sorting to loading. Figure 3 As shown, the simulation model simulates the sorting operation in the warehouse. Order information is obtained through the order receiving terminal, the corresponding product is taken out from the warehouse and sent to the conveyor belt. After reaching the sensor position (i.e., the packing station), the product is packed by the machine. After packing, it goes to the labeling station to affix the express label containing the order delivery information to the surface of the package. Then it enters the circulating conveyor belt and is sorted before reaching the loading station. After sorting, it reaches the loading station and is finally loaded into the vehicle.

[0071] Step S02. In the constructed simulation model, a deep reinforcement learning model based on a dual-objective network is used for sorting task simulation training. Performance index data for specified workstations and location nodes are obtained during the simulation training process. The dual-objective network includes a local network and a target network. DQN approximates the target network through a neural network. Q The value function is used by the local network to select the current optimal action, and the target network is used to calculate the target. Q The value is set and parameters are periodically synchronized from the local network.

[0072] In this embodiment, a deep reinforcement learning model with a dual-objective network is used for sorting task simulation training. The model architecture is as follows: Figure 4 As shown, the algorithm architecture after adding the improved strategy to the DQN (Bi-objective Network) model is based on the bi-objective network as the main framework. During the backpropagation of parameter updates, the learning rate is dynamically adjusted, and during the experience replay, priority experience sampling is performed.

[0073] Specifically, Deep Q-Networks (DQNs) approximate neural networks. Q Value function This indicates that an action is performed in state s. a The long-term cumulative reward expectation is obtained through neural network parameters. θ approximate Q Value function, i.e. The agent is guided to choose the optimal action, and its update follows the Bellman equation:

[0074] (1)

[0075] in, For instant rewards, As a discount factor, For the target network parameters, These represent the next state and action to be updated, respectively.

[0076] The update objective of DQN is to minimize the temporal difference error (TD error), that is, to minimize the error between the predicted Q-value and the target Q-value to update the network parameters. Therefore, the loss function is defined as:

[0077] (2)

[0078] in, These are the main network parameters.

[0079] This embodiment introduces a dual-target network to separate action selection and Q-value estimation, which can alleviate the bias problem in traditional DQN's Q-value estimation caused by the lag in target network updates. The local network in the dual-target network ( ) is used to select the current optimal action; the target network ( ) used to calculate the target Q The value is periodically synchronized from the local network, and the target network is calculated according to the following formula. Q value:

[0080] (3)

[0081]

[0082] in, Indicates the target network. Indicates the local network. Indicates the discount factor. Indicates an immediate reward. Indicates local network parameters, Indicates the target network parameters. and These represent the next state and action to be updated, respectively.

[0083] Furthermore, during the experience replay process, the deep reinforcement learning model employs a priority sampling strategy based on temporal difference error (TD-error). Specifically, the sampling priority of experiences is set based on the absolute value of the temporal difference error, prioritizing the replay of historical experiences that are more valuable for training, which can accelerate convergence. This embodiment uses a priority sampling strategy based on temporal difference error, which can accelerate the learning of key experiences by focusing on high-error samples.

[0084] Preferably, the expression for calculating the sampling probability of a sample is:

[0085] (4)

[0086] (5)

[0087] in, For the first i TD error for each sample To control the degree of sampling bias, A preset small constant is used to prevent zero probability. j This represents the sample number, ranging from 1 to N, where N represents the total number of samples.

[0088] Furthermore, this embodiment is based on the sampling probability. Adjust the importance sampling weights for the samples:

[0089] (6)

[0090] in, For the first i The importance sampling weights for each sample, where β controls the strength of the bias correction. This indicates the total number of samples.

[0091] This embodiment uses sampling probability. Adjusting the importance sampling weights can be done based on the sampling probability. The importance sampling weight is dynamically increased to effectively focus on high-error samples.

[0092] Furthermore, deep reinforcement learning models employ an adaptive learning rate mechanism based on training error to adjust the learning rate: (7)

[0093] in, The initial learning rate, The current step number at time t. To decay step size, The attenuation coefficient is... Let t be the learning rate at time t.

[0094] This embodiment uses the aforementioned adaptive learning rate mechanism to dynamically adjust the learning rate at different times. A high learning rate in the early stage leads to rapid convergence, while a low learning rate in the later stage fine-tunes the parameters, preventing oscillations caused by an excessively high learning rate in the later stages of training.

[0095] This embodiment, by sampling the above method and based on the improved strategy within a dual-objective network framework, can achieve efficient sampling of experience replay, dynamic optimization of parameter updates, and accurate calculation of the target Q value, accelerating the learning of key experiences. By correcting deviations through priority sampling based on TD error, it can also maintain the stability of learning and improve the utilization rate of high-value experiences, thereby significantly improving the training efficiency and stability of sorting tasks within the warehouse. At the same time, it achieves optimization of indicators such as increasing the moving average reward value during production line adjustments and reducing sorting path nodes.

[0096] Specifically, such as Figure 5 As shown, the complete process for training a deep reinforcement learning model is as follows:

[0097] (1) Initialization: Set dual network parameters , Experience pool and initial learning rate η0.

[0098] (2) Simulation training:

[0099] state Next, local network selection action Observation after execution , The TD error is calculated according to equation (5), i.e. Storage experience ( Go to the experience pool.

[0100] (3) Parameter update:

[0101] Based on sampling probability Sampling high error experience, calculate target Q value And according to the importance weight w i Update local network: ← ,in Regular synchronization → .

[0102] (4) Synergistic effect of dynamic learning rate and priority sampling: a high learning rate is used in the initial stage. Quickly capture high-error samples, by High-probability sampling is used to accelerate convergence; a low learning rate combined with importance weights w is adopted in the later stages. i To finely adjust parameters and avoid oscillations, through TD error... It can both drive sampling priority and dynamically adjust the learning rate, forming a closed-loop optimization of "error-sampling-learning rate".

[0103] Furthermore, it also includes setting environmental constraints, such as randomly generating order types during initialization, movement time between adjacent location nodes, each workstation can only operate on one order at a time (workstation exclusivity), the number of orders operated on by each sorting workstation at one time, the maximum number of location nodes traversed in a single sorting path (path length limit), and order operation sequence compliance rules. Movement time constraints are used in state transitions and reward calculations to provide a time base; path length limit constraints are used in state definition and experience replay to limit the length of action sequences; workstation exclusivity constraints are used in state definition and experience replay, implicitly in state coding to filter illegal experiences; operation sequence compliance rules are used in state transitions, action selection, and parameter updates to ensure the legality of workstation order and dynamically filter illegal actions by adjusting the sampling priority and weight of compliant samples, as shown in Table 2.

[0104] Table 2: Constraint Configuration

[0105]

[0106] Specifically, the constraints can be configured as follows:

[0107] ① The order type constraint generated randomly during initialization can be represented as:

[0108]

[0109] Where o represents the generated order, and O is the set of order types {A,B,C,D}, where A, B, C, and D are the identifiers of different order types.

[0110] ② The movement time constraint specifically fixes the movement time between adjacent nodes to a fixed value. This provides a time reference for state transitions and reward calculations, namely:

[0111] (8)

[0112] in, and Let E represent the preceding and following nodes that are connected, and let E be the set of edges connecting the nodes.

[0113] ③ Workstation exclusivity constraint: Each workstation can only operate on one order at a time, meaning that at any given time:

[0114] (9)

[0115] Where W is the set of workstations , This indicates the packaging workstation. Indicates a conveyor belt. This indicates the workstation for attaching the order slip. For binary variables, Indicates workstation W i At time t, order o is being processed.

[0116] The operation sequence compliance rule constraint is that the workstation set follows the order of packaging workstations. Conveyor belt To the labeling station The order of events, i.e. Define the state sequence compliance function:

[0117] (10)

[0118] Where s is j represents the index of the current workstation in W (j=1,2,3 are packing, labeling, and sorting, respectively).

[0119] ④ The path length constraint is: the number of location nodes traversed in a single sorting path shall not exceed [a certain limit]. ,Right now:

[0120] (11)

[0121] Where T represents the count of the moving location node, This is an indicator function; if the limit is exceeded... By forcibly terminating the process and marking it as a sorting failure, the agent's exploration can be prevented from continuing indefinitely.

[0122] In this embodiment, the state space in the model is specifically defined as including location encoding, target workstation encoding, and order type encoding. For example, it consists of the order's location information, target workstation, and order type, and is represented by discrete encoding.

[0123] (12)

[0124] in, Location codes are used to represent information about the current location node, including order receiving systems, conveyor networks, packing stations, labeling stations, and loading stations. Code the target workstation. Encode the order type.

[0125] Specifically, each encoding can use one-hot encoding; for example, the positional encoding contains 19 bits. It contains the fields start, conveyor1-10, sensor1-2, assembly1-2, and station0-3. Here, start represents the order receiving system, and conveyor1-10 represent conveyor network 1-10. The node information for each location is shown in Table 3, along with the target workstation code. For the target workstation of the order One-hot encoding. Order type encoding. For order type One-hot encoding.

[0126] Table 3: Location Node Information in the Simulation Environment

[0127]

[0128] As shown in the table, conveyors 1-10 constitute a circular conveyor belt, and the sensor is two sensor nodes on conveyor belts 1 and 2. It is defined as a packing station for packing products for orders delivered here. The target station is equivalent to the terminal station in the simulation environment. Considering the randomness of personnel and uncertainties such as shift changes and rest, this embodiment ignores the design of manual loading in the simulation environment.

[0129] In this embodiment, the action space is defined as including actions that move along the conveyor belt path and actions that stop at a workstation and perform operations. Specifically, it can be defined as a simple binary discrete operation:

[0130] (13)

[0131] Where 0 represents movement along the conveyor belt path (e.g., the conveyor belt moves forward), and 1 represents stopping at the workstation and performing sorting operations (packaging / labeling / sorting). The action validity constraint is:

[0132] (14)

[0133] in, W represents the location node of the order, and W is the set of workstations. .

[0134] The key actions of the simulation environment are described in Table 4.

[0135] Table 4: Rules for Key Actions

[0136]

[0137] In this embodiment, the model's reward function is defined with the objectives of maximizing sorting accuracy, optimizing efficiency, and guiding key operations. Key operation guidance involves providing positive incentives to key operations to guide and reward them, ensuring the sequential execution of guided processes. A combination of sparse and dense rewards is used, with termination rewards guaranteeing the final goal, guided rewards accelerating the strategy's learning of key operations, and multi-objective optimization to achieve accuracy... Maximize, Efficiency Optimization and key operation guidance Forming the reward function:

[0138] (15)

[0139] (16)

[0140] (17)

[0141] (18)

[0142] in, This is a displacement penalty term, corresponding to the efficiency objective. As a key operation guide item, To terminate reward items so that they can be directly associated with the final task objective, ensuring that the agent prioritizes learning the correct sorting behavior, corresponding to the correctness objective. The indicator function in equation (16) is used to judge the sorting results. Indicates the order type. g Indicates the target workstation. To pre-determine the displacement penalty amount, if the order type Matching the target workstation g indicates that the sorting is correct, that is... Give strong positive rewards (e.g., +50), that is If there is a mismatch (sorting error), that is A strong negative punishment will be given. (e.g., -50), that is T represents the last displacement of the order along the sorting path. This is the maximum number of location nodes traversed in a single sorting path, only if the path length is... Timely distribution R term If timeout The reward is 0. R term This reward is only given upon completion of the mission and is considered a sparse reward, requiring other rewards to guide early exploration. (Displacement penalty item) It is distributed at every step, which is a dense reward used to provide immediate feedback and accelerate strategy learning.

[0143] Specifically, displacement penalty term This is used to guide the agent to choose a shorter path, reduce unnecessary movement, and improve sorting efficiency. This is the displacement penalty coefficient. Only when... The penalty begins at a certain point to allow the agent some margin for error during the initial exploration, when the path length... Exceeding the benchmark value (For example, a portion that can be taken as 10) is distributed proportionally. Applying negative rewards, for example, if If T=15, then... = 5×(15 10)= 25. Key Operation Guidelines To reinforce the process sequence, the indicator function in equation (18) is used to determine the state of the agent. This represents the state of the agent at time t. To preset the reward amount for key operations, when the agent is in its current state... At the packing station Or label a single workstation ,Right now This indicates that the current operation is a critical operation, and a positive reward will be given. (e.g., +10) to explicitly guide the operation steps to complete key processes in sequence: packaging → labeling → sorting, avoiding skipping or reversing the order.

[0144] This embodiment decomposes a complex task into three sub-objectives: correctness, efficiency, and process execution order. Using reward functions as shown in formulas (16), (17), and (18), it can simultaneously optimize correctness (corresponding to the termination reward item). Efficiency (corresponding displacement penalty term) ) and the sequence of operations (corresponding to key operation guidelines) ), termination of reward items Displacement penalty for sparse rewards to provide end-goal orientation Key operation guidance items Dense rewards, by providing guidance for process execution, can solve the exploration difficulties caused by sparse rewards, while avoiding the local optimum trap of dense rewards and terminating reward items. Displacement penalty only applies upon termination. Key operation guidance items The behavior is continuously adjusted during the execution process, including terminating reward items. This ensures the final goal is achieved, with process rewards accelerating policy learning through parameters (such as...). =5, ±50) can ensure that the influence of each reward is equal, avoid the dominance of a certain goal, directly reward the process sequence through the indicator function, reduce exploration costs, ensure sorting accuracy, optimize efficiency, guide key operations, and ensure that all processes are completed in sequence.

[0145] This embodiment of the Double DQN model effectively solves problems such as Q-value overestimation, low sample utilization, and unstable convergence by employing a dual-objective network, priority empirical replay, and adaptive learning rate. It can exhibit stronger policy robustness and faster convergence speed in sorting tasks.

[0146] This embodiment employs an improved algorithm framework, combining Double DQN, priority experience replay technology, and dynamic learning rate adjustment to realize a deep reinforcement learning model based on in-warehouse sorting operations. Specifically, the network structure consists of an input layer, two hidden layers, and an output layer: the input layer contains 23 neurons, receiving position information encoding; each of the two hidden layers contains 128 neurons, both using the ReLU activation function to extract nonlinear features; the output layer has 2 neurons, corresponding to the two actions the agent can choose, directly outputting the Q-value of each action. To improve training stability, a dual-network mechanism is adopted: the local network is responsible for real-time policy generation and parameter updates, while the target network gradually synchronizes the local network parameters through soft updates, reducing the problem of Q-value overestimation. The experience replay module uses a SumTree data structure to implement priority sampling, dynamically adjusting the experience weights based on the absolute value of the TD error, with the sampling batch size set to 64. During training, the initial exploration rate of 1.0 decays exponentially to the lower limit of 0.01, and the network parameters are optimized by combining the Adam optimizer and the learning rate decay strategy. During the training process, a soft update of the target network is performed after every 10 local network updates. This is achieved through prioritized experience replay and dual-network collaborative optimization, while dynamically adjusting experience priority to improve sample utilization. This balances exploration and utilization in complex path sorting tasks, resulting in efficient and stable policy learning. The parameter settings used in the model framework are shown in Table 5.

[0147] Table 5: Model Parameter Settings

[0148]

[0149] Step S03. Based on the performance index data obtained in step S02, preliminarily identify abnormal signals, and associate the identified abnormal states with the workstations to determine an optimization scheme for optimizing the production line structure of the simulation model.

[0150] This embodiment analyzes the bottlenecks in the sorting process (such as excessive load on specific workstations or excessively long order paths for specific types) based on the simulation results of step S02, and then optimizes the production line structure of the simulation model to obtain an optimization solution.

[0151] Figure 6 This displays the statistical results of various order timeout failures in 1000 training rounds using the Double DQN model. The timeouts are determined based on environmental constraints; if the timeout exceeds a certain threshold... If any of these are considered to be cyclically delayed, intervention measures are taken to avoid affecting subsequent sorting. The number of each type is 18, 19, 10, and 32, totaling 79, indicating that there is a significant problem of cyclical delay when the agent is exploring sorting strategies in the early stages. Since the target station 3 for type D orders is located at the end of the sorting path in the simulation environment, its sorting path length is much longer than that of other types of orders. Therefore, the cyclical delay situation is much higher for type D orders than for other order types. This problem can be improved by improving the production line structure and reducing the sorting path length.

[0152] Figure 7 This shows the statistical results of the sorting accuracy of the Double DQN model at four target workstations after removing orders that were interfered with due to cyclic delays, during the training process of 1000 training rounds. Figure 7 It can be seen that the total number of orders generated in the four categories of ABCD are 253, 259, 240 and 248 respectively, while the number of orders sorted by target workstations 0-3 are 269, 236, 219 and 197 respectively, showing a decreasing trend. The number of sorted orders at subsequent workstations is affected not only by the orders that are processed due to the cyclical delay problem, but also by the number of orders sorted by target workstation 1.

[0153] Analysis of the results shows that the sorting accuracy of the four target workstations is above 85%, specifically 85.1%, 92.8%, 94.5%, and 96.4%, with 40, 17, 12, and 7 orders sorted incorrectly, respectively. Based on the simulated environment's target workstation location settings, the sorting accuracy of target workstation 0 is lower than that of later target workstations, indicating that the sorting accuracy of later workstations in the sorting conveyor network may be lower. Specifically, target workstation 0 has the lowest sorting accuracy, suggesting that the model's sorting strategy for target workstation 0 is not mature enough. Since all orders pass through target workstation 0 on the circular conveyor path, this may cause the workstation to need to identify too many orders, thus affecting its sorting operation. The sorting accuracy of target workstations 1-3 is significantly higher, all exceeding 90%. This is partly because the workstations are relatively later in the path, resulting in fewer orders passing through; and partly because the earlier workstations have filtered out most of the correct orders, leaving the subsequent orders relatively homogeneous. Therefore, sorting errors in the early stages will affect the sorting accuracy of subsequent target workstations, especially target workstation 0. If there are a large number of BCD type orders with sorting errors in target workstation 0, the number of orders to be sorted in subsequent target workstations will be greatly reduced, which will seriously affect the efficiency of sorting operations. In the future, it is necessary to focus on optimizing the path selection logic or adjusting and improving the production line structure.

[0154] Figure 8 This is a statistical graph showing the access frequency of nodes at different positions in the model during 1000 training rounds. During the training process of 1000 training rounds, the total number of accesses by the agent was 10572.

[0155] The results above show that, based on the simulation environment settings and the statistics of the number of orders in categories A, B, C, and D, the access frequency of nodes at different locations can reflect different problems. For example, the agent's state and action selection may cause the number of node visits to far exceed the actual number of training sessions. For instance, when the order should be sorted to the target workstation, the agent may choose the conveyor belt path instead, causing the order to enter a loop or fail to be sorted. In this simulation environment, a total of 1000 orders were generated. Conveyor3 and conveyeyor4 are essential nodes on the sorting path in this simulation environment, and theoretically, the number should be around 1000. However, only conveyeyor3 had 224 more visits than expected. Figure 7 The range of values ​​in the later stages indicates that the agent mainly suffers from repetitive actions and looping delays due to incorrect path selection in the early stages of decision-making. Furthermore, the sum of visits to the packaging and labeling stations (sensor and assembly) on the two conveyor paths is roughly equivalent to the total number of orders, indicating that repetitive actions and delays at these two locations are relatively rare during the agent's exploration of the sorting path. Conveyors 4-7 are the nodes preceding each target station. If an order is correctly sorted at its corresponding target station, its visit count should decrease sequentially, with a difference of approximately 250 visits, roughly corresponding to the results in the figure. Looking at the correct sorting of orders of categories A, B, C, and D, stations 0 and 3 have 40 and 7 more visits respectively, while stations 1 and 2 have 6 and 2 fewer visits respectively. This indicates that, besides the intervened sorting orders, some orders are sorted to the wrong station, with station 0 having a more significant problem, possibly because this target station is relatively early in the simulation environment and requires processing a large number of orders. The number of visits to conveyors 8 and 9 were 61 and 70 respectively, indicating that some orders were not sorted to the correct workstations in time and entered the loop conveyor. Conveyor 10 was visited 918 times. Besides the approximately 500 orders entering the sorting operation via conveyor 2, which may account for half of the visits, the majority were due to issues with action selection and looping. Furthermore, the visit frequency of conveyor 1 was 9.8%, and that of conveyor 2 was 7.9%. Due to the environmental settings, these two nodes are connected to the order receiving system (start), and their combined frequency (17.7%) should be equal to this, but the result is over 80%. In particular, the number of visits to conveyor 1 alone is more than that of start, indicating that in the early exploration process, the agent repeatedly explored the same node, failing to effectively learn the critical path exploration strategy, resulting in significant looping issues with orders.

[0156] In summary, considering the abnormally high number of location node visits, the discrepancy between the number of orders and the number of visits to the target workstation, and the excessive number of visits along the circular conveyor path, the agent's actions and state selection during the warehouse sorting simulation training based on the Double DQN model frequently resulted in orders getting stuck in loops, potentially leading to low sorting efficiency and an increased risk of sorting errors. Therefore, future optimizations could focus on improving the production line structure to reduce such ineffective visits and redundant paths.

[0157] Further bottleneck analysis revealed a relatively simple simulation environment lacking complex processes. Based on key node analysis, a corresponding relationship existed between the packaging and labeling stations and the order receiving system. Increased order volume impacted the efficiency of these two types of stations, with the primary solution being to increase the number of packaging and labeling stations. However, this increased cost necessitated a comprehensive consideration of order volume and revenue. From a production line design perspective, for orders targeting stations 2 and 3, regardless of whether conveyor belt 1 or 2 was chosen, these stations, located towards the end of the circulating conveyor belt, experienced a relatively large number of nodes during sorting. Previous analysis indicated that longer sorting paths resulted in significant time penalties during path planning. Furthermore, the sorting robots at the earlier target stations 0 / 1 faced the burden of handling excessive orders, increasing the likelihood of sorting errors and order stagnation. A high volume of these two types of orders not only prolonged sorting times, significantly impacting efficiency, but also reduced accuracy and increased task difficulty. Therefore, it is advisable to optimize the design of the production line structure and correspondingly optimize the simulation environment settings in the deep reinforcement learning model.

[0158] In this embodiment, to address the bottleneck problem in the sorting operation, step S03 specifically involves dynamically identifying key problem workstations, quantifying path redundancy, and constructing a shortcut path topology reconstruction to form a generalizable production line structure optimization method, thereby determining an optimization scheme for the production line structure of the simulation model. The status parameters obtained in step S02 for each sorting workstation include the number of path nodes for each order type in each sorting workstation, the order processing volume of each sorting workstation, and the sorting accuracy of each sorting workstation. Detailed steps include:

[0159] Step S301. Identify key problem workstations, including redundant workstations with path redundancy and repeated visits, overloaded workstations with load exceeding a specified proportion of the predicted order volume, and workstations requiring optimization with a sorting accuracy rate lower than a preset threshold:

[0160] Step S311. Path Redundancy Detection: Calculate the average number of path nodes for each order type to identify workstations whose path length exceeds a preset proportion (e.g., 2 / 3) of the global average, and calculate the proportion of repeatedly accessed nodes in the path to identify redundant workstations with a risk of cyclical delays. For example, workstations whose path length exceeds 2 / 3 of the global average and whose proportion of repeatedly accessed nodes exceeds a preset proportion can be identified as redundant workstations.

[0161] Step S312. Load imbalance assessment: Obtain the order processing volume per unit time of each sorting station, and dynamically assess the matching degree between the load (order processing volume) of each sorting station and the predicted order volume according to the order volume prediction model, identify overloaded stations whose load exceeds the preset proportion of the predicted order volume, so as to provide early warning of potential bottlenecks.

[0162] Step S313. Sorting accuracy threshold judgment: Based on the preset lower limit threshold for sorting accuracy (e.g., 95%), select target workstations that are below the threshold and mark them as workstations that need optimization.

[0163] Step S302. Quantify path redundancy: Construct a quick diversion path based on the identified key problem workstations. The first end connects to the assembly workstation, the starting point of the new conveyor belt is selected from the assembly workstation with the highest order generation frequency, and the end branch is multi-directionally branched and the endpoint is connected to the loading workstation of the current path and the loading workstation of the next path, so that the intelligent agent can be transported by the new conveyor belt and can be directly sorted to the corresponding loading workstation at the end.

[0164] Specifically, the planning principles for constructing a fast path are as follows: the first end connects to the assembly station, and the starting point of the new conveyor belt is selected from the assembly station with a high order generation frequency to reduce invalid movement in the initial stage of the order; the end branching is multi-directional: the endpoint connects to the loading station of the current path and the loading station of the next path to form a "one-to-many" fast channel, and the principle of minimizing the path length is followed to ensure that the number of nodes in the new path is less than the specified proportion (such as 30%) of the original path.

[0165] Step S303. Construct a quick path topology reconstruction: Divert orders concentrated in the front part of the workstations to the back part of the workstations through the addition of a new conveyor belt to achieve order diversion. Monitor the load of the diverted workstations in real time. If the load rate of the target workstation still exceeds the threshold, a second diversion will be triggered.

[0166] In this embodiment, when determining the optimization scheme, if at least one downstream target station is found to have path redundancy or a sorting accuracy rate lower than a preset threshold, it is determined that a new conveyor belt needs to be added to the simulation model to form a fast sorting path. Orders originally concentrated at the upstream workstations are diverted to the fast path of the downstream workstations via the new conveyor belt, thereby reducing the order processing pressure on high-load workstations. The first end of the new conveyor belt connects to the assembly workstation on the current path, and the last end connects to the loading workstation on the current conveyor path and the loading workstation on the next conveyor path, respectively, so that the agent can be transported via the new conveyor belt and directly sorted to the corresponding loading workstation at the end. For example, if the order volume of target workstation 0 / 1 accounts for more than 60%, then 20% of the orders are forced to be diverted to workstation 2 / 3 via the new path.

[0167] Furthermore, the load of the workstations after the traffic diversion is monitored in real time. If the load rate of a certain workstation still exceeds the threshold, a second traffic diversion is triggered, such as adding a second shortcut path.

[0168] For example, to address the problem of orders getting stuck in a loop during the sorting process, as well as the path redundancy and low sorting accuracy of target workstations 2 and 3, this embodiment adds a conveyor belt 11 to the simulation model of the sorting operation system in this warehouse, and constructs a shortcut path of "labeling workstation 2 → conveyor belt 11 → loading workstation 2 / (conveyor belt 7 → loading workstation 3)", reconstructing the sorting logic at the topology level, reducing the path length of such orders, alleviating the sorting load of the earlier target workstations 0 and 1, and reducing the possibility of looping problems.

[0169] Path design details such as Figure 9 As shown, the first end of conveyor belt 11 connects to assembly station 2, and the last end branches to conveyor belt 6 (loading station 2) and conveyor belt 7 (loading station 3). When the target station is 2, the intelligent agent can choose to be transported via conveyor belt 11 and can be directly sorted to loading station 2 at its end; similarly, when the target station is 3, the path selection can change from the original conventional path to conveyor belt 11 → conveyor belt 7, and then be sorted to the corresponding target station.

[0170] Step S04. Optimize and reconstruct the topology of the simulation model according to the optimization scheme determined in step S03, and generate the optimized simulation model.

[0171] This embodiment first analyzes the requirements of the sorting scenario, statistically analyzes the sorting process and equipment, constructs a high-fidelity simulation environment for simulation modeling, and obtains simulation results by combining the training process of the deep reinforcement learning algorithm model. Based on the simulation results, it analyzes the system's bottlenecks and optimizes the production line structure, forming a closed loop of "algorithm-environment" collaborative optimization, thereby achieving a dual improvement in sorting accuracy and sorting efficiency. Figure 10 As shown.

[0172] The moving average reward value change curve of the Double DQN model before and after production line adjustment, obtained in a specific application embodiment, is shown below. Figure 11 As shown in the figure. Through comparison, it was found that after the production line was adjusted, the agent's exploration strategy had the same problem in the first 200 training rounds. The moving average reward value fluctuated around 20, but its initial moving average reward value fluctuated around 0, indicating that the exploration strategy problem was relatively stable in the early stage, and the occurrence of problems such as cyclical stagnation was extremely rare. Figure 12 The graph shows the moving average of the sorting path locations traversed by the Double DQN model over 1000 training rounds before and after the production line adjustment. The curves in the graph show that after the production line adjustment, the moving average of the sorting path locations traversed by the agent in the early stages (first 200 training rounds) fluctuated around point 11, exhibiting a similar problem to before the adjustment. This indicates that optimizing the sorting path for orders at target workstation 2-3 provided the agent with other, more effective sorting path exploration strategies in the early stages.

[0173] Figure 13The results show the number of effective picking orders and their corresponding picking accuracy rates for each target station in the Double DQN model over 1000 training rounds before and after production line adjustments. Throughout the training process, excluding orders that were intervened due to cyclical delays, the number of effective picking orders after the production line adjustments were 257, 214, 253, and 241, respectively, with average picking accuracy rates of 94.55%, 94.39%, 97.63%, and 97.93% over the corresponding 1000 training rounds. The number of orders with picking errors were 14, 12, 6, and 5, respectively. The number of orders with picking errors at target stations 0 and 2 decreased significantly, by 65% ​​and 50%, respectively. The average picking accuracy rates for target stations 0-3 all showed some improvement compared to before production line optimization, increasing by 11.07%, 1.71%, 3.29%, and 1.58%, respectively. In particular, compared with before the production line adjustment, the sorting accuracy of target station 0 on the sorting path was significantly improved, followed by target station 2. This indicates that the agent's early sorting path exploration strategy was improved after the production line optimization. The sorting path logic of target stations 2-3 was reconstructed, reducing the corresponding optimal sorting path length and alleviating the sorting load pressure of target stations 0-1. This reduced the number of orders that each target station needed to identify and sort, and lowered the probability of sorting errors in the early sorting decision-making process of the agent. This shows that designing a conveyor belt path that can directly reach the sorting station in the sorting path can effectively improve the impact of other target loading stations on relatively later stations. Therefore, in subsequent optimizations, for the production line structure improvement of the simulation model of the sorting operation system in this warehouse, the circular conveyor belt path can be simplified and changed to a single direct path to fundamentally solve the impact of sorting errors at the front sorting station on subsequent stations.

[0174] Figure 14 This displays the statistical results of order timeout failures due to loop delays during model training before and after production line adjustments. After the production line adjustments, orders in categories A, B, C, and D failed due to exceeding the maximum timeout limit. The number of orders deemed as sorting failures due to limitations were 4, 9, 8, and 14, respectively. After the production line adjustment, the occurrence of such problems decreased from 79 to 35, a reduction of 55.70%. The decrease was even greater for AD type orders, at 77.78% and 56.25%, respectively. This indicates that the problem of cyclical congestion during the early exploration of sorting strategies by the agent was significantly reduced compared to before the production line adjustment. This was because the sorting path length for CD type orders was shortened after the production line adjustment, reducing the load pressure on the original conventional sorting path and thus effectively improving the cyclical congestion problem.

[0175] Figure 15The data shows the number of visits and frequencies of each node in the Double DQN model during 1000 training rounds before and after the production line adjustment. The total number of visits was 8420, a decrease of approximately 20.36% compared to before the adjustment. Looking at the frequency of visits to each node, the sum of visits to conveyor1 and 2 decreased significantly, by 21.50%, indicating that issues such as repeated visits and delays during the early exploration of the sorting path by the agent before the production line adjustment were alleviated. Regarding the logical reconstruction of the sorting path, the number of visits to conveyor3 was 745, no longer exceeding the total number of orders, a decrease of 36.5%, but still higher than the theoretical sum of the total number of orders entering conveyor1 (475) and the number of visits to conveyor10 (175), i.e., 650 times. Therefore, during the sorting process, either some orders entering the conveyor1 path still experienced minor duplicate delays, or some orders with target workstations 2 or 3 may not have chosen the better path after the production line reconstruction during the early exploration process. The number of visits to conveyors 4-7 also decreased, reducing the load on the target workstations at the front of the production line before the adjustment, resulting in a more balanced distribution of nodes across all positions. Furthermore, the number of visits to conveyors 9 and 10 decreased by 21.43% and 80.94% respectively, indicating a significant reduction in repeated delays and loop transfers during the Double DQN model training process. The number of visits to conveyor 11 was 421, slightly lower than the theoretically expected total of 485 orders that should have entered conveyor 2, demonstrating that the production line restructuring was effectively utilized and optimized for improved sorting accuracy and efficiency.

[0176] In summary, this invention combines deep reinforcement learning algorithms with structural optimization of the simulation environment to form a collaborative optimization framework of positive feedback between the algorithm and the environment. This framework can systematically improve sorting performance, increase sorting accuracy, reduce the number of nodes along the optimal sorting path, improve sorting efficiency, and reduce the number of orders that fail due to looping delays. At the same time, through data analysis during simulation and algorithm training, it can accurately identify system bottlenecks (such as load imbalance and path redundancy) and adjust the production line structure at the topology level. The optimization effect is continuous and stable, and it can be easily transferred to other complex logistics sorting and production scheduling scenarios.

[0177] This embodiment further provides a computer device, including a processor and a memory, wherein the memory is used to store a computer program and the processor is used to execute the computer program to perform the method as described above.

[0178] It is understood that the method described in this embodiment can be executed by a single device, such as a computer or server, or it can be applied to a distributed scenario where multiple devices cooperate to complete the task. In a distributed scenario, one of the multiple devices may execute only one or more steps of the method described in this embodiment, and the multiple devices interact to complete the method. The processor can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit, or one or more integrated circuits, and is used to execute relevant programs to implement the method described in this embodiment. The memory can be implemented using read-only memory (ROM), random access memory (RAM), static storage devices, and dynamic storage devices. The memory can store the operating system and other applications. When the method described in this embodiment is implemented through software or firmware, the relevant program code is stored in the memory and called and executed by the processor.

[0179] This embodiment further provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.

[0180] Those skilled in the art will understand that the above embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0181] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Therefore, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention should fall within the protection scope of the present invention.

Claims

1. A simulation optimization method for an in-warehouse sorting operation system, characterized by the following steps: Comprise: Step S01. Construct a simulation model of the sorting job system in the warehouse, the model comprising a conveying network composed of multiple conveying belts, multiple sorting stations, and an order processing module for simulating the order processing flow; Step S02. In the constructed simulation model, a double-target network-based deep reinforcement learning model is used for sorting task simulation training, and state parameters of each sorting station in the simulation training process are obtained, the double-target network comprising a local network and a target network, the DQN approximating the Q value function through a neural network, the local network being used for selecting the current optimal action, and the target network being used for calculating the target Q value and periodically synchronizing parameters from the local network; Step S03. According to the state parameters of each sorting station obtained in step S02, the required optimization nodes in the line structure of the simulation model are identified, and an optimization scheme of the line structure is obtained by constructing a shunt path according to the identified required optimization nodes; Step S04. The topology structure of the simulation model is optimized and reconstructed according to the optimization scheme obtained in step S03, and an optimized simulation model is generated; In step S02, the target network calculates Q Value: wherein, denotes the target network, denotes the local network, denotes the discount factor, denotes the immediate reward, denotes the local network parameter, denotes the target network parameter, and denote the updated next state and action, respectively; In the experience replay process of the deep reinforcement learning model, a priority sampling strategy based on the time difference error is used for sampling, wherein the calculation expression of the sampling probability of the sample is: wherein, is the TD error for the i sample, is the control sampling preference degree, is a preset constant; According to the sampling probability Adjust the importance sampling weight corresponding to the sample: wherein, is the importance sampling weight for the i th sample, β is the control bias correction strength, denotes the total number of samples.

2. The in-warehouse sortation operation system simulation optimization method of claim 1, wherein, In step S01, the conveying network is a double-path circulation network, the sorting stations include a packing station, a labeling station, and multiple loading stations, the packing station is used to trigger a packing action, the labeling station is used to trigger a labeling action, the loading station is the terminal point of the order that reaches and completes sorting through the conveying network, and the order processing module is configured to simulate the whole process of the order from receiving, packing, labeling, sorting to loading, including: simulating the warehouse sorting job in the simulation modeling, obtaining the order information from the order receiving terminal, taking the corresponding product from the warehouse to the conveying belt, reaching the packing station to pack the product, going to the labeling station after packing to paste the express label containing the order delivery information on the surface of the package, and then entering the circulating conveying belt, reaching the loading station before express sorting, and reaching the loading station after sorting is completed.

3. The in-warehouse sortation operation system simulation optimization method of claim 1, wherein, In step S02, an adaptive learning rate mechanism based on training error is used in the deep reinforcement learning model to adjust the learning rate: wherein, is an initial learning rate, is a current step number at time t, is a decay step, is a decay coefficient, is a learning rate at the current time t.

4. The in-library sortation system simulation optimization method of claim 1, wherein, In step S02, an environmental constraint is also included, which includes any one or more of the following: an order type randomly generated at initialization time, a moving time between adjacent position nodes, only one order can be operated at each station at a time, the number of orders operated at each sorting station at a time, the maximum number of position nodes passed through by a single sorting path, and an order operation sequence compliance rule, which is a sequence of stations followed by the packing station , the conveyor belt to the label station ; and a state space of the model is defined as including a position code, a target station code, and an order type code, the position code is used to correspond to information representing a current position node, the position node includes an order receiving system, a conveyor belt network, a packing station, a label station, and a loading station, the target station code is used to represent information of a target position node of sorting, an action space of the model is defined as including an action of moving along a conveyor belt path and an action of stopping at a station and performing a sorting operation, the sorting operation includes packing, labeling, and sorting, a reward function of the model is defined with the goals of maximizing sorting accuracy, optimizing efficiency, and guiding key operations, wherein the guiding key operations is to give positive incentives to specified key operations to guide reward key operations.

5. The in-warehouse sortation operation system simulation optimization method of claim 4, wherein, The calculation expression of the reward function is: wherein, is a displacement penalty term to correspond to an efficiency target, is a critical operation guiding term to correspond to a procedure execution order target, is a termination reward term to correspond to an accuracy target, is an indicator function, represents an order type, g represents a target station, is a preset displacement penalty amount, if and the order type matches the target station g , it means the sorting is correct, i.e. , then to reward, if not, i.e. , then to punish, if , then , T represents the last displacement of the sorting path of the order, is the maximum number of position nodes passed by a single sorting path, is a displacement penalty coefficient, is a packing station, is a labeling station, if the current state of the agent is at the packing station or the labeling station , it means the current operation is a critical operation, then to reward, is a preset critical operation guiding reward amount.

6. The in-warehouse sortation operation system simulation optimization method according to any one of claims 1 to 5, characterized in that, Step S03 includes: Step S301. Identify key problem stations, which include redundant stations with path redundancy and repeated access, overloaded stations with load exceeding a specified proportion of predicted order quantity, and optimization required stations with sorting accuracy lower than a preset threshold; Step S302. Quantify path redundancy: according to the identified key problem stations, a shortcut shunt path is constructed, wherein the first end is connected to the assembly station, the starting point of the new conveying belt is selected as the assembly station with the highest order generation frequency, the end is branched in multiple directions and connected to the loading stations of the current path and the next path. Step S303. Path topology reconstruction: orders concentrated on the front part of the station are diverted to the rear part of the station through the newly added conveyor to realize order diversion, and the load of the diverted station is monitored in real time. If the target station load rate still exceeds the threshold, further trigger secondary diversion.

7. The in-warehouse sortation operation system simulation optimization method of claim 6, wherein, In step S301, the average path node number of each order type is counted to identify the stations with path length higher than the global average by a preset proportion, and the proportion of repeated access nodes in the path is calculated to obtain redundant stations with the risk of circular retention. The unit time order processing capacity of each sorting station is obtained, and the matching degree of the load capacity and the predicted order quantity of each sorting station is dynamically evaluated according to the order quantity prediction model to identify overloaded stations with load capacity exceeding the predicted order quantity by a preset proportion.

8. A computer device comprising a processor and a memory for storing a computer program, characterized in that, The processor is configured to execute the computer program to perform the method of any one of claims 1-7.

9. A computer readable storage medium storing a computer program, characterized in that, The computer program, when executed, implements the method of any one of claims 1-7.

Images