In-warehouse sorting operation system simulation optimization method, equipment and medium

Abstract

The invention discloses an in-library sorting operation system simulation optimization method, equipment and a medium. The method comprises the following steps: step S01, constructing a simulation model of an in-library sorting operation system; s02, sorting task simulation training is carried out in the constructed simulation model based on a deep reinforcement learning model of a double-target network, and state parameters of all sorting stations in the simulation training process are obtained; s03, according to the state parameters, obtained in the step S02, of all the sorting stations, identifying nodes needing to be optimized in the production line structure of the simulation model, and according to the identified nodes needing to be optimized, constructing a shunting path to obtain an optimization scheme of the production line structure; and step S04, performing optimization reconstruction on the topological structure of the simulation model according to the obtained optimization scheme, and generating an optimized simulation model. According to the method, the sorting performance can be systematically improved, the sorting accuracy and efficiency are balanced, and meanwhile it is ensured that the optimization effect is continuous and stable.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of in-warehouse sorting automation control, and particularly relates to an in-warehouse sorting operation system simulation optimization method, device and medium. BACKGROUND

[0002] With the rapid development of e-commerce and the individualization and diversification of consumer demand, the logistics and warehousing industry is facing the challenge of insufficient sorting efficiency of multi-category and small-batch orders. As a core link of logistics and warehousing, the efficiency of in-warehouse sorting operation directly affects the operation efficiency of the entire supply chain. The traditional sorting method mainly relies on manual experience and fixed sorting process, lacks scientific optimization means, and is difficult to cope with dynamic and complex order environment, which easily leads to problems such as low sorting efficiency, redundant path, order circulation retention and low sorting accuracy.

[0003] To realize intelligent optimization of the sorting system, the existing technology usually uses simulation modeling software (such as PlantSimulation) to perform static analysis and optimization on the sorting process, or combines heuristic algorithms (such as genetic algorithm) for scheduling. However, the above methods are mostly static or offline optimization, and are difficult to cope with dynamic changes and randomness in the sorting process. Although reinforcement learning is introduced into the path planning problem, the traditional Q-learning or basic deep Q network (DQN) algorithm has problems such as Q-value overestimation, low exploration efficiency and slow convergence speed, and lacks strategy robustness and generalization ability in complex in-warehouse sorting environment. At the same time, the optimization scheme in the existing technology often focuses on a single aspect of algorithm or production line structure, and cannot realize the collaborative optimization of algorithm and "simulation environment structural optimization, which makes it difficult to fundamentally and systematically improve the sorting efficiency.

[0004] Specifically, the existing technology has the following problems for in-warehouse sorting optimization scheme: (1) Insufficient dynamic decision-making capability: the traditional static scheduling model cannot respond to order changes and equipment status in real time, leading to suboptimal path planning and easy generation of circulation retention and path redundancy.

[0005] (2) Algorithm performance limitation: the basic reinforcement learning algorithm is prone to Q-value overestimation, unstable training and slow convergence in the sorting scene, affecting the strategy quality.

[0006] (3) Conflict between sorting accuracy and efficiency: the front sorting station has a heavy load, a high error recognition rate, and the error has a cumulative effect on subsequent stations, while the rear sorting station has a long sorting path, affecting the overall efficiency.

[0007] (4) Single optimization dimension: the existing technology scheme is mostly limited to algorithm improvement or process fine-tuning, lacks structural reconstruction from the system topology level, and is difficult to fundamentally eliminate the bottleneck of the collaborative optimization method. SUMMARY

[0008] The technical problem to be solved by the present application is that in view of the above problems existing in the prior art, the present application provides a warehouse sorting operation system simulation optimization method, device and medium, which has the advantages of simple implementation method, high sorting efficiency and accuracy, and continuous and stable optimization effect, and can systematically improve the sorting performance and balance the sorting accuracy and efficiency.

[0009] To solve the above technical problems, the technical solution provided by the present application is: A warehouse sorting operation system simulation optimization method, comprising the following steps: Step S01. Construct a simulation model of the warehouse sorting operation system, the model comprising a conveying network composed of multiple conveying belts, multiple sorting stations, and an order processing module for simulating an order processing flow; Step S02. Perform sorting task simulation training based on a double-target network deep reinforcement learning model in the constructed simulation model, and obtain state parameters of each sorting station in the simulation training process, the double-target network comprising a local network and a target network, the DQN approximating a Q value function through a neural network, the local network being used for selecting a current optimal action, and the target network being used for calculating a target Q value and periodically synchronizing parameters from the local network; Step S03. Identify a required optimization node in a production line structure of the simulation model according to the state parameters of each sorting station obtained in step S02, and construct a shunt path according to the identified required optimization node to obtain an optimization scheme of the production line structure; Step S04. Optimize and reconstruct the topological structure of the simulation model according to the optimization scheme obtained in step S03 to generate an optimized simulation model.

[0010] Further, in step S01, the conveying network is a double-path circulation network, the sorting stations comprise a packing station, a label sticking station, and multiple loading stations, the packing station is used to trigger a packing action, the label sticking station is used to trigger a label sticking action, the loading station is the terminal point of an order that has arrived through the conveying network and completed sorting, and the order processing module is configured to simulate the whole process of an order from receiving, packing, label sticking, sorting to loading, comprising: simulating warehouse sorting in simulation modeling, obtaining order information through an order receiving terminal, taking corresponding products from the warehouse to the conveying belt, performing product boxing and packing after arriving at the packing station, sticking an express label containing order delivery information to the surface of the package after packing, then entering the circulating conveying belt, performing express sorting before arriving at the loading station, and arriving at the loading station after completing the sorting.

[0011] Further, in step S02, the target network calculates the target value according to the following formula: Q ​

[0012] in, Indicates the target network. Indicates the local network. Indicates the discount factor. Indicates an immediate reward. Indicates local network parameters, Indicates the target network parameters. and These represent the next state and the action to be updated, respectively. During experience replay, the deep reinforcement learning model employs a priority sampling strategy based on temporal difference error for sampling. The expression for calculating the sampling probability of a sample is as follows:

[0013]

[0014] in, For the first i TD error for each sample To control the degree of sampling bias, This is a preset constant; Based on sampling probability Adjust the importance sampling weights for the samples:

[0015] in, For the first i The importance sampling weights for each sample, where β controls the strength of the bias correction. This represents the total number of samples.

[0016] Furthermore, in step S02, the deep reinforcement learning model employs an adaptive learning rate mechanism based on training error to adjust the learning rate:

[0017] in, The initial learning rate, The current step number at time t. To decay step size, The attenuation coefficient is... Let t be the learning rate at time t.

[0018] Further, in step S02, the environment constraint conditions are also set, including any one or more of the following: the order type is randomly generated at initialization, the moving time between adjacent position nodes, each workstation can only operate one order at a time, the number of orders operated by each sorting workstation at a time, the maximum number of position nodes passed by a single sorting path, and the order operation sequence compliance rule, which is that the workstation set follows the order of packing workstation , conveyor belt to label workstation in turn; the state space of the model is defined as including position encoding, target workstation encoding, and order type encoding, the position encoding is used to correspond to the information representing the current position node, the position node includes order receiving system, conveyor belt network, packing workstation, label workstation, and loading workstation, the target workstation encoding is used to represent the information of the target position node of sorting, the action space is defined as including the action of moving along the conveyor belt path and the action of stopping at the workstation and performing the sorting operation, the sorting operation includes packing, labeling, and sorting, the reward function of the model is defined with the goal of maximizing the sorting accuracy, optimizing the efficiency, and guiding the key operation, the key operation guidance is to guide the reward key operation by giving positive incentive to the specified key operation.

[0019] Further, the calculation expression of the reward function is:

[0020]

[0021]

[0022]

[0023] wherein, is the displacement penalty term corresponding to the efficiency target, is the key operation guidance term, is the termination reward term corresponding to the accuracy target, is the indicator function, represents the order type, g represents the target workstation, is a preset displacement penalty amount, if the order type matches the target workstation g , it means that the sorting is correct, i.e. , then is rewarded, if it does not match, i.e. , then is punished, T represents the last displacement of the sorting path of the order, the maximum number of position nodes passed by a single sorting path, a displacement penalty coefficient, a packing station, a labeling station, if the current state of the agent is at the packing station or the labeling station then it means that the current operation belongs to a critical operation, then a reward is given, a preset critical operation guidance reward amount.

[0024] Further, step S03 comprises: Step S301. Identify critical problem stations, which include redundant stations with path redundancy and repeated access, overloaded stations with load exceeding a specified proportion of predicted order quantity, and optimization-required stations with sorting accuracy lower than a preset threshold; Step S302. Quantify path redundancy: construct a fast diversion path according to the identified critical problem stations, wherein the starting point of the newly added conveyor belt selects the assembly station with the highest order generation frequency, the ending point connects to the loading station of the current path and the loading station of the next path, and the path branches in multiple directions; Step S303. Path topology reconstruction: divert orders concentrated in the front part of the station to the rear part of the station through the newly added conveyor belt to achieve order diversion, and monitor the load of the diverted station in real time. If the target station load rate still exceeds the threshold, further trigger secondary diversion.

[0025] Further, in step S301, the average path node number of each order type is counted to identify stations with path length higher than a preset proportion of the global average, and the proportion of repeated access nodes in the path is calculated to obtain redundant stations with risk of circular retention. The order processing capacity of each sorting station is obtained, and the matching degree of the load of each sorting station and the predicted order quantity is dynamically evaluated according to the order quantity prediction model to identify overloaded stations with load exceeding a preset proportion of the predicted order quantity.

[0026] A computer device comprises a processor and a memory, the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0027] A computer readable storage medium storing a computer program, the computer program is executed to implement the above method.

[0028] Compared with the prior art, the advantages of the present application are: 1. The application can form a "algorithm-environment" positive feedback in-house sorting operation system structure collaborative optimization framework by combining deep reinforcement learning algorithm with structural optimization of simulation environment, can systematically improve sorting performance, improve sorting accuracy, reduce the position nodes passed by the optimal sorting path, improve sorting efficiency, and reduce the number of orders failed due to circulation retention.

[0029] 2. The application can accurately identify system structure bottlenecks by combining simulation and data analysis in the deep reinforcement learning algorithm model training process, so that the production line structure can be adjusted from the topological level, the system structure can be accurately performed, the optimization effect is continuous and stable, and it can be conveniently migrated to other complex logistics sorting and production scheduling scenes. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 is the implementation process schematic diagram of the in-house sorting operation system simulation optimization method of the embodiment.

[0031] Figure 2 is the in-house sorting operation system model schematic diagram constructed in the specific application embodiment of the application.

[0032] Figure 3 is the implementation process schematic diagram of the simulation sorting order processing in the specific application embodiment of the application.

[0033] Figure 4 is the structure principle schematic diagram of the deep reinforcement learning model based on the double target network used in the embodiment.

[0034] Figure 5 is the priority experience replay implementation process schematic diagram in the embodiment.

[0035] Figure 6 is the statistical result schematic diagram of the Double DQN model of each type of order timeout failure order obtained in the specific application embodiment.

[0036] Figure 7 is the statistical result schematic diagram of the sorting accuracy of the Double DQN model target workstations 0-3 obtained in the specific application embodiment.

[0037] Figure 8 is the statistical result schematic diagram of the Double DQN model of each position node access frequency obtained in the specific application embodiment.

[0038] Figure 9 is the principle schematic diagram of the in-house sorting operation system simulation modeling structural adjustment in the specific application embodiment.

[0039] Figure 10is a complete implementation flowchart of the simulation optimization of the in-warehouse sorting operation in the specific application embodiment of the present application.

[0040] Figure 11 is a model moving average reward value change curve diagram before and after the production line adjustment obtained in the specific application embodiment of the present application.

[0041] Figure 12 is a moving average value result diagram of the model sorting path node number before and after the production line adjustment obtained in the specific application embodiment of the present application.

[0042] Figure 13 is a sorting accuracy rate statistical result diagram of the model target workstations 0-3 before and after the production line adjustment obtained in the specific application embodiment of the present application.

[0043] Figure 14 is a statistical result diagram of each type of order timeout failure of the model before and after the production line adjustment obtained in the specific application embodiment of the present application.

[0044] Figure 15 is a statistical result diagram of the model each position node access frequency before and after the production line adjustment obtained in the specific application embodiment of the present application. DETAILED DESCRIPTION

[0045] The present application is further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the protection scope of the present application is not limited thereby.

[0046] As shown in Figure 1 , the steps of the in-warehouse sorting operation system simulation optimization method of the present embodiment include: Step S01. Construct a simulation model of the in-warehouse sorting operation system, the model including a conveying network composed of multiple conveying belts, multiple sorting workstations, and an order processing module for simulating the order processing flow.

[0047] In the present embodiment, the conveying network is a double-path circulation network, the sorting workstations include a packing workstation, a label sticking workstation, and multiple loading workstations, the packing workstation is used to trigger a packing action, the label sticking workstation is used to trigger a label sticking action, and the loading workstations are the end points of the orders that arrive through the conveying network and complete sorting.

[0048] In the specific application embodiment, the digital simulation model of the in-warehouse sorting operation system constructed based on the discrete event simulation platform such as Plant Simulation is as shown in Figure 2As shown in Table 1, a dual-path circulating conveyor network system consisting of 10 conveyor belts (conveyor belts 1-10) serves target workstations 0-3. Sorting workstations include packing workstations, labeling workstations, and multiple loading workstations. The packing workstation is located between conveyors 1 and 2 and is used to trigger the packing action. The labeling workstation is used to trigger the labeling action. The loading workstation is the final destination of the order, requiring the order to be reached via the conveyor network and correctly sorted. The distance between each workstation and the conveyor belt is equal, and the conveyor belt speed is fixed. Therefore, the time taken to travel from one location node to the next during order delivery is the same. The number of each workstation is shown in Table 1.

[0049] Table 1: Equipment Statistics Table

[0050] In this embodiment, the order processing module is configured to simulate the entire process of an order from receiving, packing, labeling, sorting to loading. This includes: simulating sorting operations in the warehouse through simulation modeling; obtaining order information through the order receiving terminal and retrieving the corresponding product from the warehouse and sending it onto the conveyor belt; packing the product at the packing station; after packing, proceeding to the labeling station to affix the courier label containing the order delivery information to the surface of the package; then entering the circulating conveyor belt; sorting the courier before reaching the loading station; and finally, arriving at the loading station after sorting.

[0051] Specifically, the simulation covers the entire process of an order, from receiving, packing, labeling, sorting to loading. Figure 3 As shown, the simulation model simulates the sorting operation in the warehouse. Order information is obtained through the order receiving terminal, the corresponding product is taken out from the warehouse and sent to the conveyor belt. After reaching the sensor position (i.e., the packing station), the product is packed by the machine. After packing, it goes to the labeling station to affix the express label containing the order delivery information to the surface of the package. Then it enters the circulating conveyor belt and is sorted before reaching the loading station. After sorting, it reaches the loading station and is finally loaded into the vehicle.

[0052] Step S02. In the constructed simulation model, a deep reinforcement learning model based on a dual-objective network is used for sorting task simulation training. Performance index data for specified workstations and location nodes are obtained during the simulation training process. The dual-objective network includes a local network and a target network. DQN approximates the target network through a neural network. Q The value function is used by the local network to select the current optimal action, and the target network is used to calculate the target. Q The value is set and parameters are periodically synchronized from the local network.

[0053] In this embodiment, a deep reinforcement learning model with a dual-objective network is used for sorting task simulation training. The model architecture is as follows: Figure 4The algorithm architecture after adding the improved strategy in the DQN (Double Deep Q Network) model is shown, taking the double deep Q network as the main framework, performing dynamic learning rate adjustment when parameter updating back propagation, and performing priority experience sampling when experience replay.

[0054] Specifically, the deep Q network (DQN) approximates the value function Q by a neural network , which represents the long-term cumulative reward expectation of performing action a in state s, and is approximated by neural network parameters θ . Q The value function guides the agent to select the optimal action, and its update follows the Bellman equation: (1) wherein is the immediate reward, is the discount factor, is the target network parameter, respectively represent the updated next state and action.

[0055] The update target of the DQN is to minimize the time difference error (TD error), that is, to minimize the error between the predicted Q value and the target Q value to update the network parameters, and therefore, the loss function is defined as: (2) wherein is the main network parameter.

[0056] The embodiment introduces a double deep Q network to separate action selection and Q value estimation, which can alleviate the problem that the Q value estimation of the traditional DQN is prone to deviation due to the lag of the target network update. In the double deep Q network, the local network is used to select the current optimal action; the target network is used to calculate the target Q value, and the parameters are synchronized from the local network at regular intervals. The target network calculates the target Q value according to the following formula: (3)

[0057] wherein represents the target network, represents the local network, represents the discount factor, represents the immediate reward, represents the local network parameter, represents the target network parameter, and These represent the next state and action to be updated, respectively.

[0058] Furthermore, during the experience replay process, the deep reinforcement learning model employs a priority sampling strategy based on temporal difference error (TD-error). Specifically, the sampling priority of experiences is set based on the absolute value of the temporal difference error, prioritizing the replay of historical experiences that are more valuable for training, which can accelerate convergence. This embodiment uses a priority sampling strategy based on temporal difference error, which can accelerate the learning of key experiences by focusing on high-error samples.

[0059] Preferably, the expression for calculating the sampling probability of a sample is: (4) (5) in, For the first i TD error for each sample To control the degree of sampling bias, A preset small constant is used to prevent zero probability. j This represents the sample number, ranging from 1 to N, where N represents the total number of samples.

[0060] Furthermore, this embodiment is based on the sampling probability. Adjust the importance sampling weights for the samples: (6) in, For the first i The importance sampling weights for each sample, where β controls the strength of the bias correction. This indicates the total number of samples.

[0061] This embodiment uses sampling probability. Adjusting the importance sampling weights can be done based on the sampling probability. The importance sampling weight is dynamically increased to effectively focus on high-error samples.

[0062] Furthermore, deep reinforcement learning models employ an adaptive learning rate mechanism based on training error to adjust the learning rate: (7) in, The initial learning rate, Let t be the number of steps at the current time. To decay step size, The attenuation coefficient is... Let t be the learning rate at time t.

[0063] This embodiment uses the aforementioned adaptive learning rate mechanism to dynamically adjust the learning rate at different times. A high learning rate in the early stage leads to rapid convergence, while a low learning rate in the later stage fine-tunes the parameters, preventing oscillations caused by an excessively high learning rate in the later stages of training.

[0064] This embodiment, by sampling the above method and based on the improved strategy within a dual-objective network framework, can achieve efficient sampling of experience replay, dynamic optimization of parameter updates, and accurate calculation of the target Q value, accelerating the learning of key experiences. By correcting deviations through priority sampling based on TD error, it can also maintain the stability of learning and improve the utilization rate of high-value experiences, thereby significantly improving the training efficiency and stability of sorting tasks within the warehouse. At the same time, it achieves optimization of indicators such as increasing the moving average reward value during production line adjustments and reducing sorting path nodes.

[0065] Specifically, such as Figure 5 As shown, the complete process for training a deep reinforcement learning model is as follows: (1) Initialization: Set dual network parameters , Experience pool and initial learning rate η0.

[0066] (2) Simulation training: state Next, local network selection action Observation after execution , The TD error is calculated according to equation (5), i.e. Storage experience ( Go to the experience pool.

[0067] (3) Parameter update: Based on sampling probability Sampling high error experience, calculate target Q value And according to the importance weight w i Update local network: ← ,in Regular synchronization → .

[0068] (4) Synergistic effect of dynamic learning rate and priority sampling: a high learning rate is used in the initial stage. Quickly capture high-error samples, by High-probability sampling is used to accelerate convergence; a low learning rate combined with importance weights w is adopted in the later stages. i To finely adjust parameters and avoid oscillations, the TD error is used. It can both drive sampling priority and dynamically adjust the learning rate, forming a closed-loop optimization of "error-sampling-learning rate".

[0069] Further, the environment constraints are also included, which include the order type randomly generated at initialization, the moving time between adjacent position nodes, the exclusive operation of each station on one order at a time (station exclusive), the number of orders operated by each sorting station at a time, the maximum number of position nodes passed by a single sorting path (path length limit), and the order operation sequence compliance rules, etc. The moving time constraint is used in state transition and reward calculation processes to provide a time benchmark. The path length limit constraint is used in state definition and experience replay processes to limit the length of the action sequence. The station exclusive constraint is used in state definition and experience replay processes by being implicitly included in the state encoding to filter illegal experiences. The operation sequence compliance rules are used in state transition, action selection, and parameter update processes to ensure the legality of the station order and dynamically filter illegal actions by adjusting the sampling priority and weight of compliant samples, as shown in Table 2.

[0070] Table 2: Constraint condition configuration

[0071] Specifically, each constraint condition can be configured as follows: ① The order type randomly generated at initialization constraint can be represented as:

[0072] where o represents the generated order, O is the order type set {A, B, C, D}, and A, B, C, and D are the identifiers of different order types.

[0073] ② The moving time constraint is to fix the moving time between adjacent position nodes as to provide a time benchmark for state transition and reward calculation, i.e.: (8) where and represent the connected front and rear position nodes, and E is the edge set of connected position nodes.

[0074] ③ The station exclusive constraint is that each station can only operate on one order at a time, i.e. for any time: (9) where W is the station set , represents the packing station, represents the conveyor belt, represents the labeling station, is a binary variable, represents the station W iThe order o is being processed at time t.

[0075] The operation sequence compliance rule constrains the workstations set to follow the order of packing workstation , conveyor to the order sticking workstation , that is , and defines the state sequence compliance function: (10) Wherein, s is , j represents the index of the current workstation in W (j=1, 2, 3 respectively for packing, order sticking and sorting).

[0076] The path length limit constraint is that the number of position nodes passed by a single sorting path does not exceed , that is: (11) Wherein, T represents the count of moving position nodes, is an indicator function, if the limit is exceeded , it is forced to terminate and marked as sorting failure, which can avoid the agent from exploring endlessly.

[0077] In this embodiment, the state space in the model is specifically defined as including position encoding, target workstation encoding and order type encoding, for example, composed of order position information, target workstation and order type, represented by discrete encoding: (12) Wherein, is the position encoding for corresponding information representing the current position node, including order receiving system, conveyor network, packing workstation, order sticking workstation and loading workstation, is the target workstation encoding, is the order type encoding.

[0078] Specifically, each encoding can use one-hot encoding. Taking the position encoding containing 19 bits as an example, , containing start, conveyor1-10, sensor1-2, assembly1-2, station0-3 fields, wherein, start represents the order receiving system, conveyor1-10 represents the conveyor network 1-10, the position node information is shown in Table 3, the target workstation encoding is the one-hot encoding of the order target workstation . The order type encoding is the one-hot encoding of the order type .

[0079] Table 3: Location node information of the simulation environment

[0080] As shown in the table, conveyor1-10 constitutes a circulating conveyor, sensor is two sensor nodes on conveyors 1 and 2, defined as a packing station for product packaging of orders conveyed thereto, the target station is equivalent to the terminal station of the simulation environment, considering the randomness of personnel and uncertain situations such as shift rest, the embodiment ignores the manual loading link design in the simulation environment.

[0081] In this embodiment, the action space is defined as including actions of moving along the conveyor path and actions of stopping at the station and performing operations. Specifically, it can be defined as a simple binary discrete operation: (13) Where 0 represents moving along the conveyor path (such as the conveyor advancing), and 1 represents stopping at the station and performing sorting operations (packing / labeling / sorting). The action validity constraint is: (14) Where, represents the location node where the order is located, and W is the set of stations . The key actions of the simulation environment are introduced and explained, as shown in Table 4.

[0082] Table 4: Key action rules

[0083] In this embodiment, the reward function of the model is defined with the goals of maximizing sorting accuracy, optimizing efficiency, and guiding key operations. The guiding key operation is to guide the reward key operation by giving positive incentives to key operations to ensure the execution order of the guiding process. Through the combination of sparse-dense rewards, the final goal is guaranteed by terminating rewards, the guiding rewards accelerate the learning of key operations, and through multi-objective optimization, the accuracy is maximized, the efficiency is optimized, and the key operation guidance forms the reward function: (15) (16) (17) (18) Where, is the displacement penalty term, corresponding to the efficiency target, is the key operation guidance term, To terminate reward items so that they can be directly associated with the final task objective, ensuring that the agent prioritizes learning the correct sorting behavior, corresponding to the correctness objective. The indicator function in equation (16) is used to judge the sorting results. Indicates the order type. g Indicates the target workstation. To pre-determine the displacement penalty amount, if the order type Matching the target workstation g indicates that the sorting is correct, that is... Give strong positive rewards (e.g., +50), that is If there is a mismatch (sorting error), that is A strong negative punishment will be given. (e.g., -50), that is T represents the last displacement of the order along the sorting path. The maximum number of location nodes traversed in a single sorting path, only if the path length is... Timely distribution R term If timeout The reward is 0. R term This reward is only given upon completion of the mission and is considered a sparse reward, requiring other rewards to guide early exploration. (Displacement penalty item) Rewards are distributed at every step, forming an intensive reward system designed to provide immediate feedback and accelerate strategy learning.

[0084] Specifically, displacement penalty term This is used to guide the agent to choose a shorter path, reduce unnecessary movement, and improve sorting efficiency. This is the displacement penalty coefficient. Only when... The penalty begins at a certain point to allow the agent some margin for error during the initial exploration, when the path length... Exceeding the benchmark value (For example, a portion that can be taken as 10) is distributed proportionally. Applying negative rewards, for example, if If T=15, then... = 5×(15 10)= 25. Key Operation Guidelines To reinforce the process sequence, the indicator function in equation (18) is used to determine the state of the agent. This represents the state of the agent at time t. To preset the reward amount for key operations, when the agent is in its current state... At the packing station Or label a single workstation If , it means that the current operation belongs to the key operation, and a positive reward (such as +10) is given to explicitly guide the operation steps to complete the key process in order: packing → labeling → sorting, avoiding skipping or reversing the order.

[0085] This embodiment can optimize correctness (corresponding to the termination reward term ), efficiency (corresponding to the displacement penalty term ), and process execution order (corresponding to the key operation guiding term ) by decomposing complex tasks into correctness, efficiency, and process execution order, using reward functions such as equations (16), (17), and (18). The termination reward term is a sparse reward to provide final goal guidance, and the displacement penalty term and the key operation guiding term are dense rewards to provide process execution process guidance, which can solve the exploration difficulty problem caused by sparse rewards, while avoiding the local optimal trap of dense rewards. The termination reward term only takes effect at the end, and the displacement penalty term and the key operation guiding term continuously adjust behavior during execution. The termination reward term can ensure the final goal, and the process reward acceleration strategy learning can ensure that the influence of each reward is equivalent through parameters (such as =5, ±50), avoiding the dominance of a certain target. Through the indicator function, the process order is directly rewarded, which can also reduce the exploration cost, ensure the sorting correctness, efficiency optimization, key operation guidance, and ensure that all processes are completed in order.

[0086] The Double DQN model of this embodiment effectively solves the problems of Q value overestimation, low sample utilization, and unstable convergence by using double target networks, priority experience replay, and adaptive learning rate, and can perform stronger strategy robustness and faster convergence speed in the sorting task.

[0087] ​The embodiment realizes a deep reinforcement learning model based on in-warehouse sorting operation by using an improved algorithm framework, combining Double DQN, priority experience replay technology and dynamic learning rate adjustment. Specifically, the network structure is composed of an input layer, two hidden layers and an output layer: the input layer contains 23 neurons to receive position information encoding; the two hidden layers each contain 128 neurons, both using ReLU activation function to extract nonlinear features; the output layer is set to 2 neurons, corresponding to the two actions that the agent can choose, directly outputting the Q values of each action. To improve the stability of training, a double network mechanism is used, the local network is responsible for real-time policy generation and parameter update, the target network gradually synchronizes the local network parameters through soft update, reducing the problem of overestimation of Q values. The experience replay module realizes priority sampling through the SumTree data structure, dynamically adjusts the experience weight according to the absolute value of TD error, and the sampling batch size is set to 64. During training, the initial exploration rate 1.0 is exponentially decayed to the lower limit 0.01, combined with the Adam optimizer and learning rate decay strategy to optimize the network parameters. In the training process, the target network soft update is executed every 10 local network updates, and the priority experience replay and double network are optimized simultaneously, while dynamically adjusting the experience priority to improve sample utilization, balancing exploration and utilization in complex path sorting tasks, and realizing efficient and stable policy learning. The parameter settings used in the model framework are shown in Table 5.

[0088] Table 5: Model parameter value settings

[0089] Step S03. Preliminarily identify the abnormal signal according to the performance index data obtained in step S02, and associate the identified abnormal state with the station to determine the optimization scheme for optimizing the production line structure of the simulation model.

[0090] The embodiment optimizes the production line structure of the simulation model according to the simulation result analysis of the sorting process bottlenecks (such as excessive load of a specific station, long route of a specific type of order) to obtain an optimization scheme.

[0091] Figure 6 The statistical results of the timeout failure of various orders in the Double DQN model in 1000 training rounds are shown. According to the environment constraint condition setting, if the position node exceeds , it is regarded as a cycle retention, and intervention measures are taken without affecting subsequent sorting. The number of statistics of each type is 18, 19, 10 and 32, a total of 79, indicating that there is a relatively obvious cycle retention problem when the agent explores the sorting strategy in the early stage. Since the target station 3 of D type orders is located at the rear position in the simulation environment sorting path, its sorting path length is much larger than that of other types of orders, so its cycle retention situation is much higher than that of other order types. Subsequently, the line structure can be improved to reduce the sorting path length to improve such problems.

[0092] Figure 7 The statistical results of the four target station sorting accuracy of the Double DQN model after removing the orders intervened due to cycle retention are shown. The training process in 1000 training rounds. Figure 7 It can be seen that the total number of ABCD four types of orders generated is 253, 259, 240 and 248, and the number of orders sorted by target stations 0-3 is 269, 236, 219 and 197, showing a decreasing trend. The number of orders sorted by subsequent stations is affected by the target station 1 in addition to the orders processed due to the cycle retention problem.

[0093] According to the analysis of the display results, the sorting accuracy of the four target stations is more than 85%, which is 85.1%, 92.8%, 94.5% and 96.4% respectively, and the number of sorting errors is 40, 17, 12 and 7 respectively. According to the position setting of the target station in the simulation environment, the sorting accuracy of the target station 0 is lower than that of the target station at the rear position, indicating that the sorting accuracy of the station at the rear position in the sorting conveyor network may be lower. Among them, the sorting accuracy of the target station 0 is the lowest, indicating that the model's sorting strategy for the target station 0 is not mature enough. Since all orders will pass through the target station 0 on the circulating conveyor path, it may cause the station to need to recognize too many orders, thereby affecting its sorting operation. The sorting accuracy of target stations 1-3 is significantly increased, all exceeding 90%. On the one hand, the station position is relatively rear, and the number of orders passing through is reduced. On the other hand, the front station selects most of the correct orders, and the subsequent order type is relatively simple. Therefore, early sorting errors will affect the sorting accuracy of subsequent target stations, especially the target station 0. If the number of sorting errors of the target station 0 contains a large number of BCD type orders, the number of orders sorted by subsequent target stations will be greatly reduced, which will seriously affect the efficiency of sorting operation. Subsequently, the path selection logic needs to be optimized or the line structure needs to be adjusted and improved.

[0094] Figure 8 The frequency statistics chart of the model visiting different position nodes in 1000 training rounds is shown. The total number of visits by the agent is 10572 in 1000 training rounds.

[0095] From the above results, based on the simulation environment settings and ABCD four types of order quantity statistics, the access frequency of different location nodes can reflect different problems. For example, the state and action selection of the agent may cause the number of node visits to be much higher than the actual training times, and when it should be sorted to the target station, it chooses the conveyor belt path, causing the order to enter the cycle or fail to sort. In this simulation environment, there are 1000 orders generated. conveyor3 and conveyor4 are the nodes that must be passed in the sorting path in this simulation environment, and according to the theory, they should be around 1000, but only the access number of conveyor3 is 224 more, combined with Figure 7 the value range in the later period, it shows that the agent mainly has problems such as repeated actions, selection of wrong paths, and cycle retention in early decision-making. In addition, the sum of the access number of the packing and labeling stations (sensor and assembly) on the two conveyor belt paths is roughly equal to the total number of orders, indicating that the agent has relatively few problems such as repeated action selection and retention in these two location nodes during the exploration of the sorting path. conveyor4-7 are the last location nodes of each target station, and if the order is correctly sorted in the corresponding target station, the access number should decrease in turn, and the difference is about 250 times, which roughly corresponds to the result in the figure. According to the sorting correctness of the ABCD four types of orders, the access number of station0 and 3 is increased by 40 and 7 times respectively, and the access number of station1 and 2 is decreased by 6 and 2 times respectively, indicating that in addition to the intervention of sorting orders, there are orders sorted to the wrong station, among which station0 has a bigger problem, which may be due to the fact that this target station is relatively in front in the simulation environment and needs to identify and process a large number of orders. The access number of conveyor8 and 9 is 61 and 70 respectively, indicating that there are some orders that have not been sorted to the correct station in time and have entered the cycle conveyor. In addition, the access number of conveyor10 is 918 times, in addition to the 500 orders that enter the sorting work from the conveyor2 path, which may cause half of the access number, most of which are caused by action selection and cycle retention problems. In addition, the access frequency of conveyor1 (conveyor1) is 9.8%, and the access frequency of conveyor2 (conveyor2) is 7.9%. Due to the environment settings, the two location nodes are connected to the order receiving system (start) at the same time, and the sum of the two (17.7%) should be equal to it, but the result exceeds 80% or more, especially the access number of conveyor1 is more than the access number of start, which indicates that during the early exploration process, the agent repeatedly explores the same location node, and fails to effectively learn the key path strategy, resulting in obvious cycle retention problems for orders.

[0096] In summary, according to the abnormally high number of location node accesses, the inconsistency between the number of orders and the number of target station accesses, and the existence of a large number of access times in the circulating conveyor path, during the simulation training process of the warehouse sorting job based on the Double DQN model, the agent may be trapped in a cycle due to the deviation of action and state selection during the sorting process, which may lead to low order sorting efficiency and an increased risk of sorting errors. Therefore, subsequent optimization can be optimized and improved from the direction of line structure, etc. to reduce such invalid access and redundant repeated paths.

[0097] Further bottleneck analysis, due to the relatively simple simulation environment, there is no particularly complex process flow, according to the key node analysis, the packing station, the labeling station and the order receiving system have a corresponding relationship, the increase of the order quantity will affect the efficiency of these two types of stations, and the corresponding measures are mainly through increasing the packing and labeling line, and at the same time it will bring the increase of cost, therefore, for the improvement of this aspect, the order quantity and the benefit need to be considered comprehensively. From the perspective of line design, the target stations of orders are station2 and 3, no matter which conveyor belt 1 or 2 is selected for sorting, since these two target stations are located at the rear position of the circulating conveyor device, the number of location nodes passed by these two types of orders during sorting is relatively large. According to the previous result analysis, the longer the sorting path, the greater the time penalty in path planning. In addition, the sorting robots of the front target stations 0 / 1 need to identify too many orders, the load pressure is large, the possibility of sorting errors is increased, and the possibility of order trapped in cycle is increased. If the quantity of these two types of orders is relatively large, on the one hand, it will lead to a long sorting operation time, which will greatly affect the sorting efficiency, on the other hand, it will also affect the sorting accuracy, and increase the difficulty of sorting task. Therefore, the optimization design of line structure can be considered, and the simulation environment setting in the corresponding deep reinforcement learning model is optimized.

[0098] The embodiment specifically identifies the key problem station, quantifies the path redundancy, and constructs a shortcut path topology reconstruction to form a set of generalizable line structure optimization methods to determine the optimization scheme for optimizing the line structure of the simulation model. The state parameters of each sorting station obtained in step S02 include the number of path nodes of each order type in each sorting station, the order processing capacity of each sorting station, and the sorting accuracy of each sorting station. The detailed steps include: Step S301. Identify the key problem station, including the redundant station with path redundancy and repeated access, the overloaded station with load exceeding the specified proportion of the predicted order quantity, and the optimization station with sorting accuracy lower than the preset threshold: Step S311. Path redundancy detection: count the average path node number of each order type to identify the stations with path length higher than the global average by a preset proportion (e.g. 2 / 3), and calculate the proportion of repeated access nodes in the path to obtain redundant stations with the risk of cycle retention. For example, stations with path length higher than 2 / 3 of the global average and repeated access node proportion exceeding a preset proportion can be regarded as redundant stations.

[0099] Step S312. Load imbalance evaluation: obtain the order processing amount per unit time of each sorting station, and dynamically evaluate the matching degree of the load (order processing amount) of each sorting station and the predicted order amount according to the order amount prediction model, to identify overloaded stations with load exceeding the predicted order amount by a preset proportion, to early warn potential bottlenecks.

[0100] Step S313. Sorting accuracy threshold judgment: filter out target stations below the threshold according to the preset lower limit threshold of sorting accuracy (e.g. 95%) and mark them as optimization stations.

[0101] Step S302. Quantitative path redundancy: construct a fast diversion path according to the identified key problem stations, with the first end connecting the assembly station, the starting point of the new conveyor belt selecting the assembly station with the highest order generation frequency, the end multi-branching and connecting the terminal to the loading station of the current path and the loading station of the next path, so that the agent can be transferred by the new conveyor belt, and directly sorted to the corresponding loading station at the end.

[0102] Specifically, the planning principles for constructing the fast path are: the first end connects the assembly station, the starting point of the new conveyor belt selects the assembly station with high order generation frequency to reduce invalid movement in the initial stage of the order; the end multi-branching: the terminal is connected to the loading station of the current path and the loading station of the next path to form a "one-to-many" fast channel, and the path length minimization principle is followed to ensure that the number of nodes of the new path is less than a specified proportion (e.g. 30%) of the original path.

[0103] Step S303. Constructing fast path topology reconstruction: divert orders concentrated in the front part of the station to the rear part of the station through the new conveyor belt to achieve order diversion, and monitor the load of the diverted stations in real time. If the target station load rate still exceeds the threshold, further trigger secondary diversion.

[0104] In this embodiment, when determining the optimization scheme, if it is identified that at least one of the later target stations has path redundancy or the sorting accuracy is lower than a preset threshold, it is determined that a conveying belt needs to be added in the simulation model to form a shortcut sorting path. Orders originally concentrated in the front target station are distributed to the shortcut path of the later target station through the added conveying belt to reduce the order processing pressure of the high-load target station. The head end of the added conveying belt is connected to the assembly station on the current path, and the tail end is connected to the loading station on the current conveying path and the loading station on the next conveying path through the conveying belt respectively, so that the agent can be conveyed through the added conveying belt and directly sorted to the corresponding loading station at the tail end. For example, if the order quantity of the target station 0 / 1 accounts for more than 60%, 20% of the orders are forced to be distributed to the target station 2 / 3 through the added path.

[0105] Further, the load of the distributed target station is monitored in real time. If the load rate of a target station still exceeds the threshold, a secondary distribution is further triggered, such as adding a second shortcut path.

[0106] For example, to solve the problem of order circulation and stagnation in the sorting process, and the problems of path redundancy and low sorting accuracy of the target stations 2 / 3, this embodiment adds a conveying belt 11 in the simulation modeling of the warehouse sorting operation system to construct a shortcut path of “labeling station 2→conveying belt 11→loading station 2 / (conveying belt 7→loading station 3)”. The sorting logic is reconstructed from the topological level to reduce the path length of such orders, reduce the sorting load of the front target stations 0 / 1, and reduce the possibility of circulation and stagnation.

[0107] The path design details are shown in Figure 9 The head end of the conveying belt 11 is connected to the assembly station 2, and the tail end is bifurcated to the conveying belt 6 (loading station 2) and the conveying belt 7 (loading station 3). When the target station is 2, the agent can choose to be transported through the conveying belt 11 and can be directly sorted to the loading station 2 at the tail end. Similarly, when the target station is 3, the path selection can be changed from the original regular path to the conveying belt 11→conveying belt 7, and then sorted to the corresponding target station.

[0108] Step S04. The topology structure of the simulation model is optimized and reconstructed according to the optimization scheme determined in step S03 to generate an optimized simulation model.

[0109] In this embodiment, first, the demand analysis is performed for the sorting scene, the sorting process and the equipment are counted, a high-fidelity simulation environment is constructed for simulation modeling, the simulation result is obtained by combining the training process of the deep reinforcement learning algorithm model, the bottleneck of the system is analyzed based on the simulation result, and then the production line structure is optimized to form a closed loop of “algorithm-environment” collaborative optimization, so that the sorting accuracy and the sorting efficiency are improved, as shown in Figure 10 .

[0110] The sliding average reward value change curve of the Double DQN model before and after the line adjustment obtained in the specific application embodiment is shown in FIG. 6. It can be found by comparison that after the line adjustment, the early 200 training rounds of the agent exploration strategy have the same problem, the sliding average reward value fluctuates up and down, and the amplitude is about 20, but the initial value of the sliding average reward fluctuates around 0, which indicates that the early exploration strategy problem is relatively stable, and the situation of circular retention and the like almost occurs in a very small number. Figure 11 Figure 12 The sliding average value of the position node of the sorting path of the Double DQN model before and after the line adjustment in 1000 training rounds is shown in FIG. 7. From the change curve in the figure, after the line adjustment, the sliding average value of the position node of the sorting path explored by the agent in the early stage (the first 200 training rounds) fluctuates around 11, which has a similar problem as before the line adjustment, indicating that by optimizing the sorting path of the order of the target station 2-3, the early agent exploration sorting path strategy has other better choices.

[0111] Figure 13 ​The effective sorting order numbers and their corresponding sorting accuracy rates of each target station in 1000 training rounds before and after the line adjustment are shown. During the entire training process, in addition to the orders intervened due to cycle retention, the number of effective sorting orders after line adjustment is 257, 214, 253, and 241, respectively, and the average sorting accuracy rate of 1000 training rounds is 94.55%, 94.39%, 97.63%, and 97.93%, respectively. The number of sorting errors is 14, 12, 6, and 5, respectively. The sorting error orders of target stations 0 and 2 are greatly reduced, with a decrease of 65% and 50%, respectively. The average sorting accuracy rate of target stations station0-3 has increased by 11.07%, 1.71%, 3.29%, and 1.58%, respectively, compared with before the line optimization. Especially compared with before the line adjustment, the sorting accuracy rate of target station 0 on the sorting path has been greatly improved, followed by target station 2, which shows that the early sorting path exploration strategy of the agent has been improved after the line optimization, the sorting path logic of target stations 2-3 is reconstructed, which reduces the length of the corresponding optimal sorting path and reduces the sorting load pressure of target stations 0-1, so that the number of orders to be identified and sorted by each target station is reduced, and the probability of sorting error in the early sorting decision process of the agent is reduced. It is shown that the design of the conveyor belt path that can directly reach the sorting station in the sorting path can effectively improve the influence of other target loading stations on the relatively rear stations, therefore, in the subsequent optimization, the line structure improvement of the simulation modeling of the in-warehouse sorting operation system can simplify the circulating conveyor belt path and change it to a single direct path, which can solve the problem of the influence of sorting errors of the front sorting stations on the subsequent stations.

[0112] Figure 14 The statistical results of the orders of each type that failed due to cycle retention during the training process of the model before and after the line adjustment are shown. After the line adjustment, the number of orders of types ABCD that are considered to be sorting failures due to exceeding the maximum limit is 4, 9, 8, and 14, respectively. After the line adjustment, the occurrence of such problems is reduced from 79 to 35, a decrease of 55.70%, and the decrease of AD type orders is relatively larger, 77.78% and 56.25, respectively, which shows that the cycle retention problem in the early exploration of the sorting strategy of the agent is greatly reduced compared with before the line adjustment. Since the sorting path length of CD type orders is shortened after the line adjustment, the load pressure of the original regular sorting path is reduced, so the cycle retention problem is effectively improved.

[0113] Figure 15The number of visits and frequency of each position node of the Double DQN model in the sorting process of 1000 training rounds before and after the line adjustment are shown, and the total number of visits is 8420, which is about 20.36% lower than before the line adjustment. From the access frequency of each position node, the sum of the number of visits of conveyor 1 and 2 is greatly reduced by 21.50%, indicating that the problems such as repeated visits and retention of the early agent exploring the sorting path are alleviated. After the logical reconstruction of the sorting path, the number of visits of conveyor 3 is 745, which is no longer higher than the total number of orders, and the reduction is 36.5%, but it is still higher than the sum of the total number of orders entering conveyor 1 (475) and the number of visits of conveyor 10 (175), that is, 650 times. Therefore, in the sorting process, either there is still a small amount of repeated retention of orders entering the conveyor 1 path, or part of the orders with target stations of 2 or 3 may not choose the more optimal path after the line reconstruction in the early exploration process. The number of visits of conveyor 4-7 also decreases, which reduces the load of the early target station identifying sorting orders before the line adjustment, and each position node is relatively more balanced. In addition, the number of visits of conveyor 9 and conveyor 10 is reduced by 21.43% and 80.94% respectively, which also shows that in the training process of the Double DQN model, the problems of repeated retention and entering the circulation of the agent exploring the sorting path are greatly reduced. The number of visits of conveyor 11 is 421, which is slightly lower than the total number of orders that should enter conveyor 2, which is 485, indicating that the line structure adjustment is effectively utilized, and plays an optimizing role in improving the sorting accuracy and sorting efficiency.

[0114] In summary, the application combines the deep reinforcement learning algorithm with the structural optimization of the simulation environment to form a "algorithm-environment" positive feedback collaborative optimization framework, which can systematically improve the sorting performance, improve the sorting accuracy, reduce the position nodes passed by the optimal sorting path, improve the sorting efficiency, and reduce the number of orders failed due to circulation retention; at the same time, through data analysis in the simulation and algorithm training process, the system bottlenecks (such as load imbalance and path redundancy) can be accurately identified, and the line structure can be adjusted from the topological level, the optimization effect is continuous and stable, and it can be easily migrated to other complex logistics sorting and production scheduling scenarios.

[0115] The embodiment further provides a computer device, including a processor and a memory, the memory is used for storing a computer program, and the processor is used for executing the computer program to perform the method as described above.

[0116] It can be understood that the above method of the embodiment can be executed by a single device, such as a computer or a server, or can be applied to a distributed scenario and completed by multiple devices in cooperation. In the distributed scenario, one of the multiple devices can only execute one or more steps in the above method of the embodiment, and the multiple devices interact to complete the above method. The processor can be implemented in a general-purpose CPU, a microprocessor, an application-specific integrated circuit, or one or more integrated circuits, and is configured to execute a related program to implement the above method of the embodiment. The memory can be implemented in a read-only memory (ROM), a random access memory (RAM), a static storage device, and a dynamic storage device. The memory can store an operating system and other application programs. When the above method of the embodiment is implemented by software or firmware, the related program code is stored in the memory and executed by the processor.

[0117] The embodiment further provides a computer readable storage medium storing a computer program, and the computer program is executed by a processor to implement the above method.

[0118] Those skilled in the art should understand that the above-mentioned embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can be in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer readable storage media (including but not limited to a disk storage, a CD-ROM, an optical storage, etc.) containing computer usable program code. The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device implemented in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 one or more flows and / or blocks Figure 1 one or more flows and / or blocks Figure 1The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.

[0119] The above merely provides the preferred embodiment of the present application, but not for any form of limitation on the present application. Although the present application has been disclosed with the preferred embodiment as above, it is not intended to limit the present application. Therefore, any simple modification, equivalent change and modification made to the above embodiment without departing from the technical solution of the present application, and according to the technical essence of the present application, should fall within the scope of protection of the technical solution of the present application.

Claims

1. A simulation optimization method for an in-warehouse sorting operation system, characterized by the following steps: Comprise: Step S01. Construct a simulation model of the in-warehouse sorting operation system, the model comprising a conveying network composed of multiple conveying belts, multiple sorting stations, and an order processing module for simulating the order processing flow; Step S02. Perform sorting task simulation training in the constructed simulation model based on a double-target network deep reinforcement learning model, and obtain state parameters of each sorting station in the simulation training process, the double-target network comprising a local network and a target network, the DQN approximating the Q value function through a neural network, the local network being used for selecting the current optimal action, and the target network being used for calculating the target Q value and periodically synchronizing parameters from the local network; Step S03. Identify the required optimization nodes in the line structure of the simulation model according to the state parameters of each sorting station obtained in step S02, and construct a shunt path according to the identified required optimization nodes to obtain an optimization scheme of the line structure; Step S04. Optimize and reconstruct the topology structure of the simulation model according to the optimization scheme obtained in step S03 to generate an optimized simulation model.

2. The in-warehouse sortation operation system simulation optimization method of claim 1, wherein, In step S01, the conveying network is a double-path circulation network, the sorting stations include a packing station, a labeling station, and multiple loading stations, the packing station is used to trigger a packing action, the labeling station is used to trigger a labeling action, the loading station is the terminal point of the orders that arrive through the conveying network and complete sorting, and the order processing module is configured to simulate the whole process of orders from receiving, packing, labeling, sorting to loading, including: simulating in-warehouse sorting operation in simulation modeling, obtaining order information through an order receiving terminal, taking corresponding products from the warehouse to the conveying belt, performing product boxing and packing after arriving at the packing station, pasting an express label containing order delivery information to the surface of the package after packing is completed, then entering the circulating conveying belt, performing express sorting before arriving at the loading station, and arriving at the loading station after sorting is completed.

3. The in-warehouse sortation operation system simulation optimization method of claim 1, wherein, In step S02, the target network calculates Q Value: wherein, denotes the target network, denotes the local network, denotes a discount factor, denotes an immediate reward, denotes a local network parameter, denotes a target network parameter, and denote the updated next state and action, respectively; In the experience replay process of the deep reinforcement learning model, a priority sampling strategy based on the timing difference error is used for sampling, wherein the calculation expression of the sampling probability of the sample is: wherein, is the TD error for the i th sample, is the control sampling preference degree, is a preset constant; According to the sampling probability Adjust the importance sampling weight corresponding to the sample: wherein, is the importance sampling weight for the i th sample, β is the control bias correction strength, denotes the total number of samples.

4. The in-library sortation system simulation optimization method of claim 1, wherein, In step S02, an adaptive learning rate mechanism based on training error is used in the deep reinforcement learning model to adjust the learning rate: wherein, is an initial learning rate, is a current step number at time t, is a decay step, is a decay coefficient, is a learning rate at the current time t.

5. The in-library sortation system simulation optimization method of claim 1, wherein, In step S02, an environmental constraint is also included, which includes any one or more of the following: an order type randomly generated at initialization time, a moving time between adjacent position nodes, only one order can be operated at each station at a time, the number of orders operated at each sorting station at a time, the maximum number of position nodes passed through by a single sorting path, and an order operation sequence compliance rule, which is a sequence of stations followed by the packing station , the conveyor belt to the label station ; and a state space of the model is defined as including a position code, a target station code, and an order type code, the position code is used to correspond to information representing a current position node, the position node includes an order receiving system, a conveyor belt network, a packing station, a label station, and a loading station, the target station code is used to represent information of a target position node of sorting, an action space of the model is defined as including an action of moving along a conveyor belt path and an action of stopping at a station and performing a sorting operation, the sorting operation includes packing, labeling, and sorting, a reward function of the model is defined with the goals of maximizing sorting accuracy, optimizing efficiency, and guiding key operations, wherein the guiding key operations is to give positive incentives to specified key operations to guide reward key operations.

6. The in-warehouse sortation operation system simulation optimization method of claim 5, wherein, The calculation expression of the reward function is: wherein, is a displacement penalty term to correspond to an efficiency target, is a critical operation guiding term to correspond to a procedure execution order target, is a termination reward term to correspond to an accuracy target, is an indicator function, represents an order type, g represents a target station, is a preset displacement penalty amount, if and the order type matches the target station g , it means the sorting is correct, i.e. , then to reward, if not, i.e. , then to punish, if , then , T represents the last displacement of the sorting path of the order, is the maximum number of position nodes passed by a single sorting path, is a displacement penalty coefficient, is a packing station, is a labeling station, if the current state of the agent is at the packing station or the labeling station , it means the current operation is a critical operation, then to reward, is a preset critical operation guiding reward amount.

7. The in-warehouse sortation operation system simulation optimization method according to any one of claims 1 to 6, characterized in that, Step S03 includes: Step S301. Identify key problem stations, the key problem stations including redundant stations with path redundancy and repeated access, overloaded stations with a load exceeding a specified proportion of predicted order quantity, and optimization required stations with a sorting accuracy lower than a preset threshold; Step S302. Quantify path redundancy: construct a shortcut shunt path according to the identified key problem stations, wherein the first end is connected to the assembly station, the starting point of the new conveying belt is selected as the assembly station with the highest order generation frequency, and the end is branched in multiple directions and connected to the loading stations of the current path and the next path; Step S303. Path topology reconstruction: orders concentrated on the front part of the station are diverted to the rear part of the station through the newly added conveyor to realize order diversion, and the load of the diverted station is monitored in real time. If the target station load rate still exceeds the threshold, further trigger secondary diversion.

8. The in-warehouse sortation operation system simulation optimization method of claim 7, wherein, In step S301, the average path node number of each order type is counted to identify the stations with path length higher than the global average by a preset proportion, and the proportion of repeated access nodes in the path is calculated to obtain redundant stations with the risk of circular retention. The unit time order processing capacity of each sorting station is obtained, and the matching degree of the load capacity and the predicted order quantity of each sorting station is dynamically evaluated according to the order quantity prediction model to identify overloaded stations with load capacity exceeding the predicted order quantity by a preset proportion.

9. A computer device comprising a processor and a memory for storing a computer program, characterized in that, The processor is configured to execute the computer program to perform the method of any one of claims 1-8.

10. A computer readable storage medium storing a computer program, characterized in that, The computer program, when executed, implements the method of any one of claims 1-8.

Images