A dynamic picking path optimization method and system based on deep reinforcement learning

Abstract

The application relates to the technical field of warehouse logistics management, in particular to a dynamic picking path optimization method and system based on deep reinforcement learning, which comprises the following steps: obtaining a picker state vector based on warehouse structured coding data analysis; calculating the moving value density of orders in each channel according to order queue data to be processed and the picker state vector; defining a state space according to the picker state vector and the order queue data to be processed; modeling a warehouse order picking problem as a Markov decision process based on the state space and a discrete action space; solving the Markov decision process by using a deep reinforcement learning neural network model based on the real-time state of the picker and the moving value density, so that an optimal picking path decision is obtained; and controlling the picker to perform a picking operation according to the optimal picking path decision. The application combines the deep reinforcement learning neural network model to solve the Markov decision process, realizes accurate optimization of the picking path, and effectively improves the efficiency of warehouse order picking.

Description

Technical Field

[0001] This invention relates to the field of warehousing and logistics management technology, and in particular to a dynamic picking path optimization method and system based on deep reinforcement learning. Background Technology

[0002] In modern warehouse operations, order picking is a core part of warehousing and logistics operations, and its efficiency directly affects the overall supply chain performance. However, in time-sensitive scenarios such as power material warehouses, the efficient processing of dynamic orders has become a technical challenge, and traditional order picking technology has exposed significant limitations in dealing with real-time changing order flows.

[0003] Traditional order picking methods are mostly based on the assumption of a static order set, employing the Traveling Salesman Problem (TSP) based on a fixed order set. Route optimization using problem-solving (TSP) or heuristic algorithms often fails to consider the dynamic arrival characteristics of orders during modeling, making it difficult to effectively cope with dynamically changing order environments. When faced with continuously arriving new orders, repeated full calculations are necessary, leading to decision delays and path suboptimality issues. This is particularly problematic in scenarios with high order arrival rates, where system responsiveness drops sharply. While existing research on dynamic picking has attempted to introduce order batching strategies with the development of warehouse automation, most studies employ single-stage optimization strategies that separate order batching from path planning. This separation results in a lack of organic coordination between the two key decision-making stages, failing to achieve optimal matching between order combinations and path selection, and struggling to meet the real-time adjustment needs in dynamic scenarios. Some studies have attempted combinatorial optimization through iterative search, but this suffers from exponentially increasing computational complexity, making it difficult to meet the time constraints of real-time decision-making. Furthermore, a multi-objective balancing mechanism in dynamic environments has not yet been effectively established. Existing methods generally use fixed-weight strategies to balance target parameters such as travel distance and order timeliness, lacking adaptability to dynamic environmental changes. Especially in sudden scenarios such as emergency order insertion or equipment malfunctions, traditional methods struggle to dynamically adjust the weights of target parameters, resulting in insufficient system robustness.

[0004] Meanwhile, existing algorithms are mostly based on simplified models of single-block warehouses. When dealing with complex real-world scenarios such as multi-block layouts and multi-device collaboration, they generally suffer from model mismatch. In particular, when it is necessary to coordinate the path planning of multiple picking devices, traditional methods are difficult to effectively solve path conflicts and task allocation problems between devices, which can easily lead to operational bottlenecks. Therefore, developing a picking path optimization method with dynamic adaptability has become a technical bottleneck that urgently needs to be overcome in the field of warehouse automation. Summary of the Invention

[0005] To address the above technical problems, this invention provides a dynamic picking path optimization method and system based on deep reinforcement learning.

[0006] In a first aspect, the present invention provides a dynamic picking path optimization method based on deep reinforcement learning, the method comprising the following steps:

[0007] Collect warehouse aisle layout parameters and picker physical parameters to form warehouse basic structure data, and perform structured encoding on the warehouse basic structure data to obtain warehouse structured encoded data;

[0008] Based on the warehouse structured coded data, the current position of the picker and the remaining loading capacity of the picker are parsed to obtain the picker state vector;

[0009] Real-time acquisition of pending order queue data in the warehouse, and calculation of the movement value density of orders in each channel based on the pending order queue data and the picker state vector;

[0010] Define the state space based on the picker state vector and the order queue data to be processed, and construct the discrete execution action space of the picker at each time step;

[0011] Based on the state space and the discrete execution action space, a cumulative reward is set with the optimization objective of minimizing the average order waiting time, and the warehouse order picking problem is modeled as a Markov decision process.

[0012] Based on the real-time status of the picker and the moving value density, the Markov decision process is solved using a pre-built deep reinforcement learning neural network model to obtain the optimal picking path decision.

[0013] The optimal picking path decision controls the picker to perform picking operations.

[0014] In a further implementation, the step of performing structured encoding on the warehouse infrastructure data to obtain structured warehouse encoded data includes:

[0015] Based on the pre-collected physical structure data of the warehouse, a discretized coordinate system for the warehouse is established with the center of the entrance intersection as the origin of the coordinate system;

[0016] Based on the warehouse aisle layout parameters, all aisles in the warehouse and the storage cells on both sides of each aisle are encoded to obtain the aisle layout parameter encoding data;

[0017] The channel layout parameter encoding data is mapped to the warehouse discretized coordinate system to obtain the warehouse channel storage encoding data;

[0018] Based on the physical parameters of the picker, the initial position of the picker is fixed as the coordinates of the entrance cross passage, and the initial unloaded capacity is set to the maximum loading capacity.

[0019] Based on the initial position of the picker and the maximum loading capacity, the initial state vector of the picker is obtained;

[0020] The warehouse channel storage encoding data and the picker initial state vector are encapsulated into a structured data packet to form warehouse structured encoding data.

[0021] In a further implementation, the step of parsing the current position of the picker and the remaining loading capacity of the picker based on the warehouse structured coded data to obtain the picker state vector includes:

[0022] The real-time location data of the picker is obtained, and the real-time location data of the picker is matched with the structured coding data of the warehouse to parse out the channel number and storage cell number of the picker, and thus obtain the current location of the picker.

[0023] Obtain the current load weight data of the picker, and determine the remaining load capacity of the picker based on the maximum load capacity of the picker in the warehouse structured coding data and the current load weight data;

[0024] The current position of the picker and the remaining loading capacity of the picker are combined into a picker state vector.

[0025] In a further implementation, the pending order queue data includes the coordinates of the item storage locations in the warehouse for the pending orders;

[0026] The moving value density is the sum of the ratios of the number of orders at each picking position in each channel to the minimum relative distance between the current position of the picker and the coordinates of the item storage position when the picker moves up or down.

[0027] In a further implementation, the step of modeling the warehouse order picking problem as a Markov decision process based on the state space and the discrete action space, setting a cumulative reward with the optimization objective of minimizing the average order waiting time, includes:

[0028] Based on the state space and the discrete execution action space, determine the state transition rules for the picker to transition from the current state to the next state;

[0029] Based on the state transition rules, the state transition probability of the picker transitioning from the current state to the next state after performing an action is calculated, and a state transition probability model is constructed.

[0030] With the goal of minimizing the average order waiting time, a composite cumulative reward function is constructed based on the results of the picker's actions, which includes picking efficiency rewards, travel distance penalties, and unloading incentives. The results of the picker's actions include at least the time required for the picker to complete the picking order, the travel distance, and the order completion rate.

[0031] Based on the state space, the discrete execution action space, and the state transition probability model, the warehouse order picking problem is modeled as a Markov decision process by maximizing the optimization objective of the composite cumulative reward function.

[0032] In a further implementation, the state transition rule is specifically as follows:

[0033] Set the minimum time unit required for the picker to move from one storage location to another, and detect the picker's current position and perform the action;

[0034] When the picker is detected to be in a non-warehouse base location and performs a stationary action, the picker is controlled to remain stationary in the current location for a period of time until the minimum time unit is reached. If a new order is detected during the time the picker remains stationary, the state transition to the next state is triggered.

[0035] When the picker is detected to be in the warehouse base location and performing an unloading action, control the picker to unload all the items it is carrying and reset the picker's remaining loading capacity to the maximum loading capacity.

[0036] When a picker is detected moving at the warehouse base location, if the picker is in the rear cross passage or the front cross passage, left and right movement is allowed.

[0037] During the movement of the picker, the order status changes and the picker status changes are monitored in real time. If any one of the following conditions is met: the picker reaches the intersection and cannot continue to move in the original direction, a new order is detected during the movement of the picker, or at least one item is picked during the movement of the picker, then the state transition to the next state is triggered.

[0038] In a further implementation, the composite cumulative reward function is specifically as follows:

[0039]

[0040] In the formula, The compound cumulative reward for time step t; This is a bonus coefficient for picking efficiency; This refers to the maximum cargo capacity that the picker can carry in a single pick. The remaining loading capacity of the picker at time step t; This is the unloading incentive adjustment coefficient; The number of storage locations the picker moves within time step t; The number of items picked up by the picker within time step t; The action performed by the picker at time step t. This indicates that the picker is performing an unloading action.

[0041] In a further embodiment, the deep reinforcement learning neural network model includes an input layer, a two-branch feature extraction module, a feature fusion module, a multi-layer feature compression module, and an output layer connected in sequence.

[0042] The dual-branch feature extraction module includes a parallel order status feature extraction unit and a picker status feature extraction unit; both the order status feature extraction unit and the picker status feature extraction unit are composed of a fully connected layer and a linear rectified activation function.

[0043] The multi-layer feature compression module includes at least three fully connected layers connected in series.

[0044] In a further implementation, the step of solving the Markov decision process using a pre-built deep reinforcement learning neural network model based on the real-time state of the picker and the moving value density to obtain the optimal picking path decision includes:

[0045] The mobility value density of each channel is combined with the real-time order status to form an order status vector; the dimension of the order status vector is twice the total number of channels.

[0046] The real-time state of the picker is nonlinearly transformed by the picker state feature extraction unit to extract the high-dimensional feature vector of the picker.

[0047] The order status vector is subjected to a nonlinear transformation by the order status feature extraction unit to extract the order distribution feature vector;

[0048] The high-dimensional feature vector of the picker and the feature vector of the order distribution are concatenated by channels to generate a state fusion feature vector;

[0049] The state fusion feature vector is subjected to hierarchical feature transformation through a multi-layer feature compression module to obtain a decision-oriented high-order feature representation;

[0050] The decision-oriented high-order feature representation is mapped to the action space dimension, and the expected cumulative reward value of each action taken in the current state is calculated through a fully connected layer.

[0051] The action with the highest expected cumulative reward value is selected as the optimal action for the picker in the current state, and the optimal picking route decision is generated based on the optimal action.

[0052] Secondly, the present invention provides a dynamic picking path optimization system based on deep reinforcement learning, the system comprising:

[0053] The data acquisition module is used to collect warehouse aisle layout parameters and picker physical parameters to form warehouse basic structure data, and to perform structured encoding on the warehouse basic structure data to obtain warehouse structured encoded data.

[0054] The state parsing module is used to parse the current position and remaining loading capacity of the picker based on the warehouse structured coded data to obtain the picker state vector;

[0055] The movement analysis module is used to collect the pending order queue data of the warehouse in real time, and calculate the movement value density of orders in each channel based on the pending order queue data and the picker state vector.

[0056] The space construction module is used to define the state space based on the picker state vector and the order queue data to be processed, and to construct the discrete execution action space of the picker at each time step;

[0057] The picking modeling module is used to model the warehouse order picking problem as a Markov decision process based on the state space and the discrete execution action space, setting a cumulative reward with the optimization objective of minimizing the average order waiting time;

[0058] The path decision module is used to solve the Markov decision process based on the real-time status of the picker and the moving value density, and obtain the optimal picking path decision by using a pre-built deep reinforcement learning neural network model.

[0059] The picking execution module is used to control the picker to perform picking operations based on the optimal picking path decision.

[0060] This invention provides a dynamic picking path optimization method and system based on deep reinforcement learning. The method collects warehouse aisle layout parameters and picker physical parameters to form warehouse infrastructure data, and performs structured encoding on this data to obtain structured coded warehouse data. Based on this structured coded data, the current position and remaining capacity of the picker are analyzed to obtain the picker's state vector. Real-time data of the warehouse's pending order queue is collected, and the movement value density of orders within each aisle is calculated based on the pending order queue data and the picker's state vector. A state space is defined based on the picker's state vector and the pending order queue data, and a discrete execution action space for the picker at each time step is constructed. Based on the state space and the discrete execution action space, a cumulative reward is set with the optimization objective of minimizing the average order waiting time, and the warehouse order picking problem is modeled as a Markov decision process. Based on the picker's real-time state and the movement value density, the Markov decision process is solved using a pre-built deep reinforcement learning neural network model to obtain the optimal picking path decision. The optimal picking path decision is then used to control the picker to perform picking operations. Compared with existing technologies, this method collects and structures various warehouse data, and dynamically adjusts the picking path through deep reinforcement learning, thereby achieving real-time optimization of the dynamic picking path. This effectively improves the efficiency of warehouse order picking, reduces the average order waiting time, and enhances the response speed and intelligence level of the warehouse logistics system. Attached Figure Description

[0061] Figure 1 This is a schematic diagram of the dynamic picking path optimization method based on deep reinforcement learning provided in an embodiment of the present invention;

[0062] Figure 2 This is an example diagram of a rectangular warehouse layout provided in an embodiment of the present invention;

[0063] Figure 3 This is a schematic diagram of the deep reinforcement learning neural network model architecture provided in an embodiment of the present invention;

[0064] Figure 4 This is a block diagram of a dynamic picking path optimization system based on deep reinforcement learning provided in an embodiment of the present invention.

[0065] Figure labeling: 101, Data acquisition module; 102, Status analysis module; 103, Movement analysis module; 104, Spatial construction module; 105, Picking modeling module; 106, Path decision module; 107, Picking execution module. Detailed Implementation

[0066] The embodiments of the present invention are described in detail below with reference to the accompanying drawings. The embodiments are given for illustrative purposes only and should not be construed as limiting the present invention. The accompanying drawings are for reference and illustration only and do not constitute a limitation on the scope of patent protection of the present invention, because many changes can be made to the present invention without departing from the spirit and scope of the present invention.

[0067] Figure 1 This is a flowchart illustrating a dynamic picking path optimization method based on deep reinforcement learning provided in an embodiment of the present invention. The embodiment of the present invention provides a dynamic picking path optimization method based on deep reinforcement learning, such as... Figure 1 As shown, the method includes the following steps:

[0068] S1. Collect warehouse aisle layout parameters and picker physical parameters to form warehouse basic structure data, and perform structured encoding on the warehouse basic structure data to obtain warehouse structured encoded data.

[0069] In some implementations, the step of performing structured encoding on the warehouse infrastructure data to obtain structured warehouse data includes:

[0070] Based on the pre-collected physical structure data of the warehouse, a discretized coordinate system for the warehouse is established with the center of the entrance intersection as the origin of the coordinate system;

[0071] Based on the warehouse aisle layout parameters, all aisles in the warehouse and the storage cells on both sides of each aisle are encoded to obtain the aisle layout parameter encoding data;

[0072] The channel layout parameter encoding data is mapped to the warehouse discretized coordinate system to obtain the warehouse channel storage encoding data;

[0073] Based on the physical parameters of the picker, the initial position of the picker is fixed as the coordinates of the entrance cross passage, and the initial unloaded capacity is set to the maximum loading capacity.

[0074] Based on the initial position of the picker and the maximum loading capacity, the initial state vector of the picker is obtained;

[0075] The warehouse channel storage encoding data and the picker initial state vector are encapsulated into a structured data packet to form warehouse structured encoding data.

[0076] This embodiment primarily addresses the path planning problem of a single picker performing order picking tasks in a rectangular warehouse. The warehouse consists of a single block with multiple parallel aisles interconnected by two intersecting aisles. Inventory storage locations are distributed on both sides of each aisle. The picker needs to collect requested items from designated storage locations. While moving within the aisles, the picker can access storage locations on both sides of the aisle. To simplify the problem, this embodiment assumes that the horizontal movement distance within the aisles is negligible. When the picker needs to move between different aisles, it must utilize one of the two intersecting aisles. These intersecting aisles do not have designated storage locations; they serve only as paths for movement between aisles. It should be noted that the picker can only enter or leave an aisle through one of the intersecting aisles. Furthermore, it is assumed that all items to be picked have the same size and priority. The picker starts from the warehouse's starting point, collects assigned items, adheres to capacity limits during transport, and ultimately returns the items to the warehouse. Figure 2 This diagram illustrates a rectangular warehouse layout, providing two alternative paths for the picker to choose from to collect the next dispatch order. The path distance is shown in the diagram. The main objective of this embodiment is to minimize the average waiting time for orders, thereby improving the throughput of warehouse operations.

[0077] This embodiment is applicable to rectangular layout warehousing and logistics centers, especially small and medium-sized warehouses or automated warehousing systems using a single picker operation mode. Specifically, typical application scenarios include e-commerce warehouse order picking, retail distribution center replenishment, and pharmaceutical or precision parts warehousing. For e-commerce warehouse order picking, given the massive volume and high frequency of orders, the picker needs to efficiently plan routes and complete multiple order picking in batches within capacity constraints to minimize order waiting time. For retail distribution center replenishment, the picker needs to pick goods from the storage area to the sorting area to support goods-to-person or person-to-goods picking modes, thereby optimizing the path and efficiency of a single picking task. For pharmaceutical or precision parts warehousing, the accuracy requirements for the storage location of items are extremely high. Structured modeling ensures that the picker strictly follows the channel rules to avoid invalid paths, thereby ensuring the accuracy and efficiency of warehousing operations.

[0078] In the process of warehouse structured coding, this embodiment first performs data acquisition to obtain the warehouse's basic structural data. This basic structural data includes warehouse aisle layout parameters and picker physical parameters. Specifically, the warehouse aisle layout parameters include the number of parallel picking aisles and the number of storage cells on both sides of each aisle; these parameters collectively define the warehouse's physical layout. The picker physical parameters include its initial position, typically set to the base coordinates (0, 0) coinciding with the entrance intersection aisle, and its maximum capacity, i.e., the maximum number of items the picker can carry in a single pick. By acquiring these parameters, this embodiment can construct the warehouse's basic structural data. After completing data acquisition, this embodiment performs structured coding on the warehouse's basic structural data. In this process, this embodiment first establishes a warehouse discretized coordinate system based on the warehouse physical structural data, using the center of the entrance intersection aisle as the origin of the discrete coordinate system, and establishing discretized coordinate axes in the horizontal and vertical directions according to a pre-set discretization precision. The warehouse physical structural data can be... This includes, but is not limited to, basic information such as the size and location of each part of the warehouse (e.g., aisles, storage cells, etc.). Then, based on the warehouse aisle layout parameters, all aisles in the warehouse and the storage cells on both sides of each aisle are encoded to obtain aisle layout parameter encoding data. These aisle layout parameter encoding data are then mapped to the warehouse discretized coordinate system according to the actual position of the aisle and storage cell corresponding to each encoding in the warehouse, thereby generating warehouse aisle storage encoding data. For example, in this embodiment, the coordinate range of the aisle in the coordinate system can be determined according to the direction and length of the aisle; for the storage cells on both sides of the aisle, their specific coordinate positions in the coordinate system are determined according to their relative position to the aisle and their order in the aisle; for a certain horizontal aisle, its coordinate range is from the starting coordinate to the ending coordinate in the horizontal direction, and is a fixed value in the vertical direction; for a certain storage cell in the aisle, its specific coordinate offset in the horizontal direction is determined according to its encoding order, thereby obtaining its accurate coordinates in the coordinate system, thus determining the position information of each aisle and storage cell in the warehouse discretized coordinate system.

[0079] Simultaneously, this embodiment, based on the physical parameters of the picker, fixes the initial position of the picker to the coordinates of the entrance intersection channel. That is, the initial coordinate position of the picker in the warehouse discretized coordinate system is the coordinates corresponding to the center of the entrance intersection channel. The initial unloaded capacity of the picker is set to the maximum loaded capacity. This embodiment combines the initial position coordinates (coordinate values ​​in both horizontal and vertical directions) and the initial loaded capacity value into a vector to obtain the initial state vector of the picker. Finally, this embodiment encapsulates the warehouse channel storage code data and the picker's initial state vector. The warehouse channel storage code data is a data set containing the position information of each channel and storage cell in the warehouse discretized coordinate system; the picker's initial state vector is a separate data element. These two parts are combined into a structured data packet, organized and stored according to a predefined data structure format, for example, using... Specific data formats (such as JSON or XML) encapsulate warehouse channel storage encoding data and picker initial state vectors to form structured warehouse encoding data, ensuring data integrity and readability. In the data logic processing stage, this embodiment maps warehouse channels and storage locations to discrete coordinates in the state space. For example, the i-th storage cell on the left of channel n corresponds to the vertical coordinate (2n-1), and the storage cell on the right corresponds to 2n. The horizontal coordinate is used to distinguish between picking channels and cross channels. The order storage location is represented by a (channel number, vertical index) tuple, which facilitates the calculation of the relative distance between the picker and the target (e.g., when moving upwards, the distance is the difference between the current vertical index and the target index), where n is the channel number. This embodiment, through structured encoding, can represent the physical structure of the warehouse and the state information of the picker in a unified and standardized form, providing strong support for the intelligence and automation of the warehouse management system.

[0080] S2. Based on the warehouse structured coded data, the current position of the picker and the remaining loading capacity of the picker are parsed to obtain the picker state vector.

[0081] In some implementations, the step of parsing the current position and remaining capacity of the picker based on the warehouse structured coded data to obtain the picker state vector includes:

[0082] The real-time location data of the picker is obtained, and the real-time location data of the picker is matched with the structured coding data of the warehouse to parse out the channel number and storage cell number of the picker, and thus obtain the current location of the picker.

[0083] Obtain the current load weight data of the picker, and determine the remaining load capacity of the picker based on the maximum load capacity of the picker in the warehouse structured coding data and the current load weight data;

[0084] The current position of the picker and the remaining loading capacity of the picker are combined into a picker state vector.

[0085] Specifically, this embodiment uses a pre-deployed positioning system (such as a positioning device based on ultra-wideband technology or radio frequency identification technology) to collect the real-time location information of the picker in the warehouse, obtaining real-time location data of the picker. This data accurately reflects the specific coordinates of the picker in the warehouse space. Then, the collected real-time location data of the picker is matched with pre-obtained warehouse structured coding data. The warehouse structured coding data contains the location information of all channels in the warehouse and the storage cells on both sides of each channel in the warehouse's discrete coordinate system, and the channels and storage cells are uniquely encoded. During the matching process, the real-time location coordinates of the picker are... By comparing the coordinate range of each channel and storage cell in the warehouse structured coding data, when the real-time position coordinates of the picker fall within the coordinate range of a certain channel, the channel is determined to be the current channel of the picker, and the corresponding channel number is obtained. This allows for further determination of the relative position of the picker to the storage cells on both sides of the channel. If the picker is close to a storage cell on one side and is within the coordinate range of that storage cell, then that storage cell is determined to be the current storage cell of the picker, and the corresponding storage cell number is obtained. Thus, the current channel number and storage cell number of the picker can be obtained, and the current channel number and storage cell number of the picker are used as the current position of the picker.

[0086] Meanwhile, this embodiment utilizes a weighing sensor installed on the picker to collect the weight information of the goods currently carried by the picker in real time, obtaining the current load weight data of the picker. It also extracts the maximum load capacity information of the picker from the warehouse structured coding data. The difference between the maximum load capacity and the current load weight data is the remaining load capacity of the picker. In this embodiment, the current position of the picker (including the channel number and storage cell number) and the remaining load capacity of the picker are combined in a certain order. For example, in this embodiment, the channel number, storage cell number, and remaining load capacity can be arranged in sequence to form an ordered data combination. This data combination is the picker state vector, which comprehensively reflects the current state information of the picker in the warehouse.

[0087] S3. Collect the pending order queue data of the warehouse in real time, and calculate the movement value density of orders in each channel based on the pending order queue data and the picker state vector.

[0088] Specifically, during warehouse order processing, this embodiment collects pending order data in real time and sorts it according to the order generation time to obtain pending order queue data. This pending order queue data includes the storage location coordinates of the items in the warehouse for each pending order. Simultaneously, this embodiment combines the current status of the picker (including location and remaining capacity) and dynamically calculates the movement value density of orders within each channel based on the sorting of the pending order queue data by order generation time. and Where n represents the nth channel, N represents the total number of picking aisles in the warehouse. In this embodiment, each aisle n is composed of... and This describes the order status, thereby dynamically updating the order status values ​​within each channel. The order status value can be represented as... Specifically, This represents the movement value density of an order as the picker moves upwards at time t within the first aisle. This represents the movement value density of an order as the picker moves downwards at time t within the second channel. The picker represents the picker in ( The moving value density of orders moving upwards at time t within the channel. This indicates that the picker is in ( The value density of orders moving downwards at time t within the channel is calculated by dividing the value of each order within the channel by the distance of that order from the picker based on the direction of movement, and then summing these values ​​across all orders. It should be noted that the distance between the picker and the order refers to the minimum number of moves required for the picker to reach the order's location, expressed in units of storage locations. This calculation method comprehensively considers both the order distribution density and the picker's movement cost, providing a quantitative basis for path optimization. It can more accurately simulate decision-making during the picking process, ensuring the efficiency and real-time performance of the picking strategy. In this embodiment, for each picking location i within channel n, the number of orders at that location is counted. And calculate its minimum relative distance to the current position of the picker. or ,final, and These represent the weighted sum of the values ​​of all orders in the corresponding movement direction of the picker. Movement value density is defined as the sum of the ratios of the number of orders at each picking position within each aisle to the minimum relative distance between the current position of the picker and the coordinates of the item's storage location when the picker moves upwards or downwards. The specific calculation formula is as follows:

[0089]

[0090] In the formula, For the picker in ( The moving value density of an order when it moves upward within the channel at time t; This represents the number of pending orders at the i-th picking position in the n-th channel. L represents the minimum relative distance (in terms of the number of storage locations) for the picker to reach picking location i when moving upwards; L represents the number of picking locations in each aisle. For the picker in ( The moving value density of an order when it moves downward within the channel at time t; This is the minimum relative distance that the picker needs to reach picking position i when moving downwards.

[0091] S4. Define the state space based on the picker state vector and the pending order queue data, and construct the discrete execution action space of the picker at each time step.

[0092] S5. Based on the state space and the discrete execution action space, a cumulative reward is set with the optimization objective of minimizing the average order waiting time, and the warehouse order picking problem is modeled as a Markov decision process.

[0093] In some implementations, based on the state space and discrete action space, and setting a cumulative reward with the optimization objective of minimizing the average order waiting time, the steps of modeling the warehouse order picking problem as a Markov decision process include:

[0094] Based on the state space and discrete execution action space, determine the state transition rules for the picker to move from the current state to the next state;

[0095] Based on the state transition rules, the state transition probability of the picker transitioning from the current state to the next state after performing an action is calculated, and a state transition probability model is constructed.

[0096] With the goal of minimizing the average order waiting time, a composite cumulative reward function is constructed based on the results of the picker's actions, which includes picking efficiency rewards, travel distance penalties, and unloading incentives. The results of the picker's actions include at least the time required for the picker to complete the picking order, the travel distance, and the order completion rate.

[0097] Based on the state space, the discrete execution action space, and the state transition probability model, the warehouse order picking problem is modeled as a Markov decision process by maximizing the optimization objective of the composite cumulative reward function.

[0098] Specifically, this embodiment proposes a modeling method based on a finite Markov decision process (MDP) for the real-time warehouse order picking problem. This method models the real-time warehouse order picking problem as a finite Markov decision process, fully utilizing its dynamic and sequential decision-making characteristics. The structured description of this problem is as follows:

[0099] State space S: the state at time step t This includes the orders that need to be picked, their location in the warehouse, and the current location and status of the picker;

[0100] Discrete execution action space A: the action selected at each time step t This can include moving in a specific direction (such as left, right, up, or down), remaining stationary, and releasing items at the base;

[0101] State transition probability model: Execution of action This will lead to a new state The generation of, from state to state The transition is determined by both the action taken and the current state;

[0102] Instant reward function R: Instant reward function Rewards can be assigned based on the effectiveness of the actions taken, taking into account factors such as the time required to complete a picking order, the distance traveled, and the success rate of order completion.

[0103] The strategy obtained by the algorithm :Strategy A strategy for selecting actions based on the current state is defined, and the optimization objective is to find an optimal strategy that maximizes the cumulative reward.

[0104] in, This represents the system state at time step t; This indicates the action taken at time step t.

[0105] The algorithm optimization objective is to determine the optimal strategy that maximizes the expected cumulative reward within a finite time frame. Once a strategy is selected, the warehousing system will evolve in discrete time steps. The specific process for each step is as follows: observe the current state. Based on strategy Select Action ; Performing this action generates a new state. Received a reward This embodiment uses a Markov decision process to mathematically model the state transition process of a warehousing system. The Markov decision process provides a structured framework for modeling decision-making processes that are influenced by both random factors and the actions of the decision-maker. The Markov decision process consists of a quintuple (S, A, P, R, ...). Modeling, where S represents the system state set; A represents a finite set of actions; P represents the state transition probability; and R represents the immediate reward function; This represents the discount factor used to balance current and future rewards. The following is the specific Markov decision implementation process for this problem:

[0106] In this embodiment, the system state with time step t is represented as... ,in, The status of the picker; Regarding the status of existing orders in the warehouse, this embodiment further decomposes and defines the status as follows:

[0107] Picker status It contains four components , , and Specifically:

[0108] This indicates the horizontal position of the picker, that is, the position of the picker relative to the cross passage. Its value can be -1, 0 or 1, where -1 means that the picker is located in the rear cross passage, 1 means that the picker is located in the front cross passage, and 0 means that the picker is located in a certain picking passage.

[0109] and Both represent the vertical position, that is, the position of the picker relative to the picking aisle. When the picker is located in the nth aisle, and These two values ​​are (2n-1) and (2n), respectively. Each picking aisle is represented by two consecutive indices (instead of a single index), because... Defined as a vector of size 2N, this maintains consistency between the two components of the state and facilitates the interpretation of the neural network.

[0110] Indicates the remaining loading capacity of the picker. If the picker is full, it means that no more items can be picked up; if This indicates that the picker is empty, and K is the maximum loading capacity of the picker.

[0111] when At that time, the precise location of the picker within a picking aisle is implicitly encoded in the order status. In this approach, the state representation method ensures consistency among the various parts of the system state, thus facilitating processing using neural networks. Through this state definition and transition process modeling, this embodiment can effectively simulate decision-making during the picking process, thereby optimizing the picking strategy. The number of picking locations and the number of orders at each location jointly influence the picking strategy, while the distance required to move up or down to a picking location directly relates to the movement cost required for the picker to approach the order. This allows for a more accurate assessment of the expected effects of different actions, thereby optimizing decision-making during the picking process. In some implementations, the state transition rules are specifically as follows:

[0112] Set the minimum time unit required for the picker to move from one storage location to another, and detect the picker's current position and perform the action;

[0113] When the picker is detected to be in a non-warehouse base location and performs a stationary action, the picker is controlled to remain stationary in the current location for a period of time until the minimum time unit is reached. If a new order is detected during the time the picker remains stationary, the state transition to the next state is triggered.

[0114] When the picker is detected to be in the warehouse base location and performing an unloading action, control the picker to unload all the items it is carrying and reset the picker's remaining loading capacity to the maximum loading capacity.

[0115] When a picker is detected moving at the warehouse base location, if the picker is in the rear cross passage or the front cross passage, left and right movement is allowed.

[0116] During the movement of the picker, the order status changes and the picker status changes are monitored in real time. If any one of the following conditions is met: the picker reaches the intersection and cannot continue to move in the original direction, a new order is detected during the movement of the picker, or at least one item is picked during the movement of the picker, then the state transition to the next state is triggered.

[0117] Specifically, in this embodiment, under a given state Next, action The actions of the picker are determined. Specifically, =0 indicates that the picker unloads items when it is at the base; otherwise, it remains stationary. and The actions correspond to moving to the right and to the left, respectively, but are only permitted within intersecting aisles. It should be noted that in this embodiment, the actions are limited to feasible options, such as when the picker is located within a picking aisle. At this time, moving left or right is prohibited; and These correspond to moving upwards and downwards respectively. Additionally, when the picker is in the rear cross passage... When in a cross passage ahead, moving upwards is prohibited. When this happens, moving downwards is prohibited. This constraint mechanism effectively avoids invalid operations and ensures the actual executability of all generation strategies. During state transitions, the system evolves from one state to another based on the actions taken by the picker and the randomness of order arrival. In this embodiment, we assume... This is the minimum time unit required for the picker to move between adjacent storage locations, if performed at a non-warehouse base location. =0, then the picker stays at the current position. Time, at this moment and The same applies unless a new order arrives; if executed at the warehouse base location. =0, the picker will unload all items, resulting in the picker state being... It becomes (1, 2n-1, 2n, K); and The action is performed only in intersecting channels, and the required time is proportional to the distance between the channels. Specifically, only when... This is effective at any time, and the vertical coordinate changes in steps of 2. During operation, the picker state By increasing the step size by 2 in the vertical position portion, the picker state at time (t+1) is obtained. ; During operation, the picker state By reducing the step size by 2 in the vertical position portion, the picker state at time (t+1) is obtained. .

[0118] when and When the action is executed, if the picker reaches the cross passage and cannot continue, and at the same time a new order arrives and changes the status of existing orders in the warehouse... Either the picker collecting at least one item will result in a new state. Assuming the picker collects all items at the current storage location before transitioning to the new state, the following steps are performed: and When performing an action, regardless of whether new orders arrive, the status of existing orders in the warehouse remains unchanged. All of these changes are due to the relative distance altered by the movement of the picker, and the new picker state will change accordingly. Its horizontal position may change, but its vertical position remains the same, and the remaining capacity depends on the number of items collected during the action.

[0119] For the Markov decision process reward function, at time step t, the composite cumulative reward function is specifically defined as follows:

[0120]

[0121] In the formula, The compound cumulative reward for time step t; This is a bonus coefficient for picking efficiency; This refers to the maximum cargo capacity that the picker can carry in a single pick. The remaining loading capacity of the picker at time step t; This is the unloading incentive adjustment coefficient, with a value ranging from 0 to 1; The number of storage locations the picker moves within time step t; The number of items picked up by the picker within time step t; The action performed by the picker at time step t. This indicates that the picker is performing an unloading action.

[0122] When the picker performs at the base When the action is =0, the quantity of items is A smaller unloading incentive adjustment coefficient means that picking items is more valuable than unloading at the base, and the picker will tend to maximize capacity utilization before heading to the base, thereby minimizing travel distance. A larger unloading incentive adjustment coefficient indicates that the two actions are equally valuable, and the picker will tend to unload items as quickly as possible to minimize waiting time and delivery time. This reward mechanism optimizes the picking strategy by adjusting the unloading incentive adjustment coefficient, allowing the picker to decide whether to prioritize maximizing the number of items carried or returning to the base to unload as quickly as possible, thereby finding the optimal balance between efficiency and response speed. It should be noted that the goal of this embodiment is to train a strategy to maximize the cumulative discount reward by establishing a deep neural network model. Specifically, this embodiment uses an improved Q-learning algorithm, with the objective function being the cumulative discount reward, and the formula for calculating the cumulative discount reward is:

[0123]

[0124] In the formula, This represents the cumulative discount reward at time step t, i.e., the cumulative discount bonus. As a discount factor, in this embodiment, The discount factor is used to balance the importance of immediate rewards and future rewards; a smaller discount factor... The value indicates that the current action is more important than the future action; a larger value indicates a more important action. Value, on the other hand, places more emphasis on future rewards; This represents the discount factor after k steps in the future. For future time steps ( The cumulative reward at time t; k is the count of future steps, i.e., the increment of steps relative to the current time t; It is infinitely large.

[0125] This embodiment approximates the optimal Q-function using a deep neural network:

[0126]

[0127] In the formula, The optimal action value function represents the maximum expected reward obtained by taking action a in state s. State s can include information such as the picker location and order distribution. Reflects the state-action pair The expected maximum value of the cumulative discount reward that can be obtained by executing subsequent actions according to the optimal strategy; In the warehouse order picking problem, the action to be taken in state s can be moving to the right, moving to the left, moving up, moving down, or unloading items at the base. A strategy for selecting an action based on the current state; This is the expectation operator, used to calculate the mean of a random variable (such as an order arrival). This represents the cumulative discount reward at time step t, i.e., the cumulative discount bonus. Let t be the system state at time t, where time t indicates the temporal sequence of the state.

[0128] The input to a deep neural network model is a state-action pair, and the output is an estimated expected reward. During training, the network parameters are continuously adjusted through gradient descent, eventually deriving the optimal policy.

[0129]

[0130] In the formula, The optimal strategy for selecting an action based on the current state is the strategy that maximizes the expected cumulative reward of the discount among all possible strategies. For parameter extraction operators, they are used to find parameters that make... Action a that achieves the maximum value.

[0131] This embodiment constructs an optimal policy that maximizes rewards by adjusting the Q-learning mechanism. However, due to the complexity of the environment, the function cannot be directly accessed. Nevertheless, since neural networks are general function approximators, a neural network can be constructed and trained to approximate the function. In this way, by using the Q-learning method and approximating the optimal Q-function through a neural network to formulate a policy, it is possible to solve complex environmental decision-making problems and find action strategies that maximize long-term rewards. This relies on the powerful fitting ability of neural networks to cope with various uncertainties in the environment. In summary, this embodiment effectively solves the curse of dimensionality problem of traditional Q-learning in high-dimensional state spaces by utilizing the generalization ability of neural networks.

[0132] S6. Based on the real-time state of the picker and the moving value density, the Markov decision process is solved using a pre-built deep reinforcement learning neural network model to obtain the optimal picking path decision.

[0133] S7. Control the picker to perform picking operations based on the optimal picking path decision.

[0134] In some implementations, the deep reinforcement learning neural network model includes an input layer, a two-branch feature extraction module, a feature fusion module, a multi-layer feature compression module, and an output layer connected in sequence. The two-branch feature extraction module includes a parallel order status feature extraction unit and a picker status feature extraction unit. Both the order status feature extraction unit and the picker status feature extraction unit are composed of fully connected layers and linear rectified activation functions. The multi-layer feature compression module includes at least three fully connected layers connected in series. In this embodiment, the steps of solving the Markov decision process using a pre-built deep reinforcement learning neural network model based on the real-time state of the picker and the moving value density to obtain the optimal picking path decision include:

[0135] The mobility value density of each channel is combined with the real-time order status to form an order status vector; the dimension of the order status vector is twice the total number of channels.

[0136] The real-time state of the picker is nonlinearly transformed by the picker state feature extraction unit to extract the high-dimensional feature vector of the picker.

[0137] The order status vector is subjected to nonlinear transformation by the order status feature extraction unit to extract the order distribution feature vector;

[0138] The high-dimensional feature vector of the picker and the feature vector of the order distribution are concatenated by channels to generate a state fusion feature vector;

[0139] The state fusion feature vector is subjected to hierarchical feature transformation through a multi-layer feature compression module to obtain a decision-oriented high-order feature representation;

[0140] The decision-oriented high-order feature representation is mapped to the action space dimension, and the expected cumulative reward value of each action taken in the current state is calculated through a fully connected layer.

[0141] The action with the highest expected cumulative reward value is selected as the optimal action for the picker in the current state, and the optimal picking route decision is generated based on the optimal action.

[0142] Specifically, the deep reinforcement learning neural network model proposed in this embodiment is used to process the optimal action prediction of pickers in a dynamic warehouse environment. This deep reinforcement learning neural network model, as a Q-network, is used to approximate the action value function. The deep reinforcement learning neural network model contains multiple fully connected layers, which handle the picker state and the order state respectively, and then combine the two to make the final action prediction. Figure 3 This is a schematic diagram of a deep reinforcement learning neural network model architecture. The specific processing flow is as follows: The input of the deep reinforcement learning neural network model is divided into picker states. and order status The system consists of two parts, where the picker state is a four-dimensional vector, represented as follows: Order status It is a vector of size 2N, where N represents the total number of picking lanes in the warehouse. This structure enables the network to process information from the picker state and the order state separately. This information is received through different input layers and integrated in subsequent network layers to more accurately predict the optimal action in a given state. It effectively utilizes the power of deep learning to solve complex decision-making problems and achieves efficient order picking in a dynamic warehouse environment.

[0143] In the picker state feature extraction unit, the picker state input is processed by the first fully connected layer. This first fully connected layer uses the Rectified Linear Unit (ReLU) activation function to transform the four-dimensional picker state into a high-dimensional feature vector of the picker. The feature space dimension of the high-dimensional feature vector of the picker is... This transformation method can capture the key features of the picker's state. The weight matrix of the picker state feature extraction unit has a dimension of [missing information]. Simultaneously, in the order status feature extraction unit, the order status is input into the second fully connected layer for processing. The second fully connected layer also uses the ReLU activation function to transform the 2N-dimensional order status vector into an order distribution feature vector. The feature space dimension of the order distribution feature vector is... This second fully connected layer can extract key features from warehouse order status. The weight matrix of the order status feature extraction unit has a dimension of [missing information]. Subsequently, the feature fusion module concatenates the high-dimensional feature vector of the picker and the feature vector of the order distribution into a concatenation module of size ( ). The 10-dimensional state fusion feature vector integrates information from two states to form a comprehensive representation of the system state.

[0144] Next, in this embodiment, the concatenated state fusion feature vector is input into a multi-layer feature compression module for hierarchical feature transformation. The multi-layer feature compression module includes multiple fully connected layers. In this embodiment, for ease of description, the multi-layer feature compression module includes three fully connected layers: a third fully connected layer, a fourth fully connected layer, and a fifth fully connected layer connected sequentially. The third fully connected layer includes... Each unit is used, and the ReLU activation function is employed to transform the state and fuse the feature vector; the fourth fully connected layer reduces the dimension of the feature vector output from the third fully connected layer to [missing information]. The fourth fully connected layer also uses the ReLU activation function; the fifth fully connected layer further reduces the dimension of the feature vector output by the fourth fully connected layer to [number of units]. The fifth fully connected layer also uses the ReLU activation function, where... , and These represent the feature layer dimensions (i.e., the number of neurons) of different fully connected layers in the multi-layer feature compression module. Specifically, these are the intermediate dimensions used by the three fully connected layers to progressively reduce the dimensionality of the state-fused feature vector. The dimension of the weight matrix of the third fully connected layer; The dimension of the weight matrix for the fourth fully connected layer; The dimension of the weight matrix of the fifth fully connected layer; The dimension of the output layer weight matrix; This represents the number of neurons in the third fully connected layer, i.e., the initial processing dimension of the state fusion feature vector; This represents the number of neurons in the fourth fully connected layer, i.e., the intermediate feature compression dimension. This represents the number of neurons in the fifth fully connected layer, which is the dimension of the final decision-oriented high-order feature representation. The action space size refers to the number of different actions that the picker can take during the warehouse order picking process. The output layer corresponds to the prediction of different actions, which determines the dimension of the output layer and the type of information output by the neural network.

[0145] In this embodiment, through the steps described above, the network progressively refines and compresses information extracted from the picker status and order status, providing a more compact and higher-order feature representation for the final action prediction. Multi-layered processing helps capture complex patterns in the data, supporting more accurate decision-making. Finally, the output layer... The decision-oriented high-order feature representation of a dimension is mapped to the action space, predicts the Q-value of each possible action, and selects the action with the highest Q-value as the optimal action of the picker. The Q-value is the expected cumulative reward value of the action. This process realizes the selection from the state to the optimal action through a deep neural network, ensuring that the action with the highest expected reward is taken in a given state. This effectively guides the operation of the picker in the warehouse, improves the efficiency and accuracy of order picking in a dynamic warehouse environment, captures the complex interaction between the picker and the warehouse order, provides efficient decision support for the picker, and improves the overall operational efficiency and performance.

[0146] It should be noted that in the training process of the deep reinforcement learning neural network model, this embodiment adopts a dual-network architecture of policy network and target network. The policy network is used for real-time decision-making and selects actions based on the epsilon-greedy policy, which balances the exploration of the action space and the utilization of learned value. The target network serves as a stable reference, and its parameters are periodically updated from the learned parameters of the policy network. To enhance the learning process, this embodiment introduces an experience replay mechanism. The experience replay mechanism acts as a repository, storing the agent's experience in the form of state transitions. During training, the deep reinforcement learning agent learns from small batches of randomly sampled and transformed samples from the experience replay, which allows past experience to be effectively reused and stabilizes the learning process.

[0147] Training unfolds over Ne epochs, each containing Ns steps, where Ne is the number of training epochs and Ns is the number of steps per epoch. In each step, the agent first observes the current state. Then, in this embodiment, an epsilon-greedy method and a policy network are used to select an action, and a corresponding reward is obtained upon transitioning to the next state. The current state, action, next state, and reward together constitute a transition, stored in the experience replay with the capacity of the experience replay. Subsequently, this embodiment randomly selects mini-batch samples of transitions from the experience replay, and the policy network trains based on these experiences. If the number of transitions in the experience replay is insufficient to form a mini-batch, that training step is temporarily skipped. The training procedure for each mini-batch is a multi-step process: first, the state, next state, action, and reward within the batch are stacked for parallel processing; then, the policy network is set to training mode, and the target network enters evaluation mode. This setting allows the policy network to calculate state-action values, while the target network calculates the expected state-action values. The difference between the two is used to calculate the loss. This embodiment uses the Huber Loss function. To increase robustness, the calculated loss is backpropagated through the policy network to drive its optimization. To ensure stability, the target network is updated at a low frequency, specifically once every Nupdate steps. This update follows a soft update rule and uses the target network update coefficients. The new target network weights are updated to a weighted average of the policy network weights and the existing target network weights. This gradual update mechanism can prevent large fluctuations and promote a smoother learning process. Here, Nupdate is the number of steps for updating the target network. The target network update coefficients are used to softly update the target network weights. This embodiment effectively solves the stability problem in deep reinforcement learning by separating the policy network and the target network and combining the experience replay mechanism. It can be applied to practical application scenarios with continuous state space, such as dynamic warehouse order picking.

[0148] In summary, this embodiment proposes a dynamic picking path optimization method based on Deep Reinforcement Learning (DRL), which can solve the dynamic order picking problem in a single-block warehouse layout served by autonomous picking equipment. This not only provides a solution to the current problem but also lays the foundation for future research, enabling the exploration of more complex warehouse environments. Specifically, this embodiment can extend the deep reinforcement learning framework to multi-block layouts and scenarios involving multiple coordinated picking equipment. Through the predictive and adaptive capabilities of deep reinforcement learning, warehouse operations can balance efficiency and flexibility, quickly responding to dynamic changes in customer needs. For example, in multi-block warehouse layouts, existing models can be tested and optimized in larger and more complex warehouse structures, thereby further improving the operational efficiency of large-scale warehouses. Regarding multi-device collaboration, research can be conducted on how to achieve effective cooperation between multiple autonomous picking devices to further improve picking efficiency and reduce bottlenecks.

[0149] This invention provides a dynamic picking path optimization method based on deep reinforcement learning. The method collects warehouse aisle layout parameters and picker physical parameters to form warehouse infrastructure data, and performs structured encoding on this data to obtain structured coded warehouse data. Based on this structured coded data, the current position and remaining capacity of the picker are analyzed to obtain the picker's state vector. Real-time data of the warehouse's pending order queue is collected, and the movement value density of orders within each aisle is calculated based on the pending order queue data and the picker's state vector. A state space is defined based on the picker's state vector and the pending order queue data, and a discrete execution action space for the picker at each time step is constructed. Based on the state space and the discrete execution action space, a cumulative reward is set with the optimization objective of minimizing the average order waiting time, and the warehouse order picking problem is modeled as a Markov decision process. Based on the picker's real-time state and the movement value density, the Markov decision process is solved using a pre-built deep reinforcement learning neural network model to obtain the optimal picking path decision. The optimal picking path decision is then used to control the picker to perform picking operations. Compared with existing technologies, this method collects and structures various warehouse data, and dynamically adjusts the picking path through deep reinforcement learning, thereby achieving real-time optimization of the dynamic picking path. This effectively improves the efficiency of warehouse order picking, reduces the average order waiting time, and enhances the response speed and intelligence level of the warehouse logistics system.

[0150] It should be noted that the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0151] In one embodiment, such as Figure 4 As shown, this embodiment of the invention provides a dynamic picking path optimization system based on deep reinforcement learning, the system comprising:

[0152] The data acquisition module 101 is used to collect warehouse aisle layout parameters and picker physical parameters to form warehouse basic structure data, and to perform structured encoding on the warehouse basic structure data to obtain warehouse structured encoded data.

[0153] The state parsing module 102 is used to parse the current position of the picker and the remaining loading capacity of the picker based on the warehouse structured coded data to obtain the picker state vector;

[0154] The movement analysis module 103 is used to collect the pending order queue data of the warehouse in real time, and calculate the movement value density of orders in each channel based on the pending order queue data and the picker state vector.

[0155] The space construction module 104 is used to define the state space based on the picker state vector and the order queue data to be processed, and to construct the discrete execution action space of the picker at each time step.

[0156] The picking modeling module 105 is used to model the warehouse order picking problem as a Markov decision process based on the state space and the discrete execution action space, setting a cumulative reward with the optimization objective of minimizing the average waiting time of orders.

[0157] The path decision module 106 is used to solve the Markov decision process using a pre-built deep reinforcement learning neural network model based on the real-time status of the picker and the moving value density, so as to obtain the optimal picking path decision.

[0158] The picking execution module 107 is used to control the picker to perform picking operations based on the optimal picking path decision.

[0159] For specific limitations regarding a dynamic picking path optimization system based on deep reinforcement learning, please refer to the above-described limitations regarding a dynamic picking path optimization method based on deep reinforcement learning, which will not be repeated here. Those skilled in the art will recognize that the various modules and steps described in conjunction with the embodiments disclosed in this application can be implemented in hardware, software, or a combination of both. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0160] This invention provides a dynamic picking path optimization system based on deep reinforcement learning. The system acquires warehouse aisle layout parameters and picker physical parameters through a data acquisition module to form warehouse infrastructure data, and then performs structured encoding on this data to obtain structured coded warehouse data. A state parsing module parses the current position and remaining capacity of the picker based on the structured coded warehouse data to obtain the picker state vector. A movement analysis module collects the pending order queue data in real time and calculates the movement value density of orders within each aisle based on the pending order queue data and the picker state vector. Spatial structure... The system defines a state space based on the picker's state vector and the order queue data, and constructs a discrete action space for the picker at each time step. The picking modeling module, based on the state space and the discrete action space, sets a cumulative reward with the optimization objective of minimizing the average order waiting time, modeling the warehouse order picking problem as a Markov decision process. The path decision module, based on the picker's real-time state and the movement value density, uses a pre-built deep reinforcement learning neural network model to solve the Markov decision process, obtaining the optimal picking path decision. The picking execution module controls the picker to perform picking operations based on the optimal picking path decision. Compared with existing technologies, this system collects and structures various warehouse data, dynamically adjusts the picking path through deep reinforcement learning, and achieves real-time optimization of the dynamic picking path. This effectively improves the efficiency of warehouse order picking, reduces the average order waiting time, and enhances the response speed and intelligence level of the warehouse logistics system.

[0161] The embodiments described above are merely preferred embodiments of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various improvements and substitutions without departing from the technical principles of this invention, and these improvements and substitutions should also be considered within the scope of protection of this application. Therefore, the scope of protection of this patent application should be determined by the scope of the claims.

Claims

1. A dynamic picking path optimization method based on deep reinforcement learning, characterized in that, Includes the following steps: Collect warehouse aisle layout parameters and picker physical parameters to form warehouse basic structure data, and perform structured encoding on the warehouse basic structure data to obtain warehouse structured encoded data; Based on the warehouse structured coded data, the current position of the picker and the remaining loading capacity of the picker are parsed to obtain the picker state vector; Real-time acquisition of pending order queue data in the warehouse, and calculation of the movement value density of orders in each channel based on the pending order queue data and the picker state vector; Define the state space based on the picker state vector and the order queue data to be processed, and construct the discrete execution action space of the picker at each time step; Based on the state space and the discrete execution action space, a cumulative reward is set with the optimization objective of minimizing the average order waiting time, and the warehouse order picking problem is modeled as a Markov decision process. Based on the real-time status of the picker and the moving value density, the Markov decision process is solved using a pre-built deep reinforcement learning neural network model to obtain the optimal picking path decision. The optimal picking path decision control controls the picker to perform picking operations. The pending order queue data includes the coordinates of the item storage location of the pending orders in the warehouse; the movement value density is the sum of the ratios of the number of orders at each picking position in each channel to the minimum relative distance between the current position of the picker and the coordinates of the item storage location when the picker moves up or down. The steps of modeling the warehouse order picking problem as a Markov decision process based on the state space and the discrete action space, setting a cumulative reward with the optimization objective of minimizing the average order waiting time, include: Based on the state space and the discrete execution action space, determine the state transition rules for the picker to transition from the current state to the next state; Based on the state transition rules, the state transition probability of the picker transitioning from the current state to the next state after performing an action is calculated, and a state transition probability model is constructed. With the goal of minimizing the average order waiting time, a composite cumulative reward function is constructed based on the results of the picker's actions, which includes picking efficiency rewards, travel distance penalties, and unloading incentives. The results of the picker's actions include at least the time required for the picker to complete the picking order, the travel distance, and the order completion rate. Based on the state space, the discrete execution action space, and the state transition probability model, the warehouse order picking problem is modeled as a Markov decision process by maximizing the optimization objective of the composite cumulative reward function.

2. The dynamic picking path optimization method based on deep reinforcement learning as described in claim 1, characterized in that, The step of performing structured encoding on the warehouse infrastructure data to obtain structured warehouse data includes: Based on the pre-collected physical structure data of the warehouse, a discretized coordinate system for the warehouse is established with the center of the entrance intersection as the origin of the coordinate system; Based on the warehouse aisle layout parameters, all aisles in the warehouse and the storage cells on both sides of each aisle are encoded to obtain the aisle layout parameter encoding data; The channel layout parameter encoding data is mapped to the warehouse discretized coordinate system to obtain the warehouse channel storage encoding data; Based on the physical parameters of the picker, the initial position of the picker is fixed as the coordinates of the entrance cross passage, and the initial unloaded capacity is set to the maximum loading capacity. Based on the initial position of the picker and the maximum loading capacity, the initial state vector of the picker is obtained; The warehouse channel storage encoding data and the picker initial state vector are encapsulated into a structured data packet to form warehouse structured encoding data.

3. The dynamic picking path optimization method based on deep reinforcement learning as described in claim 1, characterized in that, The step of parsing the current position and remaining capacity of the picker based on the warehouse structured coded data to obtain the picker state vector includes: The real-time location data of the picker is obtained, and the real-time location data of the picker is matched with the structured coding data of the warehouse to parse out the channel number and storage cell number of the picker, and thus obtain the current location of the picker. Obtain the current load weight data of the picker, and determine the remaining load capacity of the picker based on the maximum load capacity of the picker in the warehouse structured coding data and the current load weight data; The current position of the picker and the remaining loading capacity of the picker are combined into a picker state vector.

4. The dynamic picking path optimization method based on deep reinforcement learning as described in claim 1, characterized in that, The state transition rule is specifically as follows: Set the minimum time unit required for the picker to move from one storage location to another, and detect the picker's current position and perform the action; When the picker is detected to be in a non-warehouse base location and performs a stationary action, the picker is controlled to remain stationary in the current location for a period of time until the minimum time unit is reached. If a new order is detected during the time the picker remains stationary, the state transition to the next state is triggered. When the picker is detected to be in the warehouse base location and performing an unloading action, control the picker to unload all the items it is carrying and reset the picker's remaining loading capacity to the maximum loading capacity. When a picker is detected moving at the warehouse base location, if the picker is in the rear cross passage or the front cross passage, left and right movement is allowed. During the movement of the picker, the order status changes and the picker status changes are monitored in real time. If any one of the following conditions is met: the picker reaches the intersection and cannot continue to move in the original direction, a new order is detected during the movement of the picker, or at least one item is picked during the movement of the picker, then the state transition to the next state is triggered.

5. The dynamic picking path optimization method based on deep reinforcement learning as described in claim 1, characterized in that, The specific composite cumulative reward function is as follows: In the formula, The compound cumulative reward for time step t; This is a bonus coefficient for picking efficiency; This refers to the maximum cargo capacity that the picker can carry in a single pick. The remaining loading capacity of the picker at time step t; This is the unloading incentive adjustment coefficient; The number of storage locations the picker moves within time step t; The number of items picked up by the picker within time step t; The action performed by the picker at time step t. This indicates that the picker is performing an unloading action.

6. The dynamic picking path optimization method based on deep reinforcement learning as described in claim 1, characterized in that: The deep reinforcement learning neural network model includes an input layer, a two-branch feature extraction module, a feature fusion module, a multi-layer feature compression module, and an output layer connected in sequence. The dual-branch feature extraction module includes a parallel order status feature extraction unit and a picker status feature extraction unit; both the order status feature extraction unit and the picker status feature extraction unit are composed of a fully connected layer and a linear rectified activation function. The multi-layer feature compression module includes at least three fully connected layers connected in series.

7. The dynamic picking path optimization method based on deep reinforcement learning as described in claim 6, characterized in that, The steps for obtaining the optimal picking path decision by solving the Markov decision process using a pre-built deep reinforcement learning neural network model based on the real-time state of the picker and the moving value density include: The mobility value density of each channel is combined with the real-time order status to form an order status vector; the dimension of the order status vector is twice the total number of channels. The real-time state of the picker is nonlinearly transformed by the picker state feature extraction unit to extract the high-dimensional feature vector of the picker. The order status vector is subjected to a nonlinear transformation by the order status feature extraction unit to extract the order distribution feature vector; The high-dimensional feature vector of the picker and the feature vector of the order distribution are concatenated by channels to generate a state fusion feature vector; The state fusion feature vector is subjected to hierarchical feature transformation through a multi-layer feature compression module to obtain a decision-oriented high-order feature representation; The decision-oriented high-order feature representation is mapped to the action space dimension, and the expected cumulative reward value of each action taken in the current state is calculated through a fully connected layer. The action with the highest expected cumulative return value is selected as the optimal action for the picker in the current state, and the optimal picking route decision is generated based on the optimal action.

8. A dynamic picking path optimization system based on deep reinforcement learning, characterized in that, The system includes: The data acquisition module is used to collect warehouse aisle layout parameters and picker physical parameters to form warehouse basic structure data, and to perform structured encoding on the warehouse basic structure data to obtain warehouse structured encoded data. The state parsing module is used to parse the current position and remaining loading capacity of the picker based on the warehouse structured coded data to obtain the picker state vector; The movement analysis module is used to collect the pending order queue data of the warehouse in real time, and calculate the movement value density of orders in each channel based on the pending order queue data and the picker state vector. The space construction module is used to define the state space based on the picker state vector and the order queue data to be processed, and to construct the discrete execution action space of the picker at each time step; The picking modeling module is used to model the warehouse order picking problem as a Markov decision process based on the state space and the discrete execution action space, setting a cumulative reward with the optimization objective of minimizing the average order waiting time; The path decision module is used to solve the Markov decision process based on the real-time status of the picker and the moving value density, and obtain the optimal picking path decision by using a pre-built deep reinforcement learning neural network model. The picking execution module is used to control the picker to perform picking operations based on the optimal picking path decision. The pending order queue data includes the coordinates of the item storage location of the pending orders in the warehouse; the movement value density is the sum of the ratios of the number of orders at each picking position in each channel to the minimum relative distance between the current position of the picker and the coordinates of the item storage location when the picker moves up or down. The picking modeling module is specifically used for: Based on the state space and the discrete execution action space, determine the state transition rules for the picker to transition from the current state to the next state; Based on the state transition rules, the state transition probability of the picker transitioning from the current state to the next state after performing an action is calculated, and a state transition probability model is constructed. With the goal of minimizing the average order waiting time, a composite cumulative reward function is constructed based on the results of the picker's actions, which includes picking efficiency rewards, travel distance penalties, and unloading incentives. The results of the picker's actions include at least the time required for the picker to complete the picking order, the travel distance, and the order completion rate. Based on the state space, the discrete execution action space, and the state transition probability model, the warehouse order picking problem is modeled as a Markov decision process by maximizing the optimization objective of the composite cumulative reward function.

Images