Warehouse goods allocation intelligent distribution method based on deep reinforcement learning

Abstract

The invention discloses an intelligent warehouse goods allocation method based on deep reinforcement learning. The method comprises the following steps: obtaining and preprocessing multi-source business data and space resource data of a warehouse system; constructing a state element set of a warehouse goods allocation environment; uniformly coding the state element set to generate a state representation vector; constructing an action set of warehouse allocation allocation and performing dynamic screening to generate a dynamic executable action set; constructing a composite reward function, and generating a training environment interaction data set in combination with the state representation vector and the dynamic executable action set; training the value network and the strategy network, and generating a goods allocation strategy model; and performing strategy reasoning based on the goods allocation strategy model, generating a target goods allocation decision, and converting the target goods allocation decision into a warehousing execution instruction set. According to the method, the dynamic goods allocation model is constructed by adopting the deep reinforcement learning method, intelligent decision and execution of the storage goods allocation are realized, and the method has the advantages of high adaptability, high resource utilization rate and excellent operation efficiency.

Description

Technical Field

[0001] This invention relates to the field of intelligent warehouse scheduling technology, and in particular to an intelligent warehouse location allocation method based on deep reinforcement learning. Background Technology

[0002] With the development of e-commerce and intelligent logistics, the types of SKUs and order volumes in warehousing systems continue to grow. Location allocation has gradually shifted from being driven by manual experience to automated scheduling based on rules or optimization algorithms. Existing technologies typically rely on fixed rules, heuristic algorithms, or static optimization models to comprehensively assess factors such as SKU attributes, storage capacity, and operational paths to achieve location allocation and in-warehouse adjustments. These methods are effective in small- to medium-scale or stable operating conditions, but they generally depend on manual rule design or offline parameter settings, making them difficult to adapt to the complex and ever-changing actual warehousing operating environment.

[0003] In practical applications, existing technologies often fail to effectively integrate multi-source business data and spatial resource data, lacking the ability to jointly model SKU relationships, order time-series changes, and warehouse dynamic status. Furthermore, the action space construction and constraint mechanisms are relatively crude, easily leading to frequent relocations or path conflicts, impacting operational efficiency. In addition, existing methods mostly employ static or single-objective evaluation functions, lacking adaptive optimization capabilities based on operational feedback, making it difficult to continuously improve the overall performance of location allocation decisions over long-term operation.

[0004] Therefore, how to provide a method for intelligent allocation of warehouse storage locations based on deep reinforcement learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose an intelligent warehouse location allocation method based on deep reinforcement learning. This invention uses deep reinforcement learning to construct a dynamic location allocation model to achieve intelligent decision-making and execution of warehouse locations, and has the advantages of strong adaptability, high resource utilization and excellent operational efficiency.

[0006] A method for intelligent allocation of warehouse storage locations based on deep reinforcement learning according to an embodiment of the present invention includes the following steps: Acquire multi-source business data and spatial resource data of the warehousing system within the target time period, perform preprocessing, and obtain a standardized warehousing dataset. Construct a set of state elements for the warehouse location allocation environment based on a standardized warehouse dataset; The set of state elements is uniformly encoded, and a multi-dimensional warehousing environment embedding representation vector is constructed and determined as the state representation vector. Based on the set of state elements and spatial resource data, a set of actions for warehouse location allocation is constructed, and a dynamic location freezing mechanism is used to filter the set of actions to generate a set of dynamically executable actions. Construct a composite reward function and combine it with a state representation vector and a set of dynamically executable actions to generate an interactive dataset for the training environment; The value network and policy network are trained based on the interactive dataset of the training environment to generate a cargo location allocation strategy model. In actual warehouse operations, the location allocation strategy model is used to perform strategy reasoning on the objects to be put on the shelves, generate target location allocation decisions, and convert them into a set of warehouse execution instructions.

[0007] Optionally, the multi-source business data includes SKU attribute data, inbound and outbound order data, historical turnover data, and product association data; the spatial resource data includes warehouse topology data, storage location capacity data, current storage location occupancy status data, and operational accessibility data; and the preprocessing includes data cleaning, alignment, and numerical processing.

[0008] Optionally, the construction of the state element set specifically includes: Extract the SKU basic attribute field set, order line time series field set, turnover statistics field set, and product association relationship field set from the multi-source business data of the standardized warehousing dataset; Perform feature filtering and numerical mapping operations on the SKU basic attribute field set to generate an initial SKU feature representation set, and construct a cargo similarity manifold space based on the initial SKU feature representation set; In the cargo similarity manifold space, a distance measurement operation is performed on the initial feature representations of any two SKUs, and the distance result is mapped to the preset maximum distance benchmark to generate a normalized distance result. Then, a normalized similarity score is generated by subtracting the normalized distance result from one. Based on the normalized similarity score, a continuous embedding learning operation is performed on the initial feature representation of SKU to obtain a set of continuous embedding vectors of SKU located in the cargo similarity manifold space; The SKU continuous embedding vector set is determined as the SKU feature vector. Time series processing operation is performed on the order line time series field set to obtain the order time series feature vector. Feature extraction operation is performed on the spatial resource data to obtain the storage space feature vector and the warehouse capacity constraint vector. Numerical encoding and aggregation operation is performed on the product association field set to generate the product association feature vector. Finally, turnover feature extraction operation is performed on the turnover statistics field set to generate the turnover feature vector. The SKU feature vector, order time sequence feature vector, storage space feature vector, warehouse capacity constraint vector, product association feature vector, and turnover feature vector are combined to generate a set of state elements for the warehouse storage location allocation environment.

[0009] Optionally, obtaining the state representation vector specifically includes: Perform missing term imputation and numerical range correction on each vector in the state element set to generate an coded input set; Dimension alignment is performed on each vector in the coded input set to generate a set of feature coded vectors of uniform length; Normalization and scaling processes are performed on the feature encoding vector set to generate a normalized encoding vector set. The normalized encoding vector set is subjected to element-level concatenation processing to form a concatenated encoding vector, and the concatenated encoding vector is subjected to interactive encoding processing to generate a multidimensional interactive encoding vector; A fusion weight generation process is performed on the multidimensional interactive coding vector to obtain a set of element fusion weights. Based on the set of element fusion weights, a weighted combination process is performed on the set of normalized coding vectors to generate a multidimensional warehousing environment embedding representation vector. The multidimensional warehousing environment embedding representation vector is then determined as the state representation vector.

[0010] Optionally, the generation of the dynamically executable action set specifically includes: Based on the correspondence between the current pending warehousing operations and the set of status elements, the pending put-away objects are determined, and based on the set of status elements and spatial resource data, the availability of the warehousing locations is determined to generate a set of candidate storage locations corresponding to the put-away objects. The put-away objects include SKUs and order lines. Based on the candidate location set and spatial resource data, an initial action set is constructed, including single location allocation action, location relocation action, and combined action. For each action in the initial action set, perform action element parsing processing to obtain the target storage location identifier, source storage location identifier, inbound path identifier, put-away object identifier, and handling path identifier associated with the action; A set of action legality constraint rules is constructed based on spatial resource data. The set of action legality constraint rules includes storage space capacity constraint rules, occupancy status constraint rules, storage area topology connectivity constraint rules, channel accessibility constraint rules, and operation path conflict constraint rules. The initial action set is processed item by item to match the action legality constraint rule set, and actions that are determined not to meet the constraints are removed to generate an executable action set. Based on historical turnover data, storage location access frequency data, and historical strategy evaluation data, a statistical sequence of storage location time windows is extracted, and the statistical sequence of storage location time windows is standardized to obtain a statistical feature sequence of storage locations. A comprehensive scoring process is performed on the statistical characteristic sequence of cargo locations to generate a cargo location freezing score sequence; Based on the freezing threshold, the freezing scores corresponding to each storage location in the storage location freezing score sequence are compared and processed to select storage locations whose freezing scores meet the freezing conditions. A dynamic storage location freezing set is generated, and action masking processing is performed on the storage location-related actions corresponding to the dynamic storage location freezing set to generate a dynamic executable action set that changes over time.

[0011] Optionally, the generation of the training environment interaction dataset specifically includes: Obtain the state representation vector, the set of dynamic executable actions, and the set of order lines corresponding to the current pending warehousing operation task. Extract the set of turnover statistics indicators based on historical turnover data and the set of order lines. Extract the set of operation path indicators based on operation channel accessibility data and warehouse topology data. Extract the set of warehouse space utilization indicators based on warehouse space capacity data and current warehouse space occupancy status data. Extract the set of goods adjacency indicators based on the set of goods association fields and the put-away object identifier. The system performs index normalization and interval mapping on the set of turnover statistics indicators to generate turnover efficiency reward items; it performs cost mapping and interval mapping on the set of operation path indicators to generate operation path cost penalty items; it performs utilization rate calculation and interval mapping on the set of storage space utilization indicators to generate storage space utilization reward items; and it performs adjacency scoring and interval mapping on the set of goods adjacency indicators to generate goods adjacency reward items. Reward weights are assigned to the turnover efficiency reward item, the operation path cost penalty item, the warehouse space utilization rate reward item and the goods adjacency reward item, and the reward results of each reward are weighted and summed to obtain the reward value of the composite reward function. A deep reinforcement learning training environment is constructed based on the state representation vector, the set of dynamically executable actions, and the reward value of the composite reward function. The correspondence between the state representation vector, the set of dynamically executable actions, the reward value of the composite reward function, and the action execution results is recorded in the training environment to generate an interactive training environment dataset.

[0012] Optionally, the generation of the storage location allocation strategy model specifically includes: Based on the interactive dataset of the training environment, determine the state representation vector, the set of dynamically executable actions, the reward value of the composite reward function, and the sample record set of action execution results; Construct a policy network and a value network composed of deep neural networks, and configure the network parameter sets and parameter update step sizes for the policy network and the value network respectively; Acquire historical operation trajectory data and online interaction sampling data, perform merge sampling processing with the sample record set, and generate a training sample sequence including state representation vector, dynamic executable action set, selected action, reward value of composite reward function and action execution result; The updated state representation vector corresponding to the current state representation vector under the selected action is extracted from the training sample sequence in chronological order. Value estimation is then performed on both the current and updated state representation vectors based on the value network, generating the current value estimation result and the next state value estimation result. ； Based on the reward value of the composite reward function, the current value estimation result, and the next state value estimation result, the target value calculation process is performed to obtain the target value estimation result; The value error is calculated based on the target value estimation result and the current value estimation result, and gradient update processing is performed on the network parameter set of the value network based on the value error; The strategy loss is calculated based on the value error, the probability distribution of the selected action and the action output by the strategy network. Then, gradient update processing is performed on the network parameter set of the strategy network based on the strategy loss, and the strategy network after parameter update is determined as the location allocation strategy model.

[0013] Optionally, the generation of the warehouse execution instruction set specifically includes: In the actual warehouse operation, acquire the objects to be processed and put on the shelves, and determine the corresponding state representation vector and dynamic executable action set based on the objects to be processed and put on the shelves. The state representation vector and the set of dynamic executable actions are input into the storage location allocation strategy model. Based on the storage location allocation strategy model, strategy reasoning is performed, and the target storage location allocation decision is output. The target storage location allocation decision includes the target storage location identifier, the inbound path identifier, the action type identifier, the source storage location identifier, and the handling path identifier. Based on the action type identifier, the target storage location allocation decision execution instruction conversion processing is performed to generate shelving instructions, storage location locking instructions and in-warehouse handling instructions; The shelving instructions, location locking instructions, and in-warehouse handling instructions are aggregated to generate a warehouse execution instruction set, which is then sent to the warehouse execution system for location allocation.

[0014] The beneficial effects of this invention are: This invention constructs a set of warehouse location allocation environment state elements that integrate multi-source business data and spatial resource data. It achieves unified modeling of SKU attributes, order time series, location space status, warehouse capacity constraints, and product association relationships. By forming a structured state representation vector through multi-dimensional embedding encoding, the warehouse operation status can be completely and continuously expressed and used in subsequent decision-making processes. This avoids the information fragmentation problem caused by relying on only local rules or single features for location allocation in existing technologies, and improves the adaptability and stability of location allocation decisions in complex warehousing scenarios.

[0015] Meanwhile, this invention introduces an action filtering method based on space availability determination and dynamic storage location freezing mechanism during the action space construction process. It systematically models single storage location allocation actions, storage location migration actions, and combined actions, and dynamically constrains the storage location status through factors such as historical turnover, access frequency, and strategy risk. This effectively reduces unnecessary frequent relocation and path conflicts, lowers the probability of invalid operations in the warehousing process, and enables the executable action set to be dynamically adjusted according to the warehousing operation status, thereby improving the overall operation efficiency and the rationality of resource utilization.

[0016] Furthermore, this invention constructs a composite reward function that simultaneously considers turnover efficiency, operational path cost, storage space utilization, and product adjacency. It also uses a deep reinforcement learning framework to jointly train the policy network and value network, enabling the storage location allocation strategy to continuously optimize under the drive of actual operational feedback. This overcomes the problems of static evaluation functions and difficulty in long-term adaptation in existing technologies. In actual warehousing operations, the trained storage location allocation strategy model can perform policy reasoning based on real-time state representation vectors and dynamic executable action sets, outputting structured target storage location allocation decisions and converting them into directly executable warehousing instructions. This achieves automated, continuous optimization, and stable execution of warehousing storage location allocation decisions. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0018] Figure 1 This is a flowchart of a warehouse location intelligent allocation method based on deep reinforcement learning proposed in this invention; Figure 2 This is a schematic diagram of the dynamic executable action set generation process of a warehouse location intelligent allocation method based on deep reinforcement learning proposed in this invention. Figure 3 This is a schematic diagram of the training process for the location allocation strategy model of the intelligent warehouse location allocation method based on deep reinforcement learning proposed in this invention. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0020] refer to Figure 1-3 A method for intelligent allocation of warehouse storage locations based on deep reinforcement learning includes the following steps: Acquire multi-source business data and spatial resource data of the warehousing system within the target time period, perform preprocessing, and obtain a standardized warehousing dataset. Construct a set of state elements for the warehouse location allocation environment based on a standardized warehouse dataset; The set of state elements is uniformly encoded, and a multi-dimensional warehousing environment embedding representation vector is constructed and determined as the state representation vector. Based on the set of state elements and spatial resource data, a set of actions for warehouse location allocation is constructed, and a dynamic location freezing mechanism is used to filter the set of actions to generate a set of dynamically executable actions. Construct a composite reward function and combine it with a state representation vector and a set of dynamically executable actions to generate an interactive dataset for the training environment; The value network and policy network are trained based on the interactive dataset of the training environment to generate a cargo location allocation strategy model. In actual warehouse operations, the location allocation strategy model is used to perform strategy reasoning on the objects to be put on the shelves, generate target location allocation decisions, and convert them into a set of warehouse execution instructions.

[0021] In this embodiment, the multi-source business data includes SKU attribute data, inbound and outbound order data, historical turnover data, and product association data; the spatial resource data includes warehouse topology data, storage location capacity data, current storage location occupancy status data, and operational accessibility data; and the preprocessing includes data cleaning, alignment, and numerical processing.

[0022] In this embodiment, the construction of the state element set specifically includes: Extract the SKU basic attribute field set, order line time series field set, turnover statistics field set, and product association relationship field set from the multi-source business data of the standardized warehousing dataset; Perform feature filtering and numerical mapping operations on the SKU basic attribute field set to generate an initial SKU feature representation set, and construct a cargo similarity manifold space based on the initial SKU feature representation set; The construction of the cargo similarity manifold space includes: performing attribute comparison processing on each item of the initial SKU feature representation set to determine the set of feature items that can participate in the structural processing; performing multi-level partitioning operations on the feature item set to form several feature subsets; performing data range sorting processing on each feature subset and establishing adjacent feature sequence sequences between subsets; performing proximity relationship extraction operations on adjacent feature sequence sequences to generate a neighborhood relationship set; performing region connection processing on the neighborhood relationship set to form an initial manifold fragment; performing fragment splicing operations on the initial manifold fragment to form a continuous manifold structure; performing structural translation processing, structural scaling processing, and structural relocation processing on the continuous manifold structure; and using the above-processed structure as the cargo similarity manifold space. Within the cargo similarity manifold space, a distance measurement operation is performed on the initial feature representations of any two SKUs, and the distance result is mapped to a preset maximum distance benchmark to generate a normalized distance result. Then, a normalized similarity score is generated by subtracting the normalized distance result from the normalized distance result. The normalized similarity score increases as the distance result decreases and decreases as the distance result increases. Based on the normalized similarity score, a continuous embedding learning operation is performed on the initial feature representation of SKU to obtain a set of continuous embedding vectors of SKU located in the cargo similarity manifold space; The continuous embedding learning operation includes: performing feature sequence rearrangement on the initial feature representation of the SKU to form an input sequence for embedding processing; performing multi-level mapping processing on the input sequence to generate several mapping results; performing similarity comparison processing on each mapping result and establishing a set of correspondences between features based on the normalized similarity score; performing stepwise compression processing on the set of correspondences to form continuous representation fragments; performing fragment connection processing on the continuous representation fragments to form a preliminary embedding structure; performing structural translation processing, structural scaling processing, and structural relocation processing on the preliminary embedding structure; and using the processed structure as a set of continuous embedding vectors of SKU located in the cargo similarity manifold space. The SKU continuous embedding vector set is determined as the SKU feature vector. Time series processing operation is performed on the order line time series field set to obtain the order time series feature vector. Feature extraction operation is performed on the spatial resource data to obtain the storage space feature vector and the warehouse capacity constraint vector. Numerical encoding and aggregation operation is performed on the product association field set to generate the product association feature vector. Finally, turnover feature extraction operation is performed on the turnover statistics field set to generate the turnover feature vector. The SKU feature vector, order time sequence feature vector, storage space feature vector, warehouse capacity constraint vector, product association feature vector, and turnover feature vector are combined to generate a set of state elements for the warehouse storage location allocation environment.

[0023] In this embodiment, obtaining the state representation vector specifically includes: Perform missing term imputation and numerical range correction on each vector in the state element set to generate an coded input set; Dimension alignment is performed on each vector in the coded input set to generate a set of element coding vectors of uniform length. The dimension alignment process includes comparing the length of each vector based on a uniform target dimension. When the vector length is greater than the uniform target dimension, truncation is performed. When the vector length is less than the uniform target dimension, zero-padding expansion is performed. The zero-padding expanded vectors are then arranged in the original element order. Normalization and scaling processes are performed on the feature encoding vector set to generate a normalized encoding vector set. The set of normalized encoding vectors is subjected to feature-level concatenation to form a concatenated encoding vector, and the concatenated encoding vector is subjected to interactive encoding to generate a multidimensional interactive encoding vector. The interactive encoding process includes performing correlation measurement between different normalized encoding vectors and performing feature rearrangement and feature aggregation accordingly. A fusion weight generation process is performed on the multidimensional interactive coding vector to obtain a set of element fusion weights. Based on the set of element fusion weights, a weighted combination process is performed on the set of normalized coding vectors to generate a multidimensional warehousing environment embedding representation vector. The multidimensional warehousing environment embedding representation vector is then determined as the state representation vector. The fusion weight generation process includes: performing numerical aggregation processing on the encoded components corresponding to each state element in the multidimensional interactive coding vector to obtain an aggregated numerical set corresponding to each state element; performing a comparison processing on the numerical distribution of each state element based on the aggregated numerical set to generate a set of element contribution values ​​reflecting the degree of difference in numerical values ​​of different state elements; performing correlation measurement processing on the encoded components corresponding to different state elements in the multidimensional interactive coding vector pairwise to generate a set of inter-element correlation strength; combining the set of element contribution values ​​and the set of inter-element correlation strength item by item to form an unnormalized set of element weight values; performing interval mapping processing and total amount constraint processing on the unnormalized set of element weight values ​​to ensure that the weight values ​​of each element meet the uniform scale requirement, and generating a set of element fusion weights that corresponds one-to-one with each state element.

[0024] In this embodiment, the generation of the dynamically executable action set specifically includes: Based on the correspondence between the current pending warehousing operations and the set of status elements, the pending put-away objects are determined, and based on the set of status elements and spatial resource data, the availability of the warehousing locations is determined to generate a set of candidate storage locations corresponding to the put-away objects. The put-away objects include SKUs and order lines. The availability determination process includes: performing an occupancy status determination on the current occupancy status data of the storage locations in the spatial resource data, and filtering out storage locations that are already occupied; performing a capacity matching determination on the storage location capacity data and the SKU volume or quantity requirements corresponding to the shelving objects, and filtering out storage locations that do not meet the capacity requirements; performing a location relationship parsing process on the warehouse topology data, determining the topological connectivity between the warehouse area to which the storage location belongs and the target operation area corresponding to the warehousing operation task to be performed, and filtering out storage locations that do not have a connectivity relationship; performing a path analysis process on the accessibility data of the operation channel, determining whether there is a complete passage path from the operation entrance to the target storage location, and filtering out storage locations that do not have a complete passage path; and summarizing the storage locations that have passed all the above determination processes to generate a candidate storage location set corresponding to the shelving objects to be processed. Based on the candidate location set and spatial resource data, an initial action set is constructed, including single location allocation action, location relocation action, and combined action. The construction of the initial action set includes: based on the candidate storage location set, determining the target storage location identifier for each item to be put into storage, and constructing a single storage location allocation action for each target storage location identifier, including the storage object identifier, the target storage location identifier, and the inbound path identifier; based on the current storage location occupancy status data in the spatial resource data, determining the set of source storage locations that can participate in relocation from the occupied storage locations, and determining the set of target storage locations that correspond one-to-one with the source storage location set based on the candidate storage location set, and constructing a storage location migration action for each source storage location and the corresponding target storage location, including the source storage location identifier, the target storage location identifier, and the handling path identifier; based on the warehouse topology data and the accessibility data of the operation channel, performing time sequence association and path association processing on multiple single storage location allocation actions and storage location migration actions to generate a combined action including multiple action units and their execution order identifiers; summarizing the single storage location allocation actions, storage location migration actions, and combined actions to form an initial action set corresponding to the candidate storage location set; The time-series association processing includes: performing consistency comparison processing on the put-away object identifier and the storage location identifier involved in the single storage location allocation action and the storage location migration action, determining the action units with resource dependencies, and determining the execution order of the action units based on the resource dependencies, and generating an execution order identifier corresponding to each action unit. The path association processing includes: performing path continuity determination processing on the warehouse path identifier and transportation path identifier corresponding to each action unit based on the accessibility data of the operation channel, determining the path combination that can be executed continuously, and binding the path continuity determination result with the corresponding action unit to generate an action association result including the path identifier; For each action in the initial action set, perform action element parsing processing to obtain the target storage location identifier, source storage location identifier, inbound path identifier, put-away object identifier, and handling path identifier associated with the action; A set of action legality constraint rules is constructed based on spatial resource data. The set of action legality constraint rules includes storage space capacity constraint rules, occupancy status constraint rules, storage area topology connectivity constraint rules, channel accessibility constraint rules, and operation path conflict constraint rules. The initial action set is processed item by item to match the action legality constraint rule set, and actions that are determined not to meet the constraints are removed to generate an executable action set. Based on historical turnover data, storage location access frequency data, and historical strategy evaluation data, a statistical sequence of storage location time windows is extracted, and the statistical sequence of storage location time windows is standardized to obtain a statistical feature sequence of storage locations. A comprehensive scoring process is performed on the statistical feature sequence of the storage location to generate a storage location freezing scoring sequence. The comprehensive scoring process includes performing interval mapping processing on the turnover activity value corresponding to historical turnover data, the access activity value corresponding to storage location access frequency data, and the strategy risk value corresponding to historical strategy evaluation data to obtain normalized turnover activity value, normalized access activity value, and normalized risk value, respectively. Then, a weighted summation processing is performed on the three normalized values ​​to obtain the storage location freezing scoring sequence. Based on the freezing threshold, the freezing scores corresponding to each storage location in the storage location freezing score sequence are compared with the freezing scores. Storage locations that meet the freezing conditions are selected, a dynamic storage location freezing set is generated, and action masking is performed on the storage location-related actions corresponding to the dynamic storage location freezing set to generate a dynamic executable action set that changes over time. The acquisition of the freezing threshold includes: performing statistical interval division processing on the storage location freezing score sequence, extracting statistical values ​​that reflect the distribution characteristics of the storage location freezing score, and performing smoothing processing on the statistical values ​​to generate a freezing threshold used to distinguish between frozen and non-frozen storage locations. The action masking process includes: performing a location association parsing process on each action in the executable action set to identify actions that are associated with locations in the dynamic location freeze set; performing an unexecutable marking process on the identified associated actions and removing them from the executable action set to generate a dynamic executable action set.

[0025] In this embodiment, the generation of the training environment interactive dataset specifically includes: Obtain the state representation vector, the set of dynamic executable actions, and the set of order lines corresponding to the current pending warehousing operation task. Extract the set of turnover statistics indicators based on historical turnover data and the set of order lines. Extract the set of operation path indicators based on operation channel accessibility data and warehouse topology data. Extract the set of warehouse space utilization indicators based on warehouse space capacity data and current warehouse space occupancy status data. Extract the set of goods adjacency indicators based on the set of goods association fields and the put-away object identifier. The system performs index normalization and interval mapping on the set of turnover statistics indicators to generate turnover efficiency reward items; it performs cost mapping and interval mapping on the set of operation path indicators to generate operation path cost penalty items; it performs utilization rate calculation and interval mapping on the set of storage space utilization indicators to generate storage space utilization reward items; and it performs adjacency scoring and interval mapping on the set of goods adjacency indicators to generate goods adjacency reward items. The normalization of the indicators includes accumulating the number of outbound shipments for each order line within the target time window, performing a reciprocal transformation on the corresponding average storage time, and multiplying the accumulated number of outbound shipments with the reciprocal transformation of the average storage time to obtain the original value of the turnover efficiency. The cost mapping includes accumulating the path lengths corresponding to the inbound and transport paths, counting the number of work nodes passed through the path, and accumulating the number of historical congestion occurrences of the path; configuring path cost weights based on work path statistics, and multiplying the path length, number of nodes, and number of congestion occurrences by the corresponding path cost weights and then performing summation to obtain the original path cost value. The utilization rate calculation includes performing a ratio calculation on the actual occupied capacity of the target storage location within the current time window and the maximum available capacity of the storage location to obtain the original value of the storage location utilization rate. The adjacency score includes performing association matching processing on the product identifier corresponding to the shelf object and the product identifiers already stored in the adjacent storage locations, and accumulating the count based on the number of matched product associations to obtain the original adjacency matching value; Reward weights are assigned to the turnover efficiency reward item, the operation path cost penalty item, the warehouse space utilization rate reward item and the goods adjacency reward item respectively, and the reward results of each reward item are subjected to weighted summation to obtain the reward value of the composite reward function. The weighted summation process includes performing product calculation on the value of each reward item and the corresponding reward weight, and performing accumulation on each product result. The configuration of the reward weights includes: extracting historical evaluation indicators corresponding to turnover efficiency reward items, operation path cost penalty items, warehouse space utilization reward items, and goods adjacency reward items based on historical turnover data, operation path statistics data, warehouse space utilization statistics data, and goods association data; performing time window statistical processing on each historical evaluation indicator to obtain statistical results reflecting the changes of each reward item in different time periods; performing interval mapping processing on the changes of each reward item based on the statistical results to generate weight base values ​​to characterize the relative importance of each reward item; performing normalization processing on each weight base value to ensure that each reward weight meets a uniform scale constraint, and determining the normalized result as the reward weight of the corresponding reward item; A deep reinforcement learning training environment is constructed based on state representation vectors, a set of dynamically executable actions, and the reward value of a composite reward function. The correspondence between the state representation vectors, the set of dynamically executable actions, the reward value of the composite reward function, and the action execution results is recorded in the training environment to generate an interactive training environment dataset. The action execution results are obtained by updating the state of spatial resource data after receiving and applying the selected actions, based on the state transition rules pre-established in the deep reinforcement learning training environment.

[0026] In this embodiment, the generation of the cargo location allocation strategy model specifically includes: Based on the interactive dataset of the training environment, determine the state representation vector, the set of dynamically executable actions, the reward value of the composite reward function, and the sample record set of action execution results; Construct a policy network and a value network composed of deep neural networks, and configure the network parameter sets and parameter update step sizes for the policy network and the value network respectively; Acquire historical operation trajectory data and online interaction sampling data, perform merge sampling processing with the sample record set, and generate a training sample sequence including state representation vector, dynamic executable action set, selected action, reward value of composite reward function and action execution result; Extract the updated state representation vector corresponding to the current state representation vector under the selected action in the training sample sequence in chronological order, and perform value estimation processing on the current state representation vector and the updated state representation vector respectively based on the value network to generate the current value estimation result and the next state value estimation result; Based on the reward value of the composite reward function, the current value estimation result, and the next state value estimation result, a target value calculation process is performed to obtain the target value estimation result. The target value calculation process includes performing a discount transformation on the next state value estimation result and accumulating the discounted next state value estimation result with the reward value of the composite reward function. The value error is calculated based on the target value estimation result and the current value estimation result, and gradient update processing is performed on the network parameter set of the value network based on the value error; The strategy loss is calculated based on the value error, the probability distribution of the selected action and the action output by the strategy network. Then, gradient update processing is performed on the network parameter set of the strategy network based on the strategy loss, and the strategy network after parameter update is determined as the location allocation strategy model.

[0027] In this embodiment, the generation of the warehouse execution instruction set specifically includes: In the actual warehouse operation, acquire the objects to be processed and put on the shelves, and determine the corresponding state representation vector and dynamic executable action set based on the objects to be processed and put on the shelves. The state representation vector and the set of dynamic executable actions are input into the storage location allocation strategy model. Based on the storage location allocation strategy model, strategy reasoning is performed, and the target storage location allocation decision is output. The target storage location allocation decision includes the target storage location identifier, the inbound path identifier, the action type identifier, the source storage location identifier, and the handling path identifier. The strategy reasoning includes: performing forward computation processing on the strategy network in the location allocation strategy model based on the state representation vector to generate action score results that correspond one-to-one with the set of dynamic executable actions; aligning the action score results with the set of dynamic executable actions to obtain a set of effective action scores; performing comparison and judgment processing on the set of effective action scores to determine the target action whose score value satisfies the action score maximization selection rule; and extracting the target location identifier, inbound path identifier, action type identifier, source location identifier, and handling path identifier corresponding to the target action based on the action element mapping relationship of the target action in the set of dynamic executable actions, and combining the extracted results to generate a target location allocation decision. Based on the action type identifier, the target storage location allocation decision execution instruction conversion processing is performed to generate put-away instruction, storage location locking instruction and warehouse handling instruction. The put-away instruction includes the identifier of the object to be put away, the identifier of the target storage location, and the identifier of the inbound path. The storage location locking instruction includes the identifier of the target storage location and the locking parameter information. The warehouse handling instruction includes the identifier of the source storage location, the identifier of the target storage location, and the identifier of the handling path. The shelving instructions, location locking instructions, and in-warehouse handling instructions are aggregated to generate a warehouse execution instruction set, which is then sent to the warehouse execution system for location allocation.

[0028] Example 1: To verify the feasibility of this invention in practice, it was applied to the automated storage and retrieval system (AS / RS) environment of a large e-commerce company. This warehousing environment is characterized by a wide variety of goods, frequent order fluctuations, complex warehouse structure, and limited operational aisles. Traditional location allocation methods based on fixed rules or static priorities are difficult to adapt to business changes in a timely manner, easily leading to problems such as high-frequency goods being assigned to remote locations, increased conflicts in internal handling paths, and frequent adjustments to local warehouse areas, directly impacting overall operational efficiency and warehouse stability.

[0029] In this scenario, the system first continuously acquires multi-source business data and spatial resource data from the warehouse management system and equipment control system, including product attributes, inbound and outbound order information, historical turnover records, storage location occupancy status, and warehouse topology relationships, and performs preprocessing and unified modeling according to the method of this invention. Based on this, a set of state elements is constructed, including product characteristics, order time sequence characteristics, storage location spatial characteristics, and product association characteristics, and a state representation vector is generated through unified encoding. Simultaneously, an action set is dynamically constructed based on the current task and warehouse space constraints. Combining historical turnover activity, storage location access frequency, and strategy risk, a dynamic storage location freezing mechanism is introduced to avoid high-frequency adjustments interfering with local areas.

[0030] In actual operation, the system utilizes a constructed composite reward function to comprehensively incorporate factors such as turnover efficiency, operational path cost, storage space utilization, and product adjacency into the training environment. Through continuous interaction, training data is generated to jointly train the policy network and value network. The resulting storage space allocation strategy model is then deployed to the actual operational phase. Each time a shelving task is triggered, it automatically infers and generates a target storage space allocation decision based on the current state, converting it into directly executable shelving, locking, and handling instructions.

[0031] From an application perspective, this method effectively alleviates the uneven utilization of storage locations caused by rigid rules in traditional strategies, allowing high-turnover goods to be more rationally distributed in areas with higher operational efficiency, reducing unnecessary in-warehouse handling and path conflicts. Simultaneously, the dynamic storage location freezing mechanism significantly reduces the impact of frequent relocation on system stability, resulting in a smoother warehousing operation rhythm. During continuous operation, the system can adaptively adjust its decision-making strategies according to business changes, demonstrating higher overall operational coordination and efficiency, validating the practicality and stability of this invention in complex warehousing environments.

[0032] Table 1. Performance Comparison of the Invention and Traditional Cargo Location Allocation Methods

[0033] As can be clearly seen from Table 1, the method of the present invention is superior to the traditional method in many indicators.

[0034] The average decision-making time for listing shows that the traditional rule-driven method has an average decision time of 128 milliseconds, while the method of this invention reduces this time to 86 milliseconds. This improvement mainly stems from the fact that this invention uniformly encodes multi-source business data and spatial resource data in the state representation stage, enabling the strategy model to directly perform action scoring and filtering based on the state representation vector during the inference stage. This avoids the computational overhead caused by matching multiple rules one by one in traditional methods, thereby significantly shortening the decision response time.

[0035] Regarding the average handling path length per order, the traditional method has an average path length of 42.6 meters, while the method of this invention reduces it to 35.8 meters. This difference indicates that the present invention introduces warehouse topological connectivity and operational channel accessibility constraints during the action space construction stage, and comprehensively considers path cost factors during strategy reasoning. This makes the generated location allocation decisions more inclined to choose the solution with shorter paths and smoother passage, thereby effectively reducing unnecessary in-warehouse movement distances.

[0036] Regarding the occurrence rate of in-database movement conflicts, the traditional method has a conflict rate of 6.4%, while the method of this invention reduces it to 4.1%. This improvement is closely related to the introduction of temporal and path association processing in the action construction stage of this invention. By explicitly modeling the resource dependencies and path continuity between actions in the initial action set, the path intersection and resource competition during the concurrent execution of multiple tasks are effectively reduced.

[0037] Regarding the average turnover rate of storage locations, the traditional method yields 1.92 times / day, while the method of this invention improves it to 2.31 times / day. This improvement stems from the introduction of a turnover efficiency reward term into the composite reward function, which continuously reinforces the strategy model's tendency to allocate high-turnover storage locations rationally during training. This, in turn, promotes faster storage and retrieval cycles for high-frequency goods, thereby improving overall storage location utilization efficiency.

[0038] Regarding storage space utilization, the traditional method achieves 81.5%, while the method of this invention improves it to 88.2%. This change indicates that the present invention considers both storage space capacity constraints and utilization indicators in the construction of state elements and the design of reward functions, avoiding the problem of overcrowding or long-term vacancy in some areas, and making the distribution of storage space more balanced.

[0039] Regarding the number of times high-frequency storage locations are repeatedly adjusted, traditional methods average 37 adjustments per day, while the method of this invention reduces this to 21 times. This significant reduction is mainly due to the introduction of a dynamic storage location freezing mechanism. By comprehensively scoring storage location activity and strategy risk and setting a freezing threshold, the frequent rearrangement of the same storage location within a short period of time is effectively suppressed, thus improving the stability of system operation.

[0040] Finally, regarding the policy stability fluctuation coefficient, the traditional method achieves 0.34, while the method of this invention reduces it to 0.21. This result demonstrates that the joint training approach based on the value network and policy network makes the policy update process smoother, maintaining relatively stable decision outputs under business fluctuations and reducing the impact of large policy fluctuations on actual operations.

[0041] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for intelligent allocation of warehouse storage locations based on deep reinforcement learning, characterized in that, Includes the following steps: Acquire multi-source business data and spatial resource data of the warehousing system within the target time period, perform preprocessing, and obtain a standardized warehousing dataset. Construct a set of state elements for the warehouse location allocation environment based on a standardized warehouse dataset; The set of state elements is uniformly encoded, and a multi-dimensional warehousing environment embedding representation vector is constructed and determined as the state representation vector. Based on the set of state elements and spatial resource data, a set of actions for warehouse location allocation is constructed, and a dynamic location freezing mechanism is used to filter the set of actions to generate a set of dynamically executable actions. Construct a composite reward function and combine it with a state representation vector and a set of dynamically executable actions to generate an interactive dataset for the training environment; The value network and policy network are trained based on the interactive dataset of the training environment to generate a cargo location allocation strategy model. In actual warehouse operations, the location allocation strategy model is used to perform strategy reasoning on the objects to be put on the shelves, generate target location allocation decisions, and convert them into a set of warehouse execution instructions.

2. The intelligent warehouse location allocation method based on deep reinforcement learning according to claim 1, characterized in that, The multi-source business data includes SKU attribute data, inbound and outbound order data, historical turnover data, and product association data. The spatial resource data includes warehouse topology data, storage location capacity data, current storage location occupancy status data, and operational accessibility data. The preprocessing includes data cleaning, alignment, and numerical processing.

3. The intelligent warehouse location allocation method based on deep reinforcement learning according to claim 1, characterized in that, The construction of the state element set specifically includes: Extract the SKU basic attribute field set, order line time series field set, turnover statistics field set, and product association relationship field set from the multi-source business data of the standardized warehousing dataset; Perform feature filtering and numerical mapping operations on the SKU basic attribute field set to generate an initial SKU feature representation set, and construct a cargo similarity manifold space based on the initial SKU feature representation set; In the cargo similarity manifold space, a distance measurement operation is performed on the initial feature representations of any two SKUs, and the distance result is mapped to the preset maximum distance benchmark to generate a normalized distance result. Then, a normalized similarity score is generated by subtracting the normalized distance result from one. Based on the normalized similarity score, a continuous embedding learning operation is performed on the initial feature representation of SKU to obtain a set of continuous embedding vectors of SKU located in the cargo similarity manifold space; The SKU continuous embedding vector set is determined as the SKU feature vector. Time series processing operation is performed on the order line time series field set to obtain the order time series feature vector. Feature extraction operation is performed on the spatial resource data to obtain the storage space feature vector and the warehouse capacity constraint vector. Numerical encoding and aggregation operation is performed on the product association field set to generate the product association feature vector. Finally, turnover feature extraction operation is performed on the turnover statistics field set to generate the turnover feature vector. The SKU feature vector, order time sequence feature vector, storage space feature vector, warehouse capacity constraint vector, product association feature vector, and turnover feature vector are combined to generate a set of state elements for the warehouse storage location allocation environment.

4. The intelligent warehouse location allocation method based on deep reinforcement learning according to claim 1, characterized in that, The process of obtaining the state representation vector specifically includes: Perform missing term imputation and numerical range correction on each vector in the state element set to generate an coded input set; Dimension alignment is performed on each vector in the coded input set to generate a set of feature coded vectors of uniform length; Normalization and scaling processes are performed on the feature encoding vector set to generate a normalized encoding vector set. The normalized encoding vector set is subjected to element-level concatenation processing to form a concatenated encoding vector, and the concatenated encoding vector is subjected to interactive encoding processing to generate a multidimensional interactive encoding vector; A fusion weight generation process is performed on the multidimensional interactive coding vector to obtain a set of element fusion weights. Based on the set of element fusion weights, a weighted combination process is performed on the set of normalized coding vectors to generate a multidimensional warehousing environment embedding representation vector. The multidimensional warehousing environment embedding representation vector is then determined as the state representation vector.

5. The intelligent warehouse location allocation method based on deep reinforcement learning according to claim 1, characterized in that, The generation of the dynamically executable action set specifically includes: Based on the correspondence between the current pending warehousing operations and the set of status elements, the pending put-away objects are determined, and based on the set of status elements and spatial resource data, the availability of the warehousing locations is determined to generate a set of candidate storage locations corresponding to the put-away objects. The put-away objects include SKUs and order lines. Based on the candidate location set and spatial resource data, an initial action set is constructed, including single location allocation action, location relocation action, and combined action. For each action in the initial action set, perform action element parsing processing to obtain the target storage location identifier, source storage location identifier, inbound path identifier, put-away object identifier, and handling path identifier associated with the action; A set of action legality constraint rules is constructed based on spatial resource data. The set of action legality constraint rules includes storage space capacity constraint rules, occupancy status constraint rules, storage area topology connectivity constraint rules, channel accessibility constraint rules, and operation path conflict constraint rules. The initial action set is processed item by item to match the action legality constraint rule set, and actions that are determined not to meet the constraints are removed to generate an executable action set. Based on historical turnover data, storage location access frequency data, and historical strategy evaluation data, a statistical sequence of storage location time windows is extracted, and the statistical sequence of storage location time windows is standardized to obtain a statistical feature sequence of storage locations. A comprehensive scoring process is performed on the statistical characteristic sequence of cargo locations to generate a cargo location freezing score sequence; Based on the freezing threshold, the freezing scores corresponding to each storage location in the storage location freezing score sequence are compared and processed to select storage locations whose freezing scores meet the freezing conditions. A dynamic storage location freezing set is generated, and action masking processing is performed on the storage location-related actions corresponding to the dynamic storage location freezing set to generate a dynamic executable action set that changes over time.

6. The intelligent warehouse location allocation method based on deep reinforcement learning according to claim 1, characterized in that, The generation of the training environment interactive dataset specifically includes: Obtain the state representation vector, the set of dynamic executable actions, and the set of order lines corresponding to the current pending warehousing operation task. Extract the set of turnover statistics indicators based on historical turnover data and the set of order lines. Extract the set of operation path indicators based on operation channel accessibility data and warehouse topology data. Extract the set of warehouse space utilization indicators based on warehouse space capacity data and current warehouse space occupancy status data. Extract the set of goods adjacency indicators based on the set of goods association fields and the put-away object identifier. The system performs index normalization and interval mapping on the set of turnover statistics indicators to generate turnover efficiency reward items; it performs cost mapping and interval mapping on the set of operation path indicators to generate operation path cost penalty items; it performs utilization rate calculation and interval mapping on the set of storage space utilization indicators to generate storage space utilization reward items; and it performs adjacency scoring and interval mapping on the set of goods adjacency indicators to generate goods adjacency reward items. Reward weights are assigned to the turnover efficiency reward item, the operation path cost penalty item, the warehouse space utilization rate reward item and the goods adjacency reward item, and the reward results of each reward are weighted and summed to obtain the reward value of the composite reward function. A deep reinforcement learning training environment is constructed based on the state representation vector, the set of dynamically executable actions, and the reward value of the composite reward function. The correspondence between the state representation vector, the set of dynamically executable actions, the reward value of the composite reward function, and the action execution results is recorded in the training environment to generate an interactive training environment dataset.

7. The intelligent warehouse location allocation method based on deep reinforcement learning according to claim 1, characterized in that, The generation of the cargo location allocation strategy model specifically includes: Based on the interactive dataset of the training environment, determine the state representation vector, the set of dynamically executable actions, the reward value of the composite reward function, and the sample record set of action execution results; Construct a policy network and a value network composed of deep neural networks, and configure the network parameter sets and parameter update step sizes for the policy network and the value network respectively; Acquire historical operation trajectory data and online interaction sampling data, perform merge sampling processing with the sample record set, and generate a training sample sequence including state representation vector, dynamic executable action set, selected action, reward value of composite reward function and action execution result; The updated state representation vector corresponding to the current state representation vector under the selected action is extracted from the training sample sequence in chronological order. Value estimation is then performed on both the current and updated state representation vectors based on the value network, generating the current value estimation result and the next state value estimation result. ； Based on the reward value of the composite reward function, the current value estimation result, and the next state value estimation result, the target value calculation process is performed to obtain the target value estimation result; The value error is calculated based on the target value estimation result and the current value estimation result, and gradient update processing is performed on the network parameter set of the value network based on the value error; The strategy loss is calculated based on the value error, the probability distribution of the selected action and the action output by the strategy network. Then, gradient update processing is performed on the network parameter set of the strategy network based on the strategy loss, and the strategy network after parameter update is determined as the location allocation strategy model.

8. The intelligent warehouse location allocation method based on deep reinforcement learning according to claim 1, characterized in that, The generation of the warehouse execution instruction set specifically includes: In the actual warehouse operation, acquire the objects to be processed and put on the shelves, and determine the corresponding state representation vector and dynamic executable action set based on the objects to be processed and put on the shelves. The state representation vector and the set of dynamic executable actions are input into the storage location allocation strategy model. Based on the storage location allocation strategy model, strategy reasoning is performed, and the target storage location allocation decision is output. The target storage location allocation decision includes the target storage location identifier, the inbound path identifier, the action type identifier, the source storage location identifier, and the handling path identifier. Based on the action type identifier, the target storage location allocation decision execution instruction conversion processing is performed to generate shelving instructions, storage location locking instructions and in-warehouse handling instructions; The shelving instructions, location locking instructions, and in-warehouse handling instructions are aggregated to generate a warehouse execution instruction set, which is then sent to the warehouse execution system for location allocation.

Images