Warehousing operation online cooperative sorting optimization method based on reinforcement learning

Abstract

The invention relates to an online collaborative sorting optimization method for warehousing operation based on reinforcement learning. Comprising the following steps: modeling a storage environment, generating a dynamic environment state diagram and outputting a plurality of dynamic node feature matrixes; inputting the dynamic node feature matrix into a preset cooperative sorting decision model; based on a serialized sub-decision and action mask mechanism, the collaborative sorting decision model outputs a joint action instruction used for assigning a triple task; training the cooperative sorting decision model, and outputting a trained cooperative sorting strategy model under a preset cost weight; performing multi-target cost weight outer layer optimization, and iteratively generating a group of optimal decision scheme set; and selecting a target strategy from the optimal decision scheme set based on the weighted demand of the multi-dimensional operation index to dynamically adjust the currently operated cooperative selection strategy model. The method can be applied to a real storage scene with random arrival of orders and complex and changeable states; the order fulfillment speed, the resource utilization rate and the system throughput can be obviously improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent logistics technology, and more specifically to an online collaborative picking optimization method for warehouse operations based on reinforcement learning. Background Technology

[0002] With the rapid development of e-commerce and on-demand delivery services, the number and volatility of orders in the warehousing process continue to increase. As the core function of warehouses, order picking accounts for a growing proportion of total operating costs year by year. Picking operations directly affect the efficiency and cost of warehousing systems and are a core issue in the research of intelligent logistics systems.

[0003] Currently, most warehouses use traditional "person-to-goods" order picking systems, where pickers move within the warehouse, retrieving items from storage locations and transporting them to the packing station. These warehouses mostly employ low-level picking, storing inventory units on low shelves for manual picking. More advanced warehouses utilize Robot Mobile Fulfillment Systems (RMFS), where mobile robots (AMRs) transport items from shelves to picking stations, where manual pickers then retrieve the required order items.

[0004] Regardless of which method is used, when faced with increasingly complex order structures and higher timeliness requirements in warehousing and logistics systems, their "serial handover" model (e.g., robots only transport goods to a transfer point, and pickers then process them) fails to fully utilize the flexible movement capabilities of pickers and AMRs within the warehouse. Furthermore, most systems are offline, relying on heuristic task allocation to pickers or AMRs after knowing all orders, lacking overall optimization for collaborative scheduling between entities within the warehouse. In dynamic operations with significant differences in order priorities and time windows, and high timeliness requirements, the absence of a unified collaborative scheduling strategy makes it difficult to achieve optimal performance in terms of service quality and resource utilization costs. Summary of the Invention

[0005] The purpose of this invention is to provide an online collaborative picking optimization method for warehouse operations based on reinforcement learning. This method can learn and optimize strategies autonomously, and is suitable for real-world warehouse scenarios where orders arrive randomly and their states are complex and changeable. It can significantly improve order fulfillment speed, resource utilization, and system throughput.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] In a first aspect, the present invention provides an online collaborative picking optimization method for warehouse operations based on reinforcement learning, comprising:

[0008] S01, acquire real-time operation data of the warehouse, model the warehouse environment, generate a dynamic environment state diagram, and output multiple dynamic node feature matrices based on the dynamic environment state diagram;

[0009] S02, the dynamic node feature matrix is ​​input into a preset collaborative picking decision model, which is constructed based on a graph neural network and a participant-evaluator framework; based on a serialized sub-decision and action masking mechanism, the collaborative picking decision model outputs a joint action instruction for assigning triplet tasks;

[0010] S03, The collaborative picking decision model is trained using the near-end strategy optimization algorithm, and the trained collaborative picking strategy model is output under a preset cost weight.

[0011] S04, perform multi-objective cost weight outer layer optimization, iteratively generate a set of optimal decision schemes; based on the weighted requirements of multi-dimensional operational indicators, select target strategies from the set of optimal decision schemes to dynamically adjust the currently running collaborative picking strategy model.

[0012] As a preferred embodiment of the present invention, the process of generating the dynamic environment state diagram in step S01 is specifically as follows:

[0013] A discretized two-dimensional grid undirected graph is constructed based on the physical space of the warehouse, where each grid cell is defined as a node, and undirected edges are established for adjacent nodes based on Manhattan distance to form a fixed topology of the dynamic environment state graph.

[0014] As a preferred embodiment of the present invention, the dynamic node feature matrix includes, but is not limited to, node type, number of orders to be picked, order remaining time window, occupancy and idle status of pickers and AMRs.

[0015] As a preferred embodiment of the present invention, in step S02, the graph neural network is constructed based on a multi-layer graph isomorphic network encoder and is used to perform message passing and feature aggregation on the dynamic environment state graph to generate node embeddings.

[0016] As a preferred embodiment of the present invention, in step S02, the execution step of the serialization sub-decision is specifically as follows:

[0017] Select a target picker from the node where the idle picker is located based on the action mask;

[0018] Select an order node as the destination for the target picker;

[0019] Match an idle AMR to the order node.

[0020] As a preferred embodiment of the present invention, step S03, when training the cooperative picking decision model using the proximal strategy optimization algorithm, further includes the following step:

[0021] The generalized advantage estimation algorithm is used to calculate the advantage function at each time step, and the state, action, and reward sequences are collected in a multi-environment parallel simulation.

[0022] As a preferred embodiment of the present invention, in step S04, the multi-objective cost weight outer layer optimization is implemented based on the non-dominated sorting genetic algorithm. The cost preference vector is used as the optimization variable, and the Pareto optimal frontier is generated through non-dominated sorting, crowding calculation and genetic operation iteration to form the optimal decision scheme set.

[0023] As a preferred embodiment of the present invention, the specific steps for selecting a target strategy from the set of optimal decision schemes to dynamically adjust the currently running collaborative picking strategy model are as follows:

[0024] Establish a mapping relationship between the optimal decision set and the business load level, and automatically select the time-priority strategy during peak operation periods to adjust the currently running collaborative picking strategy model in order to reduce the order delinquency rate;

[0025] During periods of stable operation, an energy efficiency-first strategy is automatically selected to adjust the currently operating collaborative picking strategy model, thereby shortening the total path of physical objects.

[0026] In a second aspect, the present invention also provides an electronic device, including a processor and a memory;

[0027] The processor is connected to the memory;

[0028] Memory, used to store executable program code;

[0029] The processor reads the executable program code stored in memory and runs the program corresponding to the executable program code to execute the steps of the above-mentioned reinforcement learning-based online collaborative picking optimization method for warehouse operations.

[0030] Thirdly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described reinforcement learning-based online collaborative picking optimization method for warehouse operations.

[0031] In summary, the present invention has the following beneficial effects:

[0032] This invention uses graph neural networks to uniformly model warehouse grid topology, which can simultaneously capture local neighborhood constraints and global spatial structure information, significantly enhancing the ability to express complex collaborative environments with multiple pickers and multiple AMRs. The method of this invention can autonomously learn long-term collaborative strategies and perceive the impact of current decisions on the future, thereby effectively reducing distance costs while ensuring order timeliness. It is superior to traditional rule-based or "serial handover" methods and is more suitable for real-world warehousing scenarios where orders arrive randomly and have high-dimensional and variable states.

[0033] This invention treats cost weight as an outer optimization variable and introduces a non-dominated sorting genetic algorithm to perform multi-objective evolutionary search in the cost preference space. This automatically generates a set of optimal decision schemes that complement each other between order timeliness and travel cost. This allows the method to flexibly switch the optimal strategy according to real-time business needs (such as focusing on reducing order delinquency rates during peak periods and focusing on reducing total travel distances during stable periods). It provides decision-makers with customized scheduling schemes that cover different operational preferences and demonstrates excellent performance in terms of service quality and resource usage costs. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a flowchart of the method of the present invention;

[0036] Figure 2 This is a feature diagram of the grid node in an embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram of the GNN encoder operation in an embodiment of the present invention;

[0038] Figure 4 This is a flowchart illustrating the training process of the collaborative picking decision model in this embodiment of the invention.

[0039] Figure 5 This is the reward convergence graph in an embodiment of the present invention. Detailed Implementation

[0040] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed merely to enable those skilled in the art to better understand and implement the subject matter described herein, and are not intended to limit the scope, applicability, or examples set forth in the claims. The function and arrangement of the elements discussed may be changed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the various examples. For example, the described methods may be performed in a different order than described, and steps may be added, omitted, or combined. Furthermore, features described in some examples may be combined in other examples.

[0041] The following is in conjunction with the appendix Figure 1 The technical solutions in the embodiments of the present invention will be described in detail.

[0042] This embodiment is implemented in a rectangular warehouse, whose workspace is discretized into a two-dimensional grid. Pickers and AMRs (Autonomous Mobile Robots) move within this grid environment and perform order picking tasks. Orders arrive randomly online, and each order has specific time window requirements; the core objective of the system is to coordinate the scheduling of pickers and AMRs to minimize the total order cost (including holding costs and deferred costs) and the total path cost of physical movement.

[0043] Environmental modeling and state feature extraction are performed. This step aims to transform the physical warehouse environment into a computer-processable state representation.

[0044] First, based on the actual layout of the warehouse, a discrete two-dimensional grid map is constructed as the topological structure of the environment, such as... Figure 2 As shown, each grid cell is considered a node. If two grid cells are adjacent in the east-west or north-south direction (Manhattan distance less than 1), then an undirected edge is created between them. This forms a fixed undirected graph structure. Then, at each discrete time step t, dynamic characteristics are assigned to each node based on the real-time state of the system, and a node feature matrix is ​​constructed based on the current system state. .

[0045] Specifically, first, node features are constructed, including node type (e.g., shelf area, picking station, aisle), number of orders to be picked, and remaining time window for orders, as well as the occupancy and idle status of pickers and AMRs, etc.; then, the nodes are calculated. With nodes Find the shortest path distance between them to obtain the global distance matrix. Then, construct the node feature matrix. .

[0046] Input the dynamic node feature matrix into the preset collaborative picking decision model.

[0047] The specific method for constructing the collaborative picking decision model is as follows:

[0048] The objective function is to minimize order timeliness cost and minimize physical path cost. A collaborative picking decision model is established, in which minimizing order timeliness costs includes minimizing order delay costs and holding costs.

[0049] Meanwhile, the model is constrained by the following assumptions: each order can only be picked by one picker and one AMR; the aisle width can accommodate the two types of entities to pass through in parallel without causing congestion; each picker and AMR can only match one order at a time; there are no priority restrictions for different orders; after each matching is completed, no rematching is performed until the task is completed; the speed of the two types of entities is constant during transportation; after each entity completes the current task, it can immediately start the next task from the current position.

[0050] Next, a graph neural network encoder is used to encode the environment state graph. This embodiment uses a graph isomorphic network to extract features from the state graph, such as... Figure 3 As shown, the encoder contains a multi-layer GIN convolutional structure for message passing and feature aggregation between nodes.

[0051] In the In the layer, each node's express Updated to the following formula: ; in For nodes The neighborhood group, To provide learnable bias coefficients, since in this embodiment a single node needs to simultaneously encode information about the states of all related nodes, if the hidden dimension is small, information compression can easily occur during information propagation and aggregation using the graph neural network, leading to policy degradation. Among the commonly used MLP (Multilayer Perceptron) hidden dimensions of 32, 64, and 128, to ensure response time and prevent overfitting, the hidden dimension of each MLP layer is set to 64. After stacking multiple layers, the outputs of each layer are concatenated with the initial features to obtain the final node embedding. ;

[0052] Since a 3-layer GINConv can obtain a sufficient receptive field to ensure training efficiency, a 3-layer GINConv is used as the default configuration in this embodiment. Those skilled in the art can adjust the number of layers and dimensions according to actual needs.

[0053] Through the graph convolution process described above, each node is embedded... It not only includes the local attributes of the node, but also the sum of information of its neighborhood, enabling the subsequent policy network to perceive complex structural information such as the spatial distribution of orders and the relative positions of pickers and AMRs.

[0054] Then, the collaborative picking decision model is transformed into a Markov decision process, whose state space has been determined by the GNN encoder (graph neural network encoder). The temporal progression causes the feature node matrix to be updated, and the state is updated accordingly.

[0055] The action space is determined by the Actor-Critic strategy structure. The system contains N decision-making entities, including the picker and the AMR. Each decision-making body at any time The action is recorded as: Within the same time step, all idle decision-makers make decisions in sequence. After each triplet (select order, select picker, select AMR) is completed, it is occupied on the spot and the mask is updated. The allocation of the next idle decision-maker continues. If only one of the two parties is idle, the allocation is made first and then the allocation continues after the other party becomes idle.

[0056] Picker actions include: selecting service order tasks and moving between accessible locations.

[0057] Actions for AMR include: selecting the next order task, selecting the next target location, and moving between reachable locations.

[0058] The system's joint actions are From the perspective of centralized training, joint actions are regarded as a single action in an MDP (Markov Decision Process), corresponding to a centralized policy.

[0059] The combined actions include: selecting an available picker, selecting the order location node, and selecting an AMR.

[0060] First, select the picker node and embed it into each node. The initial score logit is obtained through an Actor network (such as an MLP): Logit, as a reward given to nodes by the network, increases the likelihood of selecting the picker at that node, but it is not a probability. Then, an action mask is used to minimize the scores of nodes not occupied by pickers and nodes where pickers are not idle (e.g., negative infinity), thus filtering out illegal selections. A softmax function is applied to the remaining legal nodes to transform them into a probability distribution, which is then used to select the target picker. ; in, This represents the set of picker nodes that can be selected in the current state.

[0061] With this distribution, this embodiment samples a node during training, and in actual deployment, the node with the highest probability can be directly selected as the picker's position for the current order.

[0062] Next, order selection involves choosing the picker's target node from all candidate target nodes (nodes containing all pending orders, or nodes containing orders waiting to be published). This is done using a pre-calculated distance matrix. Extract the distance from the current picker's location node to each node. Then for each node Embed the node This distance is concatenated to form a new feature vector. This new feature vector already contains both local environmental information and the factor of "how far away from the current picker". Finally, the original score logit of the destination is obtained through an MLP. Similarly, action masks are used to block nodes that are not allowed as destinations (unreachable locations, locations without orders), and a softmax function is applied only to the set of legal destination nodes, transforming it into the probability of the picker going to each order node.

[0063] Finally, in the AMR selection process, the same principle applies as with pickers. After selecting a picker and an order, the distance from the current AMR node to the order node is extracted from the distance matrix. Then, these distances are concatenated together to form a new feature vector. The selection probability is then determined using the original score logit, masking, and softmax functions. If there are no available pickers, the selection of AMRs and matching of order nodes will proceed first.

[0064] The final combined action of the Actor's output can be denoted as: If there are multiple idle pickers or AMRs within the same time step, this Actor action is called repeatedly. Each time, a picker, an order destination, and an AMR are selected based on the current state. The environment state is updated, the mask of the assigned entity nodes is changed, and the above steps are repeated for the remaining entities until there are no idle entities or assigned orders.

[0065] Critic and Actor share the same GNN encoder, first obtaining the embedding of each node through 3 layers of GINConv. Based on this, Critic needs to summarize the state of the entire graph into a single value. It employs a node-layer MLP with global pooling structure, embedding each node... Mapping to scalars via MLP This is equivalent to the network assigning a local score to each node, representing the urgency of the current order at that node, the length of the path, and how much it will affect future rewards.

[0066] Take the average of the scalar values ​​of all nodes. This yields the current global value estimate, which serves as the reward function for the current state of the warehouse.

[0067] In the reinforcement learning framework, weighted scalarization of the objective function transforms two objective rewards into a single scalar objective, which is the reward function in the MDP model. At time [time value missing]... The order instant cost is defined as: ;

[0068] That is, when an order is publicly available but not yet completed, a new time step is generated every time step elapsed. Holding costs reflect the costs of occupying inventory and picking resources. When an order is in a delayed state, each time step incurs these costs. Delay costs are used to penalize orders that are delayed in delivery. Once an order is completed, it is removed from the system and no further costs are incurred.

[0069] At any moment The path cost is defined as: ;

[0070] That is, at time At that time, the total distance traveled by all moving entities is summed to characterize the overall travel distance cost of the system.

[0071] In the DRL framework, The three parameters are converted into three adjustable weights, and have ;

[0072] The three adjustable weights correspond to the relative cost per unit of holding, the relative cost per unit of delay, and the relative cost per unit of path, respectively.

[0073] Introduction As a moment Given the sum of the number of orders held, the sum of the number of orders delayed, and the sum of the path distances, the immediate cost is:

[0074] ;

[0075] The immediate reward used in DRL is: ;

[0076] Then, the collaborative picking decision model is trained using a proximal strategy optimization algorithm, and the trained collaborative picking strategy model is output under a preset cost weight.

[0077] Given a set of cost preference vectors (with preset cost weights), the Proximal Policy Optimization (PPO) algorithm is used for training. The current policy is run in parallel across multiple environment instances to collect experience sequences of state, action, reward, and next state.

[0078] Specifically, such as Figure 4 As shown, firstly under the current strategy Several episodes were run using several parallel environments to collect a batch of sample sequences: This includes status joint action Global rewards Joint actions refer to a set of decisions made sequentially by decision-making entities, which are then processed using a discount factor. and Critic's current estimates The discount report and advantage value are calculated at each step. The advantage function uses GAE (Generalized Advantage Estimation). To obtain the advantage value from GAE... Define time difference error: ;

[0079] Its significance lies in the prediction error, the difference between the score reward obtained in the current step and the predicted reward for the next step. If it is greater than 0, the effect of the next step is greater than the predicted effect; if it is less than 0, the effect is less than expected. The error is weighted and accumulated using GAE (Google Actions). ;

[0080] Error weighted cumulative result As a standard, it is used to observe whether the reward of the current action is better than the average over a period of time in the future.

[0081] Using the already calculated This allows for an increase in the probability of actions that offer better rewards and a decrease in the probability of actions that offer rewards below average. However, if each update is too large, training can become unstable. Therefore, PPO proposes a proximal policy optimization approach. Each policy update allows only "limited" changes near the old policy, avoiding large leaps in one step. Specifically, this is achieved by pruning the probability ratio of the new and old policies on the same action, reducing the likelihood of the policy entering local optima. This mechanism can stably and efficiently improve policy performance.

[0082] Training continues until the policy performance converges or a preset number of steps are reached, thus obtaining a collaborative picking policy model trained under that set of cost weights.

[0083] In this example, the discount factor is set to 0.99, the initial learning rate is 0.005, and the remaining parameters use the default PPO parameters, with a fixed weighting ratio. Under the given conditions, its training reward convergence graph is as follows: Figure 5 As shown.

[0084] Finally, multi-objective cost weight outer-layer optimization is performed, iteratively generating a set of optimal decision schemes. Based on the weighted requirements of multi-dimensional operational indicators, target strategies are selected from the set of optimal decision schemes to dynamically adjust the currently running collaborative picking strategy model. This step aims to automatically find the optimal strategy set under different operational preferences (such as time-priority or cost-priority).

[0085] Cost weights significantly influence the strategy's trade-off between "order holding / overdue" and "walking costs." Therefore, this invention does not pre-fix these weights but instead employs the classic NSGA-II framework of outer-layer multi-objective optimization to search for a set of Pareto-optimal weight combinations. Its core steps include non-dominated sorting, crowding calculation, tournament selection, and arithmetic crossover and mutation.

[0086] Cost weight vector As an outer layer optimization variable, the performance of the strategy trained by the inner layer reinforcement learning on two objectives (total order cost f1 and path cost f1) is used as the optimization objective.

[0087] This embodiment provides a possible optimization process, which uses the Non-Dominated Sorting Genetic Algorithm (NSGA-II) for multi-objective optimization.

[0088] First, initialization is performed by randomly generating a set of initial cost-weight vectors (population). Specifically, individual encoding is performed, with each individual being a three-dimensional real-number vector: and with attached attributes The algorithm retains a set of non-dominated candidate weights during multiple generations of evolution, eventually forming an approximate Pareto front. For any individual X, two objectives are obtained through training and evaluation of the inner GNN-PPO layer, and the non-dominated concept is used to rank the multiple objectives.

[0089] For each individual p in the population, record the set of individuals it dominates. It records how many individuals dominate it; initially, all individuals not dominated by any individual are collected to form the first frontier. Set its rank to 0. For each individual in the first frontier, decrease the number of individuals dominated by it by one. If there are any individuals in the set dominated by it, remove the individuals in the frontier with a rank of 0. Individuals that are no longer dominated by any individual can then be added to the next frontier. Repeat the non-dominated sorting process described above until all individuals have been assigned.

[0090] To ensure the diversity and uniformity of solution distribution, this embodiment introduces the concept of crowding distance to measure how crowded an individual is in the target space. By estimating the crowding degree, points with large crowding distances are identified and such individuals are retained first. Then, the next generation of candidate individuals is generated through "selection, crossover, and mutation". However, rank and crowding distance are considered together during the selection process.

[0091] Before generating offspring, parent individuals need to be selected from the current population. NSGA-II uses a binary tournament selection mechanism to first randomly select two individuals from the parent population. When comparing the two, the individual with the higher rank (i.e., the lower rank number) is given priority. If the ranks are the same, the crowding distance is compared. The individual with the greater crowding distance has a higher probability of being selected. By repeating the binary tournament mechanism multiple times, a sufficient number of parents can be selected.

[0092] Crossover mutation: Arithmetic crossover, random sampling of one The offspring's preference vector is The offspring's preferences are a compromise between the parents' preferences. Gaussian mutation, by applying random perturbations to the offspring after crossover, improves their exploratory ability for each dimension. With probability The mutation process involves adding noise to each dimension. If a dimension becomes negative after the mutation, it is brought back to 0, and finally, the process is re-normalized.

[0093] Environmental selection and generational renewal: After obtaining N offspring individuals, there are now a total of 2N individuals (parent and offspring). The current individuals need to be re-ranked non-dominated to obtain a Pareto front ranking solution. Starting from the front with rank=0, if the optimal front is less than N, all individuals from the optimal front are added to the next generation. This continues until the front with rank=1. If adding all individuals to the next generation results in a front greater than N, then selection is made from the current front, prioritizing those with larger crowding distances. When enough individuals reach N, all subsequent fronts and individuals within the current front are discarded. This results in a more evenly distributed new generation.

[0094] After multiple generations of evolution, the population will converge to a set of approximately Pareto optimal solutions, which is a set of strategies (optimal decision sets) that perform well under different cost weight preferences.

[0095] In actual deployment, decision-makers can select the corresponding target strategy from the Pareto optimal frontier based on current operational needs (such as prioritizing timeliness during "peak operating periods" to reduce order delinquency rates, and prioritizing energy efficiency during "stable operating periods" to shorten walking paths), and dynamically adjust the currently running collaborative picking strategy model to achieve flexible operation.

[0096] In addition, the following embodiments also provide an electronic device, including a processor and a memory;

[0097] The processor is connected to the memory;

[0098] Memory, used to store executable program code;

[0099] The processor reads executable program code stored in memory to run a program corresponding to the executable program code, in order to execute the steps of the reinforcement learning-based online collaborative picking optimization method for warehouse operations described in the above embodiments.

[0100] Furthermore, the following embodiments also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the reinforcement learning-based online collaborative picking optimization method for warehouse operations described in the above embodiments.

[0101] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this specification are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0102] Several embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technological improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for optimizing online collaborative picking in warehouse operations based on reinforcement learning, characterized in that, include: S01, acquire real-time operation data of the warehouse, model the warehouse environment, generate a dynamic environment state diagram, and output multiple dynamic node feature matrices based on the dynamic environment state diagram; S02, input the dynamic node feature matrix into a preset collaborative picking decision model, which is constructed based on a graph neural network and a participant-evaluator framework; Based on the serialized sub-decision and action masking mechanism, the cooperative picking decision model outputs joint action instructions for assigning triplet tasks; S03, The collaborative picking decision model is trained using the near-end strategy optimization algorithm, and the trained collaborative picking strategy model is output under a preset cost weight. S04, perform multi-objective cost weight outer layer optimization, iteratively generate a set of optimal decision schemes; based on the weighted requirements of multi-dimensional operational indicators, select target strategies from the set of optimal decision schemes to dynamically adjust the currently running collaborative picking strategy model.

2. The online collaborative picking optimization method for warehouse operations based on reinforcement learning according to claim 1, characterized in that, The process of generating the dynamic environment state diagram in step S01 is as follows: A discretized two-dimensional grid undirected graph is constructed based on the physical space of the warehouse, where each grid cell is defined as a node, and undirected edges are established for adjacent nodes based on Manhattan distance to form a fixed topology of the dynamic environment state graph.

3. The online collaborative picking optimization method for warehouse operations based on reinforcement learning according to claim 2, characterized in that, The dynamic node feature matrix includes, but is not limited to, node type, number of orders to be picked, remaining time window of the order, occupancy and idle status of pickers and AMRs.

4. The online collaborative picking optimization method for warehouse operations based on reinforcement learning according to claim 1, characterized in that, In step S02, the graph neural network is constructed based on a multi-layer graph isomorphic network encoder and is used to perform message passing and feature aggregation on the dynamic environment state graph to generate node embeddings.

5. The online collaborative picking optimization method for warehouse operations based on reinforcement learning according to claim 4, characterized in that, In step S02, the execution steps of the serialization sub-decision are specifically as follows: Select a target picker from the node where the idle picker is located based on the action mask; Select an order node as the destination for the target picker; Match an idle AMR to the order node.

6. The online collaborative picking optimization method for warehouse operations based on reinforcement learning according to claim 5, characterized in that, In step S03, when training the cooperative picking decision model using the proximal policy optimization algorithm, the following steps are also included: The generalized advantage estimation algorithm is used to calculate the advantage function at each time step, and the state, action, and reward sequences are collected in a multi-environment parallel simulation.

7. The online collaborative picking optimization method for warehouse operations based on reinforcement learning according to claim 6, characterized in that, In step S04, the multi-objective cost weight outer layer optimization is implemented based on the non-dominated sorting genetic algorithm. The cost preference vector is used as the optimization variable. The Pareto optimal front is generated iteratively through non-dominated sorting, crowding calculation and genetic operation to form the optimal decision scheme set.

8. The online collaborative picking optimization method for warehouse operations based on reinforcement learning according to claim 1, characterized in that, The specific steps for selecting a target strategy from the set of optimal decision schemes to dynamically adjust the currently running collaborative picking strategy model are as follows: Establish a mapping relationship between the optimal decision set and the business load level, and automatically select the time-priority strategy during peak operation periods to adjust the currently running collaborative picking strategy model in order to reduce the order delinquency rate; During periods of stable operation, an energy efficiency-first strategy is automatically selected to adjust the currently operating collaborative picking strategy model, thereby shortening the total path of physical objects.

9. An electronic device, characterized in that, Including the processor and memory; The processor is connected to the memory; Memory, used to store executable program code; The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, in order to perform the steps of the reinforcement learning-based online collaborative picking optimization method for warehouse operations as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When a computer program is executed by a processor, it implements the steps of the online collaborative picking optimization method for warehouse operations based on reinforcement learning as described in any one of claims 1-8.

Images