An agent-based material handling decision optimization method

Abstract

The present invention relates to an agent-based material handling decision optimization method, and relates to the technical field of material scheduling decision making. The present application constructs a material handling spatiotemporal grid and a warehouse path distribution map to describe the entire process of material handling, providing an environmental state and constraint representation for a reinforcement learning environment; orders are stored in a cache area in a first-in-first-out manner, and an order queue is constructed to place orders from the cache area; based on the material handling spatiotemporal grid, the warehouse path distribution map, and the order queue, the environmental state, action, reward, and constraint for executing reinforcement learning are constructed; based on the proximal strategy optimization reinforcement learning method, an order allocation agent and a material scheduling agent are constructed; the two agents are trained through mutual game based on the environmental state, action, reward, and constraint; order allocation and material handling decisions are made through the trained order allocation agent and material scheduling agent.

Description

Technical Field

[0001] The present invention relates to the technical field of material handling scheduling, and in particular to an agent-based material handling decision optimization method. Background Art

[0002] Modern warehouses utilize automated guided vehicles (AGVs), mobile racks, and order workstations to process order materials. The specific operating model is that materials are assigned to mobile racks at designated storage locations. When an order is assigned to an order workstation, an AGV transports the rack containing the order materials from the storage location to the order workstation. An operator then picks the materials to complete the order, and the AGV then transports the rack back to the storage location.

[0003] Automated warehouse operations present a unique set of challenges stemming from the complexity of planning spatial layouts for automated guided vehicles (AGVs) carrying mobile racks, the diverse material configuration requirements for orders, and the coordination between multiple AGVs. These challenges can be broadly categorized into three key objectives: order allocation, real-time AGV scheduling, and practical limitations of warehouse facilities, such as the maximum number of orders per workstation and the maximum number of AGVs allowed on the road at any one time. While these objectives are interrelated, each introduces distinct sub-problems that must be addressed to achieve optimal warehouse performance. In dynamic warehouse environments, real-time task generation is crucial, as tasks cannot be fully predicted in advance. This unpredictability complicates the planning process.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0005] In order to solve the above technical problems or at least partially solve the above technical problems, the present invention provides an agent-based material handling decision optimization method.

[0006] The present invention provides an agent-based material handling decision optimization method, comprising:

[0007] Training phase: Constructing a material handling spatiotemporal grid and warehouse path distribution map to describe the entire material handling process;

[0008] Order pre-processing is done to adapt to real-world scenarios: Orders are stored in a buffer on a first-in, first-out basis; orders removed from the buffer are placed in a constructed order queue;

[0009] Constructing the environment states, actions, rewards, and constraints for performing reinforcement learning based on the material handling spatiotemporal grid, warehouse path distribution map, and order queue;

[0010] Based on the proximal strategy optimization reinforcement learning method, an order allocation agent and a material scheduling agent are constructed; the two agents are trained through mutual game based on the environment state, action, reward and constraints. When the order allocation agent is trained, the material scheduling agent runs in an evaluation mode. After the number of training steps reaches the set training step, the material scheduling agent is trained in turn and the order allocation agent is evaluated. The two training processes are repeated to ensure that the order allocation agent and the material scheduling agent coordinate the game based on rewards; the order allocation agent is trained to select the best order from the order queue and assign it to the optimal order workbench according to the current environment state, and the material handling agent is trained to select the best automatic guided vehicle from the automatic guided vehicle according to the current environment state to schedule material configuration orders;

[0011] Execution phase: The trained order allocation agent and material scheduling agent make order allocation and material handling decisions.

[0012] Furthermore, the material handling space-time grid includes a discrete time axis and a warehouse facility axis, and the warehouse facility axis includes: automatic guided vehicle charging points, shelf storage locations, and order workbenches; the directed arrows between the nodes in the material handling space-time grid are used to represent the automatic guided vehicle scheduling process, and the projection of the directed arrows on the time axis is consistent with the direction of the time axis. The directed arrows include two situations: one is that the warehouse facilities connected to the head and tail of the directed arrow have not changed, and the automatic guided vehicle is idle during this time; the other is that the warehouse facilities connected to the head and tail of the directed arrow have changed, and the automatic guided vehicle reaches the head node from the tail node during this time.

[0013] Furthermore, the warehouse path distribution map includes the warehouse path distribution and the shortest path distribution corresponding to the decision; the automatic guided vehicle is set to run along the shortest path between the facility corresponding to the tail node and the facility corresponding to the head node. Within the set time, the congestion situation within the time is determined according to the number of path intersections of the shortest path in the warehouse path distribution, and the connection between scheduling and the congestion situation caused by scheduling is established.

[0014] Furthermore, if there are no new orders in the buffer to replenish the order queue, the task with the longest duration in the order queue is copied to maintain the consistency of the order queue length and increase the possibility of the order allocation agent being selected in subsequent steps; to avoid duplicate order allocation, any order or its copy is deleted from the order queue when it is allocated.

[0015] Furthermore, at each time step, the warehouse material handling environment is characterized by the environmental state, which includes:

[0016] The order status of the order queue, where each order status includes: the storage location of the order items on each shelf; the location of the order workbench to which the order is assigned; the Euclidean distance between the storage location and the order workbench, and the timestamp of the order appearing in the order queue;

[0017] AGV status, including: AGV location, AGV availability based on the material handling spatiotemporal grid and estimated completion time for ongoing tasks, and AGV battery level;

[0018] The status of the order workstation includes: the location of the order workstation, the number of orders currently being processed by the order workstation, and the order status of the current order; the order workstation has a maximum order capacity;

[0019] Charging point status, including: charging point location, charging point availability based on the material handling spatiotemporal grid, and the estimated completion time of ongoing charging tasks;

[0020] Shelf status, including: shelf storage location coordinates, shelf current location, and estimated completion time of ongoing tasks on the shelf based on the material handling spatiotemporal grid;

[0021] Path status, including: based on the material handling time and space grid The congestion situation of the warehouse path distribution map, the shortest path situation corresponding to each decision, and the contribution of the shortest path corresponding to each decision to congestion.

[0022] Furthermore, the actions performed in the environment include: selecting an order and a corresponding automated guided vehicle, an automated guided vehicle and a corresponding automated guided vehicle charging point.

[0023] Furthermore, the reward for the action-based material handling process includes a weighted sum of three parts. The first part is calculated based on the time required for the automated guided vehicle to travel from its current position to the order workbench where the order is located; the second part is calculated based on the time interval between the timestamp of the order arriving at the order queue and the start of execution by the automated guided vehicle; and the third part is the constraint violation penalty term, which is determined by the product of the constraint violation logic variable and the constraint violation penalty coefficient.

[0024] Furthermore, the environment state, actions, and rewards must all obey the following constraints:

[0025] Each order is assigned to an order workstation;

[0026] The number of orders processed by each order workstation does not exceed its maximum order capacity;

[0027] The total time for picking materials and the total time for processing shelves must not exceed the set time threshold;

[0028] In the material processing space-time grid In the above equation, shelf quantity conservation has two meanings: first, the shelf quantity flowing into and out of each node is conserved; second, at the same time, a shelf can only be dispatched by one directed arrow;

[0029] The materials allocated to the order do not exceed the total amount of materials stored on all shelves;

[0030] In the warehouse path distribution diagram at any set time, the congestion caused by the shortest paths corresponding to all directed arrows does not exceed the set congestion threshold;

[0031] AGVs are not allowed to go directly from the order workstation to the AGV charging point;

[0032] After the shelf is taken out of the storage location, it is eventually returned to the original storage location;

[0033] Automatic guided vehicle charging meets the battery charging law constraints.

[0034] Furthermore, the order allocation agent and the material scheduling agent respectively convert their current environment states into embeddings through an embedding layer based on a multi-layer perceptron:

[0035] ;

[0036] in, The embedding layer is based on the multi-layer perceptron Based on the current state of the environment of the order allocation agent The generated embedding, The embedding layer is based on the multi-layer perceptron Based on the current environment status of the material scheduling agent Generated embeddings;

[0037] The order allocation agent and the material scheduling agent respectively use the attention mechanism to convert the embedding into attention weights;

[0038] ;

[0039] As above, embed the two 、 Multilayer Perceptron 、 After mapping, the attention weights are obtained by sigmoid activation function activation processing. ;

[0040] The order allocation agent and the material scheduling agent construct context representations for decision-making based on attention weights:

[0041] ;

[0042] As above, embed the two 、 Weight the corresponding attention weights respectively Obtaining contextual representation for decision making ;

[0043] The order allocation agent and the material scheduling agent respectively use multi-layer perceptrons to make decisions based on context representation.

[0044] Furthermore, the training process includes:

[0045] Initialize the policy network parameters and value network parameters of the order allocation agent, and initialize the policy network parameters and value network parameters of the material scheduling agent;

[0046] During the data collection phase, the environment interacts with the actions decided by the decision network in the current training phase to obtain a trajectory dataset containing records of the environment state, actions, rewards, and corresponding action probabilities. The training of the order allocation agent and the material scheduling agent is switched alternately according to the switching episodes.

[0047] The value networks of the order allocation agent and the material scheduling agent use the state of the environment in the collected trajectory dataset to calculate their respective generalized advantage estimates;

[0048] The shear surrogate objective function is constructed using the probability ratio and generalized advantage estimation of the new and old decision networks of the order allocation agent and the material scheduling agent. The policy network is updated in a way that maximizes the shear surrogate objective function. The value network parameters are updated using the mean square error between the reward-based return and the value network.

[0049] The above technical solution provided by the embodiment of the present invention has the following advantages compared with the prior art:

[0050] This application constructs a material handling space-time grid and a warehouse path distribution map to describe the entire process of material handling; decouples the decision space-time and path space, uses a discretized representation of the space-time grid to reduce the complexity of state space-time, and replaces complex fluid mechanics simulation with statistical calculations of path congestion to improve the computational efficiency of the subsequent modeling process.

[0051] Order preprocessing is adapted to real scenarios: Orders are stored in the cache in a first-in-first-out manner, and an order queue is constructed to place orders from the cache; in this application, the FIFO cache is combined with a dynamic replication strategy and a dynamic deduplication strategy to manage the order queue. The FIFO cache ensures that orders are assigned to the order queue in chronological order, and combined with the dynamic replication strategy and the dynamic deduplication strategy, ensures that orders that have existed in the order queue for a long time have a greater chance of being decided and assigned. After being assigned, deduplication is performed to avoid repeated assignment of orders.

[0052] This application uses a proximal policy optimization reinforcement learning approach to construct an order allocation agent and a material scheduling agent. The two agents are trained through mutual game play to ensure coordination between the order allocation agent and the material scheduling agent. The order allocation agent is trained to select the best order from the order queue and assign it to the optimal order workstation based on the current environmental state. This promotes the gradual efficiency of order allocation and material scheduling decisions. This solution effectively solves core issues in warehouse scheduling, such as resource competition and path conflicts, through spatiotemporal coupling modeling and a dual-agent game mechanism. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0055] Figure 1 A flowchart of an agent-based material handling decision optimization method provided in an embodiment of the present invention.

[0056] Figure 2 A schematic diagram of a warehouse path distribution map provided in accordance with an embodiment of the present invention.

[0057] Figure 3 A schematic diagram of a material processing space-time grid provided in accordance with an embodiment of the present invention.

[0058] Figure 4 A schematic diagram of a network architecture provided for an embodiment of the present invention.

[0059] Figure 5 A schematic diagram of an agent-based material handling decision optimization device provided in an embodiment disclosed by the present invention. DETAILED DESCRIPTION

[0060] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0061] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0062] Example 1

[0063] See Figure 1 As shown, an embodiment of the present invention provides an agent-based material handling decision optimization method, comprising:

[0064] Constructing a space-time grid for material handling and warehouse path distribution map To describe the entire process of material handling, it provides a more convenient representation of environmental states, actions, rewards, and constraints for reinforcement learning environments.

[0065] Among them, see Figure 3 As shown, the material processing space-time grid Contains discrete time axes and warehouse facility axes, which include: automatic guided vehicle charging points, shelf storage locations, and order workbenches; setting automatic guided vehicle charging points The number of , shelf storage location The number of , Order Workbench The number of , then the material processing space-time grid The dimension is 、 and The sum of the times multiplied by the number of discrete time points T. The physical processing spatiotemporal network ignores the specific location of the warehouse facilities and the operation path of the automated guided vehicles, and focuses on the automated guided vehicles processing materials, the distribution of materials to shelves, the distribution of orders to order workbenches, and the charging time arrangement of the automated guided vehicles; the directed arrows between the nodes in the material processing spatiotemporal grid are used to represent the automated guided vehicle scheduling process. The projection of the directed arrows on the time axis is consistent with the direction of the time axis. The directed arrows include two cases: one is that the warehouse facilities connected by the head and tail of the directed arrows have not changed, and the automated guided vehicle is idle during this time, such as from the node Pointing to a node The other is that if the warehouse facilities connected by the head and tail of the directed arrow change, the AGV will reach the head node from the tail node within the time, such as node point to .like Figure 3 As shown in Figure 3 The time axis starts at t, and the leftmost directed arrow is from the order workbench (corresponding node ) points to the shelf storage location (corresponding node ), the warehouse facilities connected by the directed arrows change, and during the time period spanned by the leftmost directed arrow, the automated guided vehicle moves from the order workbench to the warehouse. Go to the shelf storage location ; and, the path uses the order workbench and shelf storage locations The second directed arrow from the left is the shortest path between the shelf storage location. Point to AGV charging point , the warehouse facilities connected by the head and tail of the directed arrows change. During the time period spanned by the leftmost directed arrow, the AGV moves from the shelf storage location to the warehouse location. Towards an AGV charging point Thus, the movement of AGVs between warehouse facilities is described by the directed arrows of the material handling space-time grid.

[0066] During the specific implementation process, since the time axis of the material processing space-time grid is a discrete time point, the actual time is rounded up to match the corresponding discrete time point.

[0067] See Figure 2As shown, the warehouse path distribution diagram includes the warehouse path distribution and the distribution of the shortest paths corresponding to the decisions. The automated guided vehicle is set to run along the shortest path between the facilities corresponding to the tail node and the facilities corresponding to the head node. Within a set time, the congestion of the warehouse within that time is determined based on the number of path intersections in the shortest path in the warehouse path distribution. This establishes a connection between the automated guided vehicle's material processing, order allocation to the order workbench, and the automated guided vehicle's charging decision-making behavior and the resulting congestion. The congestion situation is obtained based on the statistics of the intersections of the shortest paths of all automated guided vehicles within the set time, and the material processing spatiotemporal grid is calculated based on the number of shortest path intersections / total number of system intersections. The contribution of corresponding decisions to congestion.

[0068] The decision space-time and path space are decoupled, and the discretization representation of the space-time grid is used to reduce the complexity of the state space-time. The statistical calculation of path congestion is used instead of complex fluid dynamics simulation to improve the optimization calculation efficiency.

[0069] Order preprocessing is done to adapt to real-world scenarios: Orders are stored in a cache on a first-in-first-out basis, and an order queue is constructed to hold orders from the cache.

[0070] During the specific implementation process, if the number of network orders is small and there are no new tasks in the buffer to supplement the order queue, the task with the longest duration in the order queue will be copied to maintain the consistency of the order queue length and increase the possibility of selection by the order allocation agent in subsequent steps. To avoid the allocation of duplicate orders, any order or its copy will be deleted from the order queue when it is allocated. The FIFO buffer area, combined with the dynamic replication strategy and the dynamic deduplication strategy, manages the order queue. The FIFO buffer area ensures that orders are allocated to the order queue in chronological order. Combined with the dynamic replication strategy and the dynamic deduplication strategy, it ensures that orders that have existed in the order queue for a long time have a better chance of being decided and allocated. After being allocated, deduplication is performed to avoid duplicate allocation of orders.

[0071] The environment states, actions, rewards, and constraints for performing reinforcement learning are constructed based on the material handling spatiotemporal grid and warehouse path distribution map.

[0072] At each time step, the warehouse material handling environment is characterized by the environmental state, which includes:

[0073] The order status of the order queue. The content of each order status includes: the storage location of the materials in the order on each shelf; the location of the order workbench to which the order is assigned; the Euclidean distance between the storage location and the order workbench, and the timestamp when the order appears in the order queue.

[0074] AGV status, including: AGV location, AGV availability based on the material handling spatiotemporal grid and estimated completion time for ongoing tasks; AGV battery level.

[0075] The order workbench status includes: the order workbench location, the number of orders currently processed by the order workbench, and the order status of the current order; among them, the order workbench has a maximum order capacity, which limits the order status of its current order to limited.

[0076] Charging point status, including: charging point location, charging point availability based on material handling spatiotemporal grid decisions, and estimated completion time of ongoing charging tasks.

[0077] The shelf status includes the shelf's storage location coordinates, the shelf's current position, and the estimated completion time of the shelf's ongoing tasks based on the material processing spatiotemporal grid. In this application, an AGV carries a shelf from a storage location to an order workstation, and the shelf's current position is either at the storage location or at the location of the AGV carrying it. After the materials on the shelf have been picked and assigned to all required orders at all order workstations, the AGV must return the shelf to its storage location. Therefore, when controlling the AGV to assign orders, it is necessary to monitor the shelf's storage location coordinates and current position to determine the shelf's status to provide sufficient conditions for decision-making. For example, if a given shelf is on an AGV and it is assigning order A to one order workstation, and order B to another order workstation also requires materials from the given shelf, and the AGV has sufficient battery power (which depends on the AGV's status and the shortest path to the decision; if the AGV's status supports the shortest path to the decision, then the AGV has sufficient battery power), it can assign order A, then proceed to the other order workstation to assign order B. After completing order B, it can return the given shelf to its original location.

[0078] Path status, including: based on the material handling time and space grid The congestion of the warehouse path distribution map, the shortest path corresponding to each decision, and the contribution of the shortest path corresponding to each decision to congestion. In the specific implementation process, in the warehouse path distribution map, according to the spatiotemporal grid of each automated guided vehicle material handling The plan in the algorithm gives the shortest path. Within a set timeframe, the congestion level within that timeframe is determined based on the number of intersections across each shortest path in the warehouse path distribution. The contribution of the material scheduling plan corresponding to each directed arrow to congestion is determined by the ratio of the number of intersections across all shortest paths to the total number of intersections across all directed arrows. This directly links decisions to path status.

[0079] This application includes an order allocation agent and a material scheduling agent. The goal of the order allocation agent is to optimize order allocation and thus minimize the time for material processing. The actions of the order allocation agent involve selecting orders from the order queue and assigning them to the order workbench, while the material scheduling agent focuses on allocating automated guided vehicles to schedule materials and controlling automated guided vehicles for charging. The goal is to optimally assign automated guided vehicles to selected orders for material lifting to reduce overall operating costs and delays. Therefore, the actions performed in the environment include selecting orders and corresponding automated guided vehicles, automated guided vehicles and corresponding automated guided vehicle charging points.

[0080] The reward for the action-based material handling process consists of a weighted sum of three parts.

[0081] The first part is calculated based on the time it takes for the AGV to travel from its current location to the order workstation where the order is located. If the AGV is charging, the time required needs to take into account the time it takes to complete charging. If the AGV is performing other tasks, the time it takes to complete the ongoing task needs to be taken into account. If the AGV is not carrying the shelf where the order item is located, and the shelf is in storage, the time it takes to reach the shelf storage location and then to the order workstation needs to be taken into account. If the AGV is not carrying the shelf where the order item is located, and the shelf is not in storage, the time it takes for the shelf to return to its original location and the AGV to reach the shelf storage location need to be taken into account.

[0082] The second part is calculated based on the time interval between the timestamp of the order arriving in the order queue and the start of the AGV’s execution;

[0083] The third part is the constraint violation penalty term, which is determined by the product of whether the constraint is violated and the constraint violation penalty coefficient.

[0084] The environment state, actions, and rewards must all obey the following constraints:

[0085] Each order is assigned to an order workstation.

[0086] The number of orders processed by each order workstation does not exceed its maximum order capacity.

[0087] The total time for picking materials and the total time for processing shelves must not exceed the set time threshold; the time threshold limits the time for staff to select materials and the time for automatic guided vehicles to load shelves, with a maximum upper limit.

[0088] In the material processing space-time grid In the ,shelf quantity conservation,shelf quantity conservation has two meanings. One is the conservation of shelf quantity flowing into and out of each node. The other is that, at the same time, a shelf can only be scheduled by one directed arrow.

[0089] The materials allocated to an order must not exceed the total amount of materials stored on all shelves. This prevents the situation where materials on a shelf are still considered to exist and continue to be dispatched after all the materials on the shelf have been allocated.

[0090] In the warehouse path distribution diagram at any set time, the congestion caused by the shortest paths corresponding to all directed arrows does not exceed the set congestion threshold.

[0091] Taking into account the actual operation situation, the following constraints are added: AGVs are not allowed to go directly from the order workbench to the AGV charging point, because the AGV transports the shelves to the order workbench. If the AGV goes directly from the order workbench to the AGV charging point, it means that the AGV is charging with the shelves, which affects the subsequent order processing of the materials on the shelves and is not allowed; after the shelves are transferred from the storage location, they must eventually be returned to the original storage location.

[0092] The AGV's charging must meet the battery charging constraints, which determine the AGV's state.

[0093] Violation of the constraint determines the third part of the reward, thereby affecting the training process of the network model and enabling the model to have the ability to follow the constraints. Specifically, the constraint violation logical variable includes 1 and 0, 1 corresponds to violation of the constraint, and 0 corresponds to compliance with the constraint; when the constraint is complied with, 0 multiplied by the constraint violation penalty coefficient is 0, that is, when the constraint is not violated, the third part of the reward is 0, when the constraint is violated, 1 multiplied by the constraint violation penalty coefficient is the constraint violation penalty coefficient. The third part of the reward is the constraint violation penalty coefficient corresponding to the type of constraint violated, to achieve punishment, and the constraint violation penalty coefficient is given.

[0094] This application is based on the proximal policy optimization reinforcement learning method to construct an order allocation agent and a material scheduling agent. The two agents are trained through mutual game. The two agents are trained through mutual game based on the environment state, action, reward and constraint. When the order allocation agent is training, the material scheduling agent runs in evaluation mode. After the number of training steps reaches the set training step, the material scheduling agent is trained in turn and the order allocation agent is evaluated. During the entire training process, the two training methods are repeatedly alternated to ensure mutual coordination and game between the order allocation agent and the material scheduling agent. The order allocation agent is trained to select the best order from the order queue and assign it to the optimal order workbench according to the current environment state, and the material processing agent is trained to select the best automatic guided vehicle from the automatic guided vehicle to dispatch material configuration orders according to the current environment state; so as to promote the efficiency of the order allocation and material scheduling decisions made.

[0095] In the specific implementation process, the order allocation agent and the material scheduling agent respectively convert their current environment states into embeddings through the embedding layer based on the multi-layer perceptron:

[0096] ;

[0097] in, The embedding layer is based on the multi-layer perceptron Based on the current state of the environment of the order allocation agent The generated embedding, The embedding layer is based on the multi-layer perceptron Based on the current environment status of the material scheduling agent Generated embeddings.

[0098] The order allocation agent and the material scheduling agent respectively use the attention mechanism to convert the embedding into attention weights;

[0099] ;

[0100] As above, embed the two 、 Multilayer Perceptron 、 After mapping, it is activated by the Sigmoid activation function.

[0101] The order allocation agent and the material scheduling agent construct context representations for decision-making based on attention weights:

[0102] ;

[0103] As above, embed the two 、 Weight the corresponding attention weights respectively Obtaining contextual representation for decision making .

[0104] The order allocation agent and the material scheduling agent respectively use multi-layer perceptrons to make decisions based on context representation.

[0105] The training process includes:

[0106] Initialize the policy network parameters and value network parameters of the order allocation agent and the material scheduling agent, using symmetry-breaking small random numbers for initialization. Define PPO hyperparameters, including discount factor, GAE parameter, policy pruning range, and learning rate. Set the alternation cycle: switch the training agent every 40 episodes. Set the maximum number of training steps: 200 training steps per episode.

[0107] During the data collection phase, a trajectory dataset containing environmental states, actions, rewards, and corresponding action probabilities is obtained through the interaction between the environment and the decision network actions: During the specific implementation process, at each time step t, the environment returns the environmental state; if the order allocation agent is trained, the material scheduling agent is frozen in evaluation mode, and the order allocation agent decision network calculates the order selection probability distribution based on the environmental state characteristics to sample orders, and the material scheduling agent schedules the automatic guided vehicles according to the fixed decision network; if the material scheduling agent is trained, the order allocation agent is frozen in evaluation mode, and the material scheduling agent calculates the automatic guided vehicle selection probability distribution based on the environmental state characteristics to sample automatic guided vehicles, and the material scheduling agent allocates the optimal order according to a fixed strategy; through the interaction between the environment and actions, the rewards and the next moment environmental state are obtained; a trajectory dataset containing environmental states, actions, rewards, and corresponding action probabilities is recorded;

[0108] The value network uses the environment states collected from the trajectory dataset to calculate a generalized advantage estimate. Specifically, when training an order allocation agent, the generalized advantage estimate is calculated based on the updated environment states learned by the value network; when training a material scheduling agent, the generalized advantage estimate is calculated based on the updated environment states learned by the value network.

[0109] The policy network is updated by maximizing the cut-out surrogate objective function using the probability ratio and generalized advantage estimate of the new and old decision networks. The value network parameters are updated using the mean squared error between the reward-based return and the value network.

[0110] If the order allocation agent is trained, its policy network is updated in a way that maximizes its shear surrogate objective function; if the material scheduling agent is trained, its policy network is updated in a way that maximizes its material scheduling agent; if the order allocation agent is trained, the order allocation agent value network is fitted by using the mean square error regression between the reward-based return and the predicted value of the order allocation agent value network; if the material scheduling agent is trained, the material scheduling agent value network is fitted by using the mean square error regression between the reward-based return and the predicted value of the material scheduling agent value network.

[0111] In actual operation, the trained order allocation agent and material scheduling agent make order allocation and material processing decisions.

[0112] Example 2

[0113] See Figure 5 As shown, the present invention provides an agent-based material handling decision optimization device, comprising: at least one processing unit, the processing unit being connected to a storage unit via a bus unit, the storage unit storing a computer program, and when the computer program is executed by the processing unit, implementing the multi-level power optimization allocation and equipment selection method for the energy and power system, comprising:

[0114] Training phase: Constructing a material handling spatiotemporal grid and warehouse path distribution map to describe the entire material handling process;

[0115] Order pre-processing is done to adapt to real-world scenarios: Orders are stored in a buffer on a first-in-first-out basis; orders removed from the buffer are placed in a built order queue;

[0116] Constructing the environment states, actions, rewards, and constraints for performing reinforcement learning based on the material handling spatiotemporal grid, warehouse path distribution map, and order queue;

[0117] Based on the proximal strategy optimization reinforcement learning method, an order allocation agent and a material scheduling agent are constructed; the two agents are trained through mutual game based on the environment state, action, reward and constraint. When the order allocation agent is trained, the material scheduling agent runs in an evaluation mode. After setting the training steps, the material scheduling agent is trained and the order allocation agent is evaluated alternately, and the execution is repeated to ensure that the order allocation agent and the material scheduling agent coordinate the game based on rewards; the order allocation agent is trained to select the best order from the order queue and assign it to the optimal order workbench according to the current environment state, and the material handling agent is trained to select the best automatic guided vehicle from the automatic guided vehicle according to the current environment state to schedule material configuration orders;

[0118] Execution phase: The trained order allocation agent and material scheduling agent make order allocation and material handling decisions.

[0119] Of course, the computer program stored in the storage unit of the intelligent agent-based material handling decision optimization device provided by an embodiment of the present invention is not limited to the method operations described above, and can also execute related operations in the intelligent agent-based material handling decision optimization method provided by any embodiment of the present invention.

[0120] Example 3

[0121] An embodiment of the present invention provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by a processor, the agent-based material handling decision optimization method is implemented, including:

[0122] Training phase: Constructing a material handling spatiotemporal grid and warehouse path distribution map to describe the entire material handling process;

[0123] Order pre-processing is done to adapt to real-world scenarios: Orders are stored in a buffer on a first-in-first-out basis; orders removed from the buffer are placed in a built order queue;

[0124] Constructing the environment states, actions, rewards, and constraints for performing reinforcement learning based on the material handling spatiotemporal grid, warehouse path distribution map, and order queue;

[0125] Based on the proximal strategy optimization reinforcement learning method, an order allocation agent and a material scheduling agent are constructed; the two agents are trained through mutual game based on the environment state, action, reward and constraint. When the order allocation agent is trained, the material scheduling agent runs in an evaluation mode. After setting the training steps, the material scheduling agent is trained and the order allocation agent is evaluated alternately, and the execution is repeated to ensure that the order allocation agent and the material scheduling agent coordinate the game based on rewards; the order allocation agent is trained to select the best order from the order queue and assign it to the optimal order workbench according to the current environment state, and the material handling agent is trained to select the best automatic guided vehicle from the automatic guided vehicle according to the current environment state to schedule material configuration orders;

[0126] Execution phase: The trained order allocation agent and material scheduling agent make order allocation and material handling decisions.

[0127] Of course, the computer-readable storage medium provided by an embodiment of the present invention stores a computer program that is not limited to the method operations described above, but can also execute related operations in an agent-based material handling decision optimization method provided by any embodiment of the present invention.

[0128] In the embodiments provided by the present invention, it should be understood that the disclosed structures and methods can be implemented in other ways. For example, the structural embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, structure or unit, which can be electrical, mechanical or other forms.

[0129] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0130] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0131] The foregoing description is intended only to provide specific embodiments of the present invention, which will enable those skilled in the art to understand and implement the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown herein, but is intended to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. An agent-based material handling decision optimization method, characterized in that: include: Training phase: Construct a material handling spatiotemporal grid and a warehouse path distribution map to describe the entire material handling process. The material handling spatiotemporal grid includes a discrete time axis and a warehouse facility axis. The warehouse facility axis includes: AGV charging points, shelf storage locations, and order workstations. Directed arrows between nodes in the material handling spatiotemporal grid represent the AGV scheduling process. The projection of the directed arrows on the time axis is consistent with the direction of the time axis. Directed arrows include two situations: one is that if the warehouse facilities connected by the head and tail of the directed arrow do not change, the AGV is idle during this time; the other is that if the warehouse facilities connected by the head and tail of the directed arrow change, the AGV will reach the head node from the tail node during this time. The warehouse path distribution map includes the warehouse path distribution and the distribution of the shortest path corresponding to the decision. Order preprocessing is adapted to real-world scenarios: Orders are stored in a buffer on a first-in-first-out basis; orders from the buffer are placed in a constructed order queue. The buffer ensures that orders are assigned to the order queue in chronological order. If the network order volume is small and there are no new order tasks in the buffer to replenish the order queue, the longest-lasting task in the order queue is copied to maintain the consistency of the order queue length and increase the possibility of selection by the order allocation agent in subsequent steps. When any order or its copy is assigned, the order and its copy are deleted from the order queue. Constructing the environment states, actions, rewards, and constraints for performing reinforcement learning based on the material handling spatiotemporal grid, warehouse path distribution map, and order queue; Based on the proximal strategy optimization reinforcement learning method, an order allocation agent and a material scheduling agent are constructed; the two agents are trained through mutual game based on the environment state, action, reward and constraints. When the order allocation agent is trained, the material scheduling agent runs in evaluation mode. After the number of training steps reaches the set training step, the material scheduling agent is trained in turn and the order allocation agent is evaluated. During the entire training process, the two training methods are repeatedly alternated to ensure that the order allocation agent and the material scheduling agent coordinate the game based on rewards; the order allocation agent is trained to select the best order from the order queue and assign it to the optimal order workbench according to the current environment state, and the material handling agent is trained to select the best automatic guided vehicle from the automatic guided vehicle according to the current environment state to schedule material configuration orders; Execution phase: The trained order allocation agent and material scheduling agent make order allocation and material handling decisions.

2. The agent-based material handling decision optimization method according to claim 1, characterized in that: The automated guided vehicle is set to run along the shortest path between the facilities corresponding to the tail node and the facilities corresponding to the head node. Within the set time, the congestion situation within the time is determined according to the number of path intersections of the shortest path in the warehouse path distribution, and the connection between scheduling and the congestion situation caused by scheduling is established.

3. The agent-based material handling decision optimization method according to claim 1, characterized in that: At each time step, the warehouse material handling environment is characterized by the environmental state, which includes: The order status of the order queue, where each order status includes: the storage location of the order items on each shelf; the location of the order workbench to which the order is assigned; the Euclidean distance between the storage location and the order workbench, and the timestamp of the order appearing in the order queue; AGV status, including: AGV location, AGV availability based on the material handling spatiotemporal grid and estimated completion time for ongoing tasks, and AGV battery level; The status of the order workstation includes: the location of the order workstation, the number of orders currently being processed by the order workstation, and the order status of the current order; the order workstation has a maximum order capacity; Charging point status, including: charging point location, charging point availability based on the material handling spatiotemporal grid, and the estimated completion time of ongoing charging tasks; Shelf status, including: shelf storage location coordinates, shelf current location, and estimated completion time of ongoing tasks on the shelf based on the material handling spatiotemporal grid; Path status, including: based on the material handling time and space grid The congestion situation of the warehouse path distribution map, the shortest path situation corresponding to each decision, and the contribution of the shortest path corresponding to each decision to congestion.

4. The agent-based material handling decision optimization method according to claim 1, characterized in that: Actions performed in the environment include: selecting an order and a corresponding automated guided vehicle, an automated guided vehicle and a corresponding automated guided vehicle charging point.

5. The agent-based material handling decision optimization method according to claim 1, characterized in that: The reward for the action-based material handling process consists of a weighted sum of three parts. The first part is calculated based on the time required for the automated guided vehicle to travel from its current location to the order workbench where the order is located; the second part is calculated based on the time interval between the timestamp of the order arriving at the order queue and the start of execution by the automated guided vehicle; the third part is the constraint violation penalty item, which is determined by the product of the constraint violation logic variable and the constraint violation penalty coefficient.

6. The agent-based material handling decision optimization method according to claim 1, characterized in that: The environment state, actions, and rewards must all obey the following constraints: Each order is assigned to an order workstation; The number of orders processed by each order workstation does not exceed its maximum order capacity; The total time for picking materials and the total time for processing shelves must not exceed the set time threshold; In the material processing space-time grid In the above equation, shelf quantity conservation has two meanings: first, the shelf quantity flowing into and out of each node is conserved; second, at the same time, a shelf can only be dispatched by one directed arrow; The materials allocated to the order do not exceed the total amount of materials stored on all shelves; In the warehouse path distribution diagram at any set time, the congestion caused by the shortest paths corresponding to all directed arrows does not exceed the set congestion threshold; AGVs are not allowed to go directly from the order workstation to the AGV charging point; After the shelf is taken out of the storage location, it is eventually returned to the original storage location; Automatic guided vehicle charging meets the battery charging law constraints.

7. The agent-based material handling decision optimization method according to claim 1, characterized in that: The order allocation agent and the material scheduling agent each convert their current environment states into embeddings through an embedding layer based on a multi-layer perceptron: ； in, The embedding layer is based on the multi-layer perceptron Based on the current state of the environment of the order allocation agent The generated embedding, The embedding layer is based on the multi-layer perceptron Based on the current environment status of the material scheduling agent Generated embeddings; The order allocation agent and the material scheduling agent respectively use the attention mechanism to convert the embedding into attention weights; ； As above, embed the two 、 Multilayer Perceptron 、 After mapping, the attention weights are obtained by sigmoid activation function activation processing. ; The order allocation agent and the material scheduling agent construct context representations for decision-making based on attention weights: ； As above, embed the two 、 Weight the corresponding attention weights respectively Obtaining contextual representation for decision making ; The order allocation agent and the material scheduling agent respectively use multi-layer perceptrons to make decisions based on context representation.

8. The agent-based material handling decision optimization method according to claim 7, characterized in that: The training process includes: Initialize the policy network parameters and value network parameters of the order allocation agent, and initialize the policy network parameters and value network parameters of the material scheduling agent; During the data collection phase, the environment interacts with the actions decided by the decision network in the current training phase to obtain a trajectory dataset containing records of the environment state, actions, rewards, and corresponding action probabilities. The training of the order allocation agent and the material scheduling agent is switched alternately according to the switching episodes. The value networks of the order allocation agent and the material scheduling agent use the state of the environment in the collected trajectory dataset to calculate their respective generalized advantage estimates; The shear surrogate objective function is constructed using the probability ratio and generalized advantage estimation of the new and old decision networks of the order allocation agent and the material scheduling agent. The policy network is updated in a way that maximizes the shear surrogate objective function. The value network parameters are updated using the mean square error between the reward-based return and the value network.

Images