MADDPG-based digital twin dense storage sorting path optimization method

Abstract

The invention discloses a MADDPG-based digital twin dense storage sorting path optimization method, which is applied to the technical field of sorting path optimization in the storage field, and comprises the following steps: constructing a data twin storage three-dimensional model based on storage state data collected in real time, and simulating the dynamic behavior of a six-way shuttle vehicle by using a simulation engine; an improved A algorithm is adopted as a path generation tool of the six-way shuttle vehicle, and path conflict search of the six-way shuttle vehicle is completed in combination with an ECBS algorithm; a parameter adjustment problem of an ECBS algorithm is modeled as a multi-agent cooperation problem, and a parameter adjustment process is guided by using an MADDPG algorithm. According to the method, the sorting efficiency, the path conflict solving efficiency, the dynamic adaptive capacity, the continuous optimization capacity and the equipment utilization efficiency are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of picking path optimization technology in warehousing, and more specifically to a digital twin dense warehousing picking path optimization method based on MADDPG. Background Technology

[0002] Picking route optimization is a crucial research area in warehousing. Proper route planning can improve efficiency and reduce costs—this is its most direct and intuitive benefit. During order picking, the walking time of pickers / equipment accounts for over 50% of the total operation time. Optimized routes can significantly reduce unnecessary walking, directly shortening the fulfillment cycle of individual orders. Route optimization directly improves picking efficiency per person and equipment utilization. This means that with the same labor costs and equipment investment, more orders can be processed, resulting in a significant increase in throughput. For automated equipment such as AGVs and robots, shorter routes mean less energy consumption, extended battery life, reduced unnecessary walking and turning, lower mechanical wear and tear on equipment, and reduced operating costs.

[0003] Traditional techniques still play an irreplaceable role in path planning and optimization, especially in scenarios with relatively defined conditions and requirements. However, they also have certain drawbacks and limitations. For example, Dijkstra's algorithm is used to calculate the single-source shortest path in a graph, and is suitable for network routing and traffic flow analysis, guaranteeing the finding of the shortest path from the origin to other nodes; however, it only applies to graphs with non-negative weights, cannot handle paths with negative weights, and has high computational costs for large-scale networks. Algorithms are widely used in path search and navigation systems, combining heuristic information to find the shortest path; however, a suitable heuristic function needs to be designed, as an inappropriate heuristic may lead to efficiency degradation; and in path selection, the heuristic function may influence the choice rather than the actual shortest path. The Floyd-Warshall algorithm is used to calculate the shortest path between all pairs of nodes in a weighted graph; however, it only works for fully connected weighted graphs, and has high time complexity and memory requirements, making it unsuitable for large-scale graphs. The Bellman-Ford algorithm handles the shortest path problem in graphs with negatively weighted edges, making it suitable for networks with negative weights; however, its computational complexity is higher than Dijkstra's algorithm, and its performance degrades significantly when the number of edges in the graph is large. Greedy algorithms are used for path optimization and bin packing problems, achieving global optimum through local optima, and are simple and fast; however, seeking only local optima may prevent finding the global optimum, and their understanding of problem complexity is limited, making it easy to miss the optimal path. Genetic algorithms are suitable for solving complex optimization problems, including the Traveling Salesman Problem (TSP) and path planning. They find approximate solutions by simulating natural selection and genetic operations, and are applicable to large-scale combinatorial optimization problems. However, they are difficult to control the quality of solutions found, and there is a risk of convergence to local optima. They require significant computational resources and time to evaluate the fitness of solutions. Simulated annealing is a heuristic algorithm used to find the global optimum. It avoids local optima by simulating macromolecular dynamics, but it has a high time cost and low computational efficiency, often requiring multiple iterations and adjustments to achieve satisfactory results. Parameter selection (such as cooling rate) has a significant impact on the algorithm's performance and requires fine-tuning.

[0004] Traditional algorithms often rely on predefined paths and rules, lacking the ability to handle real-time emergencies. When the actual environment changes (such as accidents or path damage), the solutions are difficult to adjust in a timely manner, potentially leading to significant delays and efficiency losses. Furthermore, commonly used solutions typically lack the ability to receive and process real-time data, limiting their ability to adjust path planning to adapt to dynamic conditions. The inability to update based on real-time information (such as real-time traffic conditions) results in an inability to quickly respond to emerging problems. Simultaneously, because they typically employ fixed decision-making rules, they are difficult to flexibly adjust and optimize according to different situations, limiting their ability to handle complex and changing environments. They cannot dynamically prioritize or select alternative paths, resulting in an inability to optimally utilize existing resources for better performance. Moreover, existing commonly used solutions lack mechanisms for learning from historical experience and self-improvement, failing to accumulate knowledge to enhance the quality of subsequent decisions. When faced with recurring problems, they cannot effectively optimize and prevent similar issues, allowing the same errors to recur.

[0005] Therefore, how to provide a MADDPG-based digital twin dense collection storage and picking path optimization method that can effectively solve the above problems is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention provides a method for optimizing the storage and picking path based on MADDPG digital twin dense collection.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: A method for optimizing the picking path in a digital twin dense collection based on MADDPG includes: Step 1: Based on real-time collected warehouse status data, construct a data twin warehouse 3D model and use a simulation engine to simulate the dynamic behavior of the six-way shuttle. Step 2: Use the improved A The algorithm serves as a path generation tool for the six-way shuttle and is combined with the ECBS algorithm to complete the path conflict search for the six-way shuttle. Step 3: Model the parameter tuning problem of the ECBS algorithm as a multi-agent cooperative problem, and use the MADDPG algorithm to guide the parameter tuning process.

[0008] Optionally, in step 1, the warehouse status data includes: shelf layout, track topology, storage space occupancy, and the size, three-dimensional position, running speed, and distance between the six-way shuttle and other vehicles.

[0009] Optionally, in step 2, the improved A The algorithm, specifically: The evaluation function is as follows:

[0010] in, From the starting point to the current node The actual cost; From The heuristic cost of reaching the goal; The heuristic function is as follows: Using Manhattan distance:

[0011] in, For the current node of Axis coordinates; For the current node of Axis coordinates; For the target node Axis coordinates; For the target node Axis coordinates; The actual cost optimization is as follows: Directional constraints: Introducing the direction matrix Constraints on extended nodes: like If the current position is the mother track, the six-way shuttle can move in four directions; like If the current position is a sub-track, the six-way shuttle can only move longitudinally; Regarding the illegal expansion direction, let ; The costs of turning around are as follows: When the path direction changes, the turning time cost increases. ;

[0012] in, For indicator functions, This refers to two consecutive steps in different directions; Heuristic function enhancement, as follows: Estimate minimum number of turns, and make corrections. : Unobstructed on the same track:

[0013] Different tracks:

[0014] Obstacles on the same track:

[0015] in, For the revised version .

[0016] Optionally, in step 2, an improved A is used. The algorithm serves as a path generation tool for the six-way shuttle and, in conjunction with the ECBS algorithm, completes the path conflict search for the six-way shuttle. Specifically: Adopting improved A The algorithm generates an unconstrained initial path for each six-way shuttle (FS), constructs an ECBS constraint tree root node N, and adds the node to be expanded buffer (OPEN). Each constraint tree node contains three elements: the constraint set... ,untie and total cost ; Calculate the lower bound of the cost for each node in OPEN. , will satisfy The nodes are added to the high-potential path solution filtering list FOCAL; among them, For the first The shortest path cost for a six-way shuttle vehicle while satisfying its constraints; This refers to the number of six-way shuttle cars; It is a bounded second-best factor; Select the node with the fewest conflicts hc(N) in the FOCAL list, detect path conflicts, generate two child nodes for each conflict, add corresponding constraints according to the conflict type, and call the improved A. The algorithm performs replanning; Repeat the above operation until the termination condition is met; the termination condition is divided into successful termination and failure termination; successful termination is: there exists N∈FOCAL satisfying hc(N)=0, return. Failure to terminate: or There is no feasible solution.

[0017] Optionally, each constraint tree node contains three elements, specifically: constraint set Used to record the spatiotemporal constraints on each of the six-directional shuttles, in the form of: This indicates that the j-th FS vehicle cannot be located at position s at time t; untie : Represents the set of paths for all six-way shuttles; where, This indicates the position of the j-th six-way shuttle at time t; Total cost Total time for all six-way shuttles to complete the task.

[0018] Optionally, detect path conflicts, generate two child nodes for each conflict, add corresponding constraints based on the conflict type, and call the improved A. The algorithm performs replanning, specifically as follows: For the solution of node N traverse all discrete time steps Detect two types of conflicts: Point conflict: if it exists , making If a conflict occurs, it is called a point conflict, denoted as . ;in, The number for the six-way shuttle; A six-way shuttle with a different number than the j-th six-way shuttle; Let be the position of the j-th six-way shuttle at discrete time step t; For the first The position of a six-way shuttle at discrete time step t; The location where the conflict occurred; Edge conflict: if it exists , making If this happens, it is called a border conflict, denoted as . ;in, The position of the j-th six-way shuttle at time step t+1 is also the position of the... A six-way shuttle is positioned at time step t; When a conflict is detected, node N splits into two child nodes. and : Point conflict :

[0019] in, For newly generated nodes in the ECBS constraint tree The set of constraints; For newly generated nodes in the ECBS constraint tree The set of constraints; Border Conflict :

[0020] After splitting, call the improved A. The algorithm replans the path for the affected six-way shuttle and updates the child nodes. and .

[0021] Optionally, in step 3, the parameter tuning problem of the ECBS algorithm is modeled as a multi-agent cooperative problem, and the MADDPG algorithm is used to guide the parameter tuning process, specifically as follows: Each parameter in the ECBS algorithm is treated as an independent agent. The agent interacts with the environment based on the current Actor policy and collects experience tuples. Stored in the shared experience replay buffer D; where, This represents the current global state of the system. For the next global state; For the joint action of all intelligent agents; For each agent, there is a corresponding reward signal. A small batch of experience is randomly sampled from D to update the Critic network. At the same time, the Actor network is updated based on the optimization results of the Critic network, and the parameters of the Target Actor and Target Critic networks are slowly updated using a soft update method.

[0022] Optionally, update the Critic network, specifically: For each agent i, use its Target Critic network. The Target Actor network of all agents computes the target. The values ​​are as follows:

[0023] in, For temporary rewards to the agent; Discount factor; For the j-th agent, a Target Actor network is provided. The j-th agent is The following observations; Minimize the Critic network The mean squared error loss is as follows:

[0024] in, For the expectation operator; use gradient descent to update the Critic network parameters. .

[0025] Optionally, update the Actor network, specifically: For each agent i, the objective is to maximize the expected Q-value given by its Critic network; the policy gradient is calculated as follows:

[0026] in, Let be the gradient of the objective function of the Actor network for agent i; For the empirical expectation operator; Let be the gradient of the Critic network for the action of agent i. Replace the action marker; The Q-value output along the Critic network relative to the action of agent i itself The gradient direction is used to update the parameters of the Actor network. to increase Values; update Actor network parameters using gradient ascent. .

[0027] Optionally, the parameters of the Target Actor and Target Critic networks can be slowly updated using a soft update method, specifically:

[0028] in, These are the current training parameters of the Actor network for agent i; These are the current parameters of the Target Critic network for agent i; This is a soft update coefficient; These are the current training parameters of the Critic network for agent i; These are the current parameters of the Target Critic network for agent i.

[0029] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a digital twin dense collection storage and picking path optimization method based on MADDPG, which achieves the following beneficial effects: (1) Real-time dynamic adaptability: It can dynamically adjust the path planning according to real-time data, solve the problem that traditional algorithms cannot handle real-time emergencies (such as accidents or path damage), and improve the system's ability to cope with dynamic environmental changes.

[0030] (2) Learning and self-improvement mechanism: It has a mechanism to learn from historical experience and improve itself, and can accumulate knowledge to improve the quality of subsequent decisions and avoid the recurrence of repeated problems.

[0031] (3) Multi-agent cooperative optimization: Multi-agent cooperative optimization is achieved through the MADDPG algorithm, which enables the system to coordinate the path planning of multiple six-way shuttles more effectively and improve the overall system efficiency.

[0032] (4) Reduce operating costs: For automated equipment such as AGVs and robots, shorter paths mean less power consumption, longer battery life, less unnecessary walking and turning, and reduced mechanical wear of equipment.

[0033] (5) Efficiently resolve path conflicts: By combining the ECBS algorithm (an improved path conflict search algorithm), suboptimal solutions are allowed in exchange for computational efficiency, thus enabling rapid resolution of multi-agent path conflicts. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0035] Figure 1 This is a schematic diagram of the method flow provided by the present invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] Example 1: Embodiment 1 of this invention discloses a digital twin dense collection storage and picking path optimization method based on MADDPG, such as... Figure 1 As shown, it includes: Step 1: Based on the real-time collected warehouse status data, construct a data twin warehouse 3D model and use a simulation engine to simulate the dynamic behavior of the six-way shuttle.

[0038] Warehouse status data includes: rack layout, track topology, warehouse space occupancy, and the size, three-dimensional position, operating speed, and distance between the six-way shuttle and other vehicles.

[0039] The physical facilities used in this invention include: Space and structural facilities: automated racking, rails, pallets.

[0040] Data acquisition and transmission facilities: Data acquisition facilities: Position sensors: photoelectric encoder (located on the shuttle), RFID positioning base station (located on the automated racking), RFID electronic tag (located on the pallet and six-way shuttle), limit switch (located next to the track); Data transmission facilities: routers, access points (APs), ACs, and switches.

[0041] Picking equipment: six-way shuttle.

[0042] Computing and control facilities: computing center, edge computing gateway.

[0043] Sensor readings are transmitted via the Zigbee protocol. Shelf occupancy status is detected using limit switches, which, in conjunction with their drivers, transmit data via Wi-Fi 6 using the Modbus protocol. Wireless data is transmitted through an access point connected to a Cat6e network cable, then through an AC (Automatic Access Center), and finally to an industrial switch. The signal ultimately reaches the computing center; protocol conversion and preprocessing are performed through an edge computing gateway to achieve unified access for heterogeneous devices.

[0044] Protocol conversion and data processing: Wired Network: Industrial Ethernet is used to access AP devices within the rack, supporting high real-time control (such as shuttle movement commands). Wireless Network: Wi-Fi 6 (for APs and shuttles). Communication Protocols: MQTT (Lightweight Device Communication), HTTPS (API Interaction), OPC UA (Industrial Interoperability), WebSocket (Real-time Status Push). Protocol conversion and preprocessing are performed through an edge computing gateway to achieve unified access for heterogeneous devices. Specific steps are as follows: Protocol conversion is performed via an edge computing gateway; Data processing is performed via an edge computing gateway; The processed data is sent to the computing center.

[0045] Step 2: Use the improved A The algorithm serves as a path generation tool for the six-way shuttle and is combined with the ECBS algorithm to complete the path conflict search for the six-way shuttle.

[0046] ECBS is an improved path conflict search algorithm that enables rapid resolution of multi-agent path conflicts by allowing suboptimal solutions in exchange for computational efficiency.

[0047] Basic settings and symbol explanations: Library space: 3D discrete raster ; Intelligent agents: n six-way shuttles (SS), numbered ; Task: ,in:

[0048] Time discretization: ; path: ; Start time: (Can be set to 0, or scheduled by task); Completion Time: ; Objective function: Minimize the total time (i.e., total travel time) for all SS to complete the task:

[0049] Constraints: (1) The path is within the feasible region:

[0050] (2) Avoid obstacles: Let the obstacle set be ,but:

[0051] (3) Start and end point constraints:

[0052] Six-way movement constraints (adjacency or wait):

[0053] That is, each step can only move 1 grid along ±x, ±y, or ±z, or wait in place.

[0054] At the same time, attention should be paid to the boundaries:

[0055] Point conflict constraint (3D):

[0056] Edge conflict constraint (3D opposing movement):

[0057] This includes opposite movements on the vertical edge (e.g., A moves from z=2 to 3, and B moves from z=3 to 2, both located in the same (x,y)).

[0058] Cost function and heuristic function (3D + steering optimization): Actual cost : consider: Horizontal movement cost = 1; Vertical movement cost = α ≥ 1 (default α = 1, if the ascent and descent are slow, α = 2 can be set); Turning cost = P > 0; Let the current node be n, and the previous node be n. The previous node is Define the direction vector of movement:

[0059] but:

[0060] The first movement (without a forward direction) does not involve steering.

[0061] Heuristic function h(n): Using weighted 3D Manhattan distance:

[0062] Improved A The algorithm (lower-level mechanism) is as follows: The evaluation function is as follows:

[0063] in, From the starting point to the current node The actual cost; From The heuristic cost of reaching the goal; The heuristic function is as follows: Using Manhattan distance (because FS can only travel straight along the track):

[0064] in, For the current node of Axis coordinates; For the current node of Axis coordinates; For the target node Axis coordinates; For the target node Axis coordinates; The actual cost optimization is as follows: Directional constraints: Introducing the direction matrix Constraints on extended nodes: like If the current position is the mother track, the six-way shuttle can move in four directions; like If the current position is a sub-track, the six-way shuttle can only move longitudinally; Regarding the illegal expansion direction, let ; The costs of turning around are as follows: When the path direction changes, the turning time cost increases. ;

[0065] in, For indicator functions, This refers to two consecutive steps in different directions; Heuristic function enhancement, as follows: Estimate minimum number of turns, and make corrections. : Unobstructed on the same track:

[0066] Different tracks:

[0067] Obstacles on the same track:

[0068] in, For the revised version This design significantly reduces the number of turns and the number of extended nodes.

[0069] Adopting improved A The algorithm serves as a path generation tool for the six-way shuttle and, in conjunction with the ECBS algorithm, completes the path conflict search for the six-way shuttle. Specifically: Adopting improved A The algorithm generates an unconstrained initial path for each six-way shuttle (FS), constructs an ECBS constraint tree root node N, and adds the node to be expanded buffer (OPEN). Each constraint tree node contains three elements: the constraint set... ,untie and total cost ; Calculate the lower bound of the cost for each node in OPEN. , will satisfy The nodes are added to the high-potential path solution filtering list FOCAL; among them, For the first The shortest path cost of a six-way shuttle under its constraints (usually determined by the lower layer A) Heuristics approximate); This refers to the number of six-way shuttle cars; It is a bounded second-best factor; Select the node with the fewest conflicts hc(N) in the FOCAL list (this strategy prioritizes exploring "high-potential" nodes to avoid blindly expanding high-cost, low-quality solutions), detect path conflicts, generate two child nodes for each conflict, add corresponding constraints based on the conflict type, and call the improved A The algorithm performs replanning; Repeat the above operation until the termination condition is met; the termination condition is divided into successful termination and failure termination; successful termination is: there exists N∈FOCAL satisfying hc(N)=0, return. Failure to terminate: or There is no feasible solution.

[0070] Theoretical nature: Convergence: Due to the finite state space and the addition of effective constraints with each split, the algorithm will terminate in a finite number of steps. Bounded suboptimality: If the solution Π is returned, then:

[0071] Among them Π This is the globally optimal solution.

[0072] Each constraint tree node contains three elements, specifically: constraint set Used to record the spatiotemporal constraints on each of the six-directional shuttles, in the form of: This indicates that the j-th FS vehicle cannot be located at position s at time t; untie : Represents the set of paths for all six-way shuttles; where, This indicates the position of the j-th six-way shuttle at time t; Total cost Total time for all six-way shuttles to complete the task.

[0073] Detect path conflicts, generate two child nodes for each conflict, add corresponding constraints based on the conflict type, and call the improved A. The algorithm performs replanning, specifically as follows: For the solution of node N traverse all discrete time steps Detect two types of conflicts: Vertex Conflict: If it exists , making If a conflict occurs, it is called a point conflict, denoted as . ;in, The number for the six-way shuttle; A six-way shuttle with a different number than the j-th six-way shuttle; Let be the position of the j-th six-way shuttle at discrete time step t; For the first The position of a six-way shuttle at discrete time step t; The location where the conflict occurred; Edge Conflict: If it exists , making If this happens, it is called a border conflict, denoted as . ;in, The position of the j-th six-way shuttle at time step t+1 is also the position of the... A six-way shuttle is positioned at time step t; When a conflict is detected, node N splits into two child nodes. and : Point conflict :

[0074] in, For newly generated nodes in the ECBS constraint tree The set of constraints; For newly generated nodes in the ECBS constraint tree The set of constraints; Border Conflict :

[0075] After splitting, call the improved A. The algorithm replans the path for the affected six-way shuttle and updates the child nodes. and .

[0076] ECBS performance is highly sensitive to various parameters (such as heuristic function settings, conflict priority rules, path truncation mechanisms, etc.). The adjustment of these parameters can essentially be viewed as a multi-agent cooperative optimization problem: 1. Each parameter can be considered an individual (intelligent agent). These agents need to coordinate and adjust their respective parameter values ​​in a shared environment to collectively achieve optimal algorithm performance metrics (such as minimum path conflict rate and minimum computational cost). 2. Minimize the cost function of ECBS in the path planning task, ensuring compatibility between different parameters to achieve a synergistic improvement in global performance.

[0077] 1. Multi-Agent Modeling of the Parameter Tuning Problem: The parameter tuning problem in ECBS is modeled as a multi-agent cooperative problem, defined as follows: Agent: Each ECBS parameter (cost function, heuristic function) is considered an agent.

[0078] Reward Function: This function designs rewards based on the impact of the agent's behavior on the overall performance of the algorithm. Positive rewards: reduced conflict rate, shorter path length, and reduced total computation time.

[0079] Negative rewards: Parameter adjustments can lead to decreased computational efficiency or increased path conflict rates.

[0080] Centralized collaboration mechanism: Utilize a global optimizer to adjust the weights and actions of each parameter by uniformly evaluating the impact of all parameters on ECBS optimization.

[0081] The reinforcement learning framework is designed to utilize MADDPG to guide the parameter tuning process, including global features such as the current path conflict rate and computation time.

[0082] Rewards: The reward weight for improving the measurability of algorithm performance through standardization and reducing path conflict rate is set to... The reward weight for reducing computation time is set to The reward weight for reducing the total time to complete the task is set to .

[0083] Step 3: Model the parameter tuning problem of the ECBS algorithm as a multi-agent cooperative problem, and use the MADDPG algorithm to guide the parameter tuning process.

[0084] MADDPG is a deep reinforcement learning algorithm specifically designed for multi-agent environments. It is developed based on the Deep Deterministic Policy Gradient (DDPG) algorithm. DDPG itself is a single-agent algorithm for solving continuous action space control problems, combining deterministic policy gradient (DPG) and deep neural networks.

[0085] The basic idea of ​​MADDPG is to adopt a framework of "Centralized Training with Decentralized Execution" (CTDE): Training Phase (Centralized): Each agent has an independent Actor network (policy network), but their Critic network (value network) can access global state information and action information of all agents during training. This provides a more stable and information-rich evaluation basis for each Critic.

[0086] Execution Phase (Distributed): During the actual execution of the strategy, each agent relies solely on its own Actor network and local observations to make decisions, without needing to communicate with other agents or obtain global information. This makes the algorithm highly scalable and practical during deployment.

[0087] MADDPG is built on top of DDPG, and its core components and processes are as follows: Key components (for each agent) ): Actor Network ; Input: agent Local observation status (or ).

[0088] Output: Agent Deterministic actions .

[0089] Objective: To learn intelligent agents The optimal strategy.

[0090] Target Actor Network ′; It is a delayed update copy of the Actor network, used to calculate the target Q-value, increasing training stability.

[0091] Critic Network : Input: Global state information or joint observation and the joint actions of all intelligent agents. .

[0092] Output: Given the global state Perform actions with all intelligent agents Under these conditions, for intelligent agents Estimated future expected cumulative reward .

[0093] Objective: To evaluate the performance of an agent from a global perspective. The quality of the strategy (and its interaction with other intelligent agents).

[0094] Target Critic Network ′: It is a delayed-update copy of the Critic network used to compute the target. value.

[0095] Experience replay buffer : Store the experience tuples obtained through the joint exploration of all agents. This is a shared buffer that contains global information and rewards for all agents.

[0096] The parameter tuning problem of the ECBS algorithm is modeled as a multi-agent cooperative problem, and the MADDPG algorithm is used to guide the parameter tuning process, specifically: Each parameter in the ECBS algorithm is treated as an independent agent. The agent interacts with the environment according to the current Actor policy (incorporating exploratory noise, such as OU noise), and collects experience tuples. Stored in the shared experience replay buffer D; where, This represents the current global state of the system. For the next global state; For the joint action of all intelligent agents; For each agent, there is a corresponding reward signal. A small batch of experience is randomly sampled from D to update the Critic network. At the same time, the Actor network is updated based on the optimization results of the Critic network, and the parameters of the Target Actor and Target Critic networks are slowly updated using a soft update method.

[0097] Update the Critic network as follows: For each agent i, use its Target Critic network. The Target Actor network of all agents computes the target. The values ​​are as follows:

[0098] in, For temporary rewards to the agent; Discount factor; For the j-th agent, a Target Actor network is provided. The j-th agent is The following observations; Minimize the Critic network The mean squared error loss (MSE Loss) is as follows:

[0099] in, For the expectation operator; use gradient descent to update the Critic network parameters. .

[0100] Update the Actor network as follows: For each agent i, the objective is to maximize the expected Q-value given by its Critic network; the policy gradient is calculated as follows:

[0101] in, Let be the gradient of the objective function of the Actor network for agent i; For the empirical expectation operator; Let be the gradient of the Critic network for the action of agent i. Replace the action marker; The Q-value output along the Critic network relative to the action of agent i itself The gradient direction is used to update the parameters of the Actor network. to increase Values; update the Actor network parameters using gradient ascent (or negative gradient descent). .

[0102] To improve learning stability, the parameters of the Target Actor and Target Critic networks are updated slowly using a soft update method. Specifically:

[0103] in, These are the current training parameters of the Actor network for agent i; These are the current parameters of the Target Critic network for agent i; This is the soft update coefficient. (e.g., 0.01 or 0.001); These are the current training parameters of the Critic network for agent i; These are the current parameters of the Target Critic network for agent i.

[0104] The input to the Critic network includes the global state. and the actions of all intelligent agents This means that when training the Critic, it "knows" what other agents are doing (actions). and the overall state of the environment. This provides the Critic with a stable environment containing information about the actions of other agents for evaluation, even if their policies change. When the Actor updates, the gradient direction provided by the Critic implicitly considers the influence of other agents. (Critic network) The evaluation is performed on the global state. Perform joint actions with all intelligent agents Under these conditions, intelligent agents The expected cumulative reward. Although it cannot be perfectly separated. While it contributes significantly (especially in complex interactions), it more accurately reflects the impact of Criticism based solely on its own observations and actions (such as independent DDPG). The value of an action in the current global context.

[0105] Step 4: Command Feedback: Feedback the ECBS path planning results to the physical world in the form of control commands, and repeat steps 1-3.

[0106] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0107] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for optimizing storage and picking paths based on MADDPG's digital twin dense collection, characterized in that, include: Step 1: Based on real-time collected warehouse status data, construct a data twin warehouse 3D model and use a simulation engine to simulate the dynamic behavior of the six-way shuttle. Step 2: Use the improved A The algorithm serves as a path generation tool for the six-way shuttle and, in conjunction with the ECBS algorithm, completes the path conflict search for the six-way shuttle. Step 3: Model the parameter tuning problem of the ECBS algorithm as a multi-agent cooperative problem, and use the MADDPG algorithm to guide the parameter tuning process.

2. The method for optimizing storage and picking paths based on MADDPG in a digital twin dense collection according to claim 1, characterized in that, In step 1, the warehouse status data includes: shelf layout, track topology, warehouse space occupancy, and the size, three-dimensional position, running speed, and distance between the six-way shuttle and other vehicles.

3. The method for optimizing the storage and picking path based on MADDPG in a digital twin-based dense collection according to claim 1, characterized in that, In step 2, the improved A The algorithm, specifically: The evaluation function is as follows: in, From the starting point to the current node The actual cost; From The heuristic cost of reaching the goal; The heuristic function is as follows: Using Manhattan distance: in, For the current node of Axis coordinates; For the current node of Axis coordinates; For the target node Axis coordinates; For the target node Axis coordinates; The actual cost optimization is as follows: Directional constraints: Introducing the direction matrix Constraints on extended nodes: like If the current position is the mother track, the six-way shuttle can move in four directions; like If the current position is a sub-track, the six-way shuttle can only move longitudinally; Regarding the illegal expansion direction, let ; The costs of turning around are as follows: When the path direction changes, the turning time cost increases. ; in, For indicator functions, This refers to two consecutive steps in different directions; Heuristic function enhancement, as follows: Estimate minimum number of turns, and make corrections. : Unobstructed on the same track: Different tracks: Obstacles on the same track: in, For the revised version .

4. The method for optimizing storage and picking paths based on MADDPG in a digital twin-based dense collection according to claim 1, characterized in that, In step 2, an improved A is used. The algorithm serves as a path generation tool for the six-way shuttle and, in conjunction with the ECBS algorithm, completes the path conflict search for the six-way shuttle. Specifically: Adopting improved A The algorithm generates an unconstrained initial path for each six-way shuttle (FS), constructs an ECBS constraint tree root node N, and adds the node to be expanded buffer (OPEN). Each constraint tree node contains three elements: the constraint set... ,untie and total cost ; Calculate the lower bound of the cost for each node in OPEN. , will satisfy The nodes are added to the high-potential path solution filtering list FOCAL; among them, For the first The shortest path cost for a six-way shuttle vehicle while satisfying its constraints; This refers to the number of six-way shuttle cars; It is a bounded second-best factor; Select the node with the fewest conflicts hc(N) in the FOCAL list, detect path conflicts, generate two child nodes for each conflict, add corresponding constraints according to the conflict type, and call the improved A. The algorithm performs replanning; Repeat the above operation until the termination condition is met; the termination condition is divided into successful termination and failure termination; successful termination is: there exists N∈FOCAL satisfying hc(N)=0, return. The failure terminates as follows: or There is no feasible solution.

5. The method for optimizing the storage and picking path based on MADDPG digital twin dense collection according to claim 4, characterized in that, Each constraint tree node contains three elements, specifically: The constraint set Used to record the spatiotemporal constraints on each of the six-directional shuttles, in the form of: This indicates that the j-th FS vehicle cannot be located at position s at time t; The solution : Represents the set of paths for all six-way shuttles; where, This indicates the position of the j-th six-way shuttle at time t; The total cost Total time for all six-way shuttles to complete the task.

6. The method for optimizing the storage and picking path based on MADDPG digital twin dense collection according to claim 4, characterized in that, Detect path conflicts, generate two child nodes for each conflict, add corresponding constraints based on the conflict type, and call the improved A. The algorithm performs replanning, specifically as follows: For the solution of node N traverse all discrete time steps Detect two types of conflicts: Point conflict: if it exists , making If a conflict occurs, it is called a point conflict, denoted as . ;in, The number for the six-way shuttle; A six-way shuttle with a different number than the j-th six-way shuttle; Let be the position of the j-th six-way shuttle at discrete time step t; For the first The position of a six-way shuttle at discrete time step t; The location where the conflict occurred; Edge conflict: if it exists , making If this happens, it is called a border conflict, denoted as . ;in, The position of the j-th six-way shuttle at time step t+1 is also the position of the... A six-way shuttle is positioned at time step t; When a conflict is detected, node N splits into two child nodes. and : Point conflict : in, For newly generated nodes in the ECBS constraint tree The set of constraints; For newly generated nodes in the ECBS constraint tree The set of constraints; Border Conflict : After splitting, the improved A is called. The algorithm replans the path for the affected six-way shuttle and updates the child nodes. and .

7. The method for optimizing the storage and picking path based on MADDPG digital twin dense collection according to claim 1, characterized in that, In step 3, the parameter tuning problem of the ECBS algorithm is modeled as a multi-agent cooperative problem, and the MADDPG algorithm is used to guide the parameter tuning process, specifically as follows: Each parameter in the ECBS algorithm is considered an independent agent. This agent interacts with the environment based on the current Actor policy and collects experience tuples. Stored in the shared experience replay buffer D; where, This represents the current global state of the system. For the next global state; For the joint action of all intelligent agents; For each agent, there is a corresponding reward signal. A small batch of experience is randomly sampled from D to update the Critic network. At the same time, the Actor network is updated based on the optimization results of the Critic network, and the parameters of the Target Actor and Target Critic networks are slowly updated using a soft update method.

8. The method for optimizing the storage and picking path based on MADDPG digital twin dense collection according to claim 7, characterized in that, The update of the Critic network specifically involves: For each agent i, use its Target Critic network. The Target Actor network of all agents computes the target. The values ​​are as follows: in, For temporary rewards to the agent; Discount factor; For the j-th agent, a Target Actor network is provided. The j-th agent is The following observations; Minimize the Critic network The mean squared error loss is as follows: in, For the expectation operator; use gradient descent to update the Critic network parameters. .

9. The method for optimizing the storage and picking path based on MADDPG digital twin dense collection according to claim 7, characterized in that, The update of the Actor network specifically involves: For each agent i, the objective is to maximize the expected Q-value given by its Critic network; the policy gradient is calculated as follows: in, Let be the gradient of the objective function of the Actor network for agent i; For the empirical expectation operator; Let be the gradient of the Critic network for the action of agent i. Replace the action marker; The Q-value output along the Critic network relative to the action of agent i itself The gradient direction is used to update the parameters of the Actor network. to increase Values; update Actor network parameters using gradient ascent. .

10. The method for optimizing the storage and picking path based on MADDPG digital twin dense collection according to claim 7, characterized in that, The method of slowly updating the parameters of the Target Actor and Target Critic networks using a soft update approach specifically involves: in, These are the current training parameters of the Actor network for agent i; These are the current parameters of the Target Critic network for agent i; This is a soft update coefficient; These are the current training parameters of the Critic network for agent i; These are the current parameters of the Target Critic network for agent i.

Images