A multi-agent path planning method and system based on a logistics storage environment

Abstract

The application belongs to the technical field of multi-agent path planning, and discloses a multi-agent path planning method and system based on a logistics storage environment; wherein the method comprises the following steps: based on a plurality of agents in the logistics storage environment, acquiring part of observation information of each agent at the current time; based on the part of observation information, using a trained collaborative optimization strategy network model to perform path planning and obtaining the action of each agent at the current time. The application specifically discloses a multi-agent path planning scheme based on evolutionary reinforcement learning for the logistics storage environment, which can be used for path planning of each agent in a plurality of agents in a complex environment, and can solve the technical problem of low planning result quality in a large scale in the prior art.

Description

A multi-agent path planning method and system based on logistics and warehousing environment Technical Field

[0001] This invention belongs to the field of multi-agent path planning technology, and specifically relates to a multi-agent path planning method and system based on a logistics and warehousing environment. Background Technology

[0002] Multi-Agent Path Finding (MAPF) is a problem of planning paths from multiple agents with different starting positions to their respective target positions. Its key constraint is to ensure that the agents reach the target position without colliding with each other, and to ensure the speed and quality of path planning.

[0003] In intelligent warehousing and logistics systems, multi-agent path planning can effectively alleviate the shortage of logistics labor and improve the efficiency of collaborative optimization strategies. However, in the logistics and warehousing environment, multi-agent path planning is often large-scale, with higher environmental complexity and a larger number of agents. Therefore, studying the multi-agent path planning problem in a large-scale logistics and warehousing environment to achieve the efficient operation of intelligent warehousing and logistics systems is a highly challenging task.

[0004] Current multi-agent path planning algorithms can be divided into two categories based on their planning methods: centralized planning methods and distributed execution methods.

[0005] Centralized planning methods rely on a central controller to plan paths for all agents. This approach presupposes that the central planner possesses complete map information and details about the start and end points of all agents. Generally, current mainstream algorithms improve performance and applicability by introducing priority planning, large-domain search, and complex heuristics. Representative algorithms include A* search, conflict-based algorithms, cost-growing tree search, and reduction-based algorithms. Centralized planning algorithms, such as the classic and commonly used MAPF algorithm, can solve most path planning problems. However, all these existing algorithms require a central planner with complete information to plan the optimal path. As environmental complexity and the number of agents increase, computational complexity increases dramatically. Therefore, with technological advancements, distributed execution methods are becoming increasingly popular.

[0006] Distributed execution methods are primarily based on reinforcement learning. They assume that each agent only possesses information about the positions of itself and obstacles within its field of vision (i.e., a certain range). Agents continuously interact with the environment based on their current policy, acquiring the previous state and reward for that action, while simultaneously calculating and updating their policy. To maximize cumulative rewards, agents gradually learn to reach the target location and collaboratively avoid obstacles, thus completing the multi-agent path planning task. These algorithms can be extended to high-density environments to efficiently solve the real-time replanning problem for multi-agent paths. Representative algorithms include the expert-demonstration PRIMARL algorithm, the improved communication-based DHC algorithm, and the hierarchical planning-based Helsa algorithm. However, these existing distributed planning methods essentially introduce a decentralized setup and train policies in random environments. In warehouse environments, the quality of multi-agent planning results is low, making it difficult to compare with traditional centralized planning methods. Summary of the Invention

[0007] The purpose of this invention is to provide a multi-agent path planning method and system based on a logistics and warehousing environment to solve one or more of the aforementioned technical problems. Specifically, this invention discloses a multi-agent path planning scheme based on evolutionary reinforcement learning for a logistics and warehousing environment. This scheme can provide path planning for multiple agents in a highly complex environment, and can solve the technical problem of low-quality planning results under large-scale conditions in the prior art.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] In a first aspect, the present invention provides a multi-agent path planning method based on a logistics warehousing environment, comprising the following steps:

[0010] Based on multiple intelligent agents in a logistics and warehousing environment, obtain partial observation information of each intelligent agent at the current moment;

[0011] Based on the aforementioned partial observation information, a path planning is performed using a trained collaborative optimization strategy network model to obtain the current actions of each agent.

[0012] in,

[0013] In the step of obtaining partial observation information of each agent at the current moment, for a selected agent, the partial observation information includes field-of-view information and local guidance vectors; the field-of-view information includes a binary matrix of shelf positions within the field of view, a binary matrix of other agent positions within the field of view, a binary matrix of intersection positions within the field of view, and a binary matrix of four-channel heuristic information; the four-channel heuristic information is used to indicate whether the selected agent is close to the endpoint when moving one grid position up, down, left, or right within the field of view, and the grid positions are obtained based on the warehouse grid map obtained based on the logistics and warehousing environment; the local guidance vector consists of a unit direction vector pointing to the endpoint and the Manhattan distance from the endpoint;

[0014] The trained collaborative optimization strategy network model employs a parallel training strategy of reinforcement learning and evolutionary learning during training.

[0015] A further improvement to the method of the present invention is that the collaborative optimization strategy network model includes:

[0016] The observation encoder module is used to input partial observation information of the agent, extract features from the field-of-view information and local guidance vectors in the partial observation information, concatenate the extracted features into a matrix, and output the encoded information vector.

[0017] The GRU network is used to take the hidden value information from the previous time step and the encoded information vector as input, and after combining the current state information and historical information, it outputs information for communication.

[0018] The communication module is used to input the communication information output by the GRU network. It communicates with other neighboring agents through the graph neural network using a self-attention mechanism and outputs the hidden value information at the current moment. At any given moment, an agent communicates with at most two of its nearest neighboring agents within its field of vision.

[0019] The Q network module is used to input the hidden value information output by the communication module and output the agent's actions.

[0020] A further improvement to the method of the present invention lies in that, in the collaborative optimization strategy network model,

[0021] The initial hidden value is 0; in addition, if the agent does not find any neighboring agents in its field of vision, no communication occurs, and the information used for communication is used as the hidden value.

[0022] A further improvement to the method of the present invention is that the training steps of the collaborative optimization strategy network model include:

[0023] The D3QN algorithm is used to train the collaborative optimization strategy network model through reinforcement learning. Utilizing the Ape-X architecture, multiple participants generate experience in parallel using a greedy strategy based on the current collaborative optimization strategy, and this experience is stored together in a global experience pool. Then, the evaluation network Q(o) is used to train the model. t ,a t ;θ) and target network Q(o t ,a t Learners with θ′ select experiences from the global experience pool to update the coordinated optimization strategy. The update method involves calculating the n-step TD error, expressed as follows:

[0024] L(θ)=Huber([r t +γr t+1 +…+γ n Q(o t+1 ,arg max a Q(o t+1 ,a;θ);θ′)]-Q(o t ,a t ;θ));

[0025] In the formula, Huber(·) is the Huber loss function; r t Let γ be the reward value obtained by the agent at time t; γ∈[0,1] is the discount factor used to balance the current reward and the future reward; o t These are partial observations of the agent at time t; o t+1 The values ​​are partial observations of the agent at time t+1; a t Let t be the action performed by the agent at time t;

[0026] The gradient descent method is used to solve for the error, so as to minimize the error and continuously optimize the learner's evaluation network. Finally, after a certain number of training iterations, the participants' collaborative optimization strategy and the learner's target network are updated.

[0027] In the process of training a policy using the D3QN algorithm, the steps completed by multiple evolvers in parallel include: First, randomly generating a warehouse environment path planning task for evaluation and performing policy evaluation; then, using the current policy obtained by the D3QN algorithm as the parent policy, using an evolutionary algorithm to complete policy evolution and generate a superior offspring policy; finally, changing the original collaborative optimization policy through soft updates.

[0028] A further improvement to the method of the present invention is that the step of using the D3QN algorithm to perform reinforcement learning training on the collaborative optimization strategy network model specifically includes:

[0029] 1) Based on the logistics and warehousing environment, generate a logistics and warehousing environment map, and design the partial observation space, action space, and reward function for each agent; among which,

[0030] The steps to generate a logistics warehousing environment map include: designing a two-dimensional warehousing grid map based on the logistics warehousing environment; setting the placement of shelves in the map based on the length, width, and shelf density of the warehousing grid map; and obtaining the logistics warehousing environment map.

[0031] The steps for designing a partial observation space include: In a partially observable environment, assuming that the agent can only observe the environment within its field of view of size R×R, where R represents the field of view scale; at time t, agent A... i The observed field of view information is represented by a 7×R×R binary matrix f t i The matrix represents the binary matrix of the shelf positions within the field of view, the binary matrix of the positions of other agents within the field of view, the binary matrix of the intersection positions within the field of view, and the binary matrix of the four-channel heuristic information; at time t, agent A... i We also need to obtain the unit direction vector pointing to the endpoint. And the distance from the destination in Manhattan Constructed local guiding vector

[0032] The steps for designing the action space include: in a 4-connected grid environment, at time t, agent A... i Capable of performing 5 discrete actions The agent moves one cell in each of the four basic directions (up, down, left, and right) or remains stationary; where, when agent A... i When an agent performs an erroneous action, its current action is replaced by remaining still.

[0033] The steps for designing the reward function include: at time t, agent A... i Taking action The reward received afterwards for,

[0034]

[0035] In the formula, r move For movement penalty; r stay For static penalty; r collide For collision penalty; r reach Rewards for achieving cooperation targets;

[0036] 2) Define agent A at time t. i Partial observation values The hidden value information obtained after passing through the observation encoder module and the communication module is: Used for action selection; assuming agent A i The Q-network action value function is From the value function and dominance function It consists of two parts, represented as follows:

[0037]

[0038] In the formula, θ represents the parameters of the overall Q-network, and p and q represent the network parameters of the value function and the advantage function, respectively. Let agent A be at time t. i The set of all possible actions;

[0039] Setting up an evaluation network in the algorithm and target network Two Q-networks are used to eliminate maximization bias, where the evaluation network is used to determine the action at the next time step, and the target network is used to determine an estimate of the reward, denoted as...

[0040]

[0041]

[0042] A further improvement of the method of the present invention lies in the step of parallel completion of multiple evolvers during the training strategy using the D3QN algorithm.

[0043] 1) The steps for policy evaluation include: assuming that each agent has a maximum number of steps t max Successfully reached the finish line and received +e. reach The evaluation value; if an agent experiences c collisions in a single task, it receives -e. collide The evaluation value; assuming the Manhattan distance from the starting point to the ending point of an agent is x, and the number of steps it takes for the agent to reach the ending point for the first time is step, if the agent reaches the ending point within the specified maximum number of steps t... max If a character fails to reach the finish line, its step count is equal to the maximum step count t. max The intelligent agent obtains e length The evaluation value is

[0044] Agent A i Guided by the collaborative optimization strategy π, task planning is performed, and the final overall evaluation value is expressed as follows:

[0045] e i (π)=r_sign·e reach -e collide +e length ;

[0046] In the formula, r_sign is the success flag;

[0047] The evaluation value of the collaborative optimization strategy π is,

[0048]

[0049] In the formula, n_env represents the collaborative optimization strategy π to be evaluated in n_env multi-agent path planning tasks in a warehousing environment; a is the set of agents participating in a multi-agent path planning task; |a| is the number of agents participating in a multi-agent path planning task.

[0050] 2) During the policy evolution process, the evolution of the evolver is as follows: First, n_env multi-agent path planning tasks in the warehouse environment are randomly generated for policy evaluation; then, the current collaborative optimization policy π(θ0) with evaluation network parameters θ0 obtained from reinforcement learning training is obtained and evaluated to obtain the initial parent policy evaluation value E[π(θ0)]; next, based on the set maximum number of iterations, the original parent policy π(θ0) is iteratively mutated; where, it is assumed that the parent policy parameter of the yth generation is θ y-1 Using the random noise vector ε y,z ~N(0,σ 2 For z = 1, 2, ..., N, N pseudo-offspring policy parameters are generated, denoted as θ. y,z =θ y-1 +ε y,z j = 1, 2, ..., N; evaluate each pseudo-offspring policy, and the obtained evaluation value is denoted as E. z =E[π(θ) y,z ]; Calculate the approximate gradient change, the calculation expression is, Obtain the mutated policy parameters. Where α is the learning rate of the evolutionary algorithm; σ is the mutation rate of the evolutionary algorithm; and π(θ) is the policy of the mutated offspring. y ) to perform an evaluation, if the evaluation value E[π(θ) y If the value of the offspring policy is greater than the initial parent policy evaluation value E[π(θ0)], then soft update is used to evolve the policy, and this evolution process of the evolver ends, and a new policy evolution begins; if no dominant offspring policy is found through mutation within the maximum number of iterations, then soft update is not performed, and the evolver evolves again.

[0051] A further improvement to the method of the present invention lies in the step of changing the original collaborative optimization strategy through soft updates.

[0052] The soft update method specifically combines the superior offspring strategy with the currently trained evaluation network and target network, represented as follows:

[0053] θ=τθ+(1-τ)θ i

[0054] θ′=τθ′+(1-τ)θ i ;

[0055] In the formula, θ represents the parameters of the evaluation network; θ′ represents the parameters of the target network; θ i The parameters are for the dominant offspring strategy.

[0056] In a second aspect, the present invention provides a multi-agent path planning system based on a logistics warehousing environment, comprising:

[0057] The data acquisition module is used to acquire partial observation information of each intelligent agent at the current moment based on multiple intelligent agents in a logistics and warehousing environment.

[0058] The path planning module is used to perform path planning based on the aforementioned partial observation information and a trained collaborative optimization strategy network model to obtain the current actions of each agent.

[0059] in,

[0060] In the step of obtaining partial observation information of each agent at the current moment, for a selected agent, the partial observation information includes field-of-view information and local guidance vectors; the field-of-view information includes a binary matrix of shelf positions within the field of view, a binary matrix of other agent positions within the field of view, a binary matrix of intersection positions within the field of view, and a binary matrix of four-channel heuristic information; the four-channel heuristic information is used to indicate whether the selected agent is close to the endpoint when moving one grid position up, down, left, or right within the field of view, and the grid positions are obtained based on the warehouse grid map obtained based on the logistics and warehousing environment; the local guidance vector consists of a unit direction vector pointing to the endpoint and the Manhattan distance from the endpoint;

[0061] The trained collaborative optimization strategy network model employs a parallel training strategy of reinforcement learning and evolutionary learning during training.

[0062] A third aspect of the present invention provides an electronic device comprising:

[0063] At least one processor; and,

[0064] A memory communicatively connected to the at least one processor; wherein,

[0065] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform a multi-agent path planning method based on a logistics warehousing environment as described in any one of the first aspects of the present invention.

[0066] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the multi-agent path planning method based on a logistics and warehousing environment as described in any one of the first aspects of the present invention.

[0067] Compared with the prior art, the present invention has the following beneficial effects:

[0068] The method provided by this invention is a multi-agent path planning method based on evolutionary reinforcement learning. It employs a collaborative optimization policy network model for path planning based on acquired information. Specifically, a collaborative optimization policy network model is constructed based on reinforcement learning. Furthermore, through a policy quantification and evaluation mechanism, parallel evolutionary learning is used during the reinforcement learning training process to complete policy evaluation and self-evolution, thereby accelerating policy training and obtaining higher-quality planning results. In summary, the technical solution disclosed in this invention can realize path planning for multiple agents in highly complex environments with multiple agents. While improving the quality of planned paths, it can effectively increase the training speed of the system's collaborative optimization policy and improve the operational efficiency of warehousing and logistics systems. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art are briefly introduced below; obviously, the drawings described below are some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0070] Figure 1 is a flowchart illustrating a multi-agent path planning method based on a logistics warehousing environment in an embodiment of the present invention.

[0071] Figure 2 is a flowchart illustrating the multi-agent path planning method based on the logistics warehousing environment in a specific embodiment of the present invention.

[0072] Figure 3 is a grid map schematic diagram of the logistics warehousing environment in an embodiment of the present invention;

[0073] Figure 4 is a schematic diagram of the collaborative optimization strategy in an embodiment of the present invention;

[0074] Figure 5 is a schematic diagram of the observation encoder module in an embodiment of the present invention;

[0075] Figure 6 is a schematic diagram of the communication module in an embodiment of the present invention;

[0076] Figure 7 is a schematic diagram of the evolution process of the collaborative optimization strategy in an embodiment of the present invention;

[0077] Figure 8 is a schematic diagram of a multi-agent path planning system based on a logistics warehousing environment in an embodiment of the present invention. Detailed Implementation

[0078] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0079] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0080] The present invention will now be described in further detail with reference to the accompanying drawings:

[0081] Please refer to Figure 1. An embodiment of the present invention provides a multi-agent path planning method based on a logistics warehousing environment, comprising the following steps:

[0082] Step 1: Based on multiple intelligent agents in the logistics and warehousing environment, obtain partial observation information of each intelligent agent at the current moment;

[0083] Step 2: Based on the partial observation information obtained in Step 1, use the trained collaborative optimization strategy network model to perform path planning and obtain the current actions of each agent.

[0084] in,

[0085] For a selected agent, the partial observation information includes field-of-view information and local guidance vectors; the field-of-view information includes a binary matrix of shelf positions within the field of view, a binary matrix of other agent positions within the field of view, a binary matrix of intersection positions within the field of view, and a binary matrix of four-channel heuristic information; the four-channel heuristic information is used to indicate whether the selected agent is close to the endpoint when moving one grid position up, down, left, or right within each grid position within the field of view; the grid positions are obtained based on a warehouse grid map obtained based on the logistics and warehousing environment;

[0086] The local guiding vector consists of a unit direction vector pointing to the endpoint and the Manhattan distance from the endpoint;

[0087] The trained collaborative optimization strategy network model employs a parallel training strategy of reinforcement learning and evolutionary learning during training.

[0088] Specifically, in the multi-agent path planning method provided in this embodiment of the invention, path planning is performed using the same pre-trained collaborative optimization policy network model based on the partial observation information of each agent. The collaborative optimization policy network model is constructed based on reinforcement learning, and through a policy quantification evaluation mechanism, it utilizes parallel evolutionary learning to complete the policy evaluation and self-evolution during the reinforcement learning training process, thereby accelerating the policy training speed and ultimately obtaining higher quality planning results.

[0089] In a further preferred embodiment of the present invention, the collaborative optimization strategy network model includes:

[0090] The observation encoder module is used to input partial observation information of the agent, extract features from the field-of-view information and local guidance vectors in the partial observation information, and concatenate the extracted features into a matrix to output the encoded information vector.

[0091] The GRU network is used to take the hidden value information from the previous time step and the encoded information vector output by the observation encoder module as input, so as to combine the current state information and historical information to output information for communication; further interpretably, the hidden value information at the initial time step can be 0;

[0092] The communication module is used to input the communication information output by the GRU network. It communicates with other neighboring agents through the graph neural network using a self-attention mechanism and outputs the hidden value information at the current moment. Further, it can be explained that at any given moment, an agent communicates with at most two of its nearest neighboring agents within its field of vision.

[0093] The Q network module is used to input the hidden value information output by the communication module and output the agent's action at this moment.

[0094] Furthermore, the training steps for the collaborative optimization policy network model include:

[0095] The D3QN algorithm is used to train the collaborative optimization strategy network model using reinforcement learning. Utilizing the Ape-X architecture, multiple participants generate experience in parallel using a greedy strategy based on the current collaborative optimization strategy, and this experience is stored together in a global experience pool. Then, the evaluation network Q(o) is used to evaluate the experience. t ,a t ;θ) and target network Q(o t ,a t Learners with θ′ select experiences from the global experience pool to update the coordinated optimization strategy. The update method involves calculating the n-step TD error, expressed as follows:

[0096] L(θ)=Huber([r t +θr t+1 +…+γ n Q(o t+1 argmax a Q(o t+1 ,a;θ);θ′)]-Q(o t ,a t ;θ));

[0097] In the formula, Huber(·) is the Huber loss function; r t Let t be the reward value obtained by the agent at time t; γ∈[0,1] is the discount factor used to balance the current reward and future reward; o t For time t, some observations of the agent; o t+1 For time t+1, these are some observations of the agent; a t Let t be the action performed by the agent.

[0098] The gradient descent method is used to solve for the error, thereby minimizing the error and continuously optimizing the learner's evaluation network. Finally, after a certain number of training iterations, the participants' collaborative optimization strategy and the learner's target network are updated.

[0099] In the process of training a policy using the D3QN algorithm, multiple evolvers are used in parallel to complete the following steps: First, a warehouse environment path planning task is randomly generated for evaluation, and the policy is evaluated; then, the current policy obtained using the D3QN algorithm is used as the parent policy, and the evolutionary algorithm is used to complete the policy evolution and generate a superior offspring policy; finally, the original collaborative optimization policy is changed through soft update.

[0100] Specifically, in the preferred embodiment of the present invention, a cooperative optimization strategy network model based on reinforcement learning is constructed. Evolutionary learning is used to evaluate and evolve the cooperative optimization strategy, which can effectively improve the quality of multi-agent path planning and accelerate the training speed of the cooperative optimization strategy.

[0101] Please refer to Figure 2. Specifically, this embodiment of the invention provides a multi-agent path planning method for large-scale warehouse environments based on evolutionary reinforcement learning, comprising the following steps:

[0102] Step S101 describes the environment of the warehouse multi-agent path planning problem under a partial Markov decision framework; specifically, by combining the characteristics of the logistics and warehousing environment, a logistics and warehousing environment map is generated, and each agent A is designed. i Partial observation space, action space, and reward function; among which...

[0103] 1) The steps for generating a logistics warehousing environment map include: designing a two-dimensional warehousing grid map based on the characteristics of the logistics warehousing environment, such as large scale, structure, and high repeatability; wherein, the length l, width w, and shelf density d (0 < d < 0.6) of the warehousing grid map are set; and the placement of shelves in the map is set based on these three pieces of information. As exemplified in Figure 3, in a specific embodiment of the present invention, for medium-high density maps, shelves are densely placed, arranged in groups of one or two rows; for medium-low density maps, shelves are arranged in groups of four or one, ultimately forming a configurable structured warehousing grid map.

[0104] 2) The steps for designing a partial observation space include: In a partially observable environment, assuming the agent can only observe the environment within a field of view (FOV) of size R×R; where R represents the field of view scale, which can be set by the agent. At time t, agent A... i The observable field of view information is represented by a 7×R×R binary matrix f t i The matrix represents the binary matrix of shelf locations within the field of view (FOV), the binary matrix of other agent locations within the FOV, the binary matrix of intersection locations within the FOV, and the binary matrix of four-channel heuristic information. The four-channel heuristic information is used to instruct agent A. i Within the FOV, at each grid position, moving one grid cell up, down, left, or right indicates proximity to the endpoint; for example, if close, the element is 1, and if far away, the element is 0. Simultaneously, to further guide the path, at time t, agent A... i We also need to obtain the unit direction vector pointing to the endpoint. And the distance from the destination in Manhattan Constructed local guiding vector In summary, agent A i Partial observations at time t FOV information can be used to determine the FOV information. t i and local guiding vector It is represented in two parts.

[0105] 3) The steps for designing the action space include: in a 4-connected grid environment, at time t, agent A... i Five discrete actions can be taken. That is, moving one cell grid in each of the four basic directions (up, down, left, and right) or remaining stationary; furthermore, to prevent erroneous behavior (agent A) i Take action at time t The impact of collisions with shelves, other agents, or exceeding boundaries (when agent A) occurs... i When an erroneous action is performed, the agent's current action is replaced by remaining still.

[0106] 4) The steps for designing the reward function include: To achieve high-quality paths and cooperation among multiple agents, the reward function consists of movement penalties, stationary penalties, collision penalties, and cooperative achievement rewards. Explained, if an agent receives a small movement penalty r after each move before reaching the destination... move Remaining stationary will result in a smaller stationary penalty r. stay If an agent collides with a shelf or another agent or goes out of bounds after taking a step, it will receive a larger collision penalty r. collide Only when all agents reach their respective endpoints will all agents receive a cooperative achievement reward r. reach ;

[0107] Therefore, agent A at time t i Taking action The reward received was,

[0108]

[0109] Specifically, the optional movement penalty r move = -0.075, static penalty r stay = -0.075, collision penalty r collide =-0.5, Cooperation achievement reward r reach =3.

[0110] Step S102: Construct a collaborative optimization policy network model based on reinforcement learning; wherein,

[0111] Referring to Figure 4, in this embodiment of the invention, the overall construction of the reinforcement learning-based collaborative optimization strategy involves encoding some observations to extract feature information, then using a graph neural network for communication learning between multiple agents, and finally using the D3QN algorithm to select actions based on the communication information. By continuously increasing the difficulty of the training environment through course learning, the strategy is gradually trained, ultimately improving its efficiency in the logistics and warehousing environment. Further, the important modules in the collaborative optimization strategy are described in the order of the observation encoder module, communication module, and Q-network module.

[0112] Referring to Figure 5, in this embodiment of the invention, in order to extract features from the observed values ​​and encode them, the observed value encoder module utilizes a multilayer perceptron network to extract local guiding vectors. Based on the characteristics of the VIT (Vision Transformer) network, a multi-head attention mechanism is used to extract FOV information. t i The features are extracted and then concatenated into a matrix to obtain the encoded information vector.

[0113] Further explanation: for a given feature information X, its multi-head attention encoding is,

[0114]

[0115] MSA=[SA1(X);SA2(X);…;SA l (X)]W O ;

[0116] Where, softmax(·) is the exponential normalization function; d is the dimension of the feature information X; l is the number of heads in the multi-head attention encoding; W O Q is the weight matrix; i (X), K i (X), V i (X) represents the query, key, and value vectors corresponding to feature information X, respectively, and their calculation formulas are as follows: in, The matrix is ​​a weight matrix corresponding to the query, key, and value;

[0117] Finally, the information extracted from both is combined to obtain intermediate information that can be used for communication, that is, for agent A at time t. i Intermediate information can be obtained through the observation encoder module.

[0118] Referring to Figure 4, in this embodiment of the invention, the intermediate information obtained by the observation encoder module needs to pass through a Gate Recurrent Unit (GRU) network to combine it with the hidden value from the previous time step. Output information for communication Represented as,

[0119] In this embodiment of the invention, a graph structure is constructed at each time step, where each agent is regarded as a node and is only connected to a limited number of adjacent nodes, and then a self-attention mechanism is used for communication.

[0120] Please refer to Figure 6. At time t, agent A... i At most, only interact with the two nearest agents A within the FOV. j and A k Communication. That is, communication information between each intelligent agent. Using the weight matrix W q W k W v Calculate its query, key, and value vector, i.e.

[0121]

[0122]

[0123]

[0124] Next, calculate the information. and The similarity between them is,

[0125]

[0126] Where, d K It is the dimension of the key vector.

[0127] Finally, by weighted summation, the intelligent agent A is output. i Hidden value information at time t That is,

[0128]

[0129] In the formula, W o This is the weight matrix.

[0130] To further explain, if an agent does not find any neighboring agents within its field of vision, no communication will occur, i.e., no communication information will be exchanged. That is, hidden value information

[0131] Since each agent in a multi-agent path planning problem has a different goal, independent learning algorithms are more suitable for multi-agent path planning problems than centralized multi-agent reinforcement learning algorithms; that is, all agents share a single action selection Q-network. In this embodiment of the invention, the D3QN algorithm is selected to learn the cooperative optimization strategy.

[0132] In a specific and exemplary embodiment of the present invention, agent A is defined at time t. i Partial observation values Hidden value information obtained through the encoder module and communication module Used for action selection. Assume agent A... i The Q-network action value function is It is determined by the value function and dominance function It consists of two parts, represented as follows:

[0133]

[0134] Where θ represents the parameters of the overall Q-network, and p and q are the network parameters of the value function and the advantage function, respectively; Let agent A be at time t. i The set of all possible actions.

[0135] To eliminate the maximization bias, the algorithm incorporates two Q-networks for evaluation. and target network The evaluation network is used to determine the action at the next moment, while the target network is used to determine the estimate of the reward.

[0136]

[0137]

[0138] Where γ∈[0,1] is a discount factor used to balance current rewards and future rewards.

[0139] To accelerate reinforcement learning training of the D3QN algorithm, this invention utilizes the Ape-X architecture, in which 20 participants generate experiences in parallel using a greedy strategy based on the current collaborative optimization strategy, and store them together in a global experience pool. Then, a learner (who owns the evaluation network and the target network) selects experiences from the global experience pool to update the collaborative optimization strategy.

[0140] Its update method is to calculate the n-step TD error.

[0141] L(θ)=Huber([r t +γr t+1 +…+γ n Q(o t+1 argmax a Q(o t+1 ,a;θ);θ′)]-Q(o t ,a t ;θ));

[0142] In the formula, Huber(·) is the Huber loss function; r t Let t be the reward value obtained by the agent at time t; γ∈[0,1] is the discount factor used to balance the current reward and future reward; o t For time t, some observations of the agent; o t+1 For time t+1, these are some observations of the agent; a t Let t be the action performed by the agent.

[0143] The gradient descent method is used to solve for the error, aiming to minimize it and continuously optimize the learner's evaluation network. Finally, after a certain number of training iterations, the participants' collaborative optimization strategy and the learner's target network are updated.

[0144] Step S103: Evolutionary learning is used to evaluate and evolve collaborative optimization strategies.

[0145] In this embodiment of the invention, a strategy quantification evaluation mechanism for a logistics warehousing environment is established using actual indicators. During reinforcement learning training, parallel evolutionary learning is used to complete the evaluation and self-evolution of the strategy, accelerating the strategy training speed and obtaining better planning results. Specifically, during the strategy training process using the D3QN algorithm, multiple evolvers are used in parallel to complete the following steps: First, a warehousing environment path planning task for evaluation is randomly generated, and the strategy is evaluated; then, the current strategy obtained using the D3QN algorithm is used as the parent strategy, and an evolutionary algorithm is used to complete the strategy evolution, generating a superior offspring strategy; finally, the original collaborative optimization strategy is changed through soft updates.

[0146] Furthermore, the specific processes for strategy evaluation and strategy evolution include:

[0147] (1) Strategy evaluation: For multi-agent path planning problems, the evaluation indicators include success rate, collision rate, average number of steps, etc. Therefore, for a multi-agent path planning task, the evaluation of the quality of path planning is related to the above indicators.

[0148] In this embodiment of the invention, it is assumed that each agent reaches a maximum of t steps. max Successfully reaching the finish line will earn you +e. reach (e reach =2) evaluation value; if an agent collides c times in a single task (with a shelf or other agents), it will receive -e collide The evaluation value, e collide =0.5c; Meanwhile, to further evaluate the path length, assume the Manhattan distance from the agent's starting point to the destination is x, and the number of steps the agent takes to reach the destination for the first time is step; if the agent reaches the destination within the specified maximum number of steps t... max If a character fails to reach the finish line, its step count is equal to the maximum step count t. max An intelligent agent can obtain e lengt The evaluation value of h is,

[0149] For example, for a certain warehouse map, there are a agents performing path planning, one of which is agent A. i Guided by the collaborative optimization strategy π, task planning is performed, ultimately yielding a total evaluation value, denoted as:

[0150] e i (π)=r_sign·e reach -e collide +e length ;

[0151] In the formula, r_sign is the success flag, which indicates that the agent has successfully completed the maximum number of steps t. max If the destination is successfully reached, the value is 1; otherwise, the value is 0.

[0152] Therefore, the evaluation value of the collaborative optimization strategy π is,

[0153]

[0154] In the formula, n_env represents the collaborative optimization strategy π to be evaluated in n_env multi-agent path planning tasks in a warehousing environment; a is the set of agents participating in a multi-agent path planning task; |a| is the number of agents participating in a multi-agent path planning task.

[0155] (2) Policy Evolution: As shown in Figure 7, in this embodiment of the invention, there are 5 evolvers running in parallel for uninterrupted policy evolution. The evolution process of an evolver is as follows: First, to ensure the fairness of the evaluation, n_env multi-agent path planning tasks in a warehouse environment are randomly generated for policy evaluation; then, the current collaborative optimization policy π(θ0) with evaluation network parameters θ0 obtained based on reinforcement learning training is obtained, and the policy is evaluated to obtain the initial parent policy evaluation value E[π(θ0)]; then, the original parent policy π(θ0) is iterated and mutated for a maximum of 10 generations; interpretably, for a collaborative optimization policy π(θ), the change of its policy is the change of the network parameter θ; therefore, assuming that the parent policy parameter of the yth generation is θ y-1 Then, using the random noise vector ε y,z ~N(0,σ 2 (z = 1, 2, ..., N) can generate N pseudo-offspring policy parameters.

[0156] θ y,z =θ y-1 +ε y,z (j = 1, 2, ..., N);

[0157] Evaluate the strategy of each pseudo-offspring and obtain the evaluation value, denoted as E. z =E[π(θ) y,z )];

[0158] The approximate gradient change is calculated using the following expression:

[0159]

[0160] This allows us to obtain the mutated policy parameters. Where α is the learning rate of the evolutionary algorithm; σ is the mutation rate of the evolutionary algorithm;

[0161] The mutated offspring strategy π(θ) y ) to perform an evaluation, if the evaluation value E[π(θ) y If the value of the policy is greater than the initial parent policy evaluation value E[π(θ0)], then the policy is superior to the original policy to some extent.

[0162] This allows for policy evolution using soft updates, which involves combining the dominant offspring policy with the currently trained evaluation and target networks.

[0163] In the formula, θ represents the parameters of the evaluation network; θ′ represents the parameters of the target network; θ i The parameters are for the dominant offspring strategy.

[0164] At the same time, this evolutionary process of the evolver will end, and a new strategy evolution will begin. If a superior offspring strategy is not found through mutation within 10 generations, then no soft update will be performed, and the evolver will repeat the above evolutionary process.

[0165] In summary, this invention, using a partially Markov decision framework, describes a warehouse multi-agent path planning problem environment. It utilizes attention mechanisms, graph neural networks, and the D3QN algorithm to build and learn a collaborative optimization strategy, and constructs a strategy evaluation and evolution method based on an evolutionary reinforcement learning framework. Ultimately, this achieves the goal of optimizing the efficiency of training strategies and improving the metrics (success rate, collision rate, average number of steps) of multi-agent path planning. More specifically, due to the complexity of large-scale path planning problems, existing traditional methods suffer from huge computational overhead and require repeated planning, making them difficult to solve and resulting in low planning quality. This invention, based on reinforcement learning, constructs a decentralized collaborative optimization strategy that outputs actions in real time based on local observations at the current moment. It can be extended to high-density warehouse environments to efficiently solve the real-time replanning problem of multi-agent paths, achieving better planning results. Specifically, as exemplified, Table 1 shows a comparison of simulation results for multi-agent path planning. Table 1 illustrates the comparison of average steps and success rate between the DHC method (a classic multi-agent path planning method) and the method of this invention, under the same high-density warehouse environment test set, with a maximum step count of 256, after the same number of training rounds. The average step count refers to the average number of steps required for all agents to successfully reach the destination during the test; the success rate refers to the proportion of test sets where all agents successfully reach the destination out of the total test set. Compared to the DHC method, the technical solution of this invention can reduce the average step count while increasing the task success rate, demonstrating that the new solution of this invention can effectively improve the performance of the strategy and enhance the quality of multi-agent path planning in a warehouse environment.

[0166] Table 1. Comparison of Simulation Results for Multi-Agent Path Planning

[0167]

[0168] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0169] Please refer to Figure 8. In another embodiment of the present invention, a multi-agent path planning system based on a logistics warehousing environment is provided, comprising:

[0170] The data acquisition module is used to acquire partial observation information of each intelligent agent at the current moment based on multiple intelligent agents in a logistics and warehousing environment.

[0171] The path planning module is used to perform path planning based on the aforementioned partial observation information and a trained collaborative optimization strategy network model to obtain the current actions of each agent.

[0172] in,

[0173] In the step of obtaining partial observation information of each agent at the current moment, for a selected agent, the partial observation information includes field-of-view information and local guidance vectors; the field-of-view information includes a binary matrix of shelf positions within the field of view, a binary matrix of other agent positions within the field of view, a binary matrix of intersection positions within the field of view, and a binary matrix of four-channel heuristic information; the four-channel heuristic information is used to indicate whether the selected agent is close to the endpoint when moving one grid position up, down, left, or right within the field of view, and the grid positions are obtained based on the warehouse grid map obtained based on the logistics and warehousing environment; the local guidance vector consists of a unit direction vector pointing to the endpoint and the Manhattan distance from the endpoint;

[0174] The trained collaborative optimization strategy network model employs a parallel training strategy of reinforcement learning and evolutionary learning during training.

[0175] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions from the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a multi-agent path planning method.

[0176] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the multi-agent path planning method based on a logistics warehousing environment in the above embodiments.

[0177] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0178] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0179] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0180] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0181] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A multi-agent path planning method based on a logistics warehousing environment, characterized in that, Includes the following steps: Based on multiple intelligent agents in a logistics and warehousing environment, obtain partial observation information of each intelligent agent at the current moment; Based on the aforementioned partial observation information, a pre-trained collaborative optimization strategy network model is used for path planning to obtain the current actions of each agent. In the step of obtaining partial observation information for each agent at the current moment, for a selected agent, the partial observation information includes field-of-view information and a local guidance vector. The field-of-view information includes a binary matrix of shelf positions, a binary matrix of other agent positions, a binary matrix of intersection positions, and a binary matrix of four-channel heuristic information. The four-channel heuristic information indicates whether the selected agent is close to the endpoint when moving one grid position up, down, left, or right within the field of view. The grid positions are obtained based on a warehouse grid map obtained from the logistics and warehousing environment. The local guidance vector consists of a unit direction vector pointing to the endpoint and the Manhattan distance from the endpoint. The pre-trained collaborative optimization strategy network model employs a parallel training strategy of reinforcement learning and evolutionary learning during training. The collaborative optimization strategy network model includes: an observation encoder module, used to input partial observation information of the agent, extract features from the field-of-view information and local guidance vectors in the partial observation information, concatenate the extracted features into a matrix, and output an encoded information vector; a GRU network, used to input the hidden value information of the previous time step and the encoded information vector, and output communication information after combining the current state information and historical information; a communication module, used to input the communication information output by the GRU network, communicate with other neighboring agents through a graph neural network using a self-attention mechanism, and output the hidden value information of the current time step; wherein, at most, the agent communicates with the two nearest neighboring agents in the field of view at any given time step; and a Q network module, used to input the hidden value information output by the communication module, and output the agent's action; the training steps of the collaborative optimization strategy network model include: using the D3QN algorithm to perform reinforcement learning training on the collaborative optimization strategy network model; wherein, using the Ape-X architecture, multiple participants generate experience in parallel using a greedy strategy according to the current collaborative optimization strategy, and store it together in a global experience pool; then, the evaluation network... and target network Learners select experiences from the global experience pool and update the coordination optimization strategy by computation. The step TD error is calculated using the following expression: In the formula, Huber loss function; Let t be the reward value obtained by the agent at time t; This is a discount factor used to balance current and future rewards; These are partial observations of the agent at time t; These are partial observations of the agent at time t+1; Let t be the action performed by the agent at time t; To evaluate the parameters of the network; The parameters of the target network are defined; the error is solved using gradient descent to minimize the error and continuously optimize the learner's evaluation network; finally, after a certain number of training iterations, the participants' collaborative optimization strategy and the learner's target network are updated; during the training process using the D3QN algorithm, the steps completed by multiple evolvers in parallel include: first, randomly generating a warehouse environment path planning task for evaluation and performing strategy evaluation; then, using the current strategy obtained using the D3QN algorithm as the parent strategy, using an evolutionary algorithm to complete the strategy evolution and generate a dominant offspring strategy; finally, changing the original collaborative optimization strategy through soft updates.

2. The multi-agent path planning method according to claim 1, characterized in that, In the collaborative optimization strategy network model, the hidden value information is 0 at the initial time. In addition, if the agent does not find any neighboring agents in its field of vision, no communication occurs, and the information used for communication is used as the hidden value information.

3. The multi-agent path planning method according to claim 1, characterized in that, The specific steps for training the collaborative optimization strategy network model using the D3QN algorithm include: 1) Generating a logistics warehousing environment map based on the logistics warehousing environment, and designing the partial observation space, action space, and reward function for each agent; wherein, the step of generating the logistics warehousing environment map includes: designing a two-dimensional warehousing grid map based on the logistics warehousing environment, and setting the placement of shelves in the map according to the length, width, and shelf density of the warehousing grid map to obtain the logistics warehousing environment map; the step of designing the partial observation space includes: in the partially observable environment, assuming that the agent can only observe objects of size... The environment within the field of vision, Indicates the scale of the field of vision; in At any moment, intelligent agent The observed field of view information is in a binary matrix The representations are, respectively, the binary matrices of the shelf locations within the field of view, the binary matrices of the locations of other agents within the field of view, the binary matrices of the intersection locations within the field of view, and the binary matrices of the four-channel heuristic information; in Time-based intelligent agent We also need to obtain the unit direction vector pointing to the endpoint. And the distance from the destination in Manhattan Constructed local guiding vector The steps for designing the motion space include: in a 4-connected mesh environment, in Time-based intelligent agent Capable of performing 5 discrete actions These represent moving one cell grid in each of the four basic directions: up, down, left, and right, or remaining stationary; where, when the agent... When an agent performs an erroneous action, its current action is replaced by remaining still; the steps in designing the reward function include: Time-based intelligent agent Taking action The reward received afterwards for, In the formula, Penalty for movement; Penalty for immobility; Penalty for collision; 2) Defined as a reward for achieving cooperation targets; Time-based intelligent agent Partial observation values The hidden value information obtained after passing through the observation encoder module and the communication module is Used for action selection; assuming an intelligent agent The Q-network action value function is By value function and dominance function It consists of two parts, represented as follows: In the formula, These are the parameters of the overall Q-network. and These are the network parameters for the value function and the advantage function, respectively; In order to be in Time-based intelligent agent The set of all possible actions; the evaluation network is set in the algorithm. and target network Two Q-networks are used to eliminate maximization bias, where the evaluation network is used to determine the action at the next time step, and the target network is used to determine an estimate of the reward, denoted as... 。 4. The multi-agent path planning method according to claim 3, characterized in that, In the process of training a policy using the D3QN algorithm, the step of parallel evolution by multiple agents includes, 1) the policy evaluation step, which includes: assuming that each agent completes the process within a specified maximum number of steps. Successfully reached the finish line and obtained... The evaluation value; if an agent fails in a task The second collision yields... The evaluation value; assuming the Manhattan distance from the starting point to the ending point of an agent is... The number of steps it takes for the agent to reach the destination for the first time is If the agent reaches the specified maximum number of steps If a character fails to reach the finish line, then its number of steps is... Maximum number of steps The intelligent agent obtains The evaluation value is intelligent agent In collaborative optimization strategy Under the guidance of [the relevant authority], task planning is carried out, and the final overall evaluation value is expressed as [the result]. In the formula, Successful identification; collaborative optimization strategy The evaluation value is, In the formula, Represents the collaborative optimization strategy In The evaluation was conducted in a multi-agent path planning task within a warehouse environment. A set of agents participating in a multi-agent path planning task; 1) The number of agents participating in a multi-agent path planning task; 2) During the policy evolution process, the evolutionary process of an evolver is as follows: First, randomly generate... A multi-agent path planning task in a warehouse environment is used to evaluate the strategy; then, the parameters of the evaluation network obtained from reinforcement learning training are obtained. Collaborative optimization strategy And evaluate to obtain the initial parent strategy evaluation value. Next, based on the set maximum number of iterations, the original parent strategy is modified. Perform iterative mutation; where, assume the parent policy parameter of generation y is... Using random noise vectors generate The pseudo-child policy parameters are represented as follows: The evaluation value for each pseudo-offspring strategy is represented as follows: ; Calculate the approximate gradient change, the calculation expression is, ; Obtain the mutated policy parameters, ;in, The learning rate for the evolutionary algorithm; The mutation rate of the evolutionary algorithm; the strategy for the mutated offspring. An evaluation will be conducted, and if the evaluation value is... Greater than the initial parent strategy evaluation value If a policy evolution is successful, soft updates are used to evolve the policy, ending the evolutionary process and initiating a new one. If no dominant offspring policy is found through mutation within the maximum number of iterations, soft updates are not performed, and the policy evolves again.

5. The multi-agent path planning method according to claim 4, characterized in that, In the step of changing the original collaborative optimization strategy through soft updates, the soft update method specifically involves combining the dominant offspring strategy with the currently trained evaluation network and target network, expressed as follows: In the formula, The parameters for the dominant offspring strategy.

6. A multi-agent path planning system based on a logistics warehousing environment, characterized in that, The method for implementing the multi-agent path planning method of claim 1 includes: a data acquisition module for acquiring partial observation information of each agent at the current moment based on multiple agents in a logistics warehousing environment; and a path planning module for performing path planning based on the partial observation information using a trained collaborative optimization strategy network model to obtain the actions of each agent at the current moment. In the step of acquiring partial observation information of each agent at the current moment, for a selected agent, the partial observation information includes field-of-view information and a local guidance vector. The field-of-view information includes a binary matrix of shelf positions within the field of view, a binary matrix of other agent positions within the field of view, a binary matrix of intersection positions within the field of view, and a binary matrix of four-channel heuristic information. The four-channel heuristic information is used to indicate whether the selected agent is close to the endpoint when moving one grid position up, down, left, or right within the field of view. The grid positions are obtained based on a warehousing grid map obtained from the logistics warehousing environment. The local guidance vector consists of a unit direction vector pointing to the endpoint and a Manhattan distance from the endpoint. The collaborative optimization strategy network model is trained using a parallel training strategy of reinforcement learning and evolutionary learning.

7. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the multi-agent path planning method based on a logistics warehousing environment as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the multi-agent path planning method based on the logistics warehousing environment as described in any one of claims 1 to 5.

Images