Multi-agent cluster three-dimensional path planning and re-planning method and system

Through the combination of particle swarm algorithm and deep Q network, the problem of low efficiency and poor adaptability of agent path planning in complex and dynamic environments is solved, efficient three-dimensional path planning and re-planning are realized, and the maximum path of agent path is ensured. Excellence and real-time.

CN119043322APending Publication Date: 2024-11-29杭州智元研究院有限公司

Patent Information

Application Number
CN202411136762.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-19
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

The existing agent path planning algorithms have limitations when dealing with complex and dynamic environments, and it is difficult to adapt to environmental changes in real time, resulting in low path planning efficiency and poor re-planning capabilities, easy to fall into local optimal solutions, and unable to find global optimal paths.

Method used

Three-dimensional path planning is used to adopt particle swarm algorithm, and path re-planning is carried out in combination with deep Q network. Through the global search of particle swarm algorithm and the dynamic adaptability of DQN, a multi-agent cluster three-dimensional path planning and re-planning method is designed, and the gate is used. The grid method builds a three-dimensional environmental model, the particle swarm algorithm optimizes the path length and obstacle avoidance, and the DQN network optimizes dynamic paths.

Benefits of technology

Improve the efficiency and search speed of path planning, improve the path planning performance in complex dynamic environments, solve the performance problems of existing algorithms in limitations and dynamic changes, and ensure the optimality and real-time path of the agent sex.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119043322A_ABST
    Figure CN119043322A_ABST
Patent Text Reader

Abstract

The invention provides a multi-agent cluster three-dimensional path planning and re-planning method and a multi-agent cluster three-dimensional path planning and re-planning system. A multi-agent cluster three-dimensional path planning and re-planning method comprises the following steps: S1, constructing a map environment, and constructing a three-dimensional environment model through sensor data or map data; s2, aiming at the three-dimensional environment model, designing a particle swarm algorithm to carry out overall path planning so as to obtain a global three-dimensional path; s3, aiming at the three-dimensional environment model, designing a depth Q network to re-plan a global three-dimensional path to obtain an optimal three-dimensional path; and S4, converting the obtained optimal three-dimensional path into a motion instruction of the intelligent agent to obtain an actual execution path of the intelligent agent. Particle swarm algorithm global planning and DQN algorithm re-planning are adopted to carry out intelligent agent path planning, on one hand, the problem of limitation of an existing single-class algorithm on path planning is solved, and on the other hand, the problem that the existing algorithm is poor in performance in a dynamically changing scene is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of path planning for intelligent agent control, and more specifically, relates to a three-dimensional path planning and re-planning method and system for a multi-agent cluster. Background Art

[0002] With the development of technology, multi-agent clusters are widely used in home life, power inspection, aerospace and scientific research. In the process of performing tasks, agents can carry various sensor devices, use their unique technical characteristics to monitor environmental conditions in real time, determine their own positions, flexibly adjust their postures, and cleverly avoid various obstacles to successfully complete the task. To achieve all this, the agent must first plan a collision-free and shortest distance path, which is the premise and guarantee for the success of the task. However, in the real environment, agents face many uncontrollable factors in performing tasks. Uncertain situations and the dynamic nature of the environment make path planning complex and critical. Therefore, it is extremely important to study the path planning algorithm of agents. The goal of path planning is to design an optimal path from the starting point to the task point based on comprehensive consideration of factors such as energy consumption, avoiding collision risks and minimizing path length, so as to minimize energy consumption and ensure collision-free. An excellent path is the key to the successful execution of tasks by agents, and choosing a suitable path planning algorithm is crucial. An excellent path planning algorithm can not only ensure the optimality of the global path, but also respond to the emergence of dynamic obstacles in real time, thereby improving overall efficiency and ensuring the smooth completion of tasks. Therefore, the research on intelligent agent path planning algorithms is crucial to the success of modern intelligent agent applications.

[0003] At present, researchers at home and abroad have conducted extensive and in-depth research in the field of intelligent agent path planning algorithms. Path planning algorithms can be divided into three categories. The first category is the traditional intelligent agent path optimization algorithm, including artificial potential field algorithm (APF), fuzzy logic algorithm, rapidly expanding random tree algorithm (RRT), etc. The characteristic of this type of algorithm is to find an optimal path through mathematical modeling and optimization techniques based on map information and target location. The second category is the heuristic intelligent agent path optimization algorithm, which includes A* algorithm, Dijkstra algorithm, simulated annealing method, etc. This type of algorithm can quickly find a better path by using heuristic search and priority queue techniques. The last category is the swarm intelligence bionic path optimization algorithm, which is one of the most commonly used path planning methods. These algorithms are inspired by the behavior of biological groups in nature, including genetic algorithm (GA), ant colony algorithm (ACO), particle swarm algorithm (PSO), neural network algorithm, etc. This type of algorithm uses the principles of swarm intelligence and distributed computing to find high-quality paths in complex environments.

[0004] Although existing agent path planning algorithms perform well in many cases, they also have some problems. These problems may have a negative impact on the performance of the algorithms in specific situations. They sometimes get stuck in local optimal solutions and fail to find the global optimal path. In some cases, this may lead to less than ideal path planning results. Global search algorithms such as A* algorithm, Dijkstra algorithm and ant colony algorithm, although they can find the global optimal path, take a lot of time to search when dealing with large-scale and complex scenes, resulting in low overall search efficiency. This may be a challenge for real-time applications and large-scale agent tasks. In addition, in dynamic environments, existing algorithms may show poor re-planning capabilities when responding to emergencies. Agent path planning needs to adapt to environmental changes in a timely manner, such as avoiding obstacles or responding to changes in tasks, so the real-time and adaptability of the algorithm are crucial to the safety of the agent and the success of the task. Summary of the invention

[0005] The purpose of the present invention is to provide a multi-agent cluster three-dimensional path planning and re-planning method and system to solve the problems in the background technology.

[0006] The technical solution for achieving the purpose of the present invention is:

[0007] A multi-agent cluster three-dimensional path planning and replanning method, comprising:

[0008] Step S1, map environment construction, constructing a three-dimensional environment model through sensor data or map data;

[0009] Step S2, designing a particle swarm algorithm for overall path planning based on the three-dimensional environment model to obtain a global three-dimensional path;

[0010] Step S3, designing a deep Q network to replan the global three-dimensional path for the three-dimensional environment model to obtain the optimal three-dimensional path;

[0011] Step S4, converting the obtained optimal three-dimensional path into the motion instructions of the intelligent agent to obtain the actual execution path of the intelligent agent.

[0012] Furthermore, constructing a three-dimensional environment model through sensor data or map data specifically includes:

[0013] Use the grid method to construct a map, and use the grid to represent the actual map size. The three-dimensional information on the map is represented by multiple grid cells:

[0014]

[0015] Among them, the ceil function represents rounding to the nearest integer, m, n, and h represent the length, width, and height of the map area, l is the side length of the grid unit, and each grid is represented by the corresponding coordinates; when there is no obstacle in the grid, it is represented as a movable area, and the grid is assigned to 0; when there is an obstacle in the grid, it is represented as an immovable area, and the grid is assigned to -1, and constraints are set based on the flight conditions of the agent.

[0016] Furthermore, designing a particle swarm algorithm for overall path planning includes: taking speed and position as attributes of the particle swarm algorithm, each particle individually searches for the optimal solution in the search space, and records it as the current individual extreme value, and shares the individual extreme value with other particles in the entire particle swarm, and finds the optimal individual extreme value as the current global optimal solution of the entire particle swarm, and all particles in the particle swarm adjust their own speed and position according to the current individual extreme value they find and the current global optimal solution shared by the entire particle swarm to obtain a global three-dimensional path.

[0017] Furthermore, the step S2 specifically includes:

[0018] Step S201, initializing the particle swarm, including the particle swarm size, particle dimension, number of iterations, inertia weight, learning factor, and iteration step range. During initialization, a group of particles is randomly generated, and a random starting position and speed are assigned to each particle;

[0019] Step S202: For each particle, the path length and obstacle avoidance fitness are calculated as:

[0020]

[0021] Among them, fit_value is the fitness value of the path length, distance represents the path distance between each particle and the target point. The larger the distance, the lower the corresponding fitness. collision represents the fitness coefficient of collision.

[0022] Step S203: for each particle, update its individual best position P id,pbest and the group's best position P d,gbest :

[0023] P id,pbest =(p i1,pbest p i2,pbest , ..., p iD,pbest )

[0024] P a,gbest =(p 1,gbest , p 2,gbest , ..., p D,gbest );

[0025] Step S204, update the speed and position of each particle to:

[0026]

[0027] Among them, ω is the inertia factor, which is a non-negative value. The larger its value is, the stronger the global optimization ability is and the weaker the local optimization ability is. On the contrary, the weaker the global optimization ability is, the stronger the local optimization ability is. c 1 is the individual learning factor, c 2 is the group learning factor, r 1 With r 2 is a random number in the interval [0, 1];

[0028] S205: Iteration status judgment. If the termination condition is not met, the current population is iteratively updated and steps S202, S203, and S204 are repeated; otherwise, a path with the best fitness value is selected from the particle swarm as the global three-dimensional path of the agent.

[0029] Furthermore, the updating strategy of the inertia factor is:

[0030]

[0031] Among them, G k is the maximum number of iterations, ω ini is the initial inertia weight, ω end is the inertia weight when it iterates to the maximum number of iterations, and g is the current number of iterations.

[0032] Furthermore, the design of a deep Q network to replan the global three-dimensional path specifically includes:

[0033] Step S301, construct the state space s as:

[0034] s={p,v,z,β}

[0035] Among them, p represents the position of the agent, v represents the velocity of the agent, z represents the posture of the agent, and β represents the angular velocity of the agent;

[0036] Step S302, constructing the action space A as:

[0037] A={u,d,l,r,f,b}

[0038] Among them, u means moving upward, d means moving downward, l means moving left, r means moving right, f means moving forward, and b means moving backward;

[0039] Step S303: Input the action of the agent into the DQN network. DQN uses the Q-value function to estimate the value of taking an action in a given state, which is:

[0040] Q(s,a;w)≈Q(s,a)

[0041] a t = argmaxQ(s t ,a;w)

[0042] Among them, Q(s,a) represents the optimal action value function, and Q(s,a;w) represents the action value function in the form of a neural network. t represents the action at time t, and w is the parameter of the neural network;

[0043] Calculate the target Q value;

[0044] Step S304, determine the next action according to the decision of DQN and add it to the path planning;

[0045] Step S305, DQN uses experience replay to store and reuse previous experience learning. The agent stores the experience tuple (s, a, r, s') in the experience buffer, where s represents the current state, a represents the action taken, r represents the reward obtained, and s' represents the next state; during training, the agent randomly extracts a certain amount of experience tuples from the experience pool for learning;

[0046] Step S306, design a loss function, and use the gradient descent method to optimize the network parameters, and obtain the optimal three-dimensional path through the optimized DQN.

[0047] Furthermore, the Q-value function uses the result of time difference learning as the true label, which is:

[0048] Q(s t ,a t ; w)≈r t +γ*Q(s t+1 ,a t+1 ; w)

[0049] Among them, r t Is performing action a t The reward returned is γ, which is a discount factor with a value range of [0, 1], s t+1 is the state space at time t+1, a t+1 Action at time t+1;

[0050] The target Q value is:

[0051]

[0052] Furthermore, the loss function is:

[0053]

[0054] Among them, Q(s t ,a t ; w) is the predicted Q value of the current Q-network, y t is the target Q value calculated by the target network.

[0055] Furthermore, the gradient descent method updates the parameters of the neural network as follows:

[0056]

[0057] Where θ represents the coefficient.

[0058] A multi-agent cluster three-dimensional path planning and re-planning system for implementing the method includes a map environment construction unit, a global path planning unit, a re-planning unit and a path mapping unit, wherein:

[0059] The map environment construction unit constructs a three-dimensional environment model through sensor data or map data;

[0060] The global path planning unit is used to design a particle swarm algorithm to perform overall path planning to obtain a global three-dimensional path;

[0061] The re-planning unit is used to design a deep Q network to re-plan the global three-dimensional path to obtain an optimal three-dimensional path;

[0062] The path mapping unit is used to convert the obtained optimal three-dimensional path into the motion instruction of the intelligent agent to obtain the actual execution path of the intelligent agent.

[0063] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention designs and improves the particle swarm algorithm and the DQN algorithm according to the scenario, and reduces the search time of the global path and improves the overall search efficiency by taking advantage of the high efficiency and fast convergence speed of the particle swarm algorithm; and improves the path planning performance of the intelligent agent in a complex and dynamic environment by taking advantage of the strong adaptability of the deep Q network to the dynamic environment. The present invention solves the limitation problem of the existing single-class algorithm in path planning, and on the other hand solves the problem that the existing algorithms perform poorly in dynamically changing scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 This is the overall flow chart of 3D path planning and re-planning.

[0065] Figure 2 This is the flow chart of particle swarm algorithm.

[0066] Figure 3 This is the flow chart of the DQN algorithm. DETAILED DESCRIPTION

[0067] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0068] Figure 1 is a flow chart of the three-dimensional path planning and re-planning of the intelligent agent of the present invention, such as Figure 1 As shown, the multi-agent cluster three-dimensional path planning and re-planning method of the present invention includes the following steps:

[0069] S1: Map environment construction. Create a three-dimensional environment model through sensor data or map data, including terrain, obstacles, restrictions, etc., to abstract the actual environment and simplify the movement space of the intelligent agent. Use the grid method to build the map, represent the actual map size with a grid, and use multiple grid units to represent the three-dimensional information on the map:

[0070]

[0071] Among them, the ceil function represents rounding to the nearest integer, the map area is m*n*h, the side length of the grid unit is l, and each grid is represented by the corresponding coordinates.

[0072] When there is no obstacle in the square, it is represented as a movable area, and the value of the grid is 0; when there is an obstacle in the square, it is represented as an immovable area, and the value of the grid is -1:

[0073]

[0074] The agent needs to be subject to some constraints when moving in the map, such as the flight height fly_h of the agent needs to be greater than a certain range, and the minimum distance dis_h between the agent and the obstacle needs to be greater than a certain value.

[0075] S2: Global path planning. Use the particle swarm algorithm for overall path planning. The particle swarm algorithm contains two attributes: speed and position. Speed ​​represents the speed of movement, and position represents the direction of movement. Each particle searches for the optimal solution in the search space individually, and records it as the current individual extreme value. The individual extreme value is shared with other particles in the entire particle swarm, and the optimal individual extreme value is found as the current global optimal solution of the entire particle swarm. All particles in the particle swarm adjust their speed and position according to the current individual extreme value they find and the current global optimal solution shared by the entire particle swarm, thereby obtaining a global three-dimensional path.

[0076] S3: Replanning. The local path is replanned using a deep Q network (DQN), and a new motion path is obtained by inputting the current path nodes into the neural network. When the global path encounters unknown obstacles or the target point changes dynamically during the movement, it is determined whether to replan it according to the threshold parameter θ, and finally the optimal three-dimensional path is found.

[0077] S4: Path mapping. The optimal three-dimensional path is converted into the motion instructions of the agent, and the actual execution path of the agent is obtained through the parameters such as the speed and angular velocity of the agent.

[0078] Figure 2 This is a flow chart of the global path planning based on the particle swarm algorithm of the present invention. Figure 2 As shown, the process of using the particle swarm algorithm to perform path planning in step S2 of the present invention includes the following steps:

[0079] S201: Initialize the particle swarm. Before starting path planning, the particle swarm needs to be initialized, including the particle swarm size, particle dimension, number of iterations, inertia weight, learning factor, and iteration step range. Each particle represents a possible path solution. During initialization, a group of particles is randomly generated and a random starting position and speed are assigned to each particle.

[0080] The position of a particle can be expressed as:

[0081]

[0082] Where i represents the i-th particle in the particle swarm, d represents the d-th dimension in the D-dimensional particle dimension, k represents the k-th iteration,

[0083] Similarly, the velocity of a particle can be expressed as:

[0084]

[0085] S202: Fitness calculation. For each particle, calculate its fitness value, that is, the quality of the path solution. The fitness value calculation includes path length and obstacle avoidance. By measuring the total length of the path, the shortest path is found; by evaluating whether obstacles are avoided on the path, the safe flight of the intelligent agent is ensured. The fitness calculation formula is:

[0086]

[0087] Among them, distance represents the path distance of each particle from the target point. The larger the distance, the lower the corresponding fitness. Collision represents the fitness coefficient of the collision. If a collision occurs, the coefficient is 0, otherwise, the coefficient is 0.1.

[0088] S203: Update individual and group positions. For each particle, update its individual best position (denoted as pbest) and group best position (denoted as gbest). The individual best position is the best path solution found by the particle itself in history, while the group best position is the best path solution found by the entire particle group. The optimal position searched by the i-th particle is:

[0089] P id,pbest =(p i1,pbest , p i2,pbest , ..., p iD,pbest )

[0090] The optimal position searched by the group is:

[0091] P d,gbest =(p 1,gbest , p 2,gbest , ..., p D,gbest )

[0092] S204: Update particle speed and position. According to the principle of particle swarm algorithm, update the speed and position of each particle to try a better path solution. The speed and position are updated using the following formula:

[0093]

[0094] Among them, ω is the inertia factor, which is a non-negative value. The larger its value, the stronger the global optimization ability and the weaker the local optimization ability; conversely, the weaker the global optimization ability and the stronger the local optimization ability. 1 is the individual learning factor, c 2 is the group learning factor, r 1 With r 2 It is a random number in the interval [0, 1], which is obtained by the random function rand, so as to increase the randomness of the search.

[0095] Dynamic ω can achieve better optimization results than fixed values. Dynamic ω can change linearly during the search process of the particle swarm algorithm, or it can change dynamically according to the measurement function in the particle swarm. Here, a linear decreasing weight strategy is adopted to adjust the global and local search capabilities of the particle swarm algorithm. This strategy can be expressed as:

[0096]

[0097] Among them, G k is the maximum number of iterations, ω ini is the initial inertia weight, ω end is the inertia weight when it iterates to the maximum number of iterations, and g is the current number of iterations.

[0098] S205: Iteration status judgment. During the iteration process, check whether the termination condition is met. The termination condition is to reach the maximum number of iterations or to reach a satisfactory path solution. If the termination condition is not met, iteratively update the current population and repeat steps S202, S203, and S204; otherwise, output the optimal path result.

[0099] S206: Output the optimal path result. After the termination condition is met, a path solution with the best fitness value is selected from the particle swarm as the optimal path planning of the intelligent agent.

[0100] Figure 3 This is a flowchart of the re-planning based on DQN in the present invention. Figure 3 As shown, the process of using DQN to perform local path planning in step S3 of the present invention includes the following steps:

[0101] S301: Define the state space. Define the current state of the agent cluster as part of the state space, including the position, speed, posture and other information of each agent. This state information will be used to construct the state representation of DQN. Use the symbol s to represent the state space:

[0102] s={p,v,z,β}

[0103] Among them, p represents the position of the agent, v represents the velocity of the agent, z represents the posture of the agent, and β represents the angular velocity of the agent.

[0104] S302: Define the action space. Define the set of actions that no one can take. Actions represent the direction in which the agent moves, such as up, down, left, right, forward, backward, etc. The action space is represented by symbol A:

[0105] A={u,d,l,r,f,b}

[0106] Among them, u means moving up, d means moving down, l means moving left, r means moving right, f means moving forward, and b means moving backward.

[0107] S303: Input the deep Q network. The action of the agent is input into the DQN network for value evaluation and next action reasoning. DQN uses the Q-value function to estimate the value of taking an action in a given state. The Q-value function is represented by Q(s,a;w), which represents the expected cumulative reward for taking action a in state s, a∈A, and w is the parameter of the neural network. Use the ε-greedy strategy to balance between exploration and exploitation. Select a random action with probability ε, and select the action with the highest Q value with probability 1-ε. The goal of DQN is to learn and fit the Q-value function through a neural network to ensure that the correct action can be obtained through the output of the network.

[0108] Q(s,a;w)≈Q(s,a)

[0109] a t = argmaxQ(s t ,a;w)

[0110] Among them, Q(s,a) represents the optimal action-value function, and Q(s,a;w) represents the action-value function in the form of a neural network.

[0111] Using the result of time difference learning (TD,Learning) as the true label, the target Q-value function can be expressed as:

[0112] Q(s t ,a t ; w)≈r t +γ*Q(s t+1 ,a t+1 ; w)

[0113] Among them, r t Is performing action a t The reward received is due to the inability to accurately calculate r t Therefore, only one reward at a time is used for approximate calculation. γ is a discount factor with a value range of [0, 1], which is used to balance current benefits and future benefits. The TD target can finally be expressed as:

[0114]

[0115] S304: Output the next action. Determine the next action based on the decision of DQN and add it to the path planning.

[0116] S305: Use experience replay. In order to stabilize training, DQN uses experience replay to store and reuse previous experience. The agent stores the experience tuple (s, a, r, s') in the experience buffer, where s represents the current state, a represents the action taken, r represents the reward obtained, and s' represents the next state. During training, the agent randomly extracts a certain amount of experience tuples from the experience pool for learning, so that it can learn both current and past experiences. At the same time, random extraction can also reduce overfitting caused by the correlation between samples.

[0117] S306: Optimize network parameters. In order to train the Q-network, a loss function needs to be defined, using the mean squared error (MSE) to measure the difference between the predicted Q value and the actual Q value of the Q-network. Therefore, the loss function can be expressed as:

[0118]

[0119] Among them, Q(s t ,a t ; w) is the predicted Q value of the current Q-network, y t It is the target Q value calculated by the target network.

[0120] Use the gradient descent method to optimize the parameters w of the Q-network to minimize the loss function, where θ represents the descent factor parameter in the gradient descent process, which can be set by yourself. The formula for updating the gradient can be expressed as:

[0121]

[0122] Where θ represents the coefficient.

[0123] This embodiment also provides a multi-agent cluster three-dimensional path planning and re-planning system for implementing the method, including a map environment construction unit, a global path planning unit, a re-planning unit and a path mapping unit, wherein:

[0124] The map environment construction unit constructs a three-dimensional environment model through sensor data or map data;

[0125] The global path planning unit is used to design a particle swarm algorithm to perform overall path planning to obtain a global three-dimensional path;

[0126] The re-planning unit is used to design a deep Q network to re-plan the global three-dimensional path to obtain an optimal three-dimensional path;

[0127] The path mapping unit is used to convert the obtained optimal three-dimensional path into the motion instruction of the intelligent agent to obtain the actual execution path of the intelligent agent.

[0128] The present invention designs a particle swarm algorithm for global planning and a DQN algorithm for replanning the path planning of the intelligent agent, which solves the limitation problem of the existing single-class algorithm in path planning on the one hand and solves the problem that the existing algorithm performs poorly in dynamically changing scenes on the other hand.

[0129] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0130] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A three-dimensional path planning and replanning method for multi-agent clusters, It is characterized in that include: Step S1, map environment construction, constructing a three-dimensional environment model through sensor data or map data; Step S2, designing a particle swarm algorithm for overall path planning based on the three-dimensional environment model to obtain a global three-dimensional path; Step S3, designing a deep Q network to replan the global three-dimensional path for the three-dimensional environment model to obtain the optimal three-dimensional path; Step S4, converting the obtained optimal three-dimensional path into the motion instructions of the intelligent agent to obtain the actual execution path of the intelligent agent.

2. A multi-agent cluster three-dimensional path planning and re-planning method according to claim 1, It is characterized in that Constructing a 3D environment model using sensor data or map data specifically includes: Use the grid method to construct a map, and use the grid to represent the actual map size. The three-dimensional information on the map is represented by multiple grid cells: Among them, the ceil function represents rounding to the nearest integer, m, n, and h represent the length, width, and height of the map area, l is the side length of the grid unit, and each grid is represented by the corresponding coordinates; when there is no obstacle in the grid, it is represented as a movable area, and the grid is assigned to 0; when there is an obstacle in the grid, it is represented as an immovable area, and the grid is assigned to -1, and constraints are set based on the flight conditions of the agent.

3. A multi-agent cluster three-dimensional path planning and re-planning method according to claim 1, It is characterized in that Designing a particle swarm algorithm for overall path planning includes: using speed and position as attributes of the particle swarm algorithm, each particle searches for the optimal solution individually in the search space, and records it as the current individual extreme value, and shares the individual extreme value with other particles in the entire particle swarm, finding the optimal individual extreme value as the current global optimal solution of the entire particle swarm, and all particles in the particle swarm adjust their speed and position according to the current individual extreme value they find and the current global optimal solution shared by the entire particle swarm to obtain a global three-dimensional path.

4. A multi-agent cluster three-dimensional path planning and re-planning method according to claim 3, It is characterized in that The step S2 specifically includes: Step S201, initializing the particle swarm, including the particle swarm size, particle dimension, number of iterations, inertia weight, learning factor, and iteration step range. During initialization, a group of particles is randomly generated, and a random starting position and speed are assigned to each particle; Step S202: For each particle, the path length and obstacle avoidance fitness are calculated as: Among them, fit_value is the fitness value of the path length, distance represents the path distance between each particle and the target point. The larger the distance, the lower the corresponding fitness. collision represents the fitness coefficient of collision. Step S203: for each particle, update its individual best position P id,pbest and the group's best position P d,gbest : P id,pbest =(p i1,pbest ,p i2,pbest ,…,p iD,pbest ) P d,gbest =(p 1,gbest ,p 2,gbest ,…,p D,gbest ); Step S204, update the speed and position of each particle to: Among them, ω is the inertia factor, which is a non-negative value. The larger its value is, the stronger the global optimization ability is and the weaker the local optimization ability is. On the contrary, the weaker the global optimization ability is, the stronger the local optimization ability is. c 1 is the individual learning factor, c 2 is the group learning factor, r 1 With r 2 is a random number in the interval [0, 1]; S205: Iteration status judgment. If the termination condition is not met, the current population is iteratively updated and steps S202, S203, and S204 are repeated; otherwise, a path with the best fitness value is selected from the particle swarm as the global three-dimensional path of the agent.

5. A multi-agent cluster three-dimensional path planning and re-planning method according to claim 4, It is characterized in that The updating strategy of the inertia factor is: Among them, G k is the maximum number of iterations, ω ini is the initial inertia weight, ω end is the inertia weight when it iterates to the maximum number of iterations, and g is the current number of iterations.

6. A multi-agent cluster three-dimensional path planning and re-planning method according to claim 1, It is characterized in that Designing a deep Q network to replan the global three-dimensional path specifically includes: Step S301, construct the state space s as: s={p,v,z,β} Among them, p represents the position of the agent, v represents the velocity of the agent, z represents the posture of the agent, and β represents the angular velocity of the agent; Step S302, constructing the action space A as: A={u,d,l,r,f,b} Among them, u means moving upward, d means moving downward, l means moving left, r means moving right, f means moving forward, and b means moving backward; Step S303: Input the action of the agent into the DQN network. DQN uses the Q-value function to estimate the value of taking an action in a given state, which is: Q(s,a;w)≈Q(s,a) a t =argmaxQ(s t ,a;w) Among them, Q(s,a) represents the optimal action value function, and Q(s,a;w) represents the action value function in the form of a neural network. t represents the action at time t, and w is the parameter of the neural network; Calculate the target Q value; Step S304, determine the next action according to the decision of DQN and add it to the path planning; Step S305, DQN uses experience replay to store and reuse previous experience learning. The agent stores the experience tuple (s, a, r, s') in the experience buffer, where s represents the current state, a represents the action taken, r represents the reward obtained, and s' represents the next state; during training, the agent randomly extracts a certain amount of experience tuples from the experience pool for learning; Step S306, design a loss function, and use the gradient descent method to optimize the network parameters, and obtain the optimal three-dimensional path through the optimized DQN.

7. A multi-agent cluster three-dimensional path planning and re-planning method according to claim 6, It is characterized in that The Q-value function uses the result of temporal difference learning as the true label, which is: Q(s t ,a t ;w)≈r t +γ*Q(s t+1 ,a t+1 ;w) Among them, r t Is performing action a t The reward returned is γ, which is a discount factor with a value range of [0, 1], s t+1 is the state space at time t+1, a t+1 Action at time t+1; The target Q value is:

8. A multi-agent cluster three-dimensional path planning and re-planning method according to claim 6, It is characterized in that The loss function is: Among them, Q(s t ,a t ; w) is the predicted Q value of the current Q-network, y t is the target Q value calculated by the target network.

9. A multi-agent cluster three-dimensional path planning and re-planning method according to claim 8, It is characterized in that The gradient descent method updates the parameters of the neural network as follows: Where θ represents the coefficient.

10. A multi-agent cluster three-dimensional path planning and re-planning system implementing any of the methods described in claims 1-9, It is characterized in that It includes a map environment construction unit, a global path planning unit, a replanning unit and a path mapping unit, wherein: The map environment construction unit constructs a three-dimensional environment model through sensor data or map data; The global path planning unit is used to design a particle swarm algorithm to perform overall path planning to obtain a global three-dimensional path; The re-planning unit is used to design a deep Q network to re-plan the global three-dimensional path to obtain an optimal three-dimensional path; The path mapping unit is used to convert the obtained optimal three-dimensional path into the motion instruction of the intelligent agent to obtain the actual execution path of the intelligent agent.

Citation Information

Patent Citations

  • Unmanned aerial vehicle path planning method based on transfer learning strategy deep Q-network

    CN110703766A

  • Robot path planning method in complex environment

    CN113467472A

  • Path planning method and system based on deep reinforcement learning

    CN114415663A

  • Driving condition local path planning method based on safety potential field and DQN algorithm

    CN115031753A

  • AGV path planning method based on deep reinforcement learning

    CN117850415A

Cited By

  • Target paving path planning method and device based on deep learning and image recognition

    CN120952298A