Multi-agent deep reinforcement learning path planning method based on improved a* heuristic

Abstract

The application discloses a multi-agent deep reinforcement learning path planning method based on an improved A* heuristic, and belongs to the technical field of communication. The method first generates prior knowledge by using the heuristic search capability of a Q-Learning algorithm, so as to accelerate the training process of a MADQN algorithm. In combination with prior Q values and estimated Q values, the Q value function of the whole system is optimized, a new action selection strategy and a multi-target reward function are designed, the dynamic decision process of a UAV is improved, the learning efficiency and task execution effect of an agent in a complex environment are significantly improved by comprehensively considering exploration rewards, obstacle avoidance rewards and multi-agent cooperation rewards. The algorithm of the application is excellent in convergence speed, task success rate and path planning efficiency in comparison with traditional algorithms in a multi-agent path planning task, and further proves the high efficiency and robustness of the algorithm in a complex environment.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of communication, and particularly relates to a multi-agent deep reinforcement learning path planning method based on an improved A* heuristic. BACKGROUND

[0002] With the continuous progress of intelligent technology, unmanned aerial vehicles (UAVs) as a kind of high-mobility aerial platform have wide applications in agricultural monitoring, map making, disaster rescue and other fields. However, in a complex environment, a single UAV has low fault tolerance and limited work capacity, and other deficiencies gradually appear. Therefore, the research on multi-agent path planning problems becomes increasingly important.

[0003] At present, the multi-agent path planning technology is mainly based on the classical MAPF (multi-agent path finding) algorithm. The classical MAPF algorithm such as the A* search algorithm optimizes the search process by introducing a heuristic function. Based on the conflict-based search algorithm (CBS), a high-low double-layer search is used to build a binary search tree to add restrictions to eliminate conflicts, and compared with other methods, the CBS can solve the problem faster. The increasing cost tree search algorithm (ICTS) finds the optimal cost of each agent according to the high-level search, and then verifies the validity of these costs through the low-level search. The hybrid strategy can maintain good performance in different types of maps and scale environments.

[0004] Recent researches show that the classical MAPF algorithm performs well in solving path planning problems. However, these methods usually rely on a central planner to obtain complete map information and the positions of all agents. This centralized method needs to collect a large amount of data to plan the optimal path, resulting in huge consumption of computing resources. With the breakthrough of artificial intelligence, the MAPF algorithm based on reinforcement learning (RL) has great potential in solving dynamic changing environments. Reinforcement learning exhibits significant adaptability and scalability through the interaction and exploration of agents with unknown environments. Deep reinforcement learning (DRL) combined with deep neural networks further enhances adaptability, enabling agents to make accurate decisions in complex environments, making it a powerful tool to replace traditional MAPF methods.

[0005] Wherein the deep Q-network (Deep Q-Network, DQN) algorithm uses a deep neural network to fit the action value function Q of the Q-learning algorithm (QL) to generate estimated Q values, which significantly improves the ability to process high-dimensional perception inputs and overall learning efficiency. At the same time, the DQN algorithm introduces two key technologies: experience replay and target network. The experience replay mechanism maintains an experience replay pool that stores the state, action, reward and next state observed by the agent in the environment, and randomly selects experience samples from it to train the network. The target network provides a relatively stable target Q value, reducing the training fluctuations caused by excessive frequent target Q values during the learning process. Thus breaking the time correlation between samples, improving the stability of learning, optimizing the long-term decision-making process.

[0006] However, in the path planning research of multi-UAV systems, the traditional DQN algorithm still has many limitations. First, with the increase in the number of agents, the computational complexity rises sharply, making it difficult to effectively execute in real-time applications. Second, in a dense obstacle environment, the agent only focuses on local path optimization, which cannot guarantee the global optimal path and is prone to path conflicts, which may lead to deadlocks or bottlenecks. In addition, the algorithm assumes that the environment is static, making it difficult to adapt to dynamic changes such as obstacle movement or environmental mutations, affecting the overall path planning effect. Finally, the scalability of the traditional DQN algorithm is poor, and in a multi-agent environment, it has high demands on computing resources and memory, limiting its application in resource-constrained environments. SUMMARY

[0007] The present application aims to overcome the above-mentioned deficiencies and shortcomings of the prior art, and provides a multi-agent deep reinforcement learning path planning method based on improved A* heuristic. The method uses an improved Q-learning algorithm as a heuristic function to generate static prior path information and a Q table, which reduces the initial exploration time and computing resources of the MADQN algorithm, and improves the efficiency and stability of the initial training.

[0008] The technical scheme adopted by the present application to solve the technical problems is: a multi-agent deep reinforcement learning path planning method based on improved A* heuristic, which comprises the following steps:

[0009] Step 1: Generate prior knowledge based on the QH-A* algorithm;

[0010] Step 2: Optimize the MADQN model, use the prior knowledge explored by the QH-A* algorithm to generate global information, design dynamic weight parameters, and use prior Q values to optimize the multi-agent Q value function.

[0011] Step 3: Generate prior knowledge by exploring the unknown environment based on the heuristic function of the improved Q-learning algorithm using the QH-A* algorithm, and store it in the prior replay pool D * .

[0012] Further, the step 1 includes using the improved Q-learning algorithm as a heuristic function h(s), defined as:

[0013] h(s)=r+γmax a Q(s,a)+f(N) (2)

[0014] Where f(N) is the initial exploration reward, specifically represented as:

[0015]

[0016] Where N(s,a) is the number of times the action a is performed in state s, and β is a constant used to encourage exploration of unknown environments and obtain global information.

[0017] Design g(s), which represents the path cost of the current state, specifically represented as:

[0018] g(s)=g(s t-1 )+g a (s t-1 ,s) (4)

[0019] Where s t-1 represents the previous state, and g a (s t-1 ,s) represents the cost of taking action a to reach the current state s.

[0020] In summary, the cost function of the QH-A* algorithm is:

[0021] f(s)=g(s)+h(s) (5)

[0022] Design a prior replay pool D * to store the static prior path information of the UAV , including the current state s * , the action taken a * , the reward obtained r * , and the next state s′ * , s * is the current state, represented as:

[0023] s* = {(x, y), (x tg ,y tg ), t, (x obs ,y obs ), done} (6)

[0024] In formula (6), the current position coordinates (x i ,y i ) of the UAV i; the target position coordinates (x tg ,y tg ); the time step information t of the current position; the position coordinate set (x obs ,y obs ) of the potential collision obstacle on the path; the task completion state done, which judges whether the UAV has reached the destination, if yes, the state value is 1; otherwise, 0;

[0025] a * is the optimal action taken from the current state s * to the next moment state s′ * based on QH-A*, represented as a * = {dx, dy}, r * is the reward obtained by executing the action a * ;

[0026] s′ * is the new state generated by the UAV executing the action a * , specifically represented as:

[0027] s′ * = {(x i + dx, y i + dx), (x tg ,y tg ), t′, (x obs ′, y obs ′)} (7)

[0028] In formula (7), t′ is the time step information of the UAV path, and (x i + dx, y i + dx) is the new position coordinates;

[0029] Based on the heuristic function of formula (2), the prior Q value of each state action is extracted from the Q table, Q p The expression is:

[0030]

[0031] Further, the step 2 comprises balancing exploration and exploitation in the decision-making process by the agent through the action selection strategy based on the cumulative reward dynamic adjustment to adapt to the learning needs of different stages, designing an improved multi-objective reward function, specifically comprising the following steps:

[0032] Step 2-1: Optimize the multi-agent Q value based on prior knowledge, generate global information τ according to the prior information stored in the prior playback pool D * , the global information τ(i) of the i-th agent is expressed as:

[0033]

[0034] Wherein, is the distance between the unmanned aerial vehicle i and the destination, expressed as:

[0035]

[0036] The distance between the unmanned aerial vehicle i and the boundary and obstacles is expressed as:

[0037]

[0038] A dynamic weight parameter λ(t) is introduced to combine the reward obtained by the agent from the interaction with the unknown environment for learning and training:

[0039]

[0040] Wherein, N max is the maximum training round, and k is the current training round;

[0041] Through λ(t), the weights of the prior Q value Q τ and the estimated Q value in the global Q value Q P are reasonably adjusted according to the number of training rounds:

[0042] Q τ (s,a; θ) = (1- λ(t)) Q p (s,a) + λ(t) Q θ (s,a; θ) (13)

[0043] At the same time, based on the global information τ, a weight ω i based on the target distance and obstacles is introduced, expressed as:

[0044]

[0045] Wherein, α is a constant used to adjust the importance of the target distance;

[0046] The weight ω i, the expression is:

[0047]

[0048] The joint value function Q of the system as a whole is Q total , the expression is:

[0049]

[0050] Using the loss function L in formula (14) θ , update the deep neural network parameters θ, the expression is:

[0051]

[0052] In formula (17), Y t i is the target Q value of each agent, the expression is:

[0053]

[0054] Step 2-2: Based on the cumulative reward dynamic adjustment of the action selection strategy, by adjusting the random action probability ε adaptively, the dynamic balance of exploration and utilization is achieved;

[0055] In the early stage, the system focuses on exploration to obtain comprehensive environmental information and build more accurate estimated Q value; in the later stage, the system focuses on using known information to improve path planning efficiency;

[0056]

[0057] In formula (19), k is the current training round, ε min is the minimum value of ε set, ε o is the initial value of ε, R k is the cumulative reward of the kth round of training, is the average reward of the first k rounds of training, is the average reward of the first k-1 rounds of training, and N is the total number of training rounds;

[0058] According to the optimized exploration strategy, the system can balance exploration and utilization at different stages, so as to select better actions and get:

[0059]

[0060] Step 2-3: Establish a reward function based on multi-objective coordination, which consists of exploration reward, obstacle avoidance reward and multi-agent cooperation reward; Specifically, the reward function R consists of three parts:

[0061] R = R T + R O + RS (21)

[0062] wherein, R T is an exploration reward; R O is an obstacle avoidance reward; R S is a multi-agent coordination reward.

[0063] Further, the step 2-3 comprises:

[0064] (1) an exploration reward function

[0065] The exploration reward function R T is designed to encourage the UAV to move towards the target point, approach the target point, and apply negative feedback to reduce invalid exploration using positive feedback.

[0066]

[0067] wherein, d t ' is the distance to the target point at the next time;

[0068] (2) an obstacle avoidance reward function

[0069] The obstacle avoidance reward function R O is designed to ensure flight safety. When the distance between the UAV and the nearest obstacle is less than s f , the UAV will be punished, and the MADQN path selection will be optimized to avoid collision risks.

[0070]

[0071] wherein, s f is the safety distance of the UAV from the obstacle;

[0072] (3) a multi-agent coordination reward function

[0073] The multi-agent coordination reward function R S is designed to coordinate the path planning between UAVs to prevent collisions between UAVs. The distance between UAV i and UAV j is :

[0074]

[0075] Further, the step 3 comprises: according to the optimized MADQN model, calling D * in the priori replay pool, performing initial learning, adjusting the weights of the priori Q value Q p and the estimated Q value according to the training rounds, generating global information τ based on priori knowledge, considering the obstacle and target information in the current environment, and optimizing the joint Q value of the system.

[0076] Beneficial effects:

[0077] 1. The present application proposes a path search algorithm named QH-A*(Q-Heuristic A* Algorithm), which uses an improved Q-learning algorithm as a heuristic function to generate static prior path information and a Q table, thereby reducing the initial exploration time and computational resources of the MADQN algorithm, improving the efficiency and stability in the early stage of training.

[0078] 2. The present application designs a multi-agent deep Q network (QH-A*-MADQN) path planning algorithm combined with QH-A* prior knowledge, reasonably adjusts the weights of prior knowledge Q values and estimated Q values, generates system overall Q values, realizes the collaborative optimization between agents, further optimizes the action exploration strategy and multi-objective reward function, balances the exploration and utilization relationship, and improves the autonomous decision-making ability of the unmanned aerial vehicle, so as to be more effectively applied to the complex environment of multiple unmanned aerial vehicles.

[0079] 3. Through the simulation experiment of multi-agent path planning, the QH-A*-MADQN algorithm proposed in the present application is verified, which is significantly superior to the traditional algorithm in terms of path planning efficiency, convergence speed, task success rate and adaptability to dynamic environment, further proving its high efficiency and robustness in complex environments. BRIEF DESCRIPTION OF DRAWINGS

[0080] Figure 1 The present application generates a schematic diagram of prior knowledge.

[0081] Figure 2 The present application is a flowchart of the QH-A*-MADQN algorithm. DETAILED DESCRIPTION

[0082] The present application will be further described in detail in conjunction with the accompanying drawings of the specification.

[0083] The present application is a multi-agent deep reinforcement learning path planning method based on improved A* heuristic, which includes the following steps:

[0084] Step 1: Generate prior knowledge based on QH-A* algorithm;

[0085] A* algorithm is a commonly used global path planning algorithm in multi-agent path planning, and the cost function of A* algorithm is:

[0086] f(n)=g(n)+h(n) (1)

[0087] Wherein, g(n) is the actual cost from the starting point to the current node, and h(n) is the estimated cost from the current node to the target.

[0088] However, the heuristic function of the traditional A* algorithm leads to insufficient searching in unexplored regions, thus affecting the global optimality of path planning. Furthermore, the A* algorithm requires frequent path recalculation in dynamic environments, resulting in reduced efficiency and accuracy in path planning.

[0089] To address the above problems, this invention proposes a novel pathfinding algorithm, QH-A*, which uses an improved Q-learning algorithm as the heuristic function h(s), defined as:

[0090] h(s) = r + γmax a Q(s,a)+f(N) (2)

[0091] Where f(N) is the initial exploration reward, specifically expressed as:

[0092]

[0093] Where N(s,a) is the number of times action a is executed in state s, and β is a constant used to encourage exploration of the unknown environment and obtain global information.

[0094] Let g(s) represent the path cost of the current state, specifically expressed as:

[0095] g(s)=g(s t-1 )+g a (s t-1 ,s) (4)

[0096] Among them, s t-1 G represents the state at the previous moment. a (s t-1 ,s) represents the cost of taking action a to reach the current state s.

[0097] In summary, the cost function of the QH-A* algorithm is:

[0098] f(s)=g(s)+h(s) (5)

[0099] Design a prior playback pool D * Store the static prior path information of the drone. Includes the current state s * Actions taken (a) * Receive rewards r * and the next state s′ * .like Figure 1 As shown, s * The current state can be represented as:

[0100] s * ={(x,y),(x tg ,y tg), t, (x obs ,y obs ) (6)

[0101] In formula (6), the current position coordinates (x i ,y i ) of the UAV i, the target position coordinates (x tg ,y tg ), the time step information t of the current position, the position coordinates set (x obs ,y obs ) of the potential collision obstacle on the path, and the task completion state done (judging whether the UAV has reached the destination, if yes, the state value is 1; otherwise, 0) are included.

[0102] a * is the optimal action taken by the QH-A* algorithm from the current state s * to the next state s' * , which can be expressed as a * = {dx, dy}, r * is the reward obtained by executing the action a * .

[0103] s' * is the new state generated by the UAV executing the action a * , which can be specifically expressed as:

[0104] s' * = {(x i + dx, y i + dx), (x tg , y tg ), t', (x obs ', y obs ')} (7)

[0105] In formula (7), t' is the time step information of the UAV path, and (x i + dx, y i + dx) is the new position coordinates.

[0106] Based on the heuristic function of formula (2), the prior Q value of each state-action is extracted from the Q table, Q p , which can be expressed as:

[0107]

[0108] Based on the heuristic function of the improved Q-Learning algorithm, the UAV system can perform more efficient search and decision-making in an unknown environment. In addition, by using the prior replay pool D *The DQN model can utilize historical experience to optimize decision strategies, achieve faster learning, and more efficient task execution, based on prior knowledge stored in the memory.

[0109] Step 2: Optimize the MADQN model; use the prior knowledge explored by the QH-A* algorithm to generate global information, design dynamic weight parameters, and optimize the multi-agent Q value function using prior Q values; then, in the decision-making process, the agent balances exploration and utilization through an action selection strategy based on cumulative rewards dynamically adjusted to adapt to the learning needs of different stages; finally, an improved multi-objective reward function is designed, which significantly improves the learning efficiency and task execution effect of the agent in complex environments by considering exploration rewards, obstacle avoidance rewards, and multi-agent collaboration rewards, including:

[0110] Step 2-1: Optimize the multi-agent Q value based on prior knowledge; according to the prior information stored in the prior replay pool D * , generate global information τ, and the global information τ(i) of the i-th agent is expressed as:

[0111]

[0112] where, is the distance between UAV i and the destination, expressed as:

[0113]

[0114] The distance between UAV i and the boundary and obstacles is expressed as:

[0115]

[0116] Since the prior knowledge does not fully consider the global complex dynamic environment, there is a certain limitation. A dynamic weight parameter λ(t) is introduced to combine the rewards obtained by the agent from the interaction with the unknown environment for learning and training:

[0117]

[0118] where N max is the maximum training round, and k is the current training round.

[0119] Through λ(t), the weights of prior Q values Q P and estimated Q values in global Q values Q τ can be reasonably adjusted according to the number of training rounds:

[0120] Q τ (s,a; θ) = (1- λ(t)) Q p (s,a) + λ(t) Q θ(s, a; θ) (13)

[0121] At the same time, based on global information τ, introduce the weight ω based on target distance and obstacle i , the expression is:

[0122]

[0123] Wherein, α is a constant, used to adjust the importance of target distance.

[0124] Normalized weight ω i ', the expression is:

[0125]

[0126] Then the joint value function Q of the system as a whole Q total , the expression is:

[0127]

[0128] Using the loss function L as shown in formula (14) θ , update the deep neural network parameters θ, the expression is:

[0129]

[0130] In formula (17), Y t i is the target Q value of each agent, the expression is:

[0131]

[0132] The algorithm flow is shown in Figure 2 , according to the global information, promote the coordination and cooperation between agents, to maximize the overall reward of the system, so that the algorithm is more suitable for complex and variable practical application scenarios.

[0133] Step 2-2: action selection strategy based on cumulative reward dynamic adjustment, through adaptive adjustment of random action probability ε, to achieve the dynamic balance of exploration and utilization;

[0134] In the early stage, the system focuses on exploration, obtains comprehensive environmental information, and builds more accurate estimated Q value; In the later stage, the system focuses on using known information to improve path planning efficiency;

[0135]

[0136] In formula (19), k is the current training round, ε min is the minimum value of ε set, ε o is the initial value of ε, R k is the cumulative reward of the kth round of training, average reward of the first k training rounds, average reward of the first k-1 training rounds, N is the total number of training rounds.

[0137] According to the optimized exploration strategy, the system can balance exploration and utilization at different stages, so as to select better actions and obtain:

[0138]

[0139] Based on the cumulative reward dynamic adjustment of the action selection strategy, the intelligent agent can adapt to the changes in the environment, which not only improves the training efficiency of the algorithm, but also enhances its robustness in complex environments. The traditional MADQN algorithm cannot balance exploration and utilization in complex environments, and is prone to falling into local optimum. The core idea of this strategy is to dynamically adjust the balance between exploration and utilization according to the changes in cumulative rewards during training, which improves the adaptive ability and learning efficiency of MADQN algorithm at different stages.

[0140] Step 2-3: Establish a reward function based on multi-objective coordination, which consists of exploration reward, obstacle avoidance reward and multi-agent cooperation reward. Specifically, the reward function R consists of three parts:

[0141] R = R T + R O + R S (21)

[0142] Where R T is the exploration reward; R O is the obstacle avoidance reward; R S is the multi-agent coordination reward.

[0143] (1) Exploration reward function

[0144] Design exploration reward function R T , using positive feedback to encourage UAVs to move towards the target point, approach the target, and apply negative feedback to reduce invalid exploration, thereby improving the exploration efficiency of MADQN algorithm in the initial training stage;

[0145]

[0146] Where d t ' is the distance to the target point at the next time.

[0147] (2) Obstacle avoidance reward function

[0148] Design obstacle avoidance reward function R O , to ensure flight safety, when the distance between UAV and the nearest obstacle is less than s fThe unmanned aerial vehicle can be punished, the MADQN path selection is optimized, and the collision risk is avoided.

[0149]

[0150] Wherein, s f is the safety distance of the unmanned aerial vehicle from the obstacle.

[0151] (3) Multi-agent cooperation reward function

[0152] The multi-agent cooperation reward function R S is designed to coordinate the path planning between unmanned aerial vehicles and prevent collisions between unmanned aerial vehicles. The distance between unmanned aerial vehicle i and unmanned aerial vehicle j is:

[0153]

[0154] In summary, the multi-objective coordination reward function improves the learning efficiency of the MADQN algorithm and the robustness of the system by comprehensively considering the exploration, obstacle avoidance and cooperation factors.

[0155] Step 3: Generate prior knowledge by QH-A* algorithm based on the heuristic function of improved Q-learning algorithm to explore unknown environment and store in the prior replay pool D * ; According to the optimized MADQN model, call D * in the prior replay pool for initial learning, adjust the weights of prior Q value Q p and estimated Q value based on the training round. Based on the prior knowledge, generate global information τ, consider the obstacle and target information in the current environment, and optimize the joint Q value of the system. The specific process is shown in Table 1:

[0156] Table 1 QH-A*-MADQN algorithm flow

[0157]

[0158]

[0159] For those skilled in the art in this technical field, other advantages and modifications can be easily imagined according to the above implementation types. Therefore, the present application is not limited to the above specific examples, which are only used as examples to illustrate in detail and demonstrate one form of the present application. The technical solutions obtained by various equivalent replacements of the above specific examples by those skilled in the art within the scope of the present application should be included in the scope of the claims of the present application and its equivalent range.

Claims

1. A multi-agent deep reinforcement learning path planning method based on an improved A* heuristic, characterized in that, The method includes the following steps: Step 1: Generate prior knowledge based on the QH-A* algorithm; Using the improved Q-learning algorithm as the heuristic function h(s), it is defined as: h(s)=r+γmax a Q(s,a)+f(N) (2) Where f(N) is the initial exploration reward, specifically expressed as: Where N(s,a) is the number of times action a is executed in state s, and β is a constant used to encourage exploration of the unknown environment and obtain global information; Let g(s) represent the path cost of the current state, specifically expressed as: g(s)=g(s t-1 )+g a (s t-1 ,s) (4) Among them, s t-1 G represents the state at the previous moment. a (s t-1 ( ,s) represents the cost of taking action a to reach the current state s; In summary, the cost function of the QH-A* algorithm is: f(s)=g(s)+h(s) (5) Design a prior playback pool D * Store the static prior path information of drone i Includes the current state s * Actions taken (a) * Receive rewards r * and the next state s′ * s * The current state is represented as: s * ={(x i ,y i ),(x tg ,y tg ),t,(x obs ,y obs ),done} (6) In equation (6), the current position coordinates (x, y) of drone i are included. i ,y i ), target position coordinates (x tg ,y tg The current position's time step information t is used to determine whether a collision will occur between drones; the set of position coordinates of potential collision obstacles on the path (x obs ,y obs The mission completion status "done" indicates whether the drone has reached its destination. If it has, the status value is 1; otherwise, it is 0. a * It is based on QH-A* and takes action from the current state s. * Reaching the next state s′ * The optimal action is denoted as a. * ={dx,dy},r * It is to perform action a * The reward received; s′ * Is the drone performing action a? * The resulting new state is specifically represented as follows: s′ * ={(x i +dx,y i +dx),(x tg ,y tg ),t′,(x obs ′,y obs ′)} (7) In equation (7), t′ represents the time step information of the UAV path, (x i +dx,y i +dx) represents the new position coordinates; Based on the heuristic function of equation (2), the prior Q-values ​​of each state action are extracted from the Q-table. p The expression is: Step 2: Optimize the MADQN model by using prior knowledge explored by the QH-A* algorithm to generate global information, design dynamic weight parameters, and optimize the multi-agent Q-value function using prior Q-values. Step 3: Utilize the QH-A* algorithm, based on a heuristic function derived from an improved Q-learning algorithm, to explore the unknown environment, generate prior knowledge, and store it in the prior replay pool D. * middle.

2. The multi-agent deep reinforcement learning path planning method based on the improved A* heuristic as described in claim 1, characterized in that, Step 2 includes, during the decision-making process, the agent balances exploration and utilization by dynamically adjusting its action selection strategy based on cumulative rewards to adapt to the learning needs at different stages, and designs an improved multi-objective reward function, specifically including the following steps: Step 2-1: Optimize the Q-value of the multi-agent system based on prior knowledge, according to the prior replay pool D. * Using prior information, global information τ is generated. The expression for the global information τ(i) of the i-th agent is: in, Let i be the distance between drone i and the destination, expressed as: Distance between drone i and boundaries and obstacles The expression is: A dynamic weight parameter λ is introduced, which is used to learn and train the agent by combining the rewards it receives from interacting with the unknown environment: Where, N max The maximum number of training epochs is k, and the current training epoch is k. By using λ, the global Q-value Q is adjusted appropriately based on the number of training epochs. τ The prior Q value Q P And estimate Q value Q θ Weights: Q τ (s,a;θ)=(1-λ)Q p (s,a)+λQ θ (s,a;θ) (13) Simultaneously, based on global information τ, weights ω based on target distance and obstacles are introduced. i The expression is: Here, α is a constant used to adjust the importance of the target distance; Normalized weights ω i The expression is: Then the joint value function Q of the system as a whole is Q total The expression is: The expression for updating the parameters θ of the deep neural network is: In equation (17), Y t i The target Q-value for each agent is expressed as: Step 2-2: Based on the cumulative reward, the action selection strategy is dynamically adjusted. By adaptively adjusting the probability ε of random actions, a dynamic balance between exploration and utilization is achieved. In the initial stage, the system focuses on exploration to acquire comprehensive environmental information and construct a more accurate estimate of the Q value; in the later stage, the system focuses on utilizing known information to improve path planning efficiency. In equation (19), k is the current training round, ε min For the set minimum value of ε, ε o Let R be the initial value of ε. k This is the cumulative reward for the kth round of training. The average reward for the first k rounds of training. is the average reward for the first k-1 rounds of training, and N is the total number of training rounds; Based on the optimized exploration strategy, the system can balance exploration and utilization at different stages, thereby selecting the better action and obtaining: Steps 2-3: Establish a reward function based on multi-objective coordination, consisting of three parts: exploration reward, obstacle avoidance reward, and multi-agent cooperation reward; specifically, the reward function R consists of three parts: R=R T +R O +R S (21) Among them, R T For exploration rewards; R O Rewards for obstacle avoidance; R S For multi-agent coordinated rewards.

3. The multi-agent deep reinforcement learning path planning method based on the improved A* heuristic as described in claim 2, characterized in that, Steps 2-3 include: (1) Explore reward functions Design an exploration reward function R T Positive feedback is used to incentivize the drone to move toward the target point and approach the target, while negative feedback is applied to reduce ineffective exploration; Where, d t ′ represents the distance from the target point at the next moment; (2) Obstacle avoidance reward function Design an obstacle avoidance reward function R. O To ensure flight safety, when the distance between the drone and the nearest obstacle is less than s... f Drones will be penalized; MADQN path selection will be optimized to avoid collision risks. Among them, s f The safe distance between the drone and the obstacle; (3) Multi-agent cooperative reward function Design a multi-agent cooperative reward function R. S Coordinate the path planning between drones to prevent collisions, and control the distance between drone i and drone j. for:

4. The multi-agent deep reinforcement learning path planning method based on the improved A* heuristic as described in claim 1, characterized in that, Step 3 includes: based on the optimized MADQN model, calling the D in the prior replay pool. * In the initial learning phase, the prior Q value is adjusted according to the training rounds. p Based on prior knowledge, global information τ is generated by estimating the weights of the Q-value and considering obstacle and target information in the current environment, thus optimizing the joint Q-value of the system.

Images