A multi-unmanned aerial vehicle cooperative safe path planning method and system, a terminal and a storage medium

Abstract

This invention discloses a method, system, terminal, and storage medium for multi-UAV cooperative safe path planning. The method includes: constructing a multi-UAV cooperative planning environment; obtaining information about each UAV from the environment and inputting it into a network model; calculating the total value of joint actions; and outputting joint actions based on the total value of joint actions; executing joint actions; updating the state of each UAV; calculating the global reward value; and returning the local observation information and global state at the next time step from the environment; storing the current global state, joint actions, global reward value, local observation information at the next time step, global state at the next time step, and task termination flag in a recycling pool and sampling them; updating the network model parameters and soft-updating the corresponding target network; repeating the above process for iterative training; and outputting the flight trajectory, action sequence, and obstacle avoidance results of each UAV. This invention can effectively improve the cooperative planning capability of multiple UAVs in complex environments.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) planning technology, and in particular to a method, system, terminal, and computer-readable storage medium for multi-UAV collaborative safe path planning. Background Technology

[0002] Multi-UAV cooperative planning is an important research direction in the field of autonomous control of unmanned systems. Its core task is to generate safe and cooperative flight trajectories for multiple UAVs, given the target location, obstacle environment, and kinematic constraints. Compared with single-UAV path planning, multi-UAV cooperative planning not only needs to solve the problem of individual UAVs reaching the target, but also needs to handle problems such as cooperative obstacle avoidance and joint decision-making among multiple UAVs. Therefore, it is characterized by high dimensionality, strong coupling, strong dynamics, and high difficulty in implementation.

[0003] To address the challenges of multi-drone path planning, numerous researchers have proposed optimization solutions. For example, multi-agent reinforcement learning is employed, where each drone is modeled as an agent, and multi-agent collaborative learning is used to handle the multi-drone path planning problem. Another example is the parameter-sharing off-policy multi-agent path planning and path following method, which models each drone as an agent sharing experience and combines a planner with sampling methods to generate following points, thus solving the problem of traditional multi-drone path planning relying too heavily on grid maps.

[0004] However, existing policy learning mechanisms primarily rely on deterministic policy gradients, lacking sufficient exploration capabilities in complex environments. For reinforcement learning, the design of the reward function is paramount. Existing reward functions offer a rather coarse representation of spatial guidance relationships, often constructing them by using distance changes to guide reward value changes. While this approach can generally express task requirements, it lacks sufficient guidance, relying solely on reward value changes to indicate right and wrong actions. Consequently, the policy acquires insufficient state information during training, hindering the development of more effective obstacle avoidance and target approach behaviors in complex obstacle environments.

[0005] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0006] The main objective of this invention is to provide a method, system, terminal, and computer-readable storage medium for cooperative safe path planning of multiple unmanned aerial vehicles (UAVs). This invention aims to address the problem that existing technologies lack a unified technical solution that integrates cooperative learning frameworks, enhanced exploration mechanisms, and potential field guidance mechanisms, resulting in low robustness and weak cooperative planning capabilities of multiple UAVs in complex environments.

[0007] To achieve the above objectives, the present invention provides a multi-UAV cooperative safe path planning method, which includes the following steps: Construct an environment for collaborative planning among multiple UAVs, obtain the corresponding targets configured in the environment, and form a mapping relationship between UAVs and corresponding targets; The current local observation information and current global state of each UAV at each decision moment are obtained from the environment and input into the APF-MASAC network model to calculate the total value of joint actions and output the joint actions. Perform joint actions in the environment, update the state of each UAV, calculate the global reward value based on the reward function of the fused artificial potential field according to the mapping relationship, and return the local observation information and global state of the next time step from the environment; After sampling the joint action, the global reward value, the local observation information at the next time step, and the global state, the APF-MASAC network model parameters are updated and the corresponding target network is softly updated. The local observation information of the next moment is used as the new current local observation information to iteratively train the APF-MASAC network with updated parameters until each UAV reaches the corresponding target or meets the termination condition. The flight trajectory, action sequence and obstacle avoidance results of each UAV are output through the iteratively trained APF-MASAC network model.

[0008] Optionally, in the multi-UAV cooperative safe path planning method, the environment for multi-UAV cooperative planning includes: flight boundaries, obstacle distribution, and the initial state of each UAV, wherein the initial state includes the initial position, speed, heading, and attitude information of the UAV. Local observation information includes: the UAV's own status, the status related to the target point, the status related to obstacles, the status related to other UAVs, and the status related to the boundary; The global state includes local observation information of all UAVs, target configuration information, obstacle status, and boundary information.

[0009] Optionally, in the multi-UAV cooperative safe path planning method, the APF-MASAC network model includes multiple local policy networks, multiple local value networks, and a global hybrid network. The local policy network is a stochastic policy network based on the maximum entropy reinforcement learning framework. It is used to input the local observation information of the i-th UAV into the corresponding local policy network, output the probability distribution of actions, and sample and generate the actions of the i-th UAV at time t. : ; in, The parameter is Local policy network, This represents the local observation information of the i-th UAV at time t. Indicates that given local observation information Under the given conditions, the probability distribution of the output action is given, where i represents the index of the current drone; The local value network is used to output a local action value estimate of the i-th UAV at time t based on the local observation information and the sampled actions. : ; in, The parameter is Local value networks; The global hybrid network is used to estimate the local motion value and global state of all UAVs. As input, the total value of the joint action is output through nonlinear hybridization. : ; in, This represents a global hybrid network based on the QMIX hybrid concept. The parameters represent the global hybrid network. This represents the local action value estimate of the first drone at time t. This represents the local motion value estimate of the second drone at time t. This represents the local action value estimate of the last drone at time t. Indicates the number of drones. Represents the global state.

[0010] Optionally, in the multi-UAV cooperative safe path planning method, the optimization objective of the local policy network is to maximize the action value objective with entropy regularization. : ; Among them, E To find the expected value, let represent the optimization objective in the average sense. Represents the entropy weighting coefficient. Indicates local observation Under certain conditions, the local policy network outputs actions. The logarithmic probability, Let represent the local action value function of the i-th drone.

[0011] Optionally, in the multi-UAV cooperative safe path planning method, the global hybrid network satisfies monotonicity constraints when calculating the total value of joint actions to ensure that the cooperative strategy learned through global information during the training phase degenerates into independent decisions made by each UAV based solely on local observations during the execution phase. The monotonicity constraint is expressed as: ; Where i = 1, 2, …, N, Indicates the total value of the union The minute changes Represents the local value of the i-th drone. The minute changes Indicates the total value of the union Local value of the i-th UAV The partial derivatives, if the partial derivatives satisfy 0 indicates that when the local value increases, the total combined value does not decrease.

[0012] Optionally, in the multi-UAV cooperative safe path planning method, the step of calculating the global reward value based on the reward function of the fused artificial potential field according to the mapping relationship specifically includes: Calculate the target attraction reward value based on the change in distance between the drone and the corresponding target. : ; in, This represents the gain coefficient of the target attraction term, used to adjust the reward intensity of the drone's movement toward a preset target. This represents the position vector of the i-th drone at the previous time step. This represents the preset target location corresponding to the i-th drone. This represents the position vector of the i-th drone at time t. Represents the L2 norm; For each obstacle, determine whether the distance between the drone and the obstacle is less than the obstacle's safe distance threshold. If so, calculate the obstacle repulsion penalty value, and sum the penalties for all obstacles that triggered the penalty to obtain the total obstacle repulsion penalty value. ; ; in, This represents the total number of obstacles in the environment. Index representing obstacles, This represents the gain coefficient for obstacle removal, used to adjust the penalty intensity when the drone approaches an obstacle. This represents the distance between the i-th drone and the m-th obstacle. Indicates the safe distance threshold from obstacles. Indicates an indicator function; For other drones, determine if the distance between the current drone and other drones is less than the inter-drone safe distance threshold. If so, calculate the inter-drone safety penalty value and sum the values ​​for all other drones that triggered the penalty to obtain the total inter-drone safety penalty value. : ; in, This represents the number of drones, j represents the index of other drones, and i represents the index of the current drone. This represents the gain coefficient of the inter-drone safety constraint, used to adjust the penalty intensity when drones are too close together. This represents the distance between the i-th and j-th drones. Indicates the safe distance threshold between machines; Determine if the drone has crossed the boundary; if so, calculate the boundary constraint penalty value. ; ; in, This represents the boundary constraint gain coefficient, used to adjust the penalty intensity when the drone approaches or crosses the environmental boundary. This indicates the degree to which the i-th drone has crossed the boundary at time t; Determine whether the drone has triggered a termination event. If so, assign a termination event reward value to the drone based on the type of termination event. The termination event types include: reaching the target, colliding, or seriously crossing the boundary. The local reward value for the i-th UAV is obtained by weighting and summing the target attraction reward value, obstacle repulsion total penalty value, inter-UAV safety total penalty value, boundary constraint penalty value, and termination event reward value according to preset weights. : ; in, , , , and These are the preset weighting coefficients for each item; The local reward values ​​of all drones are aggregated to obtain the global reward value.

[0013] Optionally, in the multi-UAV cooperative safe path planning method, the step of updating the APF-MASAC network model parameters and softly updating the corresponding target network specifically includes: A batch of experience samples are randomly sampled from the experience replay pool. Each experience sample contains the current global state, joint action, global reward value, local observation information at the next time step, global state at the next time step, and task termination flag. For each sampled empirical sample, the local action for the next time step is generated based on the local observation information at the next time step through the target policy network, and the value of the local action at the next time step is calculated through the target local value network. The value of the local action at the next moment and the global state at the next moment are input into the target global hybrid network to obtain the target joint total value; After subtracting the weighted entropy terms of each drone strategy from the total joint value of the objective, the global reward value and discount factor are combined, and the future reward term in the termination state is reduced to zero using the termination flag, to construct the time-series difference objective value. The parameters of the local value network and the global hybrid network are updated by minimizing the mean square error between the total value of joint actions and the temporal difference objective value, and the parameters of the local policy network are updated by maximizing the action value objective with entropy regularization. The updated network parameters are synchronized to the corresponding target network according to a fixed ratio, thus completing the soft update of the target network.

[0014] Furthermore, to achieve the above objectives, the present invention also provides a multi-UAV cooperative safe path planning system, wherein the multi-UAV cooperative safe path planning system includes: The environment construction module is used to construct an environment for multi-UAV collaborative planning, and to obtain the corresponding targets configured in the environment to form a mapping relationship between UAVs and corresponding targets; The joint action calculation module is used to obtain the current local observation information and current global state of each UAV at each decision moment from the environment, input them into the APF-MASAC network model to calculate the total value of joint actions, and output the joint actions. The reward feedback module is used to perform joint actions in the environment, update the status of each UAV, calculate the global reward value based on the reward function of the fused artificial potential field, and return the local observation information and the global state of the next time step from the environment. The model update module is used to sample the joint action, the global reward value, the local observation information at the next time step, and the global state, update the APF-MASAC network model parameters, and softly update the corresponding target network. The iterative output module is used to take the local observation information of the next moment as the new current local observation information and iteratively train the APF-MASAC network with updated parameters until each UAV reaches the corresponding target or meets the termination condition. The module outputs the flight trajectory, action sequence and obstacle avoidance results of each UAV through the iteratively trained APF-MASAC network model.

[0015] In addition, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a multi-UAV cooperative safe path planning program stored in the memory and executable on the processor, wherein when the multi-UAV cooperative safe path planning program is executed by the processor, it implements the steps of the multi-UAV cooperative safe path planning method as described above.

[0016] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a multi-UAV cooperative safe path planning program, which, when executed by a processor, implements the steps of the multi-UAV cooperative safe path planning method as described above.

[0017] In this invention, a multi-UAV collaborative planning environment is constructed, and corresponding targets configured in the environment are obtained to form a mapping relationship between UAVs and corresponding targets. The current local observation information and current global state of each UAV at each decision moment are obtained from the environment and input into the APF-MASAC network model to calculate the total value of joint actions, outputting the joint actions. The joint actions are executed, updating the states of each UAV, calculating the global reward value, and returning the local observation information and global state of the next moment from the environment. The current global state, joint actions, global reward value, next moment local observation information, next moment global state, and task termination flag are stored in an experience replay pool. Samples are taken from the experience replay pool to update the network model parameters and softly update the corresponding target network. The above process is repeated for iterative training, finally outputting the flight trajectory, action sequence, and obstacle avoidance results of each UAV. This invention effectively avoids the problem of slow convergence caused by blind exploration in complex state spaces in reinforcement learning, improves the robustness of UAVs flying in continuous action spaces, and effectively enhances the collaborative planning capability of multiple UAVs in complex environments. Attached Figure Description

[0018] Figure 1 This is a flowchart of a preferred embodiment of the multi-UAV cooperative safe path planning method of the present invention; Figure 2 This is a schematic diagram of the principle architecture of the multi-UAV cooperative safe path planning method of the present invention; Figure 3 This is a technical framework diagram of collaborative decision-making and reward feedback in the multi-UAV collaborative safe path planning method of the present invention; Figure 4 This is a structural diagram of a preferred embodiment of the multi-UAV cooperative safe path planning system of the present invention; Figure 5 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0019] This application provides a method, system, and terminal for cooperative safe path planning for multiple unmanned aerial vehicles (UAVs). To make the objectives, technical solutions, and effects of this application clearer and more explicit, the following detailed description is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit this application.

[0020] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0021] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0022] The preferred embodiment of the multi-UAV cooperative safe path planning method of the present invention, such as... Figure 1 and Figure 2 As shown, the multi-UAV cooperative safe path planning method includes the following steps: Step S10: Construct an environment for multi-UAV collaborative planning, and obtain the corresponding targets configured in the environment to form a mapping relationship between UAVs and corresponding targets.

[0023] In this embodiment, the environment for the multi-UAV collaborative planning includes: flight boundaries, obstacle distribution, and the initial state of each UAV, wherein the initial state includes the initial position, speed, heading, and attitude information of the UAV.

[0024] It is understood that the object of this invention is the cooperative path planning problem among multiple unmanned aerial vehicles (UAVs) in complex obstacle environments. For example... Figure 2 As shown in the rule scenario and task configuration module, this embodiment constructs a multi-UAV collaborative planning environment in a three-dimensional flight space, setting flight boundaries, obstacle distribution, and the initial state of each UAV. Obstacles can be static or dynamic, and the initial state includes the UAV's initial position, speed, heading, and attitude information.

[0025] Specifically, suppose there are N drones in the environment. For each drone... Before planning begins, a target is pre-assigned, specifying the destination (which can be a target point) that each drone will fly to. Or target area ), where i represents the index of the current drone, and the set of obstacles set in the environment is represented as: Where M represents the number of obstacles, it can be understood that each drone moves within the same flight space, subject to spatial boundary constraints, obstacle constraints, and safe distance constraints between drones. The mapping relationship shown clarifies which drone corresponds to which target, establishing a one-to-one correspondence between drones and targets. For example, drones... Flying towards the target .

[0026] Step S20: Obtain the current local observation information and current global state of each UAV at each decision moment from the environment, input them into the APF-MASAC network model to calculate the total value of joint actions, and output the joint actions.

[0027] Specifically, based on discrete time steps in reinforcement learning, the planning process is divided into multiple consecutive decision nodes, each of which performs state acquisition, action decision, and environment update. The current local observation information and current global state of each UAV are read from the constructed collaborative planning environment.

[0028] The local observation information includes: the UAV's own state, the target point-related state, the obstacle-related state, the other UAV-related state, and the boundary-related state. Specifically, the UAV's own state includes its current position and speed; the target point-related state includes the UAV's relative position, relative distance, and risk information relative to its target point; the obstacle-related state includes the relative position, relative distance, and risk information of nearby obstacles relative to the UAV; the other UAV-related state includes the relative position, relative distance, and relative speed information of nearby UAVs relative to the UAV; and the boundary-related state includes the distance information of the UAV relative to the flight boundary.

[0029] Therefore, the local observation state of the i-th UAV at time t This can be abstractly represented as: ; in, This indicates the state of the drone itself. Indicates the speed of the drone. Indicates the relevant status of the target point. This indicates the relative obstacle state. Indicates the relevant status of the other drones. This indicates the boundary-related state.

[0030] Furthermore, during the training phase, it is necessary to further define the global state. This is used for joint value evaluation in global hybrid networks. Global state It is composed of the local states of all UAVs, the state of the target, the state of obstacles, and boundary information, namely: ; Where G represents target configuration information and B represents boundary information. This indicates the local observation status of all drones.

[0031] Furthermore, the APF-MASAC network model includes multiple local policy networks, multiple local value networks, and a global (high-level) hybrid network; The local policy network is a stochastic policy network based on the maximum entropy reinforcement learning framework. It is used to input the local observation information of the i-th UAV into the corresponding local policy network, output the probability distribution of actions, and sample and generate the actions of the i-th UAV at time t. : ; in, The parameter is Local policy network, This represents the local observation information of the i-th UAV at time t. Indicates that given local observation information Under the given conditions, the probability distribution of the output action is given, where i represents the index of the current drone; The local value network is used to output a local action value estimate of the i-th UAV at time t based on the local observation information and the sampled actions. : ; in, The parameter is Local value networks; The global hybrid network is used to estimate the local motion value and global state of all UAVs. As input, the total value of the joint action is output through nonlinear hybridization. : ; in, This represents a global hybrid network based on the QMIX hybrid concept. The parameters represent the global hybrid network. This represents the local action value estimate of the first drone at time t. This represents the local motion value estimate of the second drone at time t. This represents the local action value estimate of the last drone at time t. Indicates the number of drones. Represents the global state.

[0032] Understandably, this invention proposes a novel multi-agent cooperative control network model, abbreviated as APF-MASAC (Artificial Potential Field - Multi-Agent Soft Actor-Critic). This network model takes multi-UAV cooperative planning as its core objective and combines the spatial guidance capability of artificial potential fields, the exploration mechanism of maximum entropy reinforcement learning, and the multi-agent cooperative evaluation framework.

[0033] Specifically, the APF-MASAC network model adopts the CTDE (Centralized Training with Decentralized Execution) collaborative framework as its overall basic structure. Addressing the problem of multi-agent systems easily getting trapped in local optima in complex continuous action spaces, it introduces an enhanced exploration mechanism into the local policy network. Each agent's local policy network constructs a maximum entropy stochastic policy structure based on SAC, encouraging agents to conduct broader exploration to improve local exploration capabilities and training stability.

[0034] Furthermore, the optimization objective of the local policy network is to maximize the action value with entropy regularization. : ; Among them, E To find the expected value, let represent the optimization objective in the average sense. Represents the entropy weighting coefficient. Indicates local observation Under certain conditions, the local policy network outputs actions. The logarithmic probability, Let represent the local action value function of the i-th drone.

[0035] Furthermore, the global hybrid network satisfies monotonicity constraints when calculating the total value of joint actions to ensure that the cooperative strategy learned through global information during the training phase degenerates into independent decisions made by each UAV based solely on local observations during the execution phase. The monotonicity constraint is expressed as: ; Where i = 1, 2, …, N, Indicates the total value of the union The minute changes Represents the local value of the i-th drone. The minute changes Indicates the total value of the union Local value of the i-th UAV The partial derivatives, if the partial derivatives satisfy 0 indicates that when the local value increases, the total combined value does not decrease.

[0036] It is understandable that the monotonicity constraint is represented by an integral expression of the joint total value Qtot with respect to the local value of the i-th UAV. The partial derivative (in which direction will the total joint value change when the i-th local value changes slightly). If the partial derivative is ≥0, it means that when the local value increases, the total joint value will not decrease.

[0037] Furthermore, based on the aforementioned APF-MASAC network model, the total value of the joint actions is calculated according to the current local observation information and current global state of each UAV. The process of outputting the joint actions based on the total value of the joint actions is as follows: The local observation information of the i-th UAV is input into the local policy network of the APF-MASAC network model for calculation, outputting the probability distribution of actions and sampling to generate the actions of the i-th UAV at time t. ; The action of the i-th drone at time t The local value network input to the APF-MASAC network model is used for calculation, and the local action value estimate of the i-th UAV at time t is output. ; Estimate the local motion value and global state of all drones The global hybrid network input to the APF-MASAC network model performs nonlinear hybrid computation and outputs the total value of joint actions. .

[0038] Understandably, the total value of joint actions In a mathematical sense, it represents a combination of actions. The total value assessment serves as a direct input to the global hybrid network in the network implementation structure. For the joint total value assessment, it belongs to the implementation mode of "local value is calculated first, and then global hybridization".

[0039] Furthermore, the joint action is output based on the total value of the joint action, including: a training phase to maximize the total value of the joint action. With the objective of combining the individual actions of each drone, the combined action at the current moment is constituted. And output it to the environment for execution, where, These represent the local actions of each drone.

[0040] Understandably, once training is complete and the execution phase begins, the global hybrid network and global state are no longer used. Each UAV independently outputs actions based solely on its own local observation information through a pre-trained local policy network, without needing to know the state of other UAVs or global information.

[0041] Step S30: Perform joint actions in the environment, update the status of each UAV, calculate the global reward value based on the reward function of the fused artificial potential field according to the mapping relationship, and return the local observation information and global status of the next moment from the environment.

[0042] like Figure 3 As shown, it is understandable that after the UAV performs joint actions and triggers an update of the environmental state space, this scheme introduces an artificial potential field model for reward shaping to accelerate algorithm convergence and ensure flight safety. This mechanism provides an attractive reward between the UAV and the target point, continuously guiding the agent to approach the target area; it also constructs a repulsive penalty between the UAV and obstacles and other UAVs, prompting the agent to avoid external obstacles and eliminate the risk of inter-Aggression collisions. The potential field reward provides clear directional guidance and safety boundaries for the multi-agent system.

[0043] The step of calculating the global reward value based on the reward function of the fused artificial potential field according to the mapping relationship specifically includes: Calculate the target attraction reward value based on the change in distance between the drone and the corresponding target. : ; in, This represents the gain coefficient of the target attraction term, used to adjust the reward intensity of the drone's movement toward a preset target. This represents the position vector of the i-th drone at the previous time step. This represents the preset target location corresponding to the i-th drone. This represents the position vector of the i-th drone at time t. Represents the L2 norm; For each obstacle, determine whether the distance between the drone and the obstacle is less than the obstacle's safe distance threshold. If so, calculate the obstacle repulsion penalty value, and sum the penalties for all obstacles that triggered the penalty to obtain the total obstacle repulsion penalty value. ; ; in, This represents the total number of obstacles in the environment. Index representing obstacles, This represents the gain coefficient for obstacle removal, used to adjust the penalty intensity when the drone approaches an obstacle. This represents the distance between the i-th drone and the m-th obstacle. Indicates the safe distance threshold from obstacles. Indicates an indicator function; For other drones, determine if the distance between the current drone and other drones is less than the inter-drone safe distance threshold. If so, calculate the inter-drone safety penalty value and sum the values ​​for all other drones that triggered the penalty to obtain the total inter-drone safety penalty value. : ; in, This represents the number of drones, j represents the index of other drones, and i represents the index of the current drone. This represents the gain coefficient of the inter-drone safety constraint, used to adjust the penalty intensity when drones are too close together. This represents the distance between the i-th and j-th drones. Indicates the safe distance threshold between machines; Determine if the drone has crossed the boundary; if so, calculate the boundary constraint penalty value. ; ; in, This represents the boundary constraint gain coefficient, used to adjust the penalty intensity when the drone approaches or crosses the environmental boundary. This indicates the degree to which the i-th drone has crossed the boundary at time t; Determine whether the drone has triggered a termination event. If so, assign a termination event reward value to the drone based on the type of termination event. The termination event types include: reaching the target, colliding, or seriously crossing the boundary. The local reward value for the i-th UAV is obtained by weighting and summing the target attraction reward value, obstacle repulsion total penalty value, inter-UAV safety total penalty value, boundary constraint penalty value, and termination event reward value according to preset weights. : ; in, , , , and These are the preset weighting coefficients for each item; The local reward values ​​of all drones are aggregated to obtain the global reward value.

[0044] Understandably, in this embodiment, after the UAVs perform joint actions and update their states, five reward components are calculated sequentially: First, the distance change between each UAV and its corresponding target point is determined according to the mapping relationship, and a target attraction reward value is calculated, with a positive reward given for distance reduction; Second, for each obstacle, it is determined whether the distance between it and the UAV is less than the obstacle safety distance threshold, and if so, an obstacle repulsion penalty value is calculated; Third, for each other UAV, it is determined whether the distance between them is less than the inter-UAV safety distance threshold, and if so, an inter-UAV safety penalty value is calculated; Fourth, it is determined whether the UAVs have crossed the boundary and a boundary constraint penalty value is calculated; Fifth, it is determined whether a termination event such as reaching the target, a collision, or a serious boundary crossing has been triggered and corresponding reward or penalty values ​​are assigned; Finally, the above five components are weighted and summed according to preset weights to obtain the local reward value of each UAV, and all local reward values ​​are aggregated to obtain the global reward value. This global reward value is then stored in the experience replay pool for subsequent network parameter updates.

[0045] Step S40: After sampling the joint action, the global reward value, the local observation information and the global state at the next time step, update the APF-MASAC network model parameters and soft update the corresponding target network.

[0046] In this embodiment, the total value of the joint action generated by the network decision, the global reward value of the environmental feedback, and the state information of the next moment are first packaged and stored in the experience pool. After sufficient samples are accumulated, random sampling is performed, and the target value of the action at the next moment is calculated using the target network. Then, the temporal difference target value is constructed by combining the global reward and the termination flag. Finally, the network parameters are updated by minimizing the error between the current value and the target value, and the network is slowly synchronized to the target network in a soft update manner, thereby stabilizing the training process.

[0047] Furthermore, the step of updating the APF-MASAC network model parameters and softly updating the corresponding target network specifically includes: A batch of experience samples are randomly sampled from the experience replay pool. Each experience sample contains the current global state, joint action, global reward value, local observation information at the next time step, global state at the next time step, and task termination flag. For each sampled empirical sample, the local action for the next time step is generated based on the local observation information at the next time step through the target policy network, and the value of the local action at the next time step is calculated through the target local value network. The value of the local action at the next moment and the global state at the next moment are input into the target global hybrid network to obtain the target joint total value; After subtracting the weighted entropy terms of each drone strategy from the total joint value of the objective, the global reward value and discount factor are combined, and the future reward term in the termination state is reduced to zero using the termination flag, to construct the time-series difference objective value. The parameters of the local value network and the global hybrid network are updated by minimizing the mean square error between the total value of joint actions and the temporal difference objective value, and the parameters of the local policy network are updated by maximizing the action value objective with entropy regularization. The updated network parameters are synchronized to the corresponding target network according to a fixed ratio, thus completing the soft update of the target network.

[0048] Understandably, after each UAV performs joint actions and interacts with the environment, the state transition samples are stored in the experience replay pool. After randomly sampling a small batch of experience samples from the experience replay pool, the target policy network generates the local actions for the next time step. Subsequently, the target local value network calculates the local action value of each UAV at the next moment. Then, the target global hybrid network combines the global state and the value of each local action to calculate the target's joint total value. : ; in, This indicates the global state at the next moment. This represents the joint action generated by the target policy network at the next time step. This represents the target global hybrid network. This represents the local actions of all UAVs at the next moment, generated by the target policy network.

[0049] Furthermore, by introducing the maximum entropy term, combining the global reward value and the discount factor, and using a termination flag to reset the future reward term in the termination state to zero, a temporal difference objective value is constructed. : ; in, This represents the global reward obtained by aggregating the local rewards. This represents the discount factor for future returns. This is a termination flag used to determine whether the current time has entered a termination state. This represents the weight of the entropy regularization term in maximum entropy reinforcement learning. This represents the target local policy network.

[0050] Furthermore, the loss function of the local value network is: ; in, This represents the total joint value, which is the total value of the joint actions output by the global hybrid network.

[0051] Furthermore, after training is complete, the updated network parameters are synchronized to the corresponding target network at a fixed ratio, completing the soft update of the target network. It should be noted that the networks mentioned above and the target network (such as the local policy network and the target local policy network) are two networks with identical structures but different parameters. Taking the local policy network and the target local policy network as an example, the local policy network (parameters...) The target local policy network (parameters) is the network currently being trained and continuously updated, directly used to generate actions and calculate gradients. () is a historical version of the local policy network with delayed parameter updates, specifically used to generate the local action for the next time step when constructing the temporal difference objective value.

[0052] Step S50: Use the local observation information of the next moment as the new current local observation information, and iteratively train the APF-MASAC network with updated parameters until each UAV reaches the corresponding target or meets the termination condition. Output the flight trajectory, action sequence and obstacle avoidance results of each UAV through the iteratively trained APF-MASAC network model.

[0053] Specifically, after each round of parameter updates, the local observation information returned by the environment at the next moment is used as the new local observation information at the current moment and input into the APF-MASAC network model to start a new round of decision-making, and so on in a loop.

[0054] When the termination condition corresponding to the termination flag is triggered (such as the drone reaching the target, colliding, or seriously crossing the boundary), the current round ends; if not triggered, the above-mentioned cyclical steps of obtaining state, network decision-making, executing actions, calculating rewards, and updating the network continue until each drone reaches its corresponding target or meets the preset termination condition. After the loop ends, the APF-MASAC network model training is complete. After training, the distributed execution phase begins: each drone is decoupled from the global state and high-level hybrid network, relying only on its own trained local policy network to independently output actions based on local observation information, ultimately obtaining the flight trajectory and collaborative planning results of each drone.

[0055] The present invention has the following beneficial effects: (1) In terms of network architecture, this invention constructs an APF-MASAC architecture. At the bottom action execution end of the CTDE architecture, the local policy is designed as a maximum entropy stochastic policy model based on SAC (Soft Actor-Critic, a maximum entropy reinforcement learning method), which breaks the limitation that deterministic policies are prone to local optima and enhances the agent's active exploration capability in complex environments. At the same time, a Mixer (Mixing Network) hybrid network is introduced at the high-level training end to globally integrate the value of local actions. This combination of local action exploration and global joint value evaluation enables multi-UAV systems to take into account both the real-time performance of individual UAVs and the global coordination of swarm planning.

[0056] (2) Regarding the reward mechanism, compared to the existing technology that uses linear distance to calculate reward values, this invention proposes a reward shaping method based on an artificial potential field. On the one hand, a strict safety trigger boundary is introduced, which reduces invalid negative reward perturbations for the UAV within the safe area; on the other hand, when the UAV approaches an obstacle or another UAV, the reward function will generate a penalty gradient that increases inversely based on the potential field principle. This ensures the safety of multi-UAV dynamic shuttles, thereby achieving higher quality and faster convergence of collaborative path planning in complex obstacle environments.

[0057] Furthermore, such as Figure 4 As shown, based on the above-mentioned multi-UAV cooperative safe path planning method, the present invention also provides a multi-UAV cooperative safe path planning system, wherein the multi-UAV cooperative safe path planning system includes: The environment construction module 51 is used to construct an environment for multi-UAV collaborative planning, and to obtain the corresponding targets configured in the environment to form a mapping relationship between UAVs and corresponding targets; The joint action calculation module 52 is used to obtain the current local observation information and current global state of each UAV at each decision moment from the environment, input them into the APF-MASAC network model to calculate the total value of joint actions, and output the joint actions. The reward feedback module 53 is used to perform joint actions in the environment, update the state of each UAV, calculate the global reward value based on the reward function of the fused artificial potential field, and return the local observation information and the global state of the next time step from the environment. The model update module 54 is used to sample the joint action, the global reward value, the local observation information of the next time step, and the global state, update the APF-MASAC network model parameters, and softly update the corresponding target network. The iterative output module 55 is used to take the local observation information of the next moment as the new current local observation information and iteratively train the APF-MASAC network with updated parameters until each UAV reaches the corresponding target or meets the termination condition. The iteratively trained APF-MASAC network model outputs the flight trajectory, action sequence and obstacle avoidance results of each UAV.

[0058] Furthermore, such as Figure 5 As shown, based on the above-mentioned multi-UAV collaborative safe path planning method and system, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 5 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0059] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a multi-UAV cooperative safe path planning program 40, which can be executed by the processor 10 to implement the multi-UAV cooperative safe path planning method of this application.

[0060] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the multi-UAV cooperative safe path planning method.

[0061] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components of the terminal communicate with each other via a system bus.

[0062] In one embodiment, when the processor 10 executes the multi-UAV cooperative safe path planning program 40 in the memory 20, the following steps are performed: Construct an environment for collaborative planning among multiple UAVs, obtain the corresponding targets configured in the environment, and form a mapping relationship between UAVs and corresponding targets; The current local observation information and current global state of each UAV at each decision moment are obtained from the environment and input into the APF-MASAC network model to calculate the total value of joint actions and output the joint actions. Perform joint actions in the environment, update the state of each UAV, calculate the global reward value based on the reward function of the fused artificial potential field according to the mapping relationship, and return the local observation information and global state of the next time step from the environment; After sampling the joint action, the global reward value, the local observation information at the next time step, and the global state, the APF-MASAC network model parameters are updated and the corresponding target network is softly updated. The local observation information of the next moment is used as the new current local observation information to iteratively train the APF-MASAC network with updated parameters until each UAV reaches the corresponding target or meets the termination condition. The flight trajectory, action sequence and obstacle avoidance results of each UAV are output through the iteratively trained APF-MASAC network model.

[0063] The environment for the multi-UAV collaborative planning includes: flight boundaries, obstacle distribution, and the initial state of each UAV, wherein the initial state includes the initial position, speed, heading, and attitude information of the UAV. Local observation information includes: the UAV's own status, the status related to the target point, the status related to obstacles, the status related to other UAVs, and the status related to the boundary; The global status includes: local observation information of all UAVs, target configuration information, obstacle status, and boundary information.

[0064] The APF-MASAC network model includes multiple local policy networks, multiple local value networks, and a global hybrid network. The local policy network is a stochastic policy network based on the maximum entropy reinforcement learning framework. It is used to input the local observation information of the i-th UAV into the corresponding local policy network, output the probability distribution of actions, and sample and generate the actions of the i-th UAV at time t. : ; in, The parameter is Local policy network, This represents the local observation information of the i-th UAV at time t. Indicates that given local observation information Under the given conditions, the probability distribution of the output action is given, where i represents the index of the current drone; The local value network is used to output a local action value estimate of the i-th UAV at time t based on the local observation information and the sampled actions. : ; in, The parameter is Local value networks; The global hybrid network is used to estimate the local motion value and global state of all UAVs. As input, the total value of the joint action is output through nonlinear hybridization. : ; in, This represents a global hybrid network based on the QMIX hybrid concept. The parameters represent the global hybrid network. This represents the local action value estimate of the first drone at time t. This represents the local motion value estimate of the second drone at time t. This represents the local action value estimate of the last drone at time t. Indicates the number of drones. Represents the global state.

[0065] The optimization objective of the local policy network is to maximize the action value with entropy regularization. : ; Among them, E To find the expected value, let represent the optimization objective in the average sense. Represents the entropy weighting coefficient. Indicates local observation Under certain conditions, the local policy network outputs actions. The logarithmic probability, Let represent the local action value function of the i-th drone.

[0066] The global hybrid network satisfies monotonicity constraints when calculating the total value of joint actions to ensure that the cooperative strategy learned through global information during the training phase degenerates into independent decisions made by each UAV based solely on local observations during the execution phase. The monotonicity constraint is expressed as: ; Where i = 1, 2, …, N, Indicates the total value of the union The minute changes Represents the local value of the i-th drone. The minute changes Indicates the total value of the union Local value of the i-th UAV The partial derivatives, if the partial derivatives satisfy 0 indicates that when the local value increases, the total combined value does not decrease.

[0067] Specifically, calculating the global reward value based on the reward function of the fused artificial potential field according to the mapping relationship includes: Calculate the target attraction reward value based on the change in distance between the drone and the corresponding target. : ; in, This represents the gain coefficient of the target attraction term, used to adjust the reward intensity of the drone's movement toward a preset target. This represents the position vector of the i-th drone at the previous time step. This represents the preset target location corresponding to the i-th drone. This represents the position vector of the i-th drone at time t. Represents the L2 norm; For each obstacle, determine whether the distance between the drone and the obstacle is less than the obstacle's safe distance threshold. If so, calculate the obstacle repulsion penalty value, and sum the penalties for all obstacles that triggered the penalty to obtain the total obstacle repulsion penalty value. ; ; in, This represents the total number of obstacles in the environment. Index representing obstacles, This represents the gain coefficient for obstacle removal, used to adjust the penalty intensity when the drone approaches an obstacle. This represents the distance between the i-th drone and the m-th obstacle. Indicates the safe distance threshold from obstacles. Indicates an indicator function; For other drones, determine if the distance between the current drone and other drones is less than the inter-drone safe distance threshold. If so, calculate the inter-drone safety penalty value and sum the values ​​for all other drones that triggered the penalty to obtain the total inter-drone safety penalty value. : ; in, This represents the number of drones, j represents the index of other drones, and i represents the index of the current drone. This represents the gain coefficient of the inter-drone safety constraint, used to adjust the penalty intensity when drones are too close together. This represents the distance between the i-th and j-th drones. Indicates the safe distance threshold between machines; Determine if the drone has crossed the boundary; if so, calculate the boundary constraint penalty value. ; ; in, This represents the boundary constraint gain coefficient, used to adjust the penalty intensity when the drone approaches or crosses the environmental boundary. This indicates the degree to which the i-th drone has crossed the boundary at time t; Determine whether the drone has triggered a termination event. If so, assign a termination event reward value to the drone based on the type of termination event. The termination event types include: reaching the target, colliding, or seriously crossing the boundary. The local reward value for the i-th UAV is obtained by weighting and summing the target attraction reward value, obstacle repulsion total penalty value, inter-UAV safety total penalty value, boundary constraint penalty value, and termination event reward value according to preset weights. : ; in, , , , and These are the preset weighting coefficients for each item; The local reward values ​​of all drones are aggregated to obtain the global reward value.

[0068] Specifically, updating the APF-MASAC network model parameters and soft-updating the corresponding target network includes: A batch of experience samples are randomly sampled from the experience replay pool. Each experience sample contains the current global state, joint action, global reward value, local observation information at the next time step, global state at the next time step, and task termination flag. For each sampled empirical sample, the local action for the next time step is generated based on the local observation information at the next time step through the target policy network, and the value of the local action at the next time step is calculated through the target local value network. The value of the local action at the next moment and the global state at the next moment are input into the target global hybrid network to obtain the target joint total value; After subtracting the weighted entropy terms of each drone strategy from the total joint value of the objective, the global reward value and discount factor are combined, and the future reward term in the termination state is reduced to zero using the termination flag, to construct the time-series difference objective value. The parameters of the local value network and the global hybrid network are updated by minimizing the mean square error between the total value of joint actions and the temporal difference objective value, and the parameters of the local policy network are updated by maximizing the action value objective with entropy regularization. The updated network parameters are synchronized to the corresponding target network according to a fixed ratio, thus completing the soft update of the target network.

[0069] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a multi-UAV cooperative safe path planning program, which, when executed by a processor, implements the steps of the multi-UAV cooperative safe path planning method as described above.

[0070] In summary, this invention provides a method, system, terminal, and storage medium for multi-UAV cooperative safe path planning. The method includes: constructing a multi-UAV cooperative planning environment and obtaining corresponding targets configured in the environment to form a mapping relationship between UAVs and corresponding targets; obtaining the current local observation information and current global state of each UAV at each decision moment from the environment, inputting them into an APF-MASAC network model to calculate the total value of joint actions, and outputting joint actions; executing joint actions, updating the state of each UAV, calculating the global reward value, and returning the local observation information and global state of the next moment from the environment; storing the current global state, joint actions, global reward value, local observation information of the next moment, global state of the next moment, and task termination flag into an experience replay pool, sampling from the experience replay pool, updating the network model parameters, softly updating the corresponding target network, and repeating the above process for iterative training, finally outputting the flight trajectory, action sequence, and obstacle avoidance results of each UAV. This invention can effectively avoid the problem of slow convergence caused by blind exploration in complex state spaces in reinforcement learning, improve the robustness of UAVs flying in continuous action spaces, and effectively improve the cooperative planning capability of multiple UAVs in complex environments.

[0071] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0072] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0073] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for multi-UAV cooperative safe path planning, characterized in that, The aforementioned multi-UAV cooperative safe path planning method includes: Construct an environment for collaborative planning among multiple UAVs, obtain the corresponding targets configured in the environment, and form a mapping relationship between UAVs and corresponding targets; The current local observation information and current global state of each UAV at each decision moment are obtained from the environment and input into the APF-MASAC network model to calculate the total value of joint actions and output the joint actions. Perform joint actions in the environment, update the state of each UAV, calculate the global reward value based on the reward function of the fused artificial potential field according to the mapping relationship, and return the local observation information and global state of the next time step from the environment; After sampling the joint action, the global reward value, the local observation information at the next time step, and the global state, the APF-MASAC network model parameters are updated and the corresponding target network is softly updated. The local observation information of the next moment is used as the new current local observation information to iteratively train the APF-MASAC network with updated parameters until each UAV reaches the corresponding target or meets the termination condition. The flight trajectory, action sequence and obstacle avoidance results of each UAV are output through the iteratively trained APF-MASAC network model.

2. The method of claim 1, wherein, The environment for the multi-UAV collaborative planning includes: flight boundaries, obstacle distribution, and the initial state of each UAV, wherein the initial state includes the initial position, speed, heading, and attitude information of the UAV. Local observation information includes: the UAV's own status, the status related to the target point, the status related to obstacles, the status related to other UAVs, and the status related to the boundary; The global status includes: local observation information of all UAVs, target configuration information, obstacle status, and boundary information. 3.The method of claim 1, wherein, The APF-MASAC network model includes multiple local policy networks, multiple local value networks, and a global hybrid network. The local policy network is a stochastic policy network based on the maximum entropy reinforcement learning framework. It is used to input the local observation information of the i-th UAV into the corresponding local policy network, output the probability distribution of actions, and sample and generate the actions of the i-th UAV at time t. : ； in, The parameter is Local policy network, This represents the local observation information of the i-th UAV at time t. Indicates that given local observation information Under the given conditions, the probability distribution of the output action is given, where i represents the index of the current drone; The local value network is configured to output a local action value estimate of the ith UAV at time t according to the local observation information and a sampled action : ； wherein represents a local value network with parameters ; The global mixed network is used to estimate the joint action total value with all the local action value estimates and global state of all the drones As input, the joint action total value is output by a nonlinear mixture : ； in, This represents a global hybrid network based on the QMIX hybrid concept. These represent the parameters of the global hybrid network. This represents the local action value estimate of the first drone at time t. This represents the local motion value estimate of the second drone at time t. This represents the local action value estimate of the last drone at time t. Indicates the number of drones. Represents the global state.

4. The method of claim 3, wherein, The optimization objective of the local policy network is to maximize an action value objective with an entropy regularizer : ； where E is the expectation, representing the optimization objective in the mean sense, is the entropy weight coefficient, represents the local observation under the condition, the local policy network outputs the logarithmic probability of action , and represents the local action value function of the i-th UAV.

5. The method of claim 3, wherein, The global hybrid network satisfies monotonicity constraints when calculating the total value of joint actions to ensure that the cooperative strategy learned through global information during the training phase degenerates into independent decisions made by each UAV based solely on local observations during the execution phase. The monotonicity constraint is expressed as: ； Where i = 1, 2, …, N, Indicates the total value of the union The minute changes Represents the local value of the i-th drone. The minute changes Indicates the total value of the union Local value of the i-th UAV The partial derivatives, if the partial derivatives satisfy 0 indicates that when the local value increases, the total combined value does not decrease.

6. The method of claim 1, wherein, The step of calculating the global reward value based on the reward function of the fused artificial potential field according to the mapping relationship specifically includes: According to the distance change between the unmanned aerial vehicle and the corresponding target, a target attraction reward value is calculated : ； in, This represents the gain coefficient of the target attraction term, used to adjust the reward intensity of the drone's movement toward a preset target. This represents the position vector of the i-th drone at the previous time step. This represents the preset target location corresponding to the i-th drone. This represents the position vector of the i-th drone at time t. Represents the L2 norm; For each obstacle, determine whether the distance between the drone and the obstacle is less than the obstacle's safe distance threshold. If so, calculate the obstacle repulsion penalty value, and sum the penalties for all obstacles that triggered the penalty to obtain the total obstacle repulsion penalty value. ; ； in, This represents the total number of obstacles in the environment. Index representing obstacles, This represents the gain coefficient for obstacle removal, used to adjust the penalty intensity when the drone approaches an obstacle. This represents the distance between the i-th drone and the m-th obstacle. Indicates the safe distance threshold from obstacles. Indicates an indicator function; For other drones, determine if the distance between the current drone and other drones is less than the inter-drone safe distance threshold. If so, calculate the inter-drone safety penalty value and sum the values ​​for all other drones that triggered the penalty to obtain the total inter-drone safety penalty value. : ； in, This represents the number of drones, j represents the index of other drones, and i represents the index of the current drone. This represents the gain coefficient of the inter-drone safety constraint, used to adjust the penalty intensity when drones are too close together. This represents the distance between the i-th and j-th drones. Indicates the safe distance threshold between machines; determining whether the UAV has crossed the boundary, and if so, calculating a boundary constraint penalty value ; ； wherein, represents a boundary constraint gain coefficient, used to adjust the punishment intensity when the UAV approaches the environment boundary or crosses the boundary, represents the crossing boundary degree of the i-th UAV at time t; Determine whether the drone has triggered a termination event. If so, assign a termination event reward value to the drone based on the type of termination event. The termination event types include: reaching the target, colliding, or seriously crossing the boundary. The target attraction reward value, obstacle repulsion total penalty value, inter-aircraft safety total penalty value, boundary constraint penalty value and termination event reward value are weighted and summed according to preset weights to obtain a local reward value of the ith unmanned aerial vehicle : ； wherein, , , , and are preset weighting coefficients, respectively. The local reward values ​​of all drones are aggregated to obtain the global reward value.

7. The method of claim 1, wherein, The process of updating the APF-MASAC network model parameters and soft-updating the corresponding target network specifically includes: A batch of experience samples are randomly sampled from the experience replay pool. Each experience sample contains the current global state, joint action, global reward value, local observation information at the next time step, global state at the next time step, and task termination flag. For each sampled empirical sample, the local action for the next time step is generated based on the local observation information at the next time step through the target policy network, and the value of the local action at the next time step is calculated through the target local value network. The value of the local action at the next moment and the global state at the next moment are input into the target global hybrid network to obtain the target joint total value; After subtracting the weighted entropy terms of each drone strategy from the total joint value of the objective, the global reward value and discount factor are combined, and the future reward term in the termination state is reduced to zero using the termination flag, to construct the time-series difference objective value. The parameters of the local value network and the global hybrid network are updated by minimizing the mean square error between the total value of joint actions and the temporal difference objective value, and the parameters of the local policy network are updated by maximizing the action value objective with entropy regularization. The updated network parameters are synchronized to the corresponding target network according to a fixed ratio, thus completing the soft update of the target network.

8. A multi-UAV cooperative safe path planning system, characterized in that, The multi-UAV collaborative safe path planning system includes: The environment construction module is used to construct an environment for multi-UAV collaborative planning, and to obtain the corresponding targets configured in the environment to form a mapping relationship between UAVs and corresponding targets; The joint action calculation module is used to obtain the current local observation information and current global state of each UAV at each decision moment from the environment, input them into the APF-MASAC network model to calculate the total value of joint actions, and output the joint actions. The reward feedback module is used to perform joint actions in the environment, update the status of each UAV, calculate the global reward value based on the reward function of the fused artificial potential field, and return the local observation information and the global state of the next time step from the environment. The model update module is used to sample the joint action, the global reward value, the local observation information at the next time step, and the global state, update the APF-MASAC network model parameters, and softly update the corresponding target network. The iterative output module is used to take the local observation information of the next moment as the new current local observation information and iteratively train the APF-MASAC network with updated parameters until each UAV reaches the corresponding target or meets the termination condition. The module outputs the flight trajectory, action sequence and obstacle avoidance results of each UAV through the iteratively trained APF-MASAC network model.

9. A terminal, characterized by comprising: The terminal includes: a memory, a processor, and a multi-UAV cooperative safe path planning program stored in the memory and executable on the processor. When the multi-UAV cooperative safe path planning program is executed by the processor, it implements the steps of the multi-UAV cooperative safe path planning method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a multi-UAV cooperative safe path planning program, which, when executed by a processor, implements the steps of the multi-UAV cooperative safe path planning method as described in any one of claims 1-7.

Images