Path planning method and system for heterogeneous unmanned aerial vehicle cluster

Abstract

The invention relates to a path planning method and system for a heterogeneous unmanned aerial vehicle cluster. The method comprises the following steps: constructing a heterogeneous unmanned aerial vehicle motion model, and constructing a leader unmanned aerial vehicle-follower unmanned aerial vehicle path planning model through a Markov decision process according to the heterogeneous unmanned aerial vehicle motion model; generating a plurality of training environments; selecting a training environment for training; and judging whether remaining unselected training environments exist or not, if yes, constructing a new leader unmanned aerial vehicle-follower unmanned aerial vehicle path planning model through a knowledge migration technology and returning to execute training, and if not, taking the new leader unmanned aerial vehicle-follower unmanned aerial vehicle path planning model as a final leader unmanned aerial vehicle-follower unmanned aerial vehicle path planning model to realize path planning. The method effectively improves the training efficiency, adaptability and stability of a multi-unmanned-aerial-vehicle cooperative control system through a leader unmanned aerial vehicle-follower unmanned aerial vehicle path planning model, and is especially suitable for multi-unmanned-aerial-vehicle path planning and formation maintenance tasks in a complex environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of path planning, more particularly, to a path planning method and system for heterogeneous unmanned aerial vehicle cluster. BACKGROUND

[0002] With the rapid development of unmanned aerial vehicle technology, multi-unmanned aerial vehicle cooperative operation shows great application potential in military reconnaissance, search and rescue, border patrol, agricultural monitoring and other fields. Compared with single unmanned aerial vehicle, multi-unmanned aerial vehicle system has higher task efficiency, stronger robustness and wider coverage.

[0003] At present, the traditional multi-unmanned aerial vehicle control method has the following technical defects: (1) Static control strategy: the traditional method adopts preset rules or trajectory tracking algorithm, which is difficult to adapt to highly dynamic and unknown environmental changes.

[0004] (2) Low training efficiency: the existing method based on reinforcement learning has too large exploration space in the initial training, which leads to slow convergence and easy to fall into local optimum, especially when the task complexity increases.

[0005] (3) Poor multi-agent coordination: lack of effective leader-follower mechanism, unable to handle the coordination problem between heterogeneous unmanned aerial vehicles, and difficult to maintain formation.

[0006] (4) Difficulty in reward design: it is difficult to design a reward function that can promote long-term goal realization and ensure short-term behavior safety.

[0007] (5) Weak knowledge transfer ability: unable to effectively transfer the learned simple task knowledge to more complex scenarios, resulting in the need to learn from scratch for each new task.

[0008] Although deep reinforcement learning has made significant progress in single-agent control in recent years, in the field of multi-unmanned aerial vehicle cooperative control, especially in complex scenarios involving multiple roles and multiple constraints, there is still a lack of effective progressive learning mechanism and unified decision-making framework. SUMMARY

[0009] Therefore, it is necessary to provide a path planning method and system for heterogeneous unmanned aerial vehicle cluster for better multi-unmanned aerial vehicle cooperation effect in view of the above technical problems.

[0010] One of the path planning methods for heterogeneous unmanned aerial vehicle cluster includes: S1: constructing a heterogeneous unmanned aerial vehicle motion model, and constructing a leader unmanned aerial vehicle-follower unmanned aerial vehicle path planning model through Markov decision process according to the heterogeneous unmanned aerial vehicle motion model; S2: generating multiple training environments; S3: select a training environment, train the leader drone-follower drone path planning model, and obtain a trained leader drone-follower drone path planning model; S4: determine whether there are remaining unselected training environments, if yes, construct a new leader drone-follower drone path planning model based on the trained leader drone-follower drone path planning model through knowledge transfer technology and return to step S3, if no, take the trained leader drone-follower drone path planning model as the final leader drone-follower drone path planning model, and execute step S5; S5: realize path planning through the final leader drone-follower drone path planning model.

[0011] Further, in step S1, constructing the heterogeneous drone motion model includes: mathematically modeling a leader drone and multiple follower drones from time t to time t+1; using kinematic equations to define movement rules for all drones:

[0012]

[0013] The state of each drone is described by four key variables: and represents the two-dimensional coordinates of the drone at time t; represents the linear velocity of the drone at time t; represents the heading angle of the drone at time t; control input definition: represents linear acceleration, represents angular velocity; state update: new coordinates = old coordinates + component of velocity in that direction x time step; new velocity = old velocity + acceleration x time step; new orientation = old orientation + angular velocity x time step; using a constraint condition framework, respectively constraining a leader drone and multiple followers.

[0014] Further, using a constraint condition framework, respectively constraining a leader drone and multiple followers includes: modeling the leader drone as the main navigation unit of the swarm, constraint conditions: velocity constraint: velocity within the range of 100 m / s to 200 m / s; maneuverability constraint: maximum linear acceleration not exceeding 3 m / s², maximum angular velocity not exceeding 0.6 rad / s; environmental constraint: ensuring that its flight path is always within the map boundary and maintains a safety radius not less than all obstacles The distance; The follower drone is modeled as an agile execution unit whose core task is to closely follow the leader and maintain a pre-defined formation. Constraints for the follower drone include: speed between 100 m / s and 300 m / s; maneuverability constraints: maximum linear acceleration not exceeding 6 m / s², maximum angular velocity not exceeding 1.2 rad / s²; and environmental constraints: ensuring its flight path always remains within the map boundaries and maintains a safe radius from all obstacles. The distance; Furthermore, in step S1, the Markov decision process includes: Define the state space as a seven-dimensional vector: The following information is obtained in real time from the actual location within the simulation environment: : Normalized current position coordinates of the drone; : The current normalized velocity; : Normalized current heading angle; : Normalized target position coordinates; Obstacle warning sign; Define the action space as a two-dimensional continuous vector:

[0015] Among them; steering control: Control the change of heading angle; Acceleration control: Control the speed increase or decrease.

[0016] State transition function:

[0017] The reward function is set to a multi-objective combination form:

[0018] : The weight of each reward function.

[0019] Calculate the set reward items Target Reward:

[0020] Obstacle penalty:

[0021] Formation Rewards:

[0022] Efficiency Rewards:

[0023] Exploration Rewards:

[0024] Further, in step S1, the leader drone-follower drone path planning model comprises: a policy network, a value network and a SAC controller; wherein the policy network is an attention-based Actor network: dynamically learns and aggregates information from other agents through attention mechanism; uses a general component to map raw observations to a low-dimensional embedding , whose mapping formula is as follows:

[0025] the leader drone learns a policy according to its own observation and the observations of all follower drones ; its information flow and formula are as follows: leader drone observation encoding:

[0026] follower drone observation encoding: for each follower drone :

[0027] the leader drone uses its own encoding as the query , the encodings of all follower drones as the keys and the values ; the attention mechanism calculates the context vector: :

[0028]

[0029] In multi-head attention, this process is executed in parallel times, each head learns different attention weights, and then the results are spliced or averaged: , ,

[0030]

[0031]

[0032] feature fusion: ​​

[0033] Policy parameter output:

[0034]

[0035]

[0036]

[0037] Policy: ; Follower drone actor network: follower drones learn policy from their own observations , leader drone observations and observations of other follower drones : Own observation encoding: Context observation encoding:

[0038]

[0039] for

[0040] Follower drones can observe information of all other drones as their context and construct global context information:

[0041] Attention weighting Utilize their own encoding as query , global context encoding as key and value .

[0042]

[0043] Feature fusion:

[0044] Policy parameter output: generate and from ; Policy: ​

[0045] Value network for structured attention Critic network: evaluate joint state Take joint action Q value; its function is to encode the observation-action of the UAV into embedding ; the mapping formula is as follows:

[0046] Critic network structure and information flow: Leader UAV ; Follower UAV : ; Leader UAV Q value calculation ( ) ; Encoding of leader UAV as Query; Encoding of all follower UAVs as Key and Value.

[0047]

[0048]

[0049]

[0050] In the formula , the value of the leader UAV under the current global state-action is considered, and the behavior of all follower UAVs is considered; Follower UAV Q value calculation ( ) : Build global embedding context:

[0051] For each follower UAV i, its own encoding as Query; as Key and Value;

[0052]

[0053]

[0054] Where, ​​For the follower drone in the current global state-action context The value of this was considered, taking into account the behavior of the leader drone and all other follower drones. Follower Drone Critic Network: Each follower drone corresponds to a follower drone Critic Network, whose parameters... Critic network parameters of the leader drone Soft update:

[0055] The target network is used to compute the target Q-value to stabilize the learning of the Bellman equation; IV. Entropy Regularization SAC controller: Maximizes both expected cumulative return and policy entropy; Strategy learning objectives:

[0056] in Is the strategy in the state? Entropy below; It's a temperature parameter, a trade-off between the importance of reward and entropy.

[0057] Automatically adjust temperature parameters : Learning parameters:

[0058] loss function :

[0059] Furthermore, in step S2, generating multiple training environments specifically involves: Definition of physical space: A two-dimensional planar simulation environment was constructed, with a size of 7000*6000 meters; the space includes boundaries, obstacles, and target points; Procedural generation designed for generalization capability: To ensure that the trained model has strong generalization capability, the generation of each environment includes the following: Randomization of initial state of drones: The initial position, speed and orientation angle of all drones are randomly set within a preset effective range; Randomization of obstacle layout: The number, position and size of obstacles in the environment are randomly generated.

[0060] The complexity increases with each learning-driven lesson: each generated environment automatically increases in difficulty, introducing more followers and more complex obstacle layouts; Goal setting rules: Definition and representation of the target: the main target is for the leader drone to reach a designated target area; this target is represented as a specific coordinate point in a two-dimensional environment ); Randomization of the target: the location of the target point will be randomly placed in each environment generation; Quantification of success conditions: when the leader drone is within a straight-line distance of the target point that is less than a pre-set threshold, the task is considered successful; Integration with the training process: the achievement of the target is directly related to the end of the training round; a round will terminate when one of the following conditions occurs: Success: the leader reaches the target area; Failure: the leader collides with any obstacle or boundary; Timeout: exceeds the pre-set maximum decision step number and still does not complete the task; Target reward: monitor the distance between the leader drone and the target; when the leader drone is within a distance of the target that is less than the set threshold, a large reward is given, indicating successful arrival at the target; otherwise, a reward of -0.001 times the distance to the target is given, guiding the leader drone to approach the target.

[0061] Formation reward: if the leader drone and all follower drones are within a set distance, no reward or punishment is given; otherwise, a punishment of -0.001 times the distance between the leader drone and the farthest follower drone is given, punishing the failure of the formation; Efficiency reward: when the leader drone and all follower drones are within a threshold, each follower drone has a speed difference of less than 0.1, giving a reward of +1; otherwise, no reward or punishment is given; Boundary reward: if the leader drone and all follower drones are close to the set boundary, a punishment of -0.1 is obtained; Obstacle reward: when the leader drone and all follower drones are within a threshold distance from the obstacle, a large punishment value is given, resulting in death and the end of the round; when the agent is within a threshold distance from the obstacle, a punishment of -0.2 is given; otherwise, no reward or punishment is given.

[0062] Further, in step S3, the SAC algorithm is used to judge the degree of mastery of the leader drone-follower drone path planning model for the current environment based on the performance improvement rate during the training process, and it is judged whether it can enter the next stage.

[0063] Further, in step S4, the leader drone-follower drone path planning model after training is updated to a new leader drone-follower drone path planning model through knowledge transfer technology, specifically: S4.1 stage completion and migration trigger: when the training is completed, and its performance index reward stability satisfies the preset switching threshold, the knowledge migration process is started; S4.2 parameter inheritance of policy network: after starting, a new policy network is constructed; the parameter vector of the new network is initialized by directly copying the parameter vector of the previous stage network; the process is represented as:

[0064] S4.3 parameter reset of value network: a new value network is constructed, and the parameter vector is resampled from a predefined random distribution D; the process is represented as: S4.4 synchronous migration of entropy adjustment parameter: the learnable temperature parameter is regarded as part of the policy knowledge and is synchronously migrated; the value obtained after the stage s training is completed is denoted as , which represents the optimal exploration-exploitation balance matching the policy ; when entering the new stage s+1, the parameter is directly inherited; the process is represented as:

[0065] S4.5 new stage training start: after the above asymmetric parameter migration is completed, a new leader UAV-follower UAV path planning model is obtained.

[0066] Further, in step S5, specifically: leader UAV decision: ; follower UAV decision: ; formation maintenance index, according to the proximity of the energy consumption and the proximity of the distance of each UAV, the formation constraint is performed: formation error calculation:

[0067] speed consistency: .

[0068] In addition, the present application provides a path planning system for a heterogeneous UAV cluster, comprising: model construction module: constructing a heterogeneous UAV motion model, and constructing a leader UAV-follower UAV path planning model through a Markov decision process according to the heterogeneous UAV motion model; ​​​​​environment generating module: generating multiple training environments; training module: selecting a training environment, training the leader UAV-follower UAV path planning model, and obtaining a trained leader UAV-follower UAV path planning model; determining module: determining whether there are remaining unselected training environments, if yes, constructing a new leader UAV-follower UAV path planning model according to the trained leader UAV-follower UAV path planning model through knowledge transfer technology and returning to execute the training module, if no, taking the trained leader UAV-follower UAV path planning model as a final leader UAV-follower UAV path planning model, and executing the path planning module; path planning module: realizing path planning through the final leader UAV-follower UAV path planning model.

[0069] The method effectively improves the training efficiency, adaptability and stability of the multi-UAV cooperative control system through the leader UAV-follower UAV path planning model, and is particularly suitable for multi-UAV path planning and formation maintenance tasks in complex environments. BRIEF DESCRIPTION OF DRAWINGS

[0070] Figure 1 a flowchart of a path planning method for a heterogeneous UAV cluster in an embodiment; Figure 2 a system block diagram of a path planning method for a heterogeneous UAV cluster in an embodiment; DETAILED DESCRIPTION In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0071] Embodiment One: The present embodiment provides a path planning method for a heterogeneous UAV cluster as shown in Figure 1 The present embodiment provides a path planning method for a heterogeneous UAV cluster as shown in S1: constructing a heterogeneous UAV motion model, and constructing a leader UAV-follower UAV path planning model through a Markov decision process according to the heterogeneous UAV motion model; S2: generating multiple training environments; S3: selecting a training environment, training the leader UAV-follower UAV path planning model, and obtaining a trained leader UAV-follower UAV path planning model; S4: judging whether there is still a remaining unselected training environment, if yes, constructing a new leader UAV-follower UAV path planning model according to the leader UAV-follower UAV path planning model after training through knowledge transfer technology and returning to execute step S3, if no, taking the leader UAV-follower UAV path planning model after training as the final leader UAV-follower UAV path planning model and executing step S5; S5: realizing path planning through the final leader UAV-follower UAV path planning model.

[0072] The embodiment effectively improves the training efficiency, adaptability and stability of the multi-UAV cooperative control system through the leader UAV-follower UAV path planning model, and is particularly suitable for multi-UAV path planning and formation maintenance tasks in complex environments.

[0073] Embodiment Two: The embodiment is further disclosed on the basis of Embodiment One: Further, in step S1, constructing the heterogeneous UAV motion model comprises: mathematically modeling a leader UAV and a plurality of follower UAVs from time t to time t+1; using kinematic equations to define movement rules for all UAVs:

[0074]

[0075] The state of each UAV is described by four key variables: and represents the two-dimensional coordinates of the UAV at time point t; represents the linear velocity of the UAV at time point t; represents the heading angle of the UAV at time point t; control input definition: represents linear acceleration, represents angular velocity; state update: new coordinates = old coordinates + component of velocity in that direction x time step; new velocity = old velocity + acceleration x time step; new orientation = old orientation + angular velocity x time step; using a constraint condition framework, respectively constraining a leader UAV and a plurality of followers.

[0076] Further, using a constraint condition framework, respectively constraining a leader UAV and a plurality of followers comprises: The leader drone is modeled as the primary navigation unit of the swarm, with the following constraints: Speed ​​constraint: speed within the range of 100m / s to 200m / s; Maneuverability constraint: maximum linear acceleration not exceeding 3m / s², maximum angular velocity not exceeding 0.6rad / s; Environmental constraint: ensuring its flight path always remains within the map boundaries and maintains a safe radius from all obstacles. The distance; The follower drone is modeled as an agile execution unit whose core task is to closely follow the leader and maintain a pre-defined formation. Constraints for the follower drone include: speed between 100 m / s and 300 m / s; maneuverability constraints: maximum linear acceleration not exceeding 6 m / s², maximum angular velocity not exceeding 1.2 rad / s²; and environmental constraints: ensuring its flight path always remains within the map boundaries and maintains a safe radius from all obstacles. The distance; Furthermore, in step S1, the Markov decision process includes: Define the state space as a seven-dimensional vector: The following information is obtained in real time from the actual location within the simulation environment: : Normalized current position coordinates of the drone; : The current normalized velocity; : Normalized current heading angle; : Normalized target position coordinates; Obstacle warning sign; Define the action space as a two-dimensional continuous vector:

[0077] Among them; steering control: Control the change of heading angle; Acceleration control: Control the speed increase or decrease.

[0078] State transition function:

[0079] The reward function is set to a multi-objective combination form:

[0080] : The weight of each reward function.

[0081] Calculate the set reward items Target Reward:

[0082] Obstacle penalty:

[0083] Platoon reward:

[0084] Efficiency reward:

[0085] Exploration reward:

[0086] Further, in step S1, the leader drone-follower drone path planning model comprises: a policy network, a value network, and a SAC controller; wherein the policy network is an attention-based Actor network: dynamically learns and aggregates information from other agents through an attention mechanism; uses a universal component to map raw observations to a low-dimensional embedding , whose mapping formula is as follows:

[0087] leader drone learns a policy according to its own observation and the observations of all follower drones ; its information flow and formula are as follows: leader drone observation encoding:

[0088] follower drone observation encoding: for each follower drone :

[0089] the leader drone uses its own encoding as the query , the encodings of all follower drones as the keys and values ; the attention mechanism calculates the context vector: :

[0090]

[0091] In multi-head attention, this process is executed times in parallel, each head learning different attention weights, and then the results are spliced or averaged: , , ​​

[0092]

[0093]

[0094] Feature fusion:

[0095] Policy parameter output:

[0096]

[0097]

[0098]

[0099] Policy: ; Follower drone actor network: follower drone learns policy from its own observation , leader drone observation and other follower drones observations : Own observation encoding:

[0100] Context observation encoding:

[0101] for

[0102] Follower drone can observe all other drones' information as its context and construct global context information:

[0103] Attention weighting Utilize its own encoding as query , global context encoding as key and value .

[0104]

[0105] Feature fusion:

[0106] Strategy parameter output: From generate and ; Strategy:

[0107] Value Network is a structured attention Critic network: evaluating joint states. The following joint actions were taken. The Q value; its function is to enable drones Observation-action Encoded as an embedding The mapping formula is as follows:

[0108] Critic network structure and information flow: Leader Drone ; Follower drone : ; Leader drone Q-value calculation ( ); Leader Drone encoding As a query; All follower drone codes As Key and Value.

[0109]

[0110]

[0111]

[0112] In the formula Given the current global state-action, the value of the leader drone is determined, taking into account the behavior of all follower drones. Follower drone Q-value calculation ( ): Constructing a global embedding context:

[0113] For each follower drone i, its own encoding As a query; As Key and Value;

[0114]

[0115]

[0116] where, is the value of the follower drone under the current global state-action, and takes into account the behavior of the leader drone and all other follower drones; Follower Drone Critic Network: Each follower drone corresponds to a follower drone Critic network, whose parameters are updated by the leader drone Critic network parameters softly:

[0117] Target Network: The target network is used to calculate the target Q value to stabilize the learning of the Bellman equation. IV. Entropy Regularization SAC Controller: The SAC controller maximizes the expected cumulative reward while also maximizing the entropy of the policy. Policy Learning Objective:

[0118] where is the entropy of the policy under state is the temperature parameter, which balances the importance of rewards and entropy.

[0119] Automatic Adjustment of Temperature Parameter : Learning Parameters:

[0120] Loss Function :

[0121] Further, in step S2, multiple training environments are generated, specifically: Definition of Physical Space: A two-dimensional plane simulation environment is constructed, with a size of 7000*6000 meters; the space contains boundaries, obstacles, and target points. Programmed Generation for Generalization Ability: To ensure that the trained model has strong generalization ability, the production of each environment includes the following points: randomization of initial state of drones: the initial position, speed, and orientation angle of all drones are randomly set within the pre-set valid range; randomization of obstacle layout: the number, position, and size of obstacles in the environment are randomly generated.

[0122] ​​Course learning driven complexity increment: the difficulty of each generation environment is automatically increased, and more followers and more complex obstacle layout are introduced; Goal setting rules: Definition and representation of goals: the main goal is for the leader drone to reach a designated target area; this goal is represented as a specific coordinate point in a two-dimensional environment ) in the two-dimensional environment; Randomization of goals: the location of the target point is randomly placed in each environment generation; Quantification of success conditions: when the leader drone is within a certain distance of the target point, the task is considered successful; Combination with training process: the achievement of the goal is directly related to the end of the training round; a round will be terminated when one of the following conditions occurs: Success: the leader reaches the target area; Failure: the leader collides with any obstacle or boundary; Timeout: exceeds the maximum number of decision steps and still does not complete the task; Goal reward: monitor the distance between the leader drone and the target; when the leader drone is within a certain distance of the target, a large reward is given, indicating successful arrival at the target; otherwise, a reward of -0.001*distance is given, guiding the leader drone to approach the target.

[0123] Formation reward: if the leader drone and all follower drones are within a certain distance, no reward or punishment is given; otherwise, a punishment of -0.001*the distance between the leader drone and the farthest follower drone is given, punishing the failure of the formation; Efficiency reward: when the leader drone and all follower drones are within a certain distance, each follower drone has a speed difference of less than 0.1, and a reward of +1 is given; otherwise, no reward or punishment is given; Boundary reward: if the leader drone and all follower drones are close to the set boundary, a penalty of -0.1 is given; Obstacle reward: when the leader drone and all follower drones are within a certain distance of the obstacle, a large penalty is given, resulting in death and the end of the round; when the agent is within a certain distance of the obstacle, a penalty of -0.2 is given; otherwise, no reward or punishment is given.

[0124] Further, in step S3, it includes: using the SAC algorithm to judge the degree of mastery of the leader drone-follower drone path planning model for the current environment based on the performance improvement rate during the training process, and judging whether it can enter the next stage.

[0125] Furthermore, in step S4, the trained leader-follower drone path planning model is updated to the new leader-follower drone path planning model using knowledge transfer technology, specifically as follows: S4.1 Phase Completion and Transfer Trigger: When training is complete and its performance metrics reward stability... The knowledge transfer process is initiated when the preset switching threshold is met. S4.2 Policy Network Parameter Inheritance: After startup, a new policy network is constructed; the parameter vector of the new network will be directly copied from the parameter vector of the previous stage network. To perform initialization; this process is represented as: ←

[0126] S4.3 Parameter Resetting of Value Network: To construct a new value network, the parameter vectors are resampled from a predefined random distribution D; this process is represented as: ~D S4.4 Synchronous Transfer of Entropy Adjustment Parameters: Transferring Learnable Temperature Parameters Treated as part of policy knowledge and transferred synchronously; obtained after training in stage s. Value, denoted as This represents the strategy Matching optimal exploration - utilizing balance; this parameter will be directly inherited when entering a new stage s+1; the process is represented as: ←

[0127] S4.5 New Stage Training Start: After completing the above asymmetric parameter transfer, a new leader-follower drone path planning model is obtained.

[0128] Furthermore, in step S5, specifically: Leaders' drone decisions: ; Follower drone decision-making: ; Formation maintenance metrics are established based on the similarity in energy consumption and distance among the drones: Formation error calculation:

[0129] Speed ​​consistency: .

[0130] The embodiment effectively improves the training efficiency, adaptability and stability of the multi-unmanned aerial vehicle cooperative control system through the leader-unmanned aerial vehicle follower path planning model, and is particularly suitable for multi-unmanned aerial vehicle path planning and formation maintenance tasks in complex environments.

[0131] Embodiment three The embodiment provides a path planning system for a heterogeneous unmanned aerial vehicle cluster, and the path planning system comprises: Figure 2 A path planning system for a heterogeneous unmanned aerial vehicle cluster comprises: A model construction module is configured to construct a heterogeneous unmanned aerial vehicle motion model and construct a leader-unmanned aerial vehicle follower path planning model through a Markov decision process according to the heterogeneous unmanned aerial vehicle motion model. An environment generation module is configured to generate a plurality of training environments. A training module is configured to select a training environment, train the leader-unmanned aerial vehicle follower path planning model, and obtain a trained leader-unmanned aerial vehicle follower path planning model. A judgment module is configured to determine whether there is a remaining training environment that has not been selected, and if there is, to construct a new leader-unmanned aerial vehicle follower path planning model through a knowledge transfer technology according to the trained leader-unmanned aerial vehicle follower path planning model and return to execute the training module, and if there is not, to use the trained leader-unmanned aerial vehicle follower path planning model as a final leader-unmanned aerial vehicle follower path planning model and execute a path planning module. The path planning module is configured to implement path planning through the final leader-unmanned aerial vehicle follower path planning model.

[0132] The embodiment effectively improves the training efficiency, adaptability and stability of the multi-unmanned aerial vehicle cooperative control system through the leader-unmanned aerial vehicle follower path planning model, and is particularly suitable for multi-unmanned aerial vehicle path planning and formation maintenance tasks in complex environments.

[0133] Embodiment four The embodiment provides a path planning system for a heterogeneous unmanned aerial vehicle cluster, and the path planning system comprises: A heterogeneous unmanned aerial vehicle motion model module is configured to construct a Markov decision process modeling module for different types of unmanned aerial vehicles. An adaptive curriculum learning module is configured to switch task difficulty according to learning progress. A multi-agent soft actor critic algorithm module is configured to implement a SAC multi-agent reinforcement learning algorithm combined with an attention mechanism. An environment interface module is configured to provide state processing and reward shaping functions. A knowledge transfer module is configured to transfer value network parameters, policy network parameters and entropy parameters.

[0134] The adaptive curriculum learning module comprises: A curriculum task generation unit for creating tasks of different difficulties; A learning progress evaluation unit for calculating the performance improvement rate of the agent on the current task; The multi-agent soft actor critic algorithm module comprises: A policy network for outputting action distribution parameters and implementing action sampling; A value network for evaluating state-action values and reducing overestimation bias; An adaptive entropy coefficient adjustment mechanism for automatically balancing exploration and utilization; An experience pool for storing experiences into "new experiences" and "old experiences", focusing on utilizing old experiences for training in the early stage, and more utilizing new experiences to strengthen the current stage ability in the later stage, and old experiences are gradually reduced.

[0135] The environment interface module comprises a state processor for implementing state normalization and role differentiation processing; and according to the feedback obtained from the environment module, rewards and punishments are implemented for the master and slave.

[0136] The knowledge transfer module calculates the task similarity, obtains the transfer ratio according to the similarity, and transfers three key parameters: policy parameters including the policy network of the master and the slave; entropy parameters including the entropy parameters of the master and the slave; and value parameters including the centralized value network parameters.

[0137] The leader unmanned aerial vehicle-following unmanned aerial vehicle path planning model effectively improves the training efficiency, adaptability and stability of the multi-unmanned aerial vehicle cooperative control system, and is particularly suitable for multi-unmanned aerial vehicle path planning and formation maintenance tasks in complex environments.

[0138] Embodiment five The embodiment provides a path planning method for a heterogeneous unmanned aerial vehicle cluster, comprising the following steps: Step 1, constructing a heterogeneous unmanned aerial vehicle motion model and a Markov decision process model; Step 2, initializing an adaptive curriculum generator and a multi-agent SAC algorithm; Step 3, automatically generating and selecting training tasks according to the learning progress; Step 4, performing policy learning and updating based on a multi-agent SAC algorithm combined with an attention mechanism; Step 5, transferring knowledge in simple tasks to complex tasks using knowledge transfer technology.

[0139] The step of automatically generating and selecting training tasks according to the learning progress specifically comprises: Calculating the performance improvement index in the window:

[0140] wherein is the average of the rewards obtained in the round, is the standard deviation of the rewards obtained in the round, is the reward stability used to evaluate the calculation window.

[0141] The steps of performing policy learning and updating based on the multi-agent SAC algorithm specifically include: Step 1, obtaining the current state and performing action sampling: ; Step 2, interacting with the environment and obtaining the reward, the next state and the termination flag; Step 3, storing the experience into a dynamic decay experience replay buffer Step 4, updating the policy network, the value network and the entropy coefficient.

[0142] The update target of the multi-agent SAC algorithm is: Policy target:

[0143] Value target:

[0144] Entropy coefficient target:

[0145] The knowledge transfer technology mainly transfers the policy network parameters, value network parameters and entropy parameters in the MASAC algorithm.

[0146] The leader-unmanned aerial vehicle-following unmanned aerial vehicle path planning model effectively improves the training efficiency, adaptability and stability of the multi-unmanned aerial vehicle cooperative control system, and is particularly suitable for multi-unmanned aerial vehicle path planning and formation maintenance tasks in complex environments.

[0147] Embodiment six The embodiment further discloses based on embodiment one: an attention and curriculum-driven multi-agent SAC system for a heterogeneous unmanned aerial vehicle system, which includes five main parts, namely an environment module, a curriculum generator, a MASAC controller, a trainer and a knowledge transferor. Among them: The environment module includes two main components: a state processor and a reward shaper, which are used to provide a state processor and a reward shaper, a standardized environment interface, and improve the generalization ability of the algorithm.

[0148] The adaptive curriculum learning module in the curriculum generator can adjust the task difficulty according to the learning progress, design a task generation and selection mechanism, and realize progressive learning from simple to complex.

[0149] The MASAC controller is equipped with a multi-agent soft actor critic (MASAC) algorithm module, to implement a hierarchical multi-agent reinforcement learning algorithm, including a policy network, a value network, and an entropy regularization mechanism.

[0150] Step S10: A heterogeneous UAV model is used to mathematically model the five UAVs from time t to time t+1, to obtain:

[0151]

[0152] 1. General modeling: First, kinematic equations are used to define how all UAVs move.

[0153] This step establishes a basic, unified motion model for each UAV in the system. This model describes how a UAV changes its state within discrete time steps (dt) ) according to its control instructions.

[0154] The state of each UAV is described by four key variables: and represent the two-dimensional coordinates of the UAV at time t; represent the linear velocity of the UAV at time t; represent the heading angle of the UAV at time t. Control inputs are defined as: represent the linear acceleration, represent the angular velocity.

[0155] The state update is updated by the following three points: new coordinates = old coordinates + component of velocity in that direction × time step; new velocity = old velocity + acceleration × time step; new orientation = old orientation + angular velocity × time step.

[0156] Differentiated modeling: Then, using a constraint condition framework, different values are substituted for the leader and follower (such as maximum / minimum speed, maximum acceleration, etc.), to accurately mathematically describe their respective characteristics and behavior boundaries.

[0157] The leader drone is modeled as the main navigation unit of the swarm, with its dynamic characteristics more focused on stability to perform strategic global path planning and target guidance tasks. Its specific operational boundaries are defined by a series of strict constraints: its speed is limited to a range of 100 m / s to 200 m / s to ensure smooth trajectories; in terms of maneuverability, its maximum linear acceleration does not exceed 3 m / s², and its maximum angular velocity does not exceed 0.6 rad / s. In addition, the leader must also comply with environmental constraints, ensuring that its flight path is always within the map boundaries and maintains a distance of no less than a safety radius from all obstacles.

[0158] The follower drone is modeled as an agile execution unit, whose core task is to closely follow the leader and maintain a preset formation, which requires it to have stronger maneuverability to quickly respond to the dynamic changes of the leader. To achieve this goal, the performance parameters of the follower are improved: its speed range is expanded to 100 m / s to 300 m / s, allowing it to easily catch up or adjust its position; its maneuverability is also significantly enhanced, with a maximum linear acceleration of 6 m / s² and a maximum angular velocity of 1.2 rad / s, both of which are twice those of the leader. Like the leader, the follower must also fly within the map boundaries and maintain a safe distance from all obstacles.

[0159] In this way, a heterogeneous drone swarm system model is established, which includes different roles. Step S20, a Markov decision process model is constructed to define the state space, action space, and reward function, providing a mathematical basis for reinforcement learning.

[0160] The state space is defined as a seven-dimensional vector: The following information is obtained in real time through the actual position of the simulation environment: : normalized current position coordinates of the drone; : normalized current speed; : normalized current heading angle; : normalized target position coordinates; : obstacle warning flag; The action space is defined as a two-dimensional continuous vector:

[0161] where; steering control: controls the change in heading angle; acceleration control: controls the increase or decrease in speed.

[0162] State transition function:

[0163] The reward function is set as a multi-objective combination:

[0164] : The weight of each reward function.

[0165] The reward item calculation is set Target reward:

[0166] Obstacle penalty:

[0167] Formation reward:

[0168] Efficiency reward:

[0169] Exploration reward:

[0170] Step S30, the agent interacts with the environment module iteratively, so as to realize the reward and punishment.

[0171] In the system, the setting of the environment follows the following key principles: 1. Definition of physical space: We constructed a two-dimensional plane simulation environment with a size of 7000 x 6000 meters. This space is the main world of the agent's activities, including boundaries, obstacles and target points. The environment is developed based on the OpenAI Gym framework and visualized using Pygame, which can simulate dynamic flight scenarios.

[0172] 2. Programmatic generation for generalization: To ensure that the trained model has strong generalization ability, the environment is randomly generated programmatically at the beginning of each new training round. This includes the following points: randomization of the initial state of the agent: the initial position, speed and orientation angle of all drones are randomly set within the preset valid range; randomization of obstacle layout: the number, position and even size of obstacles in the environment are randomly generated.

[0173] 3. Curriculum learning driven complexity increment: The complexity of the environment is not fixed, but gradually increases with the progress of curriculum learning. In the early stage of training, the environment is very simple with only 1 follower and 1 obstacle, allowing the agent to quickly learn basic flight and coordination skills. As the agent's performance improves, the environment will automatically increase the difficulty, introducing more followers and more complex obstacle layouts, increasing to 4 followers and 5 obstacles.

[0174] The target setting follows the following rules: 1. Definition and representation of the goal: The main goal of the leader drone is to reach a designated goal area. This goal is represented as a specific coordinate point (x, y) in a two-dimensional environment. ) For the followers, their main goal is to coordinate with the leader and maintain a safe formation at all times.

[0175] 2. Randomization of the goal: Like other elements in the environment, the location of the goal point is also randomly placed at the beginning of each new training round.

[0176] 3. Quantification of success conditions: We define that when the leader drone's straight-line distance to the goal point is less than a pre-set threshold of 40 meters, the task is considered successful.

[0177] 4. Integration with the training process: The achievement of the goal is directly related to the end of the training round. A round will terminate when one of the following conditions occurs: Success: The leader reaches the goal area.

[0178] Failure: The leader collides with any obstacle or boundary.

[0179] Timeout: Exceeding the pre-set maximum decision-making steps without completing the task.

[0180] Goal reward: Use the environment module to monitor the distance between the host and the goal, then give a large reward when the host is less than the set threshold from the goal, indicating successful arrival at the goal; otherwise, give a reward of -0.001 times the distance to the goal, guiding the host to approach the goal.

[0181] Formation reward: If the host and all followers are within a set distance, no reward or punishment is given. Otherwise, a penalty of -0.001 times the distance between the host and the farthest follower is given, punishing the failure of the formation.

[0182] Efficiency reward: When the host and follower distances are less than the threshold, a reward of +1 is given if the speed difference between each follower is less than 0.1, otherwise no reward or punishment is given.

[0183] Boundary reward: If the agent approaches the set boundary, a penalty of -0.1 is given.

[0184] Obstacle reward: When the agent's distance to the obstacle is less than the threshold for serious collision, a large penalty is given, resulting in death and the end of the round. When the agent's distance to the obstacle is less than the threshold for approaching the obstacle, a penalty of -0.2 is given. If other conditions are met, no reward or punishment is given.

[0185] Step S40, the adaptive course information module consists of course tasks and learning progress evaluation, and the course task defines a specific configuration task in the learning process, which can be represented as a six-tuple:

[0186] wherein; : task unique identifier; : task configuration parameter set; : task difficulty value; : performance history record; : success rate statistics; : access times and completion times.

[0187] The learning progress evaluator uses the SAC algorithm to judge the mastery of the agent to the current task based on the performance improvement rate, and judges whether it can enter the next stage; Learning progress indicators:

[0188] wherein represents the reward stability, is the reward standard deviation, is the reward average.

[0189] When > 0.65, it is considered that the performance of the course task is large to enter the next course standard.

[0190] Step S50, the multi-agent soft actor critic (MASAC) algorithm realizes a multi-agent soft actor-critic algorithm for heterogeneous master and slave intelligent agent hosts, and integrates a course learning mechanism. The core idea is: Heterogeneous strategy learning: the master and the slave learn independent attention-based strategies (strategy network) Centralized value evaluation (CTDE): use a shared, centralized value network to evaluate the joint state and the value of all agent actions. This value network internally processes a variable number of slaves through attention mechanisms.

[0191] Maximum entropy regularization: by introducing an entropy term and automatically adjusting its weight (temperature coefficient α), encourage policy exploration and prevent premature convergence to suboptimal policies. The master and the slave have independent α adjustment mechanisms.

[0192] Course learning integration: the algorithm can adapt to changes in the number of agents, task difficulty, etc. in course learning, and support stage-by-stage storage, sampling of experience, and migration of policy parameters.

[0193] Its framework contains three core components: the actor network, the critic network, and the SAC controller. Among them, Attention-based Actor Network: In multi-agent systems, an agent's optimal action often depends not only on its own state but also on the states and actions of other agents. Attention mechanisms provide a powerful way to dynamically learn and aggregate information from other agents. It uses a general component to map raw observations to a low-dimensional embedding , whose mapping formula is as follows:

[0194] 1. Master learns a policy from its own observations and all slave observations . Its information flow and formula are as follows: Master observation encoding: ; Slave observation encoding: For each slave : Attention weighting (master attention to slaves): The master uses its own encoding

[0195] as the query , and all slaves' encodings as the keys and values . The attention mechanism calculates a context vector: : ( is the dimension of the key)

[0196]

[0197] In multi-head attention, this process is executed times (the number of heads) , each head learning different attention weights, and then the results are concatenated or averaged.

[0198] , ,

[0199]

[0200] ​​

[0201] Feature fusion:

[0202] Strategy parameter output:

[0203]

[0204]

[0205]

[0206] Strategy: .

[0207] 2. Slave Actor Network: Slave Based on its own observations Host observation Other slave machines Observations Learn a strategy .

[0208] Information flow and formulas: Self-observation code:

[0209] Context observation coding:

[0210] for

[0211] slave machine It can observe information from all other agents as its context and construct global context information:

[0212] Attention-weighted (Follower) (Consider the global context) Utilizing its own encoding As a query ( Global Context Coding As a key ( ) and value ( ).

[0213]

[0214] Feature fusion:

[0215] Policy parameter output: From generate and .

[0216] Policy:

[0217] Structured Attention Critic Network This is a centralized Critic, evaluating the joint state Q value of taking joint action . Its function is to encode the observation-action pair of the agent into embedding . The mapping formula is as follows:

[0218] Critic network structure and information flow: (state, action) pair encoding: Master:

[0219] Slave :

[0220] Master Q value calculation ( ): The master's encoding as Query. All slaves' encodings

[0221] as Key and Value.

[0222]

[0223]

[0224]

[0225] In the formula , the evaluation is the "contribution" of the master or its related value under the current global state-action, taking into account the behavior of all slaves.

[0226] Slave Q value calculation ( ): Build global (state, action) embedding context:

[0227] For each slave i, its own encoding as Query.​

[0228] (Except The key and value are themselves (or contain themselves but the attention mechanism can learn to ignore themselves).

[0229]

[0230]

[0231]

[0232] in, The evaluation assesses the slave device's actions within the current global state-action context. The "contribution" or its associated value is considered, taking into account the behavior of the host and all other slaves.

[0233] Target Critic Network: Correspondingly, there exists a target Critic network, whose parameters... Through the main Critic network parameters Soft update:

[0234] The target network is used to compute the target Q-value to stabilize the learning of the Bellman equation.

[0235] IV. Entropy Regularization The core idea of ​​Soft Actor-Critic (SAC) is to maximize the policy entropy while maximizing the expected cumulative return. This encourages exploration and avoids premature convergence to suboptimal deterministic policies.

[0236] Strategy learning objectives:

[0237] in Is the strategy in the state? Entropy below. It's a temperature parameter, a trade-off between the importance of reward and entropy.

[0238] Automatically adjust temperature parameters : To avoid manual adjustment This sensitive hyperparameter is addressed by SAC through an automatic adjustment mechanism. The goal is to minimize the policy entropy. Maintain at a preset target entropy Near the horizontal level.

[0239] Learning parameters: (to ensure) ) loss function :

[0240] Independent entropy adjustment: maintain independent entropy network instances for hosts and slaves. This means that for an agent (which can be a host or a slave ), it has its own temperature and target entropy .

[0241] The update of an actor uses its corresponding .

[0242] The update of a critic is based on the entropy of its own policy and

[0243] Through these mechanisms, the model can learn complex strategies that take into account multi-agent interactions, and through automatic adjustment of entropy regularization to ensure sufficient exploration.

[0244] Step S60, implementation of the knowledge transfer module: the specific implementation steps are as follows: 1. Stage completion and migration trigger: when the system completes training in the current course stage, and its performance index reward stability meets the preset switching threshold, the knowledge transfer process is started.

[0245] 2. Parameter inheritance of policy network (Actor): after starting, the system first processes the policy network. For the next stage to start, the system will build a new policy network. The parameter vector of the new network will be initialized by directly copying the parameter vector of the previous stage network . This process can be represented as: ←

[0246] This operation completely migrates the learned behavior strategy and coordination mode with generalization from the previous stage to the new stage. For network structure expansion due to an increase in the number of agents, this parameter copying operation is applied to all structurally homologous layers.

[0247] 3. Parameter reset of the value network (Critic): unlike the policy network, the value network is handled in a complete reset manner. The parameter vector of the new value network built for the next stage will be resampled from a predefined random distribution D. This process can be represented as: ~ D The purpose of this operation is to decouple the value evaluation of the new stage from the old stage task-specific value system, enabling the value network to quickly and accurately learn and adapt to the reward structure and state space of the new stage task without being disturbed by historical biases.

[0248] When the system completes training at the current course stage s and meets the preset switching threshold C_s, the knowledge transfer process is started.

[0249] 4. Synchronous transfer of entropy adjustment parameters: the learnable temperature parameter is regarded as part of the policy knowledge and is synchronously transferred. The value obtained after training at stage s is denoted as , which represents the optimal exploration-exploitation balance matching the policy . When entering the new stage s+1, this parameter will be directly inherited. This process can be represented as:

[0250] 5. New stage training starts: after completing the above asymmetric parameter transfer, the system forms a hybrid initialized new model, which includes an Actor network that inherits the past policy and a Critic network that is ready to learn new values. With this model as the starting point, the system formally begins a new training iteration in the environment of stage s+1.

[0251] Step S70, based on the decision model of S50, can make decisions for leaders and followers respectively: Leader decision:

[0252] Follower decision: ; Formation maintenance index: according to the proximity of the energy consumption of each UAV and the proximity of the distance, the formation constraint is carried out: Formation error calculation:

[0253] Speed consistency:

[0254] Step S80, the embodiment provides a multi-UAV cooperative control decision system based on gradual course enhancement, which includes the following hardware and software components: Computing platform: workstation (or equivalent computing device) equipped with NVIDIA RTX 4060 graphics card (8GB RAM); Simulation environment: custom multi-UAV cooperative simulation environment based on pygame;​​ UAV configuration: 1 leader UAV and 3 follower UAVs; The above used algorithms are trained in the trainer while retaining the best performing data for subsequent use. The UAV swarm maintains formation, reaches the target and autonomously avoids obstacles through the attention-based and curriculum-driven multi-agent SAC system.

[0255] The leader UAV-follower UAV path planning model effectively improves the training efficiency, adaptability and stability of the multi-UAV cooperative control system, and is particularly suitable for multi-UAV path planning and formation maintenance tasks in complex environments.

Claims

1. A path planning method for heterogeneous unmanned aerial vehicle (UAV) swarm, characterized in that, Comprise: S1: constructing a heterogeneous unmanned aerial vehicle motion model, and constructing a leader unmanned aerial vehicle-follower unmanned aerial vehicle path planning model through a Markov decision process according to the heterogeneous unmanned aerial vehicle motion model; S2: generating multiple training environments; S3: selecting a training environment, training the leader unmanned aerial vehicle-follower unmanned aerial vehicle path planning model, and obtaining a trained leader unmanned aerial vehicle-follower unmanned aerial vehicle path planning model; S4: determining whether there are remaining unselected training environments, if yes, constructing a new leader unmanned aerial vehicle-follower unmanned aerial vehicle path planning model through a knowledge transfer technology according to the trained leader unmanned aerial vehicle-follower unmanned aerial vehicle path planning model and returning to execute step S3, if no, taking the trained leader unmanned aerial vehicle-follower unmanned aerial vehicle path planning model as a final leader unmanned aerial vehicle-follower unmanned aerial vehicle path planning model, and executing step S5; S5: realizing path planning through the final leader unmanned aerial vehicle-follower unmanned aerial vehicle path planning model. 2.The path planning method for heterogeneous UAV swarm of claim 1, wherein, In step S1, constructing a heterogeneous unmanned aerial vehicle motion model comprises: Mathematically modeling one leader unmanned aerial vehicle and multiple follower unmanned aerial vehicles from time t to time t+1; using kinematic equations to define movement rules for all unmanned aerial vehicles: The state of each drone is described by four key variables: and denotes the two-dimensional coordinates of the drone at time point t; denotes the linear velocity of the drone at time point t; denotes the heading angle of the drone at time point t; Control inputs are defined as: denotes the linear acceleration, denotes the angular velocity; State update: new coordinates = old coordinates + component of velocity in that direction x time step; new velocity = old velocity + acceleration x time step; new orientation = old orientation + angular velocity x time step; Using a constraint condition framework, constraints are respectively set for one leader unmanned aerial vehicle and multiple followers. 3.The path planning method for heterogeneous UAV swarm of claim 2, wherein, Using a constraint condition framework, constraints are respectively set for one leader unmanned aerial vehicle and multiple followers comprises: Modeling the leader drone as the main navigation unit of the swarm, with constraints: speed constraint: speed is in the range of 100 m / s to 200 m / s; maneuverability constraint: maximum linear acceleration does not exceed 3 m / s2, maximum angular velocity does not exceed 0.6 rad / s; environmental constraint: ensuring that its flight path is always within the map boundaries and maintains a distance of no less than a safety radius from all obstacles ​ The follower drone is modeled as an agile execution unit whose core task is to closely follow the leader and maintain a pre-set formation. The follower drone constraints are: speed in the range of 100m / s to 300m / s; maneuverability constraints: maximum linear acceleration does not exceed 6m / s2, maximum angular velocity does not exceed 1.2 rad / s; environmental constraints: ensuring that its flight path is always within the map boundaries and maintains a distance of no less than a safety radius from all obstacles. The follower drone is modeled as an agile execution unit whose core task is to closely follow the leader and maintain a pre-set formation. The follower drone constraints are: speed in the range of 100m / s to 300m / s; maneuverability constraints: maximum linear acceleration does not exceed 6m / s2, maximum angular velocity does not exceed 1.2 rad / s; environmental constraints: ensuring that its flight path is always within the map boundaries and maintains a distance of no less than a safety radius from all obstacles.

4. The path planning method for heterogeneous UAV swarm according to claim 3, wherein, In step S1, the Markov decision process comprises: The state space is defined as a seven-dimensional vector: The following information is obtained in real time from the actual position of the simulation environment: : normalized current drone position coordinates; : normalized current speed; : normalized current heading angle; : normalized target position coordinates; : obstacle warning sign; The action space is defined as a two-dimensional continuous vector: wherein the steering control: controls the change in heading angle; acceleration control: , control speed increase and decrease; State transition function: Setting a reward function in a multi-objective combination form: : weight for each reward function; Setting reward item calculation Target reward: Obstacle penalty: Platoon reward: Efficiency reward: Exploration reward: .

5. The path planning method for heterogeneous UAV swarm according to claim 1, wherein, In step S1, the leader unmanned aerial vehicle-follower unmanned aerial vehicle path planning model comprises: A policy network, a value network, and a SAC controller; where the policy network is an attention-based Actor network: dynamically learns and aggregates information from other agents through attention mechanism; uses a general component to map raw observations to a low-dimensional embedding with the mapping formula as follows: Leader drone According to its own observations And all follower drones Observations Learn a strategy ; Information flow and formula thereof: Leader drone observation code: Follower drone observation encoding: for each follower drone : Leader drone utilizes its own encoding as a query , all follower drones' encodings all follower drones' encodings as keys and values ; attention mechanism computes context vector: : In multi-headed attention, this process is performed in parallel Each head learns different attention weights, and the results are then concatenated or averaged: , , Feature fusion: Policy parameter output: Strategies: ; Follower drone actor network: follower drones According to its own observations , leader drone observations and observations of other follower drones , learn strategy :​ Self-observation encoding: Context observation encoding: for Follower drone Information from all other drones can be observed as their context and build global context information: Attention weighting with its own encoding as a query ( ), global context encoding as a key ( ) and value ( ); Feature fusion: Policy parameter output: from Generate And ; Strategy: Value network for structured attention Critic network: evaluates joint state Take joint action Q value; its function is to encode the observation-action of the UAV into embedding ; the mapping formula is as follows: Critic network structure and information flow: Leader drone ; Follower drone : ; Leader drone Q-value computation ); Leader drone Encoding of as Query; Encoding of all follower drones as Key and Value; In the formula is the value of the leader drone under the current global state-action, and takes into account the behavior of all follower drones. Follower drone Q-value computation ): Building a global embedding context: For each follower drone i, which encodes As Query; as Key and Value; wherein, is the value of the follower drone under the current global state-action, taking into account the behavior of the leader drone and all other follower drones; Follower drone Critic network: Each follower drone corresponds to a follower drone Critic network, whose parameters Through the leader drone Critic network parameters Soft update: A target network is used to calculate a target Q value to stabilize the learning of the Bellman equation; IV. Entropy regularization SAC controller: maximizing the expected cumulative return while also maximizing the entropy of the policy; Policy learning goal: wherein is the entropy of the policy in state ; is a temperature parameter that trades off the importance of rewards and entropy. Automatically adjusting temperature parameters : Learning parameters: Loss function : 。 6. The path planning method for heterogeneous UAV swarm according to claim 1, wherein, In step S2, generating multiple training environments specifically comprises: Definition of physical space: a two-dimensional plane simulation environment is constructed, with a size of 7000*6000 meters; the space contains boundaries, obstacles, and target points; Programmed generation for generalization ability: to ensure that the trained model has strong generalization ability, the production of each environment includes the following points: randomization of initial states of unmanned aerial vehicles: the initial positions, velocities, and orientation angles of all unmanned aerial vehicles are randomly set within a predetermined valid range; Randomization of obstacle layout: the number, position, and size of obstacles in the environment are randomly generated; Course learning driven increasing complexity: each generated environment automatically increases in difficulty, introducing more followers and more complex obstacle layouts; Target setting rules: Definition and representation of the goal: the main goal is for the leader drone to reach a specified goal region; this goal is represented in a two-dimensional environment as a specific coordinate point (x, y) ); Randomization of the target: the position of the target point will be randomly placed in each environment generation; Quantification of success conditions: the task is considered successful when the leader drone is within a preset threshold distance from the target point; Combination with the training process: the achievement of the target is directly related to the end of the training round; a round will be terminated when one of the following conditions occurs: Success: the leader reaches the target area; Failure: the leader collides with any obstacle or boundary; Timeout: the maximum number of decision steps is exceeded without completing the task; Target reward: monitor the distance between the leader drone and the target; if the leader drone is within a set threshold distance from the target, a large reward is given, indicating successful arrival at the target; otherwise, a reward of -0.001 times the distance from the target is given, guiding the leader drone to approach the target; Formation reward: if the leader drone and all follower drones are within a set distance, no reward or punishment is given; otherwise, a punishment of -0.001 times the distance between the leader drone and the farthest follower drone is given, punishing the failure of the formation; Efficiency reward: when the leader drone and all follower drones are within a threshold distance, each follower drone is given a reward of +1 if the speed difference is less than 0.1, otherwise no reward or punishment is given; Boundary reward: if the leader drone and all follower drones approach the set boundary, a penalty of -0.1 is given; Obstacle reward: when the leader drone and all follower drones are within a threshold distance from the obstacle, a large penalty is given, resulting in death and the end of the round; when the agent is within a threshold distance from the obstacle, a penalty of -0.2 is given; otherwise, no reward or punishment is given.

7. The path planning method for heterogeneous UAV swarm according to claim 1, wherein, In step S3, the following steps are included: during the training process, the SAC algorithm is used to judge the degree of mastery of the leader drone-follower drone path planning model for the current environment based on the performance improvement rate, and it is judged whether it can enter the next stage. 8.The path planning method for heterogeneous UAV swarm of claim 1, wherein, In step S4, the trained leader drone-follower drone path planning model is updated to a new leader drone-follower drone path planning model through knowledge transfer technology, specifically: S4.1 stage completion and migration trigger: when the training is completed, and its performance index reward stability When the preset switching threshold is met, the knowledge migration process is started; S4.2 Parameter inheritance of policy network: After starting, a new policy network is constructed; the parameter vector of the new network will be initialized by directly copying the parameter vector of the network in the previous stage This process is represented as: ← S4.3 Parameter reset of the value network: a new value network is constructed, and the parameter vector will be resampled from a predefined random distribution D; this process is represented as: ~ D S4.4 Synchronous migration of entropy regulation parameters: learnable temperature parameters are considered part of the policy knowledge and are migrated synchronously; After the training of phase s is completed, we obtain the value, denoted by , which represents the optimal exploration-exploitation balance that matches the policy ; this parameter will be directly inherited when entering the new phase s+1; this process is represented as: ← S4.5 Start of new stage training: after the above asymmetric parameter transfer is completed, a new leader drone-follower drone path planning model is obtained.

9. The path planning method for heterogeneous UAV swarm according to claim 1, wherein, In step S5, the following steps are included: Leader drone decision: ; Follower drone decision: ; Formation maintenance indicators, which constrain the formation based on the similarity of the energy consumption and the distance of each drone: Formation error calculation: Speed consistency: 。 10.A path planning system for heterogeneous unmanned aerial vehicle (UAV) swarm, characterized in that, The model construction module constructs a heterogeneous drone motion model and a leader drone-follower drone path planning model based on the Markov decision process according to the heterogeneous drone motion model; The environment generation module generates multiple training environments; The training module selects a training environment and trains the leader drone-follower drone path planning model to obtain a trained leader drone-follower drone path planning model; ​ The judgment module judges whether there is still a remaining unselected training environment, if yes, a new leader-unmanned aerial vehicle-follower unmanned aerial vehicle path planning model is constructed according to the leader-unmanned aerial vehicle-follower unmanned aerial vehicle path planning model after training through a knowledge transfer technology and the training module is returned to be executed, if no, the leader-unmanned aerial vehicle-follower unmanned aerial vehicle path planning model after training is taken as a final leader-unmanned aerial vehicle-follower unmanned aerial vehicle path planning model, and the path planning module is executed. The path planning module realizes path planning through the final leader-unmanned aerial vehicle-follower unmanned aerial vehicle path planning model.

Images