An unmanned swarm motion path planning method and system based on attention allocation

Abstract

The application discloses an unmanned swarm motion path planning method and system based on attention distribution. A relationship enhancement type Transformer (RETransformer) is designed to enhance the learning of the interaction and cooperation relationship of the unmanned swarm in the motion process, and is combined with a multi-agent deep deterministic policy gradient (MADDPG) to obtain a relationship enhancement type DDPG (REDDPG). The method is high in efficiency, good in flexibility and simple in realization. The application enhances the attention of each unmanned aerial vehicle in the unmanned swarm to the nearby objects, enables safe navigation of the unmanned aerial vehicles to avoid collision and maintain the inter-aircraft communication distance, and reasonably plans the route in a collective cooperation mode, and the application has robustness and scalability to different complex scenes.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of unmanned systems, and particularly relates to an unmanned swarm motion path planning method and system based on attention distribution. BACKGROUND

[0002] In recent years, unmanned swarms have been widely studied and applied in search and rescue and environmental monitoring tasks, etc., due to their key advantages of task scalability, failure-robustness and distributed intelligence. In these tasks, unmanned aerial vehicles need to coordinate and cooperate to reach the task area from the deployment site while avoiding collisions or minimizing energy consumption. To achieve such autonomous collective swarming, each swarm member needs to make control decisions based on its perception state and interaction with neighbors, while centralized control would bring heavy communication burden and cannot adapt to complex environments. Therefore, one of the main challenges of unmanned swarms is to achieve robust and scalable motion control decisions to enable a large number of unmanned aerial vehicles to fly cooperatively while maintaining obstacle avoidance.

[0003] Current related works to achieve unmanned swarm motion can be divided into two categories: rule-based methods and reinforcement learning-based methods. In rule-based methods, agents find basic motion principles based on perceived obstacle information and neighbor states, and usually the basic rules need to be quantified by multiple parameters, and the corresponding values are different for different environments or tasks. Therefore, the main disadvantage of this method is that the control parameters set in advance cannot be changed during task execution, and it cannot handle emergency situations that are not considered in the optimization process, i.e., it is not robust to changes in environmental settings or its own state (such as swarm speed or communication range). Reinforcement learning-based methods can learn near-optimal behavior without human intervention and perform better in unpredictable environments. However, most existing reinforcement learning-based methods lack the ability to learn the interaction and cooperation relationship between swarm members, and cannot effectively utilize the perceived environmental information. In addition, most models require a long training time and even cannot converge. SUMMARY

[0004] The present application proposes an unmanned swarm motion path planning method and system based on attention distribution to solve the above problems. The method enhances the learning of interactive cooperation relationship of unmanned swarms during motion by designing a relationship-enhanced Transformer (RETransformer), and combines it with multi-agent deep deterministic policy gradient (MADDPG) to obtain a relationship-enhanced DDPG (REDDPG). The method is efficient, flexible and easy to implement.

[0005] To achieve the above purposes, the present application adopts the following technical solutions:

[0006] The application discloses an unmanned swarm motion path planning method based on attention allocation, comprising the following steps:

[0007] (1) acquiring observable information of each unmanned aerial vehicle in the unmanned swarm;

[0008] (2) constructing a dynamic optimization problem based on swarm motion characteristics, and designing a corresponding reward function;

[0009] (3) building a relationship-enhanced unmanned swarm system framework by using an attention allocation mechanism;

[0010] (4) training an unmanned swarm system model, converging to obtain an optimal model, and completing path planning of the unmanned swarm motion.

[0011] Specifically, the observable information of each unmanned aerial vehicle in the unmanned swarm in the step (1) is defined as follows:

[0012]

[0013] Wherein s i is an environmental state vector observed by the i-th unmanned aerial vehicle, including the relative distance with a target, an obstacle and a neighbor unmanned aerial vehicle and the speed of the unmanned aerial vehicle itself; p i is a position vector of the i-th unmanned aerial vehicle, v i is a speed vector of the i-th unmanned aerial vehicle, N i is a neighbor set of the i-th unmanned aerial vehicle, G is a position vector of a target site, and O is a position vector set of an obstacle.

[0014] Further, the step (2) of constructing a dynamic optimization problem based on swarm motion characteristics is specifically as follows:

[0015] (2.1) reaching a target area:

[0016] ||p i -G||≤ρ g ;

[0017] Wherein ρ g represents a tolerable deviation distance of target reaching;

[0018] (2.2) avoiding collision:

[0019]

[0020] Wherein d u and d o are a safety distance between unmanned aerial vehicles and a safety distance between an unmanned aerial vehicle and an obstacle respectively, is the radius of the m-th obstacle;

[0021] (2.3) maintaining connectivity:

[0022] Ψ = Ψ v + Ψ d ,

[0023]

[0024] wherein ψ v and ψ d represent velocity consistency and position consistency respectively, v avg is the average speed vector of the unmanned swarm, and N is the total number of unmanned aerial vehicles in the unmanned swarm.

[0025] Further, the step (2) designs a corresponding reward function, specifically: a model REDDPG based on attention distribution is constructed by combining multi-agent deep deterministic policy gradient MADDPG and RETransformer; the step is realized through the following sub-steps:

[0026] (3.1) build a multi-agent deep deterministic policy gradient MADDPG training framework:

[0027] In the multi-agent environment, MADDPG is suitable for centralized learning and decentralized execution, enabling agents to learn cooperation and competition; the actor-critic architecture in MADDPG is composed of two parts: policy network actor and value network critic, both of which are composed of current network and target network; in the unmanned aerial vehicle swarm scenario, for each unmanned aerial vehicle i, the current actor network is parameterized by θ i , and the current critic network is parameterized by φ i , that is, φ i is the parameter of the critic network, and the target networks of the two are parameterized by θ i ′ and φ i ′ respectively; the actor network determines the action, while the critic network evaluates the action by estimating the value of the state;

[0028] (3.2) design a relationship-enhanced RETransformer model:

[0029] The RETransformer based on attention allocation design integrates the distance-based adjacency matrix and attention mask for cooperative navigation and obstacle avoidance; the RETransformer first incorporates the adjacency matrix representing the distance between UAVs into the query key QK matrix, so that the model can capture and utilize the spatial relationship between UAVs and provide distance information; then the attention mask is used to ensure that each UAV only focuses on the k closest UAVs; this mechanism ensures that each UAV is only affected by its nearest neighbors during decision-making, maintaining the safety distance and connectivity of the UAV swarm, and its expression is as follows:

[0030]

[0031] where Q, K and V are linear transformations of the input sequence, d k is the dimension of the key; A is the adjacency matrix, whose element A ij = -‖p i -p j ‖ represents the distance between UAV i and j; M is the attention mask, which masks the attention scores of distant UAVs by setting them to a large negative value (-∞);

[0032] (3.3) Constructing a relationship-enhanced DDPG algorithm based on attention allocation, REDDPG:

[0033] The REDDPG algorithm is obtained by combining the two structures of steps (3.1) and (3.2), first encoding the environmental information available to the UAVs, converting the state vector of each UAV into a high-dimensional embedding vector suitable for the model dimension, which is realized through a linear layer; then introducing the RETransformer into the actor network, combining the relationship matrix with the attention mechanism; the critic network uses the ordinary Transformer for policy evaluation to maintain training stability and policy quality.

[0034] Further, the UAV swarm system framework in step (3) is obtained by combining RETransforme and MADDPG; the strategy network actor applies RETransformer to capture the complex interaction between UAVs and output reasonable actions for each UAV; the value network critic uses the standard Transformer for policy evaluation to ensure the stability of the training.

[0035] Further, the step (4) includes model initialization, training data preparation, loss optimization, model verification and evaluation, model optimization and convergence, specifically:

[0036] (4.1) Model initialization: first initialize all networks, while building experience replay pool B for storing sample set (s, a, r, s ′ ); at each step, the UAV selects an action according to the current policy, interacts with the environment, and generates a new state transition sample, including the current state, action, reward and next state;

[0037] (4.2) Training data preparation: after model initialization, store in experience replay pool, use the reward function to calculate the immediate reward of each time step;

[0038] (4.3) Loss optimization: constantly update the parameters of the critic network by minimizing the loss function of the critic network, that is, the mean square error loss; at the same time, maximize the target value output by the critic network through the gradient ascent method, and optimize the parameters of the actor network;

[0039] (4.4) Model verification and evaluation: after each training cycle, use evaluation indicators to adjust the hyperparameters of the model, including learning rate, batch size;

[0040] (4.5) Model optimization and convergence: monitor the convergence of the model as the training progresses, ensure that the model reaches an optimal state after a certain number of iterations, and complete the path planning of the UAV swarm motion.

[0041] The application also discloses an unmanned bee swarm motion path planning system based on attention allocation, comprising the following modules:

[0042] Information acquisition module: acquire observable information of each UAV in the unmanned bee swarm;

[0043] Construction design module: construct a dynamic optimization problem based on the motion characteristics of the bee swarm, and design a corresponding reward function;

[0044] Frame building module: build a relationship-enhanced unmanned bee swarm system framework using an attention allocation mechanism;

[0045] Training model module: train the unmanned bee swarm system model, and converge to obtain an optimal model to complete the path planning of the unmanned bee swarm motion.

[0046] The application object can not only be for unmanned bee swarms, but also for any unmanned system with group behavior.

[0047] Compared with the prior art, the application has the beneficial effects that:

[0048] This paper proposes a method for unmanned swarm motion path planning based on attention allocation. By learning the interactions between drones and their environment, the model enhances each drone's attention to nearby objects, enabling safe navigation while avoiding collisions and maintaining inter-drone communication distances. This allows the swarm to plan reasonable routes through collective collaboration. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0050] Figure 1 This is a model architecture diagram of the unmanned bee swarm motion path planning method based on attention allocation;

[0051] Figure 2 This is a model training flowchart for the unmanned bee swarm motion path planning method based on attention allocation. DETAILED DESCRIPTION

[0052] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different drawings represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with this application.

[0053] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0054] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments:

[0055] The first aspect of the present invention provides a method for planning the motion path of an unmanned bee swarm based on attention allocation, comprising the following steps:

[0056] Step 1: Each drone can detect environmental information through its radar, thereby obtaining the location information of surrounding objects. This information can also be transmitted through inter-machine communication. Therefore, the observable information that each drone in the swarm can obtain is defined as follows:

[0057]

[0058] where s iis the environment state vector observed by the ith UAV, including the relative distance to the target, obstacles, and neighbor UAVs, and its own velocity. i is the position vector of the ith UAV, v i is the velocity vector of the ith UAV, is the neighbor set of the ith UAV, G is the position vector of the target location, and O is the set of position vectors of obstacles.

[0059] Step 2: The motion of the UAV swarm needs to consider three key performance indicators: completing the target, avoiding collision, and maintaining connectivity, which can be constructed as the following dynamic optimization problem:

[0060] (2.1) Reach the target area.

[0061] ||p i -G||≤ρ g

[0062] where ρ g represents the tolerable deviation distance of target arrival.

[0063] (2.2) Avoid collision.

[0064]

[0065] where d u and d o are the safety distance between UAVs and the safety distance between UAVs and obstacles, is the radius of the mth obstacle.

[0066] (2.3) Maintain connectivity.

[0067] ψ=ψ v +ψ d ,

[0068]

[0069] where ψ v and ψ d represent velocity consistency and position consistency, respectively, v avg is the average velocity vector of the UAV swarm, and N is the total number of UAVs in the UAV swarm.

[0070] According to the above established dynamic optimization problem, the corresponding reward function can be designed:

[0071] (2.4) Reward for reaching the target. During flight, the closer the UAV is to the target, the more reward it will receive. Once it reaches the target area with a radius of ρ g , the UAV will receive a reward r goal .

[0072]

[0073] (2.5) Collision avoidance reward. If the distance between the UAV and the nearest obstacle is less than the safety distance, it will be penalized.

[0074]

[0075] (2.6) Connectivity maintenance reward. Cooperation in the UAV swarm requires that the UAVs maintain an appropriate communication distance between them. We use the DBSCAN algorithm to calculate the outlying degree of the UAVs, which reflects the compactness and looseness of the UAV swarm. The smaller the outlying degree, the better the effect.

[0076]

[0077] where N outlier is the number of outlying UAVs.

[0078] The distance between the UAVs remains constant as the most ideal state, when the distance between any two adjacent UAVs is too far (unable to maintain communication) or too close (less than the safety distance d u , they should be punished accordingly.

[0079]

[0080] According to the above reward function, the UAV swarm can achieve the task of reaching the target area while avoiding obstacles, maintaining consistency and communication connectivity during flight, and minimizing outlying degree through learning. The reward function for UAV i is defined as:

[0081] R i = R goal,i + R collision,i + R outlier,i + R neighbor,i .

[0082] Step 3: Combine multi-agent deep deterministic policy gradient (MADDPG) and relationship-enhanced Transformer (RETransformer) to build a model based on attention allocation, REDDPG. This step is achieved through the following sub-steps:

[0083] (3.1) Build a multi-agent deep deterministic policy gradient (MADDPG) training framework.

[0084] In multi-agent environments, MADDPG is suitable for centralized learning and decentralized execution, which enables agents to learn to cooperate and compete. The actor-critic architecture in MADDPG consists of two parts, namely the policy network actor and the value network critic, both of which are composed of a current network and a target network. In the UAV swarm scenario, for each UAV i, the current actor network is parameterized by θ i , and the current critic network is parameterized by φ i , i.e., φ i are the parameters of the critic network. Similarly, their target networks are parameterized by θ i ′ and φ i ′ , respectively. The value network critic network evaluates actions by estimating Q values, providing feedback to the policy network. The policy network actor decides actions, constantly adjusting the policy using feedback to enable the selection of better actions in future situations. This learning process enables each UAV to gradually learn to perform tasks in complex multi-agent environments and achieve the goal of swarm motion.

[0085] (3.2) Design the relationship-enhanced Transformer (RETransformer) model.

[0086] The RETransformer designed based on attention allocation significantly enhances the understanding of internal relationships in the UAV swarm by integrating distance-based adjacency matrices and attention masks, achieving more efficient and collaborative cooperative navigation and obstacle avoidance. RETransformer first incorporates the adjacency matrix representing the distance between UAVs into the query key (QK) matrix, helping the model better capture and utilize spatial relationships between UAVs and providing more explicit distance information. Then, attention masks are used to ensure that each UAV only focuses on the k closest UAVs, reducing the computational complexity of the model and reducing irrelevant information interference. This mechanism ensures that each UAV is mainly influenced by its nearest neighbors during decision-making, thereby improving the accuracy and real-time performance of decision-making and maintaining the safety distance and connectivity of the UAV swarm.

[0087]

[0088] where Q, K, and V are linear transformations of the input sequence, d k is the dimension of the key. A is the adjacency matrix, whose elements A ij = -‖p i -p j|| represents the distance between drones i and j. M is the attention mask, which masks the attention scores of distant drones by setting them to a large negative value (-∞).

[0089] (3.3) Constructing the relationship-enhanced DDPG algorithm based on attention allocation (REDDPG).

[0090] The REDDPG algorithm combines the advantages of multi-agent deep deterministic policy gradient (MADDPG) and relationship-enhanced Transformer (RETransformer), aiming to achieve efficient and coordinated control of the drone swarm. First, the environmental information available to the drones is feature-encoded, converting each drone's state vector into a high-dimensional embedding vector suitable for model dimensions. This process is achieved through a linear transformation layer, which maps the original state information to a more abstract representation space for subsequent model processing.

[0091] In the actor network, REDDPG introduces the relationship-enhanced Transformer (RETransformer), which combines relationship matrices and attention mechanisms to enhance the modeling ability of complex relationships between drones. The relationship matrix reflects the relative position and distance information between drones, while the attention mechanism dynamically adjusts the information exchange weights between different drones, enabling each drone to make more intelligent decisions based on its relationship with other drones.

[0092] In the critic network part, REDDPG adopts a normal Transformer architecture for policy evaluation, which simplifies the complexity of the model while maintaining good training stability and policy quality, avoiding overfitting problems that may be caused by introducing too much relationship information in the critic network.

[0093] Step 4: Train the drone swarm system model, which is achieved through the following sub-steps:

[0094] (4.1) Model initialization: Initialize the parameters of all neural networks, including the actor network, critic network, and RETransformer model. To enhance the stability of the model, a uniform distribution random initialization method is used, and appropriate learning rates are set for the parameters of each network.

[0095] (4.2) Training data preparation: Flight data of the unmanned swarm is generated from the simulation environment, including state information of each UAV, adjacency matrix, target position, and obstacle position, etc. The experience replay pool B is constructed to store the state transition sample set for model training. At each step, the UAV selects an action according to the current strategy, interacts with the environment, and generates a new state transition sample (including the current state, action, reward, and next state, i.e., (s, a, r, s ′ )) stored in the experience replay pool.

[0096] (4.3) Loss optimization: The immediate reward at each time step is calculated using the reward function defined above. The parameters of the critic network are updated by minimizing the loss function (i.e., mean square error loss) of the critic network. At the same time, the parameters of the actor network are optimized by maximizing the target value output by the critic network through the gradient ascent method. Specifically, the current actor network generates the action of UAV i at time step t Then a small batch of samples is randomly extracted from the experience replay pool B, and the current critic network is trained using the gradient descent method and its parameters are updated. This process is achieved by minimizing the following loss function at time step t:

[0097]

[0098]

[0099] where is the target Q value, calculated by the target critic network at time step (t+1).

[0100] The current actor network updates its parameters by maximizing the Q value output by the current critic network to output better actions, through the following gradient ascent process:

[0101]

[0102] To maintain the stability of training, a soft update strategy is adopted to update the parameters of the target network, i.e., periodically copying the parameters of the target network from the current network:

[0103]

[0104] where τ << 1. In addition, an L2 regularization strategy is adopted to prevent overfitting.

[0105] (4.4) Model Validation and Evaluation: After each training cycle, the model is validated on unseen datasets to evaluate its performance in different scenarios, including target arrival rate, obstacle avoidance success rate, and connectivity maintenance. Using these evaluation metrics, model hyperparameters such as learning rate and batch size are adjusted to gradually improve model performance.

[0106] (4.5) Model Optimization and Convergence: During training, a soft update strategy is used to update the parameters of the target network, so that the target network more smoothly approaches the current network. As training progresses, the model's convergence is monitored to ensure that the model reaches a stable optimal state after a certain number of iterations.

[0107] The second aspect of the present invention is an attention allocation-based unmanned bee swarm motion path planning system, comprising the following modules:

[0108] Information acquisition module: obtains observable information of each drone in the swarm;

[0109] Build a design module: Construct a dynamic optimization problem based on the swarm's motion characteristics and design a corresponding reward function;

[0110] Build a framework module: Use the attention allocation mechanism to build a relationship-enhanced unmanned swarm system framework;

[0111] Training model module: trains the unmanned swarm system model, converges to obtain the optimal model, and completes the path planning of the unmanned swarm movement.

[0112] The unmanned swarm motion path planning method based on attention allocation proposed in the present invention enhances the attention of each drone to nearby objects by learning the interaction between the drone and the environment, thereby achieving safe navigation that can avoid collisions while maintaining the communication distance between machines, and enables the unmanned swarm to perform reasonable route planning in a collective and collaborative manner. After considering the specification and practicing the contents disclosed herein, those skilled in the art will easily think of other embodiments of the present application. This application is intended to cover any variations, uses or adaptive changes of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the technical field that are not disclosed in this application.

[0113] It will be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A method for unmanned bee swarm motion path planning based on attention allocation, characterized in that: include: (1) Obtain observable information of each drone in the swarm; (2) Construct a dynamic optimization problem based on the swarm motion characteristics and design a corresponding reward function. Specifically, the design of the corresponding reward function is to construct an attention allocation model REDDPG by combining the multi-agent deep deterministic policy gradient MADDPG and RETransformer. This step is achieved through the following sub-steps: (3.1) Building a multi-agent deep deterministic policy gradient MADDPG training framework: In a multi-agent environment, MADDPG is suitable for centralized learning and decentralized execution, enabling agents to learn to collaborate and compete. The actor-critic architecture in MADDPG consists of two parts: the policy network actor and the value network critic, both of which are composed of the current network and the target network. In the drone swarm scenario, for each drone i, the current actor network By θ i Parameterization, current critic network By φ i Parameterization, i.e. φ i are the parameters of the critic network, and the target networks of the two are respectively represented by θ i ′ and φ i ' parameterization; the actor network decides the action, while the critic network evaluates the action by estimating the value of the state; (3.2) Design of relation-enhanced RETransformer model: The RETransformer, designed based on attention allocation, integrates a distance-based adjacency matrix and an attention mask for collaborative navigation and obstacle avoidance. The RETransformer first incorporates the adjacency matrix representing the distance between drones into the query key QK matrix, enabling the model to capture and utilize the spatial relationship between drones and provide distance information. The attention mask is then used to ensure that each drone only pays attention to the k drones closest to it. This mechanism ensures that each drone is only influenced by its nearest neighbors in the decision-making process, maintaining a safe distance and connectivity within the drone swarm. Its expression is as follows: Where Q, K and V are linear transformations of the input sequence, d k is the dimension of the key; A is the adjacency matrix, whose elements A ij =-‖p i -p j ‖ represents the distance between drones i and j; M is the attention mask, which is masked by setting the attention score of distant drones to a negative value of -∞; (3.3) Constructing the relation-enhanced DDPG algorithm REDDPG based on attention allocation: The REDDPG algorithm is obtained by combining the two structures of steps (3.1) and (3.2). First, the environmental information accessible to the drone is feature encoded, and the state vector of each drone is converted into a high-dimensional embedding vector suitable for the model dimension. This process is implemented through a linear layer. Then, RETransformer is introduced into the actor network to combine the relationship matrix with the attention mechanism. The critic network uses a standard Transformer for policy evaluation to maintain training stability and policy quality. (3) Using the attention allocation mechanism to build a relationship-enhanced unmanned swarm system framework; (4) Train the unmanned swarm system model, converge to the optimal model, and complete the path planning of the unmanned swarm movement.

2. The unmanned bee swarm motion path planning method based on attention allocation according to claim 1 is characterized in that: The observable information of each drone in the unmanned bee swarm in step (1) is defined as follows: where s i is the environmental state vector observed by the i-th UAV, including the relative distance to the target, obstacles, neighboring UAVs and its own speed; p i is the position vector of the i-th UAV, v i is the velocity vector of the i-th UAV, N i is the neighbor set of the i-th UAV, G is the position vector of the target location, and O is the position vector set of obstacles.

3. The unmanned bee swarm motion path planning method based on attention allocation according to claim 1 is characterized in that: In step (2), a dynamic optimization problem is constructed based on the swarm movement characteristics, specifically: (2.1) Arriving at the target area: ||p i -G||≤ρ g ; where ρ g Indicates the tolerable deviation distance of the target; (2.2) Avoid collision: where d u and d o They are the safe distance between drones and the safe distance between drones and obstacles. is the radius of the mth obstacle; (2.3) Maintaining connectivity: ψ=ψ v +ψ d , where ψ v and ψ d Represent velocity consistency and position consistency, v avg is the average velocity vector of the drone swarm, and N is the total number of drones in the drone swarm.

4. The method for unmanned bee swarm motion path planning based on attention allocation according to claim 1, characterized in that: The unmanned bee swarm system framework in step (3) is obtained by combining RETransforme and MADDPG; The policy network actor uses RETransformer to capture the complex interactions between drones and output reasonable actions for each drone; the value network critic uses standard Transformer for policy evaluation to ensure training stability.

5. The unmanned bee swarm motion path planning method based on attention allocation according to claim 1 is characterized in that: The step (4) includes model initialization, training data preparation, loss optimization, model verification and evaluation, model optimization and convergence, specifically: (4.1) Model initialization: First, all networks are initialized and an experience replay pool B is constructed to store the sample set (s, a, r, s′). At each step, the drone selects an action based on the current strategy, interacts with the environment, and generates a new state transition sample, including the current state, action, reward, and next state. (4.2) Training data preparation: After the model is initialized, it is stored in the experience replay pool and the reward function is used to calculate the immediate reward for each time step; (4.3) Loss Optimization: By minimizing the loss function of the critic network, i.e., the mean squared error loss, the parameters of the critic network are continuously updated. At the same time, the target value output by the critic network is maximized through the gradient ascent method to optimize the parameters of the actor network. (4.4) Model Validation and Evaluation: After each training cycle, use evaluation metrics to adjust the model's hyperparameters, including learning rate and batch size; (4.5) Model optimization and convergence: As training progresses, monitor the convergence of the model to ensure that the model reaches the optimal state after a certain number of iterations and completes the path planning of the unmanned swarm movement.

6. An unmanned bee swarm motion path planning system based on attention allocation, characterized in that: Includes the following modules: Information acquisition module: obtains observable information of each drone in the swarm; Constructing a design module: Based on the characteristics of swarm motion, we construct a dynamic optimization problem and design a corresponding reward function. Specifically, we construct an attention allocation model, REDDPG, by combining the multi-agent deep deterministic policy gradient (MADDPG) and RETransformer. This is achieved through the following: Build a multi-agent deep deterministic policy gradient MADDPG training framework: In a multi-agent environment, MADDPG is suitable for centralized learning and decentralized execution, enabling agents to learn to collaborate and compete. The actor-critic architecture in MADDPG consists of two parts: the policy network actor and the value network critic, both of which are composed of the current network and the target network. In the drone swarm scenario, for each drone i, the current actor network By θ i Parameterization, current critic network By φ i Parameterization, i.e. φ i are the parameters of the critic network, and the target networks of the two are respectively represented by θ i ′ and φ i ' parameterization; the actor network decides the action, while the critic network evaluates the action by estimating the value of the state; Designing a relation-enhanced RETransformer model: The RETransformer, designed based on attention allocation, integrates a distance-based adjacency matrix and an attention mask for collaborative navigation and obstacle avoidance. The RETransformer first incorporates the adjacency matrix representing the distance between drones into the query key QK matrix, enabling the model to capture and utilize the spatial relationship between drones and provide distance information. The attention mask is then used to ensure that each drone only pays attention to the k drones closest to it. This mechanism ensures that each drone is only influenced by its nearest neighbors in the decision-making process, maintaining a safe distance and connectivity within the drone swarm. Its expression is as follows: Where Q, K and V are linear transformations of the input sequence, d k is the dimension of the key; A is the adjacency matrix, whose elements A ij =-‖p i -p j ‖ represents the distance between drones i and j; M is the attention mask, which is masked by setting the attention score of distant drones to a negative value of -∞; Construct a relation-enhanced DDPG algorithm REDDPG based on attention allocation: The REDDPG algorithm is obtained by combining the two structures of steps (3.1) and (3.2). First, the environmental information accessible to the drone is feature encoded, and the state vector of each drone is converted into a high-dimensional embedding vector suitable for the model dimension. This process is implemented through a linear layer. Then, RETransformer is introduced into the actor network to combine the relationship matrix with the attention mechanism. The critic network uses a standard Transformer for policy evaluation to maintain training stability and policy quality. Build a framework module: Use the attention allocation mechanism to build a relationship-enhanced unmanned swarm system framework; Training model module: trains the unmanned swarm system model, converges to obtain the optimal model, and completes the path planning of the unmanned swarm movement.

Images