Unmanned bee colony motion path planning method and system based on attention distribution

Abstract

The invention discloses an unmanned bee colony motion path planning method and system based on attention distribution. The method comprises the following steps: designing a relation enhanced Transform (RETransform) to enhance the learning of an interactive cooperation relation of an unmanned bee colony in a motion process, and combining the relation enhanced Transform with a multi-agent depth deterministic strategy gradient (MADDPG) to obtain a relation enhanced DDPG (REDDPG); the method is high in efficiency, good in flexibility and easy and convenient to implement. According to the method, the attention of each unmanned aerial vehicle in the unmanned bee colony to nearby objects is enhanced, so that the unmanned aerial vehicle can avoid collision and maintain safe navigation of communication distance between the unmanned aerial vehicles, reasonable route planning is carried out in a collective cooperation mode, and the method has robustness and expandability for different complex scenes.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned systems, and in particular relates to an unmanned bee colony motion path planning method and system based on attention allocation. Background Art

[0002] In recent years, unmanned swarms have been widely studied and applied in tasks such as search and rescue and environmental monitoring due to their key advantages such as mission scalability, robustness against failures, and distributed intelligence. In these tasks, unmanned aerial vehicles need to coordinate and cooperate to reach the mission area from the deployment site while avoiding collisions or minimizing energy consumption. Achieving this autonomous collective swarming requires each swarm member to make control decisions based on its sensed state and interactions with its neighbors, while centralized control brings a heavy communication burden and cannot adapt to complex environments. Therefore, one of the main challenges of unmanned swarms is to achieve robust and scalable motion control decisions that enable a large number of drones to fly collaboratively while maintaining obstacle avoidance.

[0003] At present, the related work on realizing the motion of unmanned swarms can be divided into two categories: rule-based methods and reinforcement learning-based methods. In the rule-based method, the intelligent agent finds the basic motion principle based on the perceived obstacle information and neighbor status. Usually, the basic rules need to be quantified by multiple parameters, and the corresponding values ​​are different for different environments or tasks. Therefore, the main disadvantage of this method is that the pre-set control parameters cannot be changed when performing the task, and it cannot handle emergency situations that are not considered in the optimization process, that is, it is not robust to changes in environmental settings or its own state (such as cluster speed or communication range). Reinforcement learning-based methods can learn near-optimal behaviors without human intervention and work better in unpredictable environments. However, most existing reinforcement learning-based methods lack the ability to learn the interaction and collaboration relationship between cluster members and cannot effectively utilize the perceived environmental information. In addition, most models require a long training time or even fail to converge. Summary of the invention

[0004] In view of the above problems, the present invention proposes a method and system for unmanned bee colony motion path planning based on attention allocation, which enhances the learning of interactive and cooperative relationships of unmanned bee colonies during motion by designing a relation-enhanced Transformer (RETransformer), and combines it with multi-agent deep deterministic policy gradient (MADDPG) to obtain a relation-enhanced DDPG (REDDPG). The method is efficient, flexible and easy to implement.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] The present invention discloses an unmanned bee colony motion path planning method based on attention allocation, comprising:

[0007] (1) Obtain observable information of each drone in the swarm;

[0008] (2) Construct a dynamic optimization problem based on the swarm movement characteristics and design the corresponding reward function;

[0009] (3) Using the attention allocation mechanism to build a relationship-enhanced unmanned swarm system framework;

[0010] (4) Train the unmanned swarm system model, converge to the optimal model, and complete the path planning of the unmanned swarm movement.

[0011] Specifically, the observable information of each drone in the unmanned swarm in step (1) is defined as follows:

[0012]

[0013] where s i is the environmental state vector observed by the ith UAV, including the relative distance to the target, obstacles, neighboring UAVs and its own speed; p i is the position vector of the ith UAV, v i is the velocity vector of the ith UAV, N i is the neighbor set of the ith UAV, G is the position vector of the target location, and O is the position vector set of obstacles.

[0014] Furthermore, in step (2), a dynamic optimization problem is constructed based on the swarm movement characteristics, specifically:

[0015] (2.1) Arrival at the target area:

[0016] ||p i -G||≤ρ g ;

[0017] where ρ g Indicates the tolerable deviation distance of the target;

[0018] (2.2) Avoid collision:

[0019]

[0020] where d u and d o They are the safe distance between drones and the safe distance between drones and obstacles. is the radius of the mth obstacle;

[0021] (2.3) Maintaining connectivity:

[0022] ψ=ψ v +ψ d ,

[0023]

[0024] where ψ v and ψ d They represent velocity consistency and position consistency, respectively, v avg is the average velocity vector of the unmanned swarm, and N is the total number of drones in the unmanned swarm.

[0025] Furthermore, the step (2) designs a corresponding reward function, specifically: constructing an attention allocation-based model REDDPG by combining multi-agent deep deterministic policy gradient MADDPG and RETransformer; this step is implemented through the following sub-steps:

[0026] (3.1) Building a multi-agent deep deterministic policy gradient MADDPG training framework:

[0027] In a multi-agent environment, MADDPG is suitable for centralized learning and decentralized execution, enabling agents to learn to collaborate and compete. The actor-critic architecture in MADDPG consists of two parts: the policy network actor and the value network critic, both of which are composed of the current network and the target network. In the drone swarm scenario, for each drone i, the current actor network By θ i Parameterization, current critic network By φ i Parameterization, i.e. φ i are the parameters of the critic network, and the target networks of the two are respectively composed of θ i ′ and φ i ′ Parameterized; the actor network decides on the action, while the critic network evaluates the action by estimating the value of the state;

[0028] (3.2) Design of relation-enhanced RETransformer model:

[0029] The RETransformer designed based on attention allocation integrates the distance-based adjacency matrix and attention mask for collaborative navigation and obstacle avoidance. The RETransformer first incorporates the adjacency matrix representing the distance between drones into the query key QK matrix, so that the model can capture and utilize the spatial relationship between drones and provide distance information. Then, the attention mask is used to ensure that each drone only pays attention to the k drones closest to it. This mechanism ensures that each drone is only affected by its nearest neighbors in the decision-making process, maintaining the safe distance and connectivity of the drone swarm. Its expression is as follows:

[0030]

[0031] where Q, K, and V are linear transformations of the input sequence, d k is the dimension of the key; A is the adjacency matrix whose elements A ij =-‖p i -p j ‖ represents the distance between drones i and j; M is the attention mask, which is masked by setting the attention score of the distant drone to a large negative value (-∞);

[0032] (3.3) Construct a relation-enhanced DDPG algorithm REDDPG based on attention allocation:

[0033] The REDDPG algorithm is obtained by combining the two structures of step (3.1) and step (3.2). First, the environmental information accessible to the drone is feature encoded, and the state vector of each drone is converted into a high-dimensional embedding vector suitable for the model dimension. This process is implemented through a linear layer; then RETransformer is introduced into the actor network to combine the relationship matrix with the attention mechanism; the critic network uses an ordinary Transformer for strategy evaluation to maintain training stability and strategy quality.

[0034] Furthermore, the unmanned swarm system framework in step (3) is obtained by combining RETransforme and MADDPG; the policy network actor uses RETransformer to capture the complex interaction relationship between drones and output reasonable actions for each drone; the value network critic uses standard Transformer for policy evaluation to ensure the stability of training.

[0035] Furthermore, step (4) includes model initialization, training data preparation, loss optimization, model verification and evaluation, model optimization and convergence, specifically:

[0036] (4.1) Model initialization: First, initialize all networks and build an experience replay pool B to store sample sets (s, a, r, s ′ ); At each step, the drone selects an action based on the current strategy, interacts with the environment, and generates a new state transition sample, including the current state, action, reward, and next state;

[0037] (4.2) Training data preparation: After the model is initialized, it is stored in the experience replay pool, and the reward function is used to calculate the immediate reward for each time step;

[0038] (4.3) Loss optimization: By minimizing the loss function of the critic network, i.e., the mean square error loss, the parameters of the critic network are continuously updated; at the same time, the target value of the critic network output is maximized through the gradient ascent method to optimize the parameters of the actor network;

[0039] (4.4) Model validation and evaluation: After each training cycle, use the evaluation indicators to adjust the model's hyperparameters, including learning rate and batch size;

[0040] (4.5) Model optimization and convergence: As training progresses, monitor the convergence of the model to ensure that the model reaches the optimal state after a certain number of iterations and completes the path planning of the unmanned swarm movement.

[0041] The present invention also discloses an unmanned bee colony motion path planning system based on attention allocation, comprising the following modules:

[0042] Information acquisition module: obtains observable information of each drone in the drone swarm;

[0043] Build the design module: construct dynamic optimization problems based on the swarm movement characteristics and design corresponding reward functions;

[0044] Build framework module: Use attention allocation mechanism to build a relationship-enhanced unmanned swarm system framework;

[0045] Training model module: train the unmanned swarm system model, converge to the optimal model, and complete the path planning of the unmanned swarm movement.

[0046] The application object is not only unmanned bee colonies, but any unmanned system with group behavior can use this method.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] This paper proposes a method for unmanned bee swarm motion path planning based on attention allocation. The model enhances the attention of each drone to nearby objects by learning the interaction between drones and the environment, thereby achieving safe navigation that can avoid collisions and maintain inter-machine communication distance, allowing the unmanned bee swarm to perform reasonable route planning in a collective collaborative manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0050] Figure 1 This is a model architecture diagram of the unmanned bee swarm motion path planning method based on attention allocation;

[0051] Figure 2 This is a model training flowchart of the unmanned swarm motion path planning method based on attention allocation. DETAILED DESCRIPTION

[0052] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0053] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0054] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments:

[0055] The first aspect of the present invention provides a method for unmanned bee colony motion path planning based on attention allocation, comprising the following steps:

[0056] Step 1: Each drone can detect environmental information through its equipped radar, so it can obtain the location information of surrounding objects, and can transmit this information through inter-machine communication. Therefore, the observable information that each drone in the unmanned swarm can obtain is defined as follows:

[0057]

[0058] where s iis the environmental state vector observed by the ith UAV, including the relative distance to the target, obstacles, neighboring UAVs and its own speed. i is the position vector of the ith UAV, v i is the velocity vector of the ith UAV, is the neighbor set of the ith UAV, G is the position vector of the target location, and O is the position vector set of obstacles.

[0059] Step 2: The unmanned swarm movement needs to consider three key performance indicators: completing the goal, avoiding collisions, and maintaining connections. The following dynamic optimization problem can be constructed:

[0060] (2.1) Arrive at the target area.

[0061] ||p i -G||≤ρ g

[0062] where ρ g Indicates the tolerable deviation distance of the target.

[0063] (2.2) Avoid collision.

[0064]

[0065] where d u and d o They are the safe distance between drones and the safe distance between drones and obstacles. is the radius of the mth obstacle.

[0066] (2.3) Maintain connectivity.

[0067] ψ=ψ v +ψ d ,

[0068]

[0069] where ψ v and ψ d They represent velocity consistency and position consistency, respectively, v avg is the average velocity vector of the unmanned swarm, and N is the total number of drones in the unmanned swarm.

[0070] According to the dynamic optimization problem established above, the corresponding reward function can be designed:

[0071] (2.4) Reward for reaching the target. During the flight, the closer the drone is to the target, the greater the reward. Once it reaches the radius ρ g The target area, the UAV will receive a reward r goal .

[0072]

[0073] (2.5) Reward for avoiding collision. If the distance between the drone and the nearest obstacle is less than the safe distance, it will be penalized.

[0074]

[0075] (2.6) Rewards for maintaining connectivity. Cooperation in an unmanned swarm requires that the drones maintain an appropriate communication distance. We use the DBSCAN algorithm to calculate the degree of outliers of the drones. The outlier degree reflects the compactness and looseness of the unmanned swarm. The smaller the outlier degree, the better the effect.

[0076]

[0077] Where N outlier is the number of outlier drones.

[0078] The ideal state is to keep the distance between drones constant. When the distance between any two adjacent drones is too far (unable to maintain communication) or too close (less than the safe distance d u ), they should all be punished accordingly.

[0079]

[0080] According to the above reward function, the drone swarm can achieve the task of reaching the target area while avoiding obstacles, maintaining consistency and communication connectivity during flight, and minimizing the outlier degree through learning. The reward function for drone i is defined as:

[0081] R i =R goal,i +R collision,i +R outlier,i +R neighbor,i .

[0082] Step 3: Combine Multi-Agent Deep Deterministic Policy Gradient (MADDPG) and Relation Enhanced Transformer (RETransformer) to build the attention allocation based model REDDPG. This step is implemented through the following sub-steps:

[0083] (3.1) Build a multi-agent deep deterministic policy gradient (MADDPG) training framework.

[0084] In a multi-agent environment, MADDPG is suitable for centralized learning and decentralized execution, allowing agents to learn to collaborate and compete. The actor-critic architecture in MADDPG consists of two parts, namely the policy network actor and the value network critic, both of which are composed of the current network and the target network. In the drone swarm scenario, for each drone i, the current actor network By θ i Parameterization, current critic network By φ i Parameterization, i.e. φ i are the parameters of the critic network. Similarly, their target networks are respectively composed of θ i ′ and φ i ′ Parameterization. The value network critic network evaluates actions by estimating Q values ​​and provides feedback to the policy network. The policy network actor decides on actions and uses feedback to continuously adjust the strategy so that better actions can be selected in future situations. This learning process enables each drone to gradually learn to perform tasks in a complex multi-agent environment and achieve the goal of swarm movement.

[0085] (3.2) Design a relation-enhanced Transformer (RETransformer) model.

[0086] RETransformer, designed based on attention allocation, significantly enhances the understanding of the internal relationships of the drone swarm by integrating the distance-based adjacency matrix and attention mask, achieving more efficient and collaborative collaborative navigation and obstacle avoidance. RETransformer first incorporates the adjacency matrix representing the distance between drones into the query key (QK) matrix, helping the model to better capture and utilize the spatial relationship between drones and provide clearer distance information. Then, the attention mask is used to ensure that each drone only pays attention to the k drones closest to it, reducing the computational complexity of the model and reducing the interference of irrelevant information. This mechanism ensures that each drone is mainly affected by its nearest neighbors in the decision-making process, thereby improving the accuracy and real-time performance of the decision and maintaining the safe distance and connectivity of the drone swarm.

[0087]

[0088] where Q, K, and V are linear transformations of the input sequence, d k is the dimension of the key. A is the adjacency matrix whose elements A ij =-‖p i -p j‖ represents the distance between drones i and j. M is the attention mask, which is masked by setting the attention scores of distant drones to large negative values ​​(-∞).

[0089] (3.3) Construct a relation-enhanced DDPG algorithm (REDDPG) based on attention allocation.

[0090] The REDDPG algorithm combines the advantages of multi-agent deep deterministic policy gradient (MADDPG) and relation-enhanced Transformer (RETransformer) to achieve efficient and coordinated unmanned swarm control. First, the environmental information available to the drone is feature encoded, and the state vector of each drone is converted into a high-dimensional embedding vector suitable for the model dimension. This process is implemented through a linear transformation layer, which maps the original state information to a more abstract representation space for subsequent model processing.

[0091] In the actor network, REDDPG introduces the Relationship Enhanced Transformer (RETransformer), which enhances the modeling ability of complex relationships between drones by combining the relationship matrix and the attention mechanism. The relationship matrix reflects the relative position and distance information between drones, while the attention mechanism can dynamically adjust the information interaction weights between different drones, so that each drone can make more intelligent decisions based on the relationship with other drones.

[0092] In the critic network part, REDDPG uses a common Transformer architecture for policy evaluation. While maintaining good training stability and policy quality, it simplifies the complexity of the model and avoids the overfitting problem that may be caused by introducing too much relational information in the critic network.

[0093] Step 4: Train the unmanned swarm system model. This step is implemented through the following sub-steps:

[0094] (4.1) Model initialization: Initialize the parameters of all neural networks, including the actor network, critic network, and RETransformer model. In order to enhance the stability of the model, a uniformly distributed random initialization method is used, and an appropriate learning rate is set for the parameters of each network.

[0095] (4.2) Training data preparation: Generate flight data of the drone swarm from the simulation environment. The data includes the state information of each drone, adjacency matrix, target position, and obstacle position. Construct an experience replay pool B to store state transition sample sets for model training. At each step, the drone selects an action according to the current strategy, interacts with the environment, and generates a new state transition sample (including the current state, action, reward, and next state, i.e. (s, a, r, s ′ )) is stored in the experience replay pool.

[0096] (4.3) Loss optimization: Use the reward function defined above to calculate the instantaneous reward for each time step. By minimizing the loss function of the critic network (i.e., mean square error loss), the parameters of the critic network are continuously updated. At the same time, the target value output by the critic network is maximized by the gradient ascent method to optimize the parameters of the actor network. Specifically, the current actor network generates the action of drone i at time step t Then, a small batch of samples is randomly drawn from the experience replay pool B, and the current critic network is trained and its parameters are updated using the gradient descent method. This process is achieved by minimizing the following loss function at time step t:

[0097]

[0098]

[0099] in is the target Q value, Calculated by the target critic network at time step (t+1).

[0100] The current actor network updates its parameters by maximizing the Q value output by the current critic network to output a better action, which is achieved through the following gradient ascent process:

[0101]

[0102] In order to maintain the stability of training, a soft update strategy is used to update the parameters of the target network, that is, to periodically copy the parameters of the target network from the parameters of the current network:

[0103]

[0104] Where τ << 1. In addition, the L2 regularization strategy is used to prevent overfitting.

[0105] (4.4) Model Validation and Evaluation: After each training cycle, the model is validated on unseen datasets to evaluate its performance in different scenarios, including target arrival rate, obstacle avoidance success rate, and connectivity maintenance. Based on these evaluation indicators, the model’s hyperparameters, such as learning rate and batch size, are adjusted to gradually improve the model’s performance.

[0106] (4.5) Model optimization and convergence: During the training process, a soft update strategy is used to update the parameters of the target network so that the target network approaches the current network more smoothly. As the training progresses, the convergence of the model is monitored to ensure that the model reaches the optimal state after a certain number of iterations.

[0107] The second aspect of the present invention: an unmanned bee colony motion path planning system based on attention allocation, comprising the following modules:

[0108] Information acquisition module: obtains observable information of each drone in the drone swarm;

[0109] Build the design module: construct dynamic optimization problems based on the swarm movement characteristics and design corresponding reward functions;

[0110] Build framework module: Use attention allocation mechanism to build a relationship-enhanced unmanned swarm system framework;

[0111] Training model module: train the unmanned swarm system model, converge to the optimal model, and complete the path planning of the unmanned swarm movement.

[0112] The unmanned swarm motion path planning method based on attention allocation proposed in the present invention enhances the attention of each drone to nearby objects by learning the interaction between the drone and the environment, thereby achieving safe navigation that can avoid collisions and maintain the communication distance between machines, and enables the unmanned swarm to perform reasonable route planning in a collective and collaborative manner. After considering the specification and practicing the contents disclosed herein, those skilled in the art will easily think of other implementation plans of the present application. This application is intended to cover any variations, uses or adaptive changes of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the technical field that are not disclosed in this application.

[0113] It should be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A method for unmanned bee colony motion path planning based on attention allocation, characterized in that: include: (1) Obtain observable information of each drone in the swarm; (2) Construct a dynamic optimization problem based on the swarm movement characteristics and design the corresponding reward function; (3) Using the attention allocation mechanism to build a relationship-enhanced unmanned swarm system framework; (4) Train the unmanned swarm system model, converge to the optimal model, and complete the path planning of the unmanned swarm movement.

2. The unmanned bee colony motion path planning method based on attention allocation according to claim 1 is characterized in that: The observable information of each drone in the unmanned swarm in step (1) is defined as follows: where s i is the environmental state vector observed by the ith UAV, including the relative distance to the target, obstacles, neighboring UAVs and its own speed; p i is the position vector of the ith UAV, v i is the velocity vector of the ith UAV, N i is the neighbor set of the ith UAV, G is the position vector of the target location, and O is the position vector set of obstacles.

3. The unmanned bee colony motion path planning method based on attention allocation according to claim 1 is characterized in that: In step (2), a dynamic optimization problem is constructed based on the swarm movement characteristics, specifically: (2.1) Arrival at the target area: ||p i -G||≤ρ g ; where ρ g Indicates the tolerable deviation distance of the target; (2.2) Avoid collision: where d u and d o They are the safe distance between drones and the safe distance between drones and obstacles. is the radius of the mth obstacle; (2.3) Maintaining connectivity: ψ=ψ v +ψ d , where ψ v and ψ d They represent velocity consistency and position consistency, respectively, v avg is the average velocity vector of the unmanned swarm, and N is the total number of drones in the unmanned swarm.

4. The unmanned bee colony motion path planning method based on attention allocation according to claim 1 is characterized in that: The step (2) designs the corresponding reward function, specifically: constructing the attention allocation-based model REDDPG by combining the multi-agent deep deterministic policy gradient MADDPG and RETransformer; this step is implemented through the following sub-steps: (3.1) Building a multi-agent deep deterministic policy gradient MADDPG training framework: In a multi-agent environment, MADDPG is suitable for centralized learning and decentralized execution, enabling agents to learn to collaborate and compete. The actor-critic architecture in MADDPG consists of two parts: the policy network actor and the value network critic, both of which are composed of the current network and the target network. In the drone swarm scenario, for each drone i, the current actor network By θ i Parameterization, current critic network By φ i Parameterization, i.e. φ i are the parameters of the critic network, and the target networks of the two are respectively composed of θ′ i and φ′ i Parameterized; the actor network decides on the action, while the critic network evaluates the action by estimating the value of the state; (3.2) Design of relation-enhanced RETransformer model: The RETransformer designed based on attention allocation integrates the distance-based adjacency matrix and attention mask for collaborative navigation and obstacle avoidance. The RETransformer first incorporates the adjacency matrix representing the distance between drones into the query key QK matrix, so that the model can capture and utilize the spatial relationship between drones and provide distance information. Then, the attention mask is used to ensure that each drone only pays attention to the k drones closest to it. This mechanism ensures that each drone is only affected by its nearest neighbors in the decision-making process, maintaining the safe distance and connectivity of the drone swarm. Its expression is as follows: where Q, K, and V are linear transformations of the input sequence, d k is the dimension of the key; A is the adjacency matrix, whose elements A ij =-‖p i -p j ‖ represents the distance between drones i and j; M is the attention mask, which is masked by setting the attention score of the distant drone to a large negative value (-∞); (3.3) Construct a relation-enhanced DDPG algorithm REDDPG based on attention allocation: The REDDPG algorithm is obtained by combining the two structures of step (3.1) and step (3.2). First, the environmental information accessible to the drone is feature encoded, and the state vector of each drone is converted into a high-dimensional embedding vector suitable for the model dimension. This process is implemented through a linear layer; then RETransformer is introduced into the actor network to combine the relationship matrix with the attention mechanism; the critic network uses an ordinary Transformer for strategy evaluation to maintain training stability and strategy quality.

5. The unmanned bee colony motion path planning method based on attention allocation according to claim 1 is characterized in that: The unmanned swarm system framework in step (3) is obtained by combining RETransforme and MADDPG; The policy network actor uses RETransformer to capture the complex interactions between drones and output reasonable actions for each drone. The value network critic uses the standard Transformer for strategy evaluation to ensure the stability of training.

6. The unmanned bee colony motion path planning method based on attention allocation according to claim 1 is characterized in that: The step (4) includes model initialization, training data preparation, loss optimization, model verification and evaluation, model optimization and convergence, specifically: (4.1) Model initialization: First, initialize all networks and build an experience replay pool B to store sample sets (s, a, r, s′); at each step, the drone selects actions based on the current strategy, interacts with the environment, and generates new state transition samples, including the current state, action, reward, and next state; (4.2) Training data preparation: After the model is initialized, it is stored in the experience replay pool, and the reward function is used to calculate the immediate reward for each time step; (4.3) Loss optimization: By minimizing the loss function of the critic network, i.e., the mean square error loss, the parameters of the critic network are continuously updated; at the same time, the target value of the critic network output is maximized through the gradient ascent method to optimize the parameters of the actor network; (4.4) Model validation and evaluation: After each training cycle, use the evaluation indicators to adjust the model's hyperparameters, including learning rate and batch size; (4.5) Model optimization and convergence: As training progresses, monitor the convergence of the model to ensure that the model reaches the optimal state after a certain number of iterations and completes the path planning of the unmanned swarm movement.

7. An unmanned bee colony motion path planning system based on attention allocation, characterized in that: Includes the following modules: Information acquisition module: obtains observable information of each drone in the drone swarm; Build the design module: construct dynamic optimization problems based on the swarm movement characteristics and design corresponding reward functions; Build framework module: Use attention allocation mechanism to build a relationship-enhanced unmanned swarm system framework; Training model module: train the unmanned swarm system model, converge to the optimal model, and complete the path planning of the unmanned swarm movement.

Images