A mappo unmanned aerial vehicle network adaptive clustering method based on local individual potential reward

Abstract

The application belongs to the technical field of unmanned aerial vehicle self-organizing network, wireless communication network optimization and multi-agent reinforcement learning, and discloses a MAPPO unmanned aerial vehicle network adaptive clustering method based on local individual potential reward. An unmanned aerial vehicle network architecture modeling module, a communication transmission performance modeling module, a local individual potential reward modeling module and a joint optimization decision module are designed. The method proposed in the application can enhance the feedback of individual action contribution under the condition of sharing external reward, and can alleviate the reward dilution and credit allocation difficulty in multi-unmanned aerial vehicle cooperative decision making. In different unmanned aerial vehicle quantity, area scale and topology refresh cycle scenes, the number of successfully transmitted packets and the reliability of unmanned aerial vehicle network communication are effectively improved, and the overall performance is better than that of other existing methods.

Description

Technical Field

[0001] This invention relates to the fields of unmanned aerial vehicle (UAV) self-organizing networks, wireless communication network optimization, and multi-agent reinforcement learning, and particularly to an adaptive clustering method for MAPPO UAV networks based on the potential rewards of local individuals. Background Technology

[0002] With the development of applications such as low-altitude economy, emergency communication, disaster relief, environmental monitoring, and intelligent inspection, multiple drones can act as aerial communication nodes to form a self-organizing network, providing temporary communication coverage and data relay services to ground users or mission nodes. To reduce routing maintenance overhead and improve network scalability, drone networks typically adopt a clustering structure, with some drones designated as cluster heads. These cluster heads are responsible for functions such as intra-cluster management, inter-cluster forwarding, and backbone routing maintenance.

[0003] However, drone networks are characterized by high-speed movement, rapid topology changes, fluctuating link quality, and time-varying service load. Traditional clustering methods typically rely on fixed-period re-clustering or static rules for member allocation. When the update cycle is too short, the cluster structure oscillates frequently, increasing routing and control overhead; when the update cycle is too long, the cluster structure struggles to respond promptly to link changes, and ordinary drones may remain in clusters with poor link quality for extended periods, thus affecting successful data packet transmission.

[0004] In recent years, multi-agent reinforcement learning has been used for collaborative decision-making in UAV networks, enabling individual UAVs to learn cluster member selection strategies under local observation conditions. Methods such as MAPPO, which involve centralized training and distributed execution, can improve policy stability during the training phase by utilizing global state, while also meeting the distributed decision-making needs of UAVs during the execution phase. However, under conditions of shared global rewards, the actions of multiple UAVs collectively affect the final communication performance. Individual UAVs struggle to accurately identify their own contribution to the number of successfully transmitted packets, leading to reward dilution, unclear credit allocation, and unstable policy convergence.

[0005] Therefore, there is an urgent need for an adaptive cluster member decision-making method for dynamic clustering scenarios in UAV networks, which can construct clearer individual contribution feedback for each UAV while maintaining distributed execution capabilities, thereby improving the number of successfully transmitted packets and the reliability of network communication. Summary of the Invention

[0006] To address the aforementioned issues, this invention proposes an adaptive clustering method for MAPPO UAV networks based on local individual potential rewards. By constructing a self-organizing UAV clustering network model, a communication transmission performance model, and a local individual potential reward model, the method increases the number of successfully transmitted packets in the UAV network while ensuring decentralized execution.

[0007] The technical solution of the present invention is as follows: an adaptive clustering method for MAPPO UAV networks based on local individual potential rewards, comprising a UAV network architecture modeling module, a communication transmission performance modeling module, a local individual potential reward modeling module, and a joint optimization decision module; The UAV network architecture modeling module is used to construct a clustered self-organizing network composed of multiple UAVs. It discretizes the UAV network operation time into multiple equal-length time slots, and in each time slot, it describes the UAV's position, speed, remaining energy, service load, target node, cluster head identifier, cluster number, and cluster head number. It also completes cluster division, cluster head selection, and cluster head backbone route maintenance based on the UAV's position within a preset topology refresh time slot. The communication transmission performance modeling module is used to calculate the single-hop packet loss rate based on the communication distance between UAVs, transmission power, interference from heterogeneous transmitting nodes, noise power, and modulation method, and to count the number of attempted transmission packets and the number of successfully transmitted packets based on the routing path from the source UAV to the target UAV. The local individual potential reward modeling module is used to learn the potential contribution of each UAV under the action selection of different cluster members through the individual potential function network on the basis of shared external rewards, construct local individual potential rewards, and generate individual agent rewards based on local individual potential rewards. The joint optimization decision module is used to construct an adaptive clustering optimization problem for UAV networks based on the UAV network architecture module and the communication transmission performance module, with the goal of maximizing the cumulative number of successfully transmitted packets. The MAPPO method based on individual agent rewards is used to solve the optimization problem, so that ordinary UAVs can choose to stay in the current cluster or migrate to the target cluster corresponding to the visible cluster head based on local observations during the execution phase.

[0008] In the UAV network architecture modeling module, the UAV set is represented as Representing the cluster set as The operation process of the unmanned aerial vehicle system is divided into A discrete time slot; For the A drone, in a time slot The internal state includes three-dimensional position. Velocity vector Remaining energy Business load Target Node Sending data Is it a cluster head identifier? Cluster number Cluster head number Visible cluster head set Number of packets attempted to be transmitted Number of packets successfully transmitted and migration identifier ; Within the topology refresh time slot, the K-Means method is used to partition the topology based on the UAV's 3D position. The system is divided into clusters, and within each cluster, the drone closest to the corresponding cluster center is selected as the cluster head. During non-topology refresh time slots, the cluster head drone maintains its cluster head role, while ordinary drones perform cluster member selection actions according to the learned strategy.

[0009] The local individual potential reward modeling module specifically includes: Estimating shared external state value using external value networks ,in The global state is visible during the training phase; the individual potential function network is used as the input global state. and the Local observation by drone Output the potential function values ​​of the UAV for each discrete action. ; Based on the current action and the next action The corresponding potential function value, construct the first The potential individual reward for a drone: ; Based on the local individual potential reward and shared external reward Construct individual agent rewards: ; in The potential reward weight for a local individual; Constructing individual agent state value And calculate the individual agent advantage based on the individual agent reward and the individual agent state value. , used to update the policy network.

[0010] The specific adaptive clustering optimization problem of the UAV network in the joint optimization decision module is as follows: make Indicates the first A drone in a time slot Cluster member selection action, where This indicates that the current cluster should remain unchanged. This indicates selecting the corresponding cluster head from the currently visible cluster head set; let This indicates that all drones are in the time slot. Joint actions; This represents the set of valid service transmission paths generated by the current cluster structure and routing table, where any path... Represented as ,in As the source node, Let be the target node; Representing a path In the time slot The number of data packets that finally reach the target node after hop-by-hop link loss is the number of packets in the UAV system within the time slot. The number of successfully transmitted packets within the time limit is: ; The objective function of the adaptive clustering optimization problem for unmanned aerial vehicle (UAV) networks is: in, The strategy parameter is The expected cumulative number of successfully transmitted packets for the drone network. Indicating in strategy The expected value is obtained under the induced state-action trajectory distribution. Choose an indicator variable for the action. For action mask, Indicates the first Is the drone the first The cluster head of each cluster, For cluster head set, For the topology refresh time slot set, For the first The cluster head currently connected to or selected by the drone. Indicates time slot Internal drones Connected or selected cluster head Spatial location, For the corresponding communication range, The number of data packets sent. For the service load to be sent, To maximize transmission capacity, For the remaining energy, Indicates a valid service transmission path The first 1 node Representing a path Above The next node after that, This refers to the set of valid communication links in the current network. Constraint C1 restricts each UAV to select only one cluster member action in each time slot, i.e., keeping the current cluster unchanged or selecting a target cluster corresponding to a visible cluster head; Constraint C2 restricts UAVs to select only valid actions allowed by the action mask, ensuring that invisible or invalid cluster heads are not selected by the policy network; Constraint C3 restricts each cluster to have exactly one cluster head UAV under the current topology refresh result; Constraint C4 restricts cluster head UAVs to maintain their cluster head role unchanged in non-topology refresh time slots and not participate in ordinary cluster member migration decisions; Constraint C5 restricts ordinary UAVs to meet the communication distance requirements with their currently connected or selected cluster heads; Constraint C6 restricts the number of data packets sent by a UAV in a single time slot to not exceed its pending service load and maximum transmission capacity; Constraint C7 restricts the remaining energy of the UAV to not be lower than the minimum energy threshold to ensure that it has communication forwarding capabilities; Constraint C8 restricts adjacent nodes in any valid service transmission path to belong to valid communication links in the current network, thereby ensuring that the number of successfully transmitted packets is only obtained from the final delivered data packets of reachable service paths.

[0011] The solution process of the joint optimization decision module includes the cluster head backbone topology refresh sub-process and the LIPR-MAPPO cluster member decision sub-process; The K-Means method is used to handle the cluster head backbone topology refresh sub-process; The MAPPO method, based on the potential rewards of local individuals, handles the cluster member decision subprocess.

[0012] The specific sub-process for refreshing the cluster head backbone topology is as follows: Within the preset topology refresh time slot, K-Means clustering is performed based on the 3D positions of all drones to obtain... Each cluster has its own cluster center; for each cluster, the drone closest to the cluster center is selected as the cluster head, and the other drones in the cluster are treated as ordinary cluster members; Set a first transmit power and a first communication range for cluster heads, and set a second transmit power and a second communication range for ordinary cluster members; perform neighbor discovery within the cluster head set, and establish a neighbor relationship when the distance between two cluster heads does not exceed the cluster head's communication range; Each cluster head selects a multi-point relay set based on the coverage of one-hop neighbors to two-hop neighbors, updates the topology table based on the multi-point relay set, and generates a routing table to other cluster heads based on the cluster head neighbor relationship.

[0013] The specific sub-process for LIPR-MAPPO cluster membership decision-making is as follows: Each UAV is modeled as an intelligent agent, where the policy network takes the local observations and action masks of a single UAV as input and outputs a discrete action distribution on selectable actions. The local observations include self-state features and candidate cluster head state features. Self-state features include normalized position, normalized velocity, velocity magnitude, workload, remaining energy, and one-hot encoding of the current cluster number. Candidate cluster head state features include the relative position, velocity, workload, energy, distance, estimated packet loss rate, cluster size, average cluster workload, average cluster energy, and whether it is the current cluster head. Action space is The actions This indicates that the current cluster should remain unchanged; the action... To action This indicates selecting the corresponding cluster head from the currently visible cluster head set; the action in the action mask. Always optional, action To action This option is only available if the corresponding cluster head is visible. During the training phase, the policy network is updated using a truncated objective function of proximal policy optimization, the external value network is updated by minimizing the mean square error between the external reward temporal difference objective and the external value estimate, and the individual potential function network is updated by a loss function that aligns the local individual potential reward with the external advantage direction. During the execution phase, each UAV independently selects cluster member actions based solely on its own local observations and action mask.

[0014] The beneficial effects of this invention are as follows: This invention proposes an adaptive clustering method for MAPPO UAV networks based on local individual potential rewards. By introducing a time-slot-level cluster member adaptive decision-making mechanism in the joint optimization decision module, based on periodic cluster head initialization and cluster head backbone routing maintenance, ordinary UAVs can choose to stay within a cluster or migrate across clusters based on current local observations. The local individual potential reward mechanism constructs individual agent rewards related to each UAV's actions, mitigating the contribution dilution problem caused by sharing global rewards. Through a centralized training and distributed execution framework, the training phase utilizes global state to improve policy learning stability, while the execution phase relies only on local observations and action masks. Using the number of successfully transmitted packets as the main evaluation index, under different UAV numbers, regional scales, and communication range configurations, it can improve or maintain a high number of successfully transmitted packets overall, thereby enhancing the reliability of UAV network communication. Attached Figure Description

[0015] Figure 1 This is a diagram illustrating the overall framework of the MAPPO UAV network adaptive clustering method based on local individual potential rewards.

[0016] Figure 2This is a schematic diagram of the initialization of the UAV network cluster structure and the maintenance of the cluster head backbone route. (A) is the initialization of the cluster structure, (B) is the maintenance of the cluster head backbone route, and (C) is the collaborative maintenance process of the cluster structure and backbone route.

[0017] Figure 3 This is a schematic diagram of the local observation, action masking, and LIPR-MAPPO policy training process.

[0018] Figure 4 This is a schematic diagram comparing the training performance of LIPR-MAPPO (the method of this invention) and the baseline algorithm with 20 drones, a 500×500×10 meter area, and a topology refresh cycle of 5.

[0019] Figure 5 This is a schematic diagram comparing the training performance of LIPR-MAPPO (the method of this invention) and the baseline algorithm with 20 drones, a 500×500×10 meter area, and a topology refresh cycle of 10.

[0020] Figure 6 This is a schematic diagram comparing the training performance of LIPR-MAPPO (the method of this invention) and the baseline algorithm with 20 drones, a 1000×1000×10 meter area, and a topology refresh cycle of 5.

[0021] Figure 7 This is a schematic diagram comparing the training performance of LIPR-MAPPO (the method of this invention) and the baseline algorithm with 20 drones, a 1000×1000×10 meter area, and a topology refresh cycle of 10.

[0022] Figure 8 This is a schematic diagram comparing the training performance of LIPR-MAPPO (the method of this invention) and MAPPO in an area of ​​30 UAVs, 500×500×10 meters, and a topology refresh cycle of 5.

[0023] Figure 9 This is a schematic diagram comparing the training performance of LIPR-MAPPO (the method of this invention) and MAPPO in an area of ​​500×500×10 meters with 30 drones and a topology refresh cycle of 10.

[0024] Figure 10 This is a schematic diagram comparing the training performance of LIPR-MAPPO (the method of this invention) and the baseline algorithm with 30 drones, a 1000×1000×10 meter area, and a topology refresh cycle of 5.

[0025] Figure 11 This is a schematic diagram comparing the training performance of LIPR-MAPPO (the method of this invention) and the baseline algorithm with 30 drones, a 1000×1000×10 meter area, and a topology refresh cycle of 5.

[0026] Figure 12 This is a schematic diagram comparing the test performance of LIPR-MAPPO (the method of this invention) and the baseline algorithm with 20 UAVs, a 500×500×10 meter area, and a topology refresh cycle of 5.

[0027] Figure 13 This is a schematic diagram comparing the test performance of LIPR-MAPPO (the method of this invention) and the baseline algorithm with 20 UAVs, a 500×500×10 meter area, and a topology refresh cycle of 10.

[0028] Figure 14 This is a schematic diagram comparing the test performance of LIPR-MAPPO (the method of this invention) and the baseline algorithm with 20 UAVs, a 1000×1000×10 meter area, and a topology refresh cycle of 5.

[0029] Figure 15 This is a schematic diagram comparing the test performance of LIPR-MAPPO (the method of this invention) and the baseline algorithm with 20 UAVs, a 1000×1000×10 meter area, and a topology refresh cycle of 10.

[0030] Figure 16 This is a schematic diagram comparing the test performance of LIPR-MAPPO (the method of this invention) and the baseline algorithm with 30 UAVs, a 500×500×10 meter area, and a topology refresh cycle of 5.

[0031] Figure 17 This is a schematic diagram comparing the test performance of LIPR-MAPPO (the method of this invention) and the baseline algorithm with 30 UAVs, a 500×500×10 meter area, and a topology refresh cycle of 10.

[0032] Figure 18 This is a schematic diagram comparing the test performance of LIPR-MAPPO (the method of this invention) and the baseline algorithm with 30 UAVs, a 1000×1000×10 meter area, and a topology refresh cycle of 5.

[0033] Figure 19 This is a schematic diagram comparing the test performance of LIPR-MAPPO (the method of this invention) and the baseline algorithm with 30 UAVs, a 1000×1000×10 meter area, and a topology refresh cycle of 10. Detailed Implementation

[0034] The core idea of ​​this invention is to retain the periodic cluster head initialization and cluster head backbone routing maintenance mechanism in the UAV clustered self-organizing network, while introducing the MAPPO cluster member decision mechanism based on the potential reward of local individuals in the non-topology refresh time slot. This enables ordinary UAVs to make adaptive choices between maintaining the current cluster and migrating to a visible cluster head based on local observations, with the goal of increasing the number of successfully transmitted packets.

[0035] An adaptive clustering method for MAPPO UAV networks based on local individual potential rewards, such as... Figure 1 As shown, it includes a UAV network architecture modeling module, a communication transmission performance modeling module, a local individual potential reward modeling module, and a joint optimization decision-making module; The UAV network architecture modeling module is used to construct a clustered self-organizing network composed of multiple UAVs, and maintain the position, speed, energy, load, target node, cluster number, cluster head number, cluster head identifier, and set of visible cluster heads of each UAV in discrete time slots; The communication transmission performance modeling module is used to calculate the transmission loss of each data packet on the multi-hop path based on the current cluster structure and routing path, and to count the number of packets attempted to be transmitted and the number of packets successfully transmitted. The local individual potential reward modeling module is used to construct local individual potential rewards and individual agent rewards based on shared external rewards, individual potential function networks, and policy actions. The joint optimization decision module is used to train and execute the LIPR-MAPPO strategy, enabling ordinary UAVs to independently select cluster members based on their local observations and action masks during the execution phase.

[0036] The specific details of the UAV network architecture modeling module are as follows: Consider a drone network consisting of n drones, each of which acts as both a communication node and an intelligent agent. The system runtime is divided into T discrete time slots. At the start of each training round, the system randomly selects the trajectory start time, reads the 3D position, velocity, target node, and service data of each drone at that time, and initializes the drone state. The drone service load is determined by the target node and task data; data to be sent is generated when a drone has a target node, otherwise no service data is generated.

[0037] During the topology refresh time slot, the system uses the K-Means method to cluster the locations of all UAVs. After clustering, the system selects the UAV closest to the cluster center within each cluster as the cluster head, and the other UAVs as ordinary cluster members. The cluster head uses a larger communication range and transmission power, while the ordinary cluster members use a smaller communication range and transmission power. Cluster heads form a cluster head backbone network through neighbor discovery, multi-point relay selection, topology table updates, and routing table updates.

[0038] The specific details of the communication transmission performance modeling module are as follows: For any single-hop link, the free-space path loss is first calculated based on the distance between the transmitting and receiving nodes, and then the received signal strength is calculated based on the transmitting power and path loss. Subsequently, the signal-to-interference-plus-noise ratio (SIR) is calculated based on the received signal strength, the interference power caused by heterogeneous transmitting nodes, and the noise power, and then the threshold corresponding to the modulation scheme is applied. The function calculates the single-hop packet loss rate.

[0039] When both the source and destination nodes are cluster heads, data is transmitted via the cluster head backbone routing. When the source node is a regular cluster member and the destination node is a cluster head, data is first transmitted from the source node to the source cluster head, and then forwarded via the cluster head backbone network. When the destination node is a regular cluster member, data is sent to the destination node only after reaching the destination cluster head. For a multi-hop path, the system calculates the number of lost packets and received packets hop by hop, ultimately obtaining the number of packets attempted and successfully transmitted by the source node.

[0040] During training, the environment only returns the shared external reward. The shared external reward is obtained from the cumulative number of successfully transmitted packets, and is represented as: ; in Indicates the cutoff time slot The cumulative number of successfully transmitted packets. The shared external reward reflects the overall network communication transmission effect and is not subject to manual individual reward decomposition on the environment side.

[0041] The specific local observation and action space are as follows: No. The local observations of the UAV include its own state characteristics and candidate cluster head state characteristics. The UAV's own state characteristics include normalized position, normalized velocity vector, normalized velocity magnitude, normalized workload, normalized remaining energy, and the one-hot encoding of the current cluster number. Candidate cluster head state characteristics include the relative position of the candidate cluster head, candidate cluster head velocity, candidate cluster head workload, candidate cluster head energy, normalized distance to the candidate cluster head, estimated packet loss rate, candidate cluster size, average candidate cluster workload, average candidate cluster energy, and whether the candidate cluster head is the current cluster head. When the number of visible cluster heads is insufficient... When there are missing candidate cluster head features, zero-padding is performed.

[0042] Action space is The actions This indicates that the current cluster should remain unchanged; the action... To action This indicates selecting the corresponding cluster head from the currently visible cluster head set; the action in the action mask. Always optional, action To action This option is only available if the corresponding cluster head is visible. When the policy network outputs the action distribution, the logits corresponding to actions that cannot be selected are masked.

[0043] The specific details of the local individual potential reward modeling module are as follows: LIPR stands for Local Individual Potential-based Reward. The local individual potential reward is learned by an individual potential function network, rather than being manually decomposed into global rewards by the environment. For the... A drone is deployed, and the individual potential function network is configured with a global state. and local observation As input, output the potential function values ​​of the UAV for each action. .

[0044] Based on the current action and the next action The corresponding potential function value, construct the first The potential individual reward for a drone: ; Based on the local individual potential reward and shared external reward Construct individual agent rewards: ; Individual agent state value: ; Individual agent advantage can be calculated based on individual agent rewards and individual agent state values. This individual agent advantage is used to replace the fully shared advantage in policy network updates, enabling each drone to receive training feedback that is more relevant to its own actions.

[0045] The joint optimization decision-making module is as follows: The training phase employs a centralized training and distributed execution framework. The policy network takes as input the local observations and action masks of a single UAV and outputs a discrete action distribution; the external value network takes as input the global state and outputs a shared external state value; and the individual potential function network takes as input the global state and the local observations of a single UAV and outputs potential function values ​​in the action dimension.

[0046] In each training batch, the system stores local observations, global state, actions, log probabilities of old actions, shared external rewards, action masks, next local observations, next global state, and next action masks. During updates, the external value objective and external advantage are first calculated based on the shared external reward; then, the local individual potential reward, individual agent reward, and individual agent advantage are calculated through the individual potential function network; finally, the policy network is updated using PPO truncation, the external value network is updated using mean squared error loss, and the individual potential function network is updated using a loss that aligns the direction of local individual potential rewards with that of external advantages.

[0047] During the execution phase, each UAV does not need to acquire a global state or access the policy network input of other UAVs; it selects actions solely based on its own local observations and action masks. When the action is... When the drone remains in the current cluster, the action is greater than... When the operation is successful, the drone migrates to the cluster to which the corresponding visible cluster head belongs. The primary metric for the experiment is Successfully Transmitted Packets, i.e., the number of packets successfully transmitted.

[0048] The specific embodiments of the present invention will be described in detail below.

[0049] Step 1: Construct a UAV cluster network simulation environment.

[0050] Step one specifically includes UAV node initialization, trajectory status reading, periodic cluster head backbone topology refresh, local observation generation, action mask generation, service transmission path generation, and communication transmission feedback statistics. The system numbers the n UAVs as follows: Up to n, and treat them as n agents in multi-agent reinforcement learning; set the number of clusters to k, and divide the time in each training round into... A discrete time slot.

[0051] At the start of each training round, the environment randomly selects a starting time from the trajectory data and reads the current time from the trajectory file for each drone. Three-dimensional coordinates Velocity components, velocity magnitude, target node number, and mission data. For each UAV, initialize its position, velocity, target node, remaining energy, workload, cluster head identifier, cluster number, cluster head number, neighbor set, multipoint relay set, topology table, routing table, visible cluster head set, migration identifier, number of attempted transmission packets, and number of successful transmission packets.

[0052] When a target node exists for the drone, it is considered to have pending services in the current time slot, and newly generated data is added to the service load. When the target node is empty, the drone does not generate new end-to-end services in the current time slot. Therefore, the service load... The number of data packets sent changes dynamically with time slots. It is limited by the drone's remaining load, node role, and maximum transmission capacity.

[0053] During the topology refresh time slot, i.e. belong At that time, the system performs K-Means clustering based on the three-dimensional positions of all UAVs, resulting in K clusters and their cluster centers. For the k-th cluster, the UAV closest to the cluster center is selected as the cluster head, and the remaining UAVs are designated as ordinary cluster members. The cluster head is configured with higher transmission power and communication range, while the ordinary cluster members are configured with lower transmission power and communication range.

[0054] After the cluster heads are determined, the system performs backbone network maintenance within the cluster head set: if the distance between two cluster heads does not exceed the cluster head communication range, a one-hop neighbor relationship is established; each cluster head selects a multi-point relay set based on the coverage capability of the one-hop neighbor to the two-hop neighbor; subsequently, the topology table is updated based on the multi-point relay set, and a routing table to other cluster heads is generated according to the neighbor relationship between cluster heads. This backbone routing is used to support the construction of cross-cluster service transmission paths.

[0055] During non-topology refresh time slots, the cluster head drone maintains its cluster head role; ordinary drones select cluster members based on the actions output by the policy network. When an action... When the drone is in its current cluster, it remains unchanged; when the action... When this action corresponds to a valid visible cluster head, the UAV updates the cluster head number to the selected cluster head number and updates its own cluster number to the cluster number corresponding to the selected cluster head. If the selected cluster head is different from the original cluster head, a migration is recorded.

[0056] The local observations of each UAV consist of its own state characteristics and candidate cluster head state characteristics. The own state characteristics include normalized 3D position, normalized 3D velocity, normalized velocity magnitude, normalized workload, normalized remaining energy, and one-hot encoding of the current cluster number; the candidate cluster head state characteristics include the relative position, velocity, velocity magnitude, workload, energy, distance, estimated packet loss rate, candidate cluster size, average workload of candidate clusters, average energy of candidate clusters, and whether the candidate cluster head is the current cluster head.

[0057] Action masks are used to constrain the policy network to select only actions that can be executed in the current time slot. This indicates that the current cluster should be kept, and it is always set to optional; action To action These correspond to candidate cluster heads in the currently visible cluster head set. An action is only made optional if the corresponding candidate cluster head is within the current UAV's communication range. When the policy network outputs the action distribution, actions that are not selectable are... It is set to a minimum value to prevent invalid cluster heads from being selected.

[0058] Step 2: Construct a communication transmission performance model and count the number of successfully transmitted packets.

[0059] Within each time slot, the system first determines the number of data packets to send based on the drone's workload and node role. If the drone's remaining energy is not greater than the minimum energy threshold, then the number of data packets it sends is... If the drone is a cluster head, the number of data packets it sends cannot exceed its own pending service load and the cluster head's maximum sending capacity. If the drone is a regular cluster member, the number of data packets it sends is also limited by the shared sending capacity of regular members within the cluster. The number of data packets sent satisfies: ; in For the first A drone in a time slot The pending data load, This represents its maximum transmission capacity within that time slot. After transmission, the service load is updated as follows: ; in For time slots Newly generated business data. The above load update is used to ensure that the statistics on the number of successfully transmitted packets are derived from the actual business to be sent, rather than a fixed value independent of the business load.

[0060] For any two drone nodes and First, calculate the communication distance between the two: ; Given the carrier frequency f and the speed of light c, the link path loss is calculated using the free-space path loss model: ; Based on transmission power Calculate the received signal strength using path loss: ; Receiver node The interference is generated by a drone from a different cluster that is transmitting in the current time slot. Let... For receiving nodes Total interference power at the location, For noise power, then the link arrive The signal-to-interference-plus-noise ratio is expressed as: ; When the modulation scheme is QPSK, 16-QAM, or 64-QAM, the corresponding signal-to-interference-plus-noise ratio (SINNR) threshold should be used. and steepness coefficient ,pass Function to calculate single-hop packet loss rate: ; For a valid service transmission path Path first node The source node and the end node of the path Let b_l^0(t) = s_{v_0^l}(t) represent the initial number of packets sent along this path, then the th... The remaining number of packets after the jump is calculated recursively as follows: ; in For the first The remaining transmission capacity of the hop sending node in the current time slot. The final number of successfully delivered packets along the path is: ; Therefore, the system in time slots The number of successfully transmitted packets within the specified range is the sum of the total number of successfully delivered packets across all valid end-to-end service paths: ; This definition only counts the final delivered data packets that arrive at the target node, and does not repeatedly count intermediate forwarding packets from the source cluster head, relay cluster head, or target cluster head in the number of successfully transmitted packets in the system.

[0061] The system also updates the UAV's remaining energy based on communication transmission and flight motion. For a one-hop transmission and reception process within the path, transmission energy consumption is related to the number of packets transmitted, communication distance, and power amplification term, while reception energy consumption is related to the number of packets received. Simultaneously, the flight energy consumption is updated for each time slot based on the UAV's speed. The energy state can be represented as: ; When the energy of the sending or receiving node is below the minimum energy threshold, the number of packets that can be transmitted on the corresponding link for this path is: This ensures that communication statistics are constrained by energy state.

[0062] Step 3: Train a MAPPO cluster member decision model based on local individual potential rewards.

[0063] The training phase employs a centralized training and distributed execution framework. The policy network (Actor) takes the local observations and action masks of a single UAV as input and outputs a discrete action distribution; the external value network (Critic) takes the global state as input and estimates the shared external state value. PotentialNet, an individual potential function network, uses global state... Local observations of the i-th UAV As input, output the potential function values ​​of the UAV for all available actions. .

[0064] The specific training process is as follows:

[0065] Step 4: Conduct simulation experiments and verify the algorithm performance.

[0066] In one specific embodiment, the number of drones is set to 20 or 30, the number of clusters is set to 4, the average speed is set to 15 m / s, the area size is set to 500×500×10 meters or 1000×1000×10 meters, the training round length is set to 128 time slots, the batch size is set to 128, the number of training update rounds is set to 5, the basic transmission capacity per time slot is set to 200, the cluster head communication range is set to 250 meters or 500 meters, the ordinary cluster member communication range is set to 130 meters or 300 meters, the topology refresh cycle is set to 5 or 10, the learning rate is set to 0.003, and the discount factor is set to 0.99.

[0067] The comparison algorithms include K-Means-improve, MAPPO, QMIX, and IPPO. K-Means-improve represents a traditional baseline relying solely on periodic K-Means clustering and cluster head backbone routing maintenance; MAPPO represents a centralized training and decentralized execution proximal policy optimization method with shared external rewards; QMIX represents a value decomposition method based on individual Q-networks and hybrid networks; IPPO represents a proximal policy optimization method where each UAV trains independently based on local observations; and LIPR-MAPPO represents the MAPPO method proposed in this invention based on local individual potential rewards.

[0068] The experiment uses "Successfully Transmitted Packets" as the primary evaluation metric, which is the total number of packets that successfully reach the target node across all valid end-to-end service paths within a training round. This metric aligns with the optimization objective and directly reflects the data transmission reliability of the UAV network under the combined effects of its current cluster structure, routing paths, link packet loss, interference, and energy status.

[0069] like Figures 4 to 11 As shown, the training curves are used to evaluate the learning ability of each algorithm under different training scenarios. Among them, Figures 4 to 7 The scenarios involve 20 drones, covering areas of 500×500×10 meters and 1000×1000×10 meters respectively, as well as cases with topology refresh cycles of 5 and 10. Figures 8 to 11 The training scenarios involved 30 drones, covering areas of 500×500×10 meters and 1000×1000×10 meters respectively, with topology refresh cycles of 5 and 10. The training results show that, under different drone sizes and network parameters, LIPR-MAPPO maintains a high overall successful packet transmission rate curve, indicating that the local individual potential reward mechanism can improve policy learning feedback under shared reward conditions and enhance the training effect of multi-drone cluster member selection strategies.

[0070] like Figures 12 to 19As shown, the test curves are used to evaluate the successful transmission performance of the trained policy during the actual execution phase. Among them, Figures 12 to 15 To supplement the test scenarios for 20 drones, covering different area sizes and topology refresh cycles; Figures 16 to 19 Thirty drones were added to the test scenarios, covering different area sizes and topology refresh cycles. Test results show that LIPR-MAPPO can achieve a high or stable number of successfully transmitted packets overall under different scenarios, indicating that the method not only has good learning capabilities during the training phase but also effectively improves the data transmission reliability of the drone network during the execution phase.

Claims

1. An adaptive clustering method for MAPPO unmanned aerial vehicle networks based on local individual potential rewards, characterized in that, It includes a UAV network architecture modeling module, a communication transmission performance modeling module, a local individual potential reward modeling module, and a joint optimization decision-making module; The UAV network architecture modeling module is used to construct a clustered self-organizing network composed of multiple UAVs. It discretizes the UAV network operation time into multiple equal-length time slots, and in each time slot, it describes the UAV's position, speed, remaining energy, service load, target node, cluster head identifier, cluster number, and cluster head number. It also completes cluster division, cluster head selection, and cluster head backbone route maintenance based on the UAV's position within a preset topology refresh time slot. The communication transmission performance modeling module is used to calculate the single-hop packet loss rate based on the communication distance between UAVs, transmission power, interference from heterogeneous transmitting nodes, noise power, and modulation method, and to count the number of attempted transmission packets and the number of successfully transmitted packets based on the routing path from the source UAV to the target UAV. The local individual potential reward modeling module is used to learn the potential contribution of each UAV under the action selection of different cluster members through the individual potential function network on the basis of shared external rewards, construct local individual potential rewards, and generate individual agent rewards based on local individual potential rewards. The joint optimization decision module is used to construct an adaptive clustering optimization problem for UAV networks based on the UAV network architecture module and the communication transmission performance module, with the goal of maximizing the cumulative number of successfully transmitted packets. The MAPPO method based on individual agent rewards is used to solve the optimization problem, so that ordinary UAVs can choose to stay in the current cluster or migrate to the target cluster corresponding to the visible cluster head based on local observations during the execution phase.

2. The adaptive clustering method for MAPPO UAV networks based on local individual potential rewards according to claim 1, characterized in that, In the UAV network architecture modeling module, the UAV set is represented as Representing the cluster set as The operation process of the unmanned aerial vehicle system is divided into A discrete time slot; For the A drone, in a time slot The internal state includes three-dimensional position. velocity vector Remaining energy Business load Target Node Sending data Is it a cluster head identifier? Cluster number Cluster head number Visible cluster head set Number of packets already attempted to be transmitted Number of packets successfully transmitted and migration identifier ; Within the topology refresh time slot, the K-Means method is used to partition the topology based on the UAV's 3D position. The system is divided into clusters, and within each cluster, the drone closest to the corresponding cluster center is selected as the cluster head. During non-topology refresh time slots, the cluster head drone maintains its cluster head role, while ordinary drones perform cluster member selection actions according to the learned strategy.

3. The adaptive clustering method for MAPPO UAV networks based on local individual potential rewards according to claim 2, characterized in that, The local individual potential reward modeling module specifically includes: Estimating shared external state value using external value networks ,in The global state is visible during the training phase; the individual potential function network is used as the input global state. and the Local observation by drone Output the potential function values ​​of the UAV for each discrete action. ; Based on the current action and the next action The corresponding potential function value, construct the first The potential individual reward for a drone: ; Based on the local individual potential reward and shared external reward Construct individual agent rewards: ; in The potential reward weight for a local individual; Constructing Individual Agent State Value And calculate the individual agent advantage based on the individual agent reward and the individual agent state value. , used to update the policy network.

4. The adaptive clustering method for MAPPO UAV networks based on local individual potential rewards according to claim 3, characterized in that, The specific adaptive clustering optimization problem of the UAV network in the joint optimization decision module is as follows: make Indicates the first A drone in a time slot Cluster member selection action, where This indicates that the current cluster should remain unchanged. This indicates that the corresponding cluster head in the currently visible cluster head set is selected; make This indicates that all drones are in the time slot. Joint actions; This represents the set of valid service transmission paths generated by the current cluster structure and routing table, where any path... Represented as ,in As the source node, Let be the target node; Representing a path In the time slot The number of data packets that finally reach the target node after hop-by-hop link loss is the number of packets in the UAV system within the time slot. The number of successfully transmitted packets within the time limit is: ; The objective function of the adaptive clustering optimization problem for unmanned aerial vehicle (UAV) networks is: in, The strategy parameter is The expected cumulative number of successfully transmitted packets for the drone network. Indicating in strategy The expected value is obtained under the induced state-action trajectory distribution. Choose an indicator variable for the action. For action mask, Indicates the first Is the drone the first The cluster head of each cluster, For cluster head set, For the topology refresh time slot set, For the first The cluster head currently connected to or selected by the drone. Indicates time slot Internal drones Connected or selected cluster head Spatial location, For the corresponding communication range, The number of data packets sent. For the service load to be sent, To maximize transmission capacity, For the remaining energy, Indicates a valid service transmission path The first 1 node Representing a path Above The next node after that, This refers to the set of valid communication links in the current network. Constraint C1 restricts each UAV to select only one cluster member action in each time slot, i.e., keeping the current cluster unchanged or selecting a target cluster corresponding to a visible cluster head; Constraint C2 restricts UAVs to select only valid actions allowed by the action mask, ensuring that invisible or invalid cluster heads are not selected by the policy network; Constraint C3 restricts each cluster to have exactly one cluster head UAV under the current topology refresh result; Constraint C4 restricts cluster head UAVs to maintain their cluster head role unchanged in non-topology refresh time slots and not participate in ordinary cluster member migration decisions; Constraint C5 restricts ordinary UAVs to meet the communication distance requirements with their currently connected or selected cluster heads. Constraint C6 restricts the number of data packets sent by the UAV in a single time slot from exceeding its pending service load and maximum transmission capacity; Constraint C7 restricts the UAV's remaining energy from falling below the minimum energy threshold to ensure its communication forwarding capability; Constraint C8 restricts that adjacent nodes in any valid service transmission path must be valid communication links in the current network, thereby ensuring that the number of successfully transmitted packets is only obtained from the final delivered data packets of the reachable service path.

5. The adaptive clustering method for MAPPO UAV networks based on local individual potential rewards according to claim 4, characterized in that, The solution process of the joint optimization decision module includes the cluster head backbone topology refresh sub-process and the LIPR-MAPPO cluster member decision sub-process; The K-Means method is used to handle the cluster head backbone topology refresh sub-process; The MAPPO method, based on the potential rewards of local individuals, handles the cluster member decision subprocess.

6. The adaptive clustering method for MAPPO UAV networks based on local individual potential rewards according to claim 5, characterized in that, The specific sub-process for refreshing the cluster head backbone topology is as follows: Within the preset topology refresh time slot, K-Means clustering is performed based on the 3D positions of all drones to obtain... Each cluster has its own cluster center; for each cluster, the drone closest to the cluster center is selected as the cluster head, and the other drones in the cluster are treated as ordinary cluster members; Set a first transmit power and a first communication range for cluster heads, and set a second transmit power and a second communication range for ordinary cluster members; perform neighbor discovery within the cluster head set, and establish a neighbor relationship when the distance between two cluster heads does not exceed the cluster head's communication range; Each cluster head selects a multi-point relay set based on the coverage of one-hop neighbors to two-hop neighbors, updates the topology table based on the multi-point relay set, and generates a routing table to other cluster heads based on the cluster head neighbor relationship.

7. The adaptive clustering method for MAPPO UAV networks based on local individual potential rewards according to claim 5, characterized in that, The specific sub-process for LIPR-MAPPO cluster membership decision-making is as follows: Each UAV is modeled as an intelligent agent, where the policy network takes the local observations and action masks of a single UAV as input and outputs a discrete action distribution on selectable actions. The local observations include self-state features and candidate cluster head state features. Self-state features include normalized position, normalized velocity, velocity magnitude, workload, remaining energy, and one-hot encoding of the current cluster number. Candidate cluster head state features include the relative position, velocity, workload, energy, distance, estimated packet loss rate, cluster size, average cluster workload, average cluster energy, and whether it is the current cluster head. Action space is The actions This indicates that the current cluster should remain unchanged; the action... To action This indicates selecting the corresponding cluster head from the currently visible cluster head set; the action in the action mask. Always optional, action To action This option is only available if the corresponding cluster head is visible. During the training phase, the policy network is updated using a truncated objective function of proximal policy optimization, the external value network is updated by minimizing the mean square error between the external reward temporal difference objective and the external value estimate, and the individual potential function network is updated by a loss function that aligns the local individual potential reward with the external advantage direction. During the execution phase, each UAV independently selects cluster member actions based solely on its own local observations and action mask.

Images