Multi-agent reinforcement learning method for minimizing information age of uav swarm

Abstract

This invention discloses a method for minimizing the information age of UAV swarms using multi-agent reinforcement learning, comprising two stages: offline initialization and online reinforcement learning. In the offline stage, initial patrol paths are assigned to UAVs by calculating user service priorities, region clustering, and path planning. In the online stage, a heterogeneous graph integrating spatial and semantic relationships is constructed. Graph attention mechanisms with edge attributes and Transformer-encoded entity features are utilized, followed by multi-agent near-end policy optimization (MAPPO) for collaborative decision-making and real-time scheduling. This method can jointly optimize task scheduling and path planning, significantly reducing the weighted information age (by approximately 28.6%), improving path generation efficiency (by approximately 3.7 times), and increasing task coverage and response speed in high-user-density scenarios, adapting to complex dynamic environments.

Description

Technical Field

[0001] This invention belongs to the field of UAV swarm planning, and in particular relates to a method for optimizing the minimum age of UAV swarm information using multi-agent reinforcement learning. Background Technology

[0002] With the development of communication and computer technologies, next-generation wireless networks can be deployed on unmanned aerial vehicles (UAVs) for real-time data acquisition. Due to the flexibility, low maintenance costs, and comprehensive information acquisition capabilities of UAVs, wireless communication systems supporting UAVs have attracted increasing attention and are commonly used in scenarios with high real-time data requirements, such as military reconnaissance, urban emergency response, forest patrols, and environmental hazard avoidance. However, existing UAV communication systems often prioritize energy efficiency or communication quality when optimizing path planning and task allocation, neglecting data real-time performance. This is a significant drawback in UAV applications. To specifically measure the real-time performance of UAV data acquisition, researchers have proposed the concept of Information Age (AoI) to describe the time from UAV data acquisition to unloading and analysis. AoI quantifies the time elapsed since the target node last received the information and is generally preferred over latency in real-time scenarios.

[0003] The core problem of UAV swarm mission planning is minimizing the AOI of the UAV swarm, which is an NP-hard multi-constraint combinatorial optimization problem. Due to the limited energy and flight speed of UAVs, and the constraints on collision avoidance and communication power for information exchange within the swarm, balancing power distribution within the swarm to achieve multi-task cooperation while ensuring real-time data acquisition by the UAVs has become a major challenge in UAV swarm control algorithms. Therefore, achieving task cooperation within a UAV swarm under multiple environmental constraints places high demands on the algorithm.

[0004] Traditional algorithms for minimizing the Area of ​​Interest (AoI) of drone swarms are generally heuristic algorithms used to solve the Traveling Salesman Problem (TSP), such as Genetic Algorithms (GA) and Particle Swarm Optimization (PSO). GA, based on Darwinian evolution, simulates the "survival of the fittest" process and is particularly suitable for handling discrete combinatorial optimization problems. PSO, on the other hand, simulates the process by which individuals in a swarm cooperate to find the optimal solution, handling continuous optimization problems. However, traditional heuristic algorithms are only suitable for optimization problems in fixed, static environments and have poor adaptability to the dynamic environments of drone cooperation, making them unsuitable for practical applications.

[0005] Recent research has begun to leverage the powerful decision-making and environmental adaptability of deep learning and reinforcement learning to minimize the Area of ​​Interest (AoI) of drone swarms. However, commonly used basic deep learning algorithms such as transformers and CNNs have high computational resource requirements when processing drone datasets, making them difficult to deploy directly on drones for real-time control. Commonly used reinforcement learning algorithms such as DQN and DDPG do not perform well in multi-drone collaboration, especially due to the overly complex design of reward functions, unstable training processes, and poor interpretability.

[0006] Therefore, there is currently no complete algorithm for minimizing the AoI (Aspect of Interest) of drone swarms that can achieve task collaboration among multiple drones while minimizing computational resource requirements, thus ensuring a globally minimized AoI throughout the entire task. Reinforcement learning is a promising method in the field of drone collaboration, but ensuring the stability and real-time performance of the reinforcement learning training process is a pressing issue that needs to be addressed. Summary of the Invention

[0007] The technical problem this invention aims to solve is that while existing heuristic algorithms have a certain success rate in solving UAV path planning and task scheduling problems, these algorithms rely on fixed strategies and lack adaptability to dynamic environments. Most existing machine learning methods model sub-tasks such as path optimization, task scheduling, and power control in isolation, neglecting their global collaborative impact on system energy consumption and communication timeliness in time-series decision-making. Current reinforcement learning methods typically optimize paths based only on a single UAV environment or a single round of interaction, failing to fully model the spatial relationships, energy coupling, and dynamic evolution of task priorities among multiple UAVs, users, and data centers. This results in poor policy generalization ability, slow algorithm response speed, and decreased performance of multi-UAV collaboration. Therefore, this invention proposes a multi-agent reinforcement learning-based optimization method for minimizing the information age of UAV swarms.

[0008] To achieve the above-mentioned objectives, the present invention specifically adopts the following technical solution:

[0009] A method for minimizing the age of UAV swarm information using multi-agent reinforcement learning includes the following steps:

[0010] S1. In the offline initialization phase, first obtain three basic data of ground users: perception frequency, task criticality level, and information age, in order to calculate the service priority weight of each ground user.

[0011] S2. Clustering algorithms are used to cluster ground users according to their geographical location coordinates, thereby dividing ground users into multiple service areas, and the Hungarian algorithm is combined to assign an initial service area to each drone.

[0012] S3. Using a heuristic algorithm for solving the Traveling Salesman Problem, the access order of ground users within the service area of ​​each drone is optimized to obtain the initial patrol path that minimizes the total flight path, and the offline initialization phase ends.

[0013] S4. For each time step in the online reinforcement learning phase, a dynamic task generation method is used to filter the age and service status of ground users to obtain a set of tasks to be assigned; the feature vectors of ground users and their geographical locations are jointly encoded to obtain high-dimensional embedding features of ground users; the feature vectors of UAVs and their locations are jointly encoded to obtain high-dimensional embedding features of UAVs; a relative position encoding method is used to encode the spatial relationship between any two entities to obtain the edge attribute representation between the two entities, where the entity is either a ground user or a UAV.

[0014] S5. Construct a heterogeneous graph containing multiple types of edges based on the distance between entities. Use the scaled dot product attention method that fuses edge attributes to calculate the attention weight of each type of edge in the heterogeneous graph. Use the attention weighted aggregation method to sum the features of neighboring entities by attention weight to obtain the entity aggregation features that fuse multi-source information.

[0015] S6. The entity aggregation feature input is processed through multiple layers of Transformer. The final feature representation output by the last Transformer layer is integrated with the real-time state information of the UAV to obtain the complete state vector of the UAV. The feature-action transformation method is used to perform policy network mapping and masking on the complete state vector of the UAV to obtain the action probability distribution of task selection.

[0016] S7. When the drone is idle, the action probability distribution is randomly sampled. A multi-agent near-end strategy optimization algorithm is used to jointly train the actor network and the critic network on the sampled discrete actions. The trained actor network and critic network are used as prediction models. A greedy strategy is used on the action probability distribution. The sampled new discrete actions are input into the prediction model. The critic network predicts the value function, and the actor network outputs the action prediction probability for interaction with the environment and execution of user tasks.

[0017] Based on the above scheme, each step can be implemented in the following preferred manner.

[0018] Preferably, in step S1, the first The service priority weight for each ground user is calculated using the following formula:

[0019] ;

[0020] in, Indicates the first A ground user Service priority weights for time steps; Indicates the first A ground user The information about age in the time step; Indicates the first The sensing frequency of each ground user; For the first The mission criticality level of each ground user; , , Weighting coefficients for information age, task criticality level, and perception frequency; This represents the total number of ground users; For the first A ground user The information about age in the time step; The sum of weighted ages for all ground users; For the first The mission criticality level of each ground user; The sum of the weighted mission criticality levels for all ground users; For the first The sensing frequency of each ground user; The sum of weighted sensing frequencies for all ground users; This is the index of the current time step, and it should be a positive integer.

[0021] As a preferred embodiment, the specific process of step S2 is as follows:

[0022] S21. Using the K-means clustering algorithm, for Clustering is performed on the geographic coordinates of each ground user to obtain Each ground user cluster and the cluster center location of each ground user cluster Each ground user cluster constitutes a service area; Number of drones;

[0023] S22. For each UAV, calculate the Euclidean distance between its initial position and the position of each cluster center, and use the calculated distance as the distance cost between the UAV and the cluster center to construct a distance cost matrix.

[0024] S23. Using the Hungarian algorithm, the distance-cost matrix is ​​solved by optimal bisection matching to obtain a one-to-one allocation scheme between UAVs and cluster centers. The optimization objective is to minimize the total allocation distance.

[0025] ;

[0026] in, For matching matrix No. Line number Column elements are binary variables that take the values ​​0 or 1. The drones were allocated to the first When there is a cluster center, =1, otherwise =0; for The drone's initial position to the first Euclidean distance between the centers of each cluster; Indicates the index of the cluster; This indicates taking the minimum value;

[0027] S24. Based on the matching results, the first... Ground user set for initial drone services As its corresponding service area, it is used for initialization during the online operation phase.

[0028] Preferably, in step S3, path optimization must simultaneously satisfy: speed constraint Energy constraints and load constraints ;in, These represent the minimum and maximum flight speeds of the drone, respectively. For the first The drone in The flight speed of the time step; For the first Total energy consumption of a drone; This represents the maximum capacity of the drone's battery. For the first The number of ground users within the service area of ​​the drone; The maximum number of ground users that a single drone mission can serve. For the first The collection of ground users within the cluster served by the drone.

[0029] Preferably, in step S4, the first The set of tasks to be assigned at each time step The methods to obtain it are as follows:

[0030] ;

[0031] ;

[0032] in, Index for ground users; The preset age threshold for information; This represents the logical AND operation; For the set of ground users whose services have been completed; This represents the set union operation; For the first The set of tasks being executed at a given time step; Index for drones; For the first The drone in Ground user index for time-step selection service; For the first The drone service status at each time step. A value of 0 indicates an idle state. A value of 1 indicates that the service is currently serving ground users and the service is ongoing. Each time step, update upon completion. , This indicates the preset service duration in steps.

[0033] Preferably, in step S4, a multilayer perceptron coding method is used to obtain the first... Embedded features of a ground user and the Embedded features of drones :

[0034] ;

[0035] ;

[0036] in, , For feature dimensions; This refers to the multilayer sensor corresponding to the ground user. This refers to the multilayer sensor corresponding to the drone; This represents a vector concatenation operation; For the first Feature vectors of ground users; For the first The feature vector of the drone For the first The drone in Remaining battery level at the time step; For the first The geographic coordinates of a ground user; For the first The drone in The position coordinates of the time step;

[0037] For the The entity and the first The spatial relationship between two entities is determined using a multilayer perceptron. Encode to obtain edge attribute representation :

[0038] ;

[0039] in, and The first The entity and the first The location coordinates of each entity.

[0040] Preferably, in step S5, a heterogeneous graph containing four types of edges is constructed: when the distance between two drones does not exceed their communication range. At that time, an M2M edge is established, starting from the first drone and connecting to the second drone; when the distance between the drone and the ground user does not exceed their communication range. At the same time, establish an M2N edge originating from the UAV and connecting to the ground user, and an N2M edge originating from the ground user and connecting to the UAV; when the distance between the two ground users does not exceed their communication range. At that time, establish an N2N edge starting from the first ground user and connecting to the second ground user;

[0041] M2M edge-normalized attention weights Calculate using the following formula:

[0042] ;

[0043] ;

[0044] in, For the first The drone to the first The original attention score of the drone; This is the projection matrix for the first query; This is the first key projection matrix; This represents the attribute weight vector of the first side; For the first Embedded features of the drone; Indicates transpose; Indicates the first drones and the first The edge attribute representation of a drone; For indicator functions; For the first The drone in The position coordinates of the time step; For the first The drone in The position coordinates of the time step; Indicates L2 distance; It is an exponential function; To prevent division by zero of constants; For the first The drone in The position coordinates of the time step; For the first The drone to the first The original attention score of the drone;

[0045] M2N edge-normalized attention weights Calculate using the following formula:

[0046] ;

[0047] ;

[0048] in, For the first The drone to the first The original attention score of each ground user; This is the projection matrix for the second query; This is the projection matrix of the second bond; This represents the attribute weight vector on the second side; Indicates the first drones and the first The edge attribute representation of a ground user; For the first The geographic coordinates of a ground user; For the first The geographic coordinates of a ground user; For the first The drone to the first The original attention score of each ground user;

[0049] N2M edge-normalized attention weights Calculate using the following formula:

[0050] ;

[0051] ;

[0052] in, For the first The ground user for the first The original attention score of the drone; For the third query projection matrix; The third bond projection matrix; This represents the attribute weight vector of the third side; Indicates the first The ground user and the first The edge attribute representation of a drone; For the first The ground user for the first The original attention score of the drone;

[0053] N2N edge-normalized attention weights Calculate using the following formula:

[0054] ;

[0055] ;

[0056] in, For the first The ground user for the first The original attention score of each ground user; For the fourth query projection matrix; This is the projection matrix of the fourth bond; This represents the fourth side attribute weight vector; For the first Embedded features of a ground user; Indicates the first The ground user and the first The edge attribute representation of a ground user; For the first The geographic coordinates of a ground user; For the first The ground user for the first The original attention score of each ground user;

[0057] Entity aggregation features include drone aggregation features after fusing neighbor information. Features of aggregation with ground users :

[0058] ;

[0059] ;

[0060] in, , , , For the value projection matrix, the residual connection is implemented using addition operations.

[0061] Preferably, in step S6, the action probability distribution is obtained in the following way:

[0062] S61, calculate the output of the strategy network according to the following formula. The original action score of each ground user :

[0063] ;

[0064] in, This represents a multilayer perceptron containing two hidden layers (dimensions 256 and 128); For the first The drone in The complete state vector representation of a time step; For the first The final characteristic representation of the drone;

[0065] S62, calculate the following formula: Action score after applying a mask to a ground user The action scores after applying the mask are then normalized using softmax to obtain the action probability distribution. :

[0066] ;

[0067] ;

[0068] in, This is a normalized exponential function used to convert action scores into action probability distributions.

[0069] Preferably, in step S7, only when Action sampling is performed during the training phase, and random sampling is performed according to the following formula:

[0070] ;

[0071] in, This is a classification distribution used to sample discrete actions from the action probability distribution; Indicates the first Discrete actions obtained at each time step; a greedy strategy is used during the execution phase:

[0072] ;

[0073] in, This indicates that the index of the ground user that maximizes the function value should be returned. .

[0074] Preferably, in the multi-agent proximal policy optimization algorithm of step S7, each UAV is treated as an independent agent for distributed decision-making: state space The network is used for training the critic network; the action space is discrete, used for drone selection of ground users; the generalized advantage estimation (GAE) method is used to estimate the advantage function of the actor network.

[0075] ;

[0076] ;

[0077] ;

[0078] in, For the first Timing difference error at time steps; For the first Real-time reward signals at time steps are used to quantitatively assess the service quality of drones; Discount factor; The value function estimate output by the critic network; For the first Time step state space; For the first The advantage function estimate at each time step; This represents the total number of time steps in the current round. The summation index for time step offsets; The GAE balancing parameter; the negative sign in the instant reward signal indicates that the optimization objective is to minimize the weighted information age;

[0079] The actor network parameters are optimized and updated using a pruning strategy gradient method.

[0080] ;

[0081] ;

[0082] in, The loss function for the pruned strategy; Indicates taking the expected value; For strategy ratio; These are the trainable parameters of the current actor network; For the clipping function, Limited to the range Inside; These are the trimming parameters; Given a state in the current actor network Select action The probability of; Given a state in the policy network and actor network before parameter updates. Select action The probability of; The parameters of the actor network before the update;

[0083] The mean squared error loss method is used to optimize and update the parameters of the commentator network.

[0084] ;

[0085] ;

[0086] in, The mean squared error loss for the commentator network; For the first The target value is calculated based on actual returns at each time step. For the first The estimated value function output by the time-step critic network; No. Instant reward signals for time steps;

[0087] The total loss function of the joint training of the actor network and the critic network Defined as:

[0088] ;

[0089] in, and These are preset weighting coefficients; The policy entropy encourages agents to explore.

[0090] Compared with the prior art, the present invention has the following advantages:

[0091] This invention constructs a heterogeneous relationship graph based on spatial clustering and graph neural networks, and utilizes a multi-agent reinforcement learning framework to jointly optimize task scheduling and path planning. Experimental results show that, compared with traditional task scheduling schemes based on heuristic algorithms, this invention can reduce the weighted information age (AoI) by approximately 28.6% under the same UAV resource configuration, significantly improving the timeliness and freshness of tasks.

[0092] This invention introduces a multi-head self-attention mechanism and residual connection structure to replace the point-by-point optimization process in traditional stepwise path planning. It also combines the Transformer architecture to efficiently fuse multi-hop spatial relationships, reducing the path generation time from an average of 2.5 seconds to 0.68 seconds and improving inference efficiency by nearly 3.7 times, thus meeting the real-time scheduling requirements in highly dynamic scenarios.

[0093] This invention models the multi-layered spatial-semantic relationship between UAV and user, and introduces heterogeneous edge types and attention weights to distinguish different interaction relationships. It effectively adapts to the policy migration requirements under different user densities, task distributions and communication interference conditions. In simulation tests, the policy generalization accuracy was improved by 14.2%.

[0094] Compared to single-agent or centralized scheduling schemes, this invention employs the Multi-Agent Proximity Policy Optimization (MAPPO) method, combined with heterogeneous graph neural networks, to support UAVs in making collaborative decisions based on local information in dense task areas. Experiments demonstrate that in scenarios with a user density of 100+, task coverage is improved by 17.5%, and average task response latency is reduced by 21.3%. Attached Figure Description

[0095] Figure 1 This is a flowchart of the steps of the method of the present invention;

[0096] Figure 2 A schematic diagram of the multi-intelligence reinforcement learning-heterogeneous graph multi-head self-attention algorithm structure;

[0097] Figure 3 This is a diagram showing the AoI optimization results in this embodiment.

[0098] Figure 4 This is a graph showing the variation of AoI with energy in this embodiment;

[0099] Figure 5 This is a graph showing the variation of AoI with the number of drones in this embodiment;

[0100] Figure 6 This is a graph showing how AoI changes with the number of users. Detailed Implementation

[0101] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0102] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0103] This invention relates to a method for minimizing the information age of a UAV swarm using multi-agent reinforcement learning. The method aims to minimize the weighted information age (AoI) of the system's terminals, achieving path planning and task scheduling for the UAV swarm while ensuring energy constraints and task completion. In this invention, the UAV data acquisition system includes... drones and For each ground user, a two-stage architecture of "offline initialization + online reinforcement learning" is adopted. This represents the total number of ground users. This indicates the total number of drones.

[0104] like Figure 1 As shown, in a preferred embodiment of the present invention, the multi-agent reinforcement learning method for minimizing the age of UAV swarm information includes the following steps S1 to S7. The specific implementation process of each step will be described in detail below.

[0105] S1. In the offline initialization phase, first obtain three basic data points for ground users: perception frequency, task criticality level, and information age, in order to calculate the service priority weight for each ground user.

[0106] It should be noted that in step S1 of the present invention, the first The service priority weight of each ground user is calculated using the following formula, which serves as the basis for subsequent task scheduling and reward function design:

[0107]

[0108] in, Indicates the first A ground user The service priority weight for each time step has a value range of [0,1]. Indicates the first A ground user Information Age (AoI) at time step; Indicates the first The sensing frequency of each ground user; For the first The mission criticality level of each ground user; , , In this embodiment, the weighting coefficients for information age, task criticality level, and perception frequency are used. , , ; This represents the total number of ground users; For the first A ground user The information about age in the time step; The sum of weighted ages for all ground users; For the first The mission criticality level of each ground user; The sum of the weighted mission criticality levels for all ground users; For the first The sensing frequency of each ground user; The sum of weighted sensing frequencies for all ground users. This is the index of the current time step, and it should be a positive integer.

[0109] The above-mentioned service priority weight calculation method is designed by this invention. Its core innovation lies in the normalization process achieved by summing the denominator, which integrates three different indicators of information age (dynamic change), task criticality level (static attribute), and perception frequency (task requirement) into a unified range of [0,1], thereby performing weighted fusion.

[0110] In this embodiment, the aforementioned information age is an indicator representing the "freshness" of user data. It can be obtained by calculating the information age using the information age calculation method, which calculates the time difference of the most recent service time step for each ground user.

[0111]

[0112] in, Indicates the first The time step of the last time a ground user was served is set to 0 during initialization. The time step (unit: seconds) is used in this embodiment. .

[0113] In this embodiment, the aforementioned sensing frequency (unit: times / second) adopts a sensing frequency definition method, which calculates the frequency of data update requirements based on the ground user's preset data collection cycle:

[0114]

[0115] in, For the first A data collection cycle pre-set by each ground user.

[0116] In this embodiment, the above-mentioned task criticality level is a predefined static parameter: The larger the value, the more urgent the task.

[0117] S2. Clustering algorithms are used to cluster ground users according to their geographical location coordinates, thereby dividing ground users into multiple service areas, and the Hungarian algorithm is combined to assign an initial service area to each drone.

[0118] It should be noted that the specific process in step S2 of this invention is as follows:

[0119] S21. Using the K-means clustering algorithm, for Clustering is performed on the geographic coordinates of each ground user to obtain Each ground user cluster and the cluster center location of each ground user cluster Each ground user cluster constitutes a service area.

[0120] In this embodiment S21, the K-means clustering algorithm is an existing algorithm. This invention uses it for the initial division of the UAV service area. The geographic coordinates of the ground users are clustered as follows: The ground user clusters are as follows:

[0121]

[0122] in, This indicates taking the minimum value; This represents the set of ground user geographic location coordinates for all clusters. This represents the set of cluster center locations for all clusters; For the first The location of the cluster center; For the first A set of geographic location coordinates of ground users in a cluster; Indicates the index of the cluster; This represents the total number of clusters obtained after clustering. For the geographic coordinates of ground users; Represents the square of the L2 norm; It is the squared Euclidean distance from the geographic location coordinates of the ground user to the cluster center.

[0123] It should also be noted that in this embodiment S21, the K-means clustering algorithm can be replaced by DBSCAN or spectral clustering, and therefore no limitation is made in this invention.

[0124] S22. For each UAV, calculate the Euclidean distance between its initial position and the position of each cluster center. Use the calculated distance as the distance cost between the UAV and the cluster center to construct a distance cost matrix.

[0125] In this embodiment S22, the first The drone's initial position to the first Euclidean distance between the centers of each cluster That is, the above-mentioned number The drone and the first The distance cost between cluster centers is expressed as follows:

[0126]

[0127] in, Indicates the first The initial position of the drone; This represents the L2 norm (Euclidean distance). The value calculated from this is... This value is then used as an element in the distance cost matrix. By iterating through all the drones, the complete distance cost matrix can be obtained.

[0128] S23. Using the Hungarian Algorithm, the distance-cost matrix is ​​solved for optimal bisection matching to obtain a one-to-one allocation scheme between UAVs and cluster centers. The optimization objective is to minimize the total allocation distance.

[0129]

[0130] in, For matching matrix No. Line number Column elements are binary variables that take the values ​​0 or 1. The drones were allocated to the first When there is a cluster center, =1, otherwise =0; This represents the number of drones, and its value is the same as the total number of clusters.

[0131] In this embodiment S23, the Hungarian algorithm is an existing algorithm, which is used in this invention for optimal matching between UAVs and service areas. The constraint condition guarantees a one-to-one match: that is, each UAV serves only one cluster, and each cluster is uniquely served by a single UAV.

[0132] S24. Based on the matching results, the first... Ground user set for initial drone services As its corresponding service area, it is used for initialization during the online operation phase.

[0133] In this embodiment S24, the set of ground users served during the online operation phase is denoted as... Initial value of ground user set during online operation phase .

[0134] S3. Using a heuristic algorithm for solving the Traveling Salesman Problem (TSP), the order of visits to ground users within the service area of ​​each UAV is optimized to obtain the initial patrol path that minimizes the total flight path, thus ending the offline initialization phase.

[0135] In this embodiment S3, the Nearest Neighbor Rule is used as the heuristic algorithm to solve the initial patrol path of each UAV. Its implementation is based on existing technology, and this invention uses it for initial path planning within the cluster. Here, the selected Nearest Neighbor Rule can be replaced by a Genetic Algorithm (GA) or an Ant Colony Algorithm (ACO), and is not limited in this invention.

[0136] The following example uses the nearest neighbor rule to calculate the initial patrol path for each drone, minimizing the total flight path:

[0137]

[0138] in, Arrange the indexes according to the access order; For the first The collection of ground users within the cluster served by the drone; For the first The first drone visit The geographic coordinates of a ground user; For the first The first drone visit The geographic coordinates of a ground user; The Euclidean distance between two adjacent geographic coordinates;

[0139] It should be noted that in step S3 of this invention, path optimization must simultaneously satisfy: speed constraint. Energy constraints and load constraints ;in, These represent the minimum and maximum flight speeds of the drone, respectively. In this embodiment, meters per second meters per second, the drone's flight speed ranges between the aforementioned minimum and maximum flight speeds; For the first The drone in Flight speed at the time step (unit: m / s); For the first Total energy consumption of a drone (unit: joules); The maximum capacity of the drone battery (unit: joules). For the first The number of ground users within the service area of ​​the drone; This refers to the maximum number of ground users that a drone can serve in a single mission, i.e., the drone's maximum payload.

[0140] In this embodiment, the total energy consumption of the UAV during flight consists of three parts: flight, hovering, and communication.

[0141]

[0142] in, For the first The cumulative flight time of the drone (in seconds); For the first The cumulative hovering time of the drone (in seconds); For the first The cumulative communication time of the drone (in seconds); For flight power, For hovering power, In this embodiment, for communication power, , , All units are in watts.

[0143] In this embodiment, the maximum number of ground users that a drone can serve in a single mission is limited by its battery capacity:

[0144]

[0145] in, Average energy consumption (in joules) for a single ground user serving a drone. This is the floor function.

[0146] After the above S1~S3 processes, the offline initialization phase can be considered complete. In this phase, ground user service priority weight initialization needs to be performed in S1: calculating information age, sensing frequency, and task criticality level; K-means spatial clustering and Hungarian algorithm matching are performed in S2; and TSP nearest neighbor heuristic solution is performed in S3 to achieve initial path planning.

[0147] S4. For each time step in the online reinforcement learning phase, a dynamic task generation method is used to filter the age and service status of ground users to obtain a set of tasks to be assigned; the feature vectors of ground users and their geographical locations are jointly encoded to obtain high-dimensional embedding features of ground users; the feature vectors of UAVs and their locations are jointly encoded to obtain high-dimensional embedding features of UAVs; a relative position encoding method is used to encode the spatial relationship between any two entities to obtain the edge attribute representation between the two entities, where the entity is either a ground user or a UAV.

[0148] It should be noted that in S4 of this invention, the first... The set of tasks to be assigned at each time step The methods to obtain it are as follows:

[0149]

[0150]

[0151] in, Index for ground users; The preset age threshold for information; This represents the logical AND operation; For the set of ground users whose services have been completed; This represents the set union operation; For the first The set of tasks being executed at a given time step; Index for drones; For the first The drone in Ground user index for time-step selection service; For the first The drone service status at each time step. A value of 0 indicates an idle state. A value of 1 indicates that the service is currently serving ground users and the service is ongoing. Each time step, update upon completion. , This indicates the preset service duration in steps. In this embodiment, Set to 10 seconds. Set to 2.

[0152] like Figure 2 As shown, Figure 2 The "task pool" mentioned above refers to the set of tasks to be assigned. Figure 2 The “task generator” in this context refers to the process of obtaining the set of tasks to be assigned mentioned above. This is the design of the present invention, and its core innovation lies in combining information age threshold filtering and a task protection mechanism to avoid duplicate task assignment.

[0153] It should be noted that in S4 of this invention, the multilayer perceptron (MLP) encoding method is used to obtain the first... Embedded features of a ground user and the Embedded features of drones :

[0154]

[0155]

[0156] in, , In this embodiment, the feature dimension is used as the feature dimension. ; This refers to the multilayer sensor corresponding to the ground user. This refers to the multilayer sensor corresponding to the drone; This represents a vector concatenation operation; For the first Feature vectors of ground users; For the first The feature vector of the drone For the first The drone in Remaining charge at the time step (unit: joules); For the first The geographic coordinates of a ground user; For the first The drone in The position coordinates of the time step.

[0157] In this embodiment, MLP is an existing neural network structure. Each MLP contains two hidden layers (128 dimensions) and the activation function is ReLU. Based on the above MLP, the joint encoding of feature vectors and position coordinates can be achieved to realize information fusion.

[0158] It should be noted that in S4 of this invention, for the first... The entity and the first The spatial relationship between two entities is determined using a multilayer perceptron. Encode to obtain edge attribute representation :

[0159]

[0160] in, and The first The entity and the first The location coordinates of each entity.

[0161] In this embodiment, following the above process, the multilayer perceptron... It includes a hidden layer (64 dimensions) and can establish edge attribute representations between drones and ground users, between drones, and between ground users.

[0162] S5. Construct a heterogeneous graph containing multiple types of edges based on the distance between entities. Use the scaling dot product attention method that fuses edge attributes to calculate the attention weight of each type of edge in the heterogeneous graph. Use the attention weighted aggregation method to sum the features of neighboring entities by attention weighting to obtain the entity aggregation features that fuse multi-source information.

[0163] It should be noted that in S5 of this invention, a heterogeneous graph containing four types of edges is constructed: when the distance between two drones does not exceed their communication range. At that time, an M2M edge is established, starting from the first drone and connecting to the second drone; when the distance between the drone and the ground user does not exceed their communication range. At the same time, establish an M2N edge originating from the UAV and connecting to the ground user, and an N2M edge originating from the ground user and connecting to the UAV; when the distance between the two ground users does not exceed their communication range. At that time, an N2N edge is established, starting from the first ground user and connecting to the second ground user.

[0164] In this embodiment, the communication range between the drone and the ground user is The communication range of drones is meters. The communication range between ground users is meters. rice.

[0165] It should be noted that in S5 of this invention, the attention weights after M2M edge normalization are... Calculate using the following formula:

[0166]

[0167]

[0168] in, For the first The drone to the first The original attention score of the drone; This is the projection matrix for the first query; This is the first key projection matrix; This represents the attribute weight vector of the first side; For the first Embedded features of the drone; Indicates transpose; Indicates the first drones and the first The edge attribute representation of a drone; For indicator functions; For the first The drone in The position coordinates of the time step; For the first The drone in The position coordinates of the time step; Indicates L2 distance; It is an exponential function; To prevent division by zero of constants; For the first The drone in The position coordinates of the time step; For the first The drone to the first The original attention score of the drone;

[0169] M2N edge-normalized attention weights Calculate using the following formula:

[0170]

[0171]

[0172] in, For the first The drone to the first The original attention score of each ground user; This is the projection matrix for the second query; This is the projection matrix of the second bond; This represents the attribute weight vector on the second side; Indicates the first drones and the first The edge attribute representation of a ground user; For the first The geographic coordinates of a ground user; For the first The geographic coordinates of a ground user; For the first The drone to the first The original attention score of each ground user;

[0173] N2M edge-normalized attention weights Calculate using the following formula:

[0174]

[0175]

[0176] in, For the first The ground user for the first The original attention score of the drone; For the third query projection matrix; The third bond projection matrix; This represents the attribute weight vector of the third side; Indicates the first The ground user and the first The edge attribute representation of a drone; For the first The ground user for the first The original attention score of the drone;

[0177] N2N edge-normalized attention weights Calculate using the following formula:

[0178]

[0179]

[0180] in, For the first The ground user for the first The original attention score of each ground user; For the fourth query projection matrix; This is the projection matrix of the fourth bond; This represents the fourth side attribute weight vector; For the first Embedded features of a ground user; Indicates the first The ground user and the first The edge attribute representation of a ground user; For the first The geographic coordinates of a ground user; For the first The ground user for the first The raw attention score of each ground user.

[0181] In this embodiment, the attention weight of the ground user to the UAV (corresponding to the N2M edge) is calculated in a way that is symmetrical to the attention weight of the UAV to the ground user (corresponding to the M2N edge). The specific calculation process will not be described here. This heterogeneous graph attention mechanism that integrates edge attributes is designed by this invention. The core innovations are: (1) modeling multiple interaction relationships through edges of different types; (2) incorporating spatial relative position encoding into attention calculation; and (3) ensuring that the effective attention is calculated only within the communication range through a distance mask. In the above attention weight calculation, the projection matrix is ​​queried. Key projection matrix Its dimensions are The dimension of the edge attribute weight vector is To prevent small constants from being divided by zero .

[0182] It should be noted that in S5 of this invention, the entity aggregation feature includes the UAV aggregation feature after fusing neighbor information. Features of aggregation with ground users :

[0183]

[0184]

[0185] in, , , , For the value projection matrix, the residual connection is implemented using addition operations.

[0186] S6. The entity aggregation feature input is processed through multiple layers of Transformer. The final feature representation output by the last Transformer layer is integrated with the real-time state information of the UAV to obtain the complete state vector of the UAV. The feature-action transformation method is used to perform policy network mapping and masking on the complete state vector of the UAV to obtain the action probability distribution of task selection.

[0187] It should be noted that, in S6 of this invention, for the first... The Transformer layer's internal processing flow is as follows:

[0188]

[0189] in, For the first The layer's feature representation (including features of all drones and ground users); For the first Layer feature representation; This is a layer normalization operation used to stabilize the training process; This represents the heterogeneous graph attention calculation process defined in S5; For layer index, The number of Transformer layers; the initial input of the Transformer is... and Multi-layer Transformer stacking is a prior art technique. This invention combines it with a heterogeneous graph attention mechanism to obtain a final feature representation that fuses multi-hop information. .

[0190] It should be noted that in S6 of this invention, the action probability distribution is obtained as follows:

[0191] S61, calculate the output of the strategy network according to the following formula. The original action score of each ground user :

[0192]

[0193] in, This represents a multilayer perceptron containing two hidden layers (dimensions 256 and 128); For the first The drone in The complete state vector representation of a time step; For the first The final characteristic representation of the drone;

[0194] S62, calculate the following formula: Action score after applying a mask to a ground user The action scores after applying the mask are then normalized using softmax to obtain the action probability distribution. :

[0195]

[0196]

[0197] in, This is a normalized exponential function used to convert action scores into action probability distributions.

[0198] S7. When the drone is idle, the action probability distribution is randomly sampled. A multi-agent near-end strategy optimization algorithm is used to jointly train the actor network and the critic network on the sampled discrete actions. The trained actor network and critic network are used as prediction models. A greedy strategy is used on the action probability distribution. The sampled new discrete actions are input into the prediction model. The critic network predicts the value function, and the actor network outputs the action prediction probability for interaction with the environment and execution of user tasks.

[0199] It should be noted that in S7 of this invention, only when Action sampling is performed during the training phase, and random sampling is performed according to the following formula:

[0200]

[0201] in, This is a classification distribution used to sample discrete actions from the action probability distribution; Indicates the first Discrete actions obtained at each time step; a greedy strategy is used during the execution phase:

[0202]

[0203] in, This indicates that the function returns the index n of the ground user that maximizes the function value.

[0204] In the multi-agent proximal policy optimization algorithm, each drone is treated as an independent agent for distributed decision-making: state space The network is used for training the critic network; the action space is discrete, used for drone selection of ground users; the generalized advantage estimation (GAE) method is used to estimate the advantage function of the actor network.

[0205]

[0206]

[0207]

[0208] in, For the first Timing difference (TD) error at time steps; For the first The instantaneous reward signal (scalar) of the time step is used to quantitatively evaluate the service quality of the drone; Discount factor; The value function estimate output by the critic network; For the first Time step state space; For the first The advantage function estimate at each time step; This represents the total number of time steps in the current round. The summation index for time step offsets; The GAE balancing parameter; the negative sign in the instant reward signal indicates that the optimization objective is to minimize the weighted information age;

[0209] The Clipped Policy Gradient (PPO) method is used to optimize and update the actor network parameters.

[0210]

[0211]

[0212] in, The loss function for the pruned strategy; Indicates taking the expected value; This indicates taking the minimum value; For strategy ratio; These are the trainable parameters of the current actor network; For the clipping function, the variable Limited to the range Inside; These are the trimming parameters; Given a state in the current actor network Select action The probability of; Given a state in the policy network and actor network before parameter updates. Select action The probability of; The parameters of the actor network before the update;

[0213] The mean squared error loss method is used to optimize and update the parameters of the commentator network.

[0214]

[0215]

[0216] in, The mean squared error loss for the commentator network; For the first The target value is calculated based on actual returns at each time step. For the first The estimated value function output by the time-step critic network; No. Instant reward signals for time steps;

[0217] The total loss function of the joint training of the actor network and the critic network Defined as:

[0218]

[0219] in, and These are preset weighting coefficients; The policy entropy encourages agents to explore.

[0220] In this embodiment, by sampling discrete actions from the action probability distribution, this task selection masking mechanism can set the probability of invalid actions to zero, avoiding suboptimal cross-cluster decisions and duplicate task allocation. The above parameters are set to... , , , , GAE, MAPPO, and PPO are existing technologies; this invention applies them to multi-agent collaborative decision-making in UAV swarms. The instant reward signal is a feature of this invention, ensuring that the service needs of high-priority ground users are met first.

[0221] To better demonstrate the specific implementation and technical effects of the present invention, the multi-agent reinforcement learning method for minimizing the age of UAV swarm information shown in steps S1 to S7 of the above preferred implementation is applied to a specific example.

[0222] Example

[0223] The specific implementation process of the multi-agent reinforcement learning-based drone swarm information age minimization optimization method used in this embodiment is as described above and will not be repeated here.

[0224] This embodiment is based on a self-constructed heterogeneous UAV mission scheduling simulation dataset. This dataset simulates the diverse needs of user terminals for data perception and mission response in multiple urban areas, including multi-dimensional attributes such as user spatial distribution, perception frequency, mission criticality, UAV energy constraints, flight speed, and mission load. The aim is to evaluate the comprehensive performance of this invention in optimizing weighted information age (AoI) under multi-objective constraints.

[0225] The constructed dataset contains a total of 30 user nodes and 5 heterogeneous UAVs. User task requirement attributes are randomly generated according to a Beta distribution. UAV parameters, including flight energy consumption, task execution time, and communication power range, are set based on the actual hardware platform. The spatial layout is a two-dimensional geographic coordinate system (1000m × 1000m). A certain proportion of high-critical users are randomly selected in each simulation scenario to form task hotspot areas, increasing the complexity of scheduling and robustness testing requirements. The simulation environment is implemented using the Python + PyTorch framework, with the data divided into training, validation, and test sets using a ratio of 63% / 27% / 10%.

[0226] To objectively evaluate the performance of the method of this invention on this simulation platform, this embodiment was compared with the following four mainstream baseline algorithms: nearest neighbor path planning + round-robin task allocation (NN-Polling); region clustering + priority scheduling (KMeans-Priority); static task mapping + DQN reinforcement learning path planning (DQN-Schedule); and centralized reinforcement learning model with joint scheduling optimization (Centralized-PPO).

[0227] Each method was run independently 10 times under the same initial conditions, and the following key metrics were evaluated under different task densities (200, 500, and 1000 task nodes): the mean and standard deviation of the system weighted information age (AoI), the average task completion latency (in seconds), the average energy consumption per UAV (in J), and the system task coverage (number of completed tasks / total number of tasks).

[0228] The experimental results obtained are as follows Figure 3 , Figure 4 , Figure 5 , Figure 6As shown, the results demonstrate that the optimization method of this invention has significant advantages compared to other baseline algorithms. Experimental results show that the proposed method achieves optimal or near-optimal results in all scenarios, especially under high task density (1000 task nodes), where its AoI index is reduced by 28.6% compared to the second-best solution, average task response latency is shortened by 21.3%, task coverage is increased by 17.5%, and average UAV energy consumption is reduced by 12.7%. Furthermore, because the method of this invention adopts an end-to-end training paradigm, the number of training rounds is significantly less than that of policy models based on DQN and centralized PPO, and the average convergence rounds are reduced by 35%, exhibiting stronger deployability and scalability.

[0229] In summary, the embodiments of the present invention fully verify the scheduling efficiency, path optimization capability and robustness of the proposed method in complex task scenarios, and show a significant performance advantage over existing methods in the weighted AoI optimization problem.

[0230] It is understood that the multi-agent reinforcement learning method for minimizing the age of UAV swarm information described in S1-S7 above can essentially be implemented by a computer program. Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer program product corresponding to the multi-agent reinforcement learning method for minimizing the age of UAV swarm information provided in the above embodiments. This product includes a computer program / instruction, which, when executed by a processor, can implement the multi-agent reinforcement learning method for minimizing the age of UAV swarm information as described in the above embodiments.

[0231] Similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the multi-agent reinforcement learning method for minimizing the age of UAV swarm information provided in the above embodiments, which includes a memory and a processor;

[0232] The memory is used to store computer programs;

[0233] The processor is configured to, when executing the computer program, implement the multi-agent reinforcement learning method for minimizing the age of UAV swarm information in the above embodiments.

[0234] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0235] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the multi-agent reinforcement learning method for minimizing the age of UAV swarm information provided in the above embodiments. The storage medium stores a computer program, which, when executed by a processor, can implement the multi-agent reinforcement learning method for minimizing the age of UAV swarm information in the above embodiments.

[0236] Specifically, in the computer-readable storage medium of the above three embodiments, the stored computer program is executed by a processor, which can perform the aforementioned steps S1 to S7.

[0237] It is understood that the aforementioned storage media may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Furthermore, the storage media may also be various media capable of storing program code, such as USB flash drives, external hard drives, magnetic disks, or optical discs.

[0238] It is understood that the processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0239] It should also be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the embodiments provided in this application, the division of steps or modules in the system and method is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or steps may be combined or integrated together, and a module or step may also be split.

[0240] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A method for optimizing the minimum age of UAV swarm information using multi-agent reinforcement learning, characterized in that, Includes the following steps: S1. In the offline initialization phase, first obtain three basic data of ground users: perception frequency, task criticality level, and information age, in order to calculate the service priority weight of each ground user. S2. Clustering algorithms are used to cluster ground users according to their geographical location coordinates, thereby dividing ground users into multiple service areas, and the Hungarian algorithm is combined to assign an initial service area to each drone. S3. Using a heuristic algorithm for solving the Traveling Salesman Problem, the access order of ground users within the service area of ​​each drone is optimized to obtain the initial patrol path that minimizes the total flight path, and the offline initialization phase ends. S4. For each time step in the online reinforcement learning phase, a dynamic task generation method is adopted to filter the age and service status of ground users to obtain a set of tasks to be assigned; the feature vectors of ground users and their geographical locations are jointly encoded to obtain high-dimensional embedding features of ground users; the feature vectors of UAVs and their locations are jointly encoded to obtain high-dimensional embedding features of UAVs. The spatial relationship between any two entities is encoded using a relative position encoding method, resulting in an edge attribute representation between the two entities, where the entity is a ground user or a drone. S5. Construct a heterogeneous graph containing multiple types of edges based on the distance between entities. Use the scaled dot product attention method that fuses edge attributes to calculate the attention weight of each type of edge in the heterogeneous graph. Use the attention weighted aggregation method to sum the features of neighboring entities by attention weight to obtain the entity aggregation features that fuse multi-source information. S6. The entity aggregation feature input is processed through multiple layers of Transformer. The final feature representation output by the last Transformer layer is integrated with the real-time state information of the UAV to obtain the complete state vector of the UAV. A feature-action transformation method is used to perform policy network mapping and masking on the complete state vector of the UAV to obtain the action probability distribution for task selection. S7. When the drone is idle, the action probability distribution is randomly sampled. A multi-agent near-end strategy optimization algorithm is used to jointly train the actor network and the critic network on the sampled discrete actions. The trained actor network and critic network are used as prediction models. A greedy strategy is used on the action probability distribution. The sampled new discrete actions are input into the prediction model. The critic network predicts the value function, and the actor network outputs the action prediction probability for interaction with the environment and execution of user tasks.

2. The method for minimizing the age of UAV swarm information using multi-agent reinforcement learning as described in claim 1, characterized in that, In step S1, the first The service priority weight for each ground user is calculated using the following formula: ； in, Indicates the first A ground user Service priority weights for time steps; Indicates the first A ground user The information about age in the time step; Indicates the first The sensing frequency of each ground user; For the first The mission criticality level of each ground user; , , Weighting coefficients for information age, task criticality level, and perception frequency; This represents the total number of ground users; For the first A ground user The information about age in the time step; The sum of weighted ages for all ground users; For the first The mission criticality level of each ground user; The sum of the weighted mission criticality levels for all ground users; For the first The sensing frequency of each ground user; The sum of weighted sensing frequencies for all ground users; This is the index of the current time step, and it should be a positive integer.

3. The method for minimizing the age of UAV swarm information using multi-agent reinforcement learning as described in claim 2, characterized in that, The specific process of step S2 is as follows: S21. Using the K-means clustering algorithm, for Clustering is performed on the geographic coordinates of each ground user to obtain Each ground user cluster and the cluster center location of each ground user cluster Each ground user cluster constitutes a service area; Number of drones; S22. For each UAV, calculate the Euclidean distance between its initial position and the position of each cluster center, and use the calculated distance as the distance cost between the UAV and the cluster center to construct a distance cost matrix. S23. Using the Hungarian algorithm, the distance-cost matrix is ​​solved by optimal bisection matching to obtain a one-to-one allocation scheme between UAVs and cluster centers. The optimization objective is to minimize the total allocation distance. ； in, For matching matrix No. Line number Column elements are binary variables that take the values ​​0 or 1. The drones were allocated to the first When there is a cluster center, =1, otherwise =0; for The drone's initial position to the first Euclidean distance between the centers of each cluster; Indicates the index of the cluster; This indicates taking the minimum value; S24. Based on the matching results, the first... Ground user set for initial drone services As its corresponding service area, it is used for initialization during the online operation phase.

4. The method for minimizing the age of UAV swarm information using multi-agent reinforcement learning as described in claim 3, characterized in that, In step S3, path optimization must simultaneously satisfy: speed constraint Energy constraints and load constraints ;in, These represent the minimum and maximum flight speeds of the drone, respectively. For the first The drone in The flight speed of the time step; For the first Total energy consumption of a drone; This represents the maximum capacity of the drone's battery. For the first The number of ground users within the service area of ​​the drone; The maximum number of ground users that a single drone mission can serve. For the first The collection of ground users within the cluster served by the drone.

5. The method for minimizing the age of UAV swarm information using multi-agent reinforcement learning as described in claim 4, characterized in that, In step S4, the first The set of tasks to be assigned at each time step The methods to obtain it are as follows: ； ； in, Index for ground users; The preset age threshold for information; This represents the logical AND operation; For the set of ground users whose services have been completed; This represents the set union operation; For the first The set of tasks being executed at a given time step; Index for drones; For the first The drone in Ground user index for time-step selection service; For the first The drone service status at each time step. A value of 0 indicates an idle state. A value of 1 indicates that the service is currently serving ground users and the service is ongoing. Each time step, update upon completion. , This indicates the preset service duration in steps.

6. The method for minimizing the age of UAV swarm information using multi-agent reinforcement learning as described in claim 5, characterized in that, In step S4, the multilayer perceptron coding method is used to obtain the first... Embedded features of a ground user and the Embedded features of drones : ； ； in, , For feature dimensions; This refers to the multilayer sensor corresponding to the ground user. This refers to the multilayer sensor corresponding to the drone; This represents a vector concatenation operation; For the first Feature vectors of ground users; For the first The feature vector of the drone For the first The drone in Remaining battery level at the time step; For the first The geographic coordinates of a ground user; For the first The drone in The position coordinates of the time step; For the The entity and the first The spatial relationship between two entities is determined using a multilayer perceptron. Encode to obtain edge attribute representation : ； in, and The first The entity and the first The location coordinates of each entity.

7. The method for minimizing the age of UAV swarm information using multi-agent reinforcement learning as described in claim 6, characterized in that, In step S5, a heterogeneous graph containing four types of edges is constructed: when the distance between two drones does not exceed their communication range. At that time, an M2M edge is established, starting from the first drone and connecting to the second drone; when the distance between the drone and the ground user does not exceed their communication range. At the same time, establish M2N edges that originate from the drone and connect to ground users, and N2M edges that originate from the ground users and connect to the drone; When the distance between two ground users does not exceed their communication range At that time, establish an N2N edge starting from the first ground user and connecting to the second ground user; M2M edge-normalized attention weights Calculate using the following formula: ； ； in, For the first The drone to the first The original attention score of the drone; This is the projection matrix for the first query; This is the first key projection matrix; This represents the attribute weight vector of the first side; For the first Embedded features of the drone; Indicates transpose; Indicates the first drones and the first The edge attribute representation of a drone; For indicator functions; For the first The drone in The position coordinates of the time step; For the first The drone in The position coordinates of the time step; Indicates L2 distance; It is an exponential function; To prevent division by zero of constants; For the first The drone in The position coordinates of the time step; For the first The drone to the first The original attention score of the drone; M2N edge-normalized attention weights Calculate using the following formula: ； ； in, For the first The drone to the first The original attention score of each ground user; This is the projection matrix for the second query; This is the projection matrix of the second bond; This represents the attribute weight vector on the second side; Indicates the first drones and the first The edge attribute representation of a ground user; For the first The geographic coordinates of a ground user; For the first The geographic coordinates of a ground user; For the first The drone to the first The original attention score of each ground user; N2M edge-normalized attention weights Calculate using the following formula: ； ； in, For the first The ground user for the first The original attention score of the drone; For the third query projection matrix; The third bond projection matrix; This represents the attribute weight vector of the third side; Indicates the first The ground user and the first The edge attribute representation of a drone; For the first The ground user for the first The original attention score of the drone; N2N edge-normalized attention weights Calculate using the following formula: ； ； in, For the first The ground user for the first The original attention score of each ground user; For the fourth query projection matrix; This is the projection matrix of the fourth bond; This represents the fourth side attribute weight vector; For the first Embedded features of a ground user; Indicates the first The ground user and the first The edge attribute representation of a ground user; For the first The geographic coordinates of a ground user; For the first The ground user for the first The original attention score of each ground user; Entity aggregation features include drone aggregation features after fusing neighbor information. Features of aggregation with ground users : ； ； in, , , , For the value projection matrix, the residual connection is implemented using addition operations.

8. The method for minimizing the age of UAV swarm information using multi-agent reinforcement learning as described in claim 7, characterized in that, In step S6, the action probability distribution is obtained as follows: S61, calculate the output of the strategy network according to the following formula. The original action score of each ground user : ； in, This represents a multilayer perceptron containing two hidden layers (dimensions 256 and 128); For the first The drone in The complete state vector representation of a time step; For the first The final characteristic representation of the drone; S62, calculate the following formula: Action score after applying a mask to a ground user The action scores after applying the mask are then normalized using softmax to obtain the action probability distribution. : ； ； in, This is a normalized exponential function used to convert action scores into action probability distributions.

9. The method for minimizing the age of UAV swarm information using multi-agent reinforcement learning as described in claim 8, characterized in that, In step S7, only when Action sampling is performed during the training phase, and random sampling is performed according to the following formula: ； in, This is a classification distribution used to sample discrete actions from the action probability distribution; Indicates the first Discrete actions obtained at each time step; a greedy strategy is used during the execution phase: ； in, This indicates that the index of the ground user that maximizes the function value should be returned. .

10. The method for minimizing the age of UAV swarm information using multi-agent reinforcement learning as described in claim 9, characterized in that, In the multi-agent proximal policy optimization algorithm in step S7, each UAV is treated as an independent agent for distributed decision-making: state space The network is used for training the critic network; the action space is discrete, used for drone selection of ground users; the generalized advantage estimation (GAE) method is used to estimate the advantage function of the actor network. ； ； ； in, For the first Timing difference error at time steps; For the first Real-time reward signals at time steps are used to quantitatively assess the service quality of drones; Discount factor; The value function estimate output by the critic network; For the first Time step state space; For the first The advantage function estimate at each time step; This represents the total number of time steps in the current round. The summation index for time step offsets; The GAE balancing parameter; the negative sign in the instant reward signal indicates that the optimization objective is to minimize the weighted information age; The actor network parameters are optimized and updated using a pruning strategy gradient method. ； ； in, The loss function for the pruned strategy; Indicates taking the expected value; For strategy ratio; These are the trainable parameters of the current actor network; For the clipping function, Limited to the range Inside; These are the trimming parameters; Given a state in the current actor network Select action The probability of; Given a state in the policy network and actor network before parameter updates. Select action The probability of; The parameters of the actor network before the update; The mean squared error loss method is used to optimize and update the parameters of the commentator network. ； ； in, The mean squared error loss for the commentator network; For the first The target value is calculated based on actual returns at each time step. For the first The estimated value function output by the time-step critic network; No. Instant reward signals for time steps; The total loss function of the joint training of the actor network and the critic network Defined as: ； in, and These are preset weighting coefficients; The policy entropy encourages agents to explore.

Images