Multi-objective joint optimization method for multi-uav cooperative data collection under random traffic

Abstract

The application discloses a multi-target joint optimization method for cooperative data collection of multiple unmanned aerial vehicles under random traffic, establishes data generation and AoI models, communication and energy consumption models and a multi-target optimization problem model, and maps into a multi-agent multi-target Markov decision process; an MA-MP3O algorithm is proposed, a plurality of preference individuals are generated in the whole three-dimensional weight space through parallel preference learning, each individual contains a centralized critic network and an actor network of each unmanned aerial vehicle; the critic network receives a global state and branches into value heads of three targets, and after PopArt adaptive normalization and preference weighting, the comprehensive advantages are calculated; the actor network focuses on high-priority sensor nodes by using a multi-head attention mechanism, and outputs a mixed action of continuous flight action and discrete mode switching. The application realizes the cooperative optimization and flexible trade-off of three targets of total AoI, total packet loss number and total energy consumption in a random traffic environment.

Description

Technical Field

[0001] This invention relates to the field of data collection technology for multi-UAV-assisted wireless sensor networks, and in particular to a multi-objective joint optimization method for collaborative data collection of multiple UAVs under random traffic conditions. Background Technology

[0002] Sensor nodes are typically deployed widely within monitoring areas in a low-power, low-cost manner. However, their computing power, storage space, and communication range are all limited by physical size and battery capacity. When a node detects a random event (such as a sudden change in environmental parameters or the appearance of an abnormal signal), it generates a timestamped data packet. However, the timing and frequency of such events exhibit significant uncertainty and non-uniformity in time and space, causing network data traffic to display typical random burst characteristics.

[0003] UAVs, with their high mobility and strong environmental adaptability, can overcome the geographical limitations of terrestrial networks, quickly reach target areas, and establish short-range communication links with sensor nodes, thus significantly improving data collection efficiency. However, introducing UAVs into WSN data collection tasks also brings new technical challenges. First, node-generated data is random, requiring UAVs to perceive the queue status and data freshness of each node in real time without prior information, and dynamically plan flight paths and mission modes. Second, sensor nodes generally use buffer units with limited capacity. Once a sudden surge of data or prolonged lack of collection services occurs, the buffer is prone to overflow, causing data packets to be directly discarded. This packet loss not only causes permanent loss of monitoring information but also disrupts the continuity and integrity of the data. At the same time, the UAV's own onboard battery energy is very limited, and data collection and transmission must be completed within the specified mission time. To reduce node AoI and minimize packet loss, UAVs often need to frequently perform long-distance, high-speed flights, which significantly increases energy consumption; conversely, adopting a conservative energy-saving flight strategy may lead to long-term data backlog, continuous AoI aging, and even buffer overflow packet loss.

[0004] It is evident that in UAV-assisted WSN data collection tasks under random traffic conditions, there is an inherent conflict and coupling relationship between the three performance indicators: node AoI, packet loss count, and UAV flight energy consumption.

[0005] Although existing research has explored AoT and energy consumption optimization methods for UAV-assisted WSN data collection to reduce node AoI and UAV flight energy consumption, research on optimization methods for joint node packet loss in multi-UAV data collection under random traffic still faces many challenges, mainly including: 1) Existing methods are mostly based on static or deterministic flow assumptions, making it difficult to adapt to the randomness and suddenness of sensor data generation. In actual monitoring, the timing, frequency, and spatial distribution of events are significantly uncertain, and the length of node buffer queues fluctuates dynamically. Traditional convex optimization, graph search, and metaheuristic algorithms all assume that the system state can be accurately modeled, lacking online adaptive capabilities. While existing deep reinforcement learning methods can handle some dynamism, most are trained under fixed data generation patterns. Once flow characteristics change, the performance of these methods deteriorates significantly, making it difficult to maintain stable and efficient data collection in real-world random flow environments.

[0006] 2) Existing research generally transforms multiple objectives into a single objective through fixed weighted summation, neglecting the inherent competitive relationships between different optimization objectives and the differences in their learning difficulty. For example, reducing UAV flight energy consumption can be achieved directly through a relatively conservative flight strategy, but reducing node AoI requires complex spatiotemporal coordination and mode switching decisions, while suppressing packet loss relies more on timely responses to sudden traffic surges. These three objectives conflict with each other, and their convergence speed and gradient stability differ significantly. Traditional fixed-preference scalarization methods make the optimization process easily dominated by the simple objective, leading to severe degradation of the performance of other objectives, poor Pareto front coverage, and difficulty in meeting the flexible trade-off needs under different user preferences. Summary of the Invention

[0007] The purpose of this invention is to improve and innovate upon the shortcomings and problems existing in the background technology, and to provide a multi-objective joint optimization method for collaborative data collection of multiple UAVs under random traffic conditions.

[0008] A multi-objective joint optimization method for collaborative data collection by multiple UAVs under random traffic conditions includes the following steps: Step S1: Method begins; Step S2: Initialize UAV information, sensor information, and ground base station information; Step S3: Establish a data generation and AoI model and a communication and energy consumption model based on UAV information, sensor information and ground base station information. Based on the data generation and AoI model and the communication and energy consumption model, construct a multi-objective optimization problem model that jointly minimizes the total AoI of the node, the total number of packet losses and the total energy consumption of multiple UAVs. Step S4: Map the multi-objective joint optimization problem corresponding to the multi-objective optimization problem model that jointly minimizes the total AoI of the node, the total number of packet losses, and the total energy consumption of multiple UAVs to a multi-agent multi-objective Markov decision process; Step S5: Propose the MA-MP3O algorithm to solve the multi-objective joint optimization problem. The MA-MP3O algorithm initializes multiple preference individuals, each of which is pre-assigned a set of weight vectors for total AoI, total packet loss, and total energy consumption. Each individual contains a centralized critic network and an actor network for each UAV. Based on the output values ​​of the three objectives—total AoI, total packet loss, and total energy consumption—from the critic network, and based on the individual's weight vectors and values ​​for total AoI, total packet loss, and total energy consumption, a loss function is constructed to update the parameters of the critic network and the actor network to obtain the critic network and actor network for a fixed number of training rounds. Step S6: Based on the actor network obtained from fixed training rounds, output the total AoI, total number of packet losses, and total energy consumption of multiple UAVs for all corresponding sensor nodes. Sort the output total AoI, total number of packet losses, and total energy consumption of multiple UAVs, and output a diverse set of strategies covering the three-dimensional Pareto front.

[0009] Preferably, step S3, which establishes the data generation and AoI model based on UAV information, sensor information, and ground base station information, specifically includes: Define nodes In the time slot Does a data packet generate? ; This indicates the generation of a data packet. This indicates that it was not generated; Define nodes In the time slot Did packet loss occur? ; The calculation formula is as follows: ; In the formula, For nodes In the time slot The number of cached data packets; This indicates the maximum cache capacity for each node; The formula for calculating the total number of packet losses across all nodes during the entire task time is: ; In the formula, This represents the total number of nodes. Indicates the task time for UAV data collection; The node AoI is defined as the time difference between the current timeslot and the time when the latest data packet was generated by that node. Calculation, where For nodes In the time slot AoI, Represents a node In the time slot The latest data packet was generated at the time of its creation; The total AoI of all sensor nodes throughout the entire task time is calculated using the following formula: .

[0010] Preferably, step S3, which establishes a communication and energy consumption model based on UAV information, sensor information, and ground base station information, specifically includes: Get Nodes i With the k The probability of a Loss of Space (LoS) link during UAV communication is calculated using the following formula: ; In the formula and It is a constant related to the environment. It is a node i With the k Pitch angle between UAVs; node i With the k The probability of using an NLoS link when communicating with a UAV ; Get Nodes i With the k Path loss between UAVs The calculation formula is as follows: ; In the formula At the speed of light, Indicates the carrier frequency. This is the path loss index. and These are the average additional path loss for LoS and NLoS links, respectively. For time slots t node i With the k The Euclidean distance between UAVs; Get Nodes With the Average path loss between UAVs The calculation formula is as follows: ; In the formula, Represents a node i With the k Loss of the LoS link between UAVs; Represents a node i With the k NLoS link path loss between UAVs; Get Nodes With the UAV communication rate The calculation formula is as follows: ; In the formula, For system channel bandwidth, It is a node The transmission power, This is the additive white Gaussian noise power at the UAV receiver; Set time slot With nodes The set of UAVs that meet the communication conditions is: ; In the formula, Represents the set of UAVs. Indicates the data collection radius of the UAV. Indicates the size of a data packet. Indicates the first The mission mode of the UAV. Indicates the first The UAV is in uplink mode for data collection; Then it means the first The UAV is in data return mode; when , i.e., slot There exists at least one capable of interacting with the node. When a UAV establishes a communication link; Choose the UAV with the highest communication rate to establish communication, that is: ; In the formula For nodes In the time slot Associated UAV.

[0011] Preferably, step S3, which establishes the communication and energy consumption model based on UAV information, sensor information, and ground base station information, further includes: Get the UAV propulsion power The calculation formula is as follows: ; In the formula Indicates time slot t No. The speed of UAV deployment The linear velocity at the tip of the rotor blade. Indicates the induction velocity, and These are constants related to UAV induced power and hovering power, respectively. Represents air density, For rotor realism, and These are the fuselage drag coefficient and the rotor disk area, respectively. Dividing the task time into discrete time slots of fixed length 1 second, the total task time... The total energy consumption of all UAVs inside is: .

[0012] Preferably, the step of constructing a multi-objective optimization problem model that jointly minimizes the total node AoI, total packet loss, and total energy consumption of multiple UAVs based on the data generation and AoI model and the communication and energy consumption model specifically includes: Let the first The flight angle and maneuver selected by the UAV are ; Let the first Flight distance of a UAV in a single discrete time slot ;in, Indicates the minimum moving distance; Indicates the maximum distance traveled; set up Indicates the flight maneuvers of the UAV. Let the task mode selection action of the UAV be represented. Then, the multi-objective joint optimization problem corresponding to the multi-objective optimization problem model that jointly minimizes the total AoI of the node, the total number of packet losses, and the total energy consumption of multiple UAVs is: ; In the formula The flight area of ​​the UAV is [size missing] ; Indicates time slot t No. The X-axis position coordinates of the UAV in a three-dimensional Cartesian coordinate system; Indicates time slot t No. The Y-axis position coordinates of the UAV in a three-dimensional Cartesian coordinate system; Indicates time slot t No. The coordinates of the UAV's location; Indicates except the first The position coordinates of other UAVs besides the one mounted on top of the UAV. Indicates a safe distance; Indicates the first The UAV has remaining energy; Indicates the maximum onboard energy; Indicates the first Is the UAV a node? In the time slot Associated UAV; when the Build UAV as nodes In the time slot Associated UAVs The value is 1 if it is set to 1, otherwise the value is 0. Represents the UAV set, Represents a set of nodes. Represents a set of discrete time slots; Indicates that the UAV set excluding the first Other UAVs besides the standard UAV.

[0013] Preferably, step S4 specifically includes: Constructing time slots t Global state space: ; In the formula Indicates time slot t Location of each UAV Represents the set of positions of all nodes. Represents a node Location; For time slots t The remaining energy of each UAV; For each node's AoI set, For time slots t A collection of various UAV task modes This represents the set of ages of the oldest data packets in each node's local cache queue. This represents the set of ages of the oldest data packets in the onboard cache of each UAV. Indicates time slot t The total number of data packets in the cache of each UAV Indicates time slot t No. k The total number of data packets in the UAV buffer; Indicates the time slot of each node The set of cached data packets, and Indicates time slot t The number of packet losses across all nodes, via calculate; Among them, nodes In the time slot Age of the oldest data packet In the formula, Represents a node In the time slot The first in local cache The generation time of each data packet; No.k UAV in time slot Age of the oldest data packet in the onboard cache ; In the formula, Indicates the first k UAV in time slot The first in the onboard cache The generation time of each data packet; Constructing time slots t No. k Build UAV local state space ; In the formula, time slot t No. Downlink communication rate between UAV and base station ; It is the transmission power of the UAV. Noise power at the base station; For base station and the k Path loss between UAVs ; This represents the Euclidean distance between the UAV and the base station; Indicates the first k frame Azimuth angle between UAV and ground base station; Indicates the first k The set of relative positions between a UAV and other UAVs; Indicates the first k The age of the oldest data packet in the UAV's onboard buffer; and That is, time slot t No. k The total number of data packets in the UAV buffer; The sensor feature context vector, obtained by aggregation through a multi-head attention module, is calculated as follows: ; This represents a multi-head attention mechanism; among which, the first k UAV query vector Due to its position Obtained by linear transformation; and These represent all nodes in the time slot. t The key matrix and value matrix are derived from the eigenvectors of all sensor nodes. Obtained by linear transformation; in, ; In the formula, Represents a node With the k The European distance for UAVs; Represents a node With the k frame Horizontal azimuth of the UAV; Represents a node With the k frame UAV communication rate; Constructing Action Space ; Constructing a reward function ; In the formula, For time slots t Node information, age, AoI-related rewards, It is a reward for avoiding packet loss at nodes, and Energy-related rewards; in, ; In the formula, The average AoI of all nodes in time slot t is used as the AoI backlog penalty; Update rewards for AoI; pass Calculated; The average AoI of all nodes in time slot t+1; Indicates time slot t The average age of the oldest data packets in the local cache queues of all nodes; and These are non-negative weighting coefficients in the range (0,1); in, ; Indicates time slot t No. The number of data packets collected by the UAV; and These are non-negative weighting coefficients ranging from (0,1); in, ; This indicates that penalties will be imposed on the energy-consuming behavior of UAVs; pass Calculated; Indicates the first The remaining energy of the UAV in time slot t; Indicates the first The remaining energy of the UAV in time slot t+1; Indicates the warning distance; Similarly, the weighting coefficients are non-negative and range from (0,1). The soft partition penalty is calculated using the following formula: ; In the formula Indicates that the first The areas of responsibility assigned to the UAVs Indicates the first Whether the UAV is within its own area of ​​responsibility, when the first When the UAV is not within its own area of ​​responsibility, the indicator function is 1; when the first... When the UAV is within its designated area of ​​responsibility, the indicator function is 0, while For time slots t No. The signed distance from the current position of the UAV to the predefined boundary of the region.

[0014] Preferably, in step S5, the values ​​of the three objectives—total AoI, total packet loss, and total energy consumption—output by the critic network are used to update the parameters of the critic network and the actor network by constructing a loss function based on the weight vectors and values ​​of the total AoI, total packet loss, and total energy consumption of each individual, so as to obtain the critic network and actor network with fixed training epochs. Specifically, this includes: (1) Action selection and execution: Initialize the experience replay buffers of multiple preference individuals; at the beginning of each time slot, the first... k Local state space of a UAV The input is fed into its actor network, which outputs the probability distribution of mission mode switching actions and the normal distribution of flight actions; the UAV selects and executes specific actions based on the output probability distribution and normal distribution. (2) Experience collection and storage: After all UAVs perform actions, the global state space is updated to the global state space of the next time slot. ; Calculate the current time slot based on changes in the global state space t The reward function, and the experience tuple Store in the experience replay buffer of the corresponding preference individual; (3) Advantage estimation, value output and loss calculation: Once the experience replay buffer has accumulated data, a batch of experience data is randomly sampled from the buffer; the global state space is then... Input the critic network, output the normalized values ​​of the three objectives—total AoI, total packet loss, and total energy consumption—in the current global state, and then inverse normalize them to obtain the true values ​​of the three objectives. The advantage functions of the three objectives are calculated based on their true values ​​and reward functions. Based on the pre-assigned preference weight vector for each individual, the advantage functions of the three objectives are weighted and fused to obtain the comprehensive advantage. ; The actor network is updated using the PPO trimming mechanism based on its overall advantages; The returns for the three objectives are calculated based on the advantage functions and true values ​​of the three objectives; the returns for the three objectives are then normalized; and a weighted mean squared error loss function is constructed based on the normalized returns and values. (4) Parameter update: The gradient descent algorithm is used to update the actor network parameters and critic network parameters corresponding to each preference individual to obtain the critic network and actor network with fixed training rounds.

[0015] Preferably, the advantage function calculation formulas for the three objectives are as follows: ; ; ;in; This represents the overall AoI advantage function. The advantage function represents the total number of lost packets; Represents the total energy consumption advantage function; the time-series difference error of total AoI. Timing difference error of total packet loss Time-series difference error of total energy consumption ; Indicates the discount factor; These are GAE coefficients, used to balance bias and variance; The true value of total AoI in time slot t; This represents the true value of the total number of packets lost in time slot t. This represents the true value of the total energy consumption in time slot t. The formulas for calculating the rewards for the three objectives—total AoI, total packet loss, and total energy consumption—are as follows: , and ; The weighted mean squared error loss of the critic network is calculated using the following formula: ; In the formula, This represents the weighted mean square error loss; This represents the total AoI component in the weight vector. This represents the total number of lost packets component in the weight vector. This represents the total energy consumption component in the weight vector. This represents the true value of the total AoI after normalization. This represents the normalized total AoI reward. This represents the true value of the total number of lost packets after normalization. This represents the total number of lost packets after normalization. The normalized total energy consumption is the true value. This represents the normalized total energy return.

[0016] Preferably, step S6 specifically includes: Step S61: Based on the actor networks obtained from different fixed training rounds and different individuals, output multiple strategy solutions that include the total AoI, total number of packet losses, and total energy consumption of multiple UAVs for all sensor nodes. Step S62: Remove the dominant policy solutions from the policy solutions and retain the non-dominated policy solutions. Sort the retained non-dominated policy solutions independently in multiple dimensions according to the three objectives of total AoI, total number of packet losses and total energy consumption. Step S63: Calculate the crowding degree of each sorted non-dominated strategy solution; Step S64: Reorder the solutions of each non-dominated strategy according to the degree of congestion; Step S65: Retain the non-dominated strategy solutions with higher crowding rankings until the preset population size is met, and eliminate the non-dominated strategy solutions with lower rankings. Step S66: Determine whether the number of training iterations has reached the preset number of training iterations; if yes, use the non-dominated strategy solutions that meet the preset population size as the diversified strategy set covering the three-dimensional Pareto front; if no, proceed to step S61.

[0017] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention describes the generation of random traffic data in wireless sensor networks as a random process with known model parameters but unknown parameters. Each sensor node randomly generates data packets in each time slot using a time-varying and spatially non-uniform Bernoulli distribution. The local backlog of the sensor node is added to the state space to reflect the traffic status, and a multi-head attention mechanism is used to promptly address data backlog caused by sudden traffic surges. The MA-MP3O algorithm employs parallel preference learning, that is, uniformly sampling and generating multiple preference individuals in the entire three-dimensional target weight space. Each individual corresponds to a set of preference weight vectors for total AoI, total packet loss, and total energy consumption. Each individual contains a centralized commentator network and... KEach UAV has its own actor network. A centralized commentator receives the global state and branches into three independent value heads to estimate the value of total AoI, total packet loss, and total energy consumption, respectively. Then, generalized advantage estimation (GAE) is used to calculate the advantage functions of the three objectives, and they are weighted and fused into a comprehensive advantage according to individual preference weights. The actor network outputs a hybrid action containing continuous flight actions and discrete mode switching based on the local state of each UAV and the sensor features aggregated by multi-head attention. The relevant features of each sensor node are linearly transformed to obtain keys and values, and the UAV position changes to obtain queries. The multi-head attention module calculates to give higher attention weights to specific nodes, such as nodes that are about to overflow or whose data is aging. The final output sensor feature context vector is a weighted aggregation of information from all nodes, with higher priority nodes having larger feature weights. Focusing on high-priority nodes guides the network to plan flight collection strategies and branches into three action heads to avoid the curse of action dimensionality. In addition, PopArt normalization technology is introduced to adaptively adjust the magnitude difference of the rewards of each objective, and a diverse set of strategies covering the three-dimensional Pareto front is generated through parallel preference learning. During the training phase, a centralized commentator evaluates the multi-objective value based on the global state. After PopArt normalization and preference weighting, the policy gradient is generated, guiding the actor and commentator networks to update synchronously. During the execution phase, each UAV relies only on its local state. After attention aggregation, it independently outputs the mean action without triggering gradient updates. It is only used to test policy performance and filter non-dominated solutions to maintain the Pareto front. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the multi-objective joint optimization method for multi-UAV collaborative data collection under random traffic conditions provided in this embodiment of the invention; Figure 2 This is a multi-UAV-assisted WSN data collection system model provided in the embodiments of the present invention; Figure 3 This is a comparison chart of the effectiveness analysis of different algorithms provided in the embodiments of the present invention. Detailed Implementation

[0020] To make the objectives, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0022] like Figure 1 As shown, this invention provides a multi-objective joint optimization method for collaborative data collection by multiple UAVs under random traffic conditions, which consists of six steps: Step S1: Method start; Step S2: Initialize UAV information, sensor information, and ground base station information; wherein the UAV information includes the UAV flight area, UAV data collection task time, UAV flight altitude and flight start coordinates, UAV maximum flight speed and onboard maximum energy, UAV data collection radius, and safe distance between UAVs; sensor information includes the location coordinates and buffer capacity of each sensor node; ground base station information includes the location coordinates of the ground base station. Step S3: Establish a data generation and AoI model and a communication and energy consumption model based on the UAV information, sensor information, and ground base station information, and construct a multi-objective optimization problem model that jointly minimizes the total AoI of nodes, the total number of packet losses, and the total energy consumption of multiple UAVs based on the data generation and AoI model and the communication and energy consumption model. Step S4: Map the multi-objective joint optimization problem model corresponding to the joint minimization of total AoI, total packet loss, and total energy consumption of multiple UAVs to a multi-agent multi-objective Markov decision process; Step S5: Propose the MA-MP3O algorithm to solve the multi-objective joint optimization problem; wherein the MA-MP3O algorithm initializes multiple preference individuals, each individual is pre-assigned a set of weight vectors for total AoI, total packet loss, and total energy consumption, and each individual contains a centralized critic network and each UAV's own actor network; based on the output of the critic network, the total AoI, total packet loss, and total energy consumption are calculated; The values ​​of three objectives—packet loss, total energy consumption, and total AoI—are evaluated. A loss function is constructed based on the weight vectors and values ​​of the total AoI, total packet loss, and total energy consumption of each individual, and the parameters of the critic network and actor network are updated to obtain the critic network and actor network with fixed training epochs. Step S6: Based on the actor network obtained after fixed training epochs, the total AoI, total packet loss, and total energy consumption of multiple UAVs for all corresponding sensor nodes are output. The output is sorted according to the total AoI, total packet loss, and total energy consumption of multiple UAVs, and a diverse policy set covering the three-dimensional Pareto front is output. The following sections provide a detailed description of these six steps: Step S1: Method begins; Step S2: Initialize UAV information, sensor information, and ground base station information; For the multi-objective joint optimization method for multi-UAV cooperative data collection designed in this invention, it is necessary to assign values ​​to relevant parameters. For example... Figure 2 As shown, input in a three-dimensional Cartesian coordinate system The flight area of ​​the UAV is [size missing] A ground base station serving as a data processing center is deployed at the edge of this area; its location coordinates are shown below. The time required for data collection from multiple UAVs is denoted as... To simplify the processing, it is divided into a set of discrete time slots with a fixed length of 1 second, denoted as . .

[0023] UAV assembly It indicates that the mission will be conducted at a fixed altitude. Flight, the starting point of the flight is .make Indicates time slot t No. The coordinates of each UAV's location. The maximum flight speed of each UAV is... The maximum onboard energy is The data collection radius is The safe distance between UAVs is and Time slot t No. The location of the UAV is Then the first , k The relative positions of the two UAVs can be calculated as follows: .

[0024] The entire data collection task area exists There are randomly distributed ground sensor nodes, and the nodes The position is Each node generates a value of [size missing] when it detects an event. And the data packets include timestamps. The maximum cache capacity for each node is... Packet loss occurs when the cache is full and new data continues to be generated, causing the node to delete the oldest generated packet.

[0025] Step S3: Establish a data generation and AoI model and a communication and energy consumption model based on UAV information, sensor information and ground base station information. Based on the data generation and AoI model and the communication and energy consumption model, construct a multi-objective optimization problem model that jointly minimizes the total AoI of the node, the total number of packet losses and the total energy consumption of multiple UAVs. Specifically, the data generation and AoI model based on UAV information, sensor information, and ground base station information includes: Let the monitoring radius of each sensor node be... When an event occurs randomly and uniformly throughout the region, the average probability of a node generating data can be determined by the ratio of the monitored area to the region area, calculated as follows: ; Based on this probability, the data generation behavior of a sensor node within a single time slot can be described by the parameter being... The Bernoulli distribution. To better reflect the spatiotemporal nonuniformity of data generation in real-world scenarios, the average generation probability of the data is... Distinguishing by nodes, and expanding to probabilities that change over time, i.e.: ; in For nodes In the time slot The probability of generating data; Represents a node The set of probabilities that generate data.

[0026] It should be noted that UAVs cannot obtain the data generation probability of each node when performing collection tasks; they can only perceive environmental changes by observing real-time information such as the status of node cache queues. Therefore, this information will not be directly used in subsequent algorithm processes. Rather, it is reflected indirectly in the data generation of each node.

[0027] Define nodes In the time slot Does a data packet generate? ; This indicates the generation of a data packet. This indicates that it has not been generated.

[0028] Because sensor node buffer space is limited, when the buffer is full and new data is generated, the node will discard the oldest generated data packet according to the First-In-First-Out (FIFO) principle. Therefore, the node is defined... In the time slot Did packet loss occur? ; The calculation formula is as follows: ; In the formula, For nodes In the time slot The number of cached data packets; This indicates the maximum cache capacity for each node; The formula for calculating the total number of packet losses across all nodes during the entire task time is: ; In the formula, This represents the total number of nodes. Indicates the task time for UAV data collection; The node information age (AoI) is defined as the time difference between the current timeslot and the time when the node's latest data packet was generated. Calculation, where For nodes In the time slot AoI, Represents a node In the time slot The latest data packet was generated at the time of its creation; The total AoI of all sensor nodes throughout the entire task time is calculated using the following formula: .

[0029] Specifically, establishing a communication and energy consumption model based on UAV information, sensor information, and ground base station information includes: Get Nodes i With the k The probability of a Loss of Space (LoS) link during UAV communication is calculated using the following formula: ; In the formula and This is an environment-dependent constant and needs to be set according to the actual deployment scenario. It is a node i With the k The pitch angle between the UAVs is determined by their relative positions in the air, and the calculation formula is: ; In the formula For time slots t node i and No. k The Euclidean distance between UAVs. The closer the horizontal distance between the UAV and the node, the larger the pitch angle, and the higher the probability of a Loss-of-Sight (LoS) link; conversely, when the UAV is far from the node, Non-LoS (NLoS) links dominate. NLoS link probability. .

[0030] Get Nodes i With the k Path loss between UAVs The calculation formula is as follows: ; In the formula At the speed of light, Indicates the carrier frequency. This is the path loss index. and These are the average additional path loss for LoS and NLoS links, respectively.

[0031] Considering that the channel state may change dynamically during UAV flight, the node acquisition... With the Average path loss between UAVs The calculation formula is as follows: ; In the formula, Represents a node i With the k Loss of the LoS link between UAVs; Represents a node i With the k NLoS link path loss between UAVs.

[0032] During the data backhaul phase, due to the high position of the base station antenna and the relatively open environment, only the LoS (LoS) is considered for the communication link between the UAV and the base station, and the path loss calculation method is the same as that in the data collection phase. Based on the above path loss model, time slots can be further calculated. t ,node With the The communication rate of the UAV is: ; In the formula For system channel bandwidth, It is a node The transmission power, This is the additive white Gaussian noise power at the UAV receiver.

[0033] Each UAV can only serve one ground node in the same time slot. To avoid conflicts caused by multiple UAVs choosing to serve the same node, it is stipulated that... When a UAV is in collection mode, it selects data to be collected and has a communication rate within its coverage area. , This refers to the size of a data packet, i.e., the number of serviceable nodes whose communication rate between the UAV and the node is greater than one data packet. The system then selects the node with the highest communication rate for data collection. Specifically, it sets time slots... With nodes The set of UAVs that meet the communication conditions is: ; In the formula Indicates the first The mission mode of the UAV. Indicates the first The UAV is in uplink mode for data collection, at which point the... The UAV establishes links with sensor nodes within its communication coverage area and uploads data packets from the node queue to the onboard buffer. Then it means the first The UAV is in data backhaul mode, establishing a link with the base station and transmitting data packets from its onboard buffer back to the base station. , i.e., slot There exists at least one capable of interacting with the node. When establishing a communication link among UAVs, the UAV with the highest communication rate is selected to establish communication, that is: ; In the formula For nodes In the time slot Associated UAVs. During the data backhaul phase of communication between multiple UAVs and the base station, it is assumed that the base station is equipped with a multi-antenna array, which can allocate an independent beam to each UAV. Therefore, interference between multiple UAVs during the data backhaul process is not considered.

[0034] Furthermore, since the communication energy consumption in UAV data collection tasks is much smaller than the propulsion energy consumption, this invention ignores communication energy consumption and only considers the impact of propulsion energy consumption on the UAV's endurance. Therefore, the... The propulsion power of a UAV can be calculated as follows: ; In the formula Indicates time slot t No. The speed of UAV deployment The linear velocity at the tip of the rotor blade. Indicates the induction velocity, and These are constants related to UAV induced power and hovering power, respectively. Represents air density, For rotor realism, and These are the fuselage drag coefficient and the rotor disk area, respectively.

[0035] Since the task time is divided into time slots of fixed length of 1 second, the total task time... The total energy consumption of all UAVs inside is: ; Specifically, the steps described above, based on the data generation and AoI model and the communication and energy consumption model, construct a multi-objective optimization problem model that jointly minimizes the total node AoI, total packet loss, and total energy consumption of multiple UAVs, specifically including: Let the first The selectable flight angles and maneuvers of the UAV are as follows The counter-clockwise rotation with due east as the reference direction determines the spatial movement trend of the UAV; the flight distance is assumed to be a continuous variable. , Indicates the first The maximum horizontal distance traveled by a UAV within a single discrete time slot is affected by the maximum flight speed of the UAV. Constrained by a time slot length of 1 second, i.e. This invention does not consider the hovering state of the drone and sets a minimum moving distance. The objective of this invention is to minimize task time. Total AoI and total packet loss of all sensor nodes Total energy consumption of all UAVs .set up Indicates the flight maneuvers of the UAV. Let the task mode selection action of the UAV be represented. Then, the multi-objective joint optimization problem corresponding to the multi-objective optimization problem model that jointly minimizes the total AoI of the node, the total number of packet losses, and the total energy consumption of multiple UAVs is: ; In the formula and Two constraints limit the range of flight angles and distances that each UAV can choose when performing a mission; and This requires that UAVs not fly outside the mission area; Different UAVs are required to maintain a safe distance from each other when performing tasks; This means that the flight energy consumption of each UAV during the entire data collection mission does not exceed the given maximum onboard energy. The maximum number of data packets that a node can buffer is limited. Ensure that a single sensor node can only establish a connection with one UAV within the same time slot. Indicates except the first The position coordinates of other UAVs besides the one mounted on top of the UAV. Indicates the first The UAV has remaining energy; Indicates the maximum onboard energy; Indicates the first Is the UAV a node? In the time slot Associated UAV; when the Build UAV as nodes In the time slot The associated UAV is set to 1 if it is not associated with a 0 otherwise.

[0036] Step S4: Map the multi-objective joint optimization problem corresponding to the multi-objective optimization problem model that jointly minimizes the total AoI of the node, the total number of packet losses, and the total energy consumption of multiple UAVs to a multi-agent multi-objective Markov decision process; Specifically, the optimization problem (P1) is mapped to a multi-agent, multi-objective Markov decision process (MA-MOMDP). The specific mapping process is as follows: Constructing time slots t Global state space ; In the formula Indicates time slot t Location of each UAV Represents the set of positions of all nodes. Represents a node Location; For time slots t The remaining energy of each UAV; For each node's AoI set, Indicates time slot t A collection of task modes for each UAV. This represents the set of ages of the oldest data packets in each node's local cache queue. This represents the set of ages of the oldest data packets in the onboard cache of each UAV. Indicates time slot t The total number of data packets in the cache of each UAV Indicates time slot t No. k The total number of data packets in the UAV buffer; Indicates the time slot of each node The set of cached data packets, and Indicates time slot t Total packet loss across all nodes, via calculate.

[0037] Among them, nodes In the time slot Age of the oldest data packet In the formula, Represents a node In the time slot The first in local cache The generation time of each data packet; No. k UAV in time slot Age of the oldest data packet in the onboard cache ; In the formula, Indicates the first k UAV in time slot The first in the onboard cache The generation time of each data packet.

[0038] The global state space fully characterizes the system's overall situation, providing the centralized commentator network with all the information needed to evaluate the value of joint actions. The local state space, on the other hand, is used by each UAV to make decisions during the distributed execution phase.

[0039] Constructing time slots t No. k Build UAV local state space ; In the formula, time slot t No. Downlink communication rate between UAV and base station ; It is the transmission power of the UAV. Noise power at the base station; For base station and the k Path loss between UAVs ; This represents the Euclidean distance between the UAV and the base station; ; Indicates the first k frame Azimuth angle between UAV and ground base station; ; Indicates the first k The set of relative positions between a UAV and other UAVs; Indicates the first k The age of the oldest data packet in the UAV's onboard buffer; and That is, time slot t No. k The total number of data packets in the UAV buffer; The sensor feature context vector, obtained by aggregation through a multi-head attention module, is calculated as follows: ; in, This represents the multi-head attention mechanism; the first k UAV query vector Due to its position Obtained by linear transformation; and These represent all nodes in the time slot. t The key matrix and value matrix are derived from the eigenvectors of all sensor nodes. Obtained through linear transformation; in, ; In the formula, Represents a node With the kThe European distance for UAVs; Represents a node With the k frame Horizontal azimuth between UAVs; Represents a node With the k frame Communication rate between UAVs.

[0040] For example, in practical applications, the first k UAV query vector Due to its position After linear transformation by the fully connected layer, the following is obtained: ; Let the multi-head attention embedding dimension be . In the formula It is a dimension of The learnable weight matrix This is the bias vector.

[0041] The feature vector of each node Mapped to via fully connected layer dimension: ; In the formula yes A learnable weight matrix of dimension 1. yes A bias vector of dimension. It is a node In the time slot For the The encoded feature vector of the UAV is derived from the original sensor features. Mapped to via fully connected layer The dimensional embedding space is obtained. This encoding step is fundamental for generating keys and values, ensuring that all node features are in the same representation space before entering the multi-head attention mechanism. Then, key and value vectors are generated separately through two independent unbiased linear layers: ; ; In the formula and All Learnable weight matrix; then all M The key matrix is ​​obtained by stacking the key vectors and value vectors of each node row by row. Sum matrix .

[0042] It should be noted that, due to the context vector Its dimensions are fixed. This allows the number of input nodes to be variable while the output dimension remains constant. It can compress any number of node features into a fixed-length embedding vector, significantly reducing the dimensionality of the state space of the subsequent network, thereby alleviating the curse of dimensionality caused by high-dimensional state spaces.

[0043] Constructing Action Space ; In the formula and They represent time slots respectively. t No. k frame The actions of a UAV in selecting its flight angle and flight distance This is an action to switch task modes.

[0044] Constructing a reward function ; In the formula, For time slots t Node information, age, AoI-related rewards, It is a reward for avoiding packet loss at nodes, and Energy-related rewards; in, ; In the formula, The average AoI of all nodes in time slot t is used as the AoI backlog penalty; Update rewards for AoI; pass Calculated; For all nodes in the time slot t +1 average AoI; Indicates time slot t The average age of the oldest data packets in the local cache queues of all nodes; and It is a non-negative weighting coefficient in the range (0,1); used to balance the importance between backlog penalty and node local maximum age penalty, ensuring that after the UAV collects data to reduce the local age penalty, it promptly transmits the airborne data back to the base station to reduce the backlog penalty.

[0045] in, ; Indicates the time slot t. The number of data packets collected by the UAV; and The weight coefficient is a non-negative value in the range (0,1) to ensure that the UAV selects the node with the most severe backlog for collection.

[0046] in, ; This indicates that penalties will be imposed on the energy-consuming behavior of UAVs; pass Calculated; Indicates the first UAV in time slot t +1 of remaining energy; considering that it is difficult to meet hard constraints during training, a value greater than the safety distance is set. Warning distance When the distance between UAVs is less than this value, a continuous penalty is applied to incentivize the UAVs to actively avoid dangerous areas when approaching them; where... Similarly, the non-negative weighting coefficients, which take values ​​in the range (0,1), control the importance of energy consumption penalty and safe distance penalty respectively. The soft partition penalty is calculated using the following formula: .

[0047] In the formula Indicates that the first The areas of responsibility assigned to the UAVs Indicates the first Whether the UAV is within its own area of ​​responsibility, when the first When the UAV is not within its own area of ​​responsibility, the indicator function is 1; when the first... When the UAV is within its designated area of ​​responsibility, the indicator function is 0, while For time slots t No. The signed distance from the current position of the UAV to the boundary of the pre-defined responsibility area is used to distinguish deviations and impose continuous penalties. The purpose of this penalty is to guide the UAV to work in designated areas during the initial training phase, quickly learn stable local policies, suppress free-riding behavior that may occur during multi-agent training, and also prevent multiple UAVs from concentrating in the same area, reducing the risk of collisions and data collection conflicts. The coefficient of this penalty term... The performance gradually decreases with the number of training rounds, allowing UAVs to collaborate across regions in the later stages to cope with sudden traffic surges or uneven data generation between regions, thus achieving a smooth transition from regional isolation learning to global collaborative optimization.

[0048] Step S5: Propose the MA-MP3O algorithm to solve the multi-objective joint optimization problem. The MA-MP3O algorithm initializes multiple preference individuals, each of which is pre-assigned a set of weight vectors for total AoI, total packet loss, and total energy consumption. Each individual contains a centralized critic network and an actor network for each UAV. Based on the output values ​​of the three objectives—total AoI, total packet loss, and total energy consumption—from the critic network, and based on the individual's weight vectors and values ​​for total AoI, total packet loss, and total energy consumption, a loss function is constructed to update the parameters of the critic network and the actor network to obtain the critic network and actor network for a fixed number of training rounds. It should be noted that traditional multi-agent, multi-objective reinforcement learning methods face challenges when dealing with multi-objective optimization problems in UAV data collection tasks under random traffic, including difficulties in credit allocation, coupling of mixed action spaces, and significant differences in the optimization difficulty of the three optimization objectives (total AoI, total packet loss, and total energy consumption). In the classic centralized training and distributed execution framework, although the centralized commentator can use global information to evaluate the value of joint actions, it usually only outputs a single scalar value and cannot decouple the contributions of different objectives; while the distributed actor network relies on scalar rewards with fixed weights, which can easily lead to policy bias towards the energy consumption objective that is easier to converge, thus sacrificing the timeliness and reliability objectives.

[0049] To address this, this invention proposes a multi-agent, multi-objective parallel preference optimization (MA-MP3O) algorithm. Specifically, the algorithm employs parallel preference learning, which involves uniformly sampling across the entire three-dimensional objective weight space to generate multiple preference individuals, each of which is pre-assigned a set of preference weight vectors. This represents a user preference. Each individual with this preference includes a centralized network of commenters and... K Each drone has its own actor network. Among them, This represents the total AoI component in the weight vector. This represents the total number of lost packets component in the weight vector. This represents the total energy consumption component in the weight vector. The weight vector is used for subsequent advantage-weighted fusion and weighted critic network loss calculations; the weight vector is not involved in training.

[0050] In this embodiment, step S5 involves updating the parameters of the critic network and the actor network by constructing a loss function based on the values ​​of the three objectives—total AoI, total packet loss, and total energy consumption—output by the critic network, and by building the loss function based on the weight vectors and values ​​of the total AoI, total packet loss, and total energy consumption of each individual, to obtain the critic network and actor network for a fixed training epoch. Specifically, this includes: (1) Action selection and execution: Initialize the experience replay buffers of multiple preference individuals; at the beginning of each time slot, the first... Local state space of a UAV The input is fed into its actor network, which outputs the probability distribution of mission mode switching actions and the normal distribution of flight actions; the UAV selects and executes specific actions based on the output probability distribution and normal distribution. Specifically, the experience replay buffer is used to store historical experience data under the corresponding preferences. From the perspective of a single individual, at the beginning of each time slot, the local state space is first processed by the multi-head attention module. The features of each sensor node are weighted and aggregated to obtain the sensor feature context vector. This vector, along with the UAV's own state, is then input into the shared feature extraction layer of the actor network. Subsequently, the actor network branches into three action heads: a mode-switching head outputs discrete actions. The two-dimensional probability distribution vector is obtained; based on the probability of the uplink mode and the data return mode of data collection, the uplink mode or the return mode is selected; the flight angle head and the flight distance head output the unbounded mean respectively, which, together with the learnable standard deviation, form the original normal distribution; after sampling from it, it is compressed to (-1,1) by Tanh transformation, and then the specific values ​​of flight angle and flight distance are obtained by affine mapping.

[0051] (2) Experience collection and storage: After all UAVs perform actions, the global state space is updated to the global state space of the next time slot. ; Calculate the current time slot based on changes in the global state space t The reward function, and the experience tuple Store in the experience replay buffer of the corresponding preference individual; (3) Advantage estimation, value output and loss calculation: (31) After the experience replay buffer has accumulated data, a batch of experience data is randomly sampled from the buffer; the global state space is then... Input the critic network, output the normalized values ​​of the three objectives—total AoI, total packet loss, and total energy consumption—in the current global state, and then inverse normalize them to obtain the true values ​​of the three objectives. In this embodiment, for each sampling experience, the global state space is first... The input to the centralized critic network, after passing through a shared feature extraction layer, branches into three independent value output heads. To avoid the differences in the magnitude and fluctuation range of the three targets affecting training stability, the state values ​​output by the critic network adopt PopArt adaptive normalization technology, outputting the normalized values ​​of the total AoI, total packet loss, and total energy consumption of the three targets in the current state. , and By maintaining the mean and standard deviation of the three objectives, inverse normalization can be performed to obtain the true value.

[0052] (32) The advantage functions of the three objectives are calculated based on their true values ​​and reward functions; Specifically, first calculate the time-series difference errors of the three targets: Timing difference error of total AoI ; Timing differential error of total packet loss ; Timing difference error of total energy consumption ; In the formula For time slotst The true value of total AoI; For time slots t The true value of the total number of lost packets; For time slots t The true value of total energy consumption; This represents the discount factor, used to reduce the weight of future rewards. A value closer to 1 indicates that the agent prioritizes long-term cumulative rewards, while a value closer to 0 indicates a focus on immediate rewards. Then, through multi-step accumulation, the advantage functions for the three objectives are obtained: ; ; ;in; This represents the overall AoI advantage function. The advantage function represents the total number of lost packets; This represents the total energy consumption advantage function.

[0053] In the formula This is the GAE coefficient, used to balance bias and variance. Under this setting, the advantages of AoI and packet loss can effectively capture the delayed rewards of multi-step decision-making, while the energy consumption advantage focuses on the immediate flight costs.

[0054] (33) Based on the pre-assigned preference weight vector for each individual, the advantage functions of the three objectives are weighted and fused to obtain the comprehensive advantage. ; (34) The actor network is updated using the PPO pruning mechanism based on its comprehensive advantages; The loss function of the actor network is: ; In the formula Indicates the expectation. This represents the ratio of the probability of choosing the same action under the same conditions, given the new strategy and the old strategy. This indicates a truncation operation. A clipping range is defined for the adjustable hyperparameters. During the update, Limit the values ​​to this range and take the minimum value before and after clipping.

[0055] It should be noted that during the process of updating the actor network parameters, the learnable weight matrix and bias vector of the multi-head attention module and the linear transformation layer are updated simultaneously.

[0056] (35) Calculate the returns of the three objectives based on the advantage functions and real values ​​of the three objectives; normalize the returns of the three objectives; construct a weighted mean squared error loss function based on the normalized returns and values; First, the returns for the three objectives are calculated based on the calculated advantage function and the true value; the formulas for calculating the returns for the total AoI, total packet loss, and total energy consumption for the three objectives are as follows: , and ; Next, PopArt normalization technology is used to normalize the returns of the three objectives; The formula for calculating the weighted mean squared error loss of the critic network is as follows: ; In the formula, This represents the weighted mean square error loss; This represents the total AoI component in the weight vector. This represents the total number of lost packets component in the weight vector. This represents the total energy consumption component in the weight vector. This represents the true value of the total AoI after normalization. This represents the normalized total AoI reward. This represents the true value of the total number of lost packets after normalization. This represents the total number of lost packets after normalization. The normalized total energy consumption is the true value. This represents the normalized total energy return.

[0057] (4) Parameter update: The gradient descent algorithm is used to update the actor network parameters and critic network parameters corresponding to each preference individual to obtain the critic network and actor network with fixed training rounds.

[0058] The updated actor network parameters are sent back to each UAV terminal to guide subsequent action selection, thus forming a closed-loop learning and optimization process.

[0059] Step S6: Based on the actor network obtained from fixed training rounds, output the total AoI, total number of packet loss, and total energy consumption of multiple UAVs for all corresponding sensor nodes. Perform non-dominated comparison and sorting based on the output total AoI, total number of packet loss, and total energy consumption of multiple UAVs, and output a diverse set of strategies covering the three-dimensional Pareto front. In this embodiment, step S6 specifically includes: Step S61: Based on the actor networks obtained from different fixed training rounds and different individuals, output multiple strategy solutions that include the total AoI, total number of packet losses, and total energy consumption of multiple UAVs for all sensor nodes. Step S62: Remove the dominant policy solutions from the policy solutions and retain the non-dominated policy solutions. Sort the retained non-dominated policy solutions independently in multiple dimensions according to the three objectives of total AoI, total number of packet losses and total energy consumption. It should be noted that for two policy solutions, if the first policy solution is no worse than the second policy solution in terms of total AoI, total packet loss, and total energy consumption, and performs better in at least one of these objectives, then the first policy solution dominates the second policy solution, and the second policy solution is a dominant policy solution. If the first and second policy solutions have varying degrees of performance in their three objectives, then these two policy solutions do not dominate each other. Therefore, a non-dominated policy solution is one that cannot be dominated by any other policy solution in the current policy solution set.

[0060] Step S63: Calculate the crowding degree of each sorted non-dominated strategy solution; The crowding degree calculation formula for each non-dominated strategy solution is as follows: ; In the formula, Representing the solution of non-dominated strategy The degree of congestion; This indicates that when sorted by total AoI, the solution is located in the non-dominated strategy solution. The total AoI corresponding to the next policy solution, This indicates a solution located in a non-dominated strategy. The total AoI corresponding to the previous strategy solution; This indicates that when sorted by total energy consumption, the solution located at the non-dominated strategy location is... The total energy consumption corresponding to the next strategy solution. This indicates a solution located in a non-dominated strategy. The total energy consumption corresponding to the previous strategy solution; This indicates that when sorted by total packet loss, the solution is located at the non-dominated strategy solution. The total number of packets lost corresponding to the next strategy solution. This indicates a solution located in a non-dominated strategy. The total number of packets lost corresponding to the previous strategy solution; This represents the maximum total AoI across all sorted solutions; This represents the minimum total AoI across all sorted solutions; This represents the maximum total energy consumption of all sorted solutions; This represents the minimum total energy consumption of all sorted solutions; This represents the maximum total number of lost packets across all sorted solutions. This represents the minimum total number of packet losses across all sorted solutions.

[0061] Step S64: Reorder the solutions of each non-dominated strategy according to the degree of congestion; Step S65: Retain the non-dominated strategy solutions with higher crowding rankings until the preset population size is met, and eliminate the non-dominated strategy solutions with lower rankings. It should be noted that after sorting the non-dominated policy solutions in sequence according to the three objective dimensions, since the largest and smallest non-dominated policy solutions obtained by sorting have only one neighboring solution, the crowding degree of these non-dominated policy solutions is set to infinity, and therefore needs to be retained when sorting according to the crowding degree.

[0062] Step S66: Determine whether the number of training iterations has reached the preset number of training iterations; if yes, use the non-dominated strategy solutions that meet the preset population size as the diversified strategy set covering the three-dimensional Pareto front; if no, proceed to step S61.

[0063] During training, regular testing and evaluation are necessary. Taking a preset training run of 10,000 rounds as an example, every 500 rounds, a non-dominated comparison and ranking is performed on all policy solutions output by individuals with different weight vectors; the non-dominated policy solutions with higher crowding rankings are retained, while those with lower rankings are discarded. In each training round, the non-dominated policy solutions retained from the previous round will continue to participate in the non-dominated comparison and ranking in the current round. After all training is completed, an optimal set of non-dominated policies can be obtained.

[0064] To verify the effectiveness of this invention, simulation training was performed using Python 3.11 and the PyTorch 2.4.1 framework. Consider a... A simulation environment for multi-UAV-assisted WSN data collection within a rectangular area is presented, with 20 sensor nodes randomly distributed within it. The base station is located at the edge of the area, and its two-dimensional coordinates are... The antenna height is 35m. Two UAVs are deployed near the base station, with a fixed flight altitude of 75m. The two-dimensional coordinates of their initial positions are as follows: m and The region is divided into two areas, with the base station at the lower left corner as the starting point and the upper right corner as the boundary, serving as the mission responsibility areas for the two UAVs. Each mission cycle is 300 seconds with a time step of 1 second. The maximum onboard energy of a single UAV is 100kJ. The data collection radius and safe flight distance of the UAVs are 75m and 50m, respectively. The buffer capacity of the ground sensor nodes is 5 data packets, each 1MB in size. The data generation process follows a Bernoulli distribution, and the generation probability of each node is... These parameters change over time. Unless otherwise specified, please refer to the values ​​in Table 1 for other relevant parameters.

[0065] Table 1 Simulation Parameter Description

[0066] To fairly and fully evaluate the performance of the proposed method (hereinafter referred to as MA-MP3O), corresponding comparative algorithms were selected for its key design aspects. Specifically, to verify the effectiveness of the designed centralized training distributed execution architecture in alleviating the multi-agent cooperation problem, Independent Multi-Preference Proximal Policy Optimization (IMP3O) with independent learning was introduced as a control. This algorithm allows each UAV to update its policy independently based only on its own local observations, ignoring the mutual influence between agents. To compare the overall performance of multi-agent cooperative decision-making, the traditional multi-agent reinforcement learning algorithm MAPPO was selected as a benchmark. This algorithm serves as the benchmark implementation of multi-agent PPO. Furthermore, to evaluate the joint optimization capability of the proposed method in a mixed action space, the MATD3 algorithm was introduced as a comparison. This algorithm uses a deterministic policy to handle continuous actions and decouples discrete actions through a heuristic approach. The four algorithms were trained in the same environment, and the various indicators of the individuals with specific weights selected during training were recorded and compared. The quality of the Pareto front obtained by all algorithms was also evaluated.

[0067] First, during training, four preference groups were tracked: high AoI weight, high packet loss weight, high energy consumption weight, and balanced weight. Specifically, the high AoI weight, high packet loss weight, high energy consumption weight, and balanced weight are approximated as follows: , , , .like Figure 3 As shown.

[0068] Figure 3 The trends of different algorithms in terms of total AoI, total packet loss, and total energy consumption are shown. It can be seen that the method proposed in this invention maintains stable convergence characteristics and good policy adaptability under different preference configurations. For high AoI weight preference, the proposed method utilizes a global commentator to accurately evaluate the joint state, guiding the UAV to prioritize serving high AoI nodes, resulting in a significantly faster convergence speed for total AoI compared to the comparative algorithms, and also achieving the lowest final convergence value. Simultaneously, the multi-head attention mechanism is used for real-time perception of node cache states, achieving stable convergence in terms of energy consumption and effectively suppressing the number of packet losses. In contrast, the IMP3O algorithm exhibits significant oscillations in its convergence process due to a lack of coordination among agents; the MAPPO and MATD3 algorithms are limited by the mixed action space and multi-target processing capabilities, resulting in larger fluctuations in their convergence processes.

[0069] Furthermore, thanks to the PopArt normalization technique for balancing multi-objective gradients, the method of this invention maintains stable convergence in terms of energy consumption and packet loss metrics. When the preference switches to a high packet loss weight, the method of this invention coordinates the drone to frequently switch to data collection mode through a global commentator, promptly clearing node caches, so that the number of packet losses quickly converges to an extremely low level, significantly outperforming other algorithms. Although the total energy consumption under this method is slightly higher than that of the high AoI weight scenario, the entire convergence process remains stable. Under the high energy consumption weight configuration, the method of this invention adopts a conservative flight strategy, with the total energy consumption converging to the minimum level and the process stable, significantly outperforming the highly volatile MATD3 and MAPPO algorithms. Although the AoI initially increases, through the coordination of task modes, it can eventually converge to a reasonable range, and the number of packet losses does not increase significantly, avoiding the packet loss surge problem commonly seen in the energy-saving process of the compared algorithms. As for balancing weights, the method of this invention also demonstrates good adaptability to different preferences.

[0070] The above results show that the method of the present invention can adaptively adjust the multi-UAV collaborative strategy according to different user preferences, and achieve an optimal trade-off between the three objectives of total AoI, total packet loss and total energy consumption. At the same time, the design effectively solves the limitations of traditional algorithms in credit allocation and mixed action space processing.

[0071] Table 2 presents the Pareto correlation data obtained by the four algorithms. It can be seen that the method of this invention outperforms the other algorithms in terms of hypervolume, spacing, and performance on most targets. Notably, IMP3O slightly outperforms the method of this invention in terms of the range of packet loss. This may be due to the degree of competition formed during the independent training of each UAV, making it easier to cover all nodes within the entire area.

[0072] Table 2 Pareto Data .

Claims

1. A multi-objective joint optimization method for collaborative data collection by multiple UAVs under random traffic conditions, characterized in that, Specifically, the following steps are included: Step S1: Method begins; Step S2: Initialize UAV information, sensor information, and ground base station information; Step S3: Establish a data generation and AoI model and a communication and energy consumption model based on UAV information, sensor information and ground base station information. Based on the data generation and AoI model and the communication and energy consumption model, construct a multi-objective optimization problem model that jointly minimizes the total AoI of the node, the total number of packet losses and the total energy consumption of multiple UAVs. Step S4: Map the multi-objective joint optimization problem corresponding to the multi-objective optimization problem model that jointly minimizes the total AoI of the node, the total number of packet losses, and the total energy consumption of multiple UAVs to a multi-agent multi-objective Markov decision process; Step S5: Propose the MA-MP3O algorithm to solve the multi-objective joint optimization problem; The MA-MP3O algorithm initializes multiple preference individuals, each pre-assigned a set of weight vectors for total AoI, total packet loss, and total energy consumption. Each individual contains a centralized critic network and an actor network for each UAV. The critic network outputs the values ​​of the three objectives: total AoI, total packet loss, and total energy consumption. A loss function is constructed based on the individual's weight vectors and values ​​for total AoI, total packet loss, and total energy consumption to update the parameters of the critic network and actor network, thus obtaining the critic network and actor network for a fixed number of training rounds. Step S6: Based on the actor network obtained from fixed training rounds, output the total AoI, total number of packet losses, and total energy consumption of multiple UAVs for all corresponding sensor nodes. Sort the output total AoI, total number of packet losses, and total energy consumption of multiple UAVs, and output a diverse set of strategies covering the three-dimensional Pareto front.

2. The multi-objective joint optimization method for multi-UAV cooperative data collection under random traffic conditions as described in claim 1, characterized in that, Step S3, which establishes the data generation and AoI model based on UAV information, sensor information, and ground base station information, specifically includes: Define nodes In the time slot Is a data packet generated? ; This indicates the generation of a data packet. This indicates that it was not generated. Define nodes In the time slot Did packet loss occur? ; The calculation formula is as follows: ； In the formula, For nodes In the time slot The number of cached data packets; This indicates the maximum cache capacity for each node; The formula for calculating the total number of packet losses across all nodes during the entire task time is: ； In the formula, This represents the total number of nodes. Indicates the task time for UAV data collection; The node AoI is defined as the time difference between the current timeslot and the time when the latest data packet was generated by that node. Calculation, where For nodes In the time slot AoI, Represents a node In the time slot The latest data packet was generated at the time of its creation; The total AoI of all sensor nodes throughout the entire task time is calculated using the following formula: 。 3. The multi-objective joint optimization method for multi-UAV cooperative data collection under random traffic conditions as described in claim 2, characterized in that, Step S3, which establishes the communication and energy consumption model based on UAV information, sensor information, and ground base station information, specifically includes: Get Nodes i With the k The probability of a Loss of Space (LoS) link during UAV communication is calculated using the following formula: ； In the formula and It is a constant related to the environment. It is a node i With the k Pitch angle between UAVs; node i With the k The probability of using an NLoS link when communicating with UAV ; Get Nodes i With the k Path loss between UAVs The calculation formula is as follows: ； In the formula At the speed of light, Indicates the carrier frequency. This is the path loss index. and These are the average additional path loss for LoS and NLoS links, respectively. For time slots t node i With the k The Euclidean distance between UAVs; Get Nodes With the Average path loss between UAVs The calculation formula is as follows: ； In the formula, Represents a node i With the k Loss of the LoS link between UAVs; Represents a node i With the k NLoS link path loss between UAVs; Get Nodes With the UAV communication rate The calculation formula is as follows: ； In the formula, For system channel bandwidth, It is a node The transmission power, This is the additive white Gaussian noise power at the UAV receiver; Set time slot With nodes The set of UAVs that meet the communication conditions is: ； In the formula, Represents the set of UAVs. Indicates the data collection radius of the UAV. Indicates the size of a data packet. Indicates the first The mission mode of the UAV. Indicates the first The UAV is in uplink mode for data collection; Then it means the first The UAV is in data return mode; when , i.e., slot There exists at least one capable of interacting with the node. When a UAV establishes a communication link; Choose the UAV with the highest communication rate to establish communication, that is: ； In the formula For nodes In the time slot Associated UAV.

4. The multi-objective joint optimization method for multi-UAV cooperative data collection under random traffic conditions as described in claim 3, characterized in that, Step S3, which establishes the communication and energy consumption model based on UAV information, sensor information, and ground base station information, also includes: Get the UAV propulsion power The calculation formula is as follows: ； In the formula Indicates time slot t No. The speed of UAV deployment The linear velocity at the tip of the rotor blade. Indicates the induction velocity, and These are constants related to UAV induced power and hovering power, respectively. Represents air density, For rotor realism, and These are the fuselage drag coefficient and the rotor disk area, respectively. Dividing the task time into discrete time slots of fixed length 1 second, the total task time... The total energy consumption of all UAVs inside is: 。 5. The multi-objective joint optimization method for multi-UAV cooperative data collection under random traffic conditions according to claim 4, characterized in that, The steps described above, based on the data generation and AoI model and the communication and energy consumption model, construct a multi-objective optimization problem model that jointly minimizes the total node AoI, total packet loss, and total energy consumption of multiple UAVs. Specifically, these steps include: Let the first The flight angle and maneuver selected by the UAV are ; Let the first Flight distance of a UAV in a single discrete time slot ;in, Indicates the minimum distance to move; Indicates the maximum distance traveled; set up Indicates the flight maneuvers of the UAV. Let the task mode selection action of the UAV be represented. Then, the multi-objective joint optimization problem corresponding to the multi-objective optimization problem model that jointly minimizes the total AoI of the node, the total number of packet losses, and the total energy consumption of multiple UAVs is: ； In the formula The flight area of ​​the UAV is [size missing] ; Indicates time slot t No. The X-axis position coordinates of the UAV in a three-dimensional Cartesian coordinate system; Indicates time slot t No. The Y-axis position coordinates of the UAV in a three-dimensional Cartesian coordinate system; Indicates time slot t No. The coordinates of the UAV's location; Indicates except the first The position coordinates of other UAVs besides the one mounted on top of the UAV. Indicates a safe distance; Indicates the first The UAV has remaining energy; Indicates the maximum onboard energy; Indicates the first Is the UAV a node? In the time slot Associated UAV; when the Build UAV as nodes In the time slot Associated UAVs The value is 1 if it is not 1, and 0 otherwise. Represents the UAV set, Represents a set of nodes. Represents a set of discrete time slots; Indicates that the UAV set excluding the first Other UAVs besides the standard UAV.

6. The multi-objective joint optimization method for multi-UAV cooperative data collection under random traffic conditions as described in claim 5, characterized in that, Step S4 specifically includes: Constructing time slots t Global state space: ； In the formula Indicates time slot t Location of each UAV Represents the set of positions of all nodes. Represents a node Location; For time slots t The remaining energy of each UAV; For each node's AoI set, For time slots t A collection of various UAV task modes This represents the set of ages of the oldest data packets in each node's local cache queue. This represents the set of ages of the oldest data packets in the onboard cache of each UAV. Indicates time slot t The total number of data packets in the cache of each UAV Indicates time slot t No. k The total number of data packets in the UAV buffer; Indicates the time slot of each node The set of cached data packets, and Indicates time slot t The number of packet losses across all nodes, via calculate; Among them, nodes In the time slot Age of the oldest data packet In the formula, Represents a node In the time slot The first in local cache The generation time of each data packet; No. k UAV in time slot Age of the oldest data packet in the onboard cache ; In the formula, Indicates the first k UAV in time slot The first in the onboard cache The generation time of each data packet; Constructing time slots t No. k Build UAV local state space ; In the formula, time slot t No. Downlink communication rate between UAV and base station ; It is the transmission power of the UAV. This represents the noise power at the base station. For base station and the k Path loss between UAVs ; This represents the Euclidean distance between the UAV and the base station; Indicates the first k frame Azimuth angle between UAV and ground base station; Indicates the first k The set of relative positions between a UAV and other UAVs; Indicates the first k The age of the oldest data packet in the UAV's onboard buffer; and That is, time slot t No. k The total number of data packets in the UAV buffer; The sensor feature context vector, obtained by aggregation through a multi-head attention module, is calculated as follows: ； This represents a multi-head attention mechanism; among which, the first k UAV query vector Due to its position Obtained by linear transformation; and These represent all nodes in the time slot. t The key matrix and value matrix are derived from the eigenvectors of all sensor nodes. Obtained by linear transformation; in, ; In the formula, Represents a node With the k The European distance for UAVs; Represents a node With the k frame Horizontal azimuth of the UAV; Represents a node With the k frame UAV communication rate; Constructing Action Space ; Constructing a reward function ; In the formula, For time slots t Node information, age, AoI-related rewards, It is a reward for avoiding packet loss at nodes, and Energy-related rewards; in, ; In the formula, The average AoI of all nodes in time slot t is used as the AoI backlog penalty; Update rewards for AoI; pass Calculated; The average AoI of all nodes in time slot t+1; Indicates time slot t The average age of the oldest data packets in the local cache queues of all nodes; and These are non-negative weighting coefficients in the range (0,1); in, ; Indicates time slot t No. The number of data packets collected by the UAV; and These are non-negative weighting coefficients ranging from (0,1); in, ; This indicates that penalties will be imposed on the energy-consuming behavior of UAVs; pass Calculated; Indicates the first The remaining energy of the UAV in time slot t; Indicates the first The remaining energy of the UAV in time slot t+1; Indicates the warning distance; Similarly, the weighting coefficients are non-negative and range from (0,1). The soft partition penalty is calculated using the following formula: ; In the formula Indicates that the first The areas of responsibility assigned to the UAVs Indicates the first Whether the UAV is within its own area of ​​responsibility, when the first When the UAV is not within its own area of ​​responsibility, the indicator function is 1; when the first... When the UAV is within its designated area of ​​responsibility, the indicator function is 0, while For time slots t No. The signed distance from the current position of the UAV to the predefined boundary of the region.

7. The multi-objective joint optimization method for multi-UAV cooperative data collection under random traffic conditions as described in claim 6, characterized in that, In step S5, the values ​​of the three objectives—total AoI, total packet loss, and total energy consumption—output by the critic network are used to update the parameters of the critic network and the actor network by constructing a loss function based on the weight vectors and values ​​of the total AoI, total packet loss, and total energy consumption of each individual, in order to obtain the critic network and actor network with fixed training epochs. Specifically, this includes: (1) Action selection and execution: Initialize the experience replay buffers of multiple preference individuals; at the beginning of each time slot, the first... k Local state space of a UAV The input is fed into its actor network, which outputs the probability distribution of mission mode switching actions and the normal distribution of flight actions; the UAV selects and executes specific actions based on the output probability distribution and normal distribution. (2) Experience collection and storage: After all UAVs perform actions, the global state space is updated to the global state space of the next time slot. ; Calculate the current time slot based on changes in the global state space t The reward function, and the experience tuple Store in the experience replay buffer of the corresponding preference individual; (3) Advantage estimation, value output and loss calculation: Once the experience replay buffer has accumulated data, a batch of experience data is randomly sampled from the buffer; the global state space is then... Input the critic network, output the normalized values ​​of the three objectives—total AoI, total packet loss, and total energy consumption—in the current global state, and then inverse normalize them to obtain the true values ​​of the three objectives. The advantage functions of the three objectives are calculated based on their true values ​​and reward functions. Based on the pre-assigned preference weight vector for each individual, the advantage functions of the three objectives are weighted and fused to obtain the comprehensive advantage. ; The actor network is updated using the PPO trimming mechanism based on its overall advantages; The returns for the three objectives are calculated based on the advantage functions and true values ​​of the three objectives; the returns for the three objectives are then normalized; and a weighted mean squared error loss function is constructed based on the normalized returns and values. (4) Parameter update: The gradient descent algorithm is used to update the actor network parameters and critic network parameters corresponding to each preference individual to obtain the critic network and actor network with fixed training rounds.

8. The multi-objective joint optimization method for multi-UAV cooperative data collection under random traffic conditions as described in claim 7, characterized in that: The advantage functions for the three objectives are calculated as follows: ; ; ;in; This represents the overall AoI advantage function. The advantage function represents the total number of lost packets; Represents the total energy consumption advantage function; the time-series difference error of total AoI. ; Timing differential error of total packet loss ; Timing difference error of total energy consumption ; Indicates the discount factor; These are GAE coefficients, used to balance bias and variance; The true value of total AoI in time slot t; This represents the true value of the total number of packets lost in time slot t. This represents the true value of the total energy consumption in time slot t. The formulas for calculating the rewards for the three objectives—total AoI, total packet loss, and total energy consumption—are as follows: , and ; The weighted mean squared error loss of the critic network is calculated using the following formula: ； In the formula, This represents the weighted mean square error loss; This represents the total AoI component in the weight vector. This represents the total number of lost packets component in the weight vector. This represents the total energy consumption component in the weight vector. This represents the true value of the total AoI after normalization. This represents the normalized total AoI reward. This represents the true value of the total number of lost packets after normalization. This represents the total number of lost packets after normalization. The normalized total energy consumption is the true value. This represents the normalized total energy return.

9. The multi-objective joint optimization method for multi-UAV cooperative data collection under random traffic conditions according to claim 1, characterized in that, Step S6 specifically includes: Step S61: Based on the actor networks obtained from different fixed training rounds and different individuals, output multiple strategy solutions that include the total AoI, total number of packet losses, and total energy consumption of multiple UAVs for all sensor nodes. Step S62: Remove the dominant policy solutions from the policy solutions and retain the non-dominated policy solutions. Sort the retained non-dominated policy solutions independently in multiple dimensions according to the three objectives of total AoI, total number of lost packets and total energy consumption. Step S63: Calculate the crowding degree of each sorted non-dominated strategy solution; Step S64: Reorder the solutions of each non-dominated strategy according to the degree of congestion; Step S65: Retain the non-dominated strategy solutions with higher crowding rankings until the preset population size is met, and eliminate the non-dominated strategy solutions with lower rankings. Step S66: Determine whether the number of training iterations has reached the preset number of training iterations; if yes, use the non-dominated strategy solutions that meet the preset population size as the diversified strategy set covering the three-dimensional Pareto front; if no, proceed to step S61.

Images