UAV-assisted IoT data collection method for multi-institutional overlapping deployment

Abstract

The present invention provides a drone-assisted IoT data collection method for overlapping multi-agency deployments. The method involves each agency's drone collecting local observation states, merging them into a global observation state for the current time step, and then generating an output action. After each drone performs an action, it collects local observation states again, merges them into a global observation state for the next step, and calculates a reward for the current time step. A reward function is designed with the training objective of minimizing the average AoI of all IoT devices. An experience tuple consisting of the global observation state for the current time step, the joint action, the reward, and the global observation state for the next step is stored in an experience replay pool. Training data is retrieved from the experience replay pool to complete model training based on MADDPG. The trained actor network is then used to plan and deploy drone paths for data collection. This method addresses the problem of inter-agency data collection requiring both data independence and collaboration to reduce the system's average information age.

Description

Technical Field

[0001] The present invention relates to mobile communication technology and Internet of Things technology, and in particular to drone-assisted Internet of Things data collection technology for overlapping deployment of multiple institutions. Background Art

[0002] Internet of Things (IoT) technology offers unique advantages for collecting information in hazardous and harsh environments. However, these environments often lack mains power, making manual maintenance of IoT devices expensive. Therefore, IoT devices often utilize long-lasting, low-power components and operate in short-range communication to extend their operating time. This short-range communication approach limits the communication range of IoT devices, making it difficult to transmit data directly to data centers, thus hindering the real-time nature of IoT data collection. In recent years, unmanned aerial vehicles (UAVs) have emerged as a promising tool for IoT data collection, enabling efficient and real-time transmission of monitoring data back to data centers, thereby compensating for the limited communication capabilities of IoT devices. Especially in environments with strict Age of Information (AoI) requirements, utilizing UAVs to collect IoT data and transmit it back to data centers in real time allows for timely and efficient data transmission without impacting the IoT's lifespan, thus maintaining a good AoI.

[0003] In this case, by rationally planning the UAV flight trajectory, the data collection efficiency can be improved and the needs can be quickly responded to in real-time application scenarios. Therefore, UAV flight trajectory planning has become one of the hot research directions of UAV-assisted IoT data collection.

[0004] However, existing studies generally assume that all UAVs and IoT devices belong to the same organization, and there is no need for inter-institutional data isolation in the collaboration between these UAVs and IoT devices.

[0005] However, in practical applications, there are often multiple agencies monitoring environmental information in parallel within a dangerous or harsh target area due to the specific nature of the area's key monitoring, such as remote areas monitoring vegetation and emergency disaster relief areas. Due to the overlap of regions, there is the potential for collaboration between multiple agencies to improve work efficiency and data freshness. However, because each agency requires different monitoring data and their data service systems differ, IoT devices between agencies need to maintain data independence. Therefore, it is not possible to simply merge monitoring data from different agencies. Therefore, data collection between agencies must maintain relative independence while also collaborating to improve information timeliness. This poses new challenges to trajectory planning and collaboration mechanisms for UAV-based data collection between multiple agencies. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a solution that fully utilizes the collaborative data collection capabilities of drones of different organizations.

[0007] The technical solution adopted by the present invention to solve the above technical problems is a drone-assisted Internet of Things data collection method for overlapping deployment of multiple institutions, comprising the following steps:

[0008] 1) Interaction and environment sampling steps based on multi-agent deep deterministic policy gradient MADDPG:

[0009] Each organization's drone collects local observation status, which includes the AoI, location, and energy consumption of all IoT devices within the drone's observation range at the current time step, as well as the drone's location, energy consumption, and collection status tag for each IoT device;

[0010] After the local observation states of all the drones in the organization are combined to form the global observation state of the current time step, the global observation state is input into the initial Actor network of each drone. Each initial Actor network outputs an action, and exploration noise is added to the output action as the execution action of the drone. The execution actions of all the drones in the organization are combined to form the joint action of the current time step.

[0011] After each drone performs an action, it collects local observation states and combines them to obtain the global observation state for the next step, and calculates the reward for the current time step. The reward function is designed to minimize the average AoI of all IoT devices as the training objective of the actor network.

[0012] The experience tuple consisting of the global observation state of the current time step, the joint action of the current time step, the reward of the current time step and the global observation state of the next step is stored in the experience replay pool;

[0013] 2) MADDPG-based training steps: Extract training data from the experience replay pool and update the Critic network via gradient descent. Update the Actor network of each drone via gradient ascent. Soft-update the Target Actor and TargetCritic networks. Repeat this process until training is complete.

[0014] 3) Deployment steps: Each organization’s drones use their trained Actor networks to plan cross-institutional data collection trajectories for IoT devices within their observation range.

[0015] Optimizing the average AoI of the IoT system by jointly planning the trajectories of multiple agency drones is an NP-Hard problem. Traditional model optimization solutions are not suitable for multi-UAV flight trajectory planning in this environment. This paper uses the deep deterministic policy gradient algorithm MADDPG as a basis to jointly plan the flight trajectories of multiple agency drones, optimize the order and collaboration mode of IoT network data collection by multiple agency UAVs, and optimize the AoI in drone-assisted IoT data collection. By rationally planning the drone collection plan, the average AoI of the drone-assisted IoT data collection system is minimized in scenarios where multiple agencies are simultaneously monitoring a certain area.

[0016] Furthermore, the reward function is centered around the optimization model's objective function, meaning the lower the system's average AoI, the higher the reward. To balance factors like energy consumption constraints, multi-agency collaboration, and task completion, the reward function is designed with additional constraints. Specifically, when designing the reward function with minimizing the average AoI of all IoT devices as the training objective for the actor network, the reward function must also meet the following constraints: the actions output by the actor network must be within the drone's flight capabilities; the sum of the energy consumption of IoT devices and the drone's energy consumption must be less than the set system energy consumption cap; the drone's flight range must be within the preset monitoring area; each drone can only collect data from a maximum of one IoT device at a time; each IoT device can only be collected by a maximum of one drone at a time; and drones from different agencies must maintain a preset safe flight distance.

[0017] The beneficial effect of this invention is that the joint planning of flight trajectories and collaborative modes for multiple drones, based on the MADDPG algorithm, solves the problem of maintaining data independence among the agencies while simultaneously collaborating to reduce the system's average Area of ​​Interest (AoI). This fully leverages the collaborative data collection capabilities of the drones, consuming less energy and taking less time than independent operations by each agency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a multi-agency UAV-assisted data collection scenario diagram;

[0019] Figure 2 It is a schematic diagram of AoI changing over time;

[0020] Figure 3 It is a convergence diagram of CC-MADDPG;

[0021] Figure 4 is the average AoI and energy consumption curve when the number of IoT devices changes;

[0022] Figure 5 It is the average AoI curve when the number of UAVs changes. DETAILED DESCRIPTION

[0023] To facilitate those skilled in the art to understand the technical content of the present invention, the present invention is further explained below with reference to the accompanying drawings.

[0024] Considering the scenario where multiple agencies’ drones assist in IoT data collection in the same area, each agency independently plans a path to collect its own equipment, which will inevitably lead to flying across the entire area, thus consuming more time and energy. In order to reduce the average AoI of the system as much as possible, drones need to spend less flight time to collect data, which brings about the possibility of multi-agency drones collaborating to collect data. When multiple agencies monitor a certain area at the same time, the IoT devices between agencies must maintain data independence while collaborating with each other to improve the timeliness of information. This paper proposes an AoI minimization optimization model for multi-agency collaborative collection, and based on the newly proposed CC-MADDPG algorithm, obtains the optimal trajectory and collaboration mode of multi-agency drones, and ultimately optimizes the average AoI of the system.

[0025] The following is a detailed description of the technical solution:

[0026] 1. Scenario Description

[0027] like Figure 1 As shown in Figure 2, we take the scenario where two organizations overlap and cover the target area through UAV and IoT and collect data. In this scenario, both Organization 1 and Organization 2 deploy corresponding IoT devices according to their business needs to perceive the environmental information they need. Organization 1 deploys IoT devices ( ), mechanism 2 deploys non-uniformly IoT devices ( ). That is: institution Unevenly distributed IoT devices ( ), superscript 、 All are the equipment serial number variables of the organization, subscript The two organizations each have a drone that collects its own IoT data and transmits it back to its own data center in real time. IoT devices made of drones Periodically collect and transmit to the back-end data center of institution 1 , Institution 2 IoT devices by 1 drone Periodically collect and transmit to the back-end data center of institution 2 In order to simplify the analysis, the present invention divides the time into The time step set of The time set of time steps is expressed as . and All at a fixed height within the target monitoring area Flying, in In a time step, Its flight direction is , the flight speed is , The maximum flight speed is preset. Both represent institution serial numbers , one organization corresponds to one drone, and Can be equal, When used as a subscript of a certain parameter of a drone, it can also be regarded as the drone serial number. Indicator The total number of IoT devices in the Hour, .

[0028] The drones all use the same communication frequency band and system, but the data reception between agencies is independent, i.e. Unable to access , Can't access Considering that the target monitoring areas of Institution 1 and Institution 2 overlap, their IoT devices may also be located in overlapping locations. This brings opportunities for Institution 1 and Institution 2 to collaborate to improve data collection efficiency. For example, In collecting your own organization's IoT devices When There is a second organization next to it. IoT devices When Can help at a lower flight cost Collected information, thereby avoiding For collection The extra cost of this collaboration is Cannot directly Forwarding collected data, so Collected The data needs to be forwarded to , and then by Forward to . The same applies when assisting Agency 1 in collecting data.

[0029] In the process of drone-assisted IoT data collection, in order to ensure The stability of data reception, During data upload, Need to fly to IoT devices and hover until Complete the data transfer to complete the Monitoring data collection. After data reception is completed, Forward the received data in real time, if it is collected by IoT devices of the same organization The monitoring data Forward data directly back to the institution’s data center If the data collected is not collected from IoT devices of the same organization , The monitoring data is forwarded to the drone that owns the collected data. , handed over to Return the data to the corresponding .

[0030] 2. Average AoI Minimization Optimization Model for Multi-Agency Collaborative Collection

[0031] (1) AoI model

[0032] The IoT device generates new perception data at the beginning of each time step, overwriting the old data from the previous round. When a drone hovers over the device and prepares to collect data, the IoT device will start transmitting data in the next time step. As time goes by, when the IoT device data is successfully collected, its AoI is reset to 0, and if it is not collected, its AoI gradually increases. Figure 2 As shown, IoT devices The data at time step The AoI at .exist The drone just completed the The collection of perceptual information, As time went by, the drone did not collect , then The AoI also increases accordingly, such as at time step Department The AoI is .exist UAVs The information of has been collected again, then , The AoI is . is the threshold of AoI. When AoI exceeds the threshold during cooperative collection In this way, the drone will face penalties to avoid unreasonable path planning of the drone, which may cause some devices to be ignored or the collection interval to be too long.

[0033] In the At the end of each time step, the IoT device The AoI is expressed as:

[0034] ;

[0035] in, Indicates the device in this time step By drone collection.

[0036] but The average AoI of the data acquisition system within a time step is expressed as:

[0037] ;

[0038] 、 The total number of devices in institution 1 and institution 2 respectively.

[0039] If the number of organizations S is not limited to 2, the average AoI of the system is expressed as:

[0040] .

[0041] (1) Communication model

[0042] Considering the time-varying characteristics and spatial heterogeneity of wireless transmission channels, a composite fading channel model is used to characterize the channel characteristics between a UAV and another communication node, comprehensively characterizing the two key fading effects of large-scale fading and small-scale fading. Communication nodes can be UAVs, IoT devices, and data centers. , The channel coefficient between another communication node is expressed as:

[0043] ;in, Indicates that at time step The small-scale fading coefficient of Indicates that at time step Small-scale fading is often caused by multipath effects and is usually assumed to follow a Rician distribution. When there is no obvious line-of-sight (LoS) component or when multipath scattering is abundant, its average power is usually normalized to 1. Large-scale fading is usually caused by path loss and shadowing, which can cause the system transmission rate to decrease.

[0044] In practical applications, it is usually necessary to consider the randomness of LoS communication and non-line of sight (NLoS) communication. According to the 3GPP specification, at the time step , the average path loss can be used to express the channel gain:

[0045] ;

[0046] in, and express and the LoS path loss and NLoS path loss between the other communication node, and denote the probabilities of LoS and NLoS respectively, Expressed as:

[0047] ;

[0048] in, and are two constants related to the environment, yes The elevation angle between the node and another communicating node.

[0049] At time step , IoT devices arrive Data acquisition rate between Expressed as:

[0050] ;

[0051] in, express and The channel bandwidth between Represents IoT devices The transmission power when transmitting data, represents the additive white Gaussian noise power, It is an IoT device and The channel coefficient between .

[0052] At time step , drones and Data transfer rate between Expressed as:

[0053] ;

[0054] in, express and The link bandwidth between For the sender The transmission power, for and The channel coefficient between .

[0055] At time step , drones To the data center of the same organization The transmission rate when sending data Expressed as:

[0056] ;

[0057] in, express and data centers The link bandwidth between express and data centers The channel coefficient between .

[0058] Considering the characteristics of IoT devices working in dangerous and harsh environments, it can be assumed that the amount of monitoring data obtained each time remains unchanged. At the same time, the newly collected data in each time step will overwrite the old data of the previous round. , collection The time representation of the data is:

[0059] ;

[0060] in, For IoT devices The amount of data being monitored.

[0061] If IoT devices are transmitted between drones The monitoring data is , the transmission time is expressed as:

[0062] ;

[0063] At time step , The collected data Back to the same data center , the return time is expressed as:

[0064] .

[0065] (2) Energy consumption model

[0066] Total energy consumption during drone-assisted IoT device data collection It consists of two parts, at time step , the transmission energy consumption of IoT devices and drone energy consumption ,Right now .

[0067] In the process of UAV-assisted data collection, at the time step , IoT devices The energy consumption of uploading data to the collection drone is expressed as:

[0068] ;

[0069] At time step , the total energy consumption of IoT devices is expressed as:

[0070] ;

[0071] drones Energy consumption Generally consists of two parts: communication-related energy consumption and propulsion energy consumption ,Right now Communication-related energy consumption It involves the data transmission, reception and processing power consumption of communication modules during data collection, promoting energy consumption It reflects the energy consumed by the UAV to overcome gravity and air resistance and achieve hovering and flying. and flight energy consumption ,Right now .

[0072] At time step , Collecting equipment belonging to different institutions Data, communication energy consumption Including data receiving energy consumption and The energy consumption of transmitting data is expressed as:

[0073] ;

[0074] in, express The receiving device monitors the receiving power of the data, Indicates The transmit power for transmitting data.

[0075] At time step , Collected by the same institution Data, communication energy consumption It includes the energy consumption of receiving data and the energy consumption of transmitting data back to the data center, expressed as:

[0076] ;

[0077] in, express The transmit power used to transmit data back to the data center.

[0078] At time step , Help collect data, Communication energy consumption This includes the energy consumption of relaying data and the energy consumption of transmitting data back to the data center, expressed as:

[0079] ;

[0080] in, express The receiving power of the relayed data.

[0081] At time step , The hovering energy consumption is mainly used to overcome gravity, maintain a stable flight attitude and compensate for disturbances caused by air flow fluctuations. Expressed as:

[0082] ;

[0083] Among them, the first Indicates the drag power generated by the rotor in the hovering state, the second term represents the rotor induced power, the two terms reflect the energy required for the blades to move when hovering, and the third term describes the energy consumed by the body due to air resistance. is the fuselage drag coefficient, is the air density, Indicates the rigidity of the rotor; is the rotor disk area, for Flight speed.

[0084] Then at time step , Hover energy consumption for collecting device data Expressed as:

[0085] ;

[0086] drones The flight power consumption is reflected in the fact that when the drone moves horizontally, it needs to consume extra energy to overcome air resistance and other loads. Expressed as:

[0087] ;

[0088] Among them, the first term on the right side of the equation represents the drag power generated by the rotor during flight, the second term represents the rotor induced power, and the third term represents the energy consumed by the body due to air resistance. for The induced velocity generated by the rotor when hovering, is the tip speed of the UAV blade.

[0089] At time step , Uniform straight-line flight distance Flight energy consumption Expressed as:

[0090] .

[0091] (4) Average AoI minimization optimization model

[0092] When Organization 1's own equipment is collecting data, if Organization 2's equipment is nearby, Organization 1's drone can help collect data at a lower cost, thus avoiding the higher energy consumption and time required by Organization 2's drone to fly long distances. However, the cost of this collaboration is that after Organization 1's drone collects data from Organization 2, it must incur the time and energy consumption of transferring it back to its own data center.

[0093] For the above scenarios, an AoI optimization model for multi-institutional collaboration to collect IoT data was constructed, with the goal of minimizing the average AoI of the system. , ensuring the rationality and efficiency of UAV-assisted data collection from an overall perspective. Therefore, the optimization problem is expressed as:

[0094] ;

[0095] ,

[0096] ,

[0097] ,

[0098] ,

[0099] ,

[0100] , ,

[0101] ;

[0102] in, represents the constraints of the optimization problem, and Represent the rotation angle and flight speed of the drone respectively, and the first constraint and the second constraint , Indicates the preset maximum flight speed. and Together they determine the drone’s performance when performing missions and represent the drone’s flight capabilities.

[0103] Energy consumption of IoT devices and drone energy consumption It is not unlimited, the third constraint indicates that the sum of energy consumption has an upper limit During the fourth and fifth constraints auxiliary data collection process, the drone cannot fly outside the maximum and minimum range of length and width and The target monitoring area ensures the range within which the UAV can perform its mission.

[0104] Using binary variables Indicates drone and IoT devices The connection relationship between them. , indicating that at the time step drones and IoT devices Establish data collection connection relationship, on the contrary, when In order to achieve the orderly and efficient collaborative data collection of multi-agency drones, the sixth constraint states that each drone can only collect data from one IoT device at a time step, and the seventh constraint states that each IoT device can only be collected by one drone at a time step. At the same time, since multiple drones are collecting data in the same area, the eighth constraint of safe flight distance must be met. and At time steps The positions of the two UAVs at that time, Indicates the minimum safe distance.

[0105] When extended to the case of more than two organizations, the description of the above optimization problem only needs to change the sixth and seventh constraints to the upper limit of the accumulation of the variable i to the number of organizations S, and the eighth constraint to , represents the L2 norm.

[0106] 3. Multi-organization UAV trajectory planning algorithm based on CC-MADDPG

[0107] This paper designs a deep deterministic policy gradient algorithm for time-cooperative multi-agent collaboration to solve the above problems.

[0108] (1) Basic ideas and framework

[0109] To address these challenges in multi-agency drone data collection, the present invention solves the problem based on the MADDPG algorithm, in which each drone participates in interaction and exploration as an intelligent agent. However, the traditional MADDPG algorithm still has certain limitations when faced with scenarios that require both maintaining the independence of data between agencies and requiring drones to collaborate to reduce the system's average AoI.

[0110] The MADDPG (Multi-Agent DDPG) algorithm is an actor-critic-based multi-agent reinforcement learning algorithm, extending the single-agent DDPG algorithm. It is suitable for scenarios with both collaboration and competition among agents. The algorithm employs the principle of centralized training and distributed execution. During training, each agent's value function (Critic) has access to global information, including the local observed states and actions of other agents, to learn optimal policies. During execution, each agent makes independent decisions based solely on its own local observations, achieving decentralization. Using an actor-critic architecture, the actor, as the policy network, receives local observations and outputs deterministic actions. The critic, as the Q-network, receives the combined states and actions of all agents and outputs a Q-value to evaluate the quality of the action.

[0111] The MADDPG algorithm, while guiding multiple drones to dynamically explore data collection paths, is insensitive to data collection time. Drones typically select actions based on their current location and the status of the devices to be collected, tailored to the amount of data collected or the task completion rate. This results in poor data timeliness. Furthermore, the independent optimization strategies within MADDPG's Q network can cause some devices to collect data repeatedly or not collect data for extended periods, increasing the system's average Area of ​​Interest (AoI). Furthermore, inefficient drone collaboration can lead to ineffective drone flights and excessive energy consumption. These issues hinder the algorithm's convergence speed and training effectiveness during the training phase, hindering its goal of minimizing the system's average Area of ​​Interest (AoI).

[0112] Therefore, to address this problem, the embodiment proposes a new CC-MADDPG algorithm with drones as drones. The algorithm integrates the collaborative goal of minimizing the system average AoI into the reward function of the MADDPG algorithm. The design effectively addresses the challenge faced by multi-agency drone data collection, requiring both data independence and collaborative efforts to reduce the system's average Area of ​​Interest (AoI). The CC-MADDPG algorithm minimizes the average Area of ​​Interest (AoI) of IoT devices by jointly planning flight trajectories for multiple drones.

[0113] (2) Detailed design of CC-MADDPG algorithm

[0114] This paper constructs a deep deterministic policy gradient algorithm model for time-cooperative intelligent agents with drones as intelligent agents. The model defines the global observation state as , the action space is recorded as , the reward is recorded as In each training round, each UAV observes the current environment to obtain the local observation state , select an action , give rewards , and enter the next local observation state The following is a detailed introduction to the design of the specific state space, action space, and reward function.

[0115] (a) State space

[0116] The state space of the system, that is, the global observation state, is composed of the states of IoT devices and drones. IoT devices are randomly distributed in space in an uneven manner, and generate monitoring data at the beginning of each time step. Before the data is successfully collected and transmitted back, its AoI will increase with time. When the data is successfully collected and transmitted back by the drone, its corresponding AoI is reset to 0. During the data collection process, IoT devices consume energy due to data transmission. In the drone-assisted IoT data collection task, the drone needs to fly within the area to complete the collection of all devices, and its position will change dynamically over time. At the same time, the flight state, communication process, and hovering data collection of the drone during the execution of the task all consume energy. Therefore, the local observation state of each drone Each time step should contain Information:

[0117] AoI of the device ;

[0118] Location of the device ;

[0119] Energy consumption of equipment ;

[0120] Data collection status of the device , indicating the time step drones Whether the device is collected data;

[0121] Drone location ;

[0122] Energy consumption of drones .

[0123] ; In summary, the global observation state of the system , also known as the joint state is represented as:

[0124] ;

[0125] This state space construction method can widely express the key features in multi-agency drone-assisted IoT data collection tasks by parameterizing the relationship between AoI requirements, location, and energy consumption, thereby providing contextual reasoning information with spatiotemporal perception capabilities for drone decision-making in complex environments, allowing drone energy consumption to make action decisions that are in line with the global optimality of the system based on the current environmental status and target requirements.

[0126] (b) Action space

[0127] Since the actions of the drone during data collection include: flying towards an IoT device and hovering over an IoT device to collect data. Therefore, the drone It can adjust its flight direction and speed to fly towards IoT devices or hover to collect data. Motion control includes the rotation angle of the drone. and flight speed , Defined the Adjust the flight angle of each drone, This limits the UAV to The maximum displacement within, i.e. the action of the i-th UAV .

[0128] In summary, the action space Expressed as:

[0129] This action space definition method uniformly characterizes the behavior of drones during mission execution and improves the adaptability of multi-agency drone-assisted data collection.

[0130] (c) Reward function

[0131] For the UAV-assisted IoT data collection task, the embodiment designs a comprehensive reward function to accurately evaluate the impact of UAV actions on system performance. The reward each UAV receives during training consists of four parts, including: system average AoI, energy consumption limit, multi-agency collaboration, and task completion. This design aims to fully reflect the synergy and individual contribution of UAVs, thereby guiding UAVs to make better action choices, thereby improving the data collection efficiency of the entire system and reducing the system's average AoI. Specific reward function Expressed as:

[0132] ;

[0133] in, , , and is a normalization factor used to balance the dimensions. Has the highest weight.

[0134] AoI minimization is the core of the reward, which is used to improve the drone's ability to perform tasks. To reflect the system average AoI, including rewards for average AoI reduction And the penalty when the average AoI exceeds the threshold Two parts, and are normalization factors:

[0135] ;

[0136] The reward for average AoI reduction can be measured by the difference between the system’s average AoI in the previous time step and the current time step, expressed as:

[0137] To prevent some devices from not being collected for a long time, causing the system's average AoI to continue to rise, a penalty is imposed when the average AoI exceeds the threshold, expressed as:

[0138] ;

[0139] in, is the system average AoI threshold.

[0140] In order to balance the energy consumption in the data collection task, the system energy consumption is also fed back to the drone as a negative reward, so as to guide the drone to spend less energy to complete the task. Expressed as:

[0141] ;

[0142] In order to achieve the goal of reducing the system's average AoI while spending less energy, we guide multi-institutional unmanned collaboration to complete data collection tasks. Including rewards for drones to help other agencies collect data , rewards for drones not overlapping their flight ranges to improve mission execution efficiency and avoid collision penalties when drones are too close , 、 and are normalization factors:

[0143] ;

[0144] Reward representation when drones collaborate to collect data for:

[0145] ;

[0146] in, The amount of data currently collected by drones for other structural equipment, is the total amount of data collected by drones of other institutions.

[0147] In order to improve the efficiency of data collection tasks, the flight paths of drones should not overlap with each other. This will most likely reduce the average AoI of the system while consuming less energy. Therefore, the reward given when the drones’ flight areas do not overlap is Expressed as:

[0148] ;

[0149] in, for The flight coverage area, Indicates the overlapping flight area. is a union.

[0150] To avoid collisions between drones, penalties are imposed when the distance between drones is less than the safe flight distance. Expressed as:

[0151] ;

[0152] The goal of the task is to collect data from all devices in the scene. Therefore, if the task is not completed, that is, if a device is missed or the drone only repeatedly collects data from a certain device, a penalty will be given. Report task completion, including penalties for missed tasks and duplicate collection penalties , and are normalization factors:

[0153] ;

[0154] ;

[0155] in, Indicates the number of devices that have been purchased. for The total number of devices in the organization.

[0156] This reward function combines the reduction of the system's average AoI, energy consumption constraints, and multi-agency collaboration to complete data collection tasks. It encourages drones to collaborate to complete the collection of all devices in the area. At the same time, it minimizes the system's average AoI under energy consumption constraints, thereby improving the efficiency of multi-agency drone-assisted data collection.

[0157] (d) CC-MADDPG model

[0158] For drone-assisted mobile IoT data collection, this paper proposes a CC-MADDPG algorithm to minimize the average AoI. CC stands for Chrono-Coordinated, and CC-MADDPG is a deep deterministic policy gradient algorithm with temporal coordination. The CC-MADDPG framework is identical to the traditional MADDPG framework, consisting of a policy actor network, a critic network, a target network (including a target actor network and a target critic network), an experience replay pool, and a gradient update function. The difference lies in the constraints (a), (b), and (c) above for the states, actions, and rewards in the CC-MADDPG algorithm.

[0159] (d1) Division of labor among CC-MADDPG components:

[0160] Strategy Actor Network: Each drone has an independent Actor network, which is based on the local observation state of the current drone. Output deterministic action .

[0161] Critic network: During centralized training, Critic receives global observation state and joint actions , the output Q value evaluates the quality of the global observation state-action pair.

[0162] Target Actor and Target Critic Networks: These are lagged copies of the Actor and Critic, respectively, used to stabilize training. Their parameters are gradually adjusted through soft updates (slowly tracking the online network) to avoid wild fluctuations in the target value.

[0163] Experience replay pool: Used when using centralized training, it stores information such as the local observation status and joint actions of all drones. It enables the algorithm to learn efficiently from past experience and maintain training stability in a non-stationary multi-drone environment.

[0164] Gradient update function: used to update the Actor network and Critic network, minimizing the mean squared error of the Actor network through gradient descent and maximizing the Q value given by the Critic network through gradient ascent. The gradient update function is the core mechanism that enables the MADDPG algorithm to learn effective multi-UAV strategies. Through the centralized criticism of the Critic and the distributed strategy actor update method, the algorithm can handle the non-stationary problems in multi-UAV environments.

[0165] Soft update: used to smooth the parameters of the target network and stabilize training.

[0166] (d2) Workflow

[0167] Interaction and environment sampling: Each drone selects an action based on the current local observation state through the Actor network, and adds exploration noise to the selected new action to solve the exploration problem caused by the deterministic strategy; after all drones execute the action, the environment returns to the next global observation state and rewards ; Transform the experience tuple ( , , , ) is stored in the experience replay pool; the global observation state and According to part (a), the action space According to part (b), the reward Calculated as in part (c).

[0168] Training phase: First, a batch of experience is sampled from the experience replay pool. Then, the critic network is updated by minimizing the mean squared error through gradient descent, and the actor network of each drone is updated by maximizing the Q value through gradient ascent. The online network parameters are periodically and slowly synchronized to the target network (target actor and target critic network) to achieve soft update. The above process is repeated until the training completion conditions are met.

[0169] During the deployment phase: UAVs plan data collection trajectories through their own trained Actor networks.

[0170] 4. Implementation Cases

[0171] In order to verify the effectiveness of the CC-MADDPG algorithm proposed in the present invention, the present invention implemented the above mechanism based on the Python platform.

[0172] (1) Implementation scenario and parameter setting

[0173] To verify the effectiveness of the CC-MADDPG algorithm in a multi-agency drone-assisted IoT data collection scenario, we established an implementation environment. The implementation scenario was set within a 20 km × 20 km square area. Two organizations each randomly and unevenly deployed 24 IoT devices to collect environmental data. Each organization also operated a drone to assist in data collection and transmit the data back to its own data center. The algorithm and implementation environment parameter settings are shown in Tables 1 and 2.

[0174] Table 1 CC-MADDPG algorithm parameter settings

[0175] Parameter Description Parameter settings actor network learning rate 0.0001 Critic network learning rate 0.001 epsilon value 0.1 Noise rate 0.1 Discount Factor 0.95 Soft update rate 0.01 Experience replay buffer size 500000 Number of experience replay samples 256

[0176] Table 2 Implementation environment parameter settings

[0177] Parameter Description Parameter value Baseline values ​​of LoS probability in remote mountainous environments 5 Sensitivity of remote mountainous environments to elevation changes 0.5 Path loss coefficient of LoS in remote mountainous environments 2 Path loss coefficient of nLoS in remote mountainous environments 5.5 Path loss index in remote mountainous environments 3 Altitude of the UAV 50m Altitude of the UAV 50m Noise power -100dBm Communication bandwidth 1MHz UAV data receiving power 0.3W UAV relay transmission power 0.8W UAV relay receiving power 0.5W UAV backhaul power 1W UAV communication power 1W UAV hovering power 150W UAV flight power 180W IoT communication power 0.5W UAV flight speed 15m / s

[0178] This paper then comprehensively evaluated the performance of the CC-MADDPG algorithm, implementing it under the same conditions as the non-cooperative DDPG algorithm and the traditional MADDPG algorithm, and conducting a comparative analysis of their performance. These comparative analyses demonstrate the effectiveness of the CC-MADDPG algorithm in reducing the system's average AoI and improving data collection performance in multi-institutional data collection scenarios.

[0179] (2) Implementation effectiveness analysis

[0180] During the training process, one episode contains 6,000 time steps, and a total of 500 episodes are trained. During this period, the drones of the two organizations use the CC-MADDPG algorithm to plan the flight trajectory of all IoT device data in the cooperative collection scene. Figure 3The following is a graph showing the convergence of CC-MADDPG. It shows that the average AoI is high and oscillates violently during the initial training phase. This is because the drones' policy parameters are not fully optimized in the initial stages, allowing them to explore the environment in a highly random manner and failing to learn a reasonable flight strategy. As training progresses, the experience replay buffer accumulates, and the CC-MADDPG algorithm shifts from random exploration to leveraging existing experience, resulting in a decrease in the average AoI. At this point, the critic network of each drone gradually learns the global Q-value function for the joint actions of multiple drones through centralized training, providing more accurate guidance for the policy gradient update of the actor network. As experience accumulates, the average AoI decreases and stabilizes after approximately 300 episodes, indicating that the CC-MADDPG algorithm is gradually converging. The CC-MADDPG algorithm's average AoI convergence trend demonstrates its ability to effectively guide the flight trajectories of multiple drones in collaborative data collection for multi-agency drone-assisted IoT data collection.

[0181] The average AoI (AoI) of the system is crucial in the collaborative data collection process among multiple organizations. Experiment 1 analyzed the flight trajectories of two organizations, one UAV and eight IoT devices, using the CC-MADDPG algorithm to guide data collection. This was compared to the DDPG algorithm, where each organization's UAV independently collected data from its own IoT devices. After all IoT devices were collected, the CC-MADDPG algorithm achieved an average AoI of 1775, compared to the 2725 average AoI of the DDPG algorithm, where each organization collected data independently. This demonstrates that the CC-MADDPG algorithm effectively leverages the collaborative advantages of the UAVs across organizations. Furthermore, the CC-MADDPG algorithm consumed 508 kJ of energy to collect data from all IoT devices, while the DDPG algorithm consumed 606 kJ. This demonstrates that the CC-MADDPG algorithm can effectively reduce the average AoI of the system while consuming less energy. Experiment 2 analyzed the flight trajectories of two organizations, one UAV and 12 IoT devices, using the CC-MADDPG and DDPG algorithms to guide data collection. At this point, after all IoT devices have been collected, the CC-MADDPG algorithm achieves an average AoI of 2168 and an energy consumption of 661, while the DDPG algorithm achieves an average AoI of 3466 and an energy consumption of 1043. Compared to the DDPG algorithm, the CC-MADDPG algorithm effectively reduces redundant UAV flight time, thereby lowering the system's average AoI while consuming less energy. Experiment 3 analyzed data collected by two UAVs from two organizations, each using the CC-MADDPG algorithm, to collaboratively collect 8 IoT device trajectories, totaling 16. The average AoI was 1032 and the energy consumption was 606, while the DDPG algorithm achieved an average AoI of 1407 and an energy consumption of 806. Compared to the data collected by only one UAV from each organization, the energy consumption of the two UAVs increased while the overall average AoI of the system decreased. Furthermore, the CC-DDPG algorithm still outperformed the DDPG algorithm in both average AoI and energy consumption when planning UAV trajectories. Experiment 4 analyzed the flight trajectories of two UAVs from two organizations, one from each of which collected data from 12 IoT devices, using the CC-MADDPG algorithm. The average AoI (AoI) was 2140, with an energy consumption of 890. The average AoI (AoI) using the DDPG algorithm was 2282, with an energy consumption of 1216. While collecting data from 24 IoT devices resulted in higher energy consumption and a higher average AoI, the CC-MADDPG algorithm still outperformed the baseline algorithm.

[0182] Figure 4The system's average AoI and energy consumption curves are presented as the number of IoT devices changes. At this point, the number of UAVs is 2. As the number of IoT devices increases from 16 to 48, or 8 IoT devices at a time, the system's average AoI increases with the density of IoT devices on the ground. However, the average AoI of the CC-MADDPG algorithm remains lower than that of the baseline algorithm, and the CC-MADDPG algorithm's growth is slower and more stable. While system energy consumption increases with the number of IoT devices, the CC-MADDPG algorithm's energy consumption increases slowly and remains lower than that of the baseline algorithm, demonstrating that the CC-MADDPG algorithm can reduce the system's average AoI while effectively conserving energy.

[0183] Figure 5 The system's average AoI curve is further presented as the number of UAVs changes. In this scenario, the number of IoT devices is 48. It can be seen that as the number of UAVs increases, the system's average AoI decreases, but the overall performance of the CC-MADDPG algorithm consistently outperforms the baseline algorithm. In summary, the CC-MADDPG algorithm can effectively guide UAV flight trajectory planning and reduce the system's average AoI in multi-agency UAV-assisted IoT data collection.

Claims

1. A drone-assisted IoT data collection method for multi-institutional overlapping deployment, characterized by: Including steps: 1) Interaction and environment sampling steps based on multi-agent deep deterministic policy gradient MADDPG: Each organization's drone collects local observation status, which includes the age of information (AoI), location, and energy consumption of all IoT devices within the drone's observation range at the current time step, as well as the drone's location, energy consumption, and collection status tag for each IoT device; After the local observation states of all the drones in the organization are combined to form the global observation state of the current time step, the global observation state is input into the initial policy Actor network of each drone. Each Actor network outputs an action, and exploration noise is added to the output action as the drone's execution action. The execution actions of all the drones in the organization are combined to form the joint action of the current time step. After each drone performs an action, it collects local observation states and combines them to obtain the global observation state for the next step, calculating the reward for the current time step. The reward function is designed based on the training objective of the actor network, minimizing the average age (AoI) of all IoT devices. The experience tuple consisting of the global observation state of the current time step, the joint action of the current time step, the reward of the current time step and the global observation state of the next step is stored in the experience replay pool; 2) MADDPG-based training steps: Extract training data from the experience replay pool and update the critic network via gradient descent. Update each drone's actor network via gradient ascent. Soft-update the target actor and target critic networks. Repeat this process until training is complete. 3) Deployment steps: Each organization’s drones use their trained Actor networks to plan cross-institutional data collection trajectories for IoT devices within their observation range.

2. The method according to claim 1, wherein: When designing a reward function with the goal of minimizing the average AoI of all IoT devices as the training objective for the Actor network, the reward function must also meet the following constraints: the action output by the Actor network is within the flight capabilities of the drone, the sum of the energy consumption of the IoT devices and the drone is less than the set system energy consumption limit, the drone's flight range is within the preset monitoring area, each drone can only collect information from at most one IoT device at the same time step, each IoT device can only be collected by at most one drone at the same time step, and the preset safe flight distance between drones from different organizations must be met.

3. The method according to claim 2, wherein: The action includes the rotation angle and flight speed of the drone.

4. The method according to claim 3, wherein: The action output by the Actor network is within the range of the drone's flight capability, specifically the range of the drone's rotation angle. , the flight speed range is , The preset maximum flight speed.

5. The method according to claim 1, wherein: When IoT device data is successfully collected, its AoI is reset to 0. If it is not collected, its AoI will gradually increase over time. In the At the end of the time step, the mechanism No. IoT devices The AoI is expressed as: ； in, Indicates that at time step At the end, IoT devices By the institution drones Collection, subscript All represent the institution serial number. The total number of institutions is , express Other circumstances besides The average AoI of the system within a time step is expressed as: ； Representation Agency The total number of IoT devices.

6. The method according to claim 1, wherein: When designing the reward function to minimize the average AoI of all IoT devices as the training objective of the Actor network, set the device AoI threshold , when the AoI of IoT devices exceeds , they will face punishment.

7. The method according to claim 1, wherein: In the time steps, reward function Expressed as: ； in, , , and is the normalization factor; Has the highest weight; To reflect the The average AoI of all IoT devices in the time step, including the reward for reducing the average AoI and the penalty when the average AoI exceeds the threshold; To be reflected in The energy consumption in the data acquisition task of time steps is used as a negative reward; For the The multi-agency collaboration reward for each time step includes rewards for the drone helping other agencies collect data, rewards for drones not overlapping in flight range, and penalties for drones being less than the safe flight distance. For the The task completion reward for each time step includes the penalty for missed collection and the penalty for repeated collection.

Images