Multi-unmanned aerial vehicle deployment and cooperative unloading method for Internet of Things

Abstract

The invention provides a multi-UAV (unmanned aerial vehicle) deployment and cooperative unloading method for the Internet of Things, and the method comprises the steps: firstly, converting an original joint optimization problem into a UAV deployment sub-problem and a calculation unloading sub-problem; thirdly, for the UAV deployment sub-problem, providing a UAV deployment scheme based on constrained K-Means clustering; the method comprises the following steps: introducing UAV coverage constraint into K-Means clustering, and adaptively adjusting the deployment position of the UAV to improve the coverage rate and coverage balance degree of an unloading service in a system; and finally, aiming at the calculation unloading sub-problem, providing a multi-UAV cooperative calculation unloading strategy based on MARL. Through a centralized training and decentralized execution mode, the proposed strategy realizes an approximately optimal calculation unloading and UAV cooperation strategy. According to the technical scheme, better UAV deployment performance can be achieved, and coverage balance is improved while the service coverage rate is increased.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things, and in particular to a multi-UAV deployment and collaborative unloading method for the Internet of Things. Background Art

[0002] With the rapid development of the Internet of Things (IoT) and communication technology, it has gradually become popular in scenarios such as smart cities, Industry 4.0, and smart agriculture. Hundreds of millions of sensors and smart devices have formed a large-scale IoT system, spawning many emerging smart applications (such as face recognition, virtual reality, etc.). However, limited by battery capacity and hardware performance, IoT devices are difficult to meet the high real-time requirements of these computing-intensive smart applications. Although classic cloud computing can provide abundant computing and storage resources, long-distance communication significantly increases data transmission delays and seriously affects the quality of service (QoS). By sinking resources to the edge of the network, the emergence of Mobile Edge Computing (MEC) can effectively reduce data transmission delays, reduce the bandwidth load of the core network, and thus improve the performance and user experience of the IoT system.

[0003] The communication of existing IoT systems mainly relies on ground infrastructure (such as base stations). However, when facing remote areas (such as forests and oceans) and emergency situations (such as disaster relief and military exercises), the ground infrastructure may not be able to meet the communication needs. As an aerial mobile device, unmanned aerial vehicles (UAVs) can be applied to more diverse application scenarios. Thanks to the mobility and communication capabilities of UAVs, they can assist MEC to realize the construction of aerial networks and provide more flexible computing offloading services for IoT devices. However, due to the uneven distribution of IoT devices and the unreasonable deployment of UAVs, the load between UAVs may be unbalanced, resulting in increased system latency and excessive energy consumption of UAVs. Therefore, how to design an efficient multi-UAVs deployment and collaborative offloading method still faces the following challenges:

[0004] (1) Uneven distribution of IoT devices leads to low resource efficiency of UAVs. The distribution of IoT devices is usually diverse and uneven, but UAVs can only provide computing offload services for IoT devices within a limited coverage area. Therefore, unreasonable UAV deployment schemes will seriously affect the number of IoT devices that can be served, and also lead to large differences in service coverage in different regions.

[0005] (2) Diverse offloading requests lead to significant differences in load between UAVs. The number of requests and computing requirements of IoT devices within the coverage range of different UAVs vary greatly, which leads to large differences in load conditions on different UAVs and significant differences in resource utilization between different UAVs.

[0006] (3) Complex problems caused by large-scale IoT systems are difficult to solve efficiently. Offloading requests from a large number of IoT devices and real-time collaboration between UAVs significantly increase the complexity of the problem, making it difficult to solve efficiently. Existing solutions usually use control theory or iterative algorithms, resulting in excessive overhead. Some research works using classic centralized reinforcement learning are plagued by problems such as explosion of action dimensions and low exploration efficiency. Summary of the invention

[0007] In view of this, the purpose of the present invention is to provide a multi-UAV deployment and collaborative offloading method for the Internet of Things. First, to achieve better UAV deployment performance, while improving service coverage and coverage balance. Second, to efficiently perform computation offloading and UAV collaboration to reduce the total task processing delay and UAV energy consumption.

[0008] To achieve the above-mentioned purpose, the present invention adopts the following technical solutions: a multi-UAV deployment and collaborative offloading method for the Internet of Things, first, the original joint optimization problem is converted into a UAV deployment subproblem and a computational offloading subproblem; then, for the UAV deployment subproblem, a UAV deployment scheme based on constrained K-Means clustering is proposed; by introducing the UAV coverage range constraint into K-Means clustering, the deployment position of the UAV is adaptively adjusted to improve the coverage and coverage balance of the computational offloading service in the system; finally, for the computational offloading subproblem, a multi-UAV collaborative computational offloading strategy based on MARL is proposed; through the centralized training and decentralized execution mode, the proposed strategy achieves approximately optimal computational offloading and UAV collaboration strategy.

[0009] In a preferred embodiment, a UAV deployment and collaborative offloading system for IoT devices is included, wherein the UAV deployment and collaborative offloading system for IoT devices is composed of UAVs and IoT devices; IoT devices request computing offloading services from UAVs via wireless links, denoted as a set U = {u 1 ,u 2 ,...,u n}, the horizontal coordinate is represented by {w 1 ,w 2 ,...,w n}; UAVs provide computing offloading services to IoT devices within their coverage area, denoted as the set UAV = {uav 1 ,uav2 ,...,uav m}, the horizontal coordinate is represented by {q 1 ,q 2 ,...,q m}; The horizontal height of UAVs is H, then u i(1≤i≤n) With UAV j(1≤j≤m) The distance between them is:

[0010]

[0011] Collaborative offloading is required between multiple UAVs, and UAVs forward tasks from IoT devices to their collaborative UAVs for execution; the system time slot is represented as t∈{1,2,...,T}, and at the beginning of each time slot, the IoT device chooses to execute its computing request locally or offload it to the UAVs connected to it for execution; if the IoT device is not within the coverage of any UAVs, its computing request is executed locally.

[0012] In a preferred embodiment, at time slot t, the i The task is defined as a triple <D i [t],S i [t],c i >, where D i [t] represents the amount of input data for the task, S i [t] represents the computational complexity of the task, c i represents the associated indicator variable; when u i Connect to UAV j When c i =j; when u i When unable to connect to an available UAV, c i = 0; tasks from IoT devices can be executed locally or offloaded to UAVs; u i The unloading decision at time slot t is denoted as α i [t]∈{0,1}; when u i Executing the task locally or unable to connect to an available UAV (c i =0), α i [t]=0; when u i When offloading the task to a UAV for execution, α i [t] = 1; specifically, it is divided into the following two calculation modes;

[0013] (1) Local computing mode

[0014] When the task is executed locally, its latency is:

[0015]

[0016] Among them, f i Indicates u i computing power;

[0017] (2) UAV computing model

[0018] IoT devices access UAVs via OFDMA; when u i The task is offloaded to uav j When the input data is uploaded first, according to Shannon's theorem, u i With UAV j The data transfer rate between is:

[0019]

[0020] Among them, B u represents the channel bandwidth between IoT devices and UAVs, P i Represents u i The transmission power of where ρ 0 Represents the channel gain per unit distance, N 0 represents the noise power spectral density;

[0021] Therefore, u i Offload tasks to uav j The transmission delay is:

[0022]

[0023] Next, UAV j Need to process concurrently j [t] tasks; therefore, from u i The mission in UAV j The computational delay on is:

[0024]

[0025] in, f max Indicates the computing power of a UAV;

[0026] Tasks on high-load UAVs can be forwarded to low-load UAVs to achieve load balancing; specifically, UAV j The cooperative offloading decision at time slot t is defined as β ijk [t]∈{0,1}; when uav j Decided to put u i The task is forwarded to uav k When ijk [t] = 1; otherwise, β ijk[t] = 0; a task is forwarded at most once, denoted by When the task is from uav j Forward to uav k When cooperative offloading is performed, the data transfer rate between the two is:

[0027]

[0028] Among them, B uav represents the channel bandwidth between UAVs, P uav represents the transmission power between UAVs, For UAV j With UAV k Therefore, u i The mission from UAV j Forward to uav k The transmission delay is:

[0029]

[0030] The forwarded task is in uav k The computational delay on is:

[0031]

[0032] Execute u i The task delay is:

[0033]

[0034] Processing i The total delay of the task is:

[0035]

[0036] In summary, the total delay for processing all tasks in time slot t is:

[0037]

[0038] In a preferred embodiment, when tasks are offloaded to UAVs for processing, three aspects of energy consumption are generated, including the energy consumption of UAVs executing tasks, the energy consumption of forwarding tasks between UAVs, and the energy consumption of UAVs maintaining a hovering state;

[0039] uav j Execute u i The energy consumption of the task is:

[0040]

[0041] Where κ represents the effective capacitance coefficient;

[0042] uav j will u i Task forwarding to uav k The energy consumption is:

[0043]

[0044] The energy consumption of UAVs to maintain the hovering state in each time slot is e; therefore, the total energy consumption of all UAVs in time slot t is:

[0045]

[0046] in, Indicates uav j The set of tasks to be performed, Indicates from uav j The set of tasks forwarded;

[0047] Due to the limited power of UAVs, they cannot always provide computing offload services for IoT devices; when the power is lower than the threshold b min When the UAVs are in a state of low latency, they will no longer provide computing offload services and will return to their home base for battery life.

[0048] Based on the above proposed model, the optimization goal is to minimize the weighted sum of the total task processing delay and the UAV energy consumption; the optimization problem is formally defined as:

[0049]

[0050] Among them, λ 1 and λ 2 Respectively represent the weights of delay and energy consumption, b j [t] indicates uav j The remaining power at time slot t, b min represents the minimum power that UAVs must retain; C1 and C2 represent the value range constraints for offloading decisions; C3 represents the number of times a task is forwarded between UAVs; C4 represents the distance constraint between the IoT device and its associated UAV, where R is the coverage radius of the UAVs; C5 represents the value range constraint of the associated indicator variable; constraint C6 indicates that the computing resources used by UAVs to execute tasks cannot exceed their maximum computing resource constraints; C7 indicates that UAVs need to retain the minimum power b when providing computing offloading services. min Used for returning to the voyage and continuing the power supply.

[0051] In a preferred embodiment, it is further divided into two sub-problems: UAV deployment and computational offloading. First, UAV deployment optimization is performed to improve the coverage and coverage balance of UAVs computational offloading services. After UAVs are deployed, computational offloading optimization is performed to reduce the total task processing latency and UAV energy consumption.

[0052] In a preferred embodiment, for the UAV deployment problem, IoT devices are divided into multiple clusters, and then UAVs are deployed at the centroid of each cluster to maximize the service coverage; service coverage and coverage balance are defined as performance indicators of UAV deployment, and then a UAV deployment scheme based on constrained K-Means clustering is designed; specifically, clusters are divided based on UAV coverage so that the cluster size does not exceed the UAV coverage; at the same time, the balance of the number of IoT devices within the coverage of each UAV is considered when dividing the clusters.

[0053] In a preferred embodiment, in each iteration, the positions of UAVs are first randomly initialized, and the cluster set change flag is initialized, which is used to check whether the cluster division has reached convergence; then, the IoT devices within the coverage range of the UAVs are allocated to the clusters of the corresponding UAVs; specifically, unlike the classic K-Means algorithm that allocates data points based on the principle of minimum distance to the centroid, when allocating IoT devices, whether the IoT device is within the coverage radius of the UAVs is considered; when an IoT device is covered by multiple UAVs at the same time, the nearest UAV is selected and the change is marked, and the number of UAVs covered is updated; then, the current coverage rate and the load level of all UAVs are calculated, which are defined as:

[0054]

[0055] When the coverage is close to equilibrium, F 1 ≈F 2 ≈...≈F m ; The mean square error is used to evaluate the coverage balance, which is defined as:

[0056]

[0057] in,

[0058] Next, the coverage rate and coverage balance are normalized to the same numerical range; the mean of each cluster is used as the new centroid and UAVs are deployed; through iteration, the deployment location of UAVs will gradually tend to areas where IoT devices are densely distributed, enabling them to serve more IoT devices; after the iteration, the solution with the smallest weighted sum of coverage rate and coverage balance is selected for UAV deployment.

[0059] In a preferred embodiment, after solving the UAV deployment, the computation offloading problem is further separated from P1, which is expressed as:

[0060]

[0061] A multi-UAV collaborative computing offloading strategy based on MARL is proposed: the proposed multi-UAV collaborative offloading system is regarded as an environment, and multiple agents interact with the environment at the same time and select corresponding offloading actions; after receiving the reward signal fed back by the environment, each actor network is updated in a distributed manner; the critic network adopts a centralized update method; accordingly, the state space, action space and reward function are defined as follows:

[0062] State space: The state space includes the task attributes, the computing power of IoT devices, and the i , data transmission power P of IoT devices i , the remaining power of UAV b[t]; therefore, at time slot t, the state observed by agent j is expressed as:

[0063]

[0064] Among them, 1≤j≤m, and Respectively represent the set consisting of the data volume and computational complexity of all tasks;

[0065] Action space: The action space contains the offloading decision α[t] of the IoT device and the collaborative offloading decision β[t] of the UAVs. α[t] and β[t] are integrated, and the integrated offloading decision is recorded as η[t]∈{0,1,2,...,m}. Therefore, at time slot t, the action of agent j is expressed as:

[0066]

[0067] in, represents the offloading decision made by agent j for all tasks; η i [t] = 0 means u i No task is generated or the agent decides to execute the task locally; η i [t]=k, k∈{1,2,...,m} means the agent decides to offload the task to the uav k Execute on;

[0068] Reward function: The optimization objective of P2 is to minimize the weighted sum of the total task processing delay and the UAV energy consumption; therefore, at time slot t, the reward function is expressed as:

[0069] r[t]=-(λ 1 N(T u_tot [t])+λ2 N(E uav_tot [t])), (22)

[0070] Among them, λ 1 and λ 2 They represent the weights of delay and energy consumption respectively; N is a normalization function, which normalizes delay and energy consumption to the same numerical range.

[0071] In a preferred embodiment, the value network acts as a critic and estimates the advantage function of the state-action pair using multi-step temporal difference; the policy network acts as an actor to update the policy according to the advantage function; the obtained UAVs coordinates q and the associated indicator variable c will be used as the input of Algorithm 2; specifically, the value and policy networks are first initialized; in each round of iteration, each agent obtains the initial state of the environment, initializes the round end mark and the interaction data record table, where the interaction data record table is used to collect all data on the interaction between the agent and the environment in the current round; in each round, each agent selects a suitable unloading action according to its strategy, and converts η into the unloading decision α of the corresponding IoT device and the collaborative unloading decision β of the UAVs; then, α and β are executed and the reward, state and round end mark are updated; the round ends when the power of all UAVs reaches the minimum threshold, that is: when When , done = True; then, collect the interaction data of each agent and update its state; then, use the interaction data collected in the current round to train the agent; specifically, take out the interaction data and concatenate the states for centralized training of the value network, whose generalized advantage function is defined as:

[0072]

[0073] Among them, γ is the discount factor and introduces a hyperparameter When λ=0, it is the advantage obtained by one step of differentiation; when λ=1, it is the complete average of the advantages obtained by each step of differentiation; by using the advantage function, the variance of the policy gradient estimation can be significantly reduced and the stability of the training process can be improved;

[0074] Next, the strategy obtained in the previous round is used as the old strategy for this round, and the action probability distribution of each agent is calculated; then, the same batch of data is used to update the network parameters multiple times; unlike other DRL algorithms, the previously sampled data is discarded after updating the network parameters. The proposed method converts it into a training process based on different strategies by setting new and old strategies; specifically, data is collected by interacting with the old strategy and the environment, and then the new strategy is trained using these data; according to the principle of importance sampling, the data sampled by the old strategy can be used multiple times, and gradient ascent can be performed multiple times for policy updates; after each update of the policy network parameters, the action probability distribution of each agent and the loss function of the policy network are calculated; in particular, clipping is used to limit the amplitude of each policy update to ensure that the strategy does not change drastically during training. Finally, the loss function of the value network is calculated and the parameters of the policy and value networks are updated using gradient descent.

[0075] Compared with the prior art, the present invention has the following beneficial effects: the multi-UAV deployment and cooperative unloading method for the Internet of Things proposed by the present invention can achieve better UAV deployment performance in different scenarios, improve coverage balance while improving service coverage. In addition, the multi-UAV deployment and cooperative unloading method for the Internet of Things can make better unloading decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 A schematic diagram of a UAV deployment and collaborative offloading system for IoT devices according to a preferred embodiment of the present invention;

[0077] Figure 2 This is a schematic diagram of multi-UAV collaborative unloading based on MARL in a preferred embodiment of the present invention;

[0078] Figure 3 Schematic diagrams of three scenarios with different IoT device distributions according to a preferred embodiment of the present invention, wherein (a) is scenario 1, (b) is scenario 2, and (c) is scenario 3;

[0079] Figure 4 Schematic diagrams of UAV deployment in different scenarios of a preferred embodiment of the present invention, wherein (a) is scenario 1, (b) is scenario 2, and (c) is scenario 3;

[0080] Figure 5 Schematic diagram of UAV deployment performance comparison of different methods of the preferred embodiment of the present invention, wherein (a) is the weighted sum of coverage rate and coverage balance, (b) is the coverage rate, and (c) is the coverage balance;

[0081] Figure 6 This is a schematic diagram of average time slot rewards of different methods in a preferred embodiment of the present invention.

[0082] Figure 7 Schematic diagram of average time slot delay of different methods in a preferred embodiment of the present invention,

[0083] Figure 8 This is a schematic diagram of average time slot UAV energy consumption of different methods in a preferred embodiment of the present invention.

[0084] Fig. 9 In the preferred embodiment of the present invention, max Below is a diagram of the average time slot rewards for different methods.

[0085] Fig.10 Schematic diagram of the loads of different UAVs in a preferred embodiment of the present invention.

[0086] Fig.11 A schematic diagram of load balancing degrees of different methods according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0087] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0088] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0089] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0090] refer to Figure 1-11, this paper proposes MUCO, which is a new multi-UAV deployment and collaborative offloading method for IoT systems. First, the original joint optimization problem is transformed into two sub-problems: UAV deployment and computation offloading. Then, for the UAV deployment sub-problem, a UAV deployment scheme based on constrained K-Means clustering is proposed. By introducing the UAV coverage range constraint into K-Means clustering, the proposed scheme can adaptively adjust the deployment location of UAVs to improve the coverage and coverage balance of computation offloading services in the system. Finally, for the computation offloading sub-problem, a multi-UAV collaborative computation offloading strategy based on MARL is proposed. Through the centralized training and decentralized execution mode, the proposed strategy can achieve approximately optimal computation offloading and UAV collaboration strategy. In particular, the distributed task execution mode can be better adapted to large-scale IoT systems. A large number of experiments verify the effectiveness of the proposed MUCO method, which can perform adaptive UAV deployment and collaborative offloading according to the distribution changes of IoT devices. Compared with the baseline method, the proposed MUCO method can achieve better UAV deployment performance in different scenarios, improving coverage balance while improving service coverage. In addition, the MUCO method can make better offloading decisions and achieve lower weighted sum of latency and energy consumption under the UAV power constraint.

[0091] System Model and Problem Definition

[0092] like Figure 1 As shown in the figure, the proposed UAV deployment and collaborative offloading system for IoT devices consists of UAVs and IoT devices. IoT devices request computing offloading services from UAVs through wireless links, denoted as the set U = {u 1 ,u 2 ,...,u n}, the horizontal coordinate is represented by {w 1 ,w 2 ,...,w n UAVs can provide computing offloading services for IoT devices within their coverage area, denoted as the set UAV = {uav 1 ,uav 2 ,...,uav m}, the horizontal coordinate is represented by {q 1 ,q 2 ,...,q m}. The horizontal height of UAVs is H, then u i(1≤i≤n) With UAV j(1≤j≤m) The distance between them is:

[0093]

[0094] In order to improve the resource utilization efficiency of the system, multiple UAVs need to collaborate to offload. Specifically, UAVs can forward tasks from IoT devices to their collaborative UAVs for execution. The system time slot is represented as t∈{1,2,...,T}. At the beginning of each time slot, the IoT device can choose to execute its computing request locally or offload it to the UAVs connected to it. If the IoT device is not within the coverage of any UAVs, its computing request is executed locally.

[0095] Computation and Communication Model

[0096] At time slot t, from u i The task is defined as a triple <D i [t],S i [t],c i >, where D i [t] represents the amount of input data for the task, S i [t] represents the computational complexity of the task (i.e., the number of computing resources required per unit of computing task input: cycle / bit), c i represents the associated indicator variable. i Connect to UAV j When c i =j; when u i When unable to connect to an available UAV, c i = 0. Tasks from IoT devices can be executed locally or offloaded to UAVs. i The unloading decision at time slot t is denoted as α i [t]∈{0,1}. When u i Executing the task locally or unable to connect to an available UAV (c i =0), α i [t]=0; when u i When offloading the task to a UAV for execution, α i [t] = 1. Specifically, it can be divided into the following two calculation modes.

[0097] (1) Local computing mode

[0098] When the task is executed locally, its latency is:

[0099]

[0100] Among them, f i Indicates u i computing power.

[0101] (2) UAV computing model

[0102] IoT devices access UAVs through OFDMA. i The task is offloaded to uav j When u i With UAV j The data transfer rate between is:

[0103]

[0104] Among them, B u represents the channel bandwidth between IoT devices and UAVs, P i Represents u i transmission power. where ρ 0 Represents the channel gain per unit distance, N 0 represents the noise power spectral density.

[0105] Therefore, u i Offload tasks to uav j The transmission delay is:

[0106]

[0107] Next, UAV j Need to process concurrently j [t] tasks. Therefore, from u i The mission in UAV j The computational delay on is:

[0108]

[0109] in, f max Indicates the computing power of a UAV.

[0110] To improve the resource utilization efficiency of the system, tasks on high-load UAVs can be forwarded to low-load UAVs to achieve load balancing. j The cooperative offloading decision at time slot t is defined as β ijk [t]∈{0,1}. When uav j Decided to put u i The task is forwarded to uav k When ijk [t] = 1; otherwise, β ijk [t] = 0. A task is forwarded at most once, denoted by When the task is from uav j Forward to uav k When cooperative offloading is performed, the data transfer rate between the two is:

[0111]

[0112] Among them, B uav represents the channel bandwidth between UAVs, P uav represents the transmission power between UAVs, For UAV j With UAV k Therefore, u i The mission from UAV j Forward to uav k The transmission delay is:

[0113]

[0114] The forwarded task is in uav k The computational delay on is:

[0115]

[0116] Compared with the input of the task, the amount of data of the calculation result is usually small, so the delay of returning the result can be ignored. i The task delay is:

[0117]

[0118] Furthermore, processing u i The total delay of the task is:

[0119]

[0120] In summary, the total delay for processing all tasks in time slot t is:

[0121]

[0122] Energy consumption model

[0123] UAVs consume a lot of energy when providing computing offload services. Since UAVs have limited power, their energy efficiency needs to be improved to better complete tasks from IoT devices. When tasks are offloaded to UAVs for processing, three aspects of energy consumption may be generated, including the energy consumption of UAVs executing tasks, the energy consumption of forwarding tasks between UAVs, and the energy consumption of UAVs maintaining a hovering state.

[0124] uav j Execute u i The energy consumption of the task is:

[0125]

[0126] Here, κ represents the effective capacitance coefficient.

[0127] uav j will u i Task forwarding to uav k The energy consumption is:

[0128]

[0129] The energy consumption of UAVs to maintain hovering state in each time slot is e. Therefore, the total energy consumption of all UAVs in time slot t is:

[0130]

[0131] in, Indicates uav j The set of tasks to be performed, Indicates from uav j The set of tasks forwarded.

[0132] Due to the limited power of UAVs, they cannot always provide computing offload services for IoT devices. min When the UAVs are in flight, they will no longer provide computing offload services and return to their home base for battery recovery.

[0133] Based on the above proposed model, the optimization goal of the present invention is to minimize the weighted sum of the total task processing delay and the UAV energy consumption. Therefore, the optimization problem is formally defined as:

[0134]

[0135] Among them, λ 1 and λ 2 Respectively represent the weights of delay and energy consumption, b j [t] indicates uav j The remaining power in time slot t, b min Indicates the minimum amount of power that UAVs must retain. C1 and C2 represent the value range constraints for offloading decisions. C3 represents the number of times a task is forwarded between UAVs. C4 represents the distance constraint between IoT devices and their associated UAVs, where R is the coverage radius of the UAVs. C5 represents the value range constraint of the associated indicator variable. Constraint C6 indicates that the computing resources used by UAVs to perform tasks cannot exceed their maximum computing resource constraints. C7 indicates that UAVs need to retain a minimum amount of power b when providing computing offloading services. min Used for returning to the voyage and continuing the power supply.

[0136] In order to solve P1 more efficiently, the present invention further splits it into two sub-problems: UAV deployment and computational offloading. First, UAV deployment optimization is performed to improve the coverage and coverage balance of UAVs computational offloading services. IoT devices that are not within the coverage of UAVs can only perform tasks locally, and a large amount of local computing will cause excessive latency, so improving coverage helps reduce system latency. In addition, the uneven distribution of IoT devices leads to unbalanced loads on different UAVs. If the coverage balance can be improved, the number of tasks offloaded to each UAV will be more even, which can reduce the number of task forwardings between UAVs and reduce transmission latency. After deploying UAVs, computational offloading optimization is performed to obtain the best offloading and collaborative offloading decisions.

[0137] The proposed MUCO method

[0138] This paper proposes a novel Multi-UAV deployment and collaborative offloading (MUCO) method for IoT systems. Firstly, a UAV deployment scheme based on constrained K-Means clustering is designed to improve the service coverage and coverage balance. Then, a multi-UAV collaborative offloading strategy based on MARL is developed to reduce the total task processing delay and UAV energy consumption.

[0139] UAV deployment based on constrained K-Means clustering

[0140] Considering the superior partitioning clustering effect of the K-Means algorithm, the present invention introduces it for the UAV deployment problem. For the UAV deployment problem, IoT devices can be divided into multiple clusters, and then UAVs are deployed at the centroid of each cluster to maximize the service coverage. The sizes and densities of the clusters obtained by the classic K-Means algorithm may vary greatly, and a single UAV may not be able to cover a cluster well. Therefore, if UAVs are deployed directly at the centroid, the number of IoT devices served by each UAV may differ too much, which will seriously affect the quality of UAV deployment.

[0141] In order to solve this problem, the present invention defines service coverage and coverage balance as performance indicators of UAV deployment, and then designs a UAV deployment scheme based on constrained K-Means clustering. Specifically, clusters are divided based on the UAV coverage range so that the cluster size does not exceed the UAV coverage range. At the same time, the balance of the number of IoT devices within the coverage range of each UAV is considered when dividing the clusters. The main steps are shown in Algorithm 1.

[0142]

[0143]

[0144] In each round of iteration, the positions of UAVs are first randomly initialized, and the cluster set change flags are initialized (lines 3-4), which are used to check whether the cluster division has reached convergence. Next, the IoT devices within the coverage range of the UAVs are assigned to the clusters of the corresponding UAVs (lines 7-15). Specifically, unlike the classic K-Means algorithm that allocates data points based on the principle of minimum distance to the center of mass, the present invention considers whether the IoT device is within the coverage radius of the UAVs when allocating IoT devices (line 9). When an IoT device is covered by multiple UAVs at the same time, the nearest UAV will be selected and the change will be identified (lines 10-12), and the number of UAVs covered will be updated (line 13). Next, the current coverage rate and the load level of all UAVs are calculated (lines 16-17), which are defined as:

[0145]

[0146] When the coverage is close to equilibrium, F 1 ≈F 2 ≈...≈F m The mean square error is used to evaluate the coverage balance (line 18), which is defined as:

[0147]

[0148] in,

[0149] Next, the coverage rate and coverage balance are normalized to the same numerical range (line 20). The mean of each cluster is used as the new centroid and UAVs are deployed (lines 21-23). ​​Through iteration, the deployment location of UAVs will gradually tend to areas where IoT devices are densely distributed, enabling them to serve more IoT devices. After the iteration, the solution with the smallest weighted sum of coverage rate and coverage balance is selected for UAV deployment (line 26).

[0150] Multi-UAV cooperative offloading based on MARL

[0151] After solving the UAV deployment, the computation offloading problem can be further separated from P1, which is expressed as:

[0152]

[0153] Most of the existing work on computation offloading adopts centralized DRL, which can make reasonable decisions in small-scale action spaces. However, as the scale of IoT systems continues to increase, this approach will lead to dimensionality explosion and unacceptable training complexity. The emergence of MARL provides a new idea to solve this problem. A large number of offloading requests from users can be processed by multiple agents separately, which reduces the training complexity and speeds up the decision-making speed to better meet the high real-time requirements of IoT systems. At the same time, the joint strategy of centralized training and distributed execution can promote collaboration between agents to improve system performance while maintaining state consistency. Specifically, centralized training enables multiple agents to learn from a shared global perspective and accelerate their learning process by sharing experience during training. Distributed execution enables each agent to make decisions based on individual perception and local information, thereby achieving rapid response and adaptation to the environment, which also significantly reduces the size of each agent's state and action space and simplifies the complexity of training.

[0154] Based on the above analysis, this paper proposes a multi-UAV collaborative computing offloading strategy based on MARL to optimize system latency and energy consumption. Figure 2 As shown in the figure, the proposed multi-UAV cooperative unloading system is regarded as an environment, and multiple agents interact with the environment at the same time and select corresponding unloading actions. In order to reduce the training complexity of each agent, each actor network is updated in a distributed manner after receiving the reward signal fed back by the environment. In order to better grasp the system state information as a whole, the critic network adopts a centralized update method. Accordingly, the state space, action space and reward function are defined as follows.

[0155] State space: The state space includes the task attributes, the computing power of IoT devices, and the i , data transmission power P of IoT devices i , the remaining power of the UAV b[t]. Therefore, at time slot t, the state observed by agent j (1≤j≤m) is expressed as:

[0156]

[0157] in, and They respectively represent the set consisting of the data volume and computational complexity of all tasks.

[0158] Action space: The action space contains the offloading decision α[t] of the IoT device and the collaborative offloading decision β[t] of the UAVs. In order to improve the training efficiency, α[t] and β[t] are integrated, and the integrated offloading decision is recorded as η[t]∈{0,1,2,...,m}. Therefore, at time slot t, the action of agent j is expressed as:

[0159]

[0160] in, represents the offloading decision made by agent j for all tasks. i [t] = 0 means u i No task is generated or the agent decides to perform the task locally. i [t]=k, k∈{1,2,...,m} means the agent decides to offload the task to the uav k Execute on.

[0161] Reward function: The optimization objective of P2 is to minimize the weighted sum of the total task processing delay and the UAV energy consumption. Therefore, at time slot t, the reward function is expressed as:

[0162] r[t]=-(λ 1 N(T u_tot [t])+λ 2 N(E uav_tot [t])), (22)

[0163] Among them, λ 1 and λ 2 They represent the weights of delay and energy consumption respectively. N is a normalization function, which normalizes delay and energy consumption to the same value range.

[0164] The main steps of the proposed MARL-based multi-UAV cooperative offloading strategy are shown in Algorithm 2.

[0165]

[0166]

[0167] Algorithm 2 is designed based on the actor-critic DRL framework. Among them, the value network acts as a critic and uses multi-step temporal differences to estimate the advantage function of the state-action pair; the policy network acts as an actor to update the strategy according to the advantage function. The UAVs coordinates q and the associated indicator variable c obtained by Algorithm 1 will be used as the input of Algorithm 2. Specifically, the value and policy networks are initialized first (line 1). In each round of iteration, each agent obtains the initial state of the environment (line 3), initializes the round end flag and the interaction data record table (lines 4-5), where the interaction data record table is used to collect all data of the interaction between the agent and the environment in the current round. In each round, each agent selects a suitable unloading action according to its strategy (line 7), and converts η into the unloading decision α of the corresponding IoT device and the collaborative unloading decision β of the UAVs (line 8). Next, execute α and β and update the reward, state and round end flag (line 9). The round ends when the power of all UAVs reaches the minimum threshold, that is: when When , done = True. Then, collect the interaction data of each agent and update its state (lines 10-11). Then, use the interaction data collected in the current round to train the agent (lines 13-22). Specifically, take out the interaction data and concatenate the states for centralized training of the value network (lines 13-15), whose generalized advantage function is defined as:

[0168]

[0169] Among them, γ is the discount factor and introduces a hyperparameter When λ = 0, it is the advantage obtained by one step of differentiation; when λ = 1, it is the complete average of the advantages obtained by each step of differentiation. By using the advantage function, the variance of the policy gradient estimation can be significantly reduced and the stability of the training process can be improved.

[0170] Next, the strategy obtained in the previous round is used as the old strategy for this round, and the action probability distribution of each agent is calculated (line 16). Subsequently, the same batch of data is used to update the network parameters multiple times (lines 17-22). Unlike other DRL algorithms, the previously sampled data is discarded after the network parameters are updated. The proposed method converts it into a training process based on different strategies by setting the old and new strategies. Specifically, data is collected by interacting with the old strategy and the environment, and then the new strategy is trained using these data. According to the principle of importance sampling, the data sampled by the old strategy can be used multiple times, and gradient ascent can be performed multiple times for policy update. This design significantly reduces the time overhead of data sampling and improves the utilization of the same batch of data. After each update of the policy network parameters, the action probability distribution of each agent and the loss function of the policy network are calculated (lines 18-19). In particular, clipping is used to limit the amplitude of each policy update to ensure that the policy does not change drastically during the training process, which avoids instability caused by excessive updates. Finally, the loss function of the value network is calculated and the parameters of the policy and value networks are updated using gradient descent (lines 20-21).

[0171] Method evaluation

[0172] The simulation experiment was carried out on a workstation equipped with Intel (R) Xeon (R) Silver 4208CPU, NVIDIA Getforce GTX 3090GPU, where the CPU clock frequency is 2.10GHz and the memory is 96GB. The proposed system model and method are implemented based on Python, and the main simulation parameter settings are shown in Table 1. In order to better simulate the dynamics of the environment, the task traffic is set to change with time slots in the experiment (that is, different IoT devices generate tasks in each time slot), and the number of tasks and task attributes also change with time slots. The number of time slots in a round depends on the time span of all UAVs from full power to the minimum power threshold.

[0173] Table 1 Parameter settings

[0174]

[0175]

[0176] In the simulation experiment, all IoT devices are distributed in 800×800m 2 The number of IoT devices is n = 150, and the number of UAVs is m = 4. Figure 3 As shown, three scenarios with different IoT device distributions are considered, and the settings are as follows:

[0177] (1) Scenario 1: Figure 3 As shown in (a), the scene contains a local area with a high density of IoT devices. About 90% of IoT devices are concentrated in this local area, and the remaining about 10% of IoT devices are randomly distributed in other areas.

[0178] (2) Scenario 2: Figure 3 As shown in (b), the scene contains a local area with a high density of IoT devices. About 60% of the IoT devices are concentrated in this local area, and the remaining about 40% of the IoT devices are randomly distributed in other areas.

[0179] (3) Scenario 3: Figure 3 As shown in (c), the scene contains two local areas with high density of IoT devices. About 50% and 30% of IoT devices are concentrated in these two local areas, respectively, and the remaining about 20% of IoT devices are randomly distributed in other areas.

[0180] In order to verify the superiority of the proposed MUCO method, it is compared with the following UAV deployment and computation offloading methods:

[0181] (1) RD: m locations are randomly selected from the locations of IoT devices as the deployment locations of UAVs.

[0182] (2) KD: IoT devices are regarded as clusters to be divided, and the number of UAVs m is regarded as the number of clusters. Then, the classic K-Means is used to divide IoT devices into corresponding clusters, and UAVs are deployed at the centroid of each cluster.

[0183] (3) Local: All tasks of IoT devices are executed locally.

[0184] (4) MEC: The tasks of IoT devices are offloaded to UAVs for execution, and UAVs collaborate randomly.

[0185] (5) PPO: Integrate the state and action spaces of all agents and use centralized PPO to make offloading decisions.

[0186] (6) DDQN: Use DDQN to make offloading decisions.

[0187] (7)SA: Use simulated annealing algorithm to make unloading decisions.

[0188] first, Figure 4The UAV deployment using the proposed MUCO method in different scenarios is demonstrated. From the experimental results, it can be found that in the three scenarios, the proposed MUCO method can effectively search for areas with dense distribution of IoT devices and achieve relatively balanced service coverage. Within the limited UAV coverage, the MUCO method can cover areas with higher density of IoT devices to improve coverage. Compared with scenario 2, the ideal UAV deployment points in scenarios 1 and 3 are easier to find. This is because there are high-density areas for the distribution of IoT devices in scenarios 1 and 3, and only a very small number of IoT devices are distributed in other areas. In this case, UAVs with limited coverage can cover more IoT devices. In scenario 2, IoT devices are more sparsely distributed, so there are more IoT devices that cannot be covered by UAVs. In this case, even limited by the limited number of UAVs and their coverage, the MUCO method can achieve good UAV deployment results.

[0189] Then, the UAV deployment performance of each method in different scenarios was compared, including indicators such as coverage rate and coverage balance. Figure 5As shown in the figure, the proposed MUCO method can achieve the minimum weighted sum of coverage and coverage balance in different scenarios, and is basically better than the other two methods in terms of coverage and coverage balance. This is because the proposed MUCO method can combine UAV coverage and clustering methods for different IoT device distributions to better perceive the distribution density of IoT devices, and then obtain the best UAV deployment strategy. Compared with the RD method, the MUCO and KD methods have better coverage in different scenarios. This is because the RD method randomly selects the deployment location of UAVs and cannot perceive the distribution of IoT devices. It may deploy UAVs in places where IoT devices are sparsely distributed, resulting in poor coverage. The MUCO method and the KD method have similar coverage in scenarios 1 and 3, but the MUCO method performs better in scenario 2. This is because the MUCO method considers the UAV coverage constraint when dividing clusters, and controls the range of each cluster well while improving the density of each cluster. Although the KD method can divide IoT devices into multiple clusters, it cannot control the range of each cluster, resulting in UAVs unable to fully cover all IoT devices in each cluster. At the same time, the density of IoT devices in each cluster varies greatly, resulting in UAVs being deployed in some large and sparse cluster centroids to cover only a small number of IoT devices. Therefore, the KD method has a low coverage rate in scenario 2. In addition, in scenario 2, the number of IoT devices and the density of distribution in each cluster divided by the KD method vary greatly, resulting in serious coverage imbalance. The RD method has slightly better coverage balance than the MUCO method in scenario 2. This is because in this scenario, it is easier to find several UAV deployment locations with similar distribution sparsity. Compared with the KD and RD methods, the proposed MUCO method can achieve an average of about 23.82% and 28.13% weighted and improved coverage and coverage balance in different scenarios.

[0190] Based on the UAV deployment results, the multi-UAV cooperative offloading performance of the proposed MUCO method is further tested. First, the average time slot rewards of different methods are compared. Figure 6As shown in the figure, SA, MEC and Local methods use single-step decision making and there is no learning process. The performance of MEC and Local methods is not as good as other methods because they perform tasks in a too single way and do not fully consider the system state and task characteristics. For example, due to the limited computing power of UAVs, if all tasks are offloaded to UAVs for execution, it will overload them, and the computing resources allocated to each task will be greatly reduced, resulting in serious degradation of their performance. Compared with the DDQN method, MUCO, PPO and SA methods achieved higher rewards. This is because the DDQN method cannot cope well with complex and changing environments and is prone to fall into local optimality. The PPO method integrates the states and actions of each agent, resulting in a state and action space that is too large, which increases the difficulty of training and converges slowly. However, the PPO method can surpass the SA method after a certain number of rounds of training. The SA method continuously iterates the same time slot to find the optimal solution for the time slot, but it cannot perceive environmental changes and find the optimal solution, and is prone to fall into local optimality. Compared with the PPO and SA methods, the proposed MUCO method can effectively improve the average time slot reward and show the best performance among all methods. This is because the MUCO method uses a distributed execution method, which makes the state and action space that each agent faces relatively small, which reduces the training pressure and speeds up the convergence speed. At the same time, the centralized training method used by the MUCO method enables it to fully perceive the overall state of the environment, thereby better optimizing the agent's unloading decision.

[0191] Then, the average time slot delays of different methods are compared. Figure 7 As shown in the figure, with the increase of training rounds, the average time slot delay of MEC and Local methods is basically unchanged and at the highest value. Since all tasks are offloaded to UAV for execution, it leads to excessive delay, making the performance of MEC method similar to that of Local method. In addition, the MEC method adopts a random collaboration method, which achieves load balancing to a certain extent but the effect is limited. The SA method reduces the average time slot delay to a certain extent, but it no longer changes with the increase of training rounds. With the increase of training rounds, the average time slot delay of other methods will gradually decrease, but the average time slot delay that the proposed MUCO method can finally achieve is the lowest, indicating that this method can make better offloading decisions, thereby reducing the processing delay of tasks.

[0192] Secondly, the average time slot UAV energy consumption of different methods is compared. According to formula (12), when the average allocation of computing resources is adopted, the fewer tasks a UAV processes, the more computing resources can be allocated to each task, and the higher the average time slot UAV energy consumption; conversely, the average time slot UAV energy consumption is lower. Figure 8As shown in the figure, the average time slot UAV energy consumption of the MEC method is at a low value. This is because it offloads all tasks to UAVs for execution, and each UAV needs to process a large number of tasks, so its average time slot UAV energy consumption is low. With the increase of training rounds, the average time slot UAV energy consumption of the MUCO, PPO and DDQN methods gradually increases. This is because these three methods gradually learn more optimized strategies and then reasonably select tasks to offload to UAVs for execution, which better balances the load between IoT devices and UAVs and reduces system latency. Although the proposed MUCO method is slightly higher than other methods in average time slot UAV energy consumption, the reward it obtains is significantly higher than other methods (such as Figure 6 At the same time, the advantage in reducing system latency is also significant.

[0193] Then, we compared the different f max The average time slot rewards of different methods are as follows. Fig. 9 As shown, with f max As f increases, the rewards of all methods increase. This is because when the UAV has more computing resources, more computing resources are allocated to each task, which reduces the computing latency. max As f increases gradually, the rate of increase of the rewards of each method gradually slows down. This is because the increase in computing resources allocated to the task also leads to an increase in computing energy consumption. max When f is low, the reward gap between the MEC method and other methods is large. This is because the MEC method offloads all tasks to UAVs for execution, and each task can only be allocated to limited computing resources, resulting in increased computing latency. max When f is high, the rewards of the MEC method are comparable to those of other methods. This is because when computing resources are sufficient, the MEC method can allocate enough computing resources to the task. Compared with other methods, the proposed MUCO method has a higher reward at different f. max The highest reward can be obtained under all conditions, which verifies the superiority of the MUCO method.

[0194] Then, the decision-making time of different methods was compared. As shown in Table 2, the SA method requires a large number of iterations to find the optimal solution, and its decision-making time is about 10 times that of other methods. In contrast, the MUCO, PPO, and DDQN methods can make decisions quickly based on the current environmental state, which can better meet the real-time requirements of large-scale IoT systems. Compared with the PPO and DDQN methods, the decision-making time of the proposed MUCO method is shorter. This is because the MUCO method adopts a distributed execution method, and the state and action space that each agent needs to process are smaller than those of the PPO and DDQN methods.

[0195] Table 2 Decision time of different methods (ms)

[0196] method Decision Time MUCO 2.01 PPO 7.99 DDQ 8.00 SA 43.00

[0197] Finally, the load on different UAVs within a time slot and the load balancing degree of different methods are evaluated. The load of a UAV refers to the number of tasks offloaded to and calculated by the UAV. The calculation formula for the load balancing degree is:

[0198]

[0199] Among them, ld j represents the load of the jth UAV, It represents the average value of all UAVs’ load. The smaller the load balancing degree value is, the closer the loads of each UAV are.

[0200] like Fig.10 As shown in Figure 2, when the proposed MUCO method is used for collaborative unloading, the loads of each UAV are closest, and the loads between UAVs are more balanced compared to other methods. Fig.11 As shown in Figure 2, compared with other methods, the proposed MUCO method can achieve lower load balancing degree values. The above results reflect that the proposed MUCO method can effectively utilize and coordinate multiple UAVs to achieve better load balancing effects.

[0201] Process or method of use

[0202] (1) MUCO performs adaptive UAV deployment based on the distribution of ground IoT devices.

[0203] (2) In each time slot, each MARL agent collects the task information generated by all IoT devices within the UAV coverage area, including the number of tasks and the computational complexity of the tasks.

[0204] (3) Each MARL agent generates task offloading and forwarding decisions based on the task information generated by all IoT devices, the computing power of the corresponding IoT devices, the transmission power of the corresponding IoT devices, and the remaining power of the UAV.

[0205] (4) UAV allocates computing resources to existing tasks for parallel computing based on an even distribution of computing resources.

[0206] (5) During computation offloading, each MARL agent continuously records the state of each time slot, the actions taken, the rewards obtained, and the new state entered, and continuously optimizes its own performance based on the above information.

Claims

1. A multi-UAV deployment and collaborative unloading method for the Internet of Things, characterized in that: Firstly, the original joint optimization problem is transformed into UAV deployment subproblem and computation offloading subproblem. Secondly, for the UAV deployment subproblem, a UAV deployment scheme based on constrained K-Means clustering is proposed. By introducing UAV coverage range constraints into K-Means clustering, the deployment location of UAV is adaptively adjusted to improve the coverage and coverage balance of computation offloading services in the system. Finally, for the computation offloading subproblem, a multi-UAV collaborative computation offloading strategy based on MARL is proposed. Through the centralized training and decentralized execution mode, the proposed strategy achieves approximately optimal computation offloading and UAV collaboration strategy.

2. The method for multi-UAV deployment and collaborative unloading for the Internet of Things according to claim 1 is characterized in that: The UAV deployment and collaborative offloading system for IoT devices is composed of UAVs and IoT devices. The IoT device requests computing offloading services from UAVs via a wireless link, which is denoted as a set U = {u1,u2,...,u n }, the horizontal coordinates are represented by {w1,w2,...,w n }; UAVs provide computing offloading services to IoT devices within their coverage area, denoted as the set UAV = {uav1,uav2,...,uav m }, the horizontal coordinates are expressed as {q1,q2,...,q m }; The horizontal height of UAVs is H, then u i(1≤i≤n) With UAV j(1≤j≤m) The distance between them is: Collaborative offloading is required between multiple UAVs, and UAVs forward tasks from IoT devices to their collaborative UAVs for execution; the system time slot is represented as t∈{1,2,...,T}, and at the beginning of each time slot, the IoT device chooses to execute its computing request locally or offload it to the UAVs connected to it for execution; if the IoT device is not within the coverage of any UAVs, its computing request is executed locally.

3. The method for multi-UAV deployment and collaborative unloading for the Internet of Things according to claim 2 is characterized in that: At time slot t, from u i The task is defined as a triple <D i [t],S i [t],c i >, where D i [t] represents the amount of input data for the task, S i [t] represents the computational complexity of the task, c i represents the associated indicator variable; when u i Connect to UAV j When c i =j; when u i When unable to connect to an available UAV, c i = 0; tasks from IoT devices can be executed locally or offloaded to UAVs; u i The unloading decision at time slot t is denoted as α i [t]∈{0,1}; When u i Executing the task locally or unable to connect to an available UAV (c i =0), α i [t]=0; when u i When offloading the task to a UAV for execution, α i [t] = 1; specifically, it is divided into the following two calculation modes; (1) Local computing mode When the task is executed locally, its latency is: Among them, f i Indicates u i computing power; (2) UAV computing model IoT devices access UAVs via OFDMA; when u i The task is offloaded to uav j When the input data is uploaded first, according to Shannon's theorem, u i With UAV j The data transfer rate between is: Among them, B u represents the channel bandwidth between IoT devices and UAVs, P i Represents u i The transmission power of Where ρ0 represents the channel gain per unit distance, and N0 represents the noise power spectral density; Therefore, u i Offload tasks to uav j The transmission delay is: Next, UAV j Need to process concurrently j [t] tasks; therefore, from u i The mission in UAV j The computational delay on is: in, f max Indicates the computing power of a UAV; Tasks on high-load UAVs can be forwarded to low-load UAVs to achieve load balancing; specifically, UAV j The cooperative offloading decision at time slot t is defined as β ijk [t]∈{0,1}; when uav j Decided to put u i The task is forwarded to uav k When ijk [t] = 1; otherwise, β ijk [t] = 0; a task is forwarded at most once, denoted by When the task is from uav j Forward to uav k When cooperative offloading is performed, the data transfer rate between the two is: Among them, B uav represents the channel bandwidth between UAVs, P uav represents the transmission power between UAVs, For UAV j With UAV k Therefore, u i The mission from UAV j Forward to uav k The transmission delay is: The forwarded task is in uav k The computational delay on is: Execute u i The task delay is: Processing i The total delay of the task is: In summary, the total delay for processing all tasks in time slot t is:

4. The method for multi-UAV deployment and collaborative unloading for the Internet of Things according to claim 3 is characterized in that: When tasks are offloaded to UAVs for processing, three aspects of energy consumption are generated, including the energy consumption of UAVs executing tasks, the energy consumption of forwarding tasks between UAVs, and the energy consumption of UAVs maintaining a hovering state; uav j Execute u i The energy consumption of the task is: Where κ represents the effective capacitance coefficient; uav j will u i Task forwarding to uav k The energy consumption is: The energy consumption of UAVs to maintain the hovering state in each time slot is e; therefore, the total energy consumption of all UAVs in time slot t is: in, Indicates uav j The set of tasks to be performed, Indicates from uav j The set of tasks forwarded; Due to the limited power of UAVs, they cannot always provide computing offload services for IoT devices; when the power is lower than the threshold b min When the UAVs are in a state of low latency, they will no longer provide computing offload services and will return to their home base for battery life. Based on the above proposed model, the optimization goal is to minimize the weighted sum of the total task processing delay and the UAV energy consumption; the optimization problem is formally defined as: Among them, λ1 and λ2 represent the weights of delay and energy consumption respectively, b j [t] indicates uav j The remaining power at time slot t, b min represents the minimum power that UAVs must retain; C1 and C2 represent the value range constraints for offloading decisions; C3 represents the number of times a task is forwarded between UAVs; C4 represents the distance constraint between the IoT device and its associated UAV, where R is the coverage radius of the UAVs; C5 represents the value range constraint of the associated indicator variable; constraint C6 indicates that the computing resources used by UAVs to execute tasks cannot exceed their maximum computing resource constraints; C7 indicates that UAVs need to retain the minimum power b when providing computing offloading services. min Used for returning to the voyage and continuing the power supply.

5. The method for multi-UAV deployment and collaborative unloading for the Internet of Things according to claim 4 is characterized in that: The original joint optimization problem is further split into two sub-problems: UAV deployment and computational offloading. First, UAV deployment optimization is performed to improve the coverage and coverage balance of UAVs computational offloading services. After UAVs are deployed, computational offloading optimization is performed to reduce the total task processing latency and UAV energy consumption.

6. The method for multi-UAV deployment and collaborative unloading for the Internet of Things according to claim 5 is characterized in that: For the UAV deployment problem, IoT devices are divided into multiple clusters, and then UAVs are deployed at the centroid of each cluster to maximize the service coverage. Service coverage and coverage balance are defined as performance indicators of UAV deployment, and then a UAV deployment scheme based on constrained K-Means clustering is designed. Specifically, clusters are divided based on UAV coverage so that the cluster size does not exceed the UAV coverage. At the same time, the balance of the number of IoT devices within the coverage of each UAV is considered when dividing the clusters.

7. The method for multi-UAV deployment and collaborative unloading for the Internet of Things according to claim 6 is characterized in that: In each round of iteration, the positions of UAVs are first randomly initialized, and the cluster set change flag is initialized to check whether the cluster division has reached convergence; then, the IoT devices within the coverage range of the UAVs are assigned to the clusters of the corresponding UAVs; specifically, unlike the classic K-Means algorithm that allocates data points based on the principle of minimum distance to the centroid, when allocating IoT devices, whether the IoT device is within the coverage radius of the UAVs is considered; when an IoT device is covered by multiple UAVs at the same time, the nearest UAV will be selected and the change will be marked, and then the number of UAVs covered will be updated; then, the current coverage rate and the load degree of all UAVs are calculated, which are defined as: When the coverage is close to equilibrium, F1≈F2≈...≈F m ; The mean square error is used to evaluate the coverage balance, which is defined as: in, Next, the coverage rate and coverage balance are normalized to the same numerical range; the mean of each cluster is used as the new centroid and UAVs are deployed; through iteration, the deployment location of UAVs will gradually tend to areas where IoT devices are densely distributed, enabling them to serve more IoT devices; after the iteration, the solution with the smallest weighted sum of coverage rate and coverage balance is selected for UAV deployment.

8. The method for multi-UAV deployment and collaborative unloading for the Internet of Things according to claim 7 is characterized in that: After solving the UAV deployment, the computation offloading problem is further split from P1, which is expressed as: A multi-UAV collaborative computing offloading strategy based on MARL is proposed: the proposed multi-UAV collaborative offloading system is regarded as an environment, and multiple agents interact with the environment at the same time and select corresponding offloading actions; after receiving the reward signal fed back by the environment, each actor network is updated in a distributed manner; the critic network adopts a centralized update method; accordingly, the state space, action space and reward function are defined as follows: State space: The state space includes the task attributes, the computing power of IoT devices, and the i , data transmission power P of IoT devices i , the remaining power of UAV b[t]; therefore, at time slot t, the state observed by agent j is expressed as: Among them, 1≤j≤m, and Respectively represent the set consisting of the data volume and computational complexity of all tasks; Action space: The action space contains the offloading decision α[t] of the IoT device and the collaborative offloading decision β[t] of the UAVs. α[t] and β[t] are integrated, and the integrated offloading decision is recorded as η[t]∈{0,1,2,...,m}. Therefore, at time slot t, the action of agent j is expressed as: in, represents the offloading decision made by agent j for all tasks; η i [t] = 0 means u i No task is generated or the agent decides to execute the task locally; η i [t]=k, k∈{1,2,...,m} means the agent decides to offload the task to the uav k Execute on; Reward function: The optimization objective of P2 is to minimize the weighted sum of the total task processing delay and the UAV energy consumption; therefore, at time slot t, the reward function is expressed as: r[t]=-(λ1N(T u_tot [t])+λ2N(E uav_tot [t])), (22) Among them, λ1 and λ2 represent the weights of delay and energy consumption respectively; N is the normalization function, which normalizes the delay and energy consumption to the same numerical range.

9. The method for multi-UAV deployment and collaborative unloading for the Internet of Things according to claim 8, characterized in that: The value network acts as a critic and estimates the advantage function of the state-action pair using multi-step temporal differences; the policy network acts as an actor to update the policy according to the advantage function; the obtained UAVs coordinates q and the associated indicator variable c will be used as the input of Algorithm 2; specifically, the value and policy networks are initialized first; in each round of iteration, each agent obtains the initial state of the environment, initializes the round end mark and the interaction data record table, where the interaction data record table is used to collect all data on the interaction between the agent and the environment in the current round; in each round, each agent selects a suitable unloading action according to its strategy, and converts η into the unloading decision α of the corresponding IoT device and the collaborative unloading decision β of the UAVs; then, α and β are executed and the reward, state and round end mark are updated; the round ends when the power of all UAVs reaches the minimum threshold, that is: when When , done = True; then, collect the interaction data of each agent and update its state; then, use the interaction data collected in the current round to train the agent; specifically, take out the interaction data and concatenate the states for centralized training of the value network, whose generalized advantage function is defined as: Among them, γ is the discount factor and introduces a hyperparameter When λ=0, it is the advantage obtained by one step of differentiation; when λ=1, it is the complete average of the advantages obtained by each step of differentiation; by using the advantage function, the variance of the policy gradient estimation can be significantly reduced and the stability of the training process can be improved; Next, the strategy obtained in the previous round is used as the old strategy for this round, and the action probability distribution of each agent is calculated; then, the same batch of data is used to update the network parameters multiple times; unlike other DRL algorithms, the previously sampled data is discarded after updating the network parameters. The proposed method converts it into a training process based on different strategies by setting new and old strategies; specifically, data is collected by interacting with the old strategy and the environment, and then the new strategy is trained using these data; according to the principle of importance sampling, the data sampled by the old strategy can be used multiple times, and gradient ascent can be performed multiple times for policy updates; after each update of the policy network parameters, the action probability distribution of each agent and the loss function of the policy network are calculated; in particular, clipping is used to limit the amplitude of each policy update to ensure that the strategy does not change drastically during training. Finally, the loss function of the value network is calculated and the parameters of the policy and value networks are updated using gradient descent.

Images