Unmanned aerial vehicle auxiliary task unloading and power distribution optimization method

Abstract

The invention discloses a method for optimizing auxiliary task unloading and power distribution of an unmanned aerial vehicle. The method comprises the following steps: step 1, constructing a system model; 2, establishing a communication model; 3, designing a calculation model; step 4, node selection based on evolutionary game; 5, constructing an optimization problem; and step 6, implementing a problem solution. The method has the beneficial effects that a node selection problem is modeled into an evolutionary game framework, so that a user autonomously decides and connects nodes based on a revenue function; a task unloading and power distribution joint optimization problem is ingeniously modeled into a multi-agent Markov decision process, and an algorithm based on MATD3 is innovatively proposed; and the designed optimization method shows excellent environmental adaptability and stability.

Description

Technical Field

[0001] The present invention relates to the technical field of communication networks, and particularly to a method for optimizing drone-assisted task offloading and power allocation. Background Art

[0002] In today's digital age, with the gradual popularization of 5G technology and the explosive growth of Internet of Things devices, people's requirements for the performance of mobile networks have reached an unprecedented level. The multi-drone-assisted air-ground network MEC system, as a new network architecture, is gradually becoming a research and application hotspot.

[0003] The traditional network architecture with a single macro base station as the core exposes many limitations when facing the increasing data traffic and diverse service requirements. Although the macro base station has a wide coverage area, due to its relatively fixed deployment location, in some remote areas or temporary hotspots, signal coverage and service quality are difficult to be effectively guaranteed. In addition, with the sharp increase in the number of mobile devices, especially in urban dense areas, the load pressure on the macro base station has increased sharply, prone to network congestion, resulting in a significant increase in data transmission delay and a serious decline in user experience.

[0004] In this context, the two-stage optimization system of the overloaded and congestion-prone air-ground MEC network assisted by multiple drones has emerged. Drones have high flexibility and rapid deployment capabilities, and can quickly reach areas with weak signals or dense users. As mobile edge computing nodes, they can provide closer and more efficient computing and communication services for users. By equipping edge servers on drones and offloading computing tasks from terminal devices to drones or macro base stations for processing, the transmission delay can be significantly reduced and the system response speed can be improved.

[0005] However, this emerging system faces numerous challenges in practical applications. Existing network models and algorithms often fail to fully consider key factors such as the dynamics of the network, the randomness of user distribution, and the mobility of drones when dealing with these complex scenarios. The traditional probabilistic LoS channel model is difficult to accurately describe the channel characteristics in the face of various complex environments such as rural, urban, and forest areas, resulting in inaccurate prediction of communication quality. At the same time, in a multi-agent network environment, single-agent algorithms are prone to the problem of action dimension explosion, making it difficult for agents to quickly explore the optimal decision-making strategy and unable to effectively cope with the complexity and dynamic changes of the system.

[0006] In summary, in order to fully utilize the advantages of the multi-drone-assisted air-ground MEC network system, improve network performance and user experience, there is an urgent need for an innovative method to optimize drone-assisted task offloading and power allocation to solve problems such as local network congestion, limited computing resources, poor environmental adaptability, and low algorithm efficiency existing in the existing system. Summary of the Invention

[0007] To solve the above problems, especially for the deficiencies existing in the prior art, the present invention provides a method for optimizing unmanned aerial vehicle (UAV)-assisted task offloading and power allocation to solve the above problems.

[0008] To achieve the above object, the present invention adopts the following technical means:

[0009] A method for optimizing UAV-assisted task offloading and power allocation includes the following steps:

[0010] Step 1: System model construction

[0011] Build a multi-UAV-assisted overloaded and congestion-prone air-ground mobile edge computing (MEC) network system, including a macro base station and multiple UAVs;

[0012] Clarify the total computing resources and total bandwidth resources of the macro base station and UAVs;

[0013] Set a discrete time slot set and determine the relevant parameters of the tasks generated by users, including task size and computing requirements;

[0014] Define the proportion of users offloading tasks to the selected nodes and the adjustable transmission power range when users transmit tasks;

[0015] Clarify that the positions of users in a certain time slot and area follow a random distribution;

[0016] Step 2: Communication model establishment

[0017] Adopt the ground-to-air (G2A) channel model, taking into account both large-scale fading and small-scale fading;

[0018] Calculate the transmission rate between users and UAVs based on the channel model;

[0019] Step 3: Computing model design

[0020] Adopt a partial offloading strategy and model the local task processing;

[0021] Step 4: Node selection based on evolutionary game

[0022] Define the node selection problem through evolutionary game:

[0023] The players are all users within the coverage of the nodes;

[0024] The strategies are the base stations and UAVs that users can choose to connect to;

[0025] All users within the coverage of the same unmanned aerial vehicle (UAV) form a population. There are a total of G populations, where g is the index of the population;

[0026] Define the state of the population as the proportion of selected nodes in population g;

[0027] Construct a revenue function and optimize the node selection result based on the computing resources and loads of the UAVs and the base stations;

[0028] Iteratively update the user selection strategy through an evolutionary game algorithm until a stable state is reached;

[0029] Step 5: Optimization problem construction

[0030] After node selection is completed, model the optimization problem as the negative weighted sum of system delay and energy consumption;

[0031] Step 6: Problem solution implementation

[0032] Adopt a multi-agent Markov decision process (MAMDP) to solve the task offloading and power allocation problems:

[0033] Define the actions as the task offloading ratio and the transmission power;

[0034] Define the state as the current network load, user location, and node selection result;

[0035] Define the reward as the negative weighted sum of system delay and energy consumption;

[0036] Propose a task offloading and power control algorithm based on MATD3 (multi-agent twin-delayed deep deterministic policy gradient):

[0037] During the process of the algorithm solving the problem, the UAVs and base stations acting as agents continuously execute the strategies of the algorithm within the set number of iterations, and convert the executed strategies into the proportion of tasks unloaded by users and the magnitude of the transmission power;

[0038] The agents continuously interact with the environment to generate experiences for algorithm training;

[0039] Optimize the task offloading ratio and transmission power through the MATD3 algorithm to minimize system delay and energy consumption.

[0040] A further solution of the present invention is that in the above step 1, the multi-UAV-assisted air-ground MEC network system consists of 1 macro base station, N UAVs equipped with edge servers, and K users. Among them, each base station and UAV integrates a server for processing the tasks unloaded by users.

[0041] A further solution of the present invention is that in the communication model establishment of the above step 2, the specific method for calculating the transmission rate is:

[0042] By determining whether the communication link between the user and the drone is blocked by a building, the link type is determined to be LoS or NLoS, and based on this, the calculation method of large-scale fading is obtained;

[0043]

[0044] where L n,k = 20log 10 d n,k + 20log 10 f c + 20log 10 (4π / v l ) represents the spatial path loss between the nth drone and the kth user, d n,k represents the straight-line distance from the nth drone to the kth user, v l represents the speed of light, f c represents the carrier frequency, θ Los and θ NLoS represent the propagation losses of the LoS and NLoS links respectively;

[0045] Clarify the characteristics of the small-scale fading coefficient under different links, and determine the calculation formula for the channel gain from node n to user k:

[0046]

[0047] where g′ n,k is the small-scale fading coefficient;

[0048] Clarify the calculation methods of the uplink signal-to-noise ratio and transmission rate when the user communicates with the drone. The calculation method of the uplink signal-to-noise ratio is:

[0049]

[0050] where p k is the transmit power, |g n,k | 2 is the channel power gain at the tth time slot, P n represents the power of the additive white Gaussian noise at the drone receiver;

[0051] The calculation method of the transmission rate is:

[0052] r n,k = W n,k log2(1 + SNK n,k ) (4)

[0053] where W n,k represents the bandwidth used by the user for task offloading, which is determined by the total system bandwidth and the node selection result, and is calculated as The number of users who select node n.

[0054] A further solution of the present invention is that the specific steps of step 3 are as follows:

[0055] Give the calculation method of the local task processing delay and...

[0056]

[0057] Among them, represents the offloading decision of user k in time slot t, means that the user processes all the generated tasks locally, means that user k offloads all tasks to the computing node for processing, represents the local computing resource of user k. Correspondingly, the calculation method of the energy consumption generated when the task is processed locally is:

[0058]

[0059] where k l represents the user local capacitance switching coefficient;

[0060] When the task generated by the user needs to be transmitted to the computing node for processing, it needs to be transmitted to the server. The transmission delay when user k offloads the task to the node is expressed as:

[0061]

[0062] The energy consumption during the transmission process is expressed as:

[0063]

[0064] In the actual network scenario, only the delay of task uplink transmission and server-side task processing needs to be considered. The task processing delay of the server is:

[0065]

[0066] where f n,k is the computing resource allocated by node n to the task generated by user k;

[0067] Correspondingly, the power consumption generated when the task is processed in the server is:

[0068]

[0069] where κ f represents the node server capacitance switching coefficient;

[0070] To sum up, the calculation formulas for the total delay and total energy consumption of user k to complete a task are respectively:

[0071]

[0072] A further solution of the present invention is that in step 4, the state of the population is defined as the proportion of selected nodes in population g, that is where is the number of users within the coverage of node n, is the number of users who select node n in population g.

[0073] A further solution of the present invention is that in step 4, the construction of the revenue function is specifically as follows:

[0074] Define a load function f load to characterize the congestion degree of the node:

[0075] f load (ε n ) = 1 - ε n (13)

[0076] Define the revenue function for population g to select nodes as:

[0077]

[0078] where ω is the weight of the computing resources owned by node n, which can be used to adjust the influence of the node computing resource capacity on the population revenue function;

[0079] Give the replicator dynamics equation in population g:

[0080]

[0081] where ρ > 0 in the equation is the replication dynamics weight factor, which is used to adjust the speed of players' selection of node strategies, making it easier to obtain the evolutionary equilibrium, is the average revenue in population g, defined as:

[0082]

[0083] A further solution of the present invention is that in step 5, the specific form of the construction of the optimization problem is:

[0084]

[0085] C3: 0 < p k ≤ p max (17)

[0086] Among them, α is the weight factor of the optimization function, which is used to unify the order of magnitude of the delay and energy consumption and balance the importance of the delay and energy consumption. The constraint C1 means that the offloaded tasks cannot exceed the total amount of tasks, the constraint C2 means that the bandwidth resources obtained by user k connected to node n cannot exceed the total bandwidth owned by the connected node, and the constraint C3 means that the transmit power of user k cannot exceed the maximum transmit power.

[0087] A further solution of the present invention is that in step 6, the actions, states, and rewards of the Markov decision process are defined as follows:

[0088] Action A n : The action of each agent is defined as the task offloading ratio and transmit power of the user, that is:

[0089]

[0090] State S n : The utility value of each agent is closely related to the number of tasks generated by the user, the task size, and the distance between the user and the UAV, that is:

[0091] S k ={s k,1 , s k,2 ..., s k,n} (19)

[0092] Wherein, N 2 is the number of tasks generated by the user;

[0093] Reward R k : The defined utility function is used to represent the reward of the Markov decision process, that is:

[0094]

[0095] A further solution of the present invention is that in step 6, the TD3 algorithm consists of six networks, namely the Actor network ν(·|θ ν ), the target actor network ν′(·|θ ν′ ), the critic network and the target critic network and where θ is the network parameter of each part.

[0096] A further solution of the present invention is that in step 6, during the algorithm training phase, the evaluation Critic network and the target Critic network are deployed in the servers of the UAV and the macro base station, and each node can perform wireless communication to publish the training parameters and the collected states to each agent node Facilitate better training for each other;

[0097] In each iteration of the algorithm, all nodes use the network ν(S n,t |θ ν ) to select actions, and the proposed algorithm will add a random noise ζ to each output action and clip it according to the set action threshold.

[0098] Advantages of the present invention:

[0099] 1. By modeling the node selection problem as an evolutionary game framework, the present invention enables users to autonomously decide to connect nodes based on the revenue function. This innovative mechanism can effectively avoid the phenomenon that users overly aggregate on certain nodes, and achieve a reasonable and balanced distribution of network load between macro base stations and drones. Whether in the high-density user scenario in the bustling commercial area of the city or in the special situation where a large number of users pour in instantaneously during a large-scale event, it can ensure uniform load on each node, significantly improve the overall utilization rate of network resources, greatly alleviate the local network congestion problem, and effectively guarantee the stable and efficient operation of the network.

[0100] 2. The present invention ingeniously models the joint optimization problem of task offloading and power allocation as a multi-agent Markov decision process, and innovatively proposes an algorithm based on MATD3. This algorithm can keenly perceive the dynamic changes of the network state, and accurately and efficiently search for the optimal task offloading ratio and power allocation strategy. In application scenarios such as online high-definition video live broadcast and real-time cloud games that are extremely sensitive to delay and energy consumption, it can ensure that data is transmitted with just the right power, while significantly reducing the transmission delay and minimizing energy consumption to provide users with a smooth, low-latency and energy-saving network service experience.

[0101] 3. The optimization method designed by the present invention demonstrates excellent environmental adaptability and stability. On the one hand, the proposed algorithm can flexibly cope with different scales of user numbers, diverse task types, and rapidly changing network environments. Whether the number of users suddenly surges or the network environment changes drastically due to weather, geographical factors, etc., it can quickly adjust the strategy to maintain the high performance operation of the system. On the other hand, by continuously optimizing node selection, task offloading, and power allocation, it can significantly shorten the delay of user task processing, reduce device energy consumption, comprehensively improve the quality of mobile computing services, bring a more high-quality, convenient and efficient usage experience to users, and ensure the reliability and stability of network services under various complex conditions. Description of the Drawings

[0102] Figure 1 It is a system diagram of a multi-UAV assisted air-ground MEC network of the present invention;

[0103] Figure 2Comparison diagram of the number of users in the population before and after evolutionary equilibrium when the number of users in the present invention is 90;

[0104] Figure 3 Comparison diagram of the number of users in the population before and after evolutionary equilibrium when the number of users in the present invention is 100;

[0105] Figure 4 Comparison diagram of the number of users in the population before and after evolutionary equilibrium when the number of users in the present invention is 110;

[0106] Figure 5 Comparison diagram of the number of users in four populations after evolutionary equilibrium under different numbers of users in the present invention;

[0107] Figure 6 Graph of the change in population revenue value under different numbers of users in the present invention;

[0108] Figure 7 Revenue value graph of different populations in the present invention;

[0109] Figure 8 Graph of the change in different population states during the evolution process of the present invention;

[0110] Figure 9 Comparison diagram of the performance indicators of the reward value under different algorithms in the present invention;

[0111] Figure 10 Comparison diagram of the performance indicators of the energy consumption under different algorithms in the present invention;

[0112] Figure 11 Comparison diagram of the performance indicators of the time delay under different algorithms in the present invention;

[0113] Figure 12 Graph of the influence of different task sizes on the time delay under different algorithms in the present invention;

[0114] Figure 13 Graph of the influence of different task sizes on the energy consumption under different algorithms in the present invention;

[0115] Figure 14 Graph of the influence of different node selection methods on the reward value in the present invention;

[0116] Figure 15 Graph of the influence of different node selection methods on the time delay in the present invention;

[0117] Figure 16 Graph of the influence of different node selection methods on the energy consumption in the present invention. Detailed implementation manner

[0118] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of them. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0119] Embodiment

[0120] As Figure 1 shown, a method for optimizing unmanned aerial vehicle (UAV)-assisted task offloading and power allocation includes the following steps:

[0121] Step 1: System model construction

[0122] Build a multi-UAV-assisted air-ground mobile edge computing (MEC) network system, which consists of one macro base station, N UAVs equipped with edge servers, and K users. Among them, each base station and UAV integrates a server for processing the tasks offloaded by users.

[0123] Clarify the total computing resources and total bandwidth resources of the macro base station and UAVs; set a discrete time slot set, and determine the relevant parameters of the tasks generated by users, including the task size and computing requirements; define the proportion of tasks offloaded by users to the selected nodes, and the adjustable transmit power range when users transmit tasks; clarify that the positions of users in a certain time slot and area follow a random distribution;

[0124] Among them, each base station and UAV integrates a server for processing the tasks offloaded by users. The total computing resources of the macro base station and UAVs are C n , the total bandwidth resources are W, and the discrete time slot set is represented by T = {1,..., t,..., T}, where T is the total number of time slots. In this scenario, the positions of all users in a certain time slot and area follow a random distribution. In a certain time slot, user k can independently select the base station or UAV to connect to and offload the generated task part of the tasks to the selected node, where represents the size of the task generated by user k, represents the number of unit CPU cycles required to process this task, and the offloading ratio is defined as When the user transmits the task to the server, the transmit power P can be adjusted within the maximum transmit power Pmax k to balance energy consumption and transmission rate.

[0125] Step 2: Communication model establishment

[0126] A G2A (ground-to-air) channel model is established using a simplified line-of-sight (LoS) channel or statistical channel model. Traditional probabilistic LoS channel models usually cannot adapt to complex environments such as rural, urban, and forest areas simultaneously, and in actual scenarios, drones often need to change altitude during communication. The G2A (ground-to-air) channel model can take into account both large-scale fading and small-scale fading situations.

[0127] Calculate the transmission rate between the user and the drone based on the channel model. The specific method for calculating the transmission rate is as follows:

[0128] By determining whether the communication link between the user and the drone is blocked by buildings, determine the link type as LoS or NLoS, and thus obtain the calculation method for large-scale fading;

[0129]

[0130] Among them, L n,k = 20log 10 d n,k + 20log 10 f c + 20log 10 (4π / v l ) represents the spatial path loss between the nth drone and the kth user, d n,k represents the straight-line distance from the nth drone to the kth user, v l represents the speed of light, f c represents the carrier frequency, θ LoS and θ NLoS represent the propagation losses of LoS and NLoS links respectively;

[0131] Clarify the characteristics of the small-scale fading coefficient under different links, and determine the calculation formula for the channel gain from node n to user k:

[0132]

[0133] Among them, g′ n,k is the small-scale fading coefficient;

[0134] Clarify the calculation methods for the uplink signal-to-noise ratio and transmission rate when the user communicates with the drone. The calculation method for the uplink signal-to-noise ratio is as follows:

[0135]

[0136] Among them, p k is the transmit power, |g n,k | 2 is the channel power gain at the tth time slot, P n represents the power of the additive Gaussian white noise at the drone receiver;

[0137] The calculation method of the transmission rate is as follows:

[0138] r n,k =W n,k log2(1 + SNK n,k ) (4)

[0139] Among them, W n,k represents the bandwidth used by the user for task offloading, which is determined by the total system bandwidth and the node selection result, and is calculated as is the number of users selecting node n.

[0140] Step 3: Calculation model design

[0141] Adopt the partial offloading strategy and model for local task processing. The specific steps are as follows:

[0142] Give the calculation method of the local task processing delay and;

[0143]

[0144] Among them, represents the offloading decision of user k within time slot t, means that the user processes all the generated tasks locally, means that user k offloads all tasks to the computing node for processing, represents the local computing resource of user k. Correspondingly, the calculation method of the energy consumption of the generated tasks processed locally is:

[0145]

[0146] Among them, k l represents the user's local capacitance switch coefficient;

[0147] When the tasks generated by the user need to be transmitted to the computing node for processing, the tasks need to be transmitted to the server. The transmission delay when user k offloads tasks to the node is expressed as:

[0148]

[0149] The energy consumption during the transmission process is expressed as:

[0150]

[0151] In the actual network scenario, only the delay of task uplink transmission and server-side task processing needs to be considered. The task processing delay of the server is:

[0152]

[0153] Among them, fn,k The computing resources for the tasks generated by allocating node n to user k;

[0154] Correspondingly, the power consumption generated when the task is processed in the server is:

[0155]

[0156] where κ f represents the node server capacitance switching coefficient;

[0157] In summary, the calculation formulas for the total delay and total energy consumption of user k to complete a task are respectively:

[0158]

[0159] Step 4: Node selection based on evolutionary game

[0160] Before computing offloading, each user needs to connect to a suitable offloading node. And an excellent node selection scheme will balance the load of the server and play a positive role in optimizing the network system indicators. Therefore, an evolutionary game definition for the node selection problem is as follows:

[0161] Players: Only users within the coverage of the base station and UAV have the opportunity to select nodes. In the access selection game of users in the air-ground MEC network, the players are all users within the coverage of the nodes.

[0162] Strategies: A strategy refers to the actions or decision rules that participants can choose in a game. The strategy of each game player is the base station and UAV that the user can choose to connect to, X k ={MBS, UAV n}, where MBS and UAV n are the macro base station and UAV that user k can choose respectively.

[0163] Population: A population refers to a set composed of multiple individuals in a given game environment. In the population, each individual will interact with other individuals according to the strategies and behaviors it chooses, and these individuals will gradually adjust their strategies according to their own interests and environmental changes to obtain better results. In evolutionary game, all users within the coverage of the same UAV form a population, and there are a total of G populations, where g is the index of the population.

[0164] Population state: In evolutionary game, the population state is usually used to represent the distribution of different strategies in the population, that is, the proportion or frequency of each strategy in the population. Define the state of the population as the proportion of users who choose nodes in population g, that is where, is the number of users within the coverage of node n, The number of users who select node n in population g.

[0165] Construct a revenue function, optimize the node selection result based on the computing resources and load of the UAV and the base station; update the user selection strategy iteratively through the evolutionary game algorithm until a stable state is reached.

[0166] To represent the load of node n, the load factor of the node is defined as ε n = U n / |K|. The larger the load function, the higher the load of the node, where U n is the number of users associated with node n, and |K| is the total number of users in the proposed aerial MEC network scenario. Among them, the specific construction of the revenue function is as follows:

[0167] Define a load function f load to characterize the congestion degree of the node:

[0168] f load (ε n ) = 1 - ε n (13)

[0169] It can be seen from the formula that the more users associated with node n, the larger the value of its load factor. On the contrary, the smaller the value of its load function. Since the quality of service of the users associated with the node is usually related to the computing resource capacity owned by the node, the revenue function for population g to select the node is defined as:

[0170]

[0171] Among them, ω is the weight of the computing resources owned by node n, which can be used to adjust the influence of the node computing resource capacity on the population revenue function.

[0172] In the process of dynamic evolutionary game, replicators with higher revenue can replicate themselves faster. This replicator dynamics also helps to study the convergence speed of strategy adaptation, and the replicator dynamics equation in population g is given:

[0173]

[0174] Among them, ρ > 0 in the equation is the replication dynamics weight factor, which is used to adjust the speed of the player's node selection strategy and make it easier to obtain the evolutionary equilibrium. is the average revenue in population g, which is defined as:

[0175]

[0176] In an evolutionary game, an equilibrium point refers to a stable state reached by the strategies of all players during the evolutionary process. In this state, no individual player has the incentive to unilaterally change their strategy because doing so would reduce their level of success in the group. Therefore, the solution to the node selection problem at this node is the node selection decision of each user at the evolutionary equilibrium point. At this time, the value of the payoff function of each player is the same as the average payoff value of its own population, that is In this case, the strategies of all players are optimal strategies based on each other's strategies. If any player changes their strategy at this time, they will not obtain a higher payoff value. To make the process of population evolution smoother and easier to converge, a selection probability factor σ needs to be introduced. The user association algorithm based on evolutionary game is shown in Table 1.

[0177] Table 1

[0178]

[0179]

[0180] Step 5: Optimization problem construction

[0181] After the node selection is completed, the optimization problem is modeled as the negative weighted sum of system delay and energy consumption, and the specific form is:

[0182]

[0183] C3: 0 < p k ≤ p max (17)

[0184] where α is the weight factor of the optimization function, used to unify the order of magnitude of delay and energy consumption and balance the importance of delay and energy consumption. Constraint C1 means that the offloaded tasks cannot exceed the total amount of tasks. Constraint C2 means that the bandwidth resources obtained by user k connected to node n cannot exceed the total bandwidth of the node it is connected to. Constraint C3 means that the transmission power of user k cannot exceed the maximum transmission power.

[0185] Step 6: Implementation of problem solution

[0186] The multi-agent Markov decision process (MAMDP) is used to solve the task offloading and power allocation problem. Define the actions, states, and rewards of the Markov decision process, specifically:

[0187] Action A n : Define the action of each agent as the task offloading ratio and transmission power of the user, that is:

[0188]

[0189] State Sn : The utility value of each agent is closely related to the number of tasks generated by the user, the task size, and the distance between the user and the UAV, that is:

[0190] S k = {s k,1 , s k,2 ..., s k,n} (19)

[0191] Wherein, N z is the number of tasks generated by the user;

[0192] Reward R k : Use the defined utility function to represent the reward of the Markov decision process, that is:

[0193]

[0194] A task offloading and power control algorithm based on MATD3 (Multi-Agent Twin Delayed Deep Deterministic Policy Gradient) is proposed. The TD3 algorithm consists of six networks, namely the Actor network ν(·|θ ν ), the target actor network ν′(·|θ ν′ ), the critic network and the target critic network and where θ is the network parameter of each part.

[0195] In the process of the algorithm solving the problem, the UAV and the base station as agents continuously execute the strategy of the algorithm within the set number of iterations, and convert the executed strategy into the proportion of tasks unloaded by the user and the magnitude of the transmission power.

[0196] In the algorithm training stage, the evaluation Critic network and the target Critic network are deployed in the servers of the UAV and the macro base station. Each node can perform wireless communication to publish training parameters and collected states to each agent node to facilitate better training with each other. Then, each node uses the state s k,t and a k,t to estimate the strategies of other base stations and obtain the Q function of all agents When executing an action, the training objective of the Actor network is to maximize the error between the estimate of the Critic network for the current state and the action value predicted by the Actor network. The actual performance is that the Actor network receives environmental state information as input and outputs the corresponding action value. The overall algorithm flow is shown in Table 2.

[0197] In each iteration of the algorithm, all nodes use the network ν(s n,t |θ ν ) to select actions. Moreover, the proposed algorithm will add a random noise ζ to each output action and clip it according to the set action threshold. The agent generates experiences for algorithm training by continuously interacting with the environment. The MATD3 algorithm is used to optimize the task offloading ratio and transmission power to minimize the system delay and energy consumption.

[0198] The agent maps the finally output action value into the built simulation environment. Based on the actions selected by each agent, all users choose the task offloading ratio and transmission power. Then, according to the built mathematical model, the total task delay, total energy consumption, and utility value are calculated. Based on the obtained performance metric values, each agent calculates the reward value according to Equation (20). At this time, all nodes will obtain a set of experiences {a n , s n , s′ n , r n} and store them in the experience pool, where the capacity of the experience pool is H, and the larger the value of H is, the better without using the experience first-in-first-out rule. Then, each node selects experiences of size H m from the experience pool for algorithm training. During the training process, each node will use the policy gradient update method to update the weights of the target actor network, and the weight update formula is:

[0199]

[0200] When updating the parameters of the critic network, two Critic networks are used in the TD3 algorithm to evaluate the value of actions, and then the minimum value of the outputs of the two Critic networks is taken as the final action value estimate. However, even with two Critic networks, it is still impossible to completely avoid the overestimation problem of action values, resulting in a large variance and inaccurate estimation of the target estimate. Therefore, when calculating the target value, target policy smoothing regularization is used to add interference to the actions of the next state, making the actions of the next state change gradually rather than suddenly, which helps to improve the stability of training, reduce the volatility and oscillation during the training process, and further prevent overfitting of Q values, making the value evaluation more accurate. The updated target action is defined as:

[0201] a′ j = ν(s′ j |θ ν′ ) + τ (22)

[0202] where is the noise of the policy, is the standard deviation of the noise. Then the algorithm uses a dual network to obtain the Q-value corresponding to the action, and then selects the action with the highest Q-value from the Q ′ network (greedy strategy), and uses this action to calculate the Q ′ value of the network for the next state, and then obtains the target value. The calculation method is expressed as:

[0203]

[0204] where γ is the discount factor. Finally, the algorithm uses the gradient descent method to minimize the error between the target value and the estimated value. It is expressed by the formula:

[0205]

[0206] Then the weights of the three networks of each agent will be updated in the following way at a certain frequency

[0207]

[0208] where λ is the learning rate. In order to make the amplitude of the policy update appropriate, avoid the instability caused by too frequent or too sparse updates and reduce the variance introduced when updating the policy, the agent updates the evaluation actor network at a fixed frequency, so that the agent can balance the learning speed and stability to a certain extent. Finally, the weights of the target network are softly updated at the update frequency β, according to the following formula:

[0209]

[0210] Table 2

[0211]

[0212] Specific verification

[0213] Simulation results

[0214] Simulation experiments were carried out to verify the performance of the proposed algorithm. First, the simulation parameters were set, then the effectiveness and stability of the node selection algorithm were verified. Then, the results of the proposed task offloading algorithm based on MATD3 were analyzed, proving the superiority of the algorithm. Finally, the promotion effect of the node selection algorithm on the task offloading algorithm was proved, verifying the necessity of the node selection algorithm.

[0215] Simulation parameter design

[0216] Algorithm 1 and Algorithm 2 are implemented in the Python 3.8 environment using Pytorch = 1.19.0 to evaluate the performance of the proposed algorithms. The established simulation model is based on an Intel Core I9 CPU with 512GB of memory and 16GB of RAM. This paper considers the user task offloading scenario in a UAV-assisted base station scenario, which includes one macro base station, four UAVs, and one hundred users. The base station is located at the center of the Cartesian coordinate system (0, 0), and the coordinates of the four UAVs are (250, 0), (-250, 0), (0, 250), and (0, -250) respectively. The coverage range of the UAVs is large, but when the distance between the user and the UAV is very far, the communication quality is often poor. Therefore, this paper sets the effective coverage radius of the UAVs to 100 meters, and the coverage range of the macro base station to 500 meters, which meets the actual standard. The path loss of the macro base station is 128.1 +

[0217] 37.6log10(d m,k ), where d m,k is the distance between the macro base station and the user. In the simulation environment, the users are randomly distributed within the specified coverage range, and their task arrival rates follow an exponential distribution. There are more people within the coverage range of the UAVs, which is to simulate the scenario of excessive local task load. Specifically, the number of users within the coverage range of each UAV is [25, 20, 20, 15], and the remaining twenty users are only within the coverage range of the macro base station. To make the states between the indicators equal, the weight coefficients are set as [α = 100, β = 1]. Table 3 shows some of the important system parameters. For the algorithm hyperparameters, we set the total number of iterations to 8000 times, and the capacity of the experience pool H = 6000. This is because the experience storage follows the first-in, first-out principle, and some of the initially obtained experiences can be appropriately discarded in the later stage of the algorithm training to enable the algorithm to extract more excellent experiences from the experience pool. The environmental parameters and algorithm hyperparameters are summarized in Table 3 below.

[0218] Table 3

[0219]

[0220]

[0221] Analysis of Simulation Results

[0222] Node Selection

[0223] Figures 2 - 5 shows the comparison chart of the number of users in each population before and after the evolutionary equilibrium. From Figures 2 - 4It can be seen that when there are ninety users in the scenario, after the evolutionary equilibrium, the number of users choosing each drone is equal, all being 9; when there are one hundred users in the scenario, after the evolutionary equilibrium, the number of users choosing each drone is equal, all being 11; when the number of users in the scenario reaches one hundred and ten, after the evolutionary equilibrium, the number of users served by each drone is still equal, all being 12, and from Figure 5 it can be seen more clearly that as the number of users in the system increases, the number of users served by each population after reaching the evolutionary equilibrium also increases, but the number of users served by different UAVs is always equal. Therefore, it can be concluded that the node selection algorithm based on evolutionary game proposed by us has strong compatibility and will not lead to a decline in algorithm performance as the number of users increases.

[0224] From Figure 6 it can be seen that when the number of users in the scenario is different, the converged revenue values are also different. As the number of users increases, the revenue value shows a downward trend. And as the number of users increases, the convergence of its revenue value becomes slower and the volatility of the revenue curve becomes larger. This is because the computing resources in the scenario are fixed, and as the number of users increases, the resources that each user can obtain will become less and less, resulting in lower population revenue values and greater fluctuations in the revenue curve.

[0225] Figure 7 It shows the changes in the revenue values of different populations during the evolutionary game. It can be seen that after 27 iterations, the revenue values of different populations have reached the convergence state, and the convergence value is about 4.2, indicating that the node selection algorithm proposed by us has reached the evolutionary equilibrium at this time. However, the utility value of population MBS after convergence is about 4.3 and is not exactly the same as the revenue values of other populations. This is because an adaptation factor is added to our designed algorithm. The adaptation factor can make the population evolve more stably and rapidly. When the difference between the revenue value of an individual and the average revenue value of the population is within the range of the adaptation factor, it can be considered that our evolutionary game has reached the equilibrium state under a certain fitness.

[0226] From Figure 8 it can be seen that the population states of different populations reach the convergence state after 27 iterations, indicating that the evolutionary equilibrium has been reached at this time. And it can also be seen that after reaching the equilibrium state, the population states of population 1 and population 2 are the same. This is because the number of users within the coverage ranges of UAV1 and UAV2 in our simulation environment is the same. When reaching the equilibrium, the number of connected users is also the same. Therefore, according to the definition of the population state in the previous text, it can be obtained that regardless of how the initial value of the population state changes, the converged states of population 1 and population 2 after reaching the evolutionary equilibrium are the same.

[0227] From Figure 9It can be seen that the algorithm we selected can achieve a higher reward value compared to other advanced algorithms. Although the MASAC algorithm still shows an increasing trend in the later stage, its reward value lags behind the MATD3 algorithm at the end of the iteration, and the MATD3 algorithm also shows great advantages in terms of convergence speed. Compared with the MATD3 and MASAC algorithms, the reward values obtained by the single-agent TD3 and SAC algorithms are far behind those of the multi-agent algorithms. This is because single agents are prone to the drawback of action dimension explosion in a multi-agent environment, and it is difficult for the mentor agent to quickly explore excellent actions and even fail to converge. From Figure 10 It can be seen that the algorithm we proposed has the highest energy consumption. This is because we considered the safety factor problem in the air-ground MEC network environment and designed a higher time delay weight in the utility function. Therefore, during the training process, the proposed algorithm will use higher energy consumption to achieve a lower transmission rate, thereby reducing the transmission delay. Therefore, in Figure 11 the time delay achieved by the algorithm we proposed is the lowest.

[0228] From Figures 12 - 13 it can be seen that the larger the task size generated by the user, the higher the time delay and the greater the energy consumption. The reason can be seen from formula (7). When the task is larger, its transmission time delay is higher, so the total time delay will also be higher. And from formula (8), it can be seen that when the transmission power is constant, the higher the task transmission time delay, the greater the energy consumption, which fully verifies the rationality of the simulation environment we established. For the algorithm, regardless of the task size, the algorithm we proposed can always maintain the lowest time delay. Similar to Figures 9 - 11 this is also the result of sacrificing the energy consumption value, which also shows the stability of the algorithm we proposed.

[0229] Figure 14 Similar to Figure 15 It can be seen that compared with the random selection method and the method of selecting the nearest node, based on the node selection algorithm we proposed, the task offloading and power control algorithm based on MATD3 can achieve higher rewards and lower time delays. According to the simulation results described above, the proposed node selection algorithm will cause the MATD3 algorithm to generate greater energy consumption at this time. And from Figure 16 it can be seen that the result is exactly the opposite. Compared with the random selection method and the method of selecting the nearest node, the user association algorithm based on evolutionary game proposed in this paper can make the task offloading algorithm based on MATD3 generate lower energy consumption. This is because the methods of randomly selecting nodes and selecting the nearest nodes are very likely to cause local overload of the server. At this time, the users connected to the overloaded server nodes obtain less resources, forcing these users to choose a larger power to obtain a higher transmission rate. Therefore, the node selection algorithm proposed in this paper can not only balance the load of the server, but also reduce the energy consumption generated by users processing tasks.

[0230] The examples given in the present invention are illustrative rather than restrictive of the embodiments. For those of ordinary skill in the art, based on the above description, other different forms of changes or modifications can be made. It is not necessary and impossible to enumerate all the embodiments here, and the obvious changes or modifications derived therefrom still fall within the protection scope of the present invention.

Claims

1. A method for optimizing unmanned aerial vehicle (UAV)-assisted task offloading and power allocation, characterized in that It includes the following steps: Step 1: System model construction Build a multi-UAV-assisted overloaded and congestion-prone air-ground MEC network system, including a macro base station and multiple UAVs; Clarify the total computing resources and total bandwidth resources of the macro base station and UAVs; Set a discrete time slot set and determine the relevant parameters of the tasks generated by users, including task size and computing requirements; Define the proportion of users offloading tasks to the selected nodes and the adjustable transmission power range when users transmit tasks; Clarify that the positions of users in a certain time slot and area follow a random distribution; Step 2: Communication model establishment Adopt the G2A channel model, taking into account both large-scale fading and small-scale fading; Calculate the transmission rate between users and UAVs based on the channel model; Step 3: Computing model design Adopt a partial offloading strategy and model for local task processing; Step 4: Node selection based on evolutionary game Define the node selection problem through evolutionary game: The players are all users within the coverage of the nodes; The strategies are the base stations and UAVs that users can choose to connect to; All users within the coverage of the same UAV form a population. There are a total of G populations, where g is the index of the population; Define the state of the population as the proportion of nodes selected in population g; Construct a revenue function and optimize the node selection result based on the computing resources and loads of UAVs and base stations; Iteratively update the user selection strategy through the evolutionary game algorithm until a stable state is reached; Step 5: Optimization problem construction After the node selection is completed, model the optimization problem as the negative weighted sum of system delay and energy consumption; Step 6: Implementation of problem solution Adopt a multi-agent Markov decision process to solve the task offloading and power allocation problems: Define the actions as the task offloading proportion and transmission power; Define the state as the current network load, user positions, and node selection results; Define the reward as the negative weighted sum of system delay and energy consumption; Propose a task offloading and power control algorithm based on MATD3: During the process of the algorithm solving the problem, the UAVs and base stations as agents continuously execute the strategies of the algorithm within the set number of iterations and convert the executed strategies into the proportion of tasks offloaded by users and the magnitude of the transmission power; The agents continuously interact with the environment to generate experiences for algorithm training; Optimize the task offloading proportion and transmission power through the MATD3 algorithm to minimize system delay and energy consumption.

2. The method for optimizing drone-assisted task offloading and power allocation according to claim 1, wherein In the above Step 1, the multi-UAV-assisted air-ground network MEC system consists of 1 macro base station, N UAVs equipped with edge servers, and K users. Among them, each base station and UAV integrates a server for processing the tasks offloaded by users.

3. A method for optimizing drone-assisted task offloading and power allocation according to claim 1, characterized in that, In the communication model establishment of Step 2, the specific method for calculating the transmission rate is as follows: By judging whether the communication link between the user and the UAV is blocked by a building, determine the link type as LoS or NLoS, and thus obtain the calculation method of large-scale fading; where, L n,k = 20 log 10 d n,k + 20 log 10 f c + 20 log 10 (4π / v l ) represents the spatial path loss between the nth UAV and the kth user, d n,k represents the straight-line distance from the nth UAV to the kth user, v l represents the speed of light, f c represents the carrier frequency, θ Los and θ NLoS represent the propagation losses of the LoS and NLoS links, respectively; Clarify the characteristics of the small-scale fading coefficient under different links and determine the channel gain calculation formula from node n to user k; where g ′ n,k is the small-scale fading coefficient; Clarify the calculation methods of the uplink signal-to-noise ratio and transmission rate when the user communicates with the UAV. The calculation method of the uplink signal-to-noise ratio is: where p k is the transmit power, |g n,k | 2 is the channel power gain in the t-th time slot, and P n represents the power of additive white Gaussian noise at the UAV receiver; The calculation method of the transmission rate is: r n,k = W n,k log2(1 + SNK n,k ) (4) Among them, W n,k represents the bandwidth used by the user for task offloading, which is determined by the total system bandwidth and the node selection result, and is calculated as is the number of users selecting node n.

4. A method for optimizing unmanned aerial vehicle (UAV)-assisted task offloading and power allocation according to claim 1, characterized in that, The specific steps of step 3 are as follows: Give the calculation method of the local task processing delay and... Among them, represents the offloading decision of user k in time slot t, means that all the generated tasks of the user are processed locally, means that user k offloads all tasks to the computing node for processing, represents the local computing resources of user k. Correspondingly, the calculation method of the energy consumption of the generated tasks processed locally is as follows: where k l represents the user's local capacitance switch coefficient; When the task generated by the user needs to be transmitted to the computing node for processing, the task needs to be transmitted to the server. The transmission delay when user k offloads the task to the node is expressed as: The energy consumption during the transmission process is expressed as: In the actual network scenario, only the delay of task uplink transmission and task processing on the server side needs to be considered. The task processing delay of the server is: where f n,k is the computing resource of the task generated by node n for user k; Correspondingly, the power consumption generated when the task is processed in the server is: where κ f represents the capacitance switching coefficient of the node server; To sum up, the calculation formulas for the total delay and total energy consumption of user k to complete a task are respectively:

5. A method for optimizing drone-assisted task offloading and power allocation according to claim 1, characterized in that In step 4, the state of the population is defined as the proportion of selected nodes in population g, that is where is the number of users within the coverage range of node n, is the number of users who select node n in population g.

6. The method for optimizing drone-assisted task offloading and power allocation according to claim 1, wherein In step 4, the construction of the revenue function is specifically: Define a load function f load to characterize the congestion level of the node: f load (ε n ) = 1 - ε n (13) Define the revenue function for population g to select nodes as: Among them, ω is the weight of the computing resources owned by node n, which can be used to adjust the influence of the node computing resource capacity on the population revenue function; Give the replicator dynamics equation in population g: Among them, ρ>0 in the equation is the replication dynamics weight factor, which is used to adjust the speed of players' selection of node strategies, making it easier to obtain the evolutionary equilibrium. is the average payoff in population g, which is defined as:

7. A method for optimizing unmanned aerial vehicle (UAV)-assisted task offloading and power allocation according to claim 1, wherein In step 5, the specific form of the optimization problem construction is: C3: 0 < pk ≤ p max (17) Among them, α is the weight factor of the optimization function, which is used to unify the order of magnitude of delay and energy consumption and balance the importance of delay and energy consumption. Constraint C1 means that the offloaded tasks cannot exceed the total task volume. Constraint C2 means that the bandwidth resources obtained by user k connected to node n cannot exceed the total bandwidth owned by the connected node. Constraint C3 means that the transmission power of user k cannot exceed the maximum transmission power.

8. A method for optimizing unmanned aerial vehicle (UAV)-assisted task offloading and power allocation according to claim 1, characterized in that, In step 6, define the actions, states, and rewards of the Markov decision process, specifically: Action A n : Define the action of each agent as the task offloading ratio and transmission power of the user, i.e.: State S n : The utility value of each agent is closely related to the number of tasks generated by the user, the task size, and the distance between the user and the UAV, that is: S k = {s k,1 , s k,2 ..., s k,n} (19) Among them, N z is the number of tasks generated for the user; Reward R k : Use the defined utility function to represent the reward of the Markov decision process, i.e.:

9. A method for optimizing unmanned aerial vehicle (UAV)-assisted task offloading and power allocation according to claim 1, wherein, In step 6, the TD3 algorithm consists of six networks, namely Actor network ν(·|θ ν ), target actor network ν′(·|θ ν′ ), critic network and target critic network and where θ is the network parameter of each part.

10. A method for optimizing unmanned aerial vehicle-assisted task offloading and power allocation according to claim 1, characterized in that, In step 6, during the algorithm training phase, the evaluation Critic network and the target Critic network are deployed in the servers of the drones and macro base stations, and each node can perform wireless communication to publish training parameters and collected states to each agent node which facilitates better training for each other; In each iteration of the algorithm, all nodes use the network ν(S n,t |θ ν ) to select actions, and the proposed algorithm will add a random noise ζ to each output action and clip it according to the set action threshold.

Images