Multi-unmanned aerial vehicle cooperative task unloading method and system based on improved MATD3 algorithm

Through the improved MATD3 algorithm and priority experience playback mechanism, combined with genetic algorithm to optimize hyperparameters, the sensitivity problem of deep reinforcement learning in computing-intensive task processing is solved, and the robustness of the drone-assisted mobile edge computing system is improved. Efficiency, achieving faster convergence speed and lower energy consumption.

CN119012285APending Publication Date: 2024-11-22QIQIHAR UNIVERSITY
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202411107804.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-13
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

When handling computing-intensive tasks, pure mobile cloud computing and early mobile edge computing are difficult to meet real-time requirements, especially in the case of sparse communication facilities or natural disasters, and deep reinforcement learning methods are sensitive to environmental and parameter changes. , resulting in slow convergence speed and poor robustness.

Method used

Introducing an improved multi-agent dual-latency depth deterministic strategy gradient algorithm (MATD3) combined with priority experience playback mechanism and genetic algorithm to design a multi-drone collaborative task offload system, and optimize the priority and importance sampling of each sample by calculating the priority and importance sampling of each sample. The algorithm hyperparameters reduces the problem of overestimation of Q value, improves robustness and adaptability, and reduces the total delay and total energy consumption of the drone-assisted mobile edge computing system.

Benefits of technology

It improves the robustness and adaptability of reinforcement learning algorithms, reduces sensitivity to environment and parameters, speeds up convergence speed, and reduces the total delay and total energy consumption of the drone-assisted mobile edge computing system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119012285A_ABST
    Figure CN119012285A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-unmanned aerial vehicle cooperative task unloading method and system based on an improved MATD3 algorithm. The method comprises the steps that S1, a system model of multi-unmanned aerial vehicle cooperative task unloading is established; s2, establishing a communication model, a time delay model and an energy consumption model according to the system model; and S3, based on the communication model, the time delay model and the energy consumption model, a PER-MATD3 algorithm is adopted to jointly design user scheduling, a task unloading proportion and flight angles and speeds of the unmanned aerial vehicles, the total cost of the system is minimized, and cooperative task unloading of the multiple unmanned aerial vehicles is realized. According to the method, the problem of Q value overestimation possibly occurring in the DDPG algorithm is solved, the robustness and adaptability of the reinforcement learning algorithm are improved, meanwhile, the sensitivity of the reinforcement learning algorithm to the environment and parameters is reduced, the convergence speed is increased, and the total time delay and total energy consumption generated by the unmanned aerial vehicle assisted mobile edge computing system are reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of unmanned aerial vehicles, and in particular relates to a multi-unmanned aerial vehicle collaborative task unloading method and system based on an improved MATD3 algorithm. Background Art

[0002] With the development of 5G technology, the number of computationally intensive applications running on user devices has gradually increased. These applications usually require a large amount of computing resources and generate high energy consumption. However, current user devices usually have limited computing resources and limited battery capacity. Mobile cloud computing (MCC) has abundant resources and huge computing power. In the past, users usually transferred certain computationally intensive tasks to the cloud for processing. However, cloud servers are often far away from users in space, which may cause high transmission delays and adversely affect the user's service experience (Quality of Service, QoS) and service quality. For tasks with high real-time requirements, such as assisted autonomous driving, emergency rescue, battlefield communication and reconnaissance, and face recognition, simple MCC has been unable to cope with this new challenge.

[0003] With the emergence of the new paradigm of Mobile Edge Computing (MEC), the above problems have been solved. MEC not only makes up for the problem of insufficient computing power of user terminals, but also has lower latency than cloud computing to meet the computing needs of delay-sensitive tasks. MEC deploys a large number of edge servers close to the user end and sinks tasks to edge servers at the edge of the network for processing. Compared with MCC, MEC servers are closer to users, which reduces the cost of transmission, and their powerful computing power can meet most application scenarios. However, early research on MEC focused on improving the performance of MEC systems, where computing services are provided by base stations at fixed locations, but MEC services provided by fixed infrastructure cannot work effectively when communication facilities are sparsely distributed or in the event of sudden natural disasters.

[0004] In order to solve the above problems, the present invention proposes that multiple unmanned aerial vehicles (UAVs) assist MEC in task offloading. UAVs have the characteristics of fast deployment, high flexibility, strong adaptability, strong functionality and low cost, and have good application prospects in the fields of wireless communication systems and cellular networks. UAVs can act as relay communications and can also assume the role of flying mobile terminals, which are of great help in auxiliary communications and improving network capacity. Installing the mobile edge server on the drone not only combines the advantages of drones and mobile edge computing, but also can establish a good line-of-sight link with ground user equipment to process complex and delay-sensitive tasks in real time. In recent years, many existing works have only considered the task offloading scenario of a single drone and multiple users. For computationally intensive scenarios, multiple drones will be more suitable for collaboration. In addition, some current deep reinforcement learning (DRL) methods are sensitive to changes in the environment and parameters. Slight parameter adjustments and environmental changes may lead to too slow convergence speed and poor convergence performance, thereby affecting the convergence results. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes a multi-UAV collaborative task unloading method and system based on the improved MATD3 algorithm, introduces the Multi-Agent Twin Delayed Deep Deterministic policy gradient algorithm (MATD3) into the task unloading problem of multi-UAV assisted MEC, and uses the priority experience replay mechanism and importance sampling to improve the algorithm by calculating the priority of each sample; in addition, a genetic algorithm is used before training to find the optimal hyperparameter combination of the MATD3 algorithm. The present invention solves the problem of Q value overestimation that may occur in the DDPG algorithm, improves the robustness and adaptability of the reinforcement learning algorithm, and at the same time reduces the sensitivity of the reinforcement learning algorithm to the environment and parameters, accelerates the convergence speed, and reduces the total delay and total energy consumption generated by the UAV-assisted mobile edge computing system.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] The multi-UAV collaborative task offloading method based on the improved MATD3 algorithm includes the following steps:

[0008] S1: Establish a system model for multi-UAV collaborative task offloading;

[0009] S2: Establishing a communication model, a delay model and an energy consumption model according to the system model;

[0010] S3: Based on the communication model, the delay model and the energy consumption model, the PER-MATD3 algorithm is used to jointly design user scheduling, task offloading ratio, and the flight angle and speed of the UAV to minimize the total system cost and realize multi-UAV collaborative task offloading.

[0011] Preferably, in S1, the system model includes: M user devices {1, 2, ..., M} and N drones {1, 2, ..., N}, and the entire system model runs in equal time slots in discrete time; user device m generates a computationally intensive task in each time slot, and each task can be represented by a triple W m (t)=(D m (t),C m (t),T m (t)), where D m (t) represents the amount of task data generated by user equipment m in time slot t, in bits; C m (t) represents the amount of computing resources required for each bit of data in the task, in units of CPU computing cycles. m (t) represents the maximum tolerable delay of the task.

[0012] Preferably, in S2, the method for establishing a communication model includes:

[0013] The entire communication cycle is divided into T time slots of equal length. Each drone provides services to a user device within the coverage area in time slot t, where t∈{1,2,...,T};

[0014] After the user device offloads part of the computing tasks to the drone, the remaining computing tasks are performed locally;

[0015] The coordinate position of user equipment m in time slot t is in, and are the X and Y coordinates of user equipment m at time slot t;

[0016] Assuming that each UAV always maintains a fixed altitude H during the entire communication cycle, the initial coordinate position of UAV n at time slot t is The ending coordinate position is in, and are the X and Y coordinates of drone n at time slot t, and are the X and Y coordinates of drone n at time slot t+1;

[0017] The computing services provided by drones to ground users must meet the following coverage constraints:

[0018]

[0019] Among them, d m,n (t) is the horizontal distance between the user device and the drone, R max is the maximum coverage radius of the UAV, which is determined by the flight altitude H of the UAV and the maximum receiving angle θ of the receiving antenna, that is, R max =H tanθ;

[0020] The channel gain of the line-of-sight link between user equipment m and drone n is:

[0021]

[0022] Among them, g 0 is the channel gain when the reference distance is 1m, d m,n is the Euclidean distance between user device m and drone n;

[0023] The wireless transmission rate between user equipment m and drone n is:

[0024]

[0025] Where B is the communication bandwidth, P up is the transmit power of the user equipment in the uplink, σ 2 is the noise power, P NLOS is the transmission loss, f m,n (t) indicates whether there is an obstacle between user device m and drone n at time slot t, f m,n (t)=1 means existence, f m,n (t)=0 means non-existence.

[0026] Preferably, in S2, the method for establishing the delay model includes:

[0027] A partial offloading strategy is adopted for the tasks generated by the user equipment in each time slot. The transmission delay required for the user equipment m to transmit part of the task to the drone n is:

[0028]

[0029] Among them, O m,n (t)∈[0,1], represents the proportion of tasks offloaded to the UAV, D m (t) represents the amount of task data generated by user equipment m in time slot t, in bits;

[0030] The execution delay of this offloading part on the drone MEC server is:

[0031]

[0032] in, represents the CPU computing frequency of the MEC server carried by drone n, C m (t) represents the amount of computing resources required for each bit of data in the task, in units of CPU computing cycles;

[0033] The local execution delay of user equipment m in time slot t is:

[0034]

[0035] in, Indicates the CPU computing frequency of user device m, 1-0 m,n (t) represents the proportion of remaining tasks executed locally by the user;

[0036] When unloading at any ratio, the total delay is:

[0037]

[0038] Where M represents the number of user devices, α m,n (t)∈{0,1}, when user equipment m is scheduled by drone n, α m,n (t)=1, otherwise α m,n (t)=0, and satisfies That is to ensure that each drone only dispatches one user to serve in time slot t.

[0039] Preferably, in S2, the method for establishing the energy consumption model includes:

[0040] User equipment energy consumption: in, The energy consumption generated by offloading some tasks locally, The energy consumption when transmitting data from the user device to the drone;

[0041] Drone power consumption: in, The energy consumed by the drone during this flight. The energy consumed by the drone to perform the mission, E battery The maximum capacity of all drone batteries.

[0042] Preferably, in S3, based on the communication model, the delay model and the energy consumption model, the PER-MATD3 algorithm is used to jointly design user scheduling, task offloading ratio, and the flight angle and speed of the drone, and the method for minimizing the total system cost includes:

[0043] S31: Based on the communication model, the delay model and the energy consumption model, a constrained Markov decision process is established, which uses a four-tuple<S,A,R,C> express;

[0044] S32: Quad-based<S,A,R,C> It indicates that the genetic algorithm is used to find the optimal combination of hyperparameters of the MATD3 algorithm;

[0045] S33: Train the PER-MATD3 algorithm based on genetic algorithm parameter optimization to minimize the total system cost and achieve multi-UAV collaborative task offloading.

[0046] Preferably, in S31, the quaternary<S,A,R,C> It is expressed as:

[0047] State space S: It is determined by the user device information, drone information and environment information, including the task volume W generated by the user device in time slot t m (t), location information p of the user device m (t), real-time power of the drone Drone location information n (t) and whether there are obstacles between the user device and the drone. m,n (t);

[0048] Action space A: Each drone selects the user device α to be served in time slot t by observing the state of the environment and the actions of other drones m,n (t), the flight angle of the drone β(t), the flight speed of the drone v n (t) and the task offloading ratio O of the user equipment to the drone m,n (t);

[0049] Reward space R: After the drone performs an action in the current state, it will immediately receive a reward R based on the feedback from the environment. n (t); R n (t)=-U n (t) = -(λ 1 E n (t)+λ 2 T n (t)) where U n (t) represents the system cost of UAV n, which is composed of the weighted sum of energy consumption and delay; λ 1 and λ 2 are weights, indicating the importance of energy consumption or delay optimization, λ 1 >λ 2 For energy-saving scenarios, 1 <λ 2 For delay-sensitive scenarios;

[0050] Constraint C: The action selected by the agent at each time slot t needs to satisfy the following constraints:

[0051] α m,n (t)∈{0,1}, 0≤O m,n (t)≤1,0≤v n (t)≤v max , 0≤β n (t)≤2π, d m,n (t)≤R max , f m,n (t)∈{0,1},

[0053] Preferably, in S32, based on the quaternion<S,A,R,C> It indicates that the method of using genetic algorithm to find the optimal combination of hyperparameters of MATD3 algorithm includes:

[0054] S321: Randomly initialize a group of individuals as the initial population for the parameters to be optimized, each individual represents a hyperparameter combination;

[0055] S322: For each individual in the population, that is, the hyperparameter combination, the MATD3 algorithm is used for training, and the fitness is calculated according to the size of the accumulated reward in the specified episode;

[0056] S323: According to the fitness of the individuals, a part of the individuals are selected to enter the next generation, a crossover operation is performed on the selected parent individuals, and then a mutation operation is performed on the individuals of the new generation;

[0057] S324: Repeat S321 to S323 until the fitness value reaches a given threshold, and select the hyperparameter combination with the largest fitness value as the optimal set of hyperparameters.

[0058] Preferably, in S33, the method of training the PER-MATD3 algorithm based on genetic algorithm parameter optimization to minimize the total system cost and realize multi-UAV collaborative task offloading includes:

[0059] S331: Network parameter initialization;

[0060] S332: Replace the original experience playback mechanism with a priority experience playback mechanism based on the Sumtree binary tree structure;

[0061] S333: For each episode and each time slot t, each drone n moves according to the current state s n (t) Select the action to be performed Get instant rewards n (t) and the next state s n (t+1), the experience (s n (t),a n (t),r n (t),s n (t+1)) is stored in the experience pool, s n (t) = s n (t+1), calculate the priority and importance sampling weight of each group of experience;

[0062] S334: When the experience pool reaches the maximum capacity, the priority experience playback mechanism is used to extract J groups of samples from the experience pool of drone n, and then the samples of the same time slot are taken from the experience pools of other drones. The jth group of experience after splicing is expressed as (s j ,a j ,r j ,s′ j );

[0063] S335: Update Eval Critic 1 and Eval Critic 2 Network, first calculate the state s′ j The corresponding action value Then calculate the target Q value y j , and finally update the Eval Critic by minimizing the loss 1 and Eval Critic 2 network;

[0064] S336: Update the Eval Actor network. After every delayed update frequency f time slots, each drone updates the parameters of the Eval Actor network using the gradient ascent method.

[0065] S337: Update Target Actor, Target Critic 1 and Target Critic 2 Network, after every delayed update frequency f time slots, each UAV updates the parameters of the target network through soft update;

[0066] S338: Optimal strategy deployment. After completing the centralized training process, each drone obtains the optimal strategy of the Eval Actor network. The network is then deployed on each drone and enters the distributed execution phase to achieve multi-drone collaborative task offloading. That is, when the user device requests an offload service from the drone associated with it, the drone will locally observe the state s t Input to own network To select an optimal action a t And get the best reward r t (s t ,a t ).

[0067] The present invention also provides a multi-UAV collaborative task unloading system based on an improved MATD3 algorithm, comprising: a system model building module, a function model building module, and a joint design module;

[0068] The system model building module is used to establish a system model for multi-UAV collaborative task offloading;

[0069] The functional model building module is used to establish a communication model, a delay model and an energy consumption model according to the system model;

[0070] The joint design module is used to jointly design user scheduling, task offloading ratio, and flight angle and speed of the UAV based on the communication model, the delay model and the energy consumption model using the PER-MATD3 algorithm to minimize the total system cost and realize multi-UAV collaborative task offloading.

[0071] Compared with the prior art, the present invention has the following beneficial effects:

[0072] The present invention provides a multi-UAV collaborative task unloading method and system based on the improved MATD3 algorithm, which solves the problem of Q value overestimation that may occur in the DDPG algorithm, improves the robustness and adaptability of the reinforcement learning algorithm, and reduces the algorithm's sensitivity to the environment and parameters, accelerates the convergence speed, and reduces the total delay and total energy consumption generated by the UAV-assisted mobile edge computing system. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0074] Figure 1 A schematic diagram of a system for multi-UAV collaborative task offloading according to an embodiment of the present invention;

[0075] Figure 2 The framework and structure diagram of the PER-MATD3 algorithm of an embodiment of the present invention;

[0076] Figure 3 This is a comparison chart of the total system reward changes of the proposed algorithm and the benchmark algorithm as the number of rounds increases.

[0077] Figure 4The present invention is a flowchart of a multi-UAV collaborative task offloading method based on an improved MATD3 algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0078] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0079] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0080] Embodiment 1

[0081] like Figure 1-4 As shown in FIG, the multi-UAV collaborative task offloading method based on the improved MATD3 algorithm includes the following steps:

[0082] S1: Establish a multi-UAV collaborative task offloading system model;

[0083] S2: Establish communication model, delay model and energy consumption model according to the system model;

[0084] S3: The PER-MATD3 algorithm is used to jointly design user scheduling, task offloading ratio, and the flight angle and speed of the UAV to minimize the total system cost and achieve multi-UAV collaborative task offloading.

[0085] Preferably, the specific steps of S1 are as follows:

[0086] S1 establishes a multi-UAV collaborative task offloading system model. The system includes: M user devices {1,2,...,M} and N UAVs {1,2,...,N}. The entire system runs in equal time slots in discrete time. User device m generates a computationally intensive task in each time slot. Each task can be represented by a triple W m (t)=(D m (t),C m (t),T m (t)), where D m (t) represents the amount of task data generated by user equipment m in time slot t, in bits; C m (t) represents the amount of computing resources required for each bit of data in the task, in units of CPU computing cycles. m(t) represents the maximum tolerable delay of the task. Due to limited computing power, the user equipment must transfer part of the computing tasks to the drone through the wireless network for offloading. In each time slot, it is assumed that the drone takes the approach of moving first and then hovering to provide selective services to the user equipment within the coverage area; it is assumed that all user equipment moves randomly before the start of the next time slot; it is assumed that each drone can provide communication and computing services to all user equipment within the coverage area, but only provides services to one user equipment at a time.

[0087] Preferably, the specific steps of establishing the model in S2 are as follows:

[0088] S2.1 Establish a communication model: Divide the entire communication cycle into T time slots of equal length. Each drone provides services to a user device within the coverage area in time slot t, where t∈{1,2,...,T}. After the user device offloads part of the computing tasks to the drone, the remaining computing tasks are executed locally. The coordinate position of the user device m in time slot t is in, and are the X and Y coordinates of user device m at time slot t; assuming that each drone always maintains a fixed altitude H during the entire communication cycle, the initial coordinate position of drone n at time slot t is The ending coordinate position is in, and are the X and Y coordinates of drone n at time slot t, and 1) are the X and Y coordinates of drone n at time slot t+1.

[0089] The computing services provided by drones to ground users must meet the following coverage constraints:

[0090]

[0091] Among them, d m,n (t) is the horizontal distance between the user device and the drone, R max is the maximum coverage radius of the UAV, which is determined by the flight altitude H of the UAV and the maximum receiving angle θ of the receiving antenna, that is, R max =Htanθ.

[0092] The channel gain of the line-of-sight link between user equipment m and drone n is:

[0093]

[0094] Among them, g 0 is the channel gain when the reference distance is 1m, d m,nis the Euclidean distance between user device m and drone n. Due to the obstruction of obstacles, the wireless transmission rate between user device m and drone n is:

[0095]

[0096] Where B is the communication bandwidth, P up is the transmit power of the user equipment in the uplink, σ 2 is the noise power, P NLOS is the transmission loss, f m,n (t) indicates whether there is an obstacle between user device m and drone n at time slot t, f m,n (t)=1 means existence, f m,n (t)=0 means non-existence.

[0097] S2.2 Establishing a delay model: In this MEC system, a partial offloading strategy is adopted for the tasks generated by the user equipment in each time slot. The transmission delay required for the user equipment m to transmit part of the task to the drone n is:

[0098]

[0099] Among them, O m,n (t)∈[0,1], represents the proportion of tasks offloaded to the UAV.

[0100] The execution delay of the offloaded part on the drone MEC server is:

[0101]

[0102] in, represents the CPU computing frequency of the MEC server carried by drone n. This paper considers an arbitrary ratio offloading model, so the local execution delay of user device m in time slot t is:

[0103]

[0104] in, Indicates the CPU computing frequency of user device m, 1-0 m,n (t) represents the proportion of remaining tasks executed locally by the user.

[0105] In the MEC system, the data returned by the drone's MEC server after processing is usually very small and can be ignored. Therefore, the downlink transmission delay is not considered here. In addition, in order to meet the quality of service requirements, it is stipulated that the time to complete the computing task in any time slot cannot exceed the maximum tolerable delay of the task:

[0106]

[0107] It can be concluded that when unloading at any ratio, the total delay caused by the MEC system is:

[0108]

[0109] Among them, α m,n (t)∈{0,1}, when user equipment m is scheduled by drone n, α m,n (t)=1, otherwise α m,n (t)=0, and satisfies That is to ensure that each drone only dispatches one user to serve in time slot t.

[0110] S2.3 Establish energy consumption model:

[0111] S2.3.1 User device energy consumption: The process of transmitting tasks not only consumes time but also generates energy. In the partial offloading model, only part of the task is uploaded to the drone, and the other part is executed locally. The power consumption of user device m for local calculation is:

[0112]

[0113] Among them, k 1 is the energy consumed by the CPU of the user device per round, usually k 1 Take 10 -27 . Then the energy consumption generated by local partial task offloading is:

[0114]

[0115] The other part is offloaded at the edge of the drone, and part of the energy consumption is generated when transmitting data:

[0116]

[0117] Among them, P up is the transmit power of the user equipment.

[0118] It can be concluded that when unloading at any ratio, the total energy consumption of the MEC system is:

[0119]

[0120] S2.3.2 UAV power consumption: The present invention introduces a UAV power consumption model, and minimizes the UAV power consumption, including flight energy consumption and execution energy consumption, by jointly designing the flight angle and speed of the UAV.

[0121] In each time slot t, UAV n always moves from its initial hovering position q n(t) Fly to the end hovering position. The horizontal position coordinates of the drone after the flight are:

[0122]

[0123] Among them, the flight speed of the drone v n (t)∈[0,v max ], flight angle β n (t)∈[0,2π],t fly For a fixed flight time, the coordinates of the drone's end hovering position are:

[0124]

[0125] The energy consumed by the drone during this flight can be expressed as:

[0126]

[0127] in, M uav is the weight of the drone.

[0128] The energy consumption generated by executing offload on the edge side of the drone is not only the part of transmitting data, but also the energy consumption generated by executing on the drone MEC server. The power consumption is:

[0129]

[0130] The energy consumed by the drone to perform the mission is:

[0131]

[0132] Among them, k 2 is the energy consumed by the CPU of the MEC server on the drone per round. In this paper, k 2 =k 1 To ensure that the flight and computing energy consumption of the drone in all time slots does not exceed the maximum capacity of the battery, the following conditions must be met:

[0133]

[0134] Among them, E battery The maximum capacity of all drone batteries.

[0135] Preferably, the specific steps of the PER-MATD3 algorithm in S3 are as follows:

[0136] S3.1 remodels the above problem as a constrained Markov decision process, which uses a four-tuple<S,A,R,C> express;

[0137] State space S: In this MEC system, the state space is determined by the user device information, drone information, and environmental information, including the task volume W generated by the user device in time slot t m (t), location information p of the user device m (t), real-time power of the drone Drone location information n (t) and whether there are obstacles between the user device and the drone. m,n (t).

[0138] Action space A: Each drone selects the user device α to be served in time slot t by observing the state of the environment and the actions of other drones m,n (t), the flight angle of the drone β(t), the flight speed of the drone v n (t) and the task offloading ratio O of the user equipment to the drone m,n (t).

[0139] Reward space R: After the drone performs an action in the current state, it will immediately receive a reward R based on the feedback from the environment. n (t):

[0140] R n (t)=-U n (t) = -(λ 1 E n (t)+λ 2 T n (t)) (20)

[0141] Among them, U n (t) represents the system cost of UAV n, which is composed of the weighted sum of energy consumption and delay; λ 1 and λ 2 are weights, indicating the importance of energy consumption or delay optimization, λ 1 >λ 2 For energy-saving scenarios, 1 <λ 2 This is a latency-sensitive scenario.

[0142] In order to achieve the optimization goal, the agent needs to obtain the largest possible reward value. A collaborative deep reinforcement learning architecture is used to achieve the maximum expected reward with the sum of the rewards of all drones. The reward is finally defined as:

[0143]

[0144] Constraint C: The action selected by the agent at each time slot t needs to satisfy the following constraints:

[0145] α m,n(t)∈{0,1},(22)

[0146]

[0147] 0≤O m,n (t)≤1, (24)

[0148] 0≤v n (t)≤v max , (25)

[0149] 0≤β n (t)≤2π, (26)

[0150]

[0151]

[0152] d m,n (t)≤R max , (29)

[0153] f m,n (t)∈{0,1},(30)

[0154]

[0155] S3.2 Use genetic algorithm to find the optimal combination of MATD3 algorithm hyperparameters before training: Before training the MATD3 algorithm, use genetic algorithm to search for a set of optimal hyperparameters required by the MATD3 algorithm, mainly for five parameters: discount factor γ, polyak average coefficient τ, exploration noise variance Strategy Noise Variance and delay update frequency f;

[0156] S3.2.1 Randomly initialize a group of individuals as the initial population for the above parameters to be optimized, and each individual represents a hyperparameter combination;

[0157] S3.2.2 For each individual in the population, i.e., a hyperparameter combination, the MATD3 algorithm is used for training, and its fitness is calculated based on the size of its cumulative reward within the specified episode;

[0158] Define θ as a hyperparameter combination, R(θ,E) represents the cumulative reward obtained by the MATD3 algorithm under a given hyperparameter combination θ and a specified number of rounds E, and f(θ) is the fitness function, which evaluates the quality of θ based on R(θ,E). Assume that the reward obtained at the end of each episode is r e ,but:

[0159]

[0160] Considering that the cumulative rewards between different hyperparameter combinations may differ greatly, the minimum and maximum normalization is used to process them, and the fitness function is obtained as follows:

[0161]

[0162] Among them, min θ′ R(θ′,E) and max θ′ R(θ′,E) are the minimum and maximum cumulative rewards across all hyperparameter combinations, respectively.

[0163] S3.2.3 According to the fitness of the individuals, select a part of the individuals to enter the next generation, perform crossover operation on the selected parent individuals, and then perform mutation operation on the individuals of the new generation;

[0164] S3.2.4 Repeat the above steps until the fitness value reaches a given threshold, and select the hyperparameter combination with the largest fitness value as the optimal set of hyperparameters.

[0165] S3.3 The training steps of the PER-MATD3 algorithm based on genetic algorithm parameter optimization include:

[0166] S3.3.1 Network parameter initialization: Each drone n uses the TD3 algorithm to randomly initialize the Eval Actor network The parameter u n 、Eval Critic 1 network Parameters Eval Critic 2 network Parameters TargetActor Network The parameter u′ n Target Critic 1 network Parameters Target Critic 2 network Parameters The total number of training rounds max_episode, the number of time slots contained in each round T, the sampling batch size B, the optimal hyperparameter combination found by each TD3 algorithm using the genetic algorithm, and the ISW parameters α and β are initialized to 0.5.

[0167] S3.3.2 Replace the original experience replay mechanism with a priority experience replay mechanism based on the Sumtree binary tree structure: Use SumTree as a binary tree structure to store the priority of each sample, and use the absolute value of the loss value generated by the target Q value and the actual Q value, that is, |TD-error|, as the criterion for judging the importance of experience. The larger the difference, the more the prediction of the current model needs to be corrected, that is, samples with larger |TD-error| in the experience pool are sampled first. Define the |TD-error| of the h-th group of experience as |δ h |, then the probability of the hth tuple of the training sample being sampled is:

[0168]

[0169] Among them, H represents the total number of samples in the experience pool, α represents the importance of the ranking value, and is usually set to 0≤α≤1. When α=0, it means that all samples contribute the same amount to the sampling, that is, uniform sampling. When α increases, only P(h)=|δ h Only samples with larger |+ε will be sampled, and P(h) represents the same as δ h A related sorting value is set to 0<α<1 to prevent the unsampled P(h) from being zero.

[0170] Considering that the above approach may often result in high |δ h | value experience, thereby reducing the diversity of training samples, making the training process of the neural network more likely to oscillate or even diverge. Therefore, the importance sampling weight ISW is used to correct it. The following formula gives the importance sampling weight corresponding to the TD-error priority:

[0171] e h =(H·P(h)) -β (36)

[0172] Among them, e h is the importance sampling weight corresponding to the h-th group of experience, and β is the parameter of the degree of correction. β = 0 means that the importance sampling weight strategy is not used; when β = 1, the weight will eliminate the bias caused by the PER sampling probability, but it may reduce the convergence speed of the model. Therefore, 0 ≤ β ≤ 1, which can effectively sample and the model can also effectively converge. Considering the instability of convergence, equation (36) is normalized:

[0173]

[0174] S3.3.3 For each episode and each time slot t, each drone n moves according to its current state s n (t) Select the action to be performed Get instant rewards n(t) and the next state s n (t+1), the experience (s n (t),a n (t),r n (t),s n (t+1)) is stored in the experience pool, s n (t) = s n (t+1), calculate the priority and importance sampling weight of each group of experience.

[0175] S3.3.4 When the experience pool reaches its maximum capacity, the priority experience playback mechanism is used to extract J groups of samples from the experience pool of drone n, and then samples of the same time slot are taken from the experience pools of other drones. After splicing, the jth group of experience is expressed as (s j ,a j ,r j ,s′ j ).

[0176] S3.3.5 Update Eval Critic 1 and Eval Critic 2 Network: First calculate the state s′ j The corresponding action value

[0177]

[0178] in, Then the target Q value is:

[0179]

[0180] Each drone updates the Eval Critic by minimizing its loss 1 and Eval Critic 2 The network loss is calculated as:

[0181]

[0182] S3.3.6 Update Eval Actor Network: After every delayed update frequency f time slots, each drone uses the gradient ascent method to update the parameters of the Eval Actor network. The gradient calculation formula is:

[0183]

[0184] S3.3.7 Update Target Actor and Target Critic 1 and Target Critic 2Network: After every delayed update frequency f time slots, each drone updates the parameters of the target network through soft update:

[0185] u′ n =τu n +(1-τ)u′ n (42)

[0186]

[0187] S3.3.8 Optimal Strategy Deployment: After completing the centralized training process, each drone obtains the optimal strategy of the Eval Actor network The network is then deployed on each drone and enters the distributed execution phase to achieve multi-drone collaborative task offloading. That is, when the user device requests an offload service from the drone associated with it, the drone will locally observe the state s t Input to own network To select an optimal action a t And get the best reward r t (s t ,a t ).

[0188] S3.3.9 The overall process of implementing multi-UAV collaborative task offloading: It includes two stages: centralized training and decentralized execution. In the centralized training stage, each UAV forms a collaborative relationship by sharing information (such as status and action, etc.). The information of all UAVs is centrally collected and processed by splicing, so as to optimize the strategy and learning process of each UAV from a global perspective. This method enables the TD3 algorithm of each UAV to use the information of all UAVs to jointly optimize a global objective function, which is reflected in this article to achieve the overall optimal performance of the MEC system, that is, to minimize the total delay and total energy consumption generated by the UAV-assisted mobile edge computing system. In the decentralized execution stage, each UAV makes independent decisions based only on its own observations and strategies, and no longer relies on the information of other UAVs. Decentralized execution enables the system to respond to environmental changes more quickly, and each UAV no longer needs to wait for information from other UAVs or perform complex coordination.

[0189] like Figure 1As shown, this embodiment considers a mobile edge computing system consisting of three drones equipped with edge computing servers and multiple user devices. The entire communication cycle of 400s is divided into 40 time slots. Consider a square area with a side length of 100m. The drone must fly within the boundary and always fly at a fixed height of 100m. It is assumed that the drone only provides unloading services to one user device within the coverage area in each time slot. The channel bandwidth of the user device to the drone is 1MHz. The channel gain when the reference distance between the user device and the drone is 1m is -50dB, the noise power is set to -100dBm, the transmission power of the user device is set to 0.1W, and the task volume generated by the user device is randomly generated between [2,2.5]Mbits. The local CPU computing frequency is set to 0.6GHz, the CPU computing frequency on the drone is set to 1.2GHz, the maximum flight speed of the drone is set to 10m / s, the flight time is set to 1s, and the mass of the drone is 9.65kg.

[0190] like Figure 3 As shown, a reward value comparison between the proposed algorithm and the benchmark algorithm is given. In this embodiment, the PER-MATD3 algorithm that has been optimized by genetic algorithm parameter adjustment is named GA-PER-MATD3 algorithm. As can be seen from the figure, the five algorithms all have a convergence trend, the MADDPG algorithm has the slowest convergence speed and the smallest convergence amplitude; the MATD3 algorithm is superior to the MADDPG algorithm in all aspects, which proves the effectiveness of the MATD3 algorithm in solving the problem of overestimation of Q value; the convergence curves of the GA-MATD3 algorithm after genetic algorithm parameter adjustment and the PER-MATD3 algorithm using the priority experience replay mechanism are almost the same, and the convergence values ​​are higher than the MATD3 algorithm; the algorithm proposed in the present invention fully combines the advantages of GA and PER, and the convergence values ​​are higher than other algorithms and the convergence speed is the fastest. By comparing Figure 3 The comparison between the curve and the data verifies the availability and effectiveness of the proposed algorithm in reducing the total system delay and total energy consumption and accelerating the convergence speed.

[0191] Embodiment 2

[0192] The present invention also provides a multi-UAV collaborative task unloading system based on an improved MATD3 algorithm, comprising: a system model building module, a function model building module, and a joint design module;

[0193] The system model building module is used to establish a system model for multi-UAV collaborative task offloading;

[0194] The functional model building module is used to establish a communication model, a delay model and an energy consumption model according to the system model;

[0195] The joint design module is used to jointly design user scheduling, task offloading ratio, and flight angle and speed of UAVs based on the communication model, the delay model, and the energy consumption model using the PER-MATD3 algorithm to minimize the total system cost and realize multi-UAV collaborative task offloading.

[0196] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A multi-UAV collaborative task offloading method based on the improved MATD3 algorithm is characterized by: The following steps are involved: S1: Establish a system model for multi-UAV collaborative task offloading; S2: Establishing a communication model, a delay model and an energy consumption model according to the system model; S3: Based on the communication model, the delay model and the energy consumption model, the PER-MATD3 algorithm is used to jointly design user scheduling, task offloading ratio, and the flight angle and speed of the UAV to minimize the total system cost and realize multi-UAV collaborative task offloading.

2. The multi-UAV collaborative task offloading method based on the improved MATD3 algorithm according to claim 1 is characterized in that: In S1, the system model includes: M user devices {1, 2, ..., M} and N drones {1, 2, ..., N}. The entire system model runs in equal time slots in discrete time. User device m generates a computationally intensive task in each time slot. Each task can be represented by a triple W m (t)=(D m (t),C m (t),T m (t)), where D m (t) represents the amount of task data generated by user equipment m in time slot t, in bits; C m (t) represents the amount of computing resources required for each bit of data in the task, in units of CPU computing cycles. m (t) represents the maximum tolerable delay of the task.

3. The multi-UAV collaborative task offloading method based on the improved MATD3 algorithm according to claim 2 is characterized in that: In S2, the method for establishing a communication model includes: The entire communication cycle is divided into T time slots of equal length. Each drone provides services to a user device within the coverage area in time slot t, where t∈{1,2,...,T}; After the user device offloads part of the computing tasks to the drone, the remaining computing tasks are performed locally; The coordinate position of user equipment m in time slot t is in, and are the X and Y coordinates of user equipment m at time slot t; Assuming that each UAV always maintains a fixed altitude H during the entire communication cycle, the initial coordinate position of UAV n at time slot t is The ending coordinate position is in, and are the X and Y coordinates of drone n at time slot t, and are the X and Y coordinates of drone n at time slot t+1; The computing services provided by drones to ground users must meet the following coverage constraints: Among them, d m,n (t) is the horizontal distance between the user device and the drone, R max is the maximum coverage radius of the UAV, which is determined by the flight altitude H of the UAV and the maximum receiving angle θ of the receiving antenna, that is, R max =H tanθ; The channel gain of the line-of-sight link between user equipment m and drone n is: Where g0 is the channel gain when the reference distance is 1m, d m,n is the Euclidean distance between user device m and drone n; The wireless transmission rate between user equipment m and drone n is: Where B is the communication bandwidth, P up is the transmit power of the user equipment in the uplink, σ 2 is the noise power, P NLOS is the transmission loss, f m,n (t) indicates whether there is an obstacle between user device m and drone n at time slot t, f m,n (t)=1 means existence, f m,n (t)=0 means non-existence.

4. The multi-UAV collaborative task offloading method based on the improved MATD3 algorithm according to claim 3 is characterized in that: In S2, the method for establishing the delay model includes: A partial offloading strategy is adopted for the tasks generated by the user equipment in each time slot. The transmission delay required for the user equipment m to transmit part of the task to the drone n is: Among them, O m,n (t)∈[0,1], represents the proportion of tasks offloaded to the UAV, D m (t) represents the amount of task data generated by user equipment m in time slot t, in bits; The execution delay of this offloading part on the drone MEC server is: in, represents the CPU computing frequency of the MEC server carried by drone n, C m (t) represents the amount of computing resources required for each bit of data in the task, in units of CPU computing cycles; The local execution delay of user equipment m in time slot t is: in, Indicates the CPU computing frequency of user device m, 1-0 m,n (t) represents the proportion of remaining tasks executed locally by the user; When unloading at any ratio, the total delay is: Where M represents the number of user devices, α m,n (t)∈{0,1}, when user equipment m is scheduled by drone n, α m,n (t)=1, otherwise α m,n (t)=0, and satisfies That is to ensure that each drone only dispatches one user to serve in time slot t.

5. The multi-UAV collaborative task offloading method based on the improved MATD3 algorithm according to claim 4 is characterized in that: In S2, the method for establishing the energy consumption model includes: User equipment energy consumption: in, The energy consumption generated by offloading some tasks locally, The energy consumption when transmitting data from the user device to the drone; Drone power consumption: in, The energy consumed by the drone during this flight. The energy consumed by the drone to perform the mission, E battery The maximum capacity of all drone batteries.

6. The multi-UAV collaborative task offloading method based on the improved MATD3 algorithm according to claim 5 is characterized in that: In S3, based on the communication model, the delay model and the energy consumption model, the PER-MATD3 algorithm is used to jointly design user scheduling, task offloading ratio, and the flight angle and speed of the drone, and the method for minimizing the total system cost includes: S31: Based on the communication model, the delay model and the energy consumption model, a constrained Markov decision process is established, which uses a four-tuple<S,A,R,C> express; S32: Quad-based<S,A,R,C> It indicates that the genetic algorithm is used to find the optimal combination of hyperparameters of the MATD3 algorithm; S33: Train the PER-MATD3 algorithm based on genetic algorithm parameter optimization to minimize the total system cost and achieve multi-UAV collaborative task offloading.

7. The multi-UAV collaborative task offloading method based on the improved MATD3 algorithm according to claim 6 is characterized in that: In S31, the quaternion<S,A,R,C> It is expressed as: State space S: It is determined by the user device information, drone information and environment information, including the task volume W generated by the user device in time slot t m (t), location information p of the user device m (t), real-time power of the drone Drone location information n (t) and whether there are obstacles between the user device and the drone. m,n (t); Action space A: Each drone selects the user device α to be served in time slot t by observing the state of the environment and the actions of other drones m,n (t), the flight angle of the drone β(t), the flight speed of the drone v n (t) and the task offloading ratio O of the user equipment to the drone m,n (t); Reward space R: After the drone performs an action in the current state, it will immediately receive a reward R based on the feedback from the environment. n (t); R n (t)=-U n (t) = -(λ1E n (t)+λ2T n (t)) where U n (t) represents the system cost of UAV n, which is composed of the weighted sum of energy consumption and delay; λ1 and λ2 are weights, which respectively indicate the importance of energy consumption or delay optimization. λ1>λ2 is an energy-saving scenario, and λ1<λ2 is a delay-sensitive scenario. Constraint C: The action selected by the agent at each time slot t needs to satisfy the following constraints: α m,n (t)∈{0,1}, 0≤O m,n (t)≤1,0≤v n (t)≤v max ,0≤β n (t)≤2π, d m,n (t)≤R max ,f m,n (t)∈{0,1}, 8. The multi-UAV collaborative task offloading method based on the improved MATD3 algorithm according to claim 7 is characterized in that: In S32, based on the quaternion<S,A,R,C> It indicates that the method of using genetic algorithm to find the optimal combination of hyperparameters of MATD3 algorithm includes: S321: Randomly initialize a group of individuals as the initial population for the parameters to be optimized, each individual represents a hyperparameter combination; S322: For each individual in the population, that is, the hyperparameter combination, the MATD3 algorithm is used for training, and the fitness is calculated according to the size of the accumulated reward in the specified episode; S323: According to the fitness of the individuals, a part of the individuals are selected to enter the next generation, a crossover operation is performed on the selected parent individuals, and then a mutation operation is performed on the individuals of the new generation; S324: Repeat S321 to S323 until the fitness value reaches a given threshold, and select the hyperparameter combination with the largest fitness value as the optimal set of hyperparameters.

9. The multi-UAV collaborative task offloading method based on the improved MATD3 algorithm according to claim 8 is characterized in that: In S33, the method of training the PER-MATD3 algorithm based on genetic algorithm parameter optimization to minimize the total system cost and realize multi-UAV collaborative task offloading includes: S331: Network parameter initialization; S332: Replace the original experience playback mechanism with a priority experience playback mechanism based on the Sumtree binary tree structure; S333: For each episode and each time slot t, each drone n moves according to the current state s n (t) Select the action to be performed Get instant rewards n (t) and the next state s n (t+1), the experience (s n (t),a n (t),r n (t),s n (t+1)) is stored in the experience pool, s n (t) = s n (t+1), calculate the priority and importance sampling weight of each group of experience; S334: When the experience pool reaches the maximum capacity, the priority experience playback mechanism is used to extract J groups of samples from the experience pool of drone n, and then the samples of the same time slot are taken from the experience pools of other drones. The jth group of experience after splicing is expressed as (s j ,a j ,r j ,s′ j ); S335: Update Eval Critic1 and Eval Critic2 networks, first calculate the state s′ j The corresponding action value Then calculate the target Q value y j , and finally update the Eval Critic1 and Eval Critic2 networks by minimizing the loss; S336: Update the EvalActor network. After every delayed update frequency f time slots, each drone updates the parameters of the EvalActor network using the gradient ascent method. S337: Update the Target Actor, Target Critic1 and Target Critic2 networks. After every delayed update frequency f time slots, each UAV updates the parameters of the target network through soft update. S338: Optimal strategy deployment. After completing the centralized training process, each drone obtains the optimal strategy of the Eval Actor network. The network is then deployed on each drone and enters the distributed execution phase to achieve multi-drone collaborative task offloading. That is, when the user device requests an offload service from the drone associated with it, the drone will locally observe the state s t Input to own network To select an optimal action a t And get the best reward r t (s t ,a t ).

10. The multi-UAV collaborative task unloading system based on the improved MATD3 algorithm is characterized by: include: System model building module, functional model building module, joint design module; The system model building module is used to establish a system model for multi-UAV collaborative task offloading; The functional model building module is used to establish a communication model, a delay model and an energy consumption model according to the system model; The joint design module is used to jointly design user scheduling, task offloading ratio, and flight angle and speed of the UAV based on the communication model, the delay model and the energy consumption model using the PER-MATD3 algorithm to minimize the total system cost and realize multi-UAV collaborative task offloading.

Citation Information

Cited By

  • Multi-unmanned aerial vehicle air combat strategy generation method and device and computer equipment

    CN116430888A

  • Robust task unloading method for unmanned aerial vehicle assisted mobile edge calculation

    CN120321715A

  • Communication perception calculation intelligent cooperation method for low-altitude city scene

    CN120416917A

  • Unmanned aerial vehicle multi-task distribution system

    CN120578179A