A multi-unmanned aerial vehicle trajectory planning and resource allocation method and system based on multi-agent optimization

Abstract

The application discloses a multi-unmanned aerial vehicle trajectory planning and resource allocation method and system based on multi-agent optimization, wherein the multi-unmanned aerial vehicle trajectory planning and resource allocation method based on multi-agent optimization comprises the following steps: constructing a system model; after the construction of the system model is completed, designing an algorithm; training an agent neural network, and outputting, according to the designed algorithm, the computing power resources allocated to users by the unmanned aerial vehicle and the computing task offloading proportion of the users. A multi-unmanned aerial vehicle distributed collaborative computing MEC system is designed, the unmanned aerial vehicle trajectory, the task migration proportion between unmanned aerial vehicles, the unmanned aerial vehicle access selection and the pricing of unit computing power resources are jointly optimized, the unmanned aerial vehicle benefit is maximized, and the computing power resources allocated to users by the unmanned aerial vehicle and the computing task offloading proportion of the users are jointly optimized, and the user cost is minimized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of mobile communication, in particular, to a multi-UAV trajectory planning and resource allocation method and system based on multi-agent optimization. BACKGROUND

[0002] With the explosive development of 5G communication and intelligent sensing technology, the Internet of Things is also developing rapidly. Mobile edge computing (MEC) provides a low-cost and high-benefit solution by pushing computing resources to the network edge close to the end user (such as base stations) to meet the computing-intensive and delay-sensitive tasks. However, the limited coverage of static MEC brings new challenges. In particular, in the absence of available infrastructure (such as mountainous areas, earthquake response and emergency situations, etc.), static MEC will not work. Due to the low cost, high flexibility and easy deployment of UAVs, integrating MEC servers into UAVs can solve the problem of traditional fixed-position MEC. The key technology of UAV-aided MEC is the deployment and task scheduling of UAVs. In recent years, the research on single-UAV networking is relatively comprehensive, including mobile relaying, auxiliary computing, etc. However, the massive data of mobile users with computing needs are usually generated in a random or bursty manner, which may cause the computing capacity of a single UAV server to be unbalanced or even collapsed. Compared with a single UAV server, a MEC network composed of multiple UAVs has stronger networking computing capacity, flexibility and adaptability. In addition, most of the work adopts offline algorithms for system design, i.e., under the premise that the user position is constant, the user demand is fixed or known, the entire trajectory and resource allocation scheme are planned using prior environmental information. However, in practice, smart phones, tablets, wearable devices, tracking collars carried by wild animals, etc. are usually in motion, and their trajectories are likely to be random. In order to serve mobile terminals, online algorithms are needed to make decisions based on real-time information. In the case where the distribution of random events is known, dynamic programming is an effective method to find the optimal control strategy. However, in many practical applications, the prior knowledge of the parameter distribution is also lacking. In this case, the Deep Reinforcement Learning (DRL) algorithm as a suitable and powerful tool successfully solves the problem of automatic control and decision-making in a random dynamic environment by learning from its own experience. And existing work mainly focuses on the research of centralized multi-UAV-aided MEC system, however, the centralized architecture has the problem of single point of failure and limited scalability, so it is necessary to study a flexible and reliable distributed architecture.

[0003] Therefore, how to provide a multi-UAV trajectory planning and resource allocation method for improving the efficiency of UAVs in a multi-UAV distributed collaborative computing system has become a problem that needs to be solved in the field. SUMMARY

[0004] The application provides a multi-agent optimization-based multi-unmanned aerial vehicle trajectory planning and resource allocation method, including the following steps: constructing a system model; after the system model is constructed, designing an algorithm; training an agent neural network, and outputting the computing power resources allocated to users and the computing task offloading proportion of users according to the designed algorithm.

[0005] The multi-agent optimization-based multi-unmanned aerial vehicle trajectory planning and resource allocation method as described above, wherein the construction of the system model includes the following sub-steps: constructing a node movement model of users and unmanned aerial vehicles; after the construction of the node movement model of users and unmanned aerial vehicles is completed, constructing a task generation and calculation model; after the construction of the task generation and calculation model is completed, constructing an optimization target.

[0006] The multi-agent optimization-based multi-unmanned aerial vehicle trajectory planning and resource allocation method as described above, wherein the construction of the node movement model of users and unmanned aerial vehicles includes updating the horizontal coordinates q m (t) of the unmanned aerial vehicle m at time slot t∈T m (t),y m (t)].

[0007] x m (t)=x m (t-1)+v m (t)cos(α m (t))τ t , y m (t)=y m (t-1)+v m (t)sin(α m (t))τ t , x m (t-1) and y m (t-1) represent the coordinates of the unmanned aerial vehicle m at time slot t-1, v m (t) represents the flight speed of the unmanned aerial vehicle m at time slot t, α m (t) represents the direction of the unmanned aerial vehicle m at time slot t, and τ t represents the flight time of the unmanned aerial vehicle at time slot t.

[0008] The multi-agent optimization-based multi-unmanned aerial vehicle trajectory planning and resource allocation method as described above, wherein the construction of the node movement model of users and unmanned aerial vehicles includes updating the movement speed v k (t) and the angle α k (t) of the user k∈K at time slot t.

[0009] v k(t-1) represents the movement speed of user k∈K at time slot t-1, α k (t-1) represents the angle of user k∈K at time slot t-1, c1 is a pre-set constant, represents the correlation between the movement speed of user k∈K at time slot t-1 and the current time slot t, c2 is a pre-set constant, represents the correlation between the moving angle of user k∈K at time slot t-1 and the current time slot t, is the average speed of all users, is the average angle of all users, Φ k and Ψ k are two independent Gaussian random variables, respectively.

[0010] The multi-robot trajectory planning and resource allocation method based on multi-agent optimization as described above, wherein after the construction of the system model, the algorithm design includes the following sub-steps: the design of the leader decision algorithm based on MAPPO; the design of the follower decision algorithm based on two-stage iteration.

[0011] A multi-robot trajectory planning and resource allocation system based on multi-agent optimization, specifically comprising: a model construction unit, an algorithm design unit and a training output unit; the model construction unit is used to construct the system model; the algorithm design unit is used to design the algorithm; the training output unit is used to train the agent neural network, and output the computing power resources allocated by the unmanned aerial vehicle to the user and the computing task offloading ratio of the user according to the designed algorithm.

[0012] The multi-robot trajectory planning and resource allocation system based on multi-agent optimization as described above, wherein the model construction unit constructs the system model, including the following sub-steps: constructing the node movement model of the user and the unmanned aerial vehicle; after the construction of the node movement model of the user and the unmanned aerial vehicle is completed, constructing the task generation and calculation model; after the construction of the task generation and calculation model is completed, constructing the optimization target.

[0013] The multi-robot trajectory planning and resource allocation system based on multi-agent optimization as described above, wherein the model construction unit constructs the node movement model of the user and the unmanned aerial vehicle, including, updating the horizontal coordinate q m (t) of unmanned aerial vehicle m at time slot t∈T m (t), y m (t)] ; x m (t) = x m (t-1) + v m (t)cos(α m (t))τ t , y m (t) = y m (t-1) + v m (t)sin(αm (t) t , x m (t-1) and y m (t-1) represent the coordinates of the UAV m at time slot t-1, v m (t) represents the flight speed of the UAV m at time slot t, a m (t) represents the direction of the UAV m at time slot t, and t t represents the flight time of the UAV at time slot t.

[0014] The multi-UAV trajectory planning and resource allocation system based on multi-agent optimization as described above, wherein the model construction unit constructs the node movement model of the user and the UAV, comprising updating the movement speed v k (t) and the angle a k (t) of the user k e K at time slot t; v k (t-1) represents the movement speed of the user k e K at time slot t-1, a k (t-1) represents the angle of the user k e K at time slot t-1, c1 is a pre-set constant representing the correlation between the movement speed at time slot t-1 and the current time slot t, c2 is a pre-set constant representing the correlation between the angle at time slot t-1 and the current time slot t, is the average speed of all users, is the average angle of all users, and k and k are two independent Gaussian random variables, respectively.

[0015] The multi-UAV trajectory planning and resource allocation system based on multi-agent optimization as described above, wherein the algorithm design unit designs the algorithm, including the following sub-steps: designing the leader decision algorithm based on MAPPO; designing the follower decision algorithm based on two-stage iteration.

[0016] The present application has the following beneficial effects:

[0017] The present application designs a multi-UAV distributed collaborative computing MEC system, which maximizes the UAV benefit by jointly optimizing the UAV trajectory, the task migration ratio between UAVs, the UAV access selection and the pricing of unit computing resources, and minimizes the user cost by jointly optimizing the computing resources allocated by the UAV to the user and the computing task offloading ratio of the user. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced. Obviously, the accompanying drawings in the following description only represent some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0019] Figure 1 is a flowchart of a multi-unmanned aerial vehicle trajectory planning and resource allocation method based on multi-agent optimization provided according to an embodiment of the present application;

[0020] Figure 2 is a multi-unmanned aerial vehicle cooperative auxiliary MEC system model provided according to an embodiment of the present application;

[0021] Figure 3 is an internal structure schematic diagram of a multi-unmanned aerial vehicle trajectory planning and resource allocation system based on multi-agent optimization provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0023] The present application constructs a multi-unmanned aerial vehicle distributed cooperative computing architecture. Unmanned aerial vehicles serve as agents to process tasks offloaded by users through cooperative computing, fully utilize the computing resources of unmanned aerial vehicles, expand the service range, and unmanned aerial vehicles can dynamically select to join or exit the system, thereby enhancing the self-repairing capability of the system. In view of the dynamic scene that the computing demand randomly arrives and the user location constantly changes, a leader decision algorithm based on Multiple Agent Proximal Policy Optimization (MAPPO) and a follower decision algorithm based on two-stage iteration are designed, thereby improving the unmanned aerial vehicle benefit in the multi-unmanned aerial vehicle distributed cooperative computing system and reducing the user overhead.

[0024] Embodiment One

[0025] As Figure 1As shown, the embodiment provides a multi-robot trajectory planning and resource allocation method based on multi-agent optimization. First, a node movement model of users and robots, a task generation and calculation model are constructed, and the optimization objectives of robots and users are given respectively. Then, a DRL model is constructed, the state space, action space and reward function are designed according to the optimization variables and objectives, the leader decision algorithm is designed, and the follower decision algorithm is designed based on the Lagrange multiplier method. Finally, enter the training stage, get the robot trajectory, the task migration ratio between robots, the robot access selection and the pricing of unit computing power resources by the MAPPO-based leader decision algorithm; get the computing power resources allocated by the robot to the user and the computing task offloading ratio of the user by the two-stage iteration-based follower decision algorithm, and input these actions to the environment, and train the algorithm model to convergence according to the reward feedback from the environment.

[0026] Specifically, the following steps are included:

[0027] Step S110: constructing a system model.

[0028] Consider a system containing multiple robots and multiple users, and the robots equipped with MEC servers can provide computing services for users, such as Figure 2 As shown, a multi-robot cooperative auxiliary MEC system model is provided. Through path discretization technology, the flight trajectory of the robot can be discretized into T+1 waypoints, i.e. T line segments. The time at which the robot is located on any line segment is represented as time slot t∈T, where the set T is defined as T={1,2,...,T}. In time slot t, the set of robots is represented as M={1,2,...,M(t)} and the set of users is represented as K={1,2,...,K}. The robot can dynamically join or exit the system at any time slot t according to its remaining power and system load, so the number of robots M(t) changes for each time slot. The flight time of the robot in time slot t is defined as τ t . Assuming that the robot flies at a constant speed in time slot t, since the duration of each time slot is short, it can be considered to be in a quasi-static state within a single time slot. In order to ensure that the robot is in a quasi-static state within each time slot, the length of time slot t is constrained as τ t ≤δ.

[0029] Based on the above, step S110 includes the following sub-steps:

[0030] Step S1101: constructing a node movement model of users and robots.

[0031] where the flight height of all robots is fixed at H. By establishing a Cartesian coordinate system, the horizontal coordinates of robot m in time slot t∈T can be represented as q m (t)=[x m(t),y m (t)], and define the horizontal starting point of the drone m as Given the flight speed v of UAV m in time slot t m (t)∈[0,V max ] and direction α m (t), where V max The maximum flight speed of the drone, and the coordinates of the drone (x) m (t), y m The update method for (t) is:

[0032] x m (t)=x m (t-1)+v m (t)cos(α m (t))τ t (1)

[0033] y m (t)=y m (t-1)+v m (t)sin(α m (t))τ t (2)

[0034] The horizontal coordinate of user k∈K in time slot t is defined as w. k (t)=[x k (t),y k (t)]. The user's movement is modeled as a Gaussian-Markov random motion model. Therefore, the user's movement rate v in time slot t is... k (t) and angle α k The update method for (t) is as follows:

[0035]

[0036] Here, c1 and c2 are constants, representing the relationship between the previous state and the current state. and These represent the average speed and average angle for all users, respectively. Φ k and Ψ k These are two independent Gaussian random variables, reflecting the randomness of user movement.

[0037] Therefore, the horizontal coordinate (x) of user k∈K in time slot t k (t), y k (t) is represented as:

[0038] x k (t)=x k (t-1)+v k (t)cos(αk (t) = y t (5)

[0039] y k (t) = y k (t - 1) + v k (t) sin(a k (t)) t t (6)

[0040] In the real world, due to the existence of obstacles such as buildings and mountains, the actual path loss between the UAV and the user is determined by the probability of Line-of-Sight (LoS) link and Non-Line-of-Sight (NLoS) link. At time slot t, the probability that the link between user k and UAV m is LoS link is which is given by the following equation:

[0041]

[0042] where η a and η b are constants related to the type of propagation environment. θ k,m = arcsin(H / d k,m (t)) is the elevation angle on the user side, denotes the Euclidean distance between user k and UAV m, H denotes the flight height of the UAV, w k (t) = [x k (t), y k (t)] denotes the horizontal coordinates of user k ∈ K at time slot t, q m (t) = [x m (t), y m (t)] denotes the horizontal coordinates of UAV m at time slot t ∈ T.

[0043] Correspondingly, at time slot t, the probability that the link between user k and UAV m is NLoS link is Therefore, the average path loss can be modeled as:

[0044]

[0045] where denotes the propagation combination related to LoS and NLoS, denotes the free space path loss between user k and UAV m, where f c is the system frequency, and v c denotes the speed of light. Therefore, the path loss L k,m (t) between user k and UAV m can be represented as:

[0046]

[0047] where η LoS = 1.6 dB and η NLoS = 23 dB are the path loss constant coefficients for LoS and NLoS, respectively.

[0048] Consider using OFDMA scheme for data transmission, so it is assumed that the bandwidth resource is evenly allocated to all users. Therefore, the rate r k,m (t) at which user k offloads computing tasks to drone m is:

[0049]

[0050] where σ 2 is the noise power, P K is the transmit power of the user, and B K is the communication bandwidth evenly allocated to the user.

[0051] The distance d m,m′ (t) between drone m and drone m' ∈ M can be expressed as d m,m′ (t) = ||q m (t) - q m′ (t)||2, q m′ (t) represents the horizontal coordinate of drone m' at time slot t, and needs to satisfy the minimum safety distance constraint d m,m′ (t) ≥ d min . There is no obstacle blocking between drones, so it can be considered that the communication between drones is a LoS link. At time slot t, the wireless channel power gain g m,m′ (t) between drone m and drone m' is expressed as:

[0052]

[0053] where β0is the channel power gain when the reference distance is 1 meter.

[0054] Similarly, the task transmission rate r m,m′ (t) between drone m and drone m' ∈ M can be expressed as:

[0055]

[0056] where P M is the transmit power of the drone, and B M is the communication bandwidth between drones.

[0057] Step S1102: After completing the construction of the node movement model of the user and the drone, construct the task generation and calculation model.

[0058] Each user generates a computing task {J k (t),C k} at time slot t, where J k (t) is the number of bits of the task, and C k is the number of CPUs needed to compute each bit of the task. Since the computing task is divisible, the user can choose to offload part of the task to a UAV, and the offloading ratio is defined as γ k (t). The ratio of the task computed locally is 1 - γ k (t). The access variable between the user and the UAV is defined as z k,m (t) e {0, 1}, and z k,m (t) = 1 means that the user chooses to access the UAV m at time slot t, otherwise z k,m (t) = 0. Since the user can only access one UAV at a single time slot, there is a constraint:

[0059]

[0060] The transmission time T (t) spent by user k to offload the computing task to UAV m at time slot t is:

[0061]

[0062] Since the distribution of users' locations can be uneven, the number of users associated with each UAV can also vary greatly. To achieve load balancing of the system, the UAV can choose to further split the computing task and offload the part of the computing task it needs to compute to the surrounding UAVs with idle computing power. The task migration ratio variable between UAVs is defined as p k,m,m′ (t) e [0, 1], which represents the ratio of the computing task of user k that UAV m chooses to forward to UAV m'. There is an obvious constraint Therefore, the transmission time T (t) spent by UAV m to forward the computing task of user k to UAV m' at time slot t is:

[0063]

[0064] To unify the definition, when m = m', the transmission distance is 0, and the transmission rate is infinite. Therefore, the transmission time of the computing task is 0.

[0065] For the part of the computing task that is not offloaded, the user chooses to execute it locally, and the specific computation time T is represented as: ​

[0066]

[0067] where, represents the computing power size of user k, with the unit of cycle / second.

[0068] At time slot t, the time consumed by the computing task offloaded by user k to UAV m is

[0069]

[0070] where, f k,m (t) is the computing power size allocated by UAV m to user k at time slot t, with the unit of cycle / second, and satisfies is the maximum CPU frequency of UAV m.

[0071] The communication and computing of user k can be performed in parallel, so the processing time D k (t) required by the computing task generated at time slot t for user k is:

[0072]

[0073] In order to ensure that the UAV can provide the best service for users at each time slot, the time slot length of the UAV trajectory planning depends on the maximum value of the computing task processing completion time of all users, that is, For any time slot t, the UAV can be regarded as quasi-static within τ t , so the distance between the UAV and all users can be considered as invariable.

[0074] According to the dynamic analysis model of the rotor UAV, the propulsion power p m (t) of UAV m at time slot t is represented as:

[0075]

[0076] where, U1, U2, U3, U4, U5 are constants related to aerodynamics and UAV hardware, v tip is the rotor tip speed. Therefore, the propulsion energy consumption of UAV m at time slot t is represented as

[0077] At time slot t, the communication energy consumption of user k offloading computing task to UAV m is represented as The local computing energy consumption of user k is represented as:

[0078]

[0079] where, ​​The capacitance coefficient of the user k.

[0080] The energy consumed by the user k at time slot t is:

[0081]

[0082] The energy consumed by the UAV m to forward the computing task of the user k to the UAV m' at time slot t is The energy consumed by the UAV m to compute the computing task of the user k is :

[0083]

[0084] where φ m is the capacitance coefficient of the UAV m.

[0085] Step S1103: After the task generation and the construction of the computing model are completed, the construction of the optimization target is performed.

[0086] The energy E m (t) consumed by the UAV m at time slot t is:

[0087]

[0088] The total number of task bits W m (t) computed by the UAV m at time slot t is:

[0089]

[0090] In order to encourage the UAV to actively provide computing services for users, a variable b m (t) is introduced, which represents the pricing of the unit computing resource of the UAV m at time slot t. Therefore, the benefit U m (t) of the UAV m at time slot t is defined as:

[0091]

[0092] Therefore, by jointly optimizing the UAV trajectory, the task migration ratio between UAVs, the UAV access selection and the pricing of the unit computing resource, the benefit of the UAV m is maximized, and the optimization problem is expressed as follows:

[0093] (P1):

[0094]

[0095] v m (t)≤V max , 0≤α m (t)≤2π (26d)

[0096] d m,m′ (t)≥d min (26e)

[0097]

[0098] b m (t)≥0 (26g)

[0099] Among them Λ m =(v m (t), α m (t),ρ k,m,m′ (t),z k,m (t),b m (t)), ω W and ω E These are the weights for the total number of task bits and energy consumption, respectively. Formula (26a) limits the range of values ​​for the task migration ratio between drones. Formula (26b) indicates that each user can only access one drone per time slot. Formula (26c) constrains the starting coordinates of the drone. Formula (26d) limits the flight speed and flight angle of the drone. Formula (26e) guarantees the minimum safe distance between drones. Formula (26f) indicates that the energy consumed by the drone during the entire flight should not exceed the total energy consumption limit. Formula (26g) constrains the pricing of drones per unit of computing power.

[0100] For user k, the sum of user latency and computational task overhead is defined as user overhead. User overhead is minimized by jointly optimizing the computing resources allocated to the user by the drone and the user's computational task offloading ratio:

[0101] (P2):

[0102]

[0103] γ k (t)∈[0,1] (27b)

[0104] τ t ≤δ (27c)

[0105]

[0106] Where Γ k =(f k,m (t),γ k (t)). Formula (27a) imposes restrictions on the computing resources of the UAV. Formula (27b) limits the proportion of user computing tasks that can be offloaded. Formula (27c) gives the upper limit for the length of each time slot. Formula (27d) indicates that the energy consumed by the user in each time slot should not exceed its energy limit.

[0107] Step S120: After the construction of the system model is completed, the algorithm is designed.

[0108] Wherein the algorithm includes a leader decision algorithm and a follower decision algorithm, each unmanned aerial vehicle and user is an individual, the unmanned aerial vehicles compete to provide computing services for the users, the users compete for the computing resources of the unmanned aerial vehicles, and their interaction directly affects the final optimal solution of the problem. Therefore, the method of game theory is introduced, and each individual is regarded as a participant in the game. Each participant hopes to obtain a solution most beneficial to himself. However, at the same time, the participants will be mutually constrained, which causes each participant to compete with other participants. The interaction between the unmanned aerial vehicles and the users is modeled as a Stakelberg game process, in which the unmanned aerial vehicles are leaders and the users are followers. The unmanned aerial vehicles first give a price for the unit computing resource, and the users make appropriate offloading decisions according to the price.

[0109] Based on the above, step S120 includes the following substeps:

[0110] Step S1201: The leader decision algorithm based on MAPPO is designed.

[0111] For problem (P1), firstly, the flight trajectories of the unmanned aerial vehicles are associated on time slots and each unmanned aerial vehicle has its own optimization target. Secondly, there is a high coupling between the task migration ratio between the unmanned aerial vehicles and the unmanned aerial vehicle access selection variable. In addition, the random movement of the user position and the dynamic generation of the computing task give the environment dynamic characteristics, and the delay-sensitive computing task requires the optimization problem to be solved in a very short time. Therefore, the traditional optimization algorithm is difficult to independently play a role in such a scene. As one of the popular machine learning technologies, the multi-agent deep reinforcement learning algorithm can not only cope with dynamic and uncertain environments, but also solve multi-objective optimization problems of participants competing or cooperating. The decision problem of the unmanned aerial vehicles is modeled as a Markov decision process, and a multi-agent proximal policy optimization algorithm with strong robustness and capable of processing continuous action space and state space is selected for solution.

[0112] Firstly, problem (P1) is converted into a partially observable Markov decision process containing M(t) agents. The decision process is composed of a state set S, an observation set O = {O1,..., O m ,...,O M(t)}, an action set A = {A1,..., A m ,...,A M(t)}, and a reward function set R. The state set S describes the global information of the environment, including the horizontal coordinates of all unmanned aerial vehicles, the horizontal coordinates of the users, and the task amount. The observation set O mPart of the environment information observed by the UAV m, i.e., at any time slot t, the observation space o m (t) e O m is a part of the state space s(t) e S. The action space a m (t) e A m of the UAV m at time slot t is described by the action function a m (t) is the observation space o m (t) is a mapping with respect to the policy π m (t).

[0113] The specific state space, observation space, action space and reward function are defined as follows:

[0114] State space: At time slot t, the global information s(t) e S of the environment can be represented as:

[0115] s(t) = {x M(t) (t), y M(t) (t), x K (t), y K (t), J K (t)}

[0116] Observation space: At time slot t, the local information o m (t) e O m observed by the UAV m can be represented as:

[0117] o m (t) = {x m (t), y m (t), x K (t), y K (t), J K (t)}

[0118] Action space: At time slot t, the action behavior a m (t) e A m of the UAV m can be represented as:

[0119] a m (t) = {v m (t), a m (t), p 1,m,1 (t),..., p K,m,M(t) (t), z 1,m (t),..., zK,m (t),b m (t)} respectively, which include the flight speed, flight angle, the proportion of UAV m migrating tasks to other UAVs, UAV m access selection and the pricing of unit computing resources.

[0120] Reward function: At time slot t, the reward function r m (t)∈R of UAV m can be expressed as:

[0121] represents the total benefit obtained by UAV m at time slot t.

[0122] In MAPPO, each agent is composed of an actor network, a critic network and an experience buffer. In the training phase, a central controller collects and evaluates the global information of the environment to realize centralized training. In the execution phase, each agent realizes distributed execution according to the actor network deployed by itself. Specifically, the agent inputs the observation result o m (t) into the actor network, and then obtains the action a m (t) and reward r m (t) of the current training batch, and stores the experience in the experience buffer. Every certain training batch, training data is taken out from the experience buffer, and then centralized update of all actor networks and critic networks is completed in the central controller.

[0123] Step S1202: Design a follower decision algorithm based on two-stage iteration.

[0124] For problem (P2), since there is no correlation between time slots, the optimal solution of a single time slot is the long-term optimal solution. To facilitate expression, the variable time slot t is omitted here.

[0125] First, introduce an auxiliary variable χ k,m , which satisfies and Then introduce an auxiliary variable G k , which satisfies and Problem (P2) can be transformed into:

[0126] (P3):

[0127]

[0128] γ k ∈[0,1](28e)

[0129] τ t ≤δ(28f)

[0130]

[0131] Where u k =(G k ,χ k,m ,f k,m ,γ k ).

[0132] Minimizing the computational task processing time for each user can be modeled as a strategy game:

[0133] {K,{u k} k∈K ,{G′ k} k∈K} (29)

[0134] in, It is a set of users, u k It is the set of actions of user k, u -k ={u1,...,u k-1 ,u k+1 ,...,u K} represents the set of actions of all users except user k. Let $k$ represent the cost of user $k$. The goal of all strategy games is to find a stable Nash equilibrium, meaning that no user can reduce their cost by unilaterally changing their strategy during the interaction. Definition For the user's optimal strategy, the Nash equilibrium stable point satisfies:

[0135]

[0136] For user k, the optimal strategy is adopted for other users. In this case,

[0137] For any time slot t, given the UAV's strategy, a two-stage iterative optimization algorithm is proposed to solve subproblem (P3).

[0138] Fixed computing resources f k,m Solve for the unloading ratio γ of the computational task. k :

[0139] (P3-1):

[0140]

[0141] γ k ∈[0,1](31d)

[0142] τ t ≤δ(31e)

[0143]

[0144] Firstly, the Lagrange dual method is introduced, and the obtained augmented Lagrange function is as follows:

[0145]

[0146] wherein ζ, υ m , θ m o m

[0147] , ε is the Lagrange multiplier, and all are non-negative. By applying the KKT condition, we can obtain:

[0148]

[0149]

[0150] θ m γ k = 0 (39)

[0151] o m (γ k - 1) = 0 (40)

[0152]

[0153] Based on the above equations, the optimal computing task offloading ratio

[0154]

[0155] wherein

[0156] Fix the computing task offloading ratio γ k , and solve the computing resource f k,m :

[0157] (P3-2):

[0158]

[0159] τ t ≤ δ (43e)

[0160] Firstly, the Lagrange dual method is introduced, and the obtained augmented Lagrange function is as follows:

[0161]

[0162] wherein ζ, υ m , θm ο

[0163] 、、 are Lagrange multipliers and are non-negative. By applying the KKT conditions, we can obtain:

[0164]

[0165]

[0166]

[0167] Based on the above equations, the optimal computing resource

[0168]

[0169] Step S130: Perform agent neural network training, and output the computing resource allocated by the unmanned aerial vehicle to the user and the computing task offloading ratio of the user according to the designed algorithm.

[0170] Wherein, step S130 comprises the following sub-steps:

[0171] Step S1301: Perform network parameter initialization.

[0172] The network parameter initialization comprises initializing the actor networks π(e|θ π ) and π'(e|θ π ) and the critic network Q(e|θ Q ), and the weights thereof are θ π and θ Q , respectively.

[0173] Step S1302: Perform multi-agent and environment interaction.

[0174] First, initialize the agent state s(t) and the experience buffer R b . The agent uses the actor network π to output the current action a(t) based on the current state.

[0175] According to the two-stage iterative follower decision algorithm, the computing resource allocated by the unmanned aerial vehicle to the user f k,m (t) and the computing task offloading ratio γ k (t) of the user are obtained.

[0176] The obtained variables are input into the environment to obtain the average reward q(t), and enter the next state s(t+1).

[0177] Step S1303: Update network parameters.

[0178] In a cellular network with multi-drone assisted edge computing, the base station first collects user data, location information, and the amount of computing tasks to be processed based on the actual scenario. Then, it uses a neural network to calculate the drone's location, the ratio of task migration between drones, drone access selection, and the pricing of unit computing resources. A follower decision algorithm is then used to calculate the computing resources allocated by drones to users and the user's computing task offloading ratio, thereby obtaining the current reward and optimizing network parameters until the network converges. Finally, the trained network is deployed on the drones. The drones synchronize information on all user locations and user computing task amounts, and the deployed neural network calculates the optimal decision. Simultaneously, users synchronize decision information and calculate the optimal decision using the follower decision algorithm, thus achieving online execution.

[0179] Example 2

[0180] like Figure 3 As shown, this embodiment proposes a multi-agent optimization-based multi-UAV trajectory planning and resource allocation system, which specifically includes a model building unit 310, an algorithm design unit 320, and a training output unit 330.

[0181] The model building unit 310 is used to build the system model.

[0182] Consider a system comprising multiple drones and multiple users. Drones equipped with MEC servers can provide computing services to users, such as... Figure 2 The figure shows a model of a multi-UAV collaborative assisted MEC system.

[0183] Through path discretization technology, the flight trajectory of the UAV can be discretized into T+1 waypoints, i.e., T line segments. The time when the UAV is in any line segment is represented as time slot t∈T, where the set T is defined as T={1,2,...,T}. In time slot t, the set of UAVs is represented as M={1,2,...,M(t)}, and the set of users is represented as K={1,2,...,K}. UAVs can dynamically join or leave the system in any time slot t based on their remaining battery power and system load. Therefore, the number of UAVs M(t) changes for each time slot. The flight time of the UAV in time slot t is defined as τ. t Assuming the UAV flies at a constant speed in time slot t, since the duration of each time slot is short, it can be considered to be in a quasi-static state within a single time slot. To ensure that the UAV is in a quasi-static state within each time slot, a constraint is imposed on the length of time slot t, τ. t ≤δ.

[0184] Based on the above, model building unit 310 performs the following sub-steps:

[0185] Step T1: Constructing the node movement model of users and UAVs. The flight height of all UAVs is fixed as H. By establishing a Cartesian coordinate system, the horizontal coordinates of UAV m at time slot t ∈ T can be expressed as q m (t) = [x m (t), y m (t)] and the horizontal starting point of UAV m is defined as Given the flight speed v m (t) ∈ [0, V max ] and direction a m (t) of UAV m at time slot t, where V max is the maximum flight speed of the UAV, the coordinate (x m (t), y m (t)) of the UAV is updated as follows:

[0186] x m (t) = x m (t-1) + v m (t)cos(a m (t))t t , (1)

[0187] y m (t) = y m (t-1) + v m (t)sin(a m (t))t t . (2)

[0188] The horizontal coordinates of user k ∈ K at time slot t are defined as w k (t) = [x k (t), y k (t)]. The movement of the user is modeled as a Gaussian-Markov random motion model. Therefore, the movement speed v k (t) and angle a k (t) of user k ∈ K at time slot t are updated as follows:

[0189]

[0190] where c1 and c2 are constants representing the correlation between the previous state and the current state. and are the average speed and average angle of all users, respectively. Φ k and Ψ k are two independent Gaussian random variables reflecting the randomness of user movement.

[0191] Therefore, the horizontal coordinates (x k (t), y k (t)) of user k ∈ K at time slot t are updated as follows:(t) = x

[0192] x k (t) = x k (t) + v k (t) cos(a k (t)) t t (5)

[0193] y k (t) = y k (t) + v k (t) sin(a k (t)) t t (6)

[0194] In the real world, due to the existence of obstacles such as buildings and mountains, the actual path loss between the UAV and the user is determined by the probability of Line-of-Sight (LoS) link and Non-Line-of-Sight (NLoS) link. At time slot t, the probability that the link between user k and UAV m is LoS link is Given by the following formula:

[0195]

[0196] where η a and η b are constants related to the type of propagation environment. θ k,m = arcsin(H / d k,m (t)) is the elevation angle on the user side, denotes the Euclidean distance between user k and UAV m, H denotes the flight height of the UAV, w k (t) = [x k (t), y k (t)] denotes the horizontal coordinates of user k ∈ K at time slot t, q m (t) = [x m (t), y m (t)] denotes the horizontal coordinates of UAV m at time slot t ∈ T.

[0197] Correspondingly, at time slot t, the probability that the link between user k and UAV m is NLoS link is Therefore, the average path loss can be modeled as:

[0198] where, is the propagation combination related to LoS and NLoS, denotes the free space path loss between user k and UAV m, where f cis the system frequency, v c denotes the speed of light. Thus, the path loss L k,m (t) between user k and drone m can be expressed as:

[0199]

[0200] where η LoS = 1.6 dB and η NLoS = 23 dB are the path loss constant coefficients for LoS and NLoS, respectively.

[0201] It is considered that an orthogonal frequency division multiple access scheme is used for data transmission, thus it is assumed that the bandwidth resource is equally allocated to all users. Thus, the rate r k,m (t) at which user k offloads computing tasks to drone m is:

[0202]

[0203] where σ 2 is the noise power, P K is the transmit power of the user, and B K is the communication bandwidth equally allocated to the user.

[0204] The distance d m,m′ (t) between drone m and drone m' e M can be expressed as d m,m′ (t) = ||q m (t) - q m′ (t) ||2, and needs to satisfy the minimum safety distance constraint d m,m′ (t) > d min . There is no obstacle blocking between drones, thus it can be considered that the communication between drones is a LoS link. The wireless channel power gain g m,m′ (t) between drone m and drone m' at time slot t is expressed as:

[0205]

[0206] where β0is the channel power gain when the reference distance is 1 meter.

[0207] Similarly, the task transmission rate r m,m′ (t) between drone m and drone m' e M can be expressed as:

[0208]

[0209] where P M is the transmit power of the drone, and B M is the communication bandwidth between drones.

[0210] Step T2: After the construction of the node movement model of users and UAVs, the task generation and computation model is constructed. Each user will randomly generate a computation task {J k (t), C k} at time slot t, where J k (t) is the number of bits of the task, and C k is the number of CPUs required to compute each bit of the task. Since these computation tasks are divisible, the user can choose to offload part of the task to a UAV, and the offloading ratio is defined as γ k (t), i.e., the proportion of tasks computed locally is 1-γ k (t). The access variable between the user and the UAV is defined as z k,m (t) e {0, 1}, z k,m (t) = 1 means that the user chooses to access the UAV m at time slot t, and otherwise z k,m (t) = 0. Since the user can only access one UAV in a single time slot, there is a constraint: (13)

[0212] The transmission time T (t) spent by the user k to offload the computation task to the UAV m at time slot t is:

[0213]

[0214] Since the distribution of users' locations can be uneven, the number of users associated with each UAV can also vary greatly. In order to achieve load balancing of the system, the UAV can choose to continue to split the computation task and offload the part of the computation task that needs to be computed to the surrounding UAVs with idle computing power. The task migration ratio variable p k,m,m′ (t) e [0, 1] is defined, which represents the proportion of the computation task of user k that the UAV m chooses to forward to the UAV m'. Obviously, there is a constraint Therefore, the transmission time T (t) spent by the UAV m to forward the computation task of user k to the UAV m' at time slot t is:

[0215]

[0216] In order to unify the definition, when m = m', the transmission distance is 0, and the transmission rate is infinite. Therefore, the transmission time of the computation task is 0.

[0217] For the part of the computation task that is not offloaded, the user chooses to execute it locally, and the specific computation time T is represented as: ​

[0218]

[0219] where, denotes the computing power size of user k, with the unit of cycle / second.

[0220] At time slot t, the time consumed by user k's offloaded computing task is calculated by UAV m as

[0221]

[0222] where, f k,m (t) is the computing power size allocated by UAV m for user k at time slot t, with the unit of cycle / second, and satisfies is the maximum CPU frequency of UAV m.

[0223] The communication and computing of user k can be executed in parallel, so the processing time D k (t) required by the computing task generated at time slot t for user k is:

[0224] In order to ensure that the UAV can provide the best service for users at each time slot, the time slot length for UAV trajectory planning depends on the maximum value of the computing task processing completion time of all users, i.e. For any time slot t, the UAV can be regarded as quasi-static within τ t , so the distance between the UAV and all users can be considered as constant.

[0225] According to the dynamic analysis model of the rotor UAV, the propulsion power p m (t) of UAV m at time slot t is expressed as:

[0226]

[0227] where, U1, U2, U3, U4, U5 are constants related to aerodynamics and UAV hardware, v tip is the rotor tip speed. Therefore, the propulsion energy consumption of UAV m at time slot t is expressed as

[0228] At time slot t, the communication energy consumption of user k offloading computing task to UAV m is expressed as The local computing energy consumption of user k is expressed as:

[0229]

[0230] where, is the capacitance coefficient of user k. ​​​

[0231] For user k, the energy spent by UAV m in time slot t is:

[0232] The energy consumption of UAV m forwarding user k's computing task to UAV m' in time slot t is The energy consumption of UAV m computing the computing task offloaded by user k is:

[0233]

[0234] where φ m is the capacitance coefficient of UAV m.

[0235] Step T3: After the completion of task generation and the construction of the computing model, the construction of the optimization objective is performed.

[0236] For UAV m, the energy E m (t) spent by UAV m in time slot t is:

[0237]

[0238] For UAV m, the total number of task bits W m (t) computed by UAV m in time slot t is:

[0239]

[0240] In order to motivate the UAV to actively provide computing services for users, a variable b m (t) is introduced, which represents the pricing of unit computing resource by UAV m in time slot t. Therefore, the benefit U m (t) of UAV m in time slot t is defined as:

[0241]

[0242] Therefore, by jointly optimizing the UAV trajectory, the inter-UAV task migration ratio, the UAV access selection, and the pricing of unit computing resource, the benefit of UAV m is maximized, and the optimization problem is expressed as follows:

[0243] (P1):

[0244]

[0245] v m (t)≤V max , 0≤α m (t)≤2π (26d)

[0246] d m,m′ (t)≥d min (26e)

[0247]

[0248] b m (t)≥0 (26g)

[0249] where m = (v m (t), a m (t), p k,m,m′ (t), z k,m (t), b m (t)), w W and w E are the weights of the total number of task bits and energy consumption, respectively. Equation (26a) limits the range of the UAV-to-UAV task migration ratio. Equation (26b) indicates that each user can only access one UAV in each time slot. Equation (26c) constrains the starting coordinates of the UAV. Equation (26d) limits the flight speed and flight angle of the UAV. Equation (26e) ensures the minimum safety distance between UAVs. Equation (26f) indicates that the energy consumed by the UAV during the entire flight period should not exceed the total energy consumption limit. Equation (26g) constrains the pricing of the UAV for unit computing resource.

[0250] For user k, define the sum of the user delay and the computing task overhead as the user overhead, and minimize the user overhead by jointly optimizing the computing resource allocated to the user by the UAV and the user's computing task offloading ratio:

[0251] (P2):

[0252]

[0253] where k (t) e [0, 1] (27b)

[0254] where t ≤ d (27c)

[0255]

[0256] where k = (f k,m (t), g k (t)). Equation (27a) limits the computing resource of the UAV. Equation (27b) limits the offloading ratio of the user's computing task. Equation (27c) gives an upper limit to the length of each time slot. Equation (27d) indicates that the energy consumed by the user in each time slot should not exceed its energy upper limit.

[0257] The algorithm design unit 320 is configured to design an algorithm.

[0258] Wherein the algorithm includes leader decision algorithm and follower decision algorithm, each unmanned aerial vehicle and user is an individual, unmanned aerial vehicles compete to provide computing services for users, users compete for computing resources of unmanned aerial vehicles, and their interaction directly affects the final optimal solution of the problem. Therefore, the method of game theory is introduced, and the individual is regarded as a participant in the game. Each participant hopes to obtain the solution most beneficial to himself. However, at the same time, the participants will restrict each other, which causes each participant to compete with other participants. The interaction between unmanned aerial vehicles and users is modeled as a Stakelberg game process, wherein the unmanned aerial vehicle is the leader and the user is the follower. The unmanned aerial vehicle first gives a price for the unit computing resource, and the user makes a suitable offloading decision according to the price.

[0259] Based on the above, the algorithm design unit 320 performs the following sub-steps:

[0260] Step W1: Design the leader decision algorithm based on MAPPO. For problem (P1), first, the flight trajectory of the unmanned aerial vehicle is associated on the time slot and each unmanned aerial vehicle has its own optimization target. Second, there is a high coupling between the task migration ratio between unmanned aerial vehicles and the unmanned aerial vehicle access selection variable. In addition, the random movement of user positions and the dynamic generation of computing tasks give the environment dynamic characteristics, and the delay-sensitive computing task requires the optimization problem to be solved in a very short time. Therefore, the traditional optimization algorithm is difficult to play an independent role in such a scene. As one of the popular machine learning technologies, the multi-agent deep reinforcement learning algorithm can not only cope with dynamic and uncertain environments, but also solve multi-objective optimization problems of participants competing or cooperating. The decision problem of the unmanned aerial vehicle is modeled as a Markov decision process, and the multi-agent proximal policy optimization algorithm with strong robustness and capable of processing continuous action space and state space is selected for solution.

[0261] First, problem (P1) is converted into a partially observable Markov decision process containing M(t) agents. The decision process is composed of a state set S, an observation set O = {O1,..., O m ,...,O M(t)}, an action set A = {A1,..., A m ,...,A M(t)}, and a reward function set R. The state set S describes the global information of the environment, including the horizontal coordinates of all unmanned aerial vehicles, the horizontal coordinates of users, and the task amount. The observation set O m describes the partial environment information observed by the unmanned aerial vehicle m, that is, at any time slot t, the observation space o m (t) ∈ O m is part of the state space s(t) ∈ S. The action space a m(t) e A m The action behavior of UAV, i.e., action space a m (t) is the observation space o m (t) about the policy π m (t) mapping.

[0262] The specific state space, observation space, action space and reward function are defined as follows:

[0263] State space: At time slot t, the global information s(t) e S of the environment can be represented as:

[0264] s(t) = {x1(t),...,x M(t) (t), y1(t),...,y K (t), x1(t),...,x K (t), y1(t),...,y K (t), J1(t),...,J m (t)}

[0265] Observation space: At time slot t, the local information o m (t) e O m of UAV m observed by the environment can be represented as:

[0266] o m (t) = {x m (t), y K (t), x1(t),...,x K (t), y1(t),...,y K (t), J1(t),...,J m (t)} respectively containing the horizontal coordinates of UAV m and all users and the number of computing task bits.

[0267] Action space: At time slot t, the action behavior a m (t) e A m of UAV m can be represented as:

[0268] a m (t) = {v m (t), a 1,m,1 (t), p K,m,M(t) (t),..., p 1,m (t), z K,m (t),..., z m (t), b m (t)} respectively containing the flight speed, flight angle, the proportion of UAV m migrating tasks to other UAVs, access selection and unit computing resource pricing of UAV m.

[0269] Reward function: At time slot t, the reward function r m (t)∈R can be expressed as:

[0270] denotes the total benefit obtained by UAV m at time slot t.

[0271] In MAPPO, each agent is composed of an actor network, a critic network and an experience buffer. In the training phase, a central controller collects and evaluates the global information of the environment to realize centralized training. In the execution phase, each agent realizes distributed execution according to the actor network deployed by itself. Specifically, the agent inputs the observation result o m (t) into the actor network, and then obtains the action a m (t) and the reward r m (t) and stores the experience in the experience buffer. Every certain training batch, training data is taken out from the experience buffer, and then centralized update of all actor networks and critic networks is completed in the central controller.

[0272] Step W2: Design the follower decision algorithm based on two-stage iteration. For problem (P2), since there is no correlation between time slots, the optimal solution of a single time slot is the long-term optimal solution. To facilitate expression, the variable time slot t is omitted here. First, introduce an auxiliary variable χ k,m , which satisfies and Then introduce an auxiliary variable G k , which satisfies and Problem (P2) can be transformed into:

[0273] (P3):

[0274]

[0275] γ k ∈[0,1] (28e)

[0276] τ t ≤δ (28f)

[0277]

[0278] where u k =(G k ,χ k,m ,f k,m ,γ k ).

[0279] Minimizing the computation task processing time for each user can be modeled as a strategic game:

[0280] {K,{u k} k∈K ,{G′ k} k∈K} (29)

[0281] where, is the set of users, u k is the set of actions for user k, u -k = {u1,...,u k-1 ,u k+1 ,...,u K} represents the set of actions for other users except user k, represents the overhead for user k. The goal of all strategic games is to find a Nash equilibrium stable point, which means that any user cannot reduce the overhead by unilaterally changing the strategy in the interaction. Define as the optimal strategy for users, then the Nash equilibrium stable point satisfies:

[0282]

[0283] For user k, in the case that other users take the optimal strategy ,

[0284] For any time slot t, after the strategy of the given UAV is given, in order to solve the subproblem (P3), a two-stage iterative optimization algorithm is proposed.

[0285] Fix the computing power resource f k,m , solve the computation task offloading ratio γ k :

[0286] (P3-1):

[0287]

[0288] γ k ∈ [0, 1] (31d)

[0289] τ t ≤ δ (31e)

[0290]

[0291] First, the Lagrange dual method is introduced, and the obtained augmented Lagrange function is as follows:

[0292]

[0293] where ζ, υm 、 θ m ο m

[0294] 、、 ε is a Lagrange multiplier, and are all non-negative. By applying the KKT conditions, we can get:

[0295]

[0296] θ m γ k = 0 (39)

[0297] ο m (γ k - 1) = 0 (40)

[0298]

[0299] Based on the above equations, we can get the optimal computing task offloading ratio

[0300]

[0301] where

[0302] Fix the computing task offloading ratio γ k , and solve the computing resource f k,m :

[0303] (P3-2):

[0304]

[0305]

[0306] τ t ≤ δ (43e)

[0307] First, introduce the Lagrange dual method, and the obtained augmented Lagrange function is as follows:

[0308]

[0309] where ζ, υ m 、 θ m ο

[0310] 、、 are Lagrange multipliers, and are all non-negative. By applying the KKT conditions, we can get:

[0311]

[0312] Based on the above equation, the optimal computing resource can be obtained

[0313]

[0314] The training output unit 330 is configured to train the agent neural network and output the computing resource allocated to the user by the UAV and the computing task offloading ratio of the user according to the designed algorithm.

[0315] The training output unit 330 performs the following sub-steps:

[0316] Step R1: Network parameter initialization. The network parameter initialization includes initializing the actor networks π(e|θ π ) and π'(e|θ π ) and the critic network Q(e|θ Q ) with weights θ π and θ Q respectively.

[0317] Step R2: Multi-agent and environment interaction. First, initialize the agent state s(t) and the experience buffer R b . The agent uses the actor network π to output the current action a(t) based on the current state. The computing resource allocated to the user by the UAV f k,m (t) and the computing task offloading ratio γ k (t) of the user are obtained according to the follower decision algorithm of the two-stage iteration.

[0318] The obtained variables are input into the environment to obtain the average reward q(t) and enter the next state s(t+1).

[0319] Step R3: Update network parameters.

[0320] In the multi-UAV assisted edge computing cellular network, the base station first collects the number of users, location information and the amount of computing tasks to be processed according to the actual scene. Then, the location of the UAV, the task migration ratio between UAVs, the access selection of the UAV and the pricing of the unit computing resource are calculated through the neural network. The computing resource allocated to the user by the UAV and the computing task offloading ratio of the user are calculated through the follower decision algorithm, and then the current reward is obtained to optimize the network parameters until the network converges. Finally, the trained network is deployed on the UAV, and the information of all user locations and user computing task amounts are synchronized between the UAVs and calculated through the deployed neural network to obtain the optimal decision. At the same time, the decision information is synchronized between the users and calculated through the follower decision algorithm to obtain the optimal decision, thereby realizing the effect of online execution.

[0321] The present application has the following beneficial effects:

[0322] The present application designs a multi-UAV distributed cooperative computing MEC system, which maximizes the UAV benefit by jointly optimizing the UAV trajectory, the task migration ratio between UAVs, the UAV access selection and the pricing of unit computing resources, and minimizes the user cost by jointly optimizing the computing resources allocated by the UAV to the user and the computing task offloading ratio of the user.

[0323] Although the examples of the present application are described, they are only for the purpose of explanation and not limitation of the present application, changes, additions and / or deletions to the embodiments can be made without departing from the scope of the present application.

[0324] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited to this, any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for multi-UAV trajectory planning and resource allocation based on multi-agent optimization, characterized in that, Includes the following steps: Construct a system model; After completing the construction of the system model, the algorithm is designed. Train the intelligent agent neural network and output the computing resources allocated by the drone to the user and the user's computing task offloading ratio according to the designed algorithm; The construction of the system model includes the following sub-steps: Construct node movement models for users and drones; After completing the construction of the node movement model for users and drones, the task generation and computation model is constructed. After completing the task generation and computational model construction, the optimization objectives are then constructed. The construction of the optimization objective includes determining the benefit U of the UAV m in time slot t. m (t), represented as: Among them W m (t) represents the total number of task bits calculated by UAV m in time slot t, ω W E is the weight of the total number of bits in the task. m (t) represents the energy consumed by the drone m in time slot t, ω E As the weight of energy consumption, b m (t) represents the pricing of drone m for unit computing power resources in time slot t, K represents the user set, and f k,m (t) represents the computing power allocated to drone m and user k in time slot t; By jointly optimizing drone trajectories, the proportion of task migrations between drones, drone access selection, and the pricing of unit computing resources, the efficiency of drone m is maximized. The optimization problem is expressed as P1: (P1): s.t.ρ k,m,m′ (t)∈[0,1], z k,m (t)∈{0,1}, v m (t)≤V max ,0≤α m (t)≤2π d m,m′ (t)≥d min b m (t)≥0 Among them Λ m =(v m (t), α m (t),ρ k,m,m′ (t),z k,m (t),b m (t)), ρ k,m,m′ (t)∈[0,1] represents the ratio variable of mission migration between UAVs, where time slot t∈T, z k,m (t)∈{0,1} represents the access variable between the user and the drone, z k,m (t) = 1 means that the user selects to access drone m in time slot t; otherwise, z k,m (t)=0, q m (t) represents the horizontal coordinate of UAV m in time slot t∈T. v represents the horizontal starting point of the drone m. m (t) represents the flight speed of the UAV m in time slot t, V max α is the maximum flight speed of the UAV. m (t) represents the flight direction of the UAV m in time slot t, d m,m′ (t) represents the distance between drone m and drone m′∈M, d min This indicates the minimum safe distance between drones. This indicates the maximum energy consumption of the drone (m). For user k, the sum of user latency and computational task overhead is defined as user overhead. By jointly optimizing the computing resources allocated to the user by the drone and the user's computational task offloading ratio, the user overhead is minimized. The optimization problem is formulated as P2: (P2): s.t.f k,m (t)≥0, c k (t)∈[0,1] t t ≤δ Where Γ k =(f k,m (t),γ k (t)), D k (t) represents the processing time required for the computation task generated by user k in time slot t, f k,m (t) represents the computing power allocated to drone m and user k in time slot t. γ is the maximum CPU frequency of drone m. k (t) represents the proportion of tasks offloaded to the drone, τ t This represents the flight time of the UAV in time slot t. To ensure that the UAV is in a quasi-static state within each time slot, the length of time slot t is constrained, τ. t ≤δ, E k (t) represents the energy consumed by user k in time slot t. This represents the maximum energy consumption limit for user k; After completing the construction of the system model, the algorithm design includes the following sub-steps: For problem P1, a leader decision-making algorithm based on multi-agent proximal policy optimization is designed. For problem P2, a follower decision-making algorithm based on two-stage iteration is designed. The design of a follower decision-making algorithm based on a two-stage iterative process includes: Fixed computing resources f k,m Solve for the unloading ratio γ of the computational task. k To obtain the optimal computational task unloading ratio. Fixed computing task unloading ratio γ k Solve for computing resources f k,m To obtain optimal computing resources The optimal computational task offloading ratio Represented as: Where C k This represents the number of CPU cycles required to compute each bit of the task. r represents the local computing power of user k. m,m′ J represents the transmission rate at which user k unloads computing tasks from drone m in time slot t. k Indicates the number of bits in the task. Let P be the capacitance coefficient of user k. K For the user's transmission power, r k,m f represents the rate at which user k unloads computing tasks from drone m. k,m This represents the amount of computing power allocated to user k by drone m. Among them, the optimal computing resources Represented as: f k′,m The amount of computing power allocated to user k' for drone m.

2. The multi-agent optimization-based multi-UAV trajectory planning and resource allocation method as described in claim 1, characterized in that, Construct a node movement model for both the user and the drone, including updating the horizontal coordinate q of drone m in time slot t∈T. m (t)=[x m (t),y m (t)]; x m (t)=x m (t-1)+v m (t)cos(α m (t))τ t y m (t)=y m (t-1)+v m (t)sin(α m (t))τ t x m (t-1) and y m (t-1) represents the coordinates of the drone m in time slot t-1, v m (t) represents the flight speed of the UAV m in time slot t, α m (t) represents the direction of the UAV m in time slot t, τ t This represents the flight time of the drone in time slot t.

3. The multi-agent optimization-based multi-UAV trajectory planning and resource allocation method as described in claim 1, characterized in that, Construct node mobility models for users and drones, including updating the motion rate v of user k∈K in time slot t. k (t) and angle α k (t); v k (t-1) represents the motion rate of user k∈K in time slot t-1, α k (t-1) represents the angle of user k∈K in time slot t-1, c1 is a pre-set constant representing the correlation between the movement rate of user k∈K in time slot t-1 and the current time slot t, and c2 is a pre-set constant representing the correlation between the movement angle of user k∈K in time slot t-1 and the current time slot t. The average rate for all users, Φ is the average angle for all users. k and Ψ k These are two independent Gaussian random variables.

4. A multi-agent optimization-based multi-UAV trajectory planning and resource allocation system, characterized in that, Specifically, it includes: Model building unit, algorithm design unit, and training output unit; Model building unit, used to build system models; Algorithm design unit, used for algorithm design; The training output unit is used to train the intelligent agent neural network and outputs the computing resources allocated by the drone to the user and the user's computing task offloading ratio according to the designed algorithm. The model building unit constructs the system model through the following sub-steps: Construct node movement models for users and drones; After completing the construction of the node movement model for users and drones, the task generation and computation model is constructed. After completing the task generation and computational model construction, the optimization objectives are then constructed. The construction of the optimization objective includes determining the benefit U of the UAV m in time slot t. m (t), represented as: Among them W m (t) represents the total number of task bits calculated by UAV m in time slot t, ω W E is the weight of the total number of bits in the task. m (t) represents the energy consumed by the drone m in time slot t, ω E As the weight of energy consumption, b m (t) represents the pricing of drone m for unit computing power resources in time slot t, K represents the user set, and f k,m (t) represents the computing power allocated to drone m and user k in time slot t; By jointly optimizing drone trajectories, the proportion of task migrations between drones, drone access selection, and the pricing of unit computing resources, the efficiency of drone m is maximized. The optimization problem is expressed as P1: (P1): s.t.ρ k,m,m′ (t)∈[0,1], z k,m (t)∈{0,1}, v m (t)≤V max ,0≤α m (t)≤2π d m,m′ (t)≥d min b m (t)≥0 Among them Λ m =(v m (t), α m (t),ρ k,m,m′ (t),z k,m (t),b m (t)), ρ k,m,m′ (t)∈[0,1] represents the ratio variable of mission migration between UAVs, where time slot t∈T, z k,m (t)∈{0,1} represents the access variable between the user and the drone, z k,m (t) = 1 means that the user selects to access drone m in time slot t; otherwise, z k,m (t)=0, q m (t) represents the horizontal coordinate of UAV m in time slot t∈T. v represents the horizontal starting point of the drone m. m (t) represents the flight speed of the UAV m in time slot t, V max α is the maximum flight speed of the UAV. m (t) represents the flight direction of the UAV m in time slot t, d m,m′ (t) represents the distance between drone m and drone m′∈M, d min This indicates the minimum safe distance between drones. This indicates the maximum energy consumption of the drone (m). For user k, the sum of user latency and computational task overhead is defined as user overhead. By jointly optimizing the computing resources allocated to the user by the drone and the user's computational task offloading ratio, the user overhead is minimized. The optimization problem is formulated as P2: (P2): s.t.f k,m (t)≥0, c k (t)∈[0,1] t t ≤δ Where Γ k =(f k,m (t),γ k (t)), D k (t) represents the processing time required for the computation task generated by user k in time slot t, f k,m (t) represents the computing power allocated to drone m and user k in time slot t. γ is the maximum CPU frequency of drone m. k (t) represents the proportion of tasks offloaded to the drone, τ t This represents the flight time of the UAV in time slot t. To ensure that the UAV is in a quasi-static state within each time slot, the length of time slot t is constrained, τ. t ≤δ, E k (t) represents the energy consumed by user k in time slot t. This represents the maximum energy consumption limit for user k; After completing the construction of the system model, the algorithm design unit designs the algorithm, including the following sub-steps: For problem P1, a leader decision-making algorithm based on multi-agent proximal policy optimization is designed. For problem P2, a follower decision-making algorithm based on two-stage iteration is designed. The design of a follower decision-making algorithm based on a two-stage iterative process includes: Fixed computing resources f k,m Solve for the unloading ratio γ of the computational task. k To obtain the optimal computational task unloading ratio. Fixed computing task unloading ratio γ k Solve for computing resources f k,m To obtain optimal computing resources The optimal computational task offloading ratio Represented as: Where C k This represents the number of CPU cycles required to compute each bit of the task. r represents the local computing power of user k. m,m′ J represents the transmission rate at which user k unloads computing tasks from drone m in time slot t. k Indicates the number of bits in the task. Let P be the capacitance coefficient of user k. K For the user's transmission power, r k,m f represents the rate at which user k unloads computing tasks from drone m. k,m This represents the amount of computing power allocated to user k by drone m. Among them, the optimal computing resources Represented as: f k′,m The amount of computing power allocated to user k' for drone m.

5. The multi-agent optimization-based multi-UAV trajectory planning and resource allocation system as described in claim 4, characterized in that, The model building unit constructs a node movement model for the user and the drone, including updating the horizontal coordinate q of the drone m in time slot t∈T. m (t)=[x m (t),y m (t)]; x m (t)=x m (t-1)+v m (t)cos(α m (t))τ t y m (t)=y m (t-1)+v m (t)sin(α m (t))τ t x m (t-1) and y m (t-1) represents the coordinates of the drone m in time slot t-1, v m (t) represents the flight speed of the UAV m in time slot t, α m (t) represents the direction of the UAV m in time slot t, τ t This represents the flight time of the drone in time slot t.

6. The multi-agent optimization-based multi-UAV trajectory planning and resource allocation system as described in claim 4, characterized in that, The model building unit constructs node movement models for users and drones, including updating the motion rate v of user k∈K in time slot t. k (t) and angle α k (t); v k (t-1) represents the motion rate of user k∈K in time slot t-1, α k (t-1) represents the angle of user k∈K in time slot t-1, c1 is a pre-set constant representing the correlation between the motion rate of time slot t-1 and the current time slot t, and c2 is a pre-set constant representing the correlation between the angle of time slot t-1 and the current time slot t. The average rate for all users, Φ is the average angle for all users. k and Ψ k These are two independent Gaussian random variables.

Images