Unmanned aerial vehicle assisted mobile edge computing system

Abstract

The invention provides an unmanned aerial vehicle assisted mobile edge computing system, which comprises a coding module used for carrying out coding modeling operation on an environment; the training module is used for training a neural network carried on the unmanned aerial vehicle; the execution module is used for obtaining unmanned aerial vehicle actions according to unmanned aerial vehicle observation information by using the trained neural network, obtaining a task allocation ratio between the unmanned aerial vehicle and the base station by using a proposed task allocation ratio formula, and executing the unmanned aerial vehicle actions and the task allocation ratio in the environment; according to the system, a model is converted into a Markov decision process based on a designed reward function and a discount factor, and each unmanned aerial vehicle carries out Markov decision process solution through a neural network equipped by the unmanned aerial vehicle, so that an optimized flight path and user selection are obtained; and a task allocation ratio formula is creatively proposed and used, so that the purposes of reducing delay and improving the fairness of the number of times of service of the user are finally achieved.

Description

Technical Field

[0001] The present invention relates to a multi-UAV assisted mobile edge computing system based on TD3 (Twin Delayed Deep Deterministic policy gradient), belonging to the field of mobile edge computing. Background Art

[0002] With the development of information technology, the tasks that mobile users need to handle have gradually turned to computationally intensive and delay-sensitive. Mobile edge computing can provide powerful computing services to mobile users with lower transmission delays. How to effectively ensure the user's service experience is a very critical issue. User experience includes delay, energy consumption, robustness, fairness, etc. If the number of users surges, drones play an important role in mobile edge computing due to their flexible deployability.

[0003] Drone-assisted mobile edge computing is limited by channel environment, computing resources, etc. The use of traditional methods not only requires knowing the global information of the entire environment, but also faces complex continuous-discrete mixed optimization problems, which leads to poor performance of the algorithm in terms of latency and fairness. In a changing environment, each action needs to be re-solved, which makes it difficult to meet real-time requirements.

[0004] Deep reinforcement learning methods can solve the above problems very well. Deep reinforcement learning can first let the intelligent agent understand what behaviors are good and what behaviors are bad through trial and error, so that when deployed in the environment, it can ensure that the drone can make real-time decisions about the environment.

[0005] The current deep reinforcement algorithms usually consider single UAV-assisted mobile edge computing, that is, single-agent deep reinforcement learning algorithms, and the communication model uses a free space propagation model. However, in real life, most cases are multi-UAV-assisted mobile edge computing, and the channel propagation model is unlikely to be a free space propagation model. In this case, if all calculations are performed according to the free propagation model, it will lead to poor robustness and poor actual application effects. Summary of the invention

[0006] The present invention provides a delay and fairness optimization method for a multi-UAV assisted mobile edge computing system based on TD3, aiming to optimize the flight trajectory of UAVs, UAV user selection, and the task allocation ratio between ground base stations and UAVs to reduce user delays and improve the fairness of the number of times users are served, thereby ensuring user experience.

[0007] In order to solve the above optimization problem, the present invention adopts the following technical solutions:

[0008] First, model the proposed drone-assisted mobile edge computing system.

[0009] Second, we obtain the basic status information of users and drones, take drone path planning and drone user selection as actions, and give a formula to determine the task allocation ratio. Based on the reward function and discount factor we designed, we transform the model into a Markov decision process.

[0010] Third, each drone uses its equipped neural network to solve the Markov decision process, thereby obtaining the optimized flight trajectory and user selection, ultimately achieving the goal of reducing latency and improving the fairness of the number of times users are served.

[0011] What we are considering is that when the base station is heavily loaded, the drone assists the base station in processing mobile user tasks. The user will generate a delay-sensitive task at each moment. The drone determines its own flight direction and selects the user to serve at this moment based on the deep reinforcement learning algorithm. The drone can only serve one user at the same time. After the drone completes the flight, it gives a task allocation ratio to determine the task allocation between the base station and the drone. By optimizing these three behaviors, we ultimately achieve a reduction in user delays and an improvement in the fairness of the number of times users are served.

[0012] According to a first aspect of the present invention, a drone-assisted mobile edge computing system is provided, comprising:

[0013] The encoding module is used to perform encoding modeling operations on the environment.

[0014] The training module is used to train the neural network on the drone.

[0015] The execution module is used to use the trained neural network to obtain the drone action according to the drone observation information and use the proposed task allocation ratio formula to obtain the task allocation ratio between the drone and the base station, and execute the drone action and task allocation ratio in the environment.

[0016] Among them, each drone is equipped with six neural networks, namely: Actor network with parameter α, the first Critic network with parameter β1, the second Critic network with parameter β2, the first Target network corresponding to the Actor network, the second Target network corresponding to the first Critic network, and the third Target network corresponding to the second Critic network.

[0017] Furthermore, in the method for optimizing latency and fairness of a drone-assisted mobile edge computing system provided by the present invention, the training module includes a training process for performing the following:

[0018] Step A1: Initialize environmental parameters.

[0019] Step A2: Obtain UAV observation information and perform regularization.

[0020] Step A4: Critic network training.

[0021] Step A5: Actor network training.

[0022] Step A6: Target network update.

[0023] Step A7: Determine whether the UAV operation time has reached the maximum limit time. If not, return to step A2. If it has reached the maximum limit time, determine whether the set maximum number of training rounds has been reached. If not, return to step A1. If yes, end the training.

[0024] Furthermore, in the edge computing system for drone-assisted mobility provided by the present invention, step A1 includes: initializing the drone position C m (t), the number of times all users are served A n (t), base station location C k 、User location C n , time t, neural network training optimizer Adam.

[0025] Step A2 includes: observing information o m (t) is calculated as follows:

[0026]

[0027] Among them A n (t) represents whether user n is served at time t, expressed as:

[0028]

[0029] o m (t) Regularization compresses the data into the range of 0-1, and the calculation method is:

[0030]

[0031] Among them, m (t) max for o m (t) may reach the maximum value.

[0032] Furthermore, in the drone-assisted mobile edge computing system provided by the present invention, step A3 includes:

[0033] The observation information o after regularization m (t) Input the Actor neural network to get action a m (t) and assign actions and tasks to U m(t) acts on the environment to obtain a quaternion (S(t), a m (t), r m (t), S(t+1)) to complete the experience storage; where S(t) represents the observation of all drones, which can be expressed as S(t) = {o1(t), o2(t), ..., o m (t)}. Drone action Where zm(t) represents the user choice of drone m at time t, Represents the flight direction of the UAV at time t, and the task allocation ratio U m (t) is given by the task allocation ratio formula as follows:

[0034]

[0035] Among them, S is the number of CPU cycles, f k is the CPU frequency of the ground base station, f m is the CPU frequency of the drone, r m,k (t) is the transmission speed between UAV m and ground base station k at TSt.

[0036] In order to optimize the exploration of discrete and continuous mixed actions to explore more paths and avoid falling into local optimality, the user selection z of the discrete variable is m (t) is set to:

[0037]

[0038] Where ε represents a given exploration parameter, exp represents a normal function with a mean of 0 and a variance of 1, It is the Actor network carried by the drone.

[0039] The continuous variable flight direction φ m (t) is set to:

[0040]

[0041] where ∈ represents the noise exploration parameter.

[0042] In the quadruple m (t) indicates that the drone takes the output action a m (t) and task allocation ratio U m (t) The reward obtained from the environment. The reward is used to evaluate the output action a of the drone. m The quality of (t), the reward function is expressed as:

[0043]

[0044] Among them, T mrepresents the total delay of the task processed by drone m, P col , P ue , P range They represent the collision penalty between drones, the penalty for drones choosing the same user, and the penalty for drones exceeding the boundary.

[0045] Furthermore, in the method for optimizing delay and fairness of a drone-assisted mobile edge computing system provided by the present invention, step A4 comprises:

[0046] Randomly sample k experiences in the experience replay buffer and use the sampled quadruple (S(t), a m (t), r m (t), S(t+1)) updates the two critic networks of the agent, and uses the gradient descent method based on the loss function of the critic network to update the two critic networks; the loss function of the critic network is:

[0047]

[0048] Among them, S(t) represents all drones observing 0 m (t) collection, y m (t) represents td-target, which is expressed as:

[0049]

[0050] in, Indicates the second or third Target network output. It means that in the drone m, o m (t+1) is the action obtained by the first Target network of the input drone m, and γ represents the Markov discount factor.

[0051] Critic neural network is used to predict the current drone action a based on all drone observation information S(t). m (t) Scoring. Further, the method for optimizing latency and fairness of the UAV-assisted mobile edge computing system provided by the present invention, step A5 includes: Actor network training adopts a policy gradient ascent algorithm, which is expressed as:

[0052]

[0053] Where, M = {1, 2, ..., m} is the set of drones, is the Actor network gradient of the drone.

[0054] Furthermore, in the edge computing system for drone-assisted mobility provided by the present invention, step A6 includes: Target network update is represented as:

[0055] α′=τα+(1-τ)α′

[0056] β′ i =τβ i +(1-τ)β′ i , i=1,2

[0057] Among them, α′ represents the parameters of the first Target network, β′ i represents the parameters of the second or third Target network, and τ is the update rate.

[0058] Furthermore, the drone-assisted mobile edge computing system provided by the present invention is used to optimize delay and fairness, and the execution module includes a module for performing the following execution process:

[0059] Initialize the environmental parameters and obtain the observation information of the drone. m (t) and regularize it, and convert the regularized observation information o m (t) Input the Actor network to get action a m (t), the action a at this time m (t) No exploration, including user selection Flight direction According to the task allocation ratio U between the UAV and the base station m (t), execute a m (t) and U m (t).

[0060] According to a second aspect of the present invention, there is provided a computer device, characterized in that it comprises:

[0061] A memory, used for storing instructions; and a processor, used for calling the instructions stored in the memory to implement the system of the first aspect.

[0062] According to a third aspect of the present invention, there is provided a computer-readable storage medium, characterized in that instructions are stored therein, and when the instructions are executed by a processor, the system of the first aspect is implemented.

[0063] Compared with the prior art, the above technical solution conceived by the present invention has at least the following beneficial effects:

[0064] (1) The probabilistic channel model is taken into account, which can better adapt to the real environment.

[0065] (2) The multi-agent deep reinforcement learning algorithm considered is more robust than the single-agent algorithm and can be deployed in real time. It also provides a new exploration method for deep reinforcement learning.

[0066] (3) An innovative task allocation ratio formula is proposed, and an exploration and processing method for continuous and discrete mixed actions is given.

[0067] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0069] Figure 1 The figure is an example diagram of a drone movement model according to an exemplary embodiment.

[0070] Figure 2 It is an example diagram of an algorithm framework according to an exemplary embodiment.

[0071] Figure 3 is a diagram showing an example of convergence of training according to an exemplary embodiment.

[0072] Figure 4 The figure is an example diagram of a flight trajectory of a UAV according to an exemplary embodiment.

[0073] Figure 5 The diagram is a diagram showing an example of system operation according to an exemplary embodiment.

[0074] Figure 6 is a diagram showing an example of training module operation according to an exemplary embodiment.

[0075] Figure 7 The figure is an example diagram of the execution module operation according to an exemplary embodiment. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0077] The present invention assumes that there are n UEs randomly distributed in a square with a side length of l. All drones in this square area with a side length of l can provide computing support. All users can be represented as a set N = {1, 2, ..., n}. At the same time, m drones are also flying in the same area to provide computing support for users. These drones can be represented as a set M = {1, 2, ..., m}. The flight height and speed of the drone are h respectively.UAV and v UAV User n will generate data of size D in each time slot (TS) t n (t) is a delay-sensitive task. All time slots constitute a complete service time TSs = {1, 2, ..., t}. UAV m first determines its flight angle And fly at this angle, then determine the service user to select z m (t) and task allocation ratio v m (t) offload part of the task to the ground base station k for synchronous processing. After the processing is completed, the task result is returned to the user n. In addition, a drone can only serve one user in one TS.

[0078] The UAV mobility model of the present invention can be expressed as:

[0079] The 3D coordinates of drone m at TS t are C m (t) = [x m (t), y m (t), h uav ], it must satisfy the following constraints:

[0080]

[0081]

[0082] The flight direction determined by the drone at TS t is Therefore, the position information of the drone at TS t+1 can be expressed as:

[0083]

[0084]

[0085]

[0086] x m (t+1)=x m (t)+d x ,

[0087] y m (t+1)=y m (t)+d y ,

[0088] The delay calculation module of the present invention can be calculated by the following series of formulas and expressed as:

[0089] The location of user n is denoted as c n , so the distance between user n and drone m at TS t is:

[0090]

[0091] Taking into account the use case blocked by tall buildings, the expected channel power gain between the two for a connection to occur is given by:

[0092]

[0093] Where β0 represents the path loss at 1m in LoS environment, k represents the discount factor, It represents the probability of a LoS connection between the two to be connected, which can be expressed as:

[0094]

[0095] Where θ is the angle between the two to be connected, and a and b are environmental modeling parameters.

[0096] Then we can get task D according to this formula n The transmission speed to the base station is:

[0097]

[0098] Where B represents the channel bandwidth, P ue represents the transmission power, σ 2 represents Gaussian noise.

[0099] So we can get the transmission delay of the task from user n to drone m It is expressed as:

[0100]

[0101] After the task is sent to drone m, the drone gives a task allocation ratio according to the following formula:

[0102]

[0103] Where S is the number of CPU cycles, f k is the CPU frequency of the ground base station, f m is the CPU frequency of the drone, r m,k (t) is the transmission speed between UAV m and ground base station k at TSt.

[0104] So we can get the size of the task that the drone needs to handle: U m (t)D n , and the rest is transmitted to the ground base station for processing. Therefore, the total delay can be expressed as:

[0105]

[0106] in It indicates that part of the task is assigned to the transmission delay of the ground base station, which can be expressed as:

[0107]

[0108] Indicates that some tasks are delayed in processing at the ground base station. Indicates the processing delay of another part of the tasks in the drone, which can be expressed as:

[0109]

[0110]

[0111] The fairness calculation module of the present invention can be calculated and expressed as follows:

[0112] At TS t, A n (t) is used to record whether the user is served, so our fairness formula is defined as:

[0113]

[0114] The present invention aims to solve the fairness and delay optimization problems. The present invention transforms the system model into a Markov decision process and regards each drone as an intelligent agent, where the state space of each intelligent agent includes:

[0115]

[0116] The action space of each agent consists of:

[0117]

[0118] where z m (t) represents the user choice of agent m at TS t.

[0119] The reward function of each agent of the present invention is set as:

[0120]

[0121] Where P col is the penalty factor for collision between drones, P ue is the penalty factor for different drones connecting to the same user at the same time, P range It is the penalty factor for drones flying outside the boundary.

[0122] The way the agent gives the user's choice is as follows:

[0123]

[0124] The agent gives the flight direction as follows:

[0125]

[0126] Among them, ε,∈ are artificially set exploration parameters, is an Actor network, and exp represents a normal distribution with a mean of 0 and a variance of 1.

[0127] Each agent is equipped with an Actor network with parameter α and two Critic networks with parameters β1 and β2, as well as their Target network. First, we initialize all Actor networks and all Critic networks of all agents, then copy their parameters to their Target networks, and initialize the experience replay buffer B. m , then set a maximum episode max Used to determine the maximum number of training rounds. For each episode, we must first initialize the deep reinforcement learning environment, then obtain the state space information required by the Actor network, and finally output a set of actions. Then the environment gives rewards according to the reward function, and the environment changes to the next state. At this time, we need to store the state space, action space, reward and next state of all agents in the experience replay buffer B m , this quaternion can be expressed as (S(t), a m (t), r m (t), S(t+1)).

[0128] If the number of quadruplets stored in the experience replay buffer reaches a certain amount, we start training the neural network and use random sampling to sample k groups of data from the experience replay buffer to optimize the flight direction and user selection strategy of the drone. The optimization process includes:

[0129] Using the sampled quadruple (S(t), a m (t), r m (t), S(t+1) updates the two critic networks of the agent, using the loss function of the critic network to perform gradient descent and update the two critic networks;

[0130] Use the gradient ascent function to update the Actor network of each agent separately;

[0131] Update all Target networks;

[0132] Repeat the above steps until the network converges and save the strategy.

[0133] Among them, the loss function of the Critic network is expressed as:

[0134]

[0135] where y m (t) represents td-target, which can be expressed as:

[0136]

[0137] in represents the Target Critic network, It means that in the drone m, o m (t+1) is the action obtained by the first Target network of the input drone m, and γ represents the Markov discount factor.

[0138] The gradient ascent function can be expressed as:

[0139]

[0140] Updating the Target network can be expressed as:

[0141] α′=τα+(1-τ)α′,

[0142] β′ i =τβ i +(1-τ)β′ i , i=1,2.

[0143] Where α′ represents the parameters of the Target Actor network, β′ i represents the parameters of the Target Critic network, and τ is the update rate.

[0144] The detailed technical scheme of the present invention is as follows: Figure 2 As shown in FIG. 1 , we model the system so that the drone can interact with the environment. The drone first obtains the required state from the environment, and gives the user selection, flight direction, and task allocation ratio through the neural network and the task allocation ratio formula proposed by the present invention. Then the environment changes to the next state and gives rewards according to the reward function. We transform the four-tuple (S(t), a m (t), r m (t), S(t+1)) are stored in the experience replay buffer. When the number of quadruple in the experience replay buffer reaches a certain number, we randomly extract k quadruple from it for training.

[0145] Specifically, we first transform S m (t) and the actions of all drones are input into the Critic network to obtain and Then for each drone m, input S(t) into the Target Actor network to obtain Then we can perform gradient descent based on the td-target formula and the loss function of the Critic network, and then train the Actor network using the policy gradient ascent formula to ensure that the generated actions can maximize the output value of the Critic network.

[0146] We set the number of users n to 20, the number of drones m to 3, and the location of the ground base station C k is (0, 0, 0), the initial position of the drone is C m (0) is (50,50,50), the side length of the square area l is 100m, the modeling parameter a is 2.42, the modeling parameter b is 0.29, the propagation discount factor k is 0.06, and the UAV flight altitude h uav is 30m, the transmitting power of the UAV is P uav is 0.2W, the user's transmission power P ue is 0.1W, and the channel bandwidth of the drone is B uav The channel bandwidth of the ground base station is 1MHZ. bs is 10MHZ, the CPU frequency of the drone is f uav is 2GHZ, the CPU frequency of the ground base station is f bs is 5GHZ, the CPU cycle number S is 1000 cycles / bit, and the user task size is D n is [2.0, 2.1]MB, the path loss β0 at 1m distance in LoS environment is -50dB, and the transmission noise σ 2 is -100dBm, the Markov discount factor γ is 0.99, and the experience replay buffer B m is 200000, the mini-batch sampling size k is 256, the Target update rate τ is 0.01, the user exploration parameter ε is 0.8, the random noise ∈ is 0.02, and the maximum number of training episodes is max is 5000, the penalty factor P for collision between drones col = -60, the penalty factor P for different drones connecting to the same user at the same time ue =-20, the penalty factor P for the drone flying out of the boundary range is -60, and the maximum time gap t is 20.

[0147] Figure 3 This is the reward graph obtained based on the environment and algorithm parameters we set above. It can be clearly seen that our algorithm basically converges around 2000 episodes.

[0148] The system proposed by the present invention is as follows Figure 5 As shown, we first perform coding modeling operations on the environment, where the modeling should include but is not limited to basic environmental parameters such as delay calculation, fairness calculation, drone location, ground user location, the number of times all ground users are served, and the location of ground base stations. Next, we extract the parameters required for drone observation from the code (drone location, number of times ground users are served). At this time, we should determine whether the neural network on the drone has completed training. If training is completed, run our execution module; if the neural network on the drone has not been trained, then we should start the neural network training module.

[0149] Training modules such as Figure 6 As shown, the environmental parameters should be initialized first, that is, the drone position C m (t), the number of times all users are served A n (t), base station location C k 、User location C n , time t, neural network training optimizer Adam, etc. Then we need to obtain the input required by the neural network, that is, the observation o of the drone m (t), which can be expressed as:

[0150]

[0151] Among them A n (t) represents whether user n is served at time t, which can be expressed as:

[0152]

[0153] Then we need to o m (t) Regularization is used to facilitate better neural network training. Without regularization, the gradient may disappear, leading to training failure. The regularization method is expressed as follows:

[0154]

[0155] Ready to use m (t) divided by o m (t) can reach the maximum value possible in order to compress the data into the range of 0-1.

[0156] Next, these regularized data can be input into the Actor neural network to complete the experience storage. Each drone is equipped with six neural networks, which are: an Actor network with parameter α and two Critic networks with parameters β1 and β2 and their target networks. The Actor neural network can output actions:

[0157]

[0158] Where zm(t) represents the user choice of drone m at time t, represents the flight direction of the drone at time t. At this time, the drone outputs action a m (t), we have added a new exploration method, which can effectively optimize the agent's exploration of discrete and continuous mixed actions compared to traditional noise exploration, allowing the agent to explore more paths and avoid falling into local optimality. For discrete variables, users select z m (t) can be expressed as:

[0159]

[0160] Where ε represents a given exploration parameter and exp is a normal function with a mean of 0 and a variance of 1.

[0161] For continuous variables It can be expressed as:

[0162]

[0163] where ∈ represents the noise exploration parameter.

[0164] Based on the given actions, rewards can be obtained from the environment. This reward is used to judge the quality of the drone's output actions. We define the reward formula as:

[0165]

[0166] Where T m represents the task delay processed by drone m, P col , P ue , P tange They represent the collision penalty between drones, the penalty for drones choosing the same user, and the penalty for drones exceeding the boundary.

[0167] The environment is affected by the drone's actions. m (t) and task allocation ratio U m (t) will change its own state, that is, the information o obtained by the drone at the next time t+1 m (t+1) will change, so we transform this quaternion (S(t), a m (t), r m (t), S(t+1)) is regarded as an experience and stored in the experience playback buffer, where S(t) = {o1(t), o2(t), ..., o m(t)}, this is the experience storage stage, and then we will carry out the Critic neural network training, by randomly sampling k experiences in the experience playback buffer, and using the sampled four-tuple (S(t), a m (t), r m (t), S(t+1)) updates the two critic networks of the agent, and uses the loss function of the critic network to perform gradient descent to update the two critic networks; wherein the loss function of the critic network is expressed as:

[0168]

[0169] Where S(t) is the set of all drone observation information, y m (t) represents td-target, which can be expressed as:

[0170]

[0171] in represents the Target Critic network, It means that in the drone m, o m (t+1) is the action obtained by the first Target network of the input drone m, and γ represents the Markov discount factor.

[0172] The role of the critic neural network is to analyze the drone’s current observation information. m The action a taken at (t+1) m The purpose of training the Critic neural network is to make the scoring more accurate.

[0173] When the Critic network training is completed, the Actor network training begins. The Actor network training uses the policy gradient ascent algorithm, which can be expressed as:

[0174]

[0175] Updating the target network can be expressed as:

[0176] α′=τα+(1-τ)α′,

[0177] β′ i =τβ i +(1-τ)β′ i , i=1,2.

[0178] Where α′ represents the parameters of the Target Actor network, β′ i represents the parameters of the Target Critic network, and τ is the update rate.

[0179] Then we enter to determine whether the drone operation time has reached the maximum limit time. If not, we continue to obtain drone observations to continue to accumulate experience and train the neural network. If yes, we determine whether the set maximum number of training rounds has been reached. max If not, return to the step of initializing environment parameters and continue training. If yes, the training is completed.

[0180] When the neural network is trained, the execution module can be executed, such as Figure 7 As shown, the environmental parameters are initialized and the observation information of the drone is obtained. m (t) and regularize it. Then input the processed data into the Actor network to get action a m (t), the action at this time does not need to be explored, so a m The flight angle of the drone contained in (t) Drone User Selection Then the task allocation ratio U between the UAV and the base station is given according to the formula m (t), the drone can then perform a m (t) and U m (t), and thus the delay value and fairness value are obtained according to the delay and fairness calculation module proposed above. The environment changes due to the action of the drone, and the above process is repeated until the time t reaches the maximum limit time.

[0181] Other embodiments of the present invention will be readily apparent to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0182] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A drone-assisted mobile edge computing system, characterized in that: include: The encoding module is used to perform encoding modeling operations on the environment; The training module is used to train the neural network on the drone; An execution module is used to obtain the UAV action based on the UAV observation information using the trained neural network and execute the UAV action; Among them, the drone is equipped with six neural networks, namely: Actor network with parameter α, the first Critic network with parameter β1, the second Critic network with parameter β2, the first Target network corresponding to the Actor network, the second Target network corresponding to the first Critic network, and the third Target network corresponding to the second Critic network.

2. The drone-assisted mobile edge computing system according to claim 1, characterized in that: The training module includes the following training procedures: Step A1: Initialize environmental parameters; Step A2: Obtain UAV observation information and perform regularization processing; Step A4: Critic network training; Step A5: Actor network training; Step A6: Target network update; Step A7: Determine whether the UAV operation time has reached the maximum limit time. If not, return to step A2. If it has reached the maximum limit time, determine whether the set maximum number of training rounds has been reached. If not, return to step A1. If yes, end the training.

3. The drone-assisted mobile edge computing system according to claim 2, characterized in that: Step A1 includes: initializing the drone position C m (t), the number of times all users are served A n (t), base station location C k 、User location C n , time t, neural network training optimizer Adam; Step A2 includes: observing information o m (t) is calculated as follows: Among them A n (t) represents whether user n is served at time t, expressed as: o m (t) Regularization compresses the data into the range of 0-1, and the calculation method is: Among them, m (t) max for o m (t) may reach the maximum value.

4. The drone-assisted mobile edge computing system according to claim 3, characterized in that: Step A3 includes: The observation information o after regularization m (t) Input the Actor neural network to get action a m (t) and assign actions and tasks to U m (t) acts on the environment to obtain the quaternion (S(t), a m (t),r m (t), S(t+1)) to complete the experience storage; where S(t) represents the observation of all drones, which can be expressed as S(t) = {o1(t), o2(t), ..., o m (t)}. Drone action where z m (t) represents the user choice of drone m at time t, Represents the flight direction of the UAV at time t, and the task allocation ratio U m (t) is given by the task allocation ratio formula as follows: Among them, S is the number of CPU cycles, f k is the CPU frequency of the ground base station, f m is the CPU frequency of the drone, r m,k (t) is the transmission speed between UAV m and ground base station k at TS t. In order to optimize the exploration of discrete and continuous mixed actions to explore more paths and avoid falling into local optimality, the user selection z of the discrete variable is m (t) is set to: Where ε represents a given exploration parameter, exp represents a normal function with a mean of 0 and a variance of 1, It is the Actor network carried by the drone; The continuous variable flight direction φ m (t) is set to: Where ∈ represents the noise exploration parameter; In the quadruple m (t) indicates that the drone takes the output action a m (t) and task allocation ratio U m (t) The reward obtained from the environment. The reward is used to evaluate the output action a of the drone. m The quality of (t), the reward function is expressed as: Among them, T m represents the total delay of the task processed by drone m, P col , P ue , P range They represent the collision penalty between drones, the penalty for drones choosing the same user, and the penalty for drones exceeding the boundary.

5. The edge computing system for drone-assisted mobility according to claim 4, characterized in that: Step A4 includes: Randomly sample k experiences in the experience replay buffer and use the sampled quadruple (S(t), a m (t),r m (t), S(t+1)) updates the two critic networks of the agent, using the gradient descent method based on the loss function of the critic network to update the two critic networks; the loss function of the critic network is: Among them, S(t) represents the observation o of all drones m (t), y m (t) represents td-target, which is expressed as: in, Indicates the second or third Target network output. It is o m (t+1) Input the action obtained by the first Target network, γ represents the Markov discount factor; Critic neural network is used to predict the current drone action a based on all drone observation information S(t). m (t) Scoring.

6. The drone-assisted mobile edge computing system according to claim 5, characterized in that: Step A5 includes: Actor network training adopts a policy gradient ascent algorithm, where: Where M = {1, 2, ..., m} is the set of drones, Represents the gradient of the Actor network.

7. The drone-assisted mobile edge computing system according to claim 6, characterized in that: Step A6 includes: Target network update is expressed as: α′=τα+(1-τ)α′ b′ i =tb i +(1-τ)β′ i ,i=1.2 Among them, α' represents the parameters of the first Target network, β' i represents the parameters of the second or third Target network, and τ is the update rate.

8. The drone-assisted mobile edge computing system according to claim 1, characterized in that: The execution module includes the following execution processes: Initialize the environmental parameters and obtain the observation information of the drone. m (t) and regularize it, and convert the regularized observation information o m (t) Input the Actor network to get action a m (t), the action a at this time m (t) No exploration, including user selection Flight direction According to the task allocation ratio U between the UAV and the base station m (t), execute a m (t) and U m (t).

9. A computer device, characterized in that: include: A memory for storing instructions; A processor, configured to call the instructions stored in the memory to implement the edge computing system for drone-assisted mobility as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: Instructions are stored, and when the instructions are executed by the processor, the edge computing system for drone-assisted mobility as described in claims 1-8 is implemented.

Images