A collaborative optimization method for edge network resource scheduling and drone motion control

Abstract

The present invention provides a method for collaborative optimization of edge network resource scheduling and unmanned aerial vehicle motion control, which belongs to the field of mobile edge computing technology. By establishing a unmanned aerial vehicle assisted mobile edge computing model, a user mobility model and an unmanned aerial vehicle dynamics model, a collaborative optimization problem model is constructed; an MDP model is constructed based on the problem model, and deep reinforcement learning is used to continuously interact with the environment, and finally an intelligent collaborative optimization strategy for edge network resource scheduling and unmanned aerial vehicle motion control is obtained. Based on this optimization strategy, the intelligent body directly outputs continuous and discrete mixed actions, so that the unmanned aerial vehicle collaboratively optimizes the edge network resource scheduling and the dynamic control of the motion trajectory, and improves the self-learning ability, environmental adaptability and dynamic real-time performance of the unmanned aerial vehicle assisted edge computing system in a dynamic environment, so as to adapt to the complex, diverse and dynamic real-time business requests in the future unmanned aerial vehicle assisted edge computing scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of mobile edge computing technology, and in particular relates to a method for collaborative optimization of edge network resource scheduling and unmanned aerial vehicle motion control. Background Art

[0002] In recent years, with the rapid development of the information industry, people have ushered in the 6G era of the Internet of Everything. Various services and applications have emerged in an endless stream, such as augmented reality, online games, and video stream analysis. These services and applications have strict requirements on data processing latency, which cannot be met by traditional cloud computing. In order to meet the latency requirements, a large number of computing tasks need to be performed in the user's local device. However, the battery capacity and computing power of user devices are limited, and high-performance computing cannot be performed continuously, making it very challenging to perform computing tasks completely locally. Drone-assisted edge computing is a new computing paradigm that can take advantage of the strong mobility and rapid deployment of drones to flexibly provide users with task offloading and computing services at the edge side close to users in wireless networks. Compared with cloud computing, drone-assisted edge computing can simultaneously reduce data processing latency and save energy consumption of user devices through edge network resource scheduling optimization.

[0003] With the rapid development of drone-assisted mobile edge computing technology, the computing system architecture is more flexible, the transmission environment is more complex, the real-time requirements are higher, and the business needs are more diversified. These development trends result in the fact that the users of the service are usually dynamic and mobile, and the user's task offloading requirements are also dynamic and time-varying. This has greatly increased the difficulty of edge network resource scheduling and real-time control of drone flight trajectories. However, traditional optimization designs usually focus on static or quasi-static application scenarios, and only complete the overall planning of edge network resource scheduling and drone flight trajectories based on the current user and environmental status. In addition, the current research lacks consideration of the flight dynamics constraints of drones, that is, the speed and acceleration of drones cannot change arbitrarily. Therefore, in the dynamic and time-varying application scenarios, the existing edge network resource scheduling and drone motion trajectory control optimization methods based on drone-assisted edge computing are insufficient in dynamic real-time performance and poor in environmental adaptability, which makes it difficult to meet the development needs of drone-assisted edge computing in the future.

[0004] Therefore, there is an urgent need to study a UAV-assisted edge computing method that can dynamically and collaboratively optimize edge network resource scheduling and UAV motion control in real time. Summary of the invention

[0005] In view of the problems existing in the prior art, the purpose of the present invention is to provide an intelligent collaborative optimization method for edge network resource scheduling and drone motion control suitable for drone-assisted edge computing. On the one hand, optimizing edge network resource scheduling can improve edge network energy efficiency and improve performance such as communication delay. On the other hand, it can adjust the drone flight trajectory in real time to adapt to the dynamic time-varying nature of user task offloading requirements, while dealing with problems such as random delays and time-varying system parameters caused by user dynamic movement. Therefore, the overall performance of the drone-assisted mobile edge computing system can be effectively improved through the collaborative optimization of edge network resource scheduling and dynamic control of motion trajectories. In addition, when collaboratively optimizing network resource scheduling and motion control, the actions output by the agent include continuous actions and discrete actions. Continuous actions include acceleration and calculation frequency, and discrete actions include user association and task offloading ratio. Therefore, the agent in the intelligent method proposed in the present invention can simultaneously output continuous and discrete mixed actions.

[0006] In view of the problems existing in the prior art, the present invention provides a method for collaborative optimization of edge network resource scheduling and UAV motion control, comprising:

[0007] Step S1: Establish a UAV-assisted mobile edge computing model; first define environmental information, calculate the UAV-assisted mobile edge computing latency and energy consumption indicators; then consider mobility, establish a user mobility model, establish a UAV dynamics model, and calculate the UAV flight energy consumption indicators;

[0008] Step S2: Based on the UAV-assisted mobile edge computing model of step S1, a collaborative optimization problem model combining edge network resource scheduling and UAV motion control is constructed;

[0009] Step S3: Based on the collaborative optimization problem model of edge network resource scheduling and UAV motion control, a corresponding Markov decision process MDP model is established;

[0010] Step S4, based on the MDP model established in step S3, deep reinforcement learning is used to obtain a collaborative optimization strategy for network resource scheduling and UAV motion control suitable for UAV-assisted mobile edge computing.

[0011] The beneficial effects of the present invention are as follows:

[0012] In this method, the maneuverability of the UAV and the constraints of the flight dynamics equation are comprehensively considered, and a collaborative optimization method for resource allocation and trajectory real-time control is proposed. Due to the addition of the flight dynamics equation constraints, the dynamic trajectory of the UAV output based on the proposed method is more consistent with the flight trajectory in actual conditions, which is not considered by existing methods.

[0013] This method modifies the loss function update method of the neural network in the DRL algorithm to be suitable for outputting continuous discrete mixed actions. The improved algorithm can train convergence and stably output continuous discrete mixed actions, thus effectively solving the continuous discrete mixed action space problem in resource allocation and real-time trajectory control problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a flow chart of an intelligent collaborative optimization method suitable for drone-assisted edge computing provided according to an embodiment of the present invention.

[0015] Figure 2 This is an architecture diagram of an intelligent collaborative optimization method suitable for drone-assisted edge computing provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] The embodiment of the present invention provides an intelligent collaborative optimization method combining edge network resource scheduling and UAV motion control suitable for UAV-assisted edge computing, so as to collaboratively optimize edge network resource scheduling and motion trajectory dynamic control in dynamic scenarios, such as Figure 1 As shown, the method includes:

[0018] Step S1, establish a UAV-assisted mobile edge computing model. First, define environmental information, calculate the UAV-assisted mobile edge computing latency and energy consumption indicators; then, consider mobility, establish a user mobility model, establish a UAV dynamics model, and calculate the UAV flight energy consumption indicators.

[0019] It should be noted that the drone in the embodiment of the present invention can provide edge computing services for multiple users at the same time. The drone can obtain real-time status information in the environment, including drone and user coordinate information, user computing task information, and drone speed and acceleration information. Through wireless communication, users can offload part of the computing tasks to the drone, and the other part of the computing tasks are calculated locally by the user. After the drone completes the computing task, it will send the result back. According to the limitations of drone flight dynamics, that is, the speed and acceleration of the drone cannot change arbitrarily, the drone needs to output acceleration control actions in real time according to the current flight status of the drone and the user's mission load requirements.

[0020] For drone-assisted mobile edge computing, it is necessary to analyze the latency and energy consumption of user computing tasks in local computing, offloading and uploading, and result return.

[0021] Step S2, based on the UAV-assisted mobile edge computing model of step S1, construct a collaborative optimization problem model combining edge network resource scheduling and UAV motion control.

[0022] It should be noted that it is of great significance to study the collaborative optimization problem of combining edge network resource scheduling with UAV motion control. Optimizing edge network resource scheduling can improve network energy efficiency and improve performance such as communication latency. In addition, combined with the dynamic optimization of UAV motion control, the flight path of the UAV can be controlled in real time to adapt to the dynamic time-varying nature of user task offloading requirements, and can cope with problems such as random delays and system parameter time-varying caused by user dynamic movement. Therefore, collaborative optimization of edge network resource scheduling and UAV motion control can significantly improve the overall performance of the UAV-assisted edge computing system.

[0023] Step S3: Establish a corresponding MDP model based on the collaborative optimization problem model of edge network resource scheduling and UAV motion control.

[0024] It should be noted that reinforcement learning (RL) problems are usually described by Markov decision process (MDP). MDP generally includes state, action, state transfer function and reward function. The MDP model of the system is established according to the system model and optimization problem.

[0025] Step S4, based on the MDP model established in step S3, deep reinforcement learning is used to obtain a collaborative optimization strategy for network resource scheduling and UAV motion control suitable for UAV-assisted mobile edge computing.

[0026] It should be noted that in this embodiment, the action space is a hybrid action space. For example, the action states of the calculation frequency and flight acceleration in the optimization variables are continuous, while the action state of the proportion of tasks unloaded by the user to the drone is discrete. Traditional artificial intelligence algorithms for a single action space, such as Actor-Critic, Deep Deterministic Policy Gradient (DDPG), etc., can only output discrete actions or continuous actions. When solving the optimization problem of the hybrid action space, the commonly used method is to convert the hybrid action space into a unified homogeneous action space by discretization or continuity. However, this processing method ignores the underlying structure of the hybrid action space, causing problems such as weak scalability and high learning approximation complexity, and even leads to serious performance deterioration. More importantly, this method destroys the interdependence between drone motion control and network resource optimization, thereby reducing the effect of collaborative optimization. In response to this problem, the embodiment of the present invention combines optimization theories and methods such as proximal policy optimization (PPO) algorithm and hybrid action decision-making to propose a collaborative optimization strategy for intelligent network resource scheduling and drone motion control. This strategy can robustly output continuous and discrete mixed action spaces at the same time, and is suitable for the dynamic optimization of continuous discrete mixed actions in drone-assisted mobile edge computing applications. The embodiment of the present invention combines the PPO algorithm with mixed action decision-making, and collects samples and trains them by continuously interacting with the environment according to the defined MDP model; optimizes the neural network parameters with the goal of maximizing the reward function; integrates the decision-making methods of continuous and discrete actions, so that the algorithm can output continuous and discrete mixed actions at the same time according to the real-time environmental state, thereby realizing intelligent collaborative optimization of edge network resource scheduling and drone motion control.

[0027] The method provided by the embodiment of the present invention establishes a UAV-assisted mobile edge computing model by analyzing the time delay and energy consumption generated by user computing tasks in local computing, offloading and uploading, and result return, and at the same time establishes a user mobility model and a UAV dynamics model, thereby constructing a collaborative optimization problem model; an MDP model is constructed based on the problem model, and deep reinforcement learning is used to continuously interact with the environment, and finally an intelligent collaborative optimization strategy for edge network resource scheduling and UAV motion control is obtained. Based on this optimization strategy, the intelligent agent directly outputs continuous and discrete mixed actions, so that the UAV can collaboratively optimize the edge network resource scheduling and dynamic control of the motion trajectory, improving the self-learning ability, environmental adaptability, and dynamic real-time performance of the UAV-assisted edge computing system in a dynamic environment, so as to adapt to the complex, diverse, and dynamic real-time business requests in future UAV-assisted edge computing scenarios.

[0028] Specifically, the UAV mobile edge computing model established in step S1 includes:

[0029] Step S1.1: Basic environment definition: Assume that there are K ground users in a certain area, with user numbers u∈{1, 2, ..., U}, and drone m acts as an air edge computing server to provide mobile edge computing services for ground users in the area. In the nth time slot, the coordinates of user u are p u ={x u [n],y u [n], 0}, where x u [n] is the x-axis coordinate of user u on the ground plane, y u [n] is the y-axis coordinate of user u on the ground plane, 0 represents the user's height; the coordinate of drone m is p m ={x m [n],y m [n], H}, where x m [n] is the x-axis coordinate of the drone m on the ground plane, y m [n] is the y-axis coordinate of the drone m on the ground plane, and H represents the height of the drone;

[0030] Step S1.2: Task offloading: The user randomly generates the amount of computing tasks to be executed in each time slot. In the nth time slot, the proportion of computing tasks that user u retains for local processing is defined as μ m，u [n]∈{0,1}, that is, the proportion of computing tasks uploaded to drone m is 1-μ m，u [n]; The drone can simultaneously Provide edge computing services to users.

[0031] Step S1.3: UAV and user association: The UAV makes association decisions for each time slot, deciding which users to associate with, that is, which users unload and upload to the UAV, λ m，u [n]∈{0,1} marks whether user u is associated with drone m in the nth time slot.

[0032] Step S1.4: Communication rate calculation: In the nth time slot, the channel gain h from user u to drone m is m，u for:

[0033]

[0034] Where β0 represents the channel gain when the reference distance is 1 meter; p m [n] is the coordinate of drone m in the nth time slot, p u [n] is the coordinate of user u in the nth time slot; ||·|| represents the modulus of the calculation vector, and the calculation vector (p m [n]-p u[n]), which is the distance from user u to drone m in a physical sense.

[0035] Considering the OFDMA mechanism, the bandwidth allocated to each user is B, the transmission power of user u is P0, and the transmission power of drone m is P m ; Calculate the unloading communication rate from user u to drone m in the nth time slot as:

[0036]

[0037] Among them, σ 2 Represents AWGN (Additive White Gaussian Noise) noise power.

[0038] The communication rate of the return result from drone m to user u in the nth time slot is calculated as:

[0039]

[0040] Step S1.5: time-consuming calculation;

[0041] User task offloading time calculation: The user offloads all or part of the computing tasks to the drone. The time consumption of the offloading upload process is:

[0042]

[0043] Among them, D u [n] is the total data volume of the computing task of user u.

[0044] User local computing time: For some computing tasks that are retained for local processing by user u, the computing time of user u is:

[0045]

[0046] Among them, f u [n] is the calculation frequency of user u, and 0≤f u [n]≤F u , F u is the maximum computing frequency of the user, and C0 is the number of CPU cycles for computing the unit task data volume.

[0047] Drone edge computing time:

[0048]

[0049] Among them, f m，u [n] is the computing frequency assigned to user u by drone m, and the maximum computing frequency of drone m is F m .

[0050] UAV result feedback time: After the UAV completes the computing task offloaded by user u, it feeds back the computing result to user u. The time consumed for transmitting the computing result is:

[0051]

[0052] Among them, D m [n] is the amount of calculation result data that drone m needs to feed back to user u.

[0053] Step S1.6: energy consumption calculation;

[0054] Energy consumption of user task offloading: Among them, P0 is the communication transmission power of the ground user.

[0055] User local computing energy consumption: η is a positive coefficient related to the user's processor chip.

[0056] Drone edge computing energy consumption: ψ is a positive coefficient related to the drone processor chip.

[0057] UAV result feedback energy consumption: P m Transmit power for UAV communication.

[0058] In summary, the nth time slot can be calculated:

[0059] (1) Total energy consumption of communication and computing:

[0060]

[0061] (2) Total communication and computing delay:

[0062]

[0063] In step S1, a user mobility model is established, which specifically includes:

[0064] The moving speed of the user in the nth time slot is expressed as in, Indicates the speed of the user in the x-axis direction in the nth time slot, represents the speed of the user in the y-axis direction in the nth time slot. Based on the Gauss-Markov mobility model, the moving speed v of user u in the n+1th time slot is u [n+1] is related to the moving speed in the nth time slot, which is specifically expressed as:

[0065]

[0066] Among them, σ u[n]~N(0,1) is the standard normal distribution, δ represents the influence of the previous time slot state on the current time slot state, They represent the asymptotic mean and asymptotic standard deviation of the speed respectively. Therefore, the coordinates of the mobile user can be updated according to the speed as follows:

[0067] p u [n+1]=p u [n]+v u [n]ΔT,

[0068] Where ΔT represents the duration of a time slot, p u [n] is the coordinate of user u in the nth time slot.

[0069] In step S1, a UAV dynamics model is established, which specifically includes:

[0070] Usually, the motion of UAV m in the continuous time domain is expressed as:

[0071]

[0072]

[0073] Among them, t represents the time in the continuous time domain, p m (t), v m (t) and a m (t) represent the position coordinates, velocity and acceleration of UAV n at time t, respectively, and τ represents the time delay; in the continuous time domain, It means that the derivative of the coordinate vector of the UAV m is the velocity vector; Derivative of the velocity vector of the drone m yields the acceleration vector. Define the motion state vector of the drone as χ m (t)=(p m (t), v m (t), a m (t)). Therefore, the dynamic model of UAV m is expressed as:

[0074]

[0075] Indicates x m (t) derivative. Its meaning is the state of motion (p m (t), v m (t), a m (t)), which is affected by the past state.

[0076]

[0077] To facilitate computer data processing and analysis, we transform the model into a discrete time domain model. Let n represent the nth discrete time slot in the discrete time domain, which can be approximated as the moment t in the continuous time domain. The discrete time dynamics model corresponding to the nth sampling interval is:

[0078] χ m [n+1]=X0x m [n]+Y1a m [n]+T2a m [n-1]

[0079] Among them, χ m [n] = x m (nΔT), X0=e XΔT , The flight energy consumption E of drone m in the nth time slot f，m The calculation formula for [n] is:

[0080]

[0081] Among them, κ1 and κ2 are system-determined parameters, M is the mass of the drone, and g is the gravitational acceleration.

[0082] In step S2, the overall energy consumption and delay indicators in the system model established in step S1 are considered as optimization targets, and the network resources and dynamic equations are considered as constraints. The optimization problem of the embodiment can be specifically established as follows:

[0083]

[0084] Among them, ζ1, ζ2 and ζ3 are weight factors. m [n], T m [n] are the total energy consumption and total delay of communication and computing in the nth time slot; E f，m [n] is the flight energy consumption of UAV m in the nth time slot; the optimization variable is: λ={λ m，u [n]} is the drone-user association action, λ m，u [n]∈{0,1} marks whether user u is associated with drone m in the nth time slot; μ={μ m，u [n]} is the user unloading ratio action, and its action space is discrete, μ m，u [n]∈{0,1} is the proportion of computing tasks that user u reserves for local processing; f={f m，u [n]} calculates the frequency allocation action for the drone, It is the acceleration control action of the drone in the x-axis and y-axis directions, and its action space is continuous. and is the acceleration of the drone m on the x-axis and y-axis, is the maximum number of users that UAV m can serve simultaneously in one time slot, which is usually limited by the number of subchannels; F m is the maximum calculation frequency of UAV m.

[0085] Specifically, in step S3, an MDP model is established; MDP is usually used to formally describe reinforcement learning problems. In each time slot, the agent observes the current state from the environment and makes decisions, obtains the next state after executing the action, and adjusts the strategy through the feedback reward value. The embodiment of the invention models the system model and the optimization problem as MDP MDP<S,A,R,P> , where S, A, and R represent the system state set, system action set, and reward function set, respectively, and P is the state transition function.

[0086] The state s[n] in the system state set S includes the coordinates of the drone and the user, and the amount of task data that the user needs to perform. The state s[n] is defined as:

[0087] s[n]={p m [n], p1[n], p2[n], …, p U [n],D1[n],D2[n],…,D U [n]}.

[0088] The action α[n] in the system action set A includes the acceleration of drone m on the x-axis and y-axis with a limited range of values and The calculation frequency assigned by the drone to the associated user, the associated actions between the drone and the user, and the user task offloading ratio. The acceleration and calculation frequency are continuous action spaces, and the association and offloading ratio are discrete action spaces. α[n] is a multi-dimensional discrete and continuous mixed action, specifically defined as:

[0089]

[0090] Among them, α[n] is composed of continuous actions and discrete actions α2[n] = {λ m，1 [n], …, λ m，U [n],μ m，1 [n], …, μ m，U [n]}.

[0091] After each state transition, the reward function calculates the reward value based on the agent's action and the resulting state, and the agent obtains the reward value. The agent uses the feedback provided by the reward to continuously optimize its action decision strategy over time. Different from the minimization optimization goal in optimization theory, the goal of the reinforcement learning intelligent algorithm is to maximize the long-term cumulative reward value, so the immediate reward function of the environment can be defined as:

[0092]

[0093] Among them, Pen[n] is the penalty, including the penalty for the drone movement beyond the boundary of the service area, and the reward function set is composed of reward functions.

[0094] Specifically, the collaborative optimization strategy in step S4 is as follows: Figure 2 As shown in the figure, the structure of the PPO-based intelligent collaborative optimization strategy mainly includes two neural networks, namely the actor network and the critic network.

[0095] The function of the Actor network is to select actions based on the current state of the environment observed by the agent. Its goal is to train a strategy that guides the agent to make action decisions based on the current state of the environment. The function of the Critic network is to evaluate the quality of the actions selected by the Actor network based on the current state of the environment. The Actor network optimizes its strategy based on the evaluation of the Critic network.

[0096] In the PPO-based intelligent collaborative optimization strategy proposed in the embodiment of the present invention, a learning and training process of the model is specifically as follows:

[0097] Define that in each iteration, the Actor network parameters before and after the update are θ old With θ new ; The parameters of the Critic network before and after the update are and

[0098] First, it is necessary to collect enough samples for the training phase. In this embodiment of the present invention, N time slots are used as an episode. In each time slot, the Actor network performs a Output an action α[n]. Then, input the action α[n] into the environment, obtain the state s[n+1] at the next moment according to the state transition function, and obtain the reward r[n]=r(s[n], α[n], s[n+1]) according to the reward function. After that, store the tuple (s[n], α[n], s[n+1], r[n]) as a sample in the experience pool. Continue to perform the above steps until N time slots are completed, and obtain a trajectory l i =(s[0], α[0], ..., s[N-1], α[N-1], s[N]). The above steps are repeated until enough samples are stored and a trajectory set Ψ = {l i}, l∈Ψ, where l represents the trajectory.

[0099] After collecting enough samples, the Actor network and the Critic network will learn based on the sample data in the experience pool.

[0100] Calculate the long-term expected discounted reward value corresponding to time slot n under the current strategy:

[0101]

[0102] Where n' represents a time slot after the start of the nth time slot, and its value range is n≤n'≤N, γ is the discount factor, Used to calculate the strategy , s[n′+1] represents the state of the n′+1th time slot, α[n′+1] represents the action of the n′+1th time slot, and r(s[n′+1], α[n′+1]) represents the reward of the n′+1th time slot;

[0103] Calculating Rewards It represents the sum of rewards after the start of the nth time slot on the trajectory:

[0104]

[0105] Where s[n′] represents the state of a time slot after the start of the nth time slot, α[n′] represents the action of a time slot after the start of the nth time slot, and r(s[n′], α[n′]) represents the reward of a time slot after the start of the nth time slot; calculate the advantage function corresponding to time slot n under the current Actor network strategy

[0106]

[0107] Calculate the loss function of the Critic network:

[0108]

[0109] in, Represents the sum of rewards after the start of the nth time slot on the trajectory; back propagation is performed, and the Critic network is updated and iterated with the help of the gradient descent algorithm and the mean square error regression method; the parameters of the Critic network are updated as follows:

[0110]

[0111] in is the feasible domain of weights and bias parameters that need to be updated in the Critic network, For each update iteration, the set of critic network parameters that minimizes the loss function within the feasible domain; calculate the probability distribution of continuous and discrete actions:

[0112] π θ (α1[n]|s[n]),π θ (α2[n]|s[n])

[0113] Calculate the continuous and discrete action probability distributions under the old strategy Actor network before the update, as follows:

[0114]

[0115] Calculate the probability distribution ratio of continuous discrete mixed actions under the new and old strategies:

[0116]

[0117] Among them, ε is a hyperparameter, usually a small integer, which is used to set the weight ratio of continuous and discrete actions.

[0118] Calculate the loss function of the Actor network:

[0119]

[0120] Among them, ε is a hyperparameter, which is used to constrain the update amplitude. The function of clip() is to convert the continuous discrete mixed action probability distribution ratio φ n (θ) is restricted to the range [1-ε, 1+ε].

[0121] Perform back propagation and use the stochastic gradient descent algorithm to update the Actor network parameters as follows:

[0122]

[0123] The embodiment of the present invention performs sufficient training for the above sampling, training and network updating to obtain an optimized Actor network parameter θ * , that is, to obtain the optimized Actor network action control strategy. Based on this, the agent can output the optimized control decision α in real time according to the observed environment state s. * [n].

[0124] The intelligent optimization method proposed in the embodiment of the present invention can continuously interact with the environment, so that it can continuously and intelligently learn and iteratively update the collaborative optimization strategy, thereby improving the adaptability to the dynamic random characteristics of user service requests and the mobility of network nodes in the mobile edge computing network. At the same time, the trained intelligent model can optimize the actions in the output mixed action space, making the proposed method more adaptable to mobile edge computing application scenarios. In addition, the proposed intelligent optimization method takes latency and energy consumption as optimization targets, ensuring the service quality and service sustainability of the drone-assisted mobile edge computing system.

[0125] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A collaborative optimization method for edge network resource scheduling and drone motion control, characterized in that: The following steps are involved: Step S1: Establish a UAV-assisted mobile edge computing model; first define environmental information, calculate the UAV-assisted mobile edge computing latency and energy consumption indicators; then consider mobility, establish a user mobility model, establish a UAV dynamics model, and calculate the UAV flight energy consumption indicators; The establishment of the UAV dynamics model specifically includes: The motion of UAV m in the continuous time domain is expressed as: Among them, t represents the time in the continuous time domain, p m (t), v m (t) and a m (t) represent the position coordinates, velocity and acceleration of UAV m at time t, respectively, and τ represents the time delay; in the continuous time domain, It means taking the derivative of the coordinate vector of the UAV m and obtaining the velocity vector; Derivative of the velocity vector of the drone m is used to obtain the acceleration vector; the motion state vector of the drone is defined as χ m (t)=(p m (t), v m (t), a m (t)), thus, the dynamic model of UAV m is expressed as: Indicates x m (t) is derived, where Let n represent the nth discrete time slot in the discrete time domain, replaced by the time t in the continuous time domain; the discrete time dynamics model corresponding to the nth sampling interval is: x m [n+1]=X0x m [n]+Y1a m [n]+Y2a m [n-1], among them,x m [n]=x m (nΔT), X0=e XΔT , ΔT represents the duration of a time slot. The flight energy consumption of UAV m in the nth time slot is calculated as: Among them, κ1 and κ2 are system-determined parameters, M is the mass of the drone, and g is the acceleration of gravity; Step S2: Based on the UAV-assisted mobile edge computing model of step S1, a collaborative optimization problem model combining edge network resource scheduling and UAV motion control is constructed; the optimization problem model is specifically established as follows: Among them, ζ1, ζ2 and ζ3 are weight factors, E m [n], T m [n] are the total energy consumption and total delay of communication and computing in the nth time slot; E f,m [n] is the flight energy consumption of UAV m in the nth time slot; the optimization variable is: λ={λ m，u [n]} is the drone-user association action, λ m，u [n]∈{0,1} marks whether user u is associated with drone m in the nth time slot; μ={μ m，u [n]} is the user unloading ratio action, whose action space is discrete, μ m，u [n]∈{0,1} is the proportion of computing tasks that user u reserves for local processing; f={f m，u [n]} calculates the frequency allocation action for the drone, f m，u [n] is the computing frequency assigned to user u by drone m; It is the acceleration control action of the drone in the x-axis and y-axis directions, and its action space is continuous. and is the acceleration of the drone m on the x-axis and y-axis, is the maximum number of users that UAV m can serve simultaneously in one time slot, which is usually limited by the number of subchannels; F m is the maximum calculation frequency of UAV m; The total communication and computation delay of the nth time slot is: in, Calculate the time consumed locally for the user. Offload time-consuming tasks to users. Edge computing for drones takes time. Feedback for drone results is time-consuming; The total energy consumption of communication and calculation in the nth time slot is: in, Calculate energy consumption locally for users, Offloading energy consumption for user tasks, Energy consumption for drone edge computing, Feedback energy consumption for the UAV result; Step S3: Based on the collaborative optimization problem model of edge network resource scheduling and UAV motion control, establish the corresponding Markov decision process MDP model; Step S4, based on the MDP model established in step S3, deep reinforcement learning is used to obtain a collaborative optimization strategy for network resource scheduling and UAV motion control suitable for UAV-assisted mobile edge computing; the collaborative optimization strategy includes two neural networks, an actor network and a critic network; the learning and training process of the actor network and the critic network is specifically as follows: Define that in each iteration, the Actor network parameters before and after the update are θ old With θ new ; The parameters of the Critic network before and after the update are and Step S4.1: Get a sample data set; take N time slots as one episode, in each time slot, the Actor network performs the following operations based on the input state s[n] and the current strategy Output an action α[n]; then input the action α[n] into the environment, obtain the state s[n+1] at the next moment according to the state transfer function, and obtain the reward r[n] = r(s[n], α[n], s[n+1]) according to the reward function; then store the tuple (s[n], α[n], s[n+1], r[n]) as a sample in the experience pool; continue to perform the above steps until N time slots are completed, and obtain a trajectory l i -(s[0], α[0], ..., s[N-1], α[N-1], s[N]); repeat the above steps until enough samples are stored and a trajectory set Ψ = {l i }, l∈Ψ, where l represents the trajectory; Step S4.2: Actor network and Critic network training and learning; Calculate the long-term expected discounted reward value corresponding to time slot n under the current strategy: Where n' represents a time slot after the start of the nth time slot, and its value range is n≤n'≤N, γ is the discount factor, Used to calculate the strategy , s[n′+1] represents the state of the n′+1th time slot, α[n′+1] represents the action of the n′+1th time slot, and r(s[n′+1],α[n′+1]) represents the reward of the n′+1th time slot; Calculating Rewards It represents the sum of rewards after the start of the nth time slot on the trajectory: Where s[n′] represents the state of a time slot after the start of the nth time slot, α[n′] represents the action of a time slot after the start of the nth time slot, and r(s[n′], α[n′]) represents the reward of a time slot after the start of the nth time slot; calculate the advantage function corresponding to time slot n under the current Actor network strategy Calculate the loss function of the Critic network: in, Represents the sum of rewards after the start of the nth time slot on the trajectory; back propagation is performed, and the parameters of the Critic network are updated as follows with the help of the gradient descent algorithm and the mean square error regression method: in is the feasible domain of weights and bias parameters that need to be updated in the Critic network, For each update iteration, the set of critic network parameters that minimizes the loss function within the feasible domain; calculate the probability distribution of continuous and discrete actions: p θ (α1[n]|s[n]),π θ (α2[n]|s[n]), Calculate the continuous and discrete action probability distributions under the old strategy Actor network before the update, as follows: Calculate the probability distribution ratio of continuous discrete mixed actions under the new and old strategies: Among them, ε is a hyperparameter; Calculate the loss function of the Actor network: Among them, ∈ is a hyperparameter; the function of clip() is to convert the probability distribution ratio of continuous discrete mixed actions to n (θ) is limited to the range [1-∈, 1+∈]; Perform back propagation and use the stochastic gradient descent algorithm to update the Actor network parameters as follows: Step S4.3: Repeat steps S4.1-S4.2 to set the number of iterations and obtain the optimized Actor network parameters θ * , that is, to obtain the optimized Actor network control strategy, the drone can output the optimized control decision α in real time according to the observed environmental state s * [n].

2. The method for collaborative optimization of edge network resource scheduling and UAV motion control according to claim 1 is characterized in that: The specific process of establishing the UAV-assisted mobile edge computing model is as follows: First, define the environmental information; assume that there are K ground users in a certain area, with user numbers u∈{1, 2, ..., U}, and drone m acts as an air edge computing server to provide mobile edge computing services for ground users in the area; in the nth time slot, the coordinates of user u are P u ={x u [n],y u [n], 0}, where x u [n] is the x-axis coordinate of user u on the ground plane, y u [n] is the y-axis coordinate of user u on the ground plane, 0 represents the user's height; the coordinate of drone m is p m ={x m [n],y m [n], H}, where x m [n] is the x-axis coordinate of the drone m on the ground plane, y m [n] is the y-axis coordinate of the drone m on the ground plane, and H represents the height of the drone; The user randomly generates the amount of computing tasks to be executed in each time slot. In the nth time slot, the proportion of computing tasks that user u retains for local processing is defined as μ m，u [n]∈{0,1}, the drone can be Provide edge computing services to users; The drone makes association decisions for each time slot, deciding which users to associate with, that is, which users unload and upload to the drone, λ m，u [n]∈{0,1} marks whether user u is associated with drone m in the nth time slot; The establishing of the user mobility model specifically includes: The moving speed of the user in the nth time slot is expressed as in, Indicates the speed of the user in the x-axis direction in the nth time slot, represents the speed of the user in the y-axis direction in the nth time slot; based on the Gauss-Markov mobility model, the moving speed v of user u in the n+1th time slot u [n+1] is related to the moving speed in the nth time slot, which is specifically expressed as: Among them, σ u [n]~N(0,1) is the standard normal distribution, δ represents the influence of the previous time slot state on the current time slot state, They represent the asymptotic mean and asymptotic standard deviation of the speed respectively; the coordinates of the mobile user are updated according to the speed as follows: P u [n+1]=P u [n]+v u [n]ΔT, Where ΔT represents the duration of a time slot, p u [n] is the coordinate of user u in the nth time slot.

3. The method for collaborative optimization of edge network resource scheduling and UAV motion control according to claim 2 is characterized in that: The MDP model is MDP<S,A,R,P> , where S, A, and R represent the system state set, system action set, and reward function set, respectively, and P is the state transition function; The state s[n] in the system state set S includes the coordinates of the drone and the user, and the amount of task data that the user needs to perform; The action α[n] in the system action set A includes the acceleration of the drone m with a limited range of values ​​on the x-axis and y-axis, the calculation frequency assigned by the drone to the associated user, the associated actions of the drone and the user, and the user task offloading ratio; where the acceleration and calculation frequency are continuous action spaces, and the association and offloading ratio are discrete action spaces; The immediate reward function of the environment is defined as: r[n]=-(ζ1E m [n]+ζ2E f，m [n]+ζ3T m [n])-Pen[n], Among them, Pen[n] is the penalty, including the penalty for the drone movement beyond the boundary of the service area, and the reward function set is composed of reward functions.

4. The method for collaborative optimization of edge network resource scheduling and UAV motion control according to claim 3 is characterized in that: The total communication and computation delay is calculated as follows: In the nth time slot, the channel gain h from user u to drone m is m，u for: Where β0 represents the channel gain when the reference distance is 1 meter; p m [n] is the coordinate of drone m in the nth time slot, p u [n] is the coordinate of user u in the nth time slot; ||·|| represents the modulus of the calculation vector, and the calculation vector (p m [n]-p u The modulus of the vector [n] is the distance from user u to drone m in a physical sense. Considering the OFDMA mechanism, the bandwidth allocated to each user is B, the transmission power of user u is P0, and the transmission power of drone m is P m ; Calculate the unloading communication rate from user u to drone m in the nth time slot as: Among them, σ 2 Represents AWGN (Additive White Gaussian Noise) noise power; The communication rate of the return result from drone m to user u in the nth time slot is calculated as: User task offloading time calculation: The user offloads all or part of the computing tasks to the drone. The time consumption of the offloading upload process is: Among them, D u [n] is the total data volume of the computing task of user u; User local computing time: For some computing tasks that are retained for local processing by user u, the computing time of user u is: Among them, f u [n] is the calculation frequency of user u, and 0≤f u [n]≤F u , F u is the maximum computing frequency of the user, C0 is the number of CPU cycles for computing the unit task data volume; Drone edge computing time: Among them, f m，u [n] is the computing frequency assigned to user u by drone m, and the maximum computing frequency of drone m is F m ; UAV result feedback time: After the UAV completes the computing task offloaded by user u, it feeds back the computing result to user u. The time consumed for transmitting the computing result is: Among them, D m [n] is the amount of calculation result data that drone m needs to feed back to user u.

5. The method for collaborative optimization of edge network resource scheduling and UAV motion control according to claim 4 is characterized in that: The total communication and computation delay is calculated as follows: Energy consumption of user task offloading: Among them, P0 is the communication transmission power of the ground user; User local computing energy consumption: η is a positive coefficient related to the user processor chip; Drone edge computing energy consumption: ψ is a positive coefficient related to the UAV processor chip; UAV result feedback energy consumption: P m Transmit power for UAV communication.

Images