UAV-assisted mobile edge computing reinforcement learning method based on triple learner

Abstract

The present invention discloses a UAV-assisted mobile edge computing reinforcement learning method based on a triple learner. First, a mobile edge computing network is established with UAVs and IoT devices with edge computing capabilities, and the UAVs are deployed in the form of a grid. The UAVs serve as mobile edge servers to provide task computing for IoT devices, and a cooperative working mechanism of UAVs is established with the goal of maximizing the energy efficiency of all UAVs. A multi-agent random game including trajectory planning, application deployment and energy update is established for the maneuverability, storage space and energy of UAVs. The method described in the present invention integrates trajectory learners, application learners and energy learners, solves the long-term UAV energy efficiency optimization problem under mobile edge computing of multiple UAVs, and provides a solution for reasonable decision-making to maximize the energy efficiency of UAVs in UAV-assisted mobile edge computing.

Description

Technical Field

[0001] The present invention belongs to the technology of optimizing drone trajectory, charging scheduling and application placement in drone-assisted mobile edge computing, and specifically relates to a drone-assisted mobile edge computing reinforcement learning method based on a triple learner. Background Art

[0002] Mobile edge computing has been widely accepted as a promising technology to support resource-constrained mobile IoT devices to run computationally intensive and latency-sensitive IoT applications. Mobile edge computing emphasizes that application services and content can be localized and distributed in close proximity. Each mobile edge computing entity provides services to its users, which to a certain extent solves the business requirements of 5G enhanced mobile broadband low latency, high reliability and large-scale machine communication terminal connections. However, there are still some key issues related to the optimization of mobile edge computing for multiple drones that have not been well studied. The problems mainly include three aspects:

[0003] 1) The number and coverage of drones are limited, and it is impossible to provide edge computing services for all IoT devices at the same time. This requires drones to fly in different areas to provide services for IoT devices.

[0004] 2) Each task is calculated by the corresponding application. The storage capacity of the drone is limited and cannot store all types of applications. Due to the large amount of application data and the limited transmission rate of the drone to the ground base station, the delay of the drone deploying the application is very large. This prompts the drone to return to the ground to update the application location.

[0005] 3) Due to the high maneuverability of drones, it is not easy to renew the energy of drones in the air. This requires the drone to return to the ground for energy renewal.

[0006] However, if all the above factors are taken into account, the realization of drone-assisted mobile edge computing system will encounter certain difficulties, for the following three reasons:

[0007] a. Due to the limited number and coverage of drones, each drone can only cover some IoT devices. Therefore, the drone must fly over another area to cover some IoT devices. Optimizing the flight trajectory of drones to provide edge computing services for as many IoT devices as possible is a challenging problem.

[0008] b. Due to the large delay in placing applications in the air, the drone needs to update its application placement when it returns to the ground. Optimizing the application placement of each drone to calculate more tasks is a challenging problem.

[0009] c. Since it is inconvenient to update the energy of drones in the air, drones need to return to the ground for energy update. Under the premise of the service quality requirements of IoT devices, the optimal decision of drone energy update is a challenging problem. Summary of the invention

[0010] Purpose of the invention: In view of the problems in the above-mentioned prior art such as the limited number and coverage of drones, the large delay in placing and applying drones in the air, and the inconvenient energy update of drones in the air, the present invention provides a drone-assisted mobile edge computing reinforcement learning method based on a triple learner.

[0011] A UAV-assisted mobile edge computing reinforcement learning method based on triple learner, comprising the following steps:

[0012] (1) Setting up edge servers through drones to build a mobile edge computing network, deploying drones in a grid form, wherein the mobile edge computing network also includes an IoT device, wherein the IoT device establishes communication with the drone to offload computing tasks from the IoT device, and after receiving the computing tasks requested by the IoT device, the drone performs task computing and sends the results to the IoT device;

[0013] (2) A cooperative working mechanism between UAVs is established in the mobile edge computing network. A multi-agent random game including trajectory planning, application deployment, and energy update is established based on the maneuverability, storage space, and energy of UAVs. The goal is to maximize the energy efficiency of all UAVs. Three related Q-learning-based learners are set in each UAV to find the equilibrium of the three games, including trajectory learner, application learner, and energy learner.

[0014] In the multi-agent random game, each drone selects its own action in each time slot, including all drones cooperating with other drones to achieve the same goal, and the game process of each drone is described by a finite state Markov decision process;

[0015] (3) In each time slot, the UAV in the target area decides whether to return to the ground based on the status information of all UAVs, corresponding to the following process:

[0016] If it does not return to the ground, it will choose a direction and move from the center of one grid to the center of another adjacent grid at a constant speed v, then hover at the center of the grid, receiving and calculating the requested tasks while hovering for a period of time t;

[0017] If it returns to the ground, the drone will update the energy and update the placement of the application within the time slot t. After updating the energy and the location of the application, the drone returns to the target area and continues to provide edge computing services for IoT devices;

[0018] The trajectory learner learns to choose the direction of movement in each time slot, the application learner learns to choose the strategy of updating application placement in each time slot, and the energy learner learns to choose whether to return to the charging station to update the energy in each time slot.

[0019] Furthermore, in the calculation of seeking the equilibrium of the three games in step (2), the mathematical expression of the objective function for maximizing the energy efficiency of the drone is as follows:

[0020]

[0021] Constraints:

[0022]

[0023]

[0024] φ(t)∈[0,1],t≥0 (d)

[0025]

[0026]

[0027] |U m (t)-I m (t)| 2 ε m (t) = q 2 (g)

[0028]

[0029] Among them, constraints (a)-(c) are binary variables. Constraint (e) means that the sum of transmission time and computation time cannot be greater than the hovering time. Constraints (g) and (h) indicate that in each time slot, the drone can only move from the center of one grid to the center of another adjacent grid without updating the energy.

[0030] In the above method, in the mobile edge computing system, the IoT device with a task request will request to offload the task to a drone located in the same grid;

[0031] The method for offloading the computing task model between a single IoT device and a drone includes the following calculations:

[0032] The instantaneous rate of IoT device n offloading tasks to drone m is expressed as:

[0033]

[0034] where γ m,n(t) is the signal to interference noise ratio in time slot t, ζ m,n (t)∈{0,1} indicates whether the drone m and the IoT device n are located on the same grid at time t, where ζ m,n (t) = 1 means that the drone m and the IoT device n are located on the same grid at time t, otherwise ζ m,n (t) = 0; o n (t)∈{0,1} represents the vector of whether IoT device n has an offload task request in time slot t, where o n (t) = 1 means that IoT device n has an offload task request in time slot t, otherwise o n (t) = 0; at time slot t, the task requested by IoT device n is v n (t)∈{v n,1 (t),v n,2 (t),...,v n,C (t)}, where C is the number of mission types; the application w placed in UAV m m (t)∈{w m,1 (t),w m,2 (t),...,w m,S (t)}, where S is the number of mission types; B is the bandwidth of the wireless channel between the UAV and the IoT device, in Hz;

[0035] The time it takes to offload IoT device n to drone m is expressed in mathematical form as follows:

[0036]

[0037] Where D is the size of each task. In the UAV network considered, it is assumed that all UAVs can obtain the information of UAVs, including the location of UAVs and the placement application.

[0038] The method includes the following calculations in the construction and processing of the UAV computing model:

[0039] After drone m receives the unloading task from IoT device n, the time to calculate the unloading task of IoT device n is:

[0040]

[0041] where f U The CPU frequency calculated for the drone, in cycles / second, and the size of all received tasks calculated by drone m is:

[0042]

[0043] Among them, t hover is the hovering time of the UAV at each time point, H mFor the IoT device set m that offloads tasks to drone m, let f U Indicates the CPU cycles required to calculate each bit;

[0044] definition The energy consumption of UAV m in the calculation task in time slot t is as follows:

[0045]

[0046] Where ξ is the effective capacitance coefficient of the UAV.

[0047] The method includes establishing a propulsion power model of a rotary-wing UAV that depends only on a speed factor. For the rotary-wing UAV, the propulsion power is expressed as:

[0048]

[0049] in, Body S F It represents the equivalent flat plate area of ​​the fuselage;

[0050] From the above formula, we can see that the propulsion energy consumption of the rotorcraft at each time point consists of two parts: horizontal motion energy and hovering energy, which can be expressed as

[0051] In the method, all drones move at a uniform speed, and the flight energy consumption of each drone in each time slot is equal.

[0052] Furthermore, the method includes constructing a charging model for a drone, including the following process:

[0053] ε m It is a vector indicating whether the energy of UAV m is updated, where ε m =1 means that UAV m updates its energy at time slot t, otherwise ε m = 0, and the drone returns to the grid above the target area before updating the energy after being fully charged, and continues to provide edge computing services for IoT users;

[0054] Assuming that each type of application should place at least one drone hovering over the target area in each time slot, the following relationship exists:

[0055]

[0056] Furthermore, in the method, the mathematical modeling and description of the trajectory learner includes the following process:

[0057] (11) Trajectory learner state and action: The environmental state of the trajectory learner consists of the current position of the UAV, the installed application, and the time slot index. At time slot t, the environmental state of the trajectory learner is expressed as:

[0058] s T (t) = [{L m (t)} m∈M ,{w m (t)} m∈M ]

[0059] Without loss of generality, the trajectory learner a T The set of actions includes four actions: moving forward, moving backward, moving left, and moving right;

[0060] (12) Reward of trajectory learner: At time slot t, the immediate reward of the trajectory learner of drone m is determined by the following expression:

[0061]

[0062] Among them, the numerator represents the task scale calculated by drone m in time slot t, and the denominator represents the energy consumption of drone m in time slot t. Give the energy efficiency of drone m in time slot t in bit / J;

[0063] (13) Trajectory learner strategy: The trajectory learner strategy in drone m Represents the mapping from state set to action set, using action In a given state s T The probability distribution under state s T The trajectory strategy of the trajectory learner of drone m is recorded as Each element is the drone m in state s T Select the role The probability of The joint trajectory strategy can be defined as π T is the trajectory learner in state s T Select joint action a T probability.

[0064] In the method, applying mathematical modeling and description of the learner includes the following process:

[0065] (21) Application learner state and action: The environment state of the application learner consists of the applications placed on each drone and the task requests received from all IoT devices. At time slot t, the environment state of the application learner can be expressed as:

[0066] s T (t)=[{w m (t)} m∈M ,{θ m (t)} m∈M ]

[0067] in, Without loss of generality, the set of actions to apply the learner a is to select a task type from all types of tasks;

[0068] (22) Rewards for application learners: Considering the differences in applications placed in drones, at time t, the immediate reward of the application learner in drone m is:

[0069]

[0070] Where N A represents the number of application types placed on all drones at time slot t, represents the tasks required by the IoT device before time slot t;

[0071] (23) Strategy of applying learner: Strategy Application of learning machine Represents the mapping from state set to action set, using action In a given state s A The probability distribution in; for state s A The trajectory strategy of the trajectory learner of drone m is recorded as Each element is the drone m in state s A Select the role The probability of When the drone is at the charging station, the application learners select actions individually. Assuming that the actions selected by all application learners are independent under a given environmental state, the joint application strategy is defined as π A is to apply the learner in state s A Select Joint Action a A probability.

[0072] In the method, the mathematical modeling and description of the energy learner includes the following process:

[0073] (31) Energy learner state and action: The environmental state of the energy learner consists of the current remaining energy and position of each UAV. The environmental state of the energy learner at time slot t is expressed as:

[0074] s E (t) = [{E m (t)} m∈M ,{ε m (t)} m∈M ]

[0075] The set of actions of the energy learner includes two actions: updating energy and not updating energy;

[0076] (32) Reward of energy learner: The reward of one step of energy learner is defined as:

[0077]

[0078] where ε m (t) = {0, 1}, the energy learner should place at least one drone hovering over the target area in each time slot for each type of application, and the reward of the drone that does not update energy is equal to 0; on the contrary, the reward of the energy learner is 1 when the drone updates energy. If there is no drone on duty in the grid of the target area, the reward of each energy learner is -10;

[0079] (33) Strategy of Energy Learner: Strategy Application of learning machine Represents the mapping from state set to action set, using action In a given state s E The probability distribution in; for state s E The trajectory strategy of the application learner of drone m is recorded as Each element is the drone m in state s E Select the role The probability of When the UAV is in the target area, the energy learner selects actions individually. Assuming that under the given current environment state, the selected actions of all energy learners are independent, the joint application strategy is defined as π E is to apply the learner in state s E Select Joint Action a E probability.

[0080] Beneficial effect: The UAV-assisted mobile edge computing reinforcement learning method based on triple learner provided by the present invention solves the long-term UAV energy efficiency optimization problem under multiple UAV mobile edge computing, and proposes to use UAVs equipped with edge computing servers to realize the computing task offloading allocation problem, and ensure that there are one or more UAVs in service in the grid corresponding to the target area, which improves the stability of the system and realizes energy efficiency optimization, providing a solution for reasonable decision-making to maximize UAV energy efficiency in UAV-assisted mobile edge computing. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 Schematic diagram of a drone-assisted mobile edge computing system in an embodiment;

[0082] Figure 2A schematic diagram of the framework of multi-UAV trajectory planning, energy update and application layout in the embodiment;

[0083] Figure 3 Shown is a comparison of the long-term average energy efficiency of drones at different IoT device transmission powers between the embodiment and the prior art;

[0084] Figure 4 The figure shows the comparison of the long-term average energy efficiency of the drones of the embodiment and the prior art at different hovering times. DETAILED DESCRIPTION

[0085] In order to explain the technical solution disclosed in the present invention in detail, further elaboration is made below in conjunction with the accompanying drawings and specific embodiments of the specification.

[0086] Aiming at the optimization problems of UAV trajectory planning, application placement and energy update in the framework of triple learner, the present invention provides a UAV-assisted mobile edge computing reinforcement learning method based on triple learner. Each UAV runs three standard Q-learning algorithms to learn the optimal Q value of each state-action pair and obtain the optimal local strategies of trajectory control, application placement control and energy update control, so as to maximize the average long-term energy efficiency of the UAV.

[0087] Combination Figure 1 and Figure 2 The specific implementation of the UAV-assisted mobile edge computing reinforcement learning method based on triple learner described in the present invention includes the following considerations:

[0088] (S1) describes the network model of drone mobile edge computing, including the offloading model, drone computing model, drone propulsion energy model, and drone update energy model. The mobile edge computing system consists of drones and IoT devices. The drone acts as an edge server and provides edge computing services for many IoT devices. Each IoT device is associated with a drone for offloading requested tasks through wireless communication. After receiving the task requested by the IoT device, the drone performs task calculation and sends the result to the IoT device. In each time period, the drone chooses to fly to other areas or return to the ground to provide edge computing services for more devices to update energy and update application layout.

[0089] (S2) In order to motivate multiple drones to cooperate with each other rather than work independently, a multi-drone reinforcement learning framework is designed to maximize the energy efficiency of all drones under network dynamics. In view of the limited maneuverability, storage space and energy of drones, the energy efficiency maximization problem is modeled as three multi-agent stochastic games corresponding to trajectory planning, application deployment and energy update. The equilibrium of the game is that the energy efficiency of all drones reaches the maximum value.

[0090] (S3) In order to achieve the equilibrium of the above three games, the present invention combines the reinforcement learning method of triple learners and sets three related Q-learning-based learners in each drone to find the equilibrium of the three games. In addition, the present invention designs a new objective function to meet the quality of service constraints and maximize the energy efficiency of all drones.

[0091] Specifically, in step (S1), in the mobile edge computing system, the IoT device with a task request will offload the task to the drone in the same grid; the computing task offloading model between a single IoT device and a drone includes the following calculations:

[0092] The instantaneous rate of IoT device n offloading tasks to drone m is expressed as:

[0093]

[0094] where γ m,n (t) is the signal to interference noise ratio in time slot t, ζ m,n (t)∈{0,1} indicates whether the drone m and the IoT device n are located on the same grid at time t, where ζ m,n (t) = 1 means that the drone m and the IoT device n are located on the same grid at time t, otherwise ζ m,n (t) = 0; o n (t)∈{0,1} represents the vector of whether IoT device n has an offload task request in time slot t, where o n (t) = 1 means that IoT device n has an offload task request in time slot t, otherwise o n (t) = 0; at time slot t, the task requested by IoT device n is v n (t)∈{v n,1 (t),v n,2 (t),...,v n,C (t)}, where C is the number of mission types; the application w placed in UAV m m (t)∈{w m,1 (t),w m,2 (t),...,w m,S (t)}, where S is the number of mission types; B is the bandwidth of the wireless channel between the drone and the IoT device, in Hz.

[0095] The time it takes to offload IoT device n to drone m is expressed in mathematical form as follows:

[0096]

[0097] Where D is the size of each task. In the UAV network considered, it is assumed that all UAVs can obtain the information of UAVs, including the location of UAVs and the placement application.

[0098] Among them, the construction and processing of the drone computing model include the following calculations:

[0099] After drone m receives the unloading task from IoT device n, the time to calculate the unloading task of IoT device n is:

[0100]

[0101] where f U The CPU frequency calculated for the drone, in cycles / second, and the size of all received tasks calculated by drone m is:

[0102]

[0103] Among them, t hover is the hovering time of the UAV at each time point, H m For the IoT device set m that offloads tasks to drone m, let f U Indicates the CPU cycles required to calculate each bit;

[0104] definition The energy consumption of UAV m in the calculation task in time slot t is as follows:

[0105]

[0106] Where ξ is the effective capacitance coefficient of the UAV.

[0107] In conjunction with step (S1), it is further pointed out that the method includes establishing a propulsion power model of a rotary-wing UAV that depends only on a speed factor. For a rotary-wing UAV, the propulsion power is:

[0108]

[0109] in, Body S F It represents the equivalent flat plate area of ​​the fuselage;

[0110] From the above formula, we can see that the propulsion energy consumption of the rotorcraft at each time point consists of two parts: horizontal motion energy and hovering energy, which can be expressed as

[0111] In the method, all drones move at a uniform speed, and the flight energy consumption of each drone in each time slot is equal.

[0112] The method for constructing a UAV charging model includes the following processes:

[0113] ε m It is a vector indicating whether the energy of UAV m is updated, where ε m =1 means that UAV m updates its energy at time slot t, otherwise ε m = 0. After the drone is fully charged, it returns to the grid above the target area before renewing the energy, and continues to provide edge computing services for IoT users.

[0114] The method also takes into account different applications placed on drones. It is assumed that each type of application should place at least one drone hovering over the target area in each time slot, and the following relationship exists:

[0115]

[0116] Furthermore, the energy efficiency maximization problem of multiple UAVs in step (S2) is expressed as

[0117]

[0118]

[0119]

[0120]

[0121] φ(t)∈[0,1],t≥0 (12)

[0122]

[0123]

[0124] |U m (t)-I m (t)| 2 ε m (t) = q 2 (15)

[0125]

[0126] Among them, constraints (9)-(11) are binary variables. Constraint (13) means that the sum of transmission time and computation time cannot be greater than the hovering time. Constraints (15) and (16) indicate that in each time slot, the drone can only move from the center of one grid to the center of another adjacent grid without updating the energy. However, directly solving the random game to find the equilibrium is very challenging because each individual has multiple strategies and the relationship between them is unclear. Therefore, the present invention proposes a new method to reconstruct the problem from the perspective of cooperation and competition and derive its solution by analyzing the intrinsic characteristics of the random game and the properties of the equilibrium. It should be noted that for ease of implementation, the edge computing platform can run the random game on behalf of all drones and generate an equilibrium so that the drones have no motivation to deviate.

[0127] Furthermore, the multi-agent random game in step (S2) is defined as the tuple<M,S,A,P,R> .

[0128] 1) M is a set of agents.

[0129] 2) S is the set of environmental states. At time slot t, the environmental state is denoted as s(t).

[0130] 3) A={A1,A1,...,A M} is a set of joint actions, where A m is the set of individual actions of agent m. The joint action at time slot t is represented by a(t)∈A, and the set of individual actions of agent m is represented by a m (t)∈A m Therefore, the joint action is recorded as a(t) = {a1(t),...,a M (t)}.

[0131] 4) P is the set of state transition probabilities. ss' (a(t)) is the state transition probability from state s to s' by executing the joint action a(t)∈A.

[0132] 5) R = {R1, ..., R M} is a set of reward functions, where R M is the set of immediate returns of m.

[0133] In a multi-agent random game, each drone selects its own action individually in each time slot, and all drones cooperate with other drones to achieve the same goal. In order to achieve the same goal, all drones should reach a consensus on the choice of action. Due to the competition between drones, each drone will be affected by other drones when choosing an action. The present invention introduces a finite state Markov decision process to characterize the game process of each drone. As mentioned above, the state-behavior transition satisfies the Markov property. The present invention models the game process in each drone as an MDP, where P m represents the state transition probability matrix of drone m. Note that each drone does not know the reward and transition functions of all drones in advance.

[0134] Furthermore, in the step (S2), the UAV trajectory planning, energy update and application layout studied in the present invention are as follows: Figure 1 As shown in the figure. When the drone chooses whether to update energy and update the application location, it cannot choose the flight direction. Therefore, a triple learner is considered in each drone. The trajectory learner learns to choose the direction of movement in each time slot. The application learner learns to choose the strategy of updating the application placement in each time slot. The energy learner learns whether to return to the charging station to update the energy in each time slot.

[0135] In view of the situation that each learner in the multi-UAV network works in an unknown random environment and does not know the reward function and transfer function, a Q-learning method is proposed to maximize the expected long-term reward function of the network. Similar to centralized Q-learning, this algorithm is based on a series of state-actions. In addition, it is necessary to design a one-step reward for each UAV to evaluate the action performed by the UAV in a certain state. The goal of the triple learner algorithm is to ensure that the UAV gradually learns the optimal strategy based on a series of state-actions and one-step rewards. Therefore, the settings of the trajectory learner, application learner, and energy learner focus on designing state transitions and one-step rewards.

[0136] The goal of the trajectory learner is to design trajectory planning for multiple UAVs to control all UAVs located in the target area to choose a direction in each time slot. The trajectory planning of multiple UAVs must both ensure the quality of service of IoT devices and maximize energy efficiency. Intuitively, it is reasonable to consider energy efficiency because IoT devices cannot meet the quality of service requirements of UAVs due to limited energy. However, aiming at maximizing the total energy efficiency may lead to unfair service, which means that there is a part of IoT devices that are rarely or never served by UAVs. Therefore, the flight trajectory planning of multiple UAVs must consider the service fairness issue. As mentioned above, the setting of the trajectory learner can be described as follows.

[0137] 11) Trajectory learner state and action: The environment state of the trajectory learner consists of the current position of the UAV, the installed application, and the time slot index. In particular, at time slot t, the environment state of the trajectory learner can be expressed as

[0138] s T (t) = [{L m (t)} m∈M ,{w m (t)} m∈M ] (17)

[0139] Without loss of generality, the trajectory learner a T The set of actions includes four actions, namely 1) moving forward, 2) moving backward, 3) moving left, and 4) moving right.

[0140] 12) Rewards for trajectory learners: At time slot t, the immediate reward for the trajectory learner of drone m is given by

[0141]

[0142] Among them, the numerator represents the task scale calculated by drone m in time slot t, and the denominator represents the energy consumption of drone m in time slot t. Therefore, The energy efficiency of drone m in time slot t in bit / J is given. The reward function can guide the trajectory learner to calculate more tasks and ensure fairness among IoT devices.

[0143] 13) Trajectory learner strategy: The strategy of trajectory learner in drone m Represents a mapping from a state set to an action set, which is an available action In a given state s T In particular, for state s T The trajectory strategy of the trajectory learner of drone m is recorded as Each element is the drone m in state s T Select the role The probability of Since trajectory learners select actions individually in each time slot, we can assume that the actions selected by all trajectory learners are independent given the current environment state. Therefore, the joint trajectory policy can be defined as π T is the trajectory learner in state s T Select joint action a T probability.

[0144] During the selection time of each time slot, the drone selects an action based on its own local Q function. The action selection strategy should strike a balance between exploration and exploitation. The drone can strengthen the evaluation of actions it already knows are good, but also explore new actions. In the action selection of the energy learner, the present invention considers an e-greedy exploration strategy. Specifically, the Q function of the trajectory learner on drone m is the state s The present invention gives a definition of the action value function of the trajectory learner, namely the Q function. Specifically, the Q function of the trajectory learner on drone m is the state s T In a given strategy Next action The expected reward of Q value is expressed as:

[0145]

[0146] where s T ' is the next environment state of the trajectory learner, α t and β t is the learning rate. If the trajectory learner is in state s T 'Execute joint action a T , the action selection of the trajectory learner is affected by the energy learner and the application learner. In particular, if the energy learner in drone m chooses an action to update the energy, the trajectory learner will not choose any action. Otherwise, the trajectory learner in drone m will follow the e-greedy exploration strategy in the set So, in state s T Select Action The probability is:

[0147]

[0148] Where e∈(0,1). In addition, if there are multiple actions corresponding to the highest Q value, the trajectory learner will randomly select an action from them.

[0149] The goal of the application learner is to update the application of the drone in each time slot. The application planning of multiple drones should meet the task type requirements of IoT devices. Intuitively, the more tasks the drone computes, the better the mobile edge computing service. However, the service quality of IoT devices requires that each application is placed on at least one drone. Therefore, it is necessary to place more types of applications on all drones. As mentioned above, the setting of the application learner can be described as follows.

[0150] 21) Application learner state and action: The environment state of the application learner consists of the applications placed on each drone and the task requests received from all IoT devices. In particular, at time slot t, the environment state of the application learner can be expressed as:

[0151] s T (t)=[{w m (t)} m∈M ,{θ m (t)} m∈M ] (twenty one)

[0152] in, Without loss of generality, the set of actions to which the learner a is applied is to select a task type from among all types of tasks.

[0153] 22) Rewards for application learners: Considering the differences in applications placed in drones, at time t, the immediate reward for application learners in drone m is

[0154]

[0155] Where N A represents the number of application types placed on all drones at time slot t. Therefore, The tasks required by IoT devices before time slot t are given. The reward function can guide the application learner to calculate more tasks and ensure the service quality of IoT devices.

[0156] 23) Strategy for applying learners: Strategy Application of learning machine Represents a mapping from a state set to an action set, which is an available action In a given state s A In particular, for state s A The trajectory strategy of the trajectory learner of drone m is recorded as Each element is the drone m in state s A Select the role The probability of When the drone is at the charging station, the application learners select actions individually, so it can be assumed that the actions selected by all application learners are independent under a given environmental state. Therefore, the joint application strategy can be defined as π A is to apply the learner in state s A Select Joint Action a A probability.

[0157] The present invention provides a definition of the action value function of the trajectory learner, namely the Q function. Specifically, the Q function of the application learner on the drone m is the state s A In a given strategy Next action The expected reward of Q value is expressed as:

[0158]

[0159] If the trajectory learner is in state s A Perform joint action a A , then the next environment state of ACG is s A '. The action selection of the application learner is influenced by the energy learner. In particular, if the energy learner in drone m selects an action to update the energy, the application learner will follow the exploration strategy of e-greedy. Otherwise, the application learner in drone m will not select any action. Therefore, in state s A Select the role The probability is:

[0160]

[0161] Where e∈(0,1). In addition, if there are multiple actions corresponding to the highest Q value, the application learner will randomly select an action from them.

[0162] The goal of the energy learner is to control the updated energy of the drones under the mobile edge computing service, that is, to always meet the service quality requirements of each IoT device during the whole process. Intuitively, more drones hovering over the target area can provide better mobile edge computing services for IoT devices. However, if all drones hover over the target area until the energy is exhausted, the mobile edge computing service may not be guaranteed. Therefore, it is particularly important to control the energy update of drones while meeting the service quality.

[0163] 31) Energy learner state and action: The environmental state of the energy learner consists of the current remaining energy and position of each drone. Among them, the environmental state of the energy learner in time slot t is:

[0164] s E (t) = [{E m (t)} m∈M ,{ε m (t)} m∈M ] (25)

[0165] The set of actions of the energy learner includes two actions: updating energy and not updating energy.

[0166] 32) Reward of energy learner: Considering constraint (7), the present invention defines a one-step reward of energy learner, which is expressed as:

[0167]

[0168] where ε m (t) = {0, 1}, when the energy learner satisfies constraint (7), the drone that does not update its energy is equal to 0. On the contrary, when the drone updates its energy, the reward of the energy learner is 1, and when the constraint (7) is violated, the reward of each energy learner is -10. Therefore, the reward of the energy learner can make the drone hover over the target area as much as possible while avoiding violating the constraints.

[0169] 33) Strategy of Energy Learner: Strategy Application of learning machine Represents a mapping from a state set to an action set, which is an available action In a given state s E In particular, for state s E The trajectory strategy of the application learner of drone m is recorded as Each element is the drone m in state s E Select the role The probability of When the drone is in the target area, the energy learner selects actions individually, so it can be assumed that the selected actions of all energy learners are independent given the current environment state. Therefore, the joint application strategy can be defined as π E is to apply the learner in state s E Select Joint Action a E probability.

[0170] The present invention provides a definition of the action value function of the energy learner, namely the Q function. Specifically, the Q function of the application learner on the drone m is the state s E In a given strategy Next action The expected reward of Q value is expressed as:

[0171]

[0172] Among them, if the energy learner is in state s E Perform joint action a E , then s E ' is the next environment state of ECG. In state s E Select Action The probability of is given by the following expression:

[0173]

[0174] Where e∈(0,1). In addition, if there are multiple actions corresponding to the highest Q value, the energy learner will randomly select an action from them.

[0175] Example 2

[0176] Aiming at the optimization problems of drone trajectory planning, application placement and energy update in the framework of triple learner, the method of the present invention is implemented in combination with relevant algorithms. Each drone runs three standard Q-learning algorithms to learn the optimal Q value of each state-action pair and obtain the optimal local strategy of trajectory control, application placement control and energy update control, so as to maximize the average long-term energy efficiency of the drone.

[0177] Combination Figure 2 The mobile edge computing system framework, the implementation process of the method algorithm of the present invention is described below.

[0178] Step 1: Initialize parameters. For all drones, initialize the Q values ​​of the trajectory learner, application learner, and energy learner respectively. and Set the maximum number of iterations LOOP and initialize the iteration number variable loop.

[0179] Step 2: Within the maximum number of iterations, initialize the state s and time slot t, and the drones share the Q values ​​of their own trajectory learners, application learners, and energy learners. and

[0180] Step 3: Within the maximum number of iterations, in each time slot t, first observe all states s, and then each drone m uses the strategy of the energy learner To select an action If UAV m returns to the charging station, UAV m will apply the learner's policy To select an action Otherwise, drone m will follow the strategy of trajectory learner To select an action

[0181] Step 4: Within the maximum number of iterations, in each time slot t, obtain the reward functions of the trajectory learner, application learner, and energy learner respectively and Update the Q values ​​of the trajectory learner, application learner, and energy learner respectively and The Q-values ​​of each learner are then shared between drones.

[0182] Step 5: Within the maximum number of iterations, update the time slot plus 1 and repeat steps 3 and 4 until the time is up.

[0183] Step 6: Repeat steps 2, 3, and 4 until the maximum number of iterations is reached.

[0184] Combination Figure 3 and Figure 4 , Figure 3 The proposed TLRL algorithm is compared with the other two algorithms in terms of average energy efficiency of drones under IoT transmission power. The Distributed Multi-UAV Coordinated Reinforcement Learning (DMUCRL) algorithm does not consider the update of applications, so it cannot update the IoT devices according to the task requirements, and calculates fewer tasks than the TLRL algorithm. The Energy Minimization Trajectory Planning (EMTP) algorithm aims to minimize the average energy consumption of drones. The drones only calculate a small number of tasks, and the transmission power of IoT devices has no effect on the average energy efficiency of drones. It can be seen that based on the strategy of updating applications and maximizing the average energy efficiency of drones, the proposed TLRL algorithm performs better than the other two algorithms. Figure 4 The performance comparison of the proposed TLRL algorithm and the other two algorithms in terms of average energy efficiency under the drone hovering time is shown. The results show that as the drone hovering time increases, the average energy efficiency of the drone of IoT devices under the three algorithms shows a trend of first increasing and then decreasing. The DMUCRL algorithm does not consider updating applications, cannot update applications according to the task requirements of IoT devices, and calculates fewer tasks than the TLRL algorithm. The EMTP algorithm aims to minimize the energy consumption of the drone. The drone only needs to calculate a small number of tasks, thereby reducing the energy consumption of the drone. It can be seen that based on the strategy of updating applications and maximizing the average energy efficiency of the drone, the proposed TLRL algorithm has better experimental results than the other two algorithms.

[0185] Corresponding to the complexity of the triple learner-based reinforcement learning algorithm proposed in the present invention, the control complexity reflects the size of the information that needs to be exchanged between drones to achieve a consistent control strategy, which is crucial, especially when drones provide mobile edge computing services for IoT devices densely distributed in a large area. In the proposed reinforcement learning framework, the information exchange between drones consists of the local learning results of each drone, and its size is determined by the size of the learner's state space. The size of the application learner state space is affected by the number of drones and the number of application types placed in the drone. The size of the trajectory learner state space is affected by the number of drones and the number of small square grids. Therefore, for a given UAV communication network, the present invention can remain unchanged as the density of IoT devices in the target area increases. In addition, the present invention can divide the target area into larger grids, reducing the control complexity of the algorithm. Even if the task is requested by IoT devices distributed in a large area and densely deployed, the control complexity of the method described in the present invention can be kept at a low level. This is of great significance for the practical application of UAV mobile edge computing networks.

Claims

1. A UAV-assisted mobile edge computing reinforcement learning method based on triple learner, characterized by: The steps include: (1) Setting up edge servers through drones to build a mobile edge computing network, deploying drones in a grid form, wherein the mobile edge computing network also includes an IoT device, wherein the IoT device establishes communication with the drone to offload computing tasks from the IoT device, and after receiving the computing tasks requested by the IoT device, the drone performs task computing and sends the results to the IoT device; (2) A cooperative working mechanism between UAVs is established in the mobile edge computing network. A multi-agent random game including trajectory planning, application deployment, and energy update is established based on the maneuverability, storage space, and energy of UAVs. The goal is to maximize the energy efficiency of all UAVs. Three related Q-learning-based learners are set in each UAV to find the equilibrium of the three games, including trajectory learner, application learner, and energy learner. In the multi-agent random game, each drone selects its own action in each time slot, including all drones cooperating with other drones to achieve the same goal, and the game process of each drone is described by a finite state Markov decision process; In this step, in the calculation of seeking the equilibrium of the three games, the mathematical phenotype of the objective function for maximizing the energy efficiency of the drone is as follows: Constraints: φ(t)∈[0,1],t≥0 (d) |U m (t)-I m (t)| 2 ε m (t)=q 2 (g) Among them, constraints (a)-(c) are binary variables, constraint (e) means that the sum of transmission time and calculation time cannot be greater than the hovering time, and constraints (g) and (h) indicate that in each time slot, the drone can only move from the center of one grid to the center of another adjacent grid without updating energy; (3) In each time slot, the UAV in the target area decides whether to return to the ground based on the status information of all UAVs, corresponding to the following process: If it does not return to the ground, it will choose a direction and move from the center of one grid to the center of another adjacent grid at a constant speed v, then hover at the center of the grid, receiving and calculating the requested tasks while hovering for a period of time t; If it returns to the ground, the drone will update the energy and update the placement of the application within the time slot t. After updating the energy and the location of the application, the drone returns to the target area and continues to provide edge computing services for IoT devices; The trajectory learner learns to choose the direction of movement in each time slot, the application learner learns to choose the strategy of updating application placement in each time slot, and the energy learner learns to choose whether to return to the charging station to update energy in each time slot; Furthermore, in the mobile edge computing system, an IoT device with a task request will request to offload the task to a drone located in the same grid; The method for offloading the computing task model between a single IoT device and a drone includes the following calculations: The instantaneous rate of IoT device n offloading tasks to drone m is expressed as: where γ m,n (t) is the signal to interference noise ratio in time slot t, ζ m,n (t)∈{0,1} indicates whether the drone m and the IoT device n are located on the same grid at time t, where ζ m,n (t) = 1 means that the drone m and the IoT device n are located on the same grid at time t, otherwise ζ m,n (t) = 0; o n (t)∈{0,1} represents the vector of whether IoT device n has an offload task request in time slot t, where o n (t) = 1 means that IoT device n has an offload task request in time slot t, otherwise o n (t) = 0; at time slot t, the task requested by IoT device n is v n (t)∈{v n,1 (t),v n,2 (t),...,v n,C (t)}, where C is the number of mission types; the application w placed in UAV m m (t)∈{w m,1 (t),w m,2 (t),...,w m,S (t)}, where S is the number of mission types; B is the bandwidth of the wireless channel between the UAV and the IoT device, in Hz; The time it takes to offload IoT device n to drone m is expressed in mathematical form as follows: Where D is the size of each task. In the UAV network under consideration, it is assumed that all UAVs can obtain the information of UAVs, including the location of UAVs and the placement application; The method includes the following calculations in the construction and processing of the drone computing model: After drone m receives the unloading task from IoT device n, the time to calculate the unloading task of IoT device n is: where f U The CPU frequency calculated for the drone is in cycles / second. The size of all received tasks calculated by drone m is expressed as: Among them, t hover is the hovering time of the UAV at each time point, H m For the IoT device set m that offloads tasks to drone m, let f U Indicates the CPU cycles required to calculate each bit; definition The energy consumption of UAV m in the calculation task in time slot t is as follows: Where ξ is the effective capacitance coefficient of the UAV; The method includes establishing a propulsion power model of a rotary-wing UAV that depends only on a speed factor. For a rotary-wing UAV, the propulsion power is expressed as: in, Body S F It represents the equivalent flat plate area of ​​the fuselage; From the above formula, we can see that the propulsion energy consumption of the rotorcraft at each time point consists of two parts: horizontal motion energy and hovering energy, which can be expressed as In the method, all drones move at a uniform speed, and the flight energy consumption of each drone in each time slot is equal; Furthermore, the method includes constructing a charging model for a drone, including the following process: ε m It is a vector indicating whether the energy of UAV m is updated, where ε m =1 means that UAV m updates its energy at time slot t, otherwise ε m = 0, and the drone returns to the grid above the target area before updating the energy after being fully charged, and continues to provide edge computing services for IoT users; Assuming that each type of application should place at least one drone hovering over the target area in each time slot, the following relationship exists: In the method, the mathematical modeling and description of the trajectory learner includes the following process: (11) Trajectory learner state and action: The environmental state of the trajectory learner consists of the current position of the UAV, the installed application, and the time slot index. At time slot t, the environmental state of the trajectory learner is expressed as: s T (t)=[{L m (t)} m∈M ,{w m (t)} m∈M ] Trajectory Learner T The set of actions includes four actions: moving forward, moving backward, moving left, and moving right; (12) Reward of trajectory learner: At time slot t, the immediate reward of the trajectory learner of drone m is determined by the following expression: Among them, the numerator represents the task scale calculated by drone m in time slot t, and the denominator represents the energy consumption of drone m in time slot t. Give the energy efficiency of drone m in time slot t in bit / J; (13) Trajectory learner strategy: The trajectory learner strategy in drone m Represents the mapping from state set to action set, using action In a given state s T The probability distribution under state s T The trajectory strategy of the trajectory learner of drone m is recorded as Each element is the drone m in state s T Select the role The probability of The joint trajectory strategy can be defined as π T is the trajectory learner in state s T Select joint action a T probability; In the method, applying mathematical modeling and description of the learner includes the following process: (21) Application learner state and action: The environment state of the application learner consists of the applications placed on each drone and the task requests received from all IoT devices. At time slot t, the environment state of the application learner can be expressed as: s T (t)=[{w m (t)} m∈M ,{θ m (t)} m∈M ] in, The action set of applying learner a is to select a task type from all types of tasks; (22) Rewards for application learners: Considering the differences in applications placed in drones, at time t, the immediate reward of the application learner in drone m is: Where N A represents the number of application types placed on all drones at time slot t, represents the tasks required by the IoT device before time slot t; (23) Strategy of applying learner: Strategy Application of learning machine Represents the mapping from state set to action set, using action In a given state s A The probability distribution in; for state s A The trajectory strategy of the trajectory learner of drone m is recorded as Each element is the drone m in state s A Select the role The probability of When the drone is at the charging station, the application learners select actions individually. Assuming that the actions selected by all application learners are independent under a given environmental state, the joint application strategy is defined as π A is the application of the learner in state s A Select Joint Action a A probability; In the method, the mathematical modeling and description of the energy learner includes the following process: (31) Energy learner state and action: The environmental state of the energy learner consists of the current remaining energy and position of each UAV. The environmental state of the energy learner at time slot t is expressed as: s E (t)=[{E m (t)} m∈M ,{ε m (t)} m∈M ] The set of actions of the energy learner includes two actions: updating energy and not updating energy; (32) Reward of energy learner: The reward of one step of energy learner is defined as: where ε m (t) = {0,1}, the energy learner places at least one drone hovering over the target area in each time slot for each type of application. The reward of the drone that does not update its energy is equal to 0; on the contrary, the reward of the energy learner is 1 when the drone updates its energy. If there is no drone on duty in the grid of the target area, the reward of each energy learner is -10; (33) Strategy of Energy Learner: Strategy Application of learning machine Represents the mapping from state set to action set, using action In a given state s E The probability distribution in; for state s E The trajectory strategy of the application learner of drone m is recorded as Each element is the drone m in state s E Select the role The probability of When the UAV is in the target area, the energy learner selects actions individually. Assuming that under the given current environment state, the selected actions of all energy learners are independent, the joint application strategy is defined as π E is the application of the learner in state s E Select Joint Action a E probability.

Images