Multi-unmanned aerial vehicle assisted mobile edge unloading method for open sea user terminal
Through the multi-drone-assisted mobile edge unloading method, DQN and DDPG algorithms are used to optimize task allocation, which solves the problems of high latency and energy consumption of long-sea user terminals, and improves computing efficiency and resource utilization.
Patent Information
- Application Number
- CN202410971079.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2024-11-12
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Yuanhai user terminals generate a large amount of delay and energy consumption when handling complex computing tasks, and have low computing efficiency and performance.
Using multi-drone-assisted mobile edge unloading method, by building a wireless communication model of remote sea user terminals and drones, using the MEC-DDPG framework of DQN and DDPG algorithms, optimize the task unloading strategy and allocating computing tasks to the drone and local Terminal, reduce overall computing delay and energy consumption.
It realizes the minimization of the average computing task processing delay of user terminals in the remote sea communication network, improves the computing efficiency and resource utilization rate, and improves the efficiency of the overall system.
Smart Images

Figure CN118945726A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of communication technology, and in particular relates to a multi-UAV assisted mobile edge unloading method for offshore user terminals. Background Art
[0002] In the offshore communication scenario, compared with satellite communication or traditional maritime communication methods based on marine radio, drones have the characteristics of rapid deployment and rapid mobility, which facilitates the rapid deployment and deployment of offshore drone communication networks in the offshore. In addition, compared with surface communication methods such as buoys based on the sea surface, drones have a certain height and can be free from the interference of transmission clutter signals on the sea surface. When drones communicate with offshore ships, they can avoid interference from obstacles, reduce interference in the communication process, stabilize the communication link, and improve the channel gain in offshore communications.
[0003] For the current far-sea aircraft carrier formation, various equipment in the aircraft carrier formation, such as electronic reconnaissance ships, conventional attack submarines and combat aircraft, can form a complete fleet regional far-sea communication network through regional networking. In this far-sea communication network, a large number of IoT devices such as sensors will generate a large number of complex computing tasks. These tasks will be calculated by the ship where the sensor is located, which will cause a lot of delay and energy consumption, and reduce the computing efficiency and effectiveness of the ship. Summary of the invention
[0004] The purpose of the present invention is to provide a multi-UAV assisted mobile edge offloading method for far sea user terminals, so as to solve the technical problems in the prior art that the far sea user terminals themselves generate a large amount of delay and energy consumption when processing complex computing tasks, and have low computing efficiency and effectiveness.
[0005] In order to solve the above technical problems, the present invention provides a multi-UAV assisted mobile edge unloading method for offshore user terminals, comprising the following steps:
[0006] Step S1: construct a mobile edge computing offloading model for multiple drones assisting multiple offshore user terminals;
[0007] The model consists of N UAVs and M offshore user terminals. I represents the set of offshore user terminals, and J represents the set of UAVs in the offshore communication network. The offshore user terminal is represented by UE i , i∈I={1…i…M}; UAV is j , j∈J={1…j…N};
[0008] Step S2: constructing a task model of a remote sea user terminal;
[0009] User terminal UE at time slot t i The task model is:
[0010]
[0011] where s i (t) represents the size of the current user task, e i (t) represents the number of CPU clock cycles required to process the task. Indicates the maximum tolerable delay of the task;
[0012] Step S3: constructing a wireless communication model between the offshore user terminal and the UAV;
[0013] Step S4: constructing a task offloading model for a remote sea user terminal, and performing computation offloading for the remote sea user terminal task by using a partial offloading method;
[0014] Step S5: construct a UAV movement model, calculate the distance between any two UAVs, and make the distance between any two UAVs greater than the minimum collision avoidance distance;
[0015] Step S6: construct an unloading calculation model;
[0016] Step S7: construct each offshore user terminal as an intelligent agent of a model-free Markov decision process;
[0017] Step S8: The MEC-DDPG offshore multi-UAV assisted mobile edge offloading framework combining the DQN algorithm and the DDPG algorithm is used to obtain the final edge offloading strategy.
[0018] Furthermore, in step S3, the position of the i-th offshore user terminal in the wireless communication model between the offshore user terminal and the UAV is expressed as The position coordinates of the jth UAV are expressed as
[0019] Among them, X i (t) represents the horizontal coordinate of the location of the offshore user terminal; i (t) represents the ordinate of the location of the offshore user terminal; j (t) represents the horizontal coordinate of the UAV position; Y j (t) represents the ordinate of the drone’s position; H represents the height of the drone from the sea surface;
[0020] UAV j With the offshore user terminal UE i Gain of line-of-sight communication link between It is expressed as:
[0021]
[0022] Where α0 represents the channel gain at a distance of d = 1m in the open sea environment, di,j (t) is represented by the Euclidean distance between the jth UAV and the i-th marine mobile terminal; ∥·∥ 2 represents the modulo 2 norm;
[0023] The wireless transmission rate between the UAV and the offshore user terminal is set to:
[0024]
[0025] Among them, P up represents the uplink signal transmission power of the offshore user terminal in the uplink link, σ 2 represents the noise power in the offshore environment, B represents the communication transmission bandwidth in the offshore communication environment, and P NLOS represents the transmission loss in the offshore environment, f i (t) indicates that at time slot t, the UAV j With maritime mobile terminal UE i A flag indicating whether the communication link is deteriorating.
[0026] Further, in step S4, the offshore user terminal calculates the task offloading ratio R i,j (t) Conduct UAV j Offloading of computing tasks;
[0027]
[0028] Among them, R i,j (t) represents the offloading ratio of the current task of the remote sea user terminal. At the tth time slot, the remote sea user terminal UE i The tasks are offloaded to UAVs j Unloading ratio on 1-R i,j (t) represents the proportion of the remaining tasks, and the corresponding calculation and processing of this part of the tasks are undertaken by the offshore user terminal itself.
[0029] Furthermore, in the unloading calculation model of step S6,
[0030] Ocean User Terminal UE i The local computation delay is:
[0031]
[0032] f UE represents the CPU frequency of the offshore user terminal used to process tasks in the tth time slot, e i (t) represents the remote sea user terminal UE i The number of CPU clock cycles required for the task;
[0033] Ocean User Terminal UEi The energy consumption of processing tasks is:
[0034]
[0035] where k m Indicates the energy consumption factor of the chip;
[0036] For offloading to UAV j The total delay of the part of the task that is offloaded is composed of the task transmission delay and latency of processing on the drone composition;
[0037]
[0038] f in the formula UAV Represents the CPU computing frequency of the edge computing server of the drone; the total delay of offloading to the drone It is expressed as:
[0039]
[0040] The energy consumed by offloading the computational task to the UAV at time slot t It can also be divided into two parts, one part is used for the transmission energy consumed by transmission A portion of the computing energy consumed is used to calculate Therefore, its power consumption is expressed as:
[0041]
[0042]
[0043]
[0044] The flight energy consumption of the drone in time slot t is E fly (t), the hovering flight energy consumption of the UAV is ignored here;
[0045]
[0046] Among them, M UAV represents the mass of the drone, v t represents the flight speed of the drone in the time slot, δ fly Indicates the flight time of the drone in the time slot.
[0047] Furthermore, the agent state variable s constructed in step S7 t It is expressed as:
[0048]
[0049]
[0050] in It represents the remaining energy of each UAV in the current time slot; It indicates the geographical location coordinates of the offshore user terminal in the current time slot; It represents the geographical coordinates of the drone at the current time slot; T i (t) represents the task status of the offshore user terminal in the current time slot; f i (t) represents the current status of the communication channel of the offshore user terminal;
[0051] The action taken by the agent is defined as:
[0052] a t ={k i,j (t),R i,j (t),β t ,v t}
[0053] Among them, k i,j (t) represents the unloading target UAV selected by the offshore user terminal in the current time slot; R i,j (t) represents the task offloading ratio selected for the offshore user terminal; β t and v t Indicates the flight angle and speed currently selected by the drone;
[0054] The reward function is:
[0055]
[0056] in, is the i-th offshore user terminal UE i Local computation latency; Total latency of offloading to the drone.
[0057] Further, in step S8, the MEC-DDPG deep policy gradient framework environment, the DQN unit and the DDPG unit; the DQN unit selects a discrete unloading target UAV according to the UAV position, the position of the distant sea user terminal and the communication link status; the DDPG unit calculates the continuous task unloading ratio, the UAV flight speed and angle according to the unloading target UAV, the user task size and the UAV power; the optimal unloading strategy is determined according to the task unloading ratio, the UAV flight speed and angle calculated by the DDPG unit.
[0058] Compared with the prior art, the present invention has the following beneficial technical effects:
[0059] 1) The present invention proposes a multi-UAV-assisted mobile edge offloading method for far sea user terminals, which combines UAVs with mobile edge computing to efficiently combine the high mobility advantages of UAVs and the high resource utilization of edge computing; it combines the task status of the far sea user terminal itself, the location information of surrounding UAVs, the energy consumption information of UAVs and the computing task offloading ratio to minimize the average computing task processing delay of user terminals in the entire network.
[0060] 2) The present invention adopts a partial unloading mode. According to the selected unloading ratio, part of the task is unloaded and transmitted to the drone for unloading calculation according to the unloading ratio, and the other part of the calculation task is left locally and calculated and processed by the offshore user terminal itself. The partial unloading mode helps to efficiently utilize the resources and capabilities of the drone and offshore user terminal when unloading tasks, and mobilizes and utilizes the idle computing power of the offshore user terminal itself and the computing power of the edge node, thereby improving the resource utilization efficiency of the overall system and improving the computing efficiency and effectiveness. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0062] Figure 1 This is a schematic diagram of the multi-UAV assisted mobile edge offloading model for offshore user terminals of the present invention.
[0063] Figure 2 Schematic diagram of the MEC-DDPG deep policy gradient framework of the present invention.
[0064] Figure 3 It is a structural diagram of the MEC-DDPG algorithm framework of the present invention.
[0065] Figure 4 Schematic diagram of the algorithm structure of the DQN unit of the present invention.
[0066] Figure 5 Schematic diagram of the algorithm structure of the DDPG unit of the present invention.
[0067] Figure 6 It is the convergence performance of MEC-DDPG under different learning rates in the method described in the present invention.
[0068] Figure 7 It is the delay performance under different unloading strategies in the method of the present invention.
[0069] Figure 8This is a network simulation scene diagram of the method described in the present invention.
[0070] Fig. 9 This is an end-to-end delay performance diagram of the network simulation of the method described in the present invention.
[0071] Fig.10 A network jitter performance diagram of network simulation of the method described in the present invention. DETAILED DESCRIPTION
[0072] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0073] The present invention proposes a multi-UAV assisted mobile edge offloading method for far sea user terminals, which takes into account the high mobility of multiple UAVs and the energy consumption limitations of UAVs, and jointly optimizes the selection of target UAVs for offloading users' computing tasks, the offloading ratio of computing tasks, the flight angle and speed of UAVs, thereby ultimately minimizing the average task offloading processing delay of far sea terminal users within the entire time period.
[0074] The present invention proposes a multi-UAV assisted mobile edge offloading method for offshore user terminals, comprising the following steps:
[0075] Step S1: Construct a mobile edge computing offloading model for multiple drones assisting multiple offshore user terminals.
[0076] Multi-UAV assisted mobile edge offloading model for offshore user terminals Figure 1 As shown in the figure, the entire network scenario is in a far sea environment far away from the land. The scenario mainly consists of N drones hovering in the air and M far sea user terminals distributed on the sea surface. I represents the set of far sea user terminals, and J represents the set of drones in the far sea communication network. The far sea user terminal is represented by UE i , i∈I={1…i…M}; UAV is j , j∈J={1…j…N}. Each node communicates in a wireless communication mode to form the entire offshore multi-UAV assisted mobile edge computing network system.
[0077] At the sea surface layer, there are multiple offshore user terminals on the sea surface, such as offshore data monitoring buoys, user handheld radios on offshore sailboats, and so on. These offshore user terminals have low computing and processing capabilities and will generate computing tasks of a certain size. For example, offshore buoy detection nodes will generate certain monitoring data analysis tasks. The offshore user terminal carries a communication module unit, which can transmit communication signals at a certain power for communication. In addition, the terminal is also equipped with a computing task generation module, which will generate complex computing tasks of a certain size. The computing task can be processed by the computing unit of the offshore user terminal itself, or it can be transmitted to the surrounding drones through the communication module unit for offloading processing.
[0078] In the air layer, multiple drones hovering for a long time are flying in the air at a height of H from the sea surface; the drones have their own power supply units, which are responsible for providing energy for drone computing offload and maintaining their own flight. These drones provide offload computing services for user terminals on the sea surface. The drones carry their own communication module units, which are responsible for maintaining the communication link between the entire communication network and the user terminals in the far sea. Each drone can establish a communication link with the user terminals on the sea surface layer as needed. Each drone also carries an edge computing server, which has a higher computing performance than the terminal itself, and is designed to provide users with high-performance and low-latency edge computing offload services.
[0079] Step S2: Constructing a high seas user terminal mission model.
[0080] At the beginning of each time slot, each offshore user terminal UE i A computing task of a certain size will be generated. This task is computationally intensive and mainly uses CPU resources for computing and processing to complete the task. Here, the user terminal UE is defined as i The task model is:
[0081]
[0082] where s i (t) represents the size of the current user task, e i (t) represents the number of CPU clock cycles required to process the task. Indicates the maximum tolerable delay of the task. All tasks are independent and unrelated, so the tasks can be split and calculated in a certain ratio. During the entire edge computing offloading cycle, the offshore user terminal in each time slot will make an offloading decision based on the local computing power of the user device, the location information of the drone, and the computing power of the edge server, and complete the task of the current time slot as soon as possible without exceeding the specified upper limit and in the shortest time.
[0083] Step S3: Construct a wireless communication model between the offshore user terminal and the UAV.
[0084] In the entire offshore communication network, all devices use wireless communication to forward communications and transmit tasks. Define the entire edge computing offloading cycle to be divided into K′ time slots δ of equal time, and define a time slot set K = {1…t…K′}. Assume that at the beginning of each time slot, the drone has a fixed position coordinate, which is convenient for users to perform wireless signal transmission and computing task offloading. For the offshore user terminal on the sea surface, the position of the i-th offshore user terminal can be expressed as Where I = {1…i…M}. There are N drones hovering at a height H above the sea surface, and this height remains constant. At this time, the position coordinates of the jth drone are Where J = {1…i…N}. The drone can fly in a certain direction and speed and maintain a constant altitude.
[0085] The communication link between the drone swarm and the far-sea user terminal always maintains the line-of-sight communication link unchanged, and the channel state between each drone and the far-sea user terminal remains unchanged in each time slot. When a drone communicates with a far-sea user terminal within the communication coverage area, the channel at this time is related to the environment in which the drone is located and the altitude of the drone. This drone channel can be regarded as a LoS (Line of Sight, LoS) channel model. The LoS model refers to a channel model in which a direct path exists. Because the drone is at a higher altitude than the far-sea user terminal, the signal can be sent directly from the drone antenna to the far-sea user terminal antenna. At this time, the channel quality of communication mainly depends on the UAV. j and offshore user terminals UE i The communication distance between the two. In a certain time slot, each UAV first hovers at its current position and then establishes a communication link with the user terminal. The user terminal can offload part of the computing tasks to the edge computing server of the UAV for computing and processing, and the remaining computing tasks will be executed locally. UAV j With the offshore user terminal UE i Gain of line-of-sight communication link between It can be expressed as:
[0086]
[0087] Where α0 represents the remote sea user terminal UE in the remote sea environment i UAV j Channel gain at distance d = 1m, d i,j (t) is represented as the Euclidean distance between the jth UAV and the i-th marine mobile terminal; ||·|| 2In consideration of the complex and changeable communication channel quality in the offshore scenario, the link between the UAV and the offshore user terminal is considered to be randomly deteriorated, and the wireless transmission rate between the UAV and the offshore user terminal is set to:
[0088]
[0089] Where B represents the communication transmission bandwidth in the offshore communication environment, P up represents the uplink signal transmission power of the offshore user terminal in the uplink link, σ 2 Represents the noise power in the offshore environment, P NLOS represents the transmission loss in the offshore environment, f i (t) indicates that at time slot t, the UAV j With the maritime mobile terminal UE i A flag indicating whether there is a deterioration of the communication link. A value of 0 indicates that the communication link is in good condition, and a value of 1 indicates that the communication link is not good, so as to simulate the situation where the link may be difficult to communicate in the offshore communication network.
[0090] Step S4: construct a high-seas user terminal task offloading model, and the high-seas user terminal task is offloaded by partial offloading.
[0091] In the offshore multi-drone assisted mobile edge computing offloading model, the entire offloading network consists of offshore user terminals and drones. The offshore user terminals can choose different offloading strategies to offload tasks or leave all tasks to be processed locally. In order to maximize the computing power of offshore user terminals and drones, a partial offloading mode is used for computing offloading.
[0092] At a certain time slot t, when the offshore user terminal UE i A computing task of a certain size is generated, and the user terminal can offload the computing task according to the ratio R i,j (t) Conduct UAV j Offload computing tasks.
[0093]
[0094] Where R i,j (t) represents the offloading ratio of the current task of the remote sea user terminal. At the tth time slot, the remote sea user terminal UE i The tasks are offloaded to UAVs j I represents the set of user terminals in the far sea, and J represents the set of drones in the far sea communication network. i,j (t)∈[0,1], and 1-R i,j(t) represents the proportion of the remaining tasks, and the corresponding calculation and processing of this part of the tasks are undertaken by the offshore user terminal itself.
[0095] Step S5: construct a UAV movement model, calculate the distance between any two UAVs, and make the distance between any two UAVs greater than the minimum collision avoidance distance.
[0096] In the tth time slot, the UAV j The three-dimensional coordinates of can be expressed as All drones fly at a fixed altitude to avoid the impact of sea waves and air obstacles. Specifically, each drone starts from a predetermined initial coordinate {X0, Y0, H} and flies in a certain direction, and the flight action of each drone is determined by the current speed v of the drone. t ∈[0,v max ] and angle β t ∈[0,2π], so at time t+1, UAV j The coordinates of can be expressed as {X j (t+1),Y j (t+1),H}:
[0097] X j (t+1)=X j (t)+v t δ fly cos(β t )
[0098] Y j (t+1)=Y j (t)+v t δ fly sin(β t )
[0099] Among them, δ fly Indicates the flight time of the drone in the time slot.
[0100] In addition, due to the limitations of wireless communication distance at sea and maintaining network topology, all drone positions are restricted to a certain range area. The constraints are expressed as:
[0101]
[0102]
[0103] Where T represents the total unloading period, R max Indicates the maximum flight range of the drone.
[0104] In time slot t, the distance between any two UAVs can be expressed as:
[0105]
[0106] And any two drones should maintain a minimum collision avoidance distance R min , for which:
[0107]
[0108] Step S6: construct an offloading calculation model.
[0109] There is no dependency between the computing tasks of each offshore user terminal, and they are all independent and unrelated tasks. In the tth time slot, because the edge computing system adopts a partial offloading method, R i,j (t) is the offshore user terminal UE i Offload tasks to UAVs j The unloading ratio of edge computing servers is obviously R i,j (t)∈[0,1], and 1-R i,j (t) represents the proportion of the remaining tasks, and the tasks of this part are handled by the remote user terminal itself. i The local computation delay is:
[0110]
[0111] f UE represents the CPU frequency of the offshore user terminal used to process tasks in the tth time slot, e i (t) represents the remote sea user terminal UE i The number of CPU clock cycles required for a task.
[0112] In addition, the offshore user terminal UE i The energy consumption for processing this part of the task is:
[0113]
[0114] where k m Indicates the energy consumption factor of the chip, k m It is related to the chip architecture of the computing unit of the offshore user terminal.
[0115] Secondly, for the unloading to the UAV j The total delay of the part of the task that is offloaded is composed of the task transmission delay and latency of processing on the drone composition.
[0116]
[0117] f in the formulaUAV It is represented by the CPU computing frequency of the edge computing server of the drone. So the total delay of offloading to the drone It can be expressed as:
[0118]
[0119] Similarly, the energy consumed by offloading the computing task to the drone at time slot t is It can also be divided into two parts, one part is used for the transmission energy consumed by transmission A portion of the computing energy consumed by calculation Therefore, its power consumption can be expressed as:
[0120]
[0121]
[0122]
[0123] In addition, the flight energy consumption of the UAV in time slot t is E fly (t), the hovering flight energy consumption of the UAV is ignored here.
[0124]
[0125] Among them, M UAV represents the mass of the drone, v t Indicates the flight speed of the drone in the time slot.
[0126] Step S7: construct each offshore user terminal as an intelligent agent of a model-free Markov decision process.
[0127] Based on the randomness of links and offloading tasks in the offshore multi-UAV mobile edge offloading network, the transition probability between different states is unknown. The present invention regards each offshore user terminal in the offshore communication network as an intelligent agent, and models the task offloading problem of each offshore user terminal as a model-free Markov decision process. In this scenario, the position of each drone is only related to the current position of the drone and the action taken at the current position, but not to its position and action at the previous moment.
[0128] The elements of the model-free Markov decision process are defined as follows:
[0129] State space S: In the offshore user terminal multi-UAV assisted mobile edge offloading structure, the entire state space is jointly determined by the task status of all offshore user terminals, the current power status of the UAV, the location coordinates of the offshore user terminals and the UAV, and the communication channel status. At time slot t, the state variable s of this processt It can be expressed as:
[0130]
[0131]
[0132] in It represents the remaining energy of each UAV in the current time slot. It represents the geographical location coordinates of the offshore user terminal in the current time slot. It represents the geographical coordinates of the drone at the current time slot. i (t) represents the task status of the offshore user terminal in the current time slot. i (t) represents the current status of the communication channel of the offshore user terminal.
[0133] Action space A: This invention combines the selection of unloading target drones and the ratio of unloading tasks for offshore user terminals to minimize the average computing task processing delay for users. It is necessary to select the flight direction and speed of the drone in each time slot. The agent also needs to select the unloading target drone in the current time slot, the ratio of computing tasks for offshore user terminals, and the UAV according to the current state information in the scene. j Flight angles, UAVs j The flight speed of , so the action taken by the agent in time slot t can be defined as:
[0134] a t ={k i,j (t),R i,j (t),β t ,v t}
[0135] Among them, k i,j (t) represents the unloading target UAV selected by the offshore user terminal in the current time slot. i,j (t) represents the task offloading ratio selected for the offshore user terminal. t and v t Indicates the flight angle and speed currently selected by the drone.
[0136] Reward function R: Minimizing the average user computing task processing delay is selected as the performance indicator. The ultimate goal of the deep reinforcement learning algorithm is to obtain the maximum reward feedback by taking the best strategy and action. And considering that the optimization goal is to obtain the lowest delay, in order to better achieve the feedback effect of the reward function, the reward function is defined here as:
[0137]
[0138] in, is the i-th offshore user terminal UE i Local computation latency; Total latency of offloading to the drone.
[0139] Step S8: The MEC-DDPG offshore multi-UAV assisted mobile edge offloading framework combining the DQN algorithm and the DDPG algorithm is used to obtain the final edge offloading strategy.
[0140] The computing task offloading ratio in the offloading strategy is a continuous action variable, and in the offshore communication multi-UAV assisted mobile edge computing scenario, the flight speed and angle of the UAV are also continuous action variables. Therefore, the reinforcement learning algorithm to solve the offloading problem must be able to process continuous action space. The present invention uses the DDPG algorithm to process continuous actions. In addition to the processing of continuous actions in the offloading strategy, there is also a discrete variable, namely the user's offloading object target UAV selection. The present invention uses the advantages of the DQN algorithm in discrete variables and combines DQN with DDPG to realize offshore multi-UAV assisted mobile edge offloading.
[0141] MEC-DDPG deep policy gradient framework Figure 2 As shown in the figure, it includes the environment, DQN unit and DDPG unit; the environment that the DQN unit and DDPG unit rely on is unified, but the state information required for the discrete unloading target drone selection and the continuous task unloading ratio, drone flight speed and angle determination is different. The DQN unit selects the discrete unloading target drone according to the drone position, the position of the offshore user terminal, and the communication link status; the DDPG unit calculates the continuous task unloading ratio, drone flight speed and angle according to the unloading target drone, the user task size and the drone power. The optimal unloading strategy is determined based on the task unloading ratio, drone flight speed and angle calculated by the DDPG unit.
[0142] For the optimal target drone determined by the DQN unit, the location information of multiple drones and multiple remote user terminals in the remote sea communication network environment needs to be included in the decision, and because the channel gain and communication path loss in the remote sea communication network are constantly changing, these state information also need to be taken into consideration when making decisions on the target drone. Therefore, the state that the DQN unit needs to input is:
[0143]
[0144] For the optimal unloading ratio and the flight angle and speed of the drone determined by the DDPG unit, the unloading target drone has been selected. At this time, it is necessary to obtain the specific information of the unloading target drone from the environment and read the mission information of the offshore user terminal. Therefore, the input state required by the DDPG unit is:
[0145]
[0146] The structure of the MEC-DDPG algorithm framework is as follows Figure 3 As shown in Figure 1, the entire MEC-DDPG algorithm framework consists of a DQN unit and a DDPG unit, which are responsible for different decision-making processes. The DQN unit obtains the input state of the DQN unit based on the user task information obtained from the environment. The estimation network of the DQN unit outputs actions according to the greedy strategy This action determines the target drone selection in the unloading strategy of the offshore user terminal. The DDPG unit is based on the DQN unit and takes the action made by the DQN unit. Get the corresponding state from the environment The Actor-Critic network in DDPG is used to obtain the flight speed and angle of the drone, as well as the offloading ratio of the computing task.
[0147] The algorithm structure diagram of the DQN unit is as follows Figure 4 As shown, it is responsible for processing discrete actions. At the beginning of each time slot, the DQN unit reads the state from the reinforcement learning environment Based on this state, the prediction network of the DQN unit selects an action according to the ε-greedy strategy ε is the greed factor of the DQN unit. The action is selected according to the probability 1-ε or the action with the maximum Q value in the current state according to the probability ε, that is:
[0148]
[0149] Among them, ω represents the strategy adopted, Indicates the current state target Q value. The DQN unit will choose an action with a large reward value, or try to explore an action that has not been selected. Then the action Input into the current environment to interact with the action and get the state of the next moment and rewards Combining all of the above, we can get a sample data set Then put this sample data group into the experience pool. When the amount of data collected in the experience pool reaches the set upper limit, a small part of the sample training data will be randomly extracted from the experience pool for re-training.
[0150] The training process will first be based on the current state action The corresponding Q value can be obtained Then the next moment's state Input into the neural network of the DQN unit, the neural network of the DQN unit can output the Q value obtained by different actions. Next, in order to get better results, DQN will try to choose the largest Q value. The target value of the target network is:
[0151]
[0152] Among them, γ is the discount factor, ω′ represents the strategy adopted, Indicates the maximum Q value of the next state. The discount factor exists to weigh the importance of future rewards and current rewards. In addition, the DQN unit also needs a loss function to correct the effect of training. The specific DQN loss function is as follows:
[0153]
[0154] The target Q value of the current state is corrected by the loss function, and the parameters of the optimization network are continuously updated. After continuous training, the relevant parameters of the optimized prediction network are copied and updated to the target network. After multiple iterative trainings, the parameters of the optimization neural network are continuously updated and improved in this process, which can achieve better training results.
[0155] The algorithm structure of the DDPG unit is as follows Figure 5 As shown, it is responsible for processing the continuous action space. The DDPG unit contains the prediction strategy network β π , prediction strategy network β Q , target strategy network β π′ , target policy network β Q 'Four deep neural networks, used for the Actor network and the Critic network respectively, output actions and corresponding Q values.
[0156] After the DQN unit makes an action, the DDPG unit receives the output action of the DQN and read the status from the environment Actor network based on the current state Output Action The DDPG unit uses exploration noise to balance the algorithm's decision between exploration of new actions and exploitation of known actions.
[0157]
[0158] in, represents noise. Input into the environment and get the next state through interaction with the environment and feedback reward value The Actor network will then This sample data is stored in the experience pool. When the amount of data in the experience pool reaches the upper limit, the Critic network extracts some samples from it for training. The target value of the Critic target network is:
[0159]
[0160] The Q value of the next state of the policy network under the current policy is represented by the action and state, and the mean square error between the target value and the predicted value of the current action is used as the loss function:
[0161]
[0162] It represents the target Q value of the target network when taking the current action and state. The loss function is minimized here, and the gradient descent algorithm is used to update the parameters of the Critic network and the Actor network:
[0163]
[0164]
[0165] α in the formula Q is the learning rate hyperparameter of the Critic network, α π is the learning rate hyperparameter of the Actor network, It is the DDPG gradient update formula. After a long period of round training, the parameters of the Actor network and the Critic network are continuously iteratively updated to obtain the final optimal unloading strategy.
[0166] The algorithm flow of MEC-DDPG-based offshore multi-UAV assisted edge computing is shown in Table 1:
[0167] Table 1 Offloading algorithm of multi-UAV assisted mobile edge computing based on MEC-DDPG in the open sea
[0168]
[0169]
[0170] For the DDPG unit in the MEC-DDPG algorithm, both the Actor network and the Critic network contain two fully connected hidden layers with 256 and 128 neurons, respectively. The learning rates of the Actor network and the Critic network are set to 0.001 and 0.002, respectively. The Adam Optimizer is used to update the Actor and Critic networks. In addition, the soft update coefficient is set to τ = 0.01. For the DQN unit, the prediction network and the target network contain two fully connected hidden layers with 100 and 20 neurons, respectively. The learning rate is set to 0.1 and the greed factor is ε = 0.99. The update interval of the target network in the DQN unit is 200. The maximum number of training rounds is 1000 and the experience pool size is 10000.
[0171] Figure 6 The figure shows the convergence performance of the MEC-DDPG algorithm under different settings of the learning rate hyperparameter. The learning rate of the Actor network and the Critic network in the DDPG unit has an important influence on the convergence performance of the MEC-DDPG algorithm. As can be seen from the figure, when the learning rate of the Actor network and the Critic network is set to 0.0001 and 0.0002, the MEC-DDPG algorithm is difficult to converge. This is because when the learning rate is low, the learning efficiency of the deep neural network in the algorithm is slow, and more iterations are required for the algorithm to converge. When the learning rate of the Actor network and the Critic network is set to 0.1 and 0.2, the algorithm quickly converges to a local optimal solution. This is because a larger learning rate makes the update steps of the Actor network and the Critic network larger, and the optimal action cannot be taken, and it is easy to fall into an erroneous learning cycle. Therefore, the learning rate is set to the optimal 0.001 and 0.002.
[0172] Figure 7The figure shows the latency performance comparison under different offloading strategies. As can be seen from the figure, the UE_LOCAL strategy, in which all computing tasks are left locally on the mobile terminal user, shows a higher average task processing latency. The offloading strategy based on the deep Q network DQN converges slowly because the DQN unit can only process discrete variables. The latency performance of the computing offloading strategy based on the DDPG algorithm is not optimal. This is because although the DDPG algorithm has a great advantage in solving continuous variables such as offloading ratio, this scenario involves multiple drones. The DDPG algorithm cannot well select the target drone for user offloading, so it cannot achieve the optimal latency performance when making offloading decisions. The computing offloading strategy based on the MEC-DDPG algorithm finally shows the lowest latency in the current scenario. This is because the MEC-DDPG algorithm used in this strategy can efficiently make decisions on offloading target drones based on drones with different locations and communication conditions in the scene, and then use the policy gradient DDPG unit to select the appropriate offloading ratio, drone flight angle and speed, and can efficiently use different drones in the entire scene to assist edge computing. Compared with other offloading strategies, it has better latency performance.
[0173] Figure 8 The figure is a simulation scenario diagram, showing a network scenario diagram based on the OMNET platform. The proposed algorithm for multi-UAV-assisted mobile edge offloading of offshore user terminals is simulated and implemented at the network simulation level. Adhoc network is used for networking, and the network scenario of multiple UAVs and multiple ships is simulated.
[0174] Fig. 9 and Fig.10 The end-to-end delay performance and network jitter of this network scenario are shown. In the OMNET++ simulation software, the end-to-end delay in the network is calculated by the simulation statistics meanBitLifeTimePerPacket of OMNET++. Fig. 9 As shown in the figure, the end-to-end delay in the offshore multi-UAV assisted mobile edge offloading network is basically maintained at about 565ms during the entire message transmission process. And this data is always stable and consistent. Even with the continuous movement of nodes in the network and the dynamic changes in the network topology, the end-to-end delay always maintains good performance, demonstrating the stability of the proposed network structure and being able to meet the offloading network requirements of user terminals in the offshore aircraft carrier formation communication network. In addition, Fig.10It can be seen that the network jitter value of the entire network during the simulation process always maintained an extremely low jitter value. Except for a slight jitter in the initial operation stage of the network, the network jitter value was maintained below 0.001 for the rest of the time, indicating that the network structure not only has a low end-to-end latency performance, but its low latency performance can be stably maintained throughout the entire unloading process, reflecting the low latency performance and network stability of the mobile edge unloading network of the far-sea user terminal assisted by multiple drones in the far-sea user terminal.
[0175] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.
Claims
1. A multi-UAV assisted mobile edge offloading method for offshore user terminals, characterized in that: The steps include: Step S1: construct a mobile edge computing offloading model for multiple drones assisting multiple offshore user terminals; The model consists of N UAVs and M offshore user terminals. I represents the set of offshore user terminals, and J represents the set of UAVs in the offshore communication network. The offshore user terminal is represented by UE i , i∈I={1…i…M}; the UAV is denoted as UAVj, j∈J={1…j…N}; Step S2: constructing a task model of a remote sea user terminal; User terminal UE at time slot t i The task model is: where s i (t) represents the size of the current user task, e i (t) represents the number of CPU clock cycles required to process the task. Indicates the maximum tolerable delay of the task; Step S3: constructing a wireless communication model between the offshore user terminal and the UAV; Step S4: constructing a task offloading model for a remote sea user terminal, and performing computation offloading for the remote sea user terminal task by using a partial offloading method; Step S5: construct a UAV movement model, calculate the distance between any two UAVs, and make the distance between any two UAVs greater than the minimum collision avoidance distance; Step S6: construct an unloading calculation model; Step S7: construct each offshore user terminal as an intelligent agent of a model-free Markov decision process; Step S8: The MEC-DDPG offshore multi-UAV assisted mobile edge offloading framework combining the DQN algorithm and the DDPG algorithm is used to obtain the final edge offloading strategy.
2. The multi-UAV assisted mobile edge offloading method for offshore user terminals according to claim 1 is characterized in that: In step S3, the position of the i-th offshore user terminal in the wireless communication model between the offshore user terminal and the UAV is expressed as The position coordinates of the jth UAV are expressed as Among them, X i (t) represents the horizontal coordinate of the location of the offshore user terminal; Y i (t) represents the ordinate of the location of the offshore user terminal; X j (t) represents the horizontal coordinate of the UAV position; Y j (t) represents the ordinate of the drone’s position; H represents the altitude of the drone above the sea surface; UAV j With the offshore user terminal UE i Gain of line-of-sight communication link between It is expressed as: Where α0 represents the remote sea user terminal UE in the remote sea environment i UAV j Channel gain at distance d = 1m, d i,j (t) is represented as the Euclidean distance between the jth UAV and the i-th marine mobile terminal; ||·|| 2 represents the modulo 2 norm; The wireless transmission rate between the UAV and the offshore user terminal is set to: Where B represents the communication transmission bandwidth in the offshore communication environment, P up represents the uplink signal transmission power of the offshore user terminal in the uplink link, σ 2 Represents the noise power in the offshore environment, P NLOS represents the transmission loss in the offshore environment, f i (t) represents the time slot t between the UAVj and the marine mobile terminal UE i A flag indicating whether the communication link is deteriorating.
3. The multi-UAV assisted mobile edge offloading method for offshore user terminals according to claim 2 is characterized in that: In step S4, the offshore user terminal calculates the task offloading ratio R i,j (t) Conduct UAV j Offloading of computing tasks; Among them, R i,j (t) represents the offloading ratio of the current task of the remote sea user terminal. At the tth time slot, the remote sea user terminal UE i The tasks are offloaded to UAVs j Unloading ratio on 1-R i,j (t) represents the proportion of the remaining tasks, and the corresponding calculation and processing of this part of the tasks are undertaken by the offshore user terminal itself.
4. The multi-UAV assisted mobile edge offloading method for offshore user terminals according to claim 3 is characterized in that: In the unloading calculation model of step S6, Ocean User Terminal UE i The local computation delay is: f UE represents the CPU frequency of the offshore user terminal used to process tasks in the tth time slot, e i (t) represents the remote sea user terminal UE i The number of CPU clock cycles required for the task; Ocean User Terminal UE i The energy consumption of processing tasks is: where k m Indicates the energy consumption factor of the chip; For offloading to UAV j The total delay of the part of the task that is offloaded is composed of the task transmission delay and latency of processing on the drone composition; f in the formula UAV Represents the CPU computing frequency of the edge computing server of the drone; the total delay of offloading to the drone It is expressed as: The energy consumed by offloading the computational task to the UAV at time slot t It can also be divided into two parts, one part is used for the transmission energy consumed by transmission A portion of the computing energy consumed by calculation Therefore, its power consumption is expressed as: The flight energy consumption of the drone in time slot t is E fly (t), the hovering flight energy consumption of the UAV is ignored here; Among them, M UAV represents the mass of the drone, v t represents the flight speed of the drone in the time slot, δ fly Indicates the flight time of the drone in the time slot.
5. The multi-UAV assisted mobile edge offloading method for offshore user terminals according to claim 4 is characterized in that: The agent state variable s constructed in step S7 t It is expressed as: in It represents the remaining energy of each UAV in the current time slot; It represents the geographical location coordinates of the offshore user terminal in the current time slot; It represents the geographical coordinates of the drone at the current time slot; T i (t) represents the task status of the offshore user terminal in the current time slot; f i (t) represents the current status of the communication channel of the offshore user terminal; The action taken by the agent is defined as: a t ={k i,j (t),R i,j (t),β t ,v t } Among them, k i,j (t) represents the unloading target UAV selected by the offshore user terminal in the current time slot; R i,j (t) represents the task offloading ratio selected for the offshore user terminal; β t and v t Indicates the flight angle and speed currently selected by the drone; The reward function is: in, is the i-th offshore user terminal UE i Local computation latency; Total latency offloaded to the drone.
6. The multi-UAV assisted mobile edge offloading method for offshore user terminals according to claim 5 is characterized in that: In step S8, the offshore multi-UAV assisted mobile edge offloading framework includes an environment, a DQN unit and a DDPG unit; the DQN unit selects discrete offloading target UAVs according to the UAV position, the offshore user terminal position and the communication link status; the DDPG unit calculates the continuous task offloading ratio, UAV flight speed and angle according to the offloading target UAV, the user task size and the UAV power; the optimal offloading strategy is determined according to the task offloading ratio, UAV flight speed and angle calculated by the DDPG unit.
Citation Information
Patent Citations
Online trajectory and resource optimization method for multi-unmanned aerial vehicle auxiliary edge calculation
CN116321181A
MEC communication energy consumption optimization method based on unmanned aerial vehicle and backscattering assistance
CN116684949A
Distributed task unloading method for unmanned aerial vehicle assisted mobile edge computing
CN117440341A
Unmanned aerial vehicle task unloading optimization method for power grid inspection
CN117580105A
Task unloading method for multi-UAV-RIS auxiliary mobile edge computing
CN117715117A
Cited By
Underwater computing task unloading method based on improved depth deterministic strategy gradient
CN119255300A
