Multi-unmanned aerial vehicle assisted mobile edge computing unloading method in industrial internet

Abstract

The invention provides a multi-unmanned aerial vehicle auxiliary mobile edge computing unloading method in the industrial internet, and relates to the technical field of unmanned aerial vehicle auxiliary mobile edge computing. The method comprises the following steps: firstly, acquiring network state information, mobile intelligent user state information and unmanned aerial vehicle state information of each ground unit, and further constructing a digital twin model of each ground unit; then, calculating time delay and energy consumption of a mobile intelligent user processing calculation task and an unmanned aerial vehicle processing calculation task according to a digital twin model constructed by the acquired information, further introducing an optimization target, and comprehensively determining flight information of the unmanned aerial vehicle and calculating a task unloading proportion by combining various factors; and finally, modeling an optimization problem in an unmanned aerial vehicle assisted mobile intelligent user edge calculation scene into a Markov decision process, and further optimizing a reward function through a federated learning distributed training TD3 algorithm to reduce the calculation cost.

Description

Technical Field

[0001] The present invention relates to the field of drone-assisted mobile edge computing technology, and in particular to a multi-drone-assisted mobile edge computing offloading method in an industrial Internet. Background Art

[0002] Driven by the rapid development of the Internet of Things and fifth-generation technologies, more and more applications are being installed on mobile smart devices, such as face recognition, autopilot, and virtual meetings. Most of these applications are delay-sensitive, real-time, and energy-intensive. However, smart mobile devices cannot solve every computing task due to their own energy and computing power constraints. Once there are a large number of mobile smart devices that offload processing tasks to central cloud servers, the risk of computing delays and network congestion increases. To address such issues, mobile edge computing is a promising technology that places computing and storage resources at the edge of the network to provide low-latency computing services for mobile smart mobile devices. Thanks to mobile edge computing, the quality of service can be improved and network performance can be enhanced.

[0003] Considering the fixed edge computing servers, most of the existing research focuses on (1) (users) selecting appropriate target servers for offloading tasks and (2) (servers) reducing the total energy consumption of task processing in the system. In addition, the deployment location of edge computing servers remains a key challenge that severely restricts the development of mobile edge computing. Due to the mobility of users, the link between users and edge computing servers may be disconnected, resulting in unstable services. A potential solution is to combine mobile edge computing with drone platforms. Due to the rapid development of drone technology, drones have been widely used in many fields such as public services, disaster relief, and traffic maintenance. Drone aerial base stations with computing capabilities can quickly approach mobile intelligent users to provide additional computing resources. Since the computing resources provided by a single drone are limited, multiple drones can be deployed in hot spots to improve the performance of the mobile edge computing system and the service quality of mobile intelligent users. At present, there are many traditional optimization schemes to solve multi-drone assisted computing offloading. However, due to the high dynamics of drones and mobile intelligent users, traditional optimization algorithms cannot always maintain adaptability to environmental changes, which is not conducive to establishing optimal models. At the same time, considering the growing network channel state, renewable energy and many other system parameters, the amount of computation increases exponentially.

[0004] Benefiting from the great breakthroughs and progress of artificial intelligence, deep reinforcement learning has been widely used in drone-assisted computing offloading systems to learn optimal offloading strategies in dynamic and complex scenarios. By optimizing the deployment of drones, the trajectory of drones, the allocation of computing tasks, and the management of communication resources, efforts are made to reduce the energy consumption of task execution in mobile edge computing systems, reduce the task execution delay of smart mobile users, and maximize the number of computing tasks. Considering the limited computing resources and energy of drones, some researchers aim to jointly optimize energy consumption and execution delay. However, in the above works, the offloading problem is solved by training and deploying a centralized deep reinforcement learning scheme, that is, the server for training the deep reinforcement learning model is deployed in a single location. The data is stored in a single location, which faces data privacy and security issues. Therefore, once a single point of failure occurs, the deep reinforcement learning scheme requires a lot of energy and time to relearn the model. Federated learning is a distributed machine learning technology in which each local server collects data to train a local model and only transmits the updated deep reinforcement learning model weights to the central cloud server. Each local server can share the global deep reinforcement learning model at the same time to ensure data security and privacy. By combining the advantages of federated learning and deep reinforcement learning, a secure and robust multi-drone-assisted MEC framework can be obtained to provide computing services.

[0005] In recent years, the emerging digital twin has become a promising solution in mobile edge computing systems. Digital twin is an evolving intelligent system that can build a virtual replica of a physical mobile edge computing system. Digital twin models the basic rules of physical elements, entities, communications, and data computation processing. At the same time, the digital twin model is updated in the local server by monitoring the state changes of the real mobile edge computing system. Therefore, a digital twin model can be established to simulate the physical system, each local deep reinforcement learning model can be trained from the data of each local digital twin model, and the deep reinforcement learning model parameters can be shared to the cloud server. In addition, the drone does not need to be a server for training the deep reinforcement learning model. Since the model has the data change process of the real object and the basic rules of real network communication, the model-free deep reinforcement learning algorithm and the model-based deep reinforcement learning theory can be effectively combined through the digital twin model. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a multi-UAV assisted mobile edge computing offloading method in the industrial Internet in response to the deficiencies of the above-mentioned prior art, minimize the computational cost of the weighted sum of task completion delay and energy consumption, and ensure the service quality requirements of mobile intelligent users.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is: a multi-drone assisted mobile edge computing offloading method in the industrial Internet, comprising the following steps:

[0008] Step 1: Obtain the network status information, mobile intelligent user status information and drone status information of each ground unit; the ground server of each ground unit builds a digital twin model of the unit based on this information, where each ground unit contains K mobile intelligent users, and the total time slot for the drone to provide computing offload for the ground unit is T;

[0009] The ground server in each ground unit builds a digital twin model of the unit by acquiring the network, mobile intelligent users, and drone status information in the ground unit through sensors;

[0010] Step 2: The digital twin model constructed based on the acquired information calculates the latency and energy consumption of the computing tasks processed by the mobile intelligent user and the drone, and then introduces the optimization goal, combining multiple factors to comprehensively determine the flight information of the drone and the computing task offloading ratio;

[0011] The local computing delay of the mobile intelligent user processing computing tasks in each ground unit is calculated as shown in the following formula:

[0012]

[0013] in, The local computational latency of processing computational tasks for mobile intelligent user k, J k (t) is the number of CPU cycles required for the computation task, D k (t) is the proportion of computing tasks offloaded to the UAV, f k (t) is the CPU frequency of smart mobile user k, and t represents time;

[0014] Calculate the transmission delay of the task to the drone as shown in the following formula:

[0015]

[0016] in, The transmission delay of the computing task of mobile intelligent user k to the UAV is is the data transmission rate between the UAV and the mobile user, S k are the CPU cycles and byte conversion constants of smart mobile user k;

[0017] When the drone performs a computing task, the processing delay on the drone is as shown in the following formula:

[0018]

[0019] in, is the processing delay of the computing task of mobile intelligent user k on the drone, f uav (t) is the CPU frequency of the drone server;

[0020] The total time required to complete the computation task in time slot t is as follows:

[0021]

[0022] Among them, T sum (t) is the total time required to complete the computation task in time slot t;

[0023] The energy consumption of smart mobile user k performing computing tasks is shown in the following formula:

[0024]

[0025] in, The energy consumption of performing computing tasks for smart mobile user k, is the effective conversion coefficient of smart mobile user k, f k represents the computing power of smart mobile user k;

[0026] When the intelligent mobile user k offloads the computing task to the drone, the transmission energy at time slot t is as shown in the following formula:

[0027]

[0028] in, The transmission energy when the intelligent mobile user k offloads the computing task to the drone, is the transmission power of the smart mobile user offloading computing tasks;

[0029] The energy consumed by the drone to perform computing tasks is shown in the following formula:

[0030]

[0031] in, The energy consumed by the drone to perform computing tasks, is the effective conversion coefficient of the UAV, f uav Indicates the computing power of the drone;

[0032] At each time slot t, the photovoltaic module equipped by the drone provides energy to the drone battery as shown in the following formula:

[0033] E renewable (t) = γ u ψ

[0034] Among them, E renewable(t) is the energy provided by the photovoltaic module to the drone battery; u represents the utilization constant of the collected energy, ψ conforms to the normal distribution;

[0035] Therefore, the remaining battery energy of the drone in time slot t+1 is as follows:

[0036]

[0037] Among them, E fly (t) is the energy consumed by the UAV during flight in time slot t, E remain (t) represents the remaining energy of the UAV at time slot t, Indicates the maximum capacity of the battery carried by the drone;

[0038] The computational cost Ω of the mobile intelligent user processing computing tasks and the drone processing computing tasks is shown in the following formula:

[0039]

[0040] Among them, α k (t) represents the connection coefficient between the drone and the mobile intelligent user in time slot t (1 when connected, 0 when not connected), η and represents the weighted index, which represents the weights of delay and energy consumption respectively;

[0041] Therefore, the optimization problem in the UAV-assisted mobile intelligent user edge computing scenario is expressed as follows:

[0042]

[0043]

[0044]

[0045]

[0046]

[0047]

[0048] in, and is the position of the smart mobile device and the drone at time slot t, and They represent the horizontal and vertical coordinate values ​​of the mobile intelligent user k in the ground unit, and They represent the horizontal and vertical coordinates of the UAV in the ground unit, L and W represent the length and width of the ground unit, respectively. Formula (f) represents the energy consumption constraint of the UAV. represents the energy consumed by the drone to perform computing tasks, E fly (t) represents the energy consumed by the UAV during flight;

[0049] Step 3: Model the optimization problem in the UAV-assisted mobile intelligent user edge computing scenario as a Markov decision process, and further optimize the reward function to reduce the computational cost through the federated learning distributed training double-delay deep deterministic policy gradient algorithm; Model the optimization problem in the UAV-assisted mobile intelligent user edge computing scenario as a Markov decision process, specifically:

[0050] The state space at time slot t is defined as:

[0051] S={s k (t),s uav (t),β k (t)}

[0052] Among them, s k (t) is the computing task information of mobile intelligent user k in time slot t, including the current computing task size J k (t), location information pos of mobile user k k (t) and the maximum tolerable delay of the computation task s uav (t) is the state information of the drone at time slot t, including the drone position information pos uav (t), the remaining energy of the drone E remain And the remaining computing task size G remain β k (t) Express network congestion status;

[0053] The action space at time slot t is defined as:

[0054] a t ={M uav (t),D k (t)}

[0055] Among them, M uav (t) is the UAV movement information, including the UAV flight angle θ(t) and flight speed v(t); D k (t) indicates whether the UAV and mobile intelligent user k are selected to provide computing resources in time slot t;

[0056] The reward function to be optimized is expressed as:

[0057] r(t)=-[η t T sum (t)+η e E sum (t)]

[0058] Among them, r(t) represents the reward function, η t , η e are the proportional coefficients of the total time and energy consumption required for the computation task, which represent the weights of delay and energy consumption, respectively;

[0059] By defining the reward function as a negative value, the dual-delay deep deterministic policy gradient algorithm model can be better trained; in addition, the dual-delay deep deterministic policy gradient algorithm is distributedly trained using horizontal federated learning, specifically: first, a global dual-delay deep deterministic policy gradient algorithm model is constructed in the central cloud server, and a local dual-delay deep deterministic policy gradient algorithm model is constructed and trained in each ground unit; secondly, the local model in each ground unit is aggregated to the global model at a certain learning rate through federated learning; finally, the global model is downloaded and each local model parameter is updated.

[0060] The beneficial effect of adopting the above technical solution is that the multi-UAV assisted mobile edge computing offloading method in the industrial Internet provided by the present invention is a model-free deep reinforcement learning offloading solution based on a double-delay deep deterministic policy gradient algorithm, which effectively reduces the system computing cost.

[0061] (1) Considering the dynamic network environment and renewable energy in the multi-UAV assisted mobile edge computing system, the computing resource allocation and operation trajectory of the UAVs are jointly optimized. In addition, the computation offloading process is modeled as a Markov decision process, and an offloading optimization problem with minimal computational cost is proposed.

[0062] (2) An advanced offloading method based on federated learning and double-delayed deep deterministic policy gradient algorithm is designed. The double-delayed deterministic policy gradient algorithm takes advantage of the delayed update model parameters to solve the problem of overestimation of reward value. In addition, federated learning can effectively ensure the security of data privacy. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A scene diagram of a multi-drone-assisted mobile edge computing system provided in an embodiment of the present invention;

[0064] Figure 2 A flowchart of a model-free deep reinforcement learning offloading method based on a double-delayed deep deterministic policy gradient algorithm for solving a multi-UAV-assisted mobile edge computing scenario in the industrial Internet is provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0066] This embodiment is as follows Figure 1 Taking the multi-drone assisted mobile edge computing system scenario shown as an example, the multi-drone assisted mobile edge computing unloading method in the industrial Internet of the present invention is used to determine the edge computing unloading solution.

[0067] In this embodiment, a multi-drone assisted mobile edge computing offloading method in the industrial Internet is provided. Figure 2 As shown, the following steps are included:

[0068] Step 1: Limit the ground area within a certain range to ground units. Each ground unit contains a certain number of mobile intelligent users. The deployed drones are restricted to low-speed flight within this area. Obtain the network status information, mobile intelligent user status information and drone status information of each ground unit. The ground server of each ground unit builds a digital twin model of the unit based on this information. Each ground unit contains K mobile intelligent users, and the total time slot for the drone to provide computing offload for the ground unit is T.

[0069] The ground server in each ground unit builds a digital twin model of the unit by acquiring the network, mobile intelligent users, and drone status information in the ground unit through sensors;

[0070] Digital twins model the basic rules of physical elements, entities, communications, and data computing processing. At the same time, by monitoring the state changes of the real mobile edge computing system, the digital twin model is updated in the local server. A digital twin model can be established to simulate the physical system, each local deep reinforcement learning model can be trained from the data of each local twin model, and the deep reinforcement learning model parameters can be shared with the cloud server.

[0071] Step 2: The digital twin model constructed based on the acquired information calculates the latency and energy consumption of the computing tasks processed by the mobile intelligent user and the drone, and then introduces the optimization goal, combining multiple factors to comprehensively determine the flight information of the drone and the computing task offloading ratio;

[0072] The local computing delay of the mobile intelligent user processing computing tasks in each ground unit is calculated as shown in the following formula:

[0073]

[0074] in, The local computational latency of processing computational tasks for mobile intelligent user k, J k (t) is the number of CPU cycles required for the computation task, D k (t) is the proportion of computing tasks offloaded to the UAV, f k (t) is the CPU frequency of smart mobile user k, and t represents time;

[0075] Calculate the transmission delay of the task to the drone as shown in the following formula:

[0076]

[0077] in, The transmission delay of the computing task of mobile intelligent user k to the UAV is is the data transmission rate between the UAV and the mobile user, S k are the CPU cycles and byte conversion constants of smart mobile user k;

[0078] When the drone performs a computing task, the processing delay on the drone is as shown in the following formula:

[0079]

[0080] in, is the processing delay of the computing task of mobile intelligent user k on the drone, f uav (t) is the CPU frequency of the drone server;

[0081] The total time required to complete the computation task in time slot t is as follows:

[0082]

[0083] Among them, T sum (t) is the total time required to complete the computation task in time slot t;

[0084] The energy consumption of smart mobile user k performing computing tasks is shown in the following formula:

[0085]

[0086] in, The energy consumption of performing computing tasks for smart mobile user k, is the effective conversion coefficient of smart mobile user k, f k represents the computing power of smart mobile user k;

[0087] When the intelligent mobile user k offloads the computing task to the drone, the transmission energy at time slot t is as shown in the following formula:

[0088]

[0089] in, The transmission energy when the intelligent mobile user k offloads the computing task to the drone, is the transmission power of the smart mobile user offloading computing tasks;

[0090] The energy consumed by the drone to perform computing tasks is shown in the following formula:

[0091]

[0092] in, The energy consumed by the drone to perform computing tasks, is the effective conversion coefficient of the UAV, f uav Indicates the computing power of the drone;

[0093] In addition, the drone is equipped with photovoltaic panels that can absorb renewable energy to provide power to the battery. We use an energy queue model E remain (t) is used to represent and calculate the remaining power of the drone; specifically, at each time slot t, the photovoltaic module equipped by the drone provides energy to the drone battery, as shown in the following formula:

[0094] E renewable (t) = γ u ψ

[0095] Among them, E renewable (t) is the energy provided by the photovoltaic module to the drone battery; u represents the utilization constant of the collected energy, ψ conforms to the normal distribution;

[0096] Therefore, the remaining battery energy of the drone in time slot t+1 is as follows:

[0097]

[0098] Among them, E fly (t) is the energy consumed by the UAV during flight in time slot t, E remain (t) represents the remaining energy of the UAV at time slot t, Indicates the maximum capacity of the battery carried by the drone;

[0099] The computational cost Ω of the mobile intelligent user processing computing tasks and the drone processing computing tasks is shown in the following formula:

[0100]

[0101] Among them, α k (t) represents the connection coefficient between the drone and the mobile intelligent user in time slot t (1 when connected, 0 when not connected), η and represents the weighted index (the weighted coefficient after the standard normalization of delay and energy consumption), which represents the weight of delay and energy consumption respectively;

[0102] Therefore, the optimization problem in the UAV-assisted mobile intelligent user edge computing scenario is expressed as follows:

[0103]

[0104]

[0105]

[0106]

[0107]

[0108]

[0109] in, and is the position of the smart mobile device and the drone at time slot t, and They represent the horizontal and vertical coordinate values ​​of the mobile intelligent user k in the ground unit, and They represent the horizontal and vertical coordinates of the UAV in the ground unit, L and W represent the length and width of the ground unit, respectively. Formula (f) represents the energy consumption constraint of the UAV. represents the energy consumed by the drone to perform computing tasks, E fly (t) represents the energy consumed by the UAV during flight;

[0110] The above formulas (b) and (c) represent the offloading association constraints of mobile smart users, formulas (d) and (e) represent the motion range constraints of drones and smart mobile devices, and formula (f) represents the energy consumption constraints of drones.

[0111] Step 3: Model the optimization problem in the UAV-assisted mobile intelligent user edge computing scenario as a Markov decision process, and further optimize the reward function to reduce the computational cost through the federated learning distributed training double-delay deep deterministic policy gradient algorithm, such as Figure 2 As shown;

[0112] The optimization problem in the UAV-assisted mobile intelligent user edge computing scenario is modeled as a Markov decision process, specifically:

[0113] The state space at time slot t is defined as:

[0114] S={s k (t),s uav (t),β k (t)}

[0115] Among them, s k (t) is the computing task information of mobile intelligent user k in time slot t, including the current computing task size J k(t), location information pos of mobile user k k (t) and the maximum tolerable delay of the computation task s uav (t) is the state information of the drone at time slot t, including the drone position information pos uav (t), the remaining energy of the drone E remain And the remaining computing task size G remain β k (t) Express network congestion status;

[0116] The action space at time slot t is defined as:

[0117] a t ={M uav (t),D k (t)}

[0118] Among them, M uav (t) is the UAV movement information, including the UAV flight angle θ(t) and flight speed v(t); D k (t) indicates whether the UAV and mobile intelligent user k are selected to provide computing resources in time slot t;

[0119] The reward function to be optimized is expressed as:

[0120] r(t)=-[η t T sum (t)+η e E sum (t)]

[0121] Among them, r(t) represents the reward function, η t , η e are the proportional coefficients of the total time and energy consumption required for the computation task, which represent the weights of delay and energy consumption, respectively;

[0122] By defining the reward function as a negative value, the dual-delay deep deterministic policy gradient algorithm model can be better trained; in addition, horizontal federated learning is used to distribute the training of the dual-delay deep deterministic policy gradient algorithm; first, a global dual-delay deep deterministic policy gradient algorithm model is built in the central cloud server, and a local dual-delay deep deterministic policy gradient algorithm model is built and trained in each ground unit; secondly, the local model in each ground unit is aggregated to the global model at a certain learning rate through federated learning; finally, the global model is downloaded and each local model parameter is updated.

[0123] This embodiment uses Python as the programming language and TensorFLow to build the designed FL-TD3 model, that is, a model that uses FL (Federated Learning) to train the TD3 algorithm (i.e., double-delayed deep deterministic policy gradient algorithm). The experimental environment consists of two parts: the twin model and the system environment. The system environment describes the multi-UAV auxiliary network topology and the task requests of smart mobile devices as a Markov decision process model. In this embodiment, the system environment includes 4 ground units, each of which is 200*200m 2 , there are 6 randomly distributed smart mobile devices. The deployed drone has a fixed altitude of 100m and a mass of 9kg. The flight speed of the drone is between [0, 50]m / s. The transmission bandwidth and channel power gain are 1MHz and 3-0dB, respectively. The smart mobile devices move randomly in the ground unit and generate computing tasks of size [1.5.2]Mbits. The CPU frequencies of the drone and smart mobile devices are 1.4GHz and 0.7GHz, respectively. In the FL-TD3 model, the actor network and the critic network consist of 2 and 4 feed-forward deep neural networks, respectively, and use the ReLU activation function. Tables 1 and 2 show the three different delay weighting parameter values ​​η in the multi-drone assisted system. t =(0.2, 0.5, 0.8) and the comparison of the average computational cost of the system and the completion delay of the system tasks. The results show that when the delay weighting parameter value is larger, the average completion time of the system tasks is smaller. However, when the delay weighting parameter values ​​are η t =0.5 and η t =0.8, the average computational cost and task completion time of the system are the lowest. This is because in order to reduce time delay, the smart mobile device transfers more tasks to the drone, which provides more computing resources and requires more energy. Therefore, in order to better adjust the trade-off between system energy and delay, this implementation will t = 0.5 was set for subsequent comparative experiments.

[0124] Table 1 Average computational cost of the system under three different delay weighting parameter values ​​in the multi-UAV assisted system

[0125]

[0126]

[0127] Table 2 Average system delay under three different delay weighting parameter values ​​in the multi-UAV assisted system

[0128]

[0129] Table 3 shows the performance comparison of different federated learning rates based on the FL-TD3 algorithm. The convergence speed of the algorithm is greatly affected by the federated learning speed. When the federated learning rates are 0.01 and 0.001 respectively, the algorithm converges stably in a training set with a cycle of 200. However, when the federated learning rate is 0.0001, the algorithm can only converge when the training cycle exceeds 500. In addition, too large values ​​do not have an excessively positive effect on the convergence performance. The reason is that an appropriate federated learning rate can effectively aggregate local models, but the smaller the value, the slower the update rate of the global model, and more steps are required to converge. Therefore, this embodiment sets the federated learning rate to 0.01 to better illustrate the advantages of FL-TD3 in subsequent comparative experiments.

[0130] Table 3 Performance comparison of different federated learning rates based on FL-TD3 algorithm

[0131]

[0132] As shown in Table 4, this embodiment considers the impact of different delayed update parameters on the performance of the proposed TD3 algorithm. When the delayed update parameter is 2, the optimal average computational cost fluctuates at 2.12, while when the parameter is 8, the lowest average computational cost can be achieved. The larger the value, the more times the critic network is updated relative to the participant network before updating. This allows the Q value to stabilize first, and then update the strategy to reduce erroneous updates of some participant networks. In contrast, a smaller value may cause the participant network to be trapped in a limiting value due to erroneous updates. However, a larger value is not a better choice because the criticality requires longer time and some redundant calculations. This embodiment conducts a large number of experiments to obtain appropriate parameter values, which are then set to 8 in the following experiments to obtain better performance.

[0133] Table 4 Effect of different delay update parameters on TD3 algorithm performance

[0134]

[0135] Table 5 describes the comparison of the average computational cost of the four schemes FL-TD3, DDPG, A3C and DQN. The four DRL schemes have been trained for 1000 episodes, and they can effectively converge to a certain value as the number of iterations increases. FL-TD3 can obtain a better strategy to minimize the computational cost than the other three algorithms. The reason is that FL-TD3 can obtain the optimal dynamic resource allocation and drone flight strategy for a distributed dynamic network multi-drone assisted MEC system. In addition, DDPG and A3C are based on an actor-critic framework and do not solve the estimation problem, and their convergence performance is quite close. DQN cannot effectively obtain the optimal offloading strategy in the continuous action space. Therefore, FL-TD3 can reduce the waste of computational resources in the system through the collaborative learning strategy of multiple ground units.

[0136] Table 5 Average computational cost of four schemes

[0137]

[0138]

[0139] Tables 6 and 7 compare the latency and computational cost of the four schemes under different IMD computing capabilities. These two tables show that as the IMD computing power increases, the latency and execution cost of the four schemes decrease, and the results of the proposed algorithm are always the lowest. In addition, when the computing power of IMD increases, the system has stronger computing power and faster data processing speed. Since DQN cannot find accurate actions in the continuous action space, it cannot effectively utilize computing resources, resulting in the highest value of the offloading strategy. Therefore, the FL-TD3 method is more suitable for multi-UAV-assisted MEC environments, reducing the waste of resources in the system.

[0140] Table 6 Computational costs of four solutions under different IMD computing capabilities

[0141]

[0142] Table 7 Delay of four solutions under different IMD computing capabilities

[0143]

[0144] Table 8 compares the performance of offloading ratios among different schemes. It can be found that among the four schemes, the FL-TD3 scheme has the highest offloading ratio when converging, and FL-TD3 performs more tasks on the drone server, while DQN fluctuates greatly in early training and converges the slowest. The reason is that when we apply the uniform discretization of the offloading ratio in the action space to the DQN scheme, the high-order states of the DNN tend to output larger values, which makes it difficult to learn the optimal value. The FL-TD3 scheme can output multi-dimensional actions to ensure the stability of the offloading ratio action value between [0,1], learn the optimal value, and obtain the optimal offloading strategy.

[0145] Table 8 Performance of offloading ratio between different schemes

[0146]

[0147]

[0148] Table 9 compares the total cost of different schemes under different uplink channel bandwidths. It can be seen that the computational cost of all offloading schemes is reduced due to the increase in uplink bandwidth. The larger the bandwidth, the faster the IMD uploads task data, and the smaller the system delay and energy consumption. In addition, the computational cost of the FL-TD3 scheme decreases much faster than the other three schemes, which better demonstrates the advantages of this scheme.

[0149] Table 9 Total cost of different solutions under different uplink channel bandwidths

[0150]

[0151]

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A multi-drone assisted mobile edge computing offloading method in the industrial Internet, characterized in that: The following steps are involved: Step 1: Obtain the network status information, mobile intelligent user status information and drone status information of each ground unit; the ground server of each ground unit builds a digital twin model of the unit based on this information; Step 2: The digital twin model constructed based on the acquired information calculates the latency and energy consumption of the computing tasks processed by the mobile intelligent user and the drone, and then introduces the optimization goal, combining multiple factors to comprehensively determine the flight information of the drone and the computing task offloading ratio; Step 3: Model the optimization problem in the drone-assisted mobile intelligent user edge computing scenario as a Markov decision process, and further optimize the reward function through federated learning distributed training double-delay deep deterministic policy gradient algorithm to reduce the computational cost.

2. The method for offloading multi-drone assisted mobile edge computing in the industrial Internet according to claim 1 is characterized in that: The step 1 obtains the network, mobile intelligent user and drone status information in the ground unit through sensors, and the ground server in each ground unit builds a digital twin model of the unit.

3. The method for offloading multi-drone assisted mobile edge computing in the industrial Internet according to claim 2 is characterized in that: The specific method of calculating the time delay of the mobile intelligent user processing the computing task and the drone processing the computing task in step 2 is: The local computing delay of the mobile intelligent user processing computing tasks in each ground unit is calculated as shown in the following formula: in, The local computational latency of processing computational tasks for mobile intelligent user k, J k (t) is the number of CPU cycles required for the computation task, D k (t) is the proportion of computing tasks offloaded to the UAV, f k (t) is the CPU frequency of smart mobile user k, and t represents time; Calculate the transmission delay of the task to the drone as shown in the following formula: in, The transmission delay of the computing task of mobile intelligent user k to the UAV is is the data transmission rate between the UAV and the mobile user, S k are the CPU cycles and byte conversion constants of smart mobile user k; When the drone performs a computing task, the processing delay on the drone is as shown in the following formula: in, is the processing delay of the computing task of mobile intelligent user k on the drone, f uav (t) is the CPU frequency of the drone server; The total time required to complete the computation task in time slot t is as follows: Among them, T sum (t) is the total time required to complete the computational task in time slot t.

4. The method for offloading multi-drone assisted mobile edge computing in the industrial Internet according to claim 3 is characterized in that: The specific method of calculating the energy consumption of the mobile intelligent user processing computing tasks and the drone processing computing tasks in step 2 is: The energy consumption of smart mobile user k performing computing tasks is shown in the following formula: in, The energy consumption of performing computing tasks for smart mobile user k, is the effective conversion coefficient of smart mobile user k, f k represents the computing power of smart mobile user k; When the intelligent mobile user k offloads the computing task to the drone, the transmission energy at time slot t is as shown in the following formula: in, The transmission energy when the intelligent mobile user k offloads the computing task to the drone, is the transmission power of the smart mobile user offloading computing tasks; The energy consumed by the drone to perform computing tasks is shown in the following formula: in, The energy consumed by the drone to perform computing tasks, is the effective conversion coefficient of the UAV, f uav Indicates the computing power of the drone; At each time slot t, the photovoltaic module equipped by the drone provides energy to the drone battery as shown in the following formula: E renewable (t)=γ u ψ Among them, E renewable (t) is the energy provided by the photovoltaic module to the drone battery; u represents the utilization constant of the collected energy, ψ conforms to the normal distribution; Therefore, the remaining battery energy of the drone in time slot t+1 is as follows: Among them, Efly(t) is the energy consumed by the UAV flight at time slot t, and Eremain(t) is the remaining energy of the UAV at time slot t. Indicates the maximum capacity of the battery carried by the drone.

5. The method for offloading multi-drone assisted mobile edge computing in the industrial Internet according to claim 4 is characterized in that: The specific method of calculating the delay and energy consumption of the mobile intelligent user processing computing tasks and the drone processing computing tasks in step 2 and then introducing the optimization target is: The computational cost Ω of the mobile intelligent user processing computing tasks and the drone processing computing tasks is shown in the following formula: Among them, α k (t) represents the connection coefficient between the drone and the mobile intelligent user in time slot t, η and represents the weighted index, which represents the weights of delay and energy consumption respectively; Therefore, the optimization problem in the UAV-assisted mobile intelligent user edge computing scenario is expressed as follows: in, and is the position of the smart mobile device and the drone at time slot t, and They represent the horizontal and vertical coordinate values ​​of the mobile intelligent user k in the ground unit, and They represent the horizontal and vertical coordinates of the UAV in the ground unit, L and W represent the length and width of the ground unit, respectively. Formula (f) represents the energy consumption constraint of the UAV. represents the energy consumed by the UAV to perform computing tasks, Efly(t) represents the energy consumed by the UAV to fly; K is the number of mobile intelligent users contained in each ground unit, and T is the total time slot where the UAV provides computing offloading for the ground unit.

6. The method for offloading multi-drone assisted mobile edge computing in the industrial Internet according to claim 5 is characterized in that: Step 3 models the optimization problem in the UAV-assisted mobile intelligent user edge computing scenario as a Markov decision process, specifically: The state space at time slot t is defined as: S={s k (t),s uav (t),β k (t)} Among them, s k (t) is the computing task information of mobile intelligent user k in time slot t, including the current computing task size J k (t), location information pos of mobile user k k (t) and the maximum tolerable delay of the computation task s uav (t) is the state information of the drone at time slot t, including the drone position information pos uav (t), the remaining energy of the drone E remain And the remaining computing task size G remain β k (t) Express network congestion status; The action space at time slot t is defined as: a t ={M uav (t),D k (t)} Among them, M uav (t) is the UAV movement information, including the UAV flight angle θ(t) and flight speed v(t); D k (t) indicates whether the UAV and mobile intelligent user k are selected to provide computing resources in time slot t; The reward function to be optimized is expressed as: r(t)=-[η t T sum (t)+η e E sum (t)] Among them, r(t) represents the reward function, η t , η e are the proportional coefficients of the total time and energy consumption required for the calculation task, and represent the weights of delay and energy consumption, respectively.

7. The method for offloading multi-drone assisted mobile edge computing in the industrial Internet according to claim 6 is characterized in that: In step 3, the reward function is defined as a negative value to better train the double-delayed deep deterministic policy gradient algorithm model; in addition, the double-delayed deep deterministic policy gradient algorithm is trained in a distributed manner using horizontal federated learning, specifically: First, a global double-delay deep deterministic policy gradient algorithm model is built in the central cloud server, and a local double-delay deep deterministic policy gradient algorithm model is built and trained in each ground unit; secondly, the local model in each ground unit is aggregated to the global model at a certain learning rate through federated learning; finally, the global model is downloaded and the parameters of each local model are updated.

Images