A load-aware computing offloading energy optimization method in mobile edge computing

Abstract

The application discloses a load-aware computing offloading energy-saving optimization method in mobile edge computing, comprising the following steps: 1) constructing a MEC system offloading model; 2) modeling local computing of a task; 3) modeling task offloading processing; and 4) constructing an energy consumption cost model. The method utilizes a long short-term memory network to realize load prediction, utilizes the characteristics of a deep reinforcement learning method to realize high-performance offloading of a computing task, reduces energy consumption of a user, and improves quality of experience of the user.

Description

TECHNICAL FIELD

[0001] The present application relates to the method technology of mobile edge computing, deep learning and reinforcement learning, and particularly relates to a load-aware computing offloading energy optimization method in mobile edge computing. BACKGROUND

[0002] The 21st century is an era of continuous innovation of Internet technology. Among them, the continuous development and application of wireless communication technology and Internet of Things (IOT) technology make the era of Internet of Everything not far away. With the progress and popularization of technology, the number of intelligent mobile devices (Mobile Device, referred to as MD) such as smart phones, electric vehicles and personal PCs has increased dramatically, and the data traffic consumed by them has also increased exponentially. International Digital Company estimates that the number of connected devices worldwide will reach 41.6 billion in 2025. And Seagate estimates that the data traffic consumption in the same year will reach 163 ZB. The huge consumption of data traffic is not only due to the increase of mobile devices. The emergence of a large number of emerging applications brings people convenience, but also brings more consumption, for example, virtual reality (Virtual Reality), natural language processing, speech recognition and self-driving, etc. These computationally intensive emerging applications always have high requirements for the quality of service (Quality of Service, referred to as QoS), which brings higher resource consumption than traditional applications, but limited by the size and battery technology, the computing resources of mobile devices are limited. Moreover, with the increase of application computing demand, mobile devices have been unable to guarantee the low energy consumption and low latency requirements of computing tasks, which will affect the execution of tasks and reduce the quality of experience of the application program.

[0003] To guarantee the quality of service (QoS) of terminal task execution, a new paradigm combining cloud computing and edge computing has emerged. In 2014, the European Telecommunications Standardization Institute proposed the concept of mobile edge computing (MEC). MEC provides mobile terminals with sufficient computing capacity by sinking servers from the cloud to the edge of the mobile network. This architecture avoids long-distance data transmission, reduces task processing transmission delay, and relieves data transmission pressure. To save limited computing resources on mobile devices, mobile devices can offload complex tasks to edge servers. Edge servers are equipped with rich computing, storage, and communication resources, which can assist mobile devices in processing complex computing tasks and achieving low-latency user experience. Moreover, it can also reduce task processing energy consumption and delay, and reduce the discard rate of delay-sensitive tasks. However, the resources of edge nodes are limited, so the computing resources allocated to mobile devices depend on the load level of edge nodes (i.e., the number of tasks that edge nodes need to process). Some existing work has addressed the load level of edge nodes and proposed centralized task offloading algorithms. Lyu proposed an algorithm in X. Lyu, H. Tian, W. Ni, Y. Zhang, P. Zhang, and R. P. Liu, “Energy efficient admission of delay-sensitive tasks for mobile edge computing,” IEEE Trans. Commun., vol. 66, no. 6, pp. 2603–2616, Jun. 2018. that guarantees task delay and minimizes the energy consumption of task offloading. Zhang et al. proposed an online reward-optimal auction algorithm in D. Zhang, L. Tan, J. Ren, et al., “Near-optimal and truthful online auction for computation offloading in green edge-computing systems,” IEEE Trans. Mobile Comput., early access, Feb. 25, 2019. that optimizes the long-term reward total of processing offloaded tasks.Wang et al. in F. Wang and X. Zhang, “Dynamic interface-selection and resource allocation over heterogeneous mobile edge-computing wireless networks with energy harvesting,” in Proc. IEEE Conf. Comput. Commun. Workshops, 2018, pp. 190-195. consider optimizing resource allocation and offloading decisions aiming at maximizing the total energy saving of mobile devices and develop a progressively optimal method and a suboptimal scheme to obtain system information.

[0004] Centralized algorithms can require complete system information, which is not readily available in real-world environments, so some work has begun to focus on distributed task offloading algorithms with edge node load. Liu et al. in J. Liu, J. Ren, Y. Zhang, X. Peng, Y. Zhang, and Y. Yang, "Efficient Dependent Task Offloading for Multiple Applications in MEC-Cloud System," in IEEE Transactions on Mobile Computing, doi: 10.1109 / TMC.2021.3119200. considered task dependency constraints and proposed an algorithm based on heuristic ordering to minimize the average completion time. Li et al. in L. Li, T. Q. S. Quek, J. Ren, H. H. Yang, Z. Chen, and Y. Zhang, "An incentive-aware job offloading control framework for multi-access edge computing," IEEE Trans. Mobile Comput., early access, Sep. 17, 2019 introduced a new utility function and proposed an incentive-aware job offloading control framework. Yang et al. in L. Yang, H. Zhang, X. Li, H. Ji, and V. Leung, "A distributed computation offloading strategy in small-cell networks integrated with mobile edge computing," IEEE / ACM Trans. Netw., vol. 26, no. 6, pp. 2762-2773, Dec. 2018 proposed a game theory-based distributed offloading algorithm aimed at optimizing the energy consumption and latency of mobile devices. Tang et al. in M. Tang and V. W. S. Wong, "Deep reinforcement learning for task offloading in mobile edge computing systems," IEEE Trans. Mobile Comput., early access, Nov. 10, 2020 combined long short-term memory and reinforcement learning techniques to propose an algorithm to reduce long-term cost estimation in task offloading.

[0005] In the MEC offloading system, the computing resources are limited, the task completion is limited by time delay, and the communication resources need to be competed. These limiting conditions will affect the performance of task offloading, but users have different performance requirements in different offloading environments, so different limiting conditions may bring better user experience. Many existing works consider indivisible tasks, but the task dependency is generally strong, so divisible tasks may not be realistic. Moreover, most works assume that tasks can be completed in one time slot, but the execution of large data tasks may last for multiple time slots, so it is necessary to consider the queuing system at the device bottom. Considering the above factors, it is difficult to achieve high-performance computing offloading in the complex MEC environment by traditional methods. Reinforcement learning uses interaction with the environment to achieve the preset purpose, which is very suitable for solving the offloading problem in the MEC environment. SUMMARY

[0006] The purpose of the present application is to overcome the shortcomings of existing computing offloading optimization algorithms, and to provide a load-aware computing offloading energy optimization method in mobile edge computing. This method uses long short-term memory network to realize load prediction, and uses the characteristics of deep reinforcement learning method to realize high-performance offloading of computing tasks, reduce the energy consumption of users, and improve the quality of experience of users.

[0007] The technical solution to achieve the purpose of the present application is:

[0008] A load-aware computing offloading energy optimization method in mobile edge computing, comprising the following steps:

[0009] 1) Constructing a MEC system offloading model:

[0010] Based on a real scene of a public scene with a large number of mobile devices, a reasonable number of MEC servers are deployed at the network edge of mobile devices, and mobile devices communicate with MEC servers through wireless networks on orthogonal channels, wherein the number set of mobile devices is set as The number set of edge servers is set as Finally, define as the time set of the entire system offloading task, that is, one offloading period contains |T| time slots, wherein each time slot has Δ seconds, and in each time slot , the mobile device will randomly generate a new task with a probability of η∈[0,1], and the task size will be selected from a set range, define k d (t) as the unique index mark of the task, and describe the task as Where s d (t) represents the data size of the new task generated by the mobile device at time slot t, c d (t)=s d (t)ρd reflects the amount of computing resources required to complete the task, which is quantified as the total number of CPU cycles required to process the task, p d denotes the number of CPU cycles required to process one unit of data, denotes the maximum tolerated time of the task, i.e., the task needs to be completed within this time, otherwise the task will be discarded;

[0011] To focus on the performance optimization of task offloading in MEC systems, a mobile device generates new tasks at each time slot. When the mobile device's resources are insufficient, the mobile device offloads the computing task to an edge node. The edge node assists the terminal to complete the task computation and feeds back the processing result to the mobile device. Specifically, when the task decides to be processed locally, the task is assigned to the local computing queue to wait for resource allocation. Define variable x d (t)∈{0,1} indicates whether the task is processed locally; if the local resources are insufficient, the task needs to be offloaded for processing, and the task will be assigned to the transmission queue to wait for offloading. When the task transmission is completed and arrives at the edge node , it will also wait for processing in the computing queue in the edge node. Set y d,n (t)∈{0,1} indicates whether the task is offloaded to the edge node n for processing.

[0012] 2) Task local computing modeling:

[0013] The scheduler of the mobile device decides the processing method of the task. When the amount of new task data is small, the probability of local processing of the task is large. When the newly arrived task of the mobile device needs to be processed locally, x d (t) = 1, the task is put into the computing queue to wait for resource allocation, and at time slot new task k d (t) waits for processing in the computing queue, which will generate a corresponding time delay. The corresponding calculation is as follows:

[0014]

[0015] wherein denotes the time slot at which the current task starts processing after all tasks placed in the computing queue before time slot t have been processed or discarded;

[0016] After task k d (t) is put into the computing queue, the time slot at which the task is processed or discarded is calculated as follows:

[0017]

[0018] The total delay cost and energy cost of the task processed locally are calculated as follows:

[0019]

[0020] where κ is the computing capability parameter;

[0021] 3) Task offloading process modeling:

[0022] 3.1) Communication model: In the MEC system, mobile devices communicate with MEC servers through wireless networks. When task k d (t) needs to be offloaded to edge node x d (t) = 0, y d,n (t) = 1, the task is put into the transmission queue and waits, and the mobile device transmits the task to the selected edge node through the wireless network link interface, |h dn | 2 is set as the channel gain of data transmission between the mobile device and the edge node , and P is defined as the transmission power when transmitting the task to the edge node. When the mobile device transmits the task to the edge node, the data transmission rate between them is calculated as follows:

[0023]

[0024] where W is the channel bandwidth when transmitting the task, δ 2 is defined as the received noise power of the edge node;

[0025] In time slot , the task will be put into the transmission queue, but each task needs to wait for the task in front of it in the queue to be processed before it starts processing. The number of time slots for processing the task waiting is calculated as follows:

[0026]

[0027] The number of time slots for task transmission completion or discard is calculated as follows:

[0028]

[0029] Then the total delay cost and energy cost in the task transmission process are calculated as follows:

[0030]

[0031] 3.2) Edge computing model: Mobile devices decide to offload the computing task to edge nodes y d,n (t) = 1, the task will arrive at the edge node computing queue from the local transmission queue. In the edge node, each node has a queue, where all mobile devices All correspond to a task queue for processing tasks, at time slot t-1, task k d (t) is offloaded to edge node n, task k d (t) will be put into the corresponding computing queue at the next time slot, defining the unique index of the task at the edge node equal to the amount of tasks ;

[0032] at time slot When the computing queue in the edge node still has tasks to be processed, the number of queues in the edge node that still need to continue to process tasks is the load, defining B n (t) is the load size of the edge node at time slot t:

[0033]

[0034] The active queue of the edge node shares computing resources, defining as the allocable computing resources of the edge node n, set to represent the number of bits of the task discarded by the queue after time slot t,

[0035]

[0036] If the task of the mobile device is offloaded to the queue corresponding to the edge node n, defining

[0037] as the time slot for processing or discarding the task , due to the unknown load of the edge node , it is difficult to determine the exact value of before the task is processed or discarded, in order to solve this problem, defining as the time slot when the task starts to be processed;

[0038]

[0039] Before the task is processed or discarded, the size is equivalent to the amount of data accumulated processed in the processing time, that is, the amount of data of the task is not greater than the amount of data processed by the computing resources allocated by the edge node to the task from time slot to , and is greater than the amount of data completed from time slot

[0040] to ;

[0041]

[0042] Task The total cost of processing delay and the total cost of energy consumption of the task when the edge node is processed or discarded are defined as and

[0043]

[0044] 4) Build energy consumption cost model:

[0045] The computing cost involved in task offloading in the MEC system is analyzed, and the purpose is to optimize the task offloading decision to minimize the energy consumption cost under the time delay limit, wherein the total cost of offloading includes the local computing energy consumption, transmission energy consumption and edge computing energy consumption; specifically, at each time slot , the mobile device decides the processing mode of the task, and the total cost of task processing delay is represented as follows:

[0046]

[0047] Similarly, the total energy consumption cost generated after completing the task at each time slot is calculated as follows:

[0048]

[0049] The optimization problem of focusing on optimizing task offloading is to minimize the average energy consumption cost generated by the whole system, therefore, the specific system optimization problem is represented as follows:

[0050]

[0051] Wherein, constraint condition C1 represents that the computing task on the mobile device can only be processed locally or offloaded for processing, constraint condition C2 represents whether the task on the mobile device is offloaded to the edge node C3 represents that the delay cost generated by offloading the task is less than or equal to the task tolerance time;

[0052] The technical solution studies the task offloading decision and resource allocation problem under the time delay limit. In the past, the task offloading decision was solved by traditional methods such as game theory and some traditional algorithms, but the task offloading in MEC is dynamic and complex, and the traditional solution is difficult to solve the problem. With the development of machine learning and neural network, a powerful intelligent method - deep reinforcement learning brings a new solution. In order to solve the above-mentioned problem, the technical solution plans to use deep reinforcement learning to solve the offloading optimization problem of multiple mobile devices and multiple edge nodes in the MEC system.

[0053] To use the reinforcement learning method, a Markov decision process (MDP) needs to be defined, in which the interaction process between the agent and the environment is regarded as a sequential decision-making process, i.e., the agent receives the state of the environment at each step and makes an action decision according to the current state and previous experience, which depends not only on the current state but also on the goal and previous actions and their results. In this process, the agent aims to find an optimal strategy to maximize the cumulative reward in the long run;

[0054] Markov decision model:

[0055] (1) State space:

[0056] Let be the historical load level of each edge node in the previous T step time slots, i.e., the number of queues that still have tasks to be completed. Specifically, define as the load of edge node j in the previous T step time slot i.

[0057]

[0058] At the beginning of each time slot t, the mobile device needs to decide how to process the task, and the task size, task queuing time, queue information and edge node load will all affect the execution of the task. Therefore, the observed state information in the RL method is defined as:

[0059]

[0060] (2) Action space:

[0061] In the MEC system, when a mobile device d has a new task k d (t) arrives, the mobile device needs to make a decision on how to process the task, so the offloading decision is expressed as the action space as follows:

[0062] a d (t) = (x d (t), y d,n (t))

[0063] (3) Reward space:

[0064] In time slot , the mobile device makes an offloading decision under state S According to the energy cost model, the average energy cost generated by the minimization of system task offloading is minimized, but the optimization of minimization is negatively related to the reward in the RL method, so the reward space is expressed as:

[0065] r d (t)=r(s d (t),a d (t))=-E total

[0066] According to the MDP model, a task offloading and resource allocation algorithm based on Double DQN is designed, wherein different algorithms are respectively executed at the mobile device and the edge node. Specifically, the edge node assists the mobile device in training corresponding network parameters to relieve the computing pressure of the terminal, and the mobile device transmits its offloading experience to the edge node to improve the performance of the algorithm of the edge node.

[0067] (1) Energy-saving algorithm at the mobile device:

[0068]

[0069]

[0070] Compared with the prior art, the technical solution has the following characteristics:

[0071] 1. More complex MEC environment is considered: Many studies focus on task offloading in a single MEC server, but the real MEC system environment is complex, and multiple mobile devices and multiple MEC servers are a more realistic offloading environment. Moreover, most studies consider that tasks can be completed within one time slot, but large data volume tasks often take more time, so the execution of some tasks may last for multiple time slots. The technical solution not only considers multiple-to-multiple task offloading, but also considers the underlying queuing system of tasks.

[0072] 2. The load level of the edge end is predicted: It is difficult to break through the existing performance bottleneck by using traditional methods to achieve computing offloading in a complex environment, but the development of deep learning and reinforcement learning brings powerful tools. The technical solution adopts a deep reinforcement learning method and proposes a distributed offloading algorithm based on DRL. Specifically, a long short-term memory network (LSTM) is used to complete the prediction of the load level of the edge end, and Dueling DQN is used to improve the performance of the algorithm.

[0073] 3. Energy consumption optimization under delay limitation is achieved: In a complex MEC environment, the technical solution still considers optimizing energy consumption performance under delay limitation and establishes a two-part offloading model of the underlying queuing. According to different offloading modes of tasks, different offloading costs are derived, including local computing energy consumption, transmission energy consumption and offloading computing energy consumption, and the energy consumption cost minimization problem under delay limitation is formulated accordingly. BRIEF DESCRIPTION OF DRAWINGS

[0074] Figure 1A neural network structure diagram for the embodiment;

[0075] Figure 2 A training structure diagram based on the Double DQN method for the embodiment.

[0076] Figure 3 A MEC system model diagram in the embodiment;

[0077] Figure 4 A MEC offloading system model diagram in the embodiment. DETAILED DESCRIPTION

[0078] The content of the present application will be further described below in conjunction with the drawings and embodiments, but is not limited to the present application.

[0079] Embodiment:

[0080] This embodiment considers a single-cell communication application scenario, in which there are a large number of mobile devices and multiple edge servers. Because of the rise of emerging applications, the limited resources of mobile devices have been difficult to meet intensive computing requests, such as mobile applications such as augmented reality, virtual reality, and cloud gaming. Due to the size limitation of devices such as smart phones or personal computers, it is difficult to configure a large number of resources, so mobile edge computing provides an effective computing paradigm, and mobile devices can offload tasks to edge servers.

[0081] For computing-intensive tasks, this example formulates an energy optimization problem under latency constraints, and proposes a deep reinforcement learning method to realize task offloading decisions and resource allocation. Specifically, this method considers the queuing model of indivisible and latency-sensitive tasks, analyzes the cost model of task offloading, and constructs the energy cost minimization model, then converts the cost model into an MDP model, and finally realizes the energy minimization decision of task offloading by using the DDQN method.

[0082] As shown in Figure 3 , a system model in a real scenario is shown, in which the offloading model inside each mobile device is as shown in Figure 4 , in order to realize the energy minimization of task offloading, a load-aware computing offloading energy optimization method in mobile edge computing is proposed, as shown in Figure 1 , including the following steps:

[0083] 1) Construct the MEC system offloading model:

[0084] As shown in Figure 3 , based on a public scenario in a real scenario in which there are a large number of intelligent mobile devices, a reasonable number of MEC servers are deployed at the edge of the mobile device network, and the mobile devices communicate with the MEC servers through wireless networks on orthogonal channels, wherein the set of the number of smart phones is While the number of edge servers is set as Finally, the task offloading duration is set as 110 time slots, and define the time set of the whole system offloading tasks, i.e., one offloading period contains |T| time slots, where each time slot is set as 0.1 second, in each time slot , the smartphone will randomly generate a new task with probability η∈[0, 1], and the task size will be selected from the set range, define k d (t) as the unique index of the task, and describe the task as where s d (t) represents the data size of the new task generated by the mobile device at time slot t, c d (t) = s d (t)ρ d reflects the amount of computing resources required to complete the task, which can be quantified as the total number of CPU cycles required to process the task, ρ d represents the number of CPU cycles required to process one unit of data, represents the maximum tolerance time of the task, i.e., the task needs to be completed within this time, otherwise the task will be discarded;

[0085] As shown in Figure 4 , the smartphone will generate a new task at each time slot, and when the resources of the smartphone are insufficient, the smartphone will offload the computing task to the edge node, which will assist the device in completing the task computation and feed back the processing result to the mobile device. Specifically, when the task decides to be processed locally, the task will be assigned to the local computing queue to wait for resource allocation, define the variable x d (t)∈{0, 1} represents whether the task is processed locally; if the local resources are insufficient, the task needs to be offloaded for processing, the task will be assigned to the transmission queue to wait for offloading, when the task transmission is completed and arrives at the edge node , the task will also wait for processing in the computing queue of the edge node, set y d,n (t)∈{0, 1} represents whether the task is offloaded to the edge node n for processing;

[0086] 2) Task local computing modeling:

[0087] The processor of the smartphone will decide the processing method of the task, when the data volume of the new task is small, the probability of local processing of the task is large, when the newly arrived task of the smartphone needs to be processed locally x d (t) = 1, the task will be put into the computing queue to wait for resource allocation, at time slot , the new task k d(t) is put into the computation queue, the corresponding latency is calculated as follows:

[0088]

[0089] wherein denotes the time slot at which the current task starts to be processed after all tasks placed in the computation queue before time slot t have been processed or discarded;

[0090] task k d (t) is put into the computation queue, k d (t) is processed or discarded, the time slot is calculated as follows:

[0091]

[0092] The total delay cost and energy cost of the task being processed locally are calculated as follows:

[0093]

[0094] wherein κ is a computing capability parameter;

[0095] 3) Task offloading processing modeling:

[0096] 3.1) Communication model: In the MEC system, the smartphone communicates with the MEC server through the wireless network, and when task k d (t) needs to be offloaded to edge node x d (t) = 0, y d,n (t) = 1, the task is put into the transmission queue to wait, and the smartphone will transmit the task to the selected edge node through the wireless network link interface, |h dn | 2 is set as the channel gain of data transmission between the smartphone and the edge node , and P is defined as the transmission power when transmitting the task to the edge node, and the data transmission rate between the smartphone and the edge node when the smartphone transmits the task to the edge node is calculated as follows:

[0097]

[0098] wherein W is the channel bandwidth when transmitting the task, δ 2 is defined as the received noise power of the edge node;

[0099] In time slot , the task will be put into the transmission queue, but each task needs to wait for the task in front of it in the queue to be processed before it starts to be processed, and the number of time slots of the task waiting for processing is calculated as follows:

[0100]

[0101] The number of slots that the task is transmitted or discarded is calculated as follows:

[0102]

[0103] The total delay cost and energy cost during the task transmission process are calculated as follows:

[0104]

[0105] 3.2) Computing model at the edge: smart phone Decision to offload computing task to edge node y d,n (t) = 1, the task will arrive at the edge node computing queue from the local transmission queue, in the edge node, each node has a queue, where all mobile devices correspond to a task queue for processing tasks, at time slot t-1, task k d (t) is offloaded to edge node n, task k d (t) will be put into the corresponding computing queue at the next time slot, define the unique index of the task in the edge node equal to the amount of tasks ;

[0106] At time slot When the computing queue in the edge node still has tasks to be processed, the number of queues in the edge node that still need to process tasks is the load, define B n (t) as the load size of the edge node at time slot t:

[0107]

[0108] The active queues of the edge node share computing resources, define as the allocable computing resources of edge node n, set to represent the number of bits of tasks discarded by the queue after time slot t,

[0109]

[0110] If the task of the smart phone is offloaded to the queue corresponding to the edge node n, define as the time slot for processing or discarding task , due to the unknown load of the edge node , it is difficult to determine before the task the value of the task, in order to solve this problem, define for the task the time slot being processed;

[0111]

[0112] in the task before being processed or discarded, the size of the data processed during the processing time, that is, the task The amount of data is not greater than the amount of data processed from the time slot to The data processed by the edge node is less than the amount of data processed from the time slot

[0113] to The amount of data completed;

[0114]

[0115] The task When the task is processed or discarded at the edge node, the total cost of the processing delay of the task and the total cost of the energy consumption are defined as and

[0116]

[0117] 4) Construct the energy consumption cost model:

[0118] The computing cost involved in task offloading in the MEC system is analyzed, and the purpose is to optimize the task offloading decision to minimize the energy consumption cost under the delay limit, wherein the total cost of offloading includes local computing energy consumption, transmission energy consumption and edge computing energy consumption; Specifically, at each time slot , the smart phone Decide how to process the task, and the total cost of the task processing delay is represented as follows:

[0119]

[0120] Similarly, the total energy consumption cost generated after completing the task at each time slot is calculated as follows:

[0121]

[0122] The specific system optimization problem is represented as follows:

[0123]

[0124] Where constraint C1 represents that the computing task on the smartphone can only be selected to be processed locally or offloaded, and constraint C2 represents whether the task on the smartphone is offloaded to the edge node C3 represents that the time delay cost generated by offloading the task is less than or equal to the task tolerance time.

[0125] This example studies the task offloading decision and resource allocation problem under the time delay constraint. In the past, traditional methods such as game theory and some traditional algorithms have been used to solve the task offloading decision. However, the task offloading in MEC is dynamic and complex, and the traditional solutions are difficult to solve the problem. With the development of machine learning and neural networks, a powerful intelligent method - deep reinforcement learning brings a new solution. In order to solve the above problems, this example proposes to use deep reinforcement learning to solve the offloading optimization problem of multiple mobile devices and multiple edge nodes in the MEC system.

[0126] As shown in Figure 2 , in order to use the reinforcement learning method, it is necessary to define a Markov decision model MDP. In the MDP, the interaction process between the agent and the environment is regarded as a sequential decision process, that is, the agent receives the state of the environment at each step and makes an action decision according to the current state and previous experience. This action decision depends not only on the current state, but also on the goal and previous actions and their results. In this process, the agent aims to find an optimal strategy so that it can obtain the maximum cumulative reward in the long run.

[0127] Markov decision model:

[0128] (1) State space:

[0129] Let be the historical load level of each edge node in the previous T step time slots, that is, the number of queues that still have tasks to be completed. Specifically, define as the load of edge node j in the previous T step time slot i.

[0130]

[0131] At the beginning of each time slot t, the smartphone needs to decide how the task is processed. The task size, task queuing time, queue information and edge node load will all affect the execution of the task. Therefore, the observed state information in the RL method is defined as:

[0132]

[0133] (2) Action space:

[0134] In the MEC system, when a mobile device d has a new task k d(t) Upon arrival, the smartphone needs to decide how to handle the task, thus, the offloading decision is formulated as an action space as follows:

[0135] a d (t) = (x d (t), y d,n (t))

[0136] (3) Reward space:

[0137] In time slot , the smartphone makes an offloading decision in state According to the energy cost model, this example aims to minimize the average energy cost generated by task offloading in the system, but the optimization of minimization is negatively related to the reward in the RL method, so the reward space is formulated as:

[0138] r d (t) = r(s d (t), a d (t)) = -E total

[0139] According to the MDP model, a task offloading and resource allocation algorithm based on Double DQN is designed, in which different algorithms are executed at the smartphone and the edge node respectively. Specifically, the edge node will assist the mobile device in training the corresponding network parameters to alleviate the computing pressure of the terminal, while the mobile device will transmit its offloading experience to the edge node to improve its algorithm performance.

[0140] (1) Energy-saving algorithm at the smartphone:

[0141]

[0142] (2) Energy-saving algorithm at the edge node:

[0143]

[0144]

Claims

1. A load-aware computation offloading energy optimization method in mobile edge computing, characterized in that, Comprising the following steps: 1) Constructing MEC system offloading model: Based on a real-world scenario of a public place with mobile devices, a reasonable number of MEC servers are deployed at the edge of the mobile device network, and the mobile devices communicate with the MEC servers through wireless networks on orthogonal channels, wherein the number of mobile devices is set as , and the number of edge servers is set as , finally, define as the time set of the entire system offloading task, that is, one offloading period contains slots, each slot has seconds, in each slot , the mobile device will randomly generate a new task with a probability of , the size of the task will be selected from the set range, define as the unique index of the task, and describe the task as , wherein represents the data size of the new task generated by the mobile device at time slot t, reflects the amount of computing resources required to complete the task, which can be quantified as the total number of CPU cycles required to process the task, represents the number of CPU cycles required to process one unit of data, represents the maximum tolerance time of the task, that is, the task needs to be completed within this time, otherwise the task will be discarded. The mobile device end generates a new task in each time slot. When the resources of the mobile device end are insufficient, the mobile device unloads the computing task to the edge node. The edge node assists the terminal to complete the task calculation, and feeds back the processing result to the mobile device. Specifically, when the task is determined to be processed locally, the task is assigned to a local computing queue to wait for resource allocation. A variable is defined to represent whether the task is processed locally. If the local resources are insufficient, the task needs to be unloaded for processing, and the task is assigned to a transmission queue to wait for unloading. When the task transmission is completed and arrives at the edge node , the task also waits for processing in the computing queue in the edge node. A variable is set to represent whether the task is unloaded to the edge node n for processing. 2) Task local computing modeling: The scheduler of the mobile device decides the processing mode of the task. When the new task data is small, the probability of local processing is large. When the new task of the mobile device needs to be calculated locally , the task is put into a calculation queue to wait for resource allocation. In a time slot , the new task waits for processing in the calculation queue, which produces a corresponding time delay. The corresponding calculation is as follows: ， wherein tstart represents the time slot in which the current task starts to be processed after all tasks placed in the computation queue before time slot t have been processed or discarded; Task After being put into the computation queue, the task is processed or discarded the time slot of the task is calculated as follows: ， The total delay cost and energy cost of tasks processed locally are calculated as follows: ， ， wherein is a computing power parameter; 3) Task offloading processing modeling: 3.1) Communication Model: In the MEC system, the mobile device communicates with the MEC server through the wireless network. When the task decision needs to be offloaded to the edge node , the task is put into the transmission queue to wait, and the mobile device will transmit the task to the selected edge node through the wireless network link interface, be set as the channel gain of data transmission between the mobile device and the edge node , define P as the transmission power when transmitting the task to the edge node, and the data transmission rate between the mobile device and the edge node when the mobile device transmits the task to the edge node is calculated as follows: ， where W is the channel bandwidth at the time of task transmission, defined as the received noise power at the edge node; In time slots , tasks are put into the transmission queue, but each task needs to wait until the tasks in front of it are processed before it can start processing. The number of time slots that a computing task waits for processing is calculated as follows: ， The number of slots for task transmission completion or discard is calculated as follows: ， Then the total delay cost and energy cost during task transmission are calculated as follows: ， ； 3.2) Edge computing model: mobile devices Decision computing tasks are offloaded to edge nodes , The task will travel from the local transmission queue to the edge node computing queue. At the edge node, each node... All A queue containing all mobile devices Each corresponds to a task queue for processing tasks, within a time slot. ,Task Unload to edge node n, task It will be placed into the corresponding computation queue in the next time slot, defining a unique index for the task on the edge node. , , equals task The workload; At time slot When the compute queue in the edge node still has tasks to be processed, the number of queues in the edge node that still need to continue to process tasks is the load, defined as For the load size of the edge node at time slot t: ， The active queue sharing computing resources of the edge node, define Let be the allocable computing resources owned by the edge node n, denotes the number of bits of tasks discarded by the queue after time slot t, ， If the task of the mobile device is offloaded into the queue corresponding to the edge node n, define the time slot for processing or discarding the task , due to the unknown load of the edge node , it is difficult to determine the exact value before the task is processed or discarded, in order to solve this problem, define the time slot for the task to start being processed; ， In the task before being processed or discarded, the size of the data processed during the processing time, i.e. the task data size is not greater than the amount of data processed by the edge node assigned to the task in the time slot to the amount of data processed, and greater than the amount of data completed from the time slot to ; ， Task The total cost of processing delay and the total cost of energy consumption of a task when the edge node is processed or discarded are defined as and respectively. ， ； 4) Constructing energy cost model: The computing cost involved in task offloading in the MEC system is analyzed, aiming to optimize the task offloading decision to minimize the energy cost under the time delay limit, wherein the total cost of offloading includes local computing energy, transmission energy and edge computing energy; at each time slot t∈𝒯, the mobile device d∈𝒟 decides the processing method of the task, and the total cost of task processing delay is represented as follows: ， Similarly, the total energy cost generated after completing the task at each time slot is calculated as follows: ， The specific system optimization problem is represented as follows: ， ， ， ， wherein Constraint C1 indicates that a computing task on a mobile device can only be selected to be processed locally or offloaded, constraint C2 indicates whether a task on a mobile device is offloaded to an edge node , C3 indicates that a latency cost of offloading a task is less than or equal to a task tolerance time In order to use the reinforcement learning method, it is necessary to define the Markov decision model MDP, in which the interaction process between the agent and the environment is regarded as a sequential decision-making process, i.e. the agent receives the state of the environment at each step and makes an action decision according to the current state and previous experience, which depends not only on the current state but also on the goal and previous actions and their results, in this process, the agent aims to find an optimal strategy to maximize the cumulative reward in the long run; Markov decision model: (1) State space: Let be the history load level of each edge node at time ; i.e. the number of tasks still in the queue when the edge node has finished its tasks at time ; i.e. the load of edge node j at time ; i.e. the load of edge node j at time ; i.e. the load of edge node j at time ， At the beginning of each time slot t, the mobile device needs to decide how the task is processed, and the task size, task queuing time, queue information and edge node load will all affect the execution of the task, therefore, the observed state information in the RL method is defined as: ； (2) Action space: In a MEC system, when a mobile device d has a new task arrives, the mobile device needs to decide how to handle the task, thus, the offloading decision is formulated as an action space as follows: ； (3) Reward space: In a time slot , a mobile device makes an offloading decision under a state , according to an energy cost model, to minimize the average energy cost incurred by task offloading in the system, but the optimization of minimization is negatively related to the reward in the RL method, so the reward space is expressed as: ； According to the MDP model, a task offloading and resource allocation algorithm based on Double DQN is designed, in which the mobile device and the edge node execute different algorithms respectively, the edge node will assist the mobile device to train the corresponding network parameters to alleviate the computing pressure of the terminal, while the mobile device will transmit its offloading experience to the edge node to improve the performance of its algorithm; (1) Energy-saving algorithm at mobile device:

1. Initialize the mobile device in its initial unloaded state, 2. from time slot 1 to traverse, 3. If a task is generated, send a parameter request to the edge node, 4. Receive parameter vector and select action based on it , 5. Make unload go to next state , 6. A reward set is given for completed tasks , 7. Iterating over a collection from experience in 8. Experience with all offloading tasks send to edge node; (2) Energy-saving algorithm at edge node:

1. Initializing an edge node's experience replayer and iteration count Count = 0; 2. initialize the evaluation network with a random network parameter vector , 3. Initialize the target network with the random network parameter vector , 4. Repeat:

5. If the parameter request of the mobile device is received:

6. Transmit parameter vector to the mobile device; 7. If the experience information of the mobile device is received:

8. Unload experience store to ; 9. Randomly sample an experience set (defined as ) from the memory ); 10. Iterate through the set of experiences all experiences in the set:

11. Calculate the target Q value corresponding to the experience 12. Minimize the error function in the RL method to update the parameter vector 13. Increase the iteration count by one Count = count + 1; 14. If the iteration count reaches the parameter replacement condition:

15. The network parameters of the target network are replaced 16. Repeat.

Images