A collaborative edge computing task offloading method based on deep reinforcement learning
By adopting a two-layer optimization method based on deep reinforcement learning in a multi-user mobile edge computing system, combining offload decisions, channel allocation and power allocation, the problem of minimizing energy consumption under delay constraints is solved, and the system energy consumption is realized Optimization and improvement of user experience.
Patent Information
- Application Number
- CN202111278169.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-30
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-10-30
AI Technical Summary
In multi-user mobile edge computing systems, how to minimize energy consumption under delay constraints, especially when traditional communication resource allocation and computing resource allocation are complicated, and effective task offloading strategies are formulated.
Using a two-layer optimization method based on deep reinforcement learning, the collaborative mobile edge computing network scenario is built, combined with offload decisions, computing resources, channel allocation and power allocation for joint optimization, and using Dueling DQN and Double DQN technologies, combined with adaptive noise, optimized. Offload decisions, channel selection and power allocation ensure that energy consumption for all users is minimized under delay constraints.
Effectively optimize the total energy consumption of the system under delay constraints, improving the user experience and the energy saving effect of the equipment.
Smart Images

Figure CN114189936B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of wireless networks and relates to an energy consumption optimization method for task offloading and resource allocation in collaborative edge computing based on deep reinforcement learning. Background Art
[0002] With the popularity of smart mobile devices, such as smartphones, smart watches, smart bracelets, and Internet of Things (IOT) devices, such as shared power and shared bicycles, many new mobile applications have come with the wave of fashion. These new mobile applications, such as e-health, face recognition, surveillance, augmented reality, and virtual reality, are not only computationally intensive, but also energy-intensive. However, this unparalleled growth is not matched by the improvement of the battery and computing power of mobile devices. Given the huge growth in the use of mobile devices, Mobile Edge Computing (MEC) can bridge the gap between the limited capabilities of mobile devices and the growing computing demand.
[0003] To address these challenges, mobile devices can offload computational tasks to nearby MEC servers to reduce processing latency and save battery energy. Therefore, the most important issue that arises in computational offloading is selecting a MEC server among all available servers. If the selected MEC server experiences heavy workloads and unfavorable communication conditions, it may take longer for the mobile device to offload data and receive results. Therefore, in order to implement an effective and efficient offloading strategy, environmental changes such as the available energy of the mobile device, the computational workload, and the computing power of different MEC servers must be taken into account. Therefore, the offloading strategy is particularly important.
[0004] Moreover, communication resources and computing resources are particularly important for the performance of task offloading: the former determines the data rate and energy consumption during the transmission task, while the latter limits the computing time and energy consumption of the task. Traditional research on communication resource allocation mainly focuses on spectrum efficiency and energy efficiency. However, in these two resource-constrained MEC systems, the transmission and computing processes are coupled. As the number of users increases, the combinatorial optimization of the MEC system becomes more and more complex. Therefore, it is necessary to formulate an effective offloading strategy for multi-user MEC systems.
[0005] In view of the above considerations, the present invention provides an energy consumption optimization method for task offloading and resource allocation in multi-user collaborative edge computing based on deep reinforcement learning. Summary of the invention
[0006] The technical problem to be solved by the present invention is to provide an energy consumption optimization method for task offloading and resource allocation in collaborative edge computing. By jointly optimizing offloading decisions, computing resources, channel allocation and power allocation, the goal is to minimize energy consumption under delay constraints.
[0007] The technical solution of the invention is as follows:
[0008] Constructing collaborative mobile edge computing network scenarios, such as the attached Figure 1 As shown in the figure, we consider a MEC system with a cloud service, multiple base stations with MEC servers and a large number of users, and Orthogonal Frequency Division Multiple Access (OFDMA) is used as its transmission mechanism. Assume that there are |S| subcarriers available for wireless transmission, S = {1,2,...,s,...,|S|}, and the bandwidth of each subcarrier is B. The network contains 1 cloud, M = {1,2,...,m,...,|M|}, a total of |M| base stations with MEC servers. N = {1,2,...,n,...,|N|}, a total of |N| mobile users, each user has 1 task to perform. Task U of user n n =(C n ,D n ,T n max ), C n To complete the task U n The amount of computation required, in CPU cycles / bit, D n is the size of the data in bits, T n max Represents task U n The delay constraint of task M i Must be in [0,T i max ] is completed within . Since the calculation result is very small, we ignore the calculation result and its return process.
[0009] In addition, we assume that all MEC servers belong to the same network operator, so that the computation data can be split and forwarded once between MEC servers to achieve collaborative execution. When a divisible task is offloaded to a MEC server, it will decide whether the task is executed on a single MEC server or split the task and forward the request to other MEC servers or remote clouds for collaborative computation. This decision is based on the workload and computing resource status of each MEC server.
[0010] In this way, each task has four possible offloading decisions. We define For each task Un The four possible uninstallation decisions are as follows:
[0011] 1) α n =1 indicates task U n Local calculation, otherwise α n =0.
[0012] 2) When the task U n Offload to MEC server m computing, otherwise m represents the unloading location of user n. When MEC server m has sufficient computing resources to meet the needs of user n, it receives task U n After that, complete the task calculation independently.
[0013] 3) When the task U n The calculation is completed under the cooperation of MEC server m and MEC server k, otherwise When MEC server m receives task U n When there are not enough computing resources to meet the needs of user n, MEC server m will send a request to other MEC servers k with sufficient computing resources through the X2 link, and hand over the remaining computing tasks to MEC server k for calculation.
[0014] 4) When the task U n The calculation is completed under the cooperation of MEC server m and cloud server, otherwise When there is no MEC server k with sufficient computing resources to complete the remaining computing tasks of user n, MEC server m sends a request to the cloud server through the optical fiber link, and MEC server m sends the remaining computing tasks to the cloud server for computing.
[0015] We define the offloading decision matrix O = {o1,o2,...,o n} is the uninstallation decision matrix of n users. Definition is the computational resource allocation matrix, Indicates that server m is assigned to task U n computing resources.
[0016] In addition, let the subcarrier allocation matrix be W = {w n,s,m |w n,s,m ∈{0,1},n∈N,s∈S,m∈M},w n,s,m =1 means that subcarrier s will be allocated to user n, and the task offloading calculation will be used for transmission to server m. Subcarrier power allocation matrix P = {p n,s,m |p n,s,m ∈(0,Pmax ],n∈N,s∈S,m∈M}, where p n,s,m Indicates the transmission power of subcarrier s allocated between user n and server m to offload the task. n,s,m represents the channel gain between user n and server m. At the same time, we assume a flat fading environment so that the channel gain matrix remains unchanged during a scheduling process. The system noise conforms to the zero-expectation Gaussian distribution, and its variance is δ 2 express.
[0017] The technical problem to be solved by the present invention is to provide an energy-saving two-layer optimization method for joint task offloading and resource allocation of collaborative mobile edge computing based on deep reinforcement learning. By jointly optimizing offloading decisions, computing resources, channel allocation and power allocation, the goal is to minimize the energy consumption of all users under delay constraints.
[0018] The steps of the energy consumption optimization method for task offloading and resource allocation in multi-user collaborative edge computing based on deep reinforcement learning proposed in the present invention are as follows:
[0019] 1. Construct the mathematical model p1 of the collaborative edge computing system, with the goal of minimizing energy consumption. The steps are as follows:
[0020] 1) Transmission model
[0021] For all offloaded computations, mobile users must first upload data to the MEC server. Given the OFDMA mechanism, each user has exclusive access to a subcarrier due to subcarrier allocation, and the interference between user transmissions is ignored. Also, since the computation result is very small, we ignore the process of transmitting the computation result back.
[0022] User n assigns task U n The data transmission rate when transmitted to MEC server m is shown in formula (1):
[0023]
[0024] Where B is the channel bandwidth, w n,s,m Indicates that subcarrier s is allocated to user n and the task is offloaded to server m for transmission. n,s,m represents the transmission power of the task to be uploaded by allocating subcarrier s between user n and server m, g n,s,m represents the channel gain between user n and server m.
[0025] Mission U n The transmission time to MEC server m is shown in formula (2):
[0026]
[0027] Mission U n The transmission energy consumption to MEC server m is shown in formula (3):
[0028]
[0029] The MEC server m receives the task U n If the MEC server m does not have enough resources to meet user needs, the MEC server will do its best to calculate part of the data within the delay constraint. MEC server m sends the remaining data through the X2 link Forward to another MEC server k with sufficient resources. The two MEC servers collaborate to calculate, so the remaining data is transmitted to MEC server m. The transmission time to MEC server k is shown in formula (4):
[0030]
[0031] in is the X2 link capacity between MEC servers m and k. MEC server m forwards the remaining data The calculation formula is shown in formula (5):
[0032]
[0033] in Indicates that server m is assigned to task U n When no MEC server has enough computing resources to complete the remaining tasks of user n, MEC server m forwards the task request to the cloud server through the wired optical fiber link. MEC server m collaborates with the cloud server for computing, so it is necessary for MEC server m to transmit the remaining data. The transmission time to the cloud server is shown in formula (6):
[0034]
[0035] in is the link capacity between MEC server m and the cloud server.
[0036] 2) Computational Model
[0037] The MEC server receives the offloaded tasks and then executes them in parallel. Since we consider the upper limit of the processing capacity of the MEC server and the upper limit of the tasks that can be processed under the delay constraint, we do not consider the serial processing of tasks and the task waiting queue.
[0038] A)α n =1, Task U nIn the local computing mode, U n The computation time for local execution is shown in formula (7):
[0039]
[0040] in For U n Local execution of allocated computing resources.
[0041] Mission U n The energy consumption of the computation performed locally is shown in formula (8):
[0042]
[0043] Wherein k0>0 is a constant related to the CPU of the user device.
[0044] B) Mission U n When the computing mode is offloaded to MEC server m and MEC server m has sufficient computing resources to ensure that the task is completed within the delay constraint, U n The computation time of executing MEC server m is shown in formula (9):
[0045]
[0046] The computing resources allocated to server m are
[0047] Mission U n The energy consumption of computing performed by MEC server m is shown in formula (10):
[0048]
[0049] Where k1>0 is a constant related to the CPU of the MEC server.
[0050] C) When the MEC server m does not have enough computing resources to ensure that the task is completed within the delay constraint, the MEC server will do its best to calculate part of the data within the delay constraint. The remaining data Forwarded to MEC server k, since the calculation of MEC server m does not affect the processing of the remaining data, the remaining data can be forwarded to other servers for calculation at the same time, so the calculation time of MEC server m can be ignored at this time. At this time, the computing resources allocated to n by MEC server m are all the remaining computing resources of m. n The remaining data The computation time executed on MEC server k is shown in formula (11):
[0051]
[0052] in represents the computing resources allocated to server k.
[0053] U n The remaining data The computational energy consumption performed on MEC server k is shown in formula (12):
[0054]
[0055] D) When no collaborative MEC server has sufficient computing resources to complete the remaining tasks of user n, the MEC server will do its best to calculate part of the data within the delay constraint. The remaining data Forwarded to the cloud server Cloud. At this time, the computing resources allocated to n by MEC server m are all the remaining computing resources of m. n The remaining data The computation time executed on the cloud server is shown in formula (13):
[0056]
[0057] in Represents the computing resources allocated by the cloud server.
[0058] U n The remaining data The energy consumption of computing performed on the cloud server is shown in formula (14):
[0059]
[0060] Where k2>0 is a constant related to the cloud server CPU.
[0061] 3) Delay Model
[0062] For the task U under four decisions n The delay is shown in formulas (15)-(18):
[0063] A)α n =1
[0064]
[0065] B)
[0066]
[0067] C)
[0068]
[0069] D)
[0070]
[0071] Mission U n The total delay is shown in formula (19):
[0072]
[0073] 4) Energy consumption model
[0074] For the task U under four decisions n The energy consumption is shown in formulas (20)-(23):
[0075] A)α n =1
[0076]
[0077] B)
[0078]
[0079] C)
[0080]
[0081] D)
[0082]
[0083] Mission U n The total energy consumption is shown in formula (24):
[0084]
[0085] 5) Energy consumption minimization
[0086] With the goal of minimizing the total energy consumption of the entire system, including total computing and transmission energy consumption, the offloading decision, computing resources, channels, and power allocation are optimized. The mathematical model (problem) p1 is expressed as follows:
[0087]
[0088]
[0089]
[0090]
[0091]
[0092]
[0093]
[0094]
[0095] Where (25b) represents the range of the task offloading decision; (25c) states that the task can be offloaded to at most one MEC server m, and can perform collaborative computing of the remaining tasks with other servers at most once. (25d) represents the maximum computing resource allocation that can be made to MEC server m; (25e) is the maximum transmission power constraint for user n, which is less than or equal to P max ; (25f) and (25g) indicate that each subcarrier is assigned to at most one user; (25h) ensures that each task must be completed under the delay constraint.
[0096] 2. Analyze problem p1 and transform it into a two-layer problem and solve it separately. The steps are as follows:
[0097] Through further analysis of problem p1, we can find the following two characteristics: first, different unloading decision results will result in different allocated computing resources and wireless channel resources; second, different allocated computing resources and wireless channel resources will result in different optimal unloading decisions. Since the optimization variables both influence each other and can be divided into two independent sets, problem p1 meets the characteristics of a two-layer optimization problem, and problem p1 can be transformed into a two-layer optimization problem p2. Two-layer optimization is to solve the upper-layer optimization problem on the premise of ensuring the optimality of the lower-layer optimization problem. The present invention regards the unloading decision problem and the wireless channel resource allocation problem as the upper-layer optimization problem, and regards the computing resources as the lower-layer optimization problem, with the aim of minimizing the total energy consumption of all mobile users.
[0098] Specifically, through model analysis, after offloading decisions, channel and power allocation, based on the task completion time constraint (25h), all tasks must complete the task within the time constraint. At the same time, according to formula (19) The definition of , can be derived from the computational task U n The minimum lower bound of computing resources under different decisions is shown in formula (26):
[0099]
[0100] By analyzing the energy consumption calculation formulas (10), (12) and (14) of the computing model, it can be seen that the computing energy consumption is proportional to the computing resources. The computing resource lower bound formula (26) is the optimal energy consumption value, that is, the computing resource allocation.
[0101] Therefore, the problem p1 can be transformed into a two-level optimization problem p2, as shown in formula (27):
[0102]
[0103] (25b)-(25h)(27b)
[0104]
[0105] Then, we propose a two-layer optimization algorithm based on deep reinforcement learning to solve problem p2. The upper layer optimization and the lower layer optimization are iterative, and the latter is nested in the former. First, the upper layer solves the unloading decision O k , power distribution P k and channel allocation W k , provided to the lower layer. In the lower layer optimization, the monotonic optimization method is used to obtain the optimal allocation of corresponding computing resources according to the given offloading decision and communication resource allocation. k , calculate the optimal energy consumption Then feedback to the upper level.
[0106] 3. Solve the upper-level optimization problem and solve the unloading decision based on deep reinforcement learning k , power distribution P k and channel allocation W k , provided to the lower layer, the steps are as follows:
[0107] There are three key elements in deep reinforcement learning methods, namely state, action, and reward, which are defined as follows:
[0108] · Status (st): st is used to represent status. Represents the state space of the environment, n∈N is the current user, st n (n∈N) means that the nth task is currently being executed.
[0109] Action (a): a represents an action, which consists of three parts: offloading decision, channel selection, and power allocation. The offloading decision is a vector represents the offloading decision of user n. The channel selection is a vector Π=[ω1,...,ω n ,...,ω |N| ],ω n ={ω n,1 ,...,ω n,s ,...,ωn,|S|} represents the channel selection of user n, where ω n,s =1 indicates that user n is assigned channel s. The power is selected as the vector Λ=[p1,...,p n ,...,p |N| ], where p n ={p n,1 ,...,p n,k ,...,p n,max-1} represents the power selection of user n, p n,k =1 means according to the maximum transmission power P max The discretized power of the kth level is allocated to user n. Combined with the vector Π, Λ Up , the system action is as shown: a=[o1,...,o n ,...,o |N| ,ω1,...,ω n ,...,ω |N| ,p1,...,p n ,...,p |N| ].
[0110] Reward (R): Generally speaking, the reward function should be related to the objective function. In a certain state st, a reward is obtained after executing each possible action a. The goal of the optimization problem is to minimize the total energy consumption Because the reward is negatively correlated with the total energy consumption, In return, the minimum energy consumption corresponds to the maximum return.
[0111] We propose a two-layer optimization method based on deep reinforcement learning, which combines Dueling DQN and Double DQN techniques, and adds adaptive noise to the internal parameters of the neural network to improve learning efficiency. k , power distribution P k and channel allocation W k After that, the lower layer obtains the computing resource allocation r according to the provided offloading decision, power allocation and channel allocation. k , the total energy consumption of the system is calculated As a Reward Feedback to the upper layer. After the upper layer gets the feedback energy consumption, it stores it in the experience pool and updates the network weight parameters.
[0112] Initialize iteration = 0, the maximum number of iterations Ι, and set the current state st, that is, user n and the corresponding task U nThe environmental information is input into the evaluation network, and the output is the Q value corresponding to the action (the action is the unloading decision, power allocation and channel allocation). The action corresponding to the maximum Q value is selected according to the ∈-greedy strategy, or an action is randomly selected and input into the lower layer.
[0113] 4. Based on the unloading decision provided by the upper layer O k , power distribution P k and channel allocation W k , the lower layer solves the optimal allocation of computing resources through monotonic optimization k , calculate the reward (i.e. optimal energy consumption ) and then fed back to the upper layer. The steps are as follows:
[0114] By analyzing the energy consumption calculation formulas (10), (12) and (14) of the computing model, it can be seen that there is a strictly monotonically increasing relationship between computing resources and total computing energy consumption. In order to minimize the total computing energy consumption of all mobile users, the computing resources allocated to each task should be as small as possible. The computing resource lower bound formula (26) is the optimal energy consumption value, that is, the computing resource allocation. After obtaining the offloading decision O k , power distribution P k and channel allocation W k Then, according to formula (26), the lower bound of computing resources can be calculated as computing resource allocation, and then substituted into formula (27) to calculate the total energy consumption of the system After the lower-level calculations are allocated computing resources, they receive energy consumption rewards and the next state st' (i.e. the next task to be performed) are fed back to the upper layer.
[0115] 5. The upper layer receives feedback from the lower layer, stores it in the experience pool, and updates the network parameters. The steps are as follows:
[0116] The upper layer receives feedback and stores a 4-tuple into the experience pool. The 4-tuple includes the current state st, the selected action a, the next state st', and the reward obtained. Then input st' into the evaluation network and jump to step 3 until φ 4-tuples are stored in the experience pool. Skip to step 6.
[0117] 6. Train the evaluation network and the target network. The steps are as follows:
[0118] i) 4-tuple stored from the experience pool Sample a part, input the sampled multiple st as a batch into the evaluation network, and then select the corresponding Q value Q in the evaluation network according to the sampled action a eval(st,a,ω), where ω is the weight parameter of the evaluation network after adding noise, and the calculation formula is shown in (28):
[0119]
[0120] Among them, θ is the weight parameter of the evaluation network, and σ is the noise scale of adding Gaussian noise.
[0121] ii) Input the sampled multiple st' as a batch into the target network (the evaluation network has the same network structure as the target network) to obtain the Q value Q'(st',a') of action a', take the largest Q value in Q'(st',a') and multiply it by a coefficient γ, then add the sampled reward R to get Q tar (st',a',ω'), the calculation formula based on DDQN is shown in (29):
[0122]
[0123] Among them, ω' is the weight parameter of the target network after adding noise, and the calculation formula is shown in (30):
[0124]
[0125] Among them, θ' is the weight parameter of the target network.
[0126] iii) Calculate the loss function Back propagation Update the neural network weight parameters: Calculate the loss value based on the two Q values obtained by the two neural networks, that is, the result Q of step 2 eval (st,a,ω) and the result Q of step 3 tar The mean square error of (st', a', ω') is calculated by the iteration number iteration+1 as shown in (31):
[0127] L(ω)=E[Q tar (st',a',ω')-Q eval (st,a,ω)] 2 (31)
[0128] The gradient descent method is used to back propagate and update the network weight parameter ω. The calculation formula is shown in (32):
[0129]
[0130] After every θ steps, the weight ω of the evaluation network is updated to the target network ω';
[0131] 7. Determine whether the number of algorithm iterations iteration is greater than the maximum iteration number I. If iteration>I, the iteration ends, otherwise jump to step 3 to continue iterating under the new network parameters;
[0132] After the training is completed, the trained neural network can be used to input the current task to obtain the offloading decision, channel allocation, power allocation, and computing resource allocation. In this way, the current optimal solution can be obtained. * ,r * ,W * ,P *}, and the corresponding minimum energy consumption
[0133] Beneficial effects:
[0134] The present invention solves a task offloading and resource allocation method in collaborative edge computing based on deep reinforcement learning, effectively obtaining the optimal value of energy consumption under delay constraints, thereby improving user experience and device energy saving.
[0135] The present invention is further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0136] Figure 1 It is a schematic diagram of the network model of the present invention;
[0137] Figure 2 is a flow chart of the double-layer optimization method of the present invention;
[0138] Figure 3 This is a flowchart based on the deep reinforcement learning algorithm proposed in the present invention. DETAILED DESCRIPTION
[0139] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:
[0140] Embodiment 1:
[0141] In this embodiment, Figure 1 The figure shows a schematic diagram of a collaborative mobile edge computing network model, a cloud service, 3 base stations with MEC servers and 10 mobile user devices N = {1, 2, ..., 10}, all mobile users have 1 task to perform. In addition, the computational workload C required to complete the user task n (n∈N) is 2000 cycles / bit. User input data D n (n∈N) is randomly distributed in [1000,1500] bits, and the delay constraint of the task The CPU computing power of each mobile device is 15ms. Randomly distributed in [0.1, 0.9] GHz, there are a total of S = {1, 2, ..., 10} channels, the channel bandwidth B is 12.5 kHz, and the maximum transmission power P max is 1.3W, background noise δ 2 The CPU frequency of MEC servers is randomly distributed in [2.3, 2.7] GHz. The X2 link capacity between MEC servers The link capacity between the MEC server and the cloud server is 5Mbps. is 120Mbps. In addition, k0 is 5*10 -24 , k1 is 10 -26 , k3 is 10 -25 The maximum number of iterations is Ι = 300, the experience pool capacity is φ = 10000, the target network weights are updated after iteration θ = 10 times, and the noise scale of Gaussian noise is added with σ = 1.01.
[0142] The objective of the present invention to jointly optimize offloading decisions, computing resources, channel allocation and power allocation is to minimize the total energy consumption of mobile users in the entire system under the delay constraint. Figure 2 The figure shows a flow chart of the double-layer optimization method proposed by the present invention. The steps are as follows:
[0143] S1-1 Initialize the experience pool capacity to 10000, initialize the number of iterations to 1, and generate 10 tasks U for 10 mobile users. n , and its corresponding C n , D n , and S are shown in Table 1.
[0144] Table 1 Parameters of each task
[0145]
[0146] S1-2 is at the upper level, calculating the offloading decisions and resource allocation of 10 tasks and their corresponding energy consumption;
[0147] S1-2-1 starts selecting actions from the first task and obtains the offloading decision, channel allocation and power allocation as shown in Table 2, which are then input to the lower layer.
[0148] Table 2 Actions of the first task
[0149] <![CDATA[o1]]> <![CDATA[p 1,1,3 ]]> S <![CDATA[U1]]> <![CDATA[β1 3 =1]]> 1.035 1
[0150] S1-2-2 The lower layer receives the task and its offloading decision o1, channel allocation S and power allocation p 1,1,3 Then, the computing resource allocation r1 is obtained according to formula (26) through the strictly monotonic relationship3 , and then substitute into formula (24) to obtain the energy consumption In this way, the offloading decision, resource allocation and energy consumption of the first task are obtained, as shown in Table 3, and then fed back to the upper layer. After receiving the feedback, the upper layer stores it in the experience pool and selects the action for the next task.
[0151] Table 3 Offloading decision and resource allocation of the first task
[0152]
[0153] S1-2-3 repeats S1-2-1 to S1-2-2 10 times until the offloading decisions and resource allocations of 10 tasks and their corresponding energy consumption are obtained, as shown in Table 4.
[0154] Table 4 Offloading decisions and resource allocation of all 10 tasks and their corresponding energy consumption
[0155]
[0156] S1-3 In this way, one round of iteration is completed, the number of iterations is increased by 1, all data are stored in the experience pool, and the loss function is calculated according to formula (25). Back propagation updates the neural network weight parameters to train the neural network, and after each iteration θ = 10 times, that is, when iteration is a multiple of 10, the target network weight is updated and Gaussian noise is added to the two neural network parameters. Then, the algorithm performs a new round of iterative training under the new neural network parameters.
[0157] S1-4 repeats S1-2 to S1-3 until the number of iterations reaches the maximum number of iterations I=300. At this point, the neural network training is completed and the global optimal solution is obtained, including the task offloading decision O * , channel allocation W * , power distribution P * and computing resource allocation * Total energy consumption At this time, the minimum total energy consumption is 0.3774 joules, as shown in Table 5
[0158] Table 5 Global optimal solution
[0159]
Claims
1. A collaborative edge computing task offloading method based on deep reinforcement learning, characterized in that: The following steps are involved: Step 1: Construct a mathematical model p1 of the collaborative mobile edge computing system, with the goal of minimizing energy consumption; Step 2: Analyze problem p1 and transform it into a two-layer problem p2 for solution; Step 3: Solve the upper-level optimization problem and solve the unloading decision O based on deep reinforcement learning k , power distribution P k and channel allocation W k , provided to the lower level; Step 4: Based on the offloading decision, power allocation and channel allocation provided by the upper layer, the lower layer solves the optimal allocation of computing resources r k , calculate the energy consumption Then feedback to the upper level; Step 5: The upper layer receives feedback from the lower layer, stores it in the experience pool, and updates the network parameters; Step 6: Train the evaluation network and the target network; Step 7: Determine whether the number of algorithm iterations is greater than the maximum iteration number I. If iteration>I, the iteration ends, otherwise jump to step 3 to continue iterating under the new network parameters; The steps to construct the mathematical model p1 of the collaborative edge computing system in step 1 are as follows: 1) Transmission model For all offloaded computations, mobile users must first upload data to the MEC server. Given the OFDMA mechanism, each user has exclusive access to a subcarrier due to subcarrier allocation, and interference between user transmissions is ignored; Furthermore, since the calculation result is very small, we ignore the process of transmitting the calculation result back; User n assigns task U n The data transmission rate when transmitted to MEC server m is shown in formula (1): Where M represents the set of base stations with MEC servers, N represents the set of mobile users, B is the channel bandwidth, and w n,s,m Indicates that subcarrier s is allocated to user n and the task is offloaded to server m for transmission. n,s,m represents the transmission power of the task to be uploaded by allocating subcarrier s between user n and server m, g n,s,m represents the channel gain between user n and server m; Mission U n The transmission time to MEC server m is shown in formula (2): Where D n Indicates the size of the task data of user n, in bits; Mission U n The transmission energy consumption to MEC server m is shown in formula (3): The MEC server m receives the task U n If the MEC server m does not have enough resources to meet user needs, the MEC server will do its best to calculate part of the data within the delay constraint. MEC server m sends the remaining data through the X2 link Forward to another MEC server k with sufficient resources. The two MEC servers collaborate to calculate, so the remaining data is transmitted to MEC server m. The transmission time to MEC server k is shown in formula (4): in is the X2 link capacity between MEC servers m and k. MEC server m forwards the remaining data The calculation formula is shown in formula (5): Where T n max Represents task U n The delay constraint of task M i Must be in [0,T i max ] is completed within C n To complete the task U n The amount of computation required, in CPU cycles / bit, Indicates that server m is assigned to task U n When no MEC server has enough computing resources to complete the remaining tasks of user n, MEC server m forwards the task request to the cloud server through the wired optical fiber link; MEC server m collaborates with the cloud server for computing, so it is necessary for MEC server m to transmit the remaining data. The transmission time to the cloud server is shown in formula (6): in is the link capacity between MEC server m and the cloud server; 2) Computational Model The MEC server receives the offloaded tasks and then executes them in parallel; since we consider the upper limit of the processing capacity of the MEC server and the upper limit of the tasks that can be processed under the delay constraint, we do not consider the serial processing of tasks and the task waiting queue; A)α n =1, Task U n In the local computing mode, U n The computation time for local execution is shown in formula (7): in For U n Local execution of allocated computing resources; Mission U n The energy consumption of the computation performed locally is shown in formula (8): Where k0>0 is a constant related to the CPU of the user device; B) Mission U n When the computing mode is offloaded to MEC server m and MEC server m has sufficient computing resources to ensure that the task is completed within the delay constraint, U n The computation time of executing MEC server m is shown in formula (9): The computing resources allocated to server m are Mission U n The energy consumption of computing performed by MEC server m is shown in formula (10): Where k1>0 is a constant related to the CPU of the MEC server; C) Where k≠0, indicating that the task U n Offload to MEC server m. When MEC server m does not have enough resources to meet user needs, it offloads part of the tasks to MEC server k for execution. When MEC server m does not have enough computing resources to ensure that the task is completed within the delay constraint, the MEC server will do its best to calculate part of the data within the delay constraint. The remaining data D n r Forwarded to MEC server k. Since the processing of the remaining data is not affected while MEC server m is calculating, the remaining data can be forwarded to other servers for calculation at the same time, so the calculation time of MEC server m can be ignored at this time. At this time, the computing resources allocated to n by MEC server m are all the remaining computing resources of m. U n The remaining data The computation time executed on MEC server k is shown in formula (11): in represents the computing resources allocated to server k; U n The remaining data The computational energy consumption performed on MEC server k is shown in formula (12): D) When no collaborative MEC server has sufficient computing resources to complete the remaining tasks of user n, the MEC server will do its best to calculate part of the data within the delay constraint. The remaining data Forwarded to the cloud server Cloud; at this time, the computing resources allocated to n by MEC server m are all the remaining computing resources of m; U n The remaining data The computation time executed on the cloud server is shown in formula (13): in Represents the computing resources allocated by the cloud server; U n The remaining data The energy consumption of computing performed on the cloud server is shown in formula (14): Where k2>0 is a constant related to the cloud server CPU; 3) Delay Model For the task U under four decisions n The delay is shown in formulas (15)-(18): A)α n =1, Task U n The total latency of the computational mode executed locally is as follows: in Represents task U n Total latency of computational patterns executed locally; B) Mission U n The total latency of the computational mode offloaded to MEC server m is as follows: in Represents task U n The total latency of the computing mode offloaded to MEC server m; C) Mission U n When MEC server m does not have enough resources to meet user needs, the total latency of the computing mode of offloading part of the tasks to MEC server k is as follows: in Represents task U n Offloading to MEC server m. When MEC server m does not have enough resources to meet user needs, the total latency of the computing mode in which part of the tasks are offloaded to MEC server k for execution; D) Mission U n When the task is unloaded to MEC server m and no collaborative MEC server has enough computing resources to complete the remaining tasks of user n, the MEC server will do its best to calculate part of the data within the delay constraint. The remaining data The total latency of the computing mode forwarded to the cloud server Cloud for execution is calculated as follows: in Represents task U n When the MEC server m does not have enough computing resources to complete the remaining tasks of user n, the MEC server m will transfer the remaining data to the MEC server m. The total latency of the computational model forwarded to the cloud server Cloud for execution; Mission U n The total delay is shown in formula (19): 4) Energy consumption model For the task U under four decisions n The energy consumption is shown in formulas (20)-(23): A)α n =1, Task U n The total energy consumption of the computational mode executed locally is as follows: in Represents task U n Total energy consumption of computational patterns executed locally; B) Mission U n The total energy consumption of the computing mode offloaded to MEC server m is as follows: in Represents task U n The total energy consumption of the computing mode offloaded to the MEC server m; C) Mission U n When MEC server m does not have enough resources to meet user needs, the total energy consumption of the computing mode of offloading part of the tasks to MEC server k is as follows: in Represents task U n Offloading to MEC server m: When MEC server m does not have enough resources to meet user needs, the total energy consumption of the computing mode in which part of the tasks are offloaded to MEC server k for execution; D) Mission U n When the task is unloaded to MEC server m and no collaborative MEC server has enough computing resources to complete the remaining tasks of user n, the MEC server will do its best to calculate part of the data within the delay constraint. The remaining data The total energy consumption of the computing mode forwarded to the cloud server Cloud for execution is calculated as follows: in Represents task U n When the MEC server m does not have enough computing resources to complete the remaining tasks of user n, the MEC server m will transfer the remaining data to the MEC server m. The total energy consumption of the computing mode forwarded to the cloud server Cloud for execution; Mission U n The total energy consumption is shown in formula (24): 5) Energy consumption minimization With the goal of minimizing the total energy consumption of the entire system, including total computing and transmission energy consumption, the offloading decision, computing resources, channels, and power allocation are optimized. The mathematical model p1 is expressed as follows: Where O = {o1, o2, ..., o n } represents the uninstall decision matrix of n users, represents the computing resource allocation matrix, Indicates that server m is assigned to task U n The computing resources, W = {w n,s,m |w n,s,m ∈{0,1},n∈N,s∈S,m∈M} represents the subcarrier allocation matrix, w n,s,m =1 means that subcarrier s will be allocated to user n, and the task offloading calculation will be used for transmission to server m, P = {p n,s,m |p n,s,m ∈(0,P max ],n∈N,s∈S,m∈M} represents the subcarrier power allocation matrix, where p n,s,m represents the transmission power of subcarrier s allocated between user n and server m to offload the task; (25b) represents the value range of the task offloading decision; (25c) declares that the task can be offloaded to at most one MEC server m, and can perform collaborative calculation of the remaining tasks with other servers at most once; (25d) represents the maximum computing resource allocation that can be made to MEC server m. Indicates that server m is assigned to task U n of computing resources, represents the maximum computing resource of server m; (25e) is the maximum transmission power constraint of user n, which is less than or equal to P max , P max represents the maximum transmission power; (25f) and (25g) indicate that each subcarrier is assigned to at most one user; (25h) ensures that each task must be completed under the delay constraint; Step 2: Analyze problem p1 and transform it into a two-layer problem and solve them separately. The steps are as follows: Through further analysis of problem p1, we can find the following two characteristics: first, different offloading decision results will result in different allocated computing resources and wireless channel resources; second, different allocated computing resources and wireless channel resources will result in different optimal offloading decisions; since the optimization variables not only affect each other but can also be divided into two independent sets, problem p1 meets the characteristics of a two-layer optimization problem, and problem p1 can be transformed into a two-layer optimization problem p2; two-layer optimization is to solve the upper-layer optimization problem under the premise of ensuring the optimality of the lower-layer optimization problem; the present invention regards the offloading decision problem and the wireless channel resource allocation problem as the upper-layer optimization problem, and regards the computing resources as the lower-layer optimization problem, and its purpose is to minimize the total energy consumption of all mobile users; Specifically, through model analysis, after offloading decisions, channel and power allocation, based on the task completion time constraint (25h), all tasks must complete the task within the time constraint. At the same time, according to formula (19) The definition of , can be derived from the computational task U n The minimum lower bound of computing resources under different decisions is shown in formula (26): By analyzing the energy consumption calculation formula (10), formula (12) and formula (14) of the computing model, it can be seen that the computing energy consumption is proportional to the computing resources. The lower bound formula (26) of the computing resources is the optimal value of energy consumption, that is, the computing resource allocation; Therefore, the problem p1 can be transformed into a two-level optimization problem p2. Formula (27) includes (27a), (27b), and (27c). Formula (27) is as follows: (25b)-(25h)(27b) Then, we proposed a two-layer optimization algorithm based on deep reinforcement learning to solve problem p2. The upper layer optimization and the lower layer optimization are iterative, and the latter is nested in the former. First, the upper layer solves the unloading decision O k , power distribution P k and channel allocation W k , provided to the lower layer; in the lower layer optimization, the monotonic optimization method is used to obtain the optimal allocation of corresponding computing resources according to the given offloading decision and communication resource allocation r k , calculate the optimal energy consumption Then feedback to the upper level; Step 3: Solve the upper-level optimization problem and solve the unloading decision O based on deep reinforcement learning. k , power distribution P k and channel allocation W k , provided to the lower layer, the steps are as follows: There are three key elements in deep reinforcement learning methods, namely state, action, and reward, which are defined as follows: ·State: st represents the state, S{st1,st2,...,st n ,...,st |N| } represents the state space of the environment, n∈N is the current user, st n (n∈N) means that the nth task is currently executed; Action: a represents an action, which consists of three parts: offloading decision, channel selection, and power allocation. The offloading decision is a vector O = [o1, ..., o n ,...,o |N| ], represents the offloading decision of user n; the channel selection is the vector Π=[ω1,...,ω n ,...,ω |N| ],ω n ={ω n,1 ,...,ω n,s ,...,ω n,|S| } represents the channel selection of user n, where ω n,s =1 indicates that user n is assigned channel s; the power is selected as the vector Λ=[p1,...,p n ,...,p |N| ], where p n ={p n,1 ,...,p n,k ,...,p n,max-1 } represents the power selection of user n, p n,k =1 means according to the maximum transmission power P max The discretized k-th level power is allocated to user n; combined with vectors O, Π, Λ Up , the system action is as shown: a=[o1,...,o n ,...,o |N| ,ω1,...,ω n ,...,ω |N| ,p1,...,p n ,...,p |N| ]; Reward: Generally speaking, the reward function should be related to the objective function. In a certain state st, a reward R is obtained after executing each possible action a. The goal of the optimization problem is to minimize the total energy consumption E sum , because the reward is negatively correlated with the total energy consumption, so -E sum In return, the minimum energy consumption corresponds to the maximum return; We propose a two-layer optimization method based on deep reinforcement learning, which combines DuelingDQN and DoubleDQN techniques, and adds adaptive noise to the internal parameters of the neural network to improve learning efficiency; in the upper layer, the unloading decision O is obtained through action selection. k , power distribution P k and channel allocation W k After that, the lower layer obtains the computing resource allocation r according to the provided offloading decision, power allocation and channel allocation. k , the total energy consumption of the system is calculated Feedback is given to the upper layer as a reward; the upper layer gets the feedback energy consumption and stores it in the experience pool to update the network weight parameters; Initialize iteration = 0, the maximum number of iterations Ι, and set the current state st, that is, user n and the corresponding task U n As environmental information, it is input into the evaluation network, and the output is the Q value corresponding to the action, where the action is unloading decision, power allocation and channel allocation. The action corresponding to the maximum Q value is selected according to the ∈-greedy strategy, or an action is randomly selected and input into the lower layer; Step 4: According to the offloading decision provided by the upper layer k , power distribution P k and channel allocation W k , the lower layer solves the optimal allocation of computing resources through monotonic optimization k , calculate the reward R and feed it back to the upper layer. The steps are as follows: By analyzing the energy consumption calculation formulas (10), (12) and (14) of the computing model, it can be seen that there is a strictly monotonically increasing relationship between computing resources and total computing energy consumption; in order to minimize the total computing energy consumption of all mobile users, the computing resources allocated to each task should be as small as possible; the computing resource lower bound formula (26) is the optimal energy consumption value, that is, the computing resource allocation; after obtaining the offloading decision O k , power distribution P k and channel allocation W k Then, according to formula (26), the lower bound of computing resources can be calculated as computing resource allocation, and then substituted into formula (27) to calculate the total energy consumption of the system After the lower layer obtains the computing resource allocation, it obtains the energy consumption reward R and the next state st', that is, the next task to be executed, and feeds it back to the upper layer; In step 5, the upper layer receives feedback from the lower layer, stores it in the experience pool, and updates the network parameters. The steps are as follows: The upper layer receives feedback and stores a 4-tuple into the experience pool. The 4-tuple includes the current state st, the selected action a, the next state st', and the reward R. Then st' is input into the evaluation network and jumps to step 3 until φ 4-tuples (st, a, R, st') are stored in the experience pool and jumps to step 6. Step 6: Train the evaluation network and the target network. The steps are as follows: i) Sample a part of the 4-tuple (st, a, R, st') stored in the experience pool, input the sampled multiple st as a batch into the evaluation network, and then select the corresponding Q value Q in the evaluation network according to the sampled action a eval (st,a,ω), where ω is the weight parameter of the evaluation network after adding noise, and the calculation formula is shown in (28): ω=θ+N(0,σ 2 ) (28) Among them, θ is the weight parameter of the evaluation network, σ is the noise scale of adding Gaussian noise; N represents the Gaussian distribution function; ii) Input the sampled multiple st' as a batch into the target network to obtain the Q value Q'(st',a') of action a'. Take the largest Q value in Q'(st',a') and multiply it by a coefficient γ, then add the sampled reward R to get Q tar (st',a',ω'), the calculation formula based on DDQN is shown in (29): Among them, ω' is the weight parameter of the target network after adding noise, and the calculation formula is shown in (30): ω'=θ'+N(0,σ 2 ) (30) Among them, θ' is the weight parameter of the target network; iii) Calculate the loss function Back propagation Update the neural network weight parameters: Calculate the loss value based on the two Q values obtained by the two neural networks, that is, the result Q of step 2 eval (st,a,ω) and the result Q of step 3 tar The mean square error of (st', a', ω') is calculated by the iteration number iteration+1 as shown in (31): L(ω)=E[Q tar (st',a',ω')-Q eval (st,a,ω)] 2 (31) The gradient descent method is used to back propagate and update the network weight parameter ω. The calculation formula is shown in (32): Every time you pass After steps, the weight ω of the evaluation network is updated to the target network ω'; Step 7 determines whether the number of algorithm iterations iteration is greater than the maximum iteration number I. If iteration>I, the iteration ends, otherwise jump to step 3 to continue iterating under the new network parameters.
