A method for dual-timescale task offloading and resource allocation in a multi-timeslot MEC system
By employing a dual-timescale method and the D3QN algorithm based on deep reinforcement learning in a multi-timeslot MEC system, the problem of high computational complexity in task unloading and resource allocation is solved, and the optimal solution is obtained quickly.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTHEAST UNIV
- Filing Date
- 2023-10-24
- Publication Date
- 2026-05-26
AI Technical Summary
In multi-timeslot mobile edge computing systems, existing technologies struggle to effectively solve the mixed-integer nonlinear programming problem of task offloading and resource allocation, resulting in high computational complexity and slow solution speed. In particular, it is difficult to compute an optimal solution within a reasonable timeframe when dealing with large-scale base stations and users.
A dual-timescale approach is adopted, which decouples the task offloading and resource allocation problems through the Lyapunov method, and combines the D3QN algorithm of deep reinforcement learning to learn server mode selection and user-server association decisions, thereby reducing computational complexity.
It effectively reduces the computational complexity of multi-timeslot systems, improves the solution speed, and ensures that the optimal solution is obtained within the effective time.
Smart Images
Figure CN117354934B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of mobile edge computing resource allocation technology, and in particular to a dual-timescale task offloading and resource allocation method for a multi-timeslot MEC system. Background Technology
[0002] Mobile Edge Computing (MEC) leverages wireless access networks to provide telecommunications users with the IT services and cloud computing capabilities they need, creating a high-performance, low-latency, and high-bandwidth carrier-grade service environment that accelerates the rapid download of various content, services, and applications within the network. In the caching design of MEC systems, wireless access points (APs) and base stations (BSs) deploying MEC servers can pre-cache computing tasks or results. Wireless devices can then directly obtain computing results from the APs without needing to offload or perform local computation. Jointly designing computing task caching, computing resource allocation, and computing offloading effectively improves the performance of MEC systems.
[0003] In dynamic environments, the caching decisions of mobile edge servers need to adapt to both time-varying wireless channel states and randomly arriving user tasks. Currently, for task offloading and resource allocation in multi-timeslot mobile edge computing, traditional convex optimization methods are mainly used, along with heuristic algorithms to search for optimal solutions. Since task offloading and resource allocation in mobile edge computing are typically a mixed-integer nonlinear programming problem, and the multiple time slots increase the complexity of the solution, on the one hand, relaxing the 0-1 integer programming of task offloading cannot yield the optimal solution; on the other hand, when the number of base stations and users reaches a certain scale and the number of time slots increases, the computational complexity of heuristic algorithms grows exponentially, making it difficult to compute a solution to the optimization problem within a reasonable timeframe. Summary of the Invention
[0004] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a dual-timescale task offloading and resource allocation method for multi-timeslot MEC systems, which can reduce the computational complexity of solving multi-timeslot systems and improve the solution speed.
[0005] The objective of this invention can be achieved through the following technical solution: a method for dual-timescale task offloading and resource allocation in a multi-timeslot MEC system, comprising the following steps:
[0006] S1. For multi-user, multi-server MEC networks, establish a long-term average energy consumption minimization model for the system.
[0007] S2. Solve the long-term average energy consumption minimization model of the system on two time scales:
[0008] At a small time scale, by invoking the Lyapunov method, the long-term random task offloading and resource allocation problem is decoupled into a series of deterministic problems of online optimization, and the closed-form solution of offloading decision and resource allocation at each time slot is obtained.
[0009] On a large time scale, the problem of server active / sleep mode selection and user-server association decision is formulated as a constrained Markov decision process, and a dual adversarial deep Q network (D3QN) in deep reinforcement learning is used to learn mode selection and association decision.
[0010] Furthermore, the long-term average energy consumption minimization model of the system in step S1 is specifically as follows:
[0011]
[0012]
[0013]
[0014]
[0015]
[0016]
[0017]
[0018]
[0019]
[0020]
[0021]
[0022]
[0023]
[0024] Where i is the user's index, j is the server's index, t is the index of the time slot on a small time scale, and n is the index of the time frame on a large time scale, with each frame containing T time slots; ζ j (n) is a server activity / sleep indicator variable, ξ ij (n) is a user-server connection indicator variable, δ i (t) represents the proportion of tasks that are unloaded from the user to the server, p i (t) is the user's transmit power, f i (t) is the user's local computing speed, g j(t) is the server's computing speed; It calculates the queue length locally. Y is the length of the local unload queue. j (t) represents the server-side calculation queue length; This is the user's maximum local computing speed. It is the user's maximum power. It is the server's maximum computing speed, κ u It is the effective capacitance coefficient of the processor chip; E M (t) represents the energy consumption of all servers, E U (t) represents the energy consumption of all users.
[0025] Furthermore, the energy consumption of all the servers is as follows:
[0026]
[0027]
[0028]
[0029] The energy consumption of all users is as follows:
[0030]
[0031] Where τ is the time slot length of the calculation and communication timeline, and P a It is the energy consumption required to maintain an active state, P c Ps is the maximum computing power consumption, while Ps is the power consumption required to maintain sleep mode.
[0032] Furthermore, step S2 specifically includes the following steps:
[0033] S21. Decompose the dual-timescale optimization problem into server mode selection and user-server association decision-making problems on the large timescale, and task unloading and resource allocation problems on the small timescale.
[0034] S22. Construct a server active / sleep mode selection and user-server association decision problem on a large time scale, and formulate it as a constrained Markov process. Use the D3QN algorithm of deep reinforcement learning to solve it.
[0035] Furthermore, step S21 specifically includes the following steps:
[0036] S211. First, assuming that the server mode selection and user-server association decision are given, construct the task unloading and resource allocation problem on a small time scale.
[0037] S212. Construct a virtual queue and use the Lyapunov method to handle the long-term average objective and constraints, transforming the task offloading and resource allocation problem constructed in step S211 into a series of single-slot online optimization problems.
[0038] S213. The single-slot online optimization problem is decomposed into three sub-problems for solution. The three sub-problems include the unloading decision problem, the resource allocation problem on the user side, and the resource allocation problem on the server side.
[0039] Furthermore, the task unloading and resource allocation problem at a small time scale in step S211 specifically refers to:
[0040]
[0041] Furthermore, the specific process of step S212 is as follows:
[0042] First, construct a virtual queue using the following expression:
[0043]
[0044]
[0045]
[0046] Thus, the long-term average constraint can be satisfied by the average rate stability of the virtual queue, as expressed in the following expression:
[0047]
[0048]
[0049]
[0050] Construct the Lyapunov function, whose expression is:
[0051]
[0052] in, It is a combined queue vector, where α1, α2, and α3 are scaling factors;
[0053] The conditional Lyapunov drift plus penalty function is constructed, and its expression is:
[0054]
[0055] Where V is the weight of the penalty function;
[0056] The upper bound of the Lyapunov drift plus penalty function is derived as follows:
[0057]
[0058] Where C is a constant, ω(t) is a quantity that depends only on the queue length and is independent of the optimization variables; the expression for Q(t) is as follows:
[0059]
[0060] Furthermore, for the above upper bound expression to hold, it must be subject to the following three constraints:
[0061]
[0062]
[0063]
[0064] in, This is the maximum transmission rate between the user and the server;
[0065] Minimizing the upper bound of the conditional Lyapunov drift plus penalty function stabilizes the queue while minimizing the system's average energy consumption. Furthermore, increasing V yields asymptotic optimality. Based on the chance expectation minimization technique, minimizing the upper bound of the conditional Lyapunov drift plus penalty function is achieved by minimizing the deterministic problem for each time slot. The deterministic problem for each time slot is described as follows:
[0066]
[0067] Furthermore, step S22 specifically includes the following steps:
[0068] S221. Given the optimal offloading decision and resource allocation for each time slot, we obtain the server active / sleep mode selection and user-server association decision problem on a large time scale.
[0069] S222. Design a deep reinforcement learning method, which expresses the problem obtained in step S221 as a constrained Markov process, performs reinforcement learning based on the D3QN network architecture, and solves for the server active / sleep mode and user-server association decision results.
[0070] Furthermore, the server active / sleep mode selection and user-server association decision problem on a large time scale in step S221 are specifically as follows:
[0071]
[0072] Furthermore, the reinforcement learning process based on the D3QN network architecture in step S222 includes:
[0073] Define the state, action, and reward, and determine the expression of the Q function within the D3QN network architecture;
[0074] The training process updates the parameters of the evaluation network by minimizing the loss function;
[0075] After calculating the loss function, an evaluation network based on gradient descent is trained.
[0076] Furthermore, the state is as follows:
[0077]
[0078] The action is as follows:
[0079]
[0080] The reward is:
[0081]
[0082] The expression for the Q function is:
[0083]
[0084] Among them, V(S) n ;θ V D(S) is the state value function. n A n ;θ D ) is the dominance function, and Q(·;θ) represents the parameter θ=(θ V θ D ) target network, It refers to the size of the action space;
[0085] At the start of the nth frame, the system observes the queue and channel conditions to obtain state S. n ;
[0086] Then, action A is selected based on the ∈-greedy policy. n Specifically, the system either selects an action for exploration based on probability ∈ (random) or probability 1 ∈ (action). To maximize the Q value, through action A n Calculate the optimal task unloading and resource allocation within n frames, and obtain the reward R. n and the next state S n+1 ;
[0087] Set the current state S n Action A n Reward R n and the next state S n+1 Will be stored in the experience replay cache In each training step, D3QN randomly draws a mini-batch of historical experience data (Si) from the experience replay pool. n A n R n S n+1 );
[0088] The training process updates the parameters of the evaluation network by minimizing the loss function, which is expressed as:
[0089]
[0090]
[0091] Among them, y n Q′(·θ′) is the target value, and Q′(·θ′) is the target network with parameter θ′.
[0092] After calculating the loss function, an evaluation network based on gradient descent is trained. The parameters θ′ of the target network are updated by θ′=μθ+(1-μ)θ′, where μ represents the update rate.
[0093] Compared with the prior art, the present invention has the following advantages:
[0094] This invention first establishes a long-term average energy consumption minimization model for a multi-user, multi-server MEC network and solves it on two time scales: on a smaller time scale, the long-term stochastic task offloading and resource allocation problem is decoupled into a series of deterministic problems that can be optimized online by invoking the Lyapunov method; on a larger time scale, the server active / sleep mode selection and user-server association decision problem is formulated as a constrained Markov decision process, and a bi-adversarial deep Q-network (D3QN) from deep reinforcement learning is used to learn the mode selection and association decisions. Compared with traditional optimization methods, this invention can effectively reduce the computational complexity of multi-time-slot systems, thereby improving the solution speed. Attached Figure Description
[0095] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation
[0096] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.
[0097] Example
[0098] like Figure 1 As shown, a method for dual-timescale task offloading and resource allocation in a multi-timeslot MEC system includes the following steps:
[0099] S1. For multi-user, multi-server MEC networks, establish a long-term average energy consumption minimization model for the system.
[0100] S2. Solve the long-term average energy consumption minimization model of the system on two time scales:
[0101] At a small time scale, by invoking the Lyapunov method, the long-term random task offloading and resource allocation problem is decoupled into a series of deterministic problems of online optimization, and the closed-form solution of offloading decision and resource allocation at each time slot is obtained.
[0102] On a large time scale, the problem of server active / sleep mode selection and user-server association decision is formulated as a constrained Markov decision process, and a dual adversarial deep Q network (D3QN) in deep reinforcement learning is used to learn mode selection and association decision.
[0103] Specifically, the process includes the following:
[0104] Step A: For a multi-user, multi-server cellless MEC network, establish a long-term average energy minimization model A1. Model A1 is expressed as follows:
[0105]
[0106]
[0107]
[0108]
[0109]
[0110]
[0111]
[0112]
[0113]
[0114]
[0115]
[0116]
[0117]
[0118] Where i is the user's index, j is the server's index, t is the index of the time slot on a small time scale, and n is the index of the time frame on a large time scale, with each frame containing T time slots; ζ j(n) is a server activity / sleep indicator variable, ξ ij (n) is a user-server connection indicator variable, δ i (t) represents the proportion of tasks that are unloaded from the user to the server, p i (t) is the user's transmit power, f i (t) is the user's local computing speed, g j (t) is the server's computing speed; It calculates the queue length locally. Y is the length of the local unload queue. j (t) represents the server-side calculation queue length; This is the user's maximum local computing speed. It is the user's maximum power. It is the server's maximum computing speed, κ u It is the effective capacitance coefficient of the processor chip; E M (t) represents the energy consumption of all servers, E U (t) represents the energy consumption of all users.
[0119] Step B: Decompose the dual-timescale optimization problem into a server mode selection and user-server association decision problem on the large timescale, and a task unloading and resource allocation problem on the small timescale, as described below:
[0120] Step B-1: First, assuming the server mode selection and user-server association decisions are given, construct a task unloading and resource allocation problem on a small time scale, expressed as:
[0121]
[0122]
[0123]
[0124]
[0125]
[0126]
[0127]
[0128]
[0129]
[0130] Step B-2: Construct a virtual queue and use the Lyapunov method to handle the long-term average objective and constraints, transforming the problem into a series of single-slot online optimization problems. First, construct the virtual queue, expressed as:
[0131]
[0132]
[0133]
[0134] Thus, the long-term average constraint can be satisfied by the average rate stability of the virtual queue, as expressed in the following expression:
[0135]
[0136]
[0137]
[0138] Construct the Lyapunov function, whose expression is:
[0139]
[0140] in, It is a combined queue vector, where α1, α2, and α3 are scaling factors.
[0141] The conditional Lyapunov drift plus penalty function is constructed, and its expression is:
[0142]
[0143] Where V is the weight of the penalty function.
[0144] The upper bound of the Lyapunov drift plus penalty function is derived as follows:
[0145]
[0146] Where C is a constant, ω(t) is a quantity that depends only on the queue length and is independent of the optimization variables; the expression for Q(t) is as follows:
[0147]
[0148] Furthermore, for the above upper bound expression to hold, it must be subject to the following three constraints:
[0149]
[0150]
[0151]
[0152] in, It is the maximum transmission rate between the user and the server.
[0153] Minimizing the upper bound of the conditional Lyapunov drift plus penalty function stabilizes the queue, minimizes the system's average energy consumption, and asymptotic optimality can be achieved by increasing V. According to the chance expectation minimization technique, minimizing the upper bound of the conditional Lyapunov drift plus penalty function can be achieved by minimizing the deterministic problem for each time slot, which is formulated as follows:
[0154]
[0155]
[0156]
[0157]
[0158]
[0159]
[0160] Step B-3: The single-slot deterministic optimization problem described above can be decomposed into three sub-problems for solution: the offloading decision problem, the resource allocation problem on the user side, and the resource allocation problem on the server side. These three sub-problems are all convex problems, and closed-form solutions can be derived.
[0161] In this way, given the server active / sleep mode selection and user-server association decision, the optimal solution for offloading decision and resource allocation in each time slot can be obtained.
[0162] Step C: Construct a server activity / sleep mode selection and user-server association decision problem over a large time scale, and formulate it as a constrained Markov process. Solve it using the D3QN algorithm based on deep reinforcement learning. The specific steps are as follows:
[0163] Step C-1: Given the optimal offloading decision and resource allocation for each time slot, the server active / sleep mode selection and user-server association decision problem on a large time scale are obtained, which can be expressed as:
[0164]
[0165]
[0166]
[0167]
[0168]
[0169] Step C-2: Design a deep reinforcement learning method to formulate the above problem as a constrained Markov process, and define the state, action, and reward as follows:
[0170] state:
[0171] action:
[0172] award:
[0173] Step C-3: Under the D3QN network architecture, the expression for the Q function is:
[0174]
[0175] Among them, V(S) n ;θ V D(S) is the state value function. n A n ;θ D ) is the dominance function, and Q(·;θ) represents the parameter θ=(θ V θ D ) target network, It refers to the size of the action space.
[0176] At the start of the nth frame, the system observes the queue and channel conditions to obtain state S. n Then, action A is selected based on the ∈-greedy policy. n Specifically, the system either selects an action for exploration based on probability ∈ (randomly chosen) or on probability 1-∈ (selected by choice). To maximize the Q value. Through action A n Calculate the optimal task unloading and resource allocation within n frames, and obtain the reward R. n and the next state S n+1 Current state S n Action A n Reward R n and the next state S n+1 Will be stored in the experience replay cache In each training step, D3QN randomly draws a mini-batch (S) of historical experience data from the experience replay pool. n A n R n S n+1 ).
[0177] Step C-4: The training process updates the parameters of the evaluation network by minimizing the loss function, the expression of which is:
[0178]
[0179] Among them, y n It is the target value, expressed as:
[0180]
[0181] Where Q′(·;θ′) is the target network with parameter θ′.
[0182] Step C-5: After calculating the loss function, train the evaluation network based on gradient descent. The parameters θ′ of the target network are updated using θ′=μθ+(1μ)θ′, where μ represents the update rate.
[0183] This embodiment applies the above technical solution to a cellless MEC network consisting of M servers and U users, where each server is integrated into an access point. The sets of users and servers are respectively... and Users have the ability to perform computational tasks locally. However, due to limited computing power for executing computationally intensive applications, users can offload their tasks to a server, with the time slot length of the computation and communication timeline being τ.
[0184] To achieve a green network, servers can selectively switch to sleep mode, reducing power consumption while maintaining network efficiency. For this purpose, a dual-time-scale framework is considered. Specifically, each consecutive T time slots are grouped into a time frame. The frame index is denoted by n, n∈{0,1,…,N-1}, and the set of time slots for the nth frame is denoted by n. express.
[0185] The first step is to analyze the binary active / sleep indicator variable ζ on a large timescale. j (n) = 1 indicates that the server is active. Binary user-server association indicator variable ξ ij (n) = 1 indicates that user i can offload its task to server j. The server active / sleep mode selection and user-server association rules can be expressed as follows:
[0186]
[0187]
[0188]
[0189]
[0190] The second step, on a small time scale, uses A i (t) represents the arrival of a user's task. The arrival of tasks is independent and identically distributed in time, and the average arrival rate is λ. i ,Right now Because the communication circuits and computing units are separate, each user can simultaneously perform local computation and task offloading. Therefore, each user divides their task arrival into two disjoint parts, namely... and in and δ represents the amount of tasks executed locally and the amount of tasks offloaded to the server, respectively. i (t)∈[0,1] is the partitioning parameter. Let (in bits) represents the length of the local computation queue for the i-th user, then the dynamic expression for the queue is:
[0191]
[0192] Among them, f i (t) (in rounds per second) represents the local CPU cycle frequency. By utilizing dynamic voltage and frequency scaling techniques, f can be adaptively controlled. i (t). φ i >0 indicates the number of CPU rounds required to compute each bit of data locally.
[0193] For the user's task unloading queue, the queue backlog is represented as... The dynamic expression of a queue is
[0194]
[0195] Where, r ij (t) is the unloading rate.
[0196] On the server side, let Y j (t) represents the backlog of computational tasks waiting to be processed by the server, and its dynamic expression is:
[0197]
[0198] Among them, g j (t) represents the server's computing speed.
[0199] The third step, to ensure limited delay, is to impose constraints on the queue, expressed as:
[0200]
[0201]
[0202]
[0203] The fourth step is to construct an energy consumption model. First, the computing and transmission energy consumption at the user end must be allocated, satisfying the following conditions:
[0204]
[0205] User energy consumption is divided into two parts: 1) energy consumption for local computation, and 2) energy consumption for task offloading. Therefore, the total energy consumption for all users is...
[0206]
[0207] Server-side energy consumption is divided into the energy consumption of active servers and the energy consumption of sleeping servers. The energy consumption expression for active servers is as follows:
[0208]
[0209] Among them, P a It is the energy consumption required to maintain an active state, P c It is the largest energy consumption for computing.
[0210] The energy consumption expression for a sleep server is:
[0211]
[0212] Among them, P s This is the energy required to maintain a sleep state.
[0213] The total energy consumption of all servers is expressed as follows:
[0214]
[0215] Step 5: Establish a model for optimizing task offloading and resource allocation in a multi-timescale MEC system:
[0216]
[0217]
[0218]
[0219]
[0220]
[0221]
[0222]
[0223]
[0224]
[0225]
[0226]
[0227]
[0228]
[0229] The process of combining the dual-timescale Lyapunov method with deep reinforcement learning is as follows:
[0230] Step 1: Decompose the dual-timescale optimization problem into a server mode selection and user-server association decision problem on a large timescale, and a task unloading and resource allocation problem on a small timescale, as detailed below:
[0231] Step 1-1: First, assuming that the server mode selection and user-server association decisions are given, construct a task unloading and resource allocation problem on a small time scale, which is expressed as:
[0232]
[0233]
[0234]
[0235]
[0236]
[0237]
[0238]
[0239]
[0240]
[0241] Step 1-2: Construct a virtual queue and use the Lyapunov method to handle the long-term average objective and constraints, transforming the problem into a series of single-slot online optimization problems. First, construct the virtual queue, expressed as:
[0242]
[0243]
[0244]
[0245] Thus, the long-term average constraint can be satisfied by the average rate stability of the virtual queue, as expressed in the following expression:
[0246]
[0247]
[0248]
[0249] Construct the Lyapunov function, whose expression is:
[0250]
[0251] in, It is a combined queue vector, where α1, α2, and α3 are scaling factors.
[0252] The conditional Lyapunov drift plus penalty function is constructed, and its expression is:
[0253]
[0254] Where V is the weight of the penalty function.
[0255] The upper bound of the Lyapunov drift plus penalty function is derived as follows:
[0256]
[0257] Where C is a constant, ω(t) is a quantity that depends only on the queue length and is independent of the optimization variables; the expression for Q(t) is as follows:
[0258]
[0259] Furthermore, for the above upper bound expression to hold, it must be subject to the following three constraints:
[0260]
[0261]
[0262]
[0263] in, It is the maximum transmission rate between the user and the server.
[0264] Minimizing the upper bound of the conditional Lyapunov drift plus penalty function stabilizes the queue, minimizes the system's average energy consumption, and asymptotic optimality can be achieved by increasing V. According to the chance expectation minimization technique, minimizing the upper bound of the conditional Lyapunov drift plus penalty function can be achieved by minimizing the deterministic problem for each time slot, which is formulated as follows:
[0265]
[0266]
[0267]
[0268]
[0269]
[0270]
[0271] Steps 1-3: The single-slot deterministic optimization problem described above can be decomposed into three sub-problems for solution: the offloading decision problem, the resource allocation problem on the user side, and the resource allocation problem on the server side. These three sub-problems are all convex problems, and closed-form solutions can be derived.
[0272] In this way, given the server active / sleep mode selection and user-server association decision, the optimal solution for offloading decision and resource allocation in each time slot can be obtained.
[0273] Step 2: Construct a server activity / sleep mode selection and user-server association decision problem on a large time scale, and formulate it as a constrained Markov process. The D3QN algorithm based on deep reinforcement learning is used to solve it. The specific steps are as follows:
[0274] Step 2-1: Given the optimal offloading decision and resource allocation for each time slot, the server active / sleep mode selection and user-server association decision problem on a large time scale are obtained, which can be expressed as:
[0275]
[0276]
[0277]
[0278]
[0279]
[0280] Step 2-2: Design a deep reinforcement learning method, formulating the above problem as a constrained Markov process, and defining the state, action, and reward as follows:
[0281] state:
[0282] action:
[0283] award:
[0284] Steps 2-3: Under the D3QN network architecture, the expression for the Q function is:
[0285]
[0286] Among them, V(S) n ;θ V D(S) is the state value function. n A n ;θ D ) is the dominance function, and Q(·;θ) represents the parameter θ=(θ V θ D ) target network, It refers to the size of the action space.
[0287] At the start of the nth frame, the system observes the queue and channel conditions to obtain state S. n Then, action A is selected based on the ∈-greedy policy. n Specifically, the system either selects an action for exploration based on probability ∈ (random) or probability 1 ∈ (action ≠ 1). To maximize the Q value. Through action A n Calculate the optimal task unloading and resource allocation within n frames, and obtain the reward R. n and the next state S n+1 Current state S n Action A n Reward R n and the next state S n+1 Will be stored in the experience replay cache In each training step, D3QN randomly draws a mini-batch (S) of historical experience data from the experience replay pool. n A n R n S n+1 ).
[0288] Steps 2-4: The training process updates the parameters of the evaluation network by minimizing the loss function, the expression of which is:
[0289]
[0290] Among them, y n It is the target value, expressed as:
[0291]
[0292] Where Q′(·;θ′) is the target network with parameter θ′.
[0293] Steps 2-5: After calculating the loss function, train the evaluation network based on gradient descent. The parameters θ′ of the target network are updated using θ′=μθ+(1-μ)θ′, where μ represents the update rate.
[0294] In summary, this solution is designed for the optimization of multi-user, multi-timeslot mobile edge computing systems. By establishing a model that minimizes the long-term average energy consumption of the system and solving it on two time scales, the computational complexity of the multi-timeslot system can be effectively reduced, thereby ensuring that the optimal solution can be obtained quickly within the effective time.
Claims
1. A method for dual-timescale task offloading and resource allocation in a multi-timeslot MEC system, characterized in that, Includes the following steps: S1. For multi-user, multi-server MEC networks, establish a long-term average energy consumption minimization model for the system. S2. Solve the long-term average energy consumption minimization model of the system on two time scales: At a small time scale, by invoking the Lyapunov method, the long-term random task offloading and resource allocation problem is decoupled into a series of deterministic problems of online optimization, and the closed-form solution of offloading decision and resource allocation at each time slot is obtained. On a large time scale, the server active / sleep mode selection and user-server association decision problem is formulated as a constrained Markov decision process, and D3QN in deep reinforcement learning is used to learn the mode selection and association decision. Specifically, we first assume that the server mode selection and user-server association decisions are given, and then construct a task unloading and resource allocation problem on a small time scale: ； Then, a virtual queue is constructed, and the Lyapunov method is used to handle the long-term average objective and constraints, transforming the constructed task unloading and resource allocation problem into a series of single-slot online optimization problems; The single-slot online optimization problem is further decomposed into three sub-problems for solution. These three sub-problems include the offloading decision problem, the resource allocation problem on the user side, and the resource allocation problem on the server side. The specific problems concerning server active / sleep mode selection and user-server association decision-making on a large time scale are as follows: ； in, It is the user's index. It is the server's index. It is an index of time slots on a small time scale. It is an index of time frames on a large time scale, each frame containing One time slot; It is a server activity / sleep indicator variable. It is a user-server connection indicator variable. It represents the percentage of tasks that are unloaded from the user to the server. It is the user's transmit power. It is the user's local computing speed. It refers to the server's computing speed; It calculates the queue length locally. This is the length of the local unload queue. The server calculates the queue length. This is the user's maximum local computing speed. It is the user's maximum power. It is the server's maximum computing speed. It is the effective capacitance coefficient of the processor chip; This is the energy consumption of all servers. This represents the energy consumption of all users.
2. The method for dual-timescale task offloading and resource allocation in a multi-timeslot MEC system according to claim 1, characterized in that, The system's long-term average energy consumption minimization model in step S1 is specifically as follows: ； in, It is the user's index. It is the server's index. It is an index of time slots on a small time scale. It is an index of time frames on a large time scale, each frame containing One time slot; It is a server activity / sleep indicator variable. It is a user-server connection indicator variable. It represents the percentage of tasks that are unloaded from the user to the server. It is the user's transmit power. It is the user's local computing speed. It refers to the server's computing speed; It calculates the queue length locally. This is the length of the local unload queue. The server calculates the queue length. This is the user's maximum local computing speed. It is the user's maximum power. It is the server's maximum computing speed. It is the effective capacitance coefficient of the processor chip; This is the energy consumption of all servers. This represents the energy consumption of all users.
3. The method for dual-timescale task offloading and resource allocation in a multi-timeslot MEC system according to claim 2, characterized in that, The energy consumption of all servers is as follows: ； ； ； The energy consumption of all users is as follows: ； in, To calculate and communicate the time slot length, It is the energy required to maintain an active state. It is the largest computing power consumption. This is the energy required to maintain a sleep state.
4. The method for dual-timescale task offloading and resource allocation in a multi-timeslot MEC system according to claim 3, characterized in that, Step S2 specifically includes the following steps: S21. Decompose the dual-timescale optimization problem into server mode selection and user-server association decision-making problems on the large timescale, and task unloading and resource allocation problems on the small timescale. S22. Construct a server active / sleep mode selection and user-server association decision problem on a large time scale, and formulate it as a constrained Markov process. Use the D3QN algorithm of deep reinforcement learning to solve it.
5. The method for dual-timescale task offloading and resource allocation in a multi-timeslot MEC system according to claim 1, characterized in that, The specific process of transforming the task unloading and resource allocation problem into a series of single-slot online optimization problems is as follows: First, construct a virtual queue using the following expression: ； Thus, the long-term average constraint can be satisfied by the average rate stability of the virtual queue, as expressed in the following expression: ； Construct the Lyapunov function, whose expression is: ； in, It is a combined queue vector. , and It is the scaling factor; The conditional Lyapunov drift plus penalty function is constructed, and its expression is: ； in, It is the weight of the penalty function; The upper bound of the Lyapunov drift plus penalty function is derived as follows: ； in, It is a constant. It is a quantity that is only related to the queue length and is unrelated to optimization variables; The expression is as follows: ； Furthermore, for the above upper bound expression to hold, it must be subject to the following three constraints: ； in, This is the maximum transmission rate between the user and the server; Minimize the upper bound of the Lyapunov drift plus penalty function to ensure queue stability, while simultaneously minimizing the system's average energy consumption, and by increasing... Asymptotic optimality can be obtained. According to the chance expectation minimization technique, minimizing the upper bound of the conditional Lyapunov drift plus penalty function is achieved by minimizing the deterministic problem of each time slot, which is expressed as follows: 。 6. The method for dual-timescale task offloading and resource allocation in a multi-timeslot MEC system according to claim 4, characterized in that, Step S22 specifically includes the following steps: S221. Given the optimal offloading decision and resource allocation for each time slot, we obtain the server active / sleep mode selection and user-server association decision problem on a large time scale. S222. Design a deep reinforcement learning method, which expresses the problem obtained in step S221 as a constrained Markov process, performs reinforcement learning based on the D3QN network architecture, and solves for the server active / sleep mode and user-server association decision results.
7. The method for dual-timescale task offloading and resource allocation in a multi-timeslot MEC system according to claim 6, characterized in that, The reinforcement learning process based on the D3QN network architecture in step S222 includes: Define the state, action, and reward, and determine the expression of the Q function within the D3QN network architecture; The training process updates the parameters of the evaluation network by minimizing the loss function; After calculating the loss function, an evaluation network based on gradient descent is trained.
8. A method for dual-timescale task offloading and resource allocation in a multi-timeslot MEC system according to claim 7, characterized in that, The state is as follows: ； The action is as follows: ； The reward is: ； The expression for the Q function is: ； in, It is a state-value function. It is the dominant function. The parameter is The target network, It refers to the size of the action space; In the At the start of a frame, the system observes the queue and channel conditions to obtain the state. ; Then, based on -greedy strategy selects action Specifically, the system depends on probability. Explore by randomly selecting actions, or by probability. Select Action To maximize Value, through action ,calculate The best task unloading and resource allocation within the frame will earn rewards. and the next state ; Current state ,action ,award and the next state Will be stored in the experience replay cache In each training step, D3QN randomly draws mini-batch data from historical experiences from the experience replay pool. ; The training process updates the parameters of the evaluation network by minimizing the loss function, which is expressed as: ； ； in, It is the target value. The parameter is The target network; After calculating the loss function, train the evaluation network based on gradient descent, and the parameters of the target network are... pass Update Indicates the update rate.