Adaptive computing offloading method based on server load balancing mechanism in MEC environment

Abstract

The application discloses a kind of self-adaptive computing unloading methods based on server load balancing mechanism under MEC environment, comprising the following steps: S1: multiple user equipment, task, edge base station and server constitute environment;S2: according to optimization target and the load condition of edge server determines system state;S3: based on environment construction Markov decision model;S4: based on the improved DDPG algorithm is solved to Markov decision model;S5: obtain optimal decision action, ensure edge server load balancing, obtain optimal computing unloading strategy.Based on edge server load balancing, introduce random early detection algorithm, propose an improved deep reinforcement learning algorithm to obtain adaptive computing unloading method, effectively relieve the pressure of server side, improve network resource utilization, achieve the goal of relieving network congestion, reducing end-to-end delay and reducing energy consumption.

Description

Technical Field

[0001] This invention relates to the field of computational offloading technology in networks, and more specifically to an adaptive computational offloading method based on server load balancing mechanisms in a MEC environment. Background Technology

[0002] Currently, with the rapid development of internet technology, a large number of smart terminals have entered people's lives. Applications with high latency requirements, such as online interactive video and AR / VR, generate a large amount of computational data during use, putting enormous pressure and challenges on the network. The number of terminals accessing the network is growing exponentially, making traditional cloud computing models unable to adequately meet users' demands for low latency and high speed. Mobile Edge Computing (MEC) effectively addresses the shortcomings of traditional cloud computing models. MEC introduces computing and storage resources at the network edge, reducing the burden on the central cloud and allowing some services and data to be computed and stored at the edge, effectively reducing data transmission latency and alleviating network congestion.

[0003] However, with the explosive growth in the number of terminals and the limited resources of edge servers, it is crucial to rationally distribute large amounts of computational data across edge servers in an edge computing environment to enable rapid task completion and improve resource utilization. Load balancing distributes workloads across multiple servers for processing, thereby preventing excessive pressure on some servers, improving resource utilization, and enhancing overall system performance. Therefore, load balancing is a very important research topic in edge computing. Compute offloading is one of the key technologies of MEC (Multi-access Edge Computing), referring to the offloading of computationally intensive and latency-sensitive tasks from terminal devices to MEC servers for processing, addressing the shortcomings of terminals in terms of resource storage, computing performance, and energy consumption.

[0004] Therefore, obtaining an effective computing offloading strategy based on load balancing mechanisms in edge computing environments is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides an adaptive computation offloading method based on server load balancing mechanism in MEC environment; based on edge server load balancing, a random early detection algorithm is introduced, and an improved deep reinforcement learning algorithm is proposed to obtain the adaptive computation offloading method, which effectively alleviates the pressure on the server side, improves the utilization of network resources, and achieves the goals of alleviating network congestion, reducing end-to-end latency and reducing energy consumption.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] An adaptive computation offloading method based on server load balancing mechanism in a MEC environment includes the following steps:

[0008] S1: Combining multiple user devices, tasks, edge base stations, and servers into an environment;

[0009] S2: Determine the system status based on the optimization objectives and the load of the edge servers;

[0010] S3: Constructing Markov decision models based on the environment;

[0011] S4: Solve the Markov decision model based on the improved DDPG algorithm;

[0012] S5: Obtain the optimal decision action, ensure load balancing of edge servers, and obtain the optimal computing offloading strategy.

[0013] Preferably, step S1 specifically includes:

[0014] The environment includes user equipment, tasks, edge base stations, and servers;

[0015] User Equipment MD m Represented as:

[0016] MD m ={F md ,P,CG,x_position,y_position};

[0017] Among them, F md The computing power of the edge user equipment is represented by CPU cycles / s, P represents the transmission power, CG represents the channel gain between the device and the MEC server, and x_position and y_position represent the coordinates.

[0018] Task m Represented as:

[0019] TASK m ={md, DS, F cpu ,T};

[0020] Where md represents the device that generates the computation task m, DS represents the amount of data to be computed, and F cpu This represents the required CPU resources, and T represents the latency constraint, meaning that the task must be completed within the constraint time, otherwise the data will be invalid.

[0021] Edge base stations are represented as:

[0022] BS n ={F bs,x_position,y_position,range_radius};

[0023] Among them, F bs This represents the server's computing power, measured in CPU cycles / s. x_position and y_position represent coordinates, and range_radius represents the signal range radius of the base station.

[0024] Preferably, step S3 specifically includes:

[0025] The Markov decision model consists of tuples M = {S, A, P, R}, including state function S, action function A, state transition probability P, and reward function R.

[0026] The state function S is specifically represented as follows:

[0027]

[0028] Among them, dec M ={dec1, dec2, ..., dec m} represents the system's unloading decision at a certain moment, pro M ={pro1, pro2, ..., pro m} represents the progress of a computation task, with 0 and 1 representing incomplete and completed tasks, respectively. This indicates the computing resources required to complete the task. This indicates the remaining available resources in the edge server;

[0029] The action function A is specifically represented as follows:

[0030] a t ={x M MEC M ,res M};

[0031] Where, x M ={x1, x2, ..., x n} represents the unloading decision vector for the computational task, x n =0 indicates local execution; x n =1 indicates that computation is offloaded to the edge server. MEC M ={MEC1, MEC2, ..., MEC m}, where MEC m ∈[1, 2, ..., n] indicates that the computational task is offloaded to a certain edge server for execution, res M = {res1, res2, ..., res m}, where resm This indicates the computing resources required for the MEC server to complete the task;

[0032] The reward function R consists of the time delay function T and the energy consumption function ε. m Composition, represented as r t (s, a)=weight*T+(1-weight)*ε m , where s represents the system state, a represents the action taken in a certain state, and weight is the weight parameter for balancing time delay and energy consumption;

[0033] The time delay function T is expressed as:

[0034] T local =F task / F md

[0035] T tran =DS / log2(1+P*CG / N)

[0036] T offloading =F task / F allocation

[0037] T = T local +T tran +T offloading

[0038] Among them, T local It is the local calculation time, T tran It is the time it takes for the task to be unloaded to the edge server, T offloading The calculation time is the time required by the server. DS represents the amount of data to be calculated, P represents the transmission power, CG represents the channel gain between the server and the MEC server, and N is the signal-to-noise power ratio.

[0039] The energy consumption function is expressed as:

[0040]

[0041] ε tran =P tran *T tran

[0042] ε m =ε local +ε tran

[0043] in, This represents the energy consumption per CPU cycle, which depends on the chip architecture on the terminal. k is an energy efficiency parameter, set to 5 * 10 in the text. -5 Ptran represents the wireless signal transmission power.

[0044] Preferably, step S4 specifically includes:

[0045] The improved DDPG algorithm consists of four networks: a local actor network, a target actor network, a local critic network, and a target critic network. The inputs of the local actor network and the target actor network are states, and the outputs are actions. The inputs of the local critic network and the target critic network are states and actions, and the outputs are the ratings of the actions taken in the corresponding states.

[0046] The local actor network interacts with the environment, obtaining action a from state s, executing the action, and the environment returns the next state s' and immediate reward r, represented by a quadruple (s, a, r, s'), stored in the experience replay pool. Samples are batched from the experience replay pool for training. s and a from (s, a, r, s') are input into the local critic network to obtain Q(s, a). Then, s' from (s, a, r, s') is input into the target actor network to obtain action a'. s' and a' are input into the target critic network to obtain Q(s', a'). The local critic network is updated to make its output Q(s, a) close to Q(s', a'). For actions output by the local actor network, the local critic network provides the Q value, and the parameters of the local actor network are updated to maximize the Q value. At fixed intervals, the parameters of the target network are updated using the parameters of the local network.

[0047] As can be seen from the above technical solutions, compared with the prior art, the present invention discloses an adaptive computing offloading method based on server load balancing mechanism in ME C environment; based on edge server load balancing, a random early detection algorithm is introduced, and an improved deep reinforcement learning algorithm is proposed to obtain an adaptive computing offloading method, which effectively alleviates the pressure on the server side, improves the utilization rate of network resources, and achieves the goals of alleviating network congestion, reducing end-to-end latency and reducing energy consumption. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0049] Figure 1 The attached figure is a schematic diagram of the process structure of the adaptive computation unloading method based on the server load balancing mechanism provided by the present invention.

[0050] Figure 2 The attached figure is a schematic diagram of the application framework of the adaptive computing offloading method based on server load balancing mechanism in the MEC environment provided by the present invention.

[0051] Figure 3 The attached figure is a schematic diagram of the DDPG algorithm structure provided by the present invention for solving Markov decision models. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] This invention discloses an adaptive computation offloading method based on a server load balancing mechanism in an MEC environment, comprising the following steps:

[0054] S1: Combining multiple user devices, tasks, edge base stations, and servers into an environment;

[0055] S2: Determine the system state based on the optimization goal and the load of the edge server, reduce the time delay and energy consumption of completing the computing task, reduce the load imbalance of the server, and determine the system state based on the goal, offloading decision, the progress of computing tasks and the available resources of the edge server.

[0056] S3: Constructing Markov decision models based on the environment;

[0057] S4: Solve the Markov decision model based on the improved DDPG algorithm;

[0058] S5: Obtain the optimal decision action, ensure load balancing of edge servers, obtain the optimal computation offloading strategy, use the DDPG algorithm to learn from past experience, update the policy network, and based on the system state, use the policy network to obtain the action that maximizes the expected reward value and minimizes the server load imbalance.

[0059] To further optimize the above technical solution, step S1 specifically includes:

[0060] The environment includes user equipment, tasks, edge base stations, and servers;

[0061] User Equipment MD m Represented as:

[0062] MD m ={F md ,P,CG,x_position,y_position};

[0063] Among them, F mdThe computing power of the edge user equipment is represented by CPU cycles / s, P represents the transmission power, CG represents the channel gain between the device and the MEC server, and x_position and y_position represent the coordinates.

[0064] Task m Represented as:

[0065] TASK m ={md, DS, F cpu ,T};

[0066] Where md represents the device that generates the computation task m, DS represents the amount of data to be computed, and F cpu This represents the required CPU resources, and T represents the latency constraint, meaning that the task must be completed within the constraint time, otherwise the data will be invalid.

[0067] Edge base stations are represented as:

[0068] BS n ={F bs , x_position, y_position, range_radius};

[0069] Among them, F bs This represents the server's computing power, measured in CPU cycles / s. x_position and y_position represent coordinates, and range_radius represents the signal range radius of the base station.

[0070] To further optimize the above technical solution, step S3 specifically includes:

[0071] The Markov decision model consists of tuples M = {S, A, P, R}, including state function S, action function A, state transition probability P, and reward function R.

[0072] The state function S is specifically represented as follows:

[0073]

[0074] Among them, dec M ={dec1, dec2, ..., dec m} represents the system's unloading decision at a certain moment, pro M ={pro1, pr02, ..., pro m} represents the progress of a computation task, with 0 and 1 representing incomplete and completed tasks, respectively. This indicates the computing resources required to complete the task. This indicates the remaining available resources in the edge server;

[0075] The action function A is specifically represented as follows:

[0076] a t ={x M MEC M ,res M};

[0077] Where, x M ={x1, x2, ..., x n} represents the unloading decision vector for the computational task, x n =0 indicates local execution; x n =1 indicates that computation is offloaded to the edge server. MEC M ={MEC1, MEC2, ..., MEC m}, where MEC m ∈[1, 2, ..., n] indicates that the computational task is offloaded to a certain edge server for execution, res M = {res1, res2, ..., res m}, where res m This indicates the computing resources required for the MEC server to complete the task;

[0078] The reward function R consists of the time delay function T and the energy consumption function ε. m Composition, represented as r t (s, a)=weight*T+(1-weight)*ε m , where s represents the system state, a represents the action taken in a certain state, and weight is the weight parameter for balancing time delay and energy consumption;

[0079] The time delay function T is expressed as:

[0080] T local =F task / F md

[0081] T tran =DS / log2(1+P*CG / N)

[0082] T offloading =F task / F allocation

[0083] T = T local +T tran +T offloading

[0084] Among them, T local It is the local calculation time, T tran It is the time it takes for the task to be unloaded to the edge server, T offloadingThe calculation time is the time required by the server. DS represents the amount of data to be calculated, P represents the transmission power, CG represents the channel gain between the server and the MEC server, and N is the signal-to-noise power ratio.

[0085] The energy consumption function is expressed as:

[0086]

[0087] E tran =P tran *T tran

[0088] ε m =ε local +ε tran

[0089] in, This represents the energy consumption per CPU cycle, which depends on the chip architecture on the terminal. k is an energy efficiency parameter, set to 5 * 10 in the text. -5 P tran This indicates the power of the wireless signal transmission.

[0090] To further optimize the above technical solution, step S4 specifically includes:

[0091] The improved DDPG algorithm consists of four networks: a local actor network, a target actor network, a local critic network, and a target critic network. The inputs of the local actor network and the target actor network are states, and the outputs are actions. The inputs of the local critic network and the target critic network are states and actions, and the outputs are the ratings of the actions taken in the corresponding states.

[0092] The local actor network interacts with the environment, obtaining action a from state s, executing the action, and the environment returns the next state s' and immediate reward r, represented by a quadruple (s, a, r, s'), stored in the experience replay pool. Samples are batched from the experience replay pool for training. s and a from (s, a, r, s') are input into the local critic network to obtain Q(s, a). Then, s' from (s, a, r, s') is input into the target actor network to obtain action a'. s' and a' are input into the target critic network to obtain Q(s', a'). The local critic network is updated to make its output Q(s, a) close to Q(s', a'). For actions output by the local actor network, the local critic network provides the Q value, and the parameters of the local actor network are updated to maximize the Q value. At fixed intervals, the parameters of the target network are updated using the parameters of the local network.

[0093] The goal of this example is to obtain the optimal compute offloading strategy to ensure load balancing on edge servers and reduce the expected total cost (latency and energy consumption). Its expression is: This expression is the objective optimization function in the DDPG algorithm, which is the reward function in the Markov decision model. It is the objective of the algorithm optimization, that is, to minimize the expected reward value obtained by performing action a in state s during the algorithm optimization process. In the previous text, our reward function was expressed as a weighted value of time delay and energy consumption.

[0094] To further optimize the above technical solution, step S5 specifically includes:

[0095] The DDPG algorithm is used to learn from past experiences. First, the agent obtains the initial state of the environment. actor network μ(s|θ) μ Based on state s t Get action a t ={x M MEC M ,res M}, perform action a t Receive the new state s′ and the immediate reward r t (s, a), will (s) t a t r t s i+1 The data is saved to the experience replay pool (Replay_buff), and samples are obtained by randomly sampling from the experience replay pool. i a i r i s i+1 ), to obtain y i =r i +γQ′(s i+1 ,μ′(S i+1 |θ μ )|θ Q Minimize the loss function Strategy for updating the actor network: Update strategy network parameters θ μ′ and θ Q′ Iterative updates are performed to obtain the optimal strategy, thereby minimizing server load imbalance.

[0096] It has the following beneficial effects:

[0097] (1) This invention introduces an edge server load balancing mechanism and Markov modeling to address the computation offloading problem. Edge computing servers in different locations serve different ranges. As the number of computing tasks increases, some edge servers may experience load, leading to network congestion and traffic overload. Load balancing is used to offload the computation, alleviating system pressure and improving efficiency. Based on the research on the computation offloading problem, this invention establishes a Markov decision model, which is beneficial for deep reinforcement learning algorithms to solve the model, find the optimal computation offloading strategy, and uses a stochastic early detection algorithm to balance server load and improve network resource utilization.

[0098] (2) This invention designs a computation offloading strategy based on an improved DDPG algorithm for edge server load balancing. The problem scenario in this invention consists of multiple edge computing servers and multiple user terminals. Different servers have different service ranges, and the terminals are distributed in different locations, generating computation task data. The computation tasks can be processed locally or offloaded to reachable edge servers for execution. Considering the time delay, energy consumption, and server load status during the computation offloading process, a stochastic early detection algorithm is introduced based on the established Markov decision model. The optimal computation offloading strategy is solved using the improved DDPG algorithm to minimize the latency and energy consumption of the entire model and reduce server load imbalance.

[0099] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0100] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1.A method for adaptive computation offloading based on server load balancing mechanism in MEC environment, characterized in that, The method comprises the following steps: S1: constructing an environment comprising a plurality of user equipment, tasks, edge base stations and servers; S2: determining a system state according to an optimization target and a load condition of an edge server; S3: constructing a Markov decision model based on the environment; Specifically comprising: The Markov decision model is composed of a tuple M={S, A, P, R}, comprising a state function S, an action function A, a state transition probability P and a reward function R; The state function S is specifically represented as follows: ； where dec M = {dec1, dec2, …, dec m} denotes the offloading decision of the system at a certain time, pro M = {pro1, pro2, …, pro m} denotes the progress of the computing task, with 0 and 1 representing not completed and completed, denotes the computing resources needed to complete the task, denotes the remaining available resources in the edge server; The action function A is specifically represented as follows: a t = {x M , MEC M , res M} ; Where, x M ={x1, x2, ..., x n } represents the unloading decision vector for the computational task, x n =0 indicates local execution; x n =1 indicates that computation is offloaded to the edge server; MEC M ={MEC1, MEC2, ..., MEC m }, where MEC m ∈[1, 2, ..., n] indicates that the computational task is offloaded to a certain edge server for execution, res M = {res1, res2, ..., res m }, where res m This indicates the computing resources required for the MEC server to complete the task; The reward function R is composed of a latency function T and an energy consumption function ε m and is expressed as r t (s, a) = weight * T + (1 - weight) * ε m where s represents the system state, a represents the action taken at a certain state, and weight is a weight parameter balancing the time delay and energy consumption. The time delay function T is represented as: T local = F task / F md T tran = DS / log2(1 + P * CG / N) T offloading = F task / F allocation T = T local + T tran + T offloading where T local is the local computation time, T tran is the time for task offloading to the edge server, T offloading is the time for computation at the server, DS represents the amount of data that needs to be computed, P represents the transmission power, CG represents the channel gain between the MEC server, and N is the signal-to-noise power ratio; The energy consumption function is represented as: ； e tran = P tran T tran ε m = ε local + ε tran wherein, represents the energy consumption per CPU cycle, which depends on the chip structure on the terminal, k is the energy efficiency parameter, and in this paper, k = 5*10 -5 , P tran represents the wireless signal transmission power; S4: solving the Markov decision model based on an improved DDPG algorithm; S5: obtaining an optimal decision action, ensuring edge server load balancing, and obtaining an optimal computing offloading strategy. 2.The method of adaptive computation offloading based on server load balancing mechanism in MEC environment according to claim 1, characterized in that, The step S1 specifically comprises: The environment comprises user equipment, tasks, edge base stations and servers; User equipment MD m is represented as: MD m = {F md , P, CG, x_position, y_position}; where F md represents the computing capability of the edge user equipment, the unit is CPU cycles / s, P represents the transmission power, CG represents the channel gain between the MEC server, and x_position and y_position represent the coordinates. Task TASK m is represented as: TASK m = {md, DS, F cpu , T}; where md represents the device that generates the computation task m, DS represents the amount of data that needs to be computed, F cpu represents the required CPU resources, and T represents the latency constraint, i.e., the task needs to be completed within the constraint time, otherwise the data will expire. The edge base station is represented as: BS n = {F bs , x_position, y_position, range_radius}; where F bs represents the computing power of the server, in CPU cycles / s, x_position and y_position represent the coordinates, and range_radius represents the signal range radius of the base station. 3.The method of claim 1, wherein, The step S4 specifically comprises: The improved DDPG algorithm is composed of four networks, namely a local actor network, a target actor network, a local critic network and a target critic network; the input of the local actor network and the target actor network is a state, and the output is an action; the input of the local critic network and the target critic network is a state and an action, and the output is a score of the action taken under the corresponding state; The local actor network interacts with the environment to obtain an action a through a state s, executes the action, and the environment returns a next time state s' and an immediate reward r, which are represented by a four-tuple (s, a, r, s'), and are stored in an experience replay pool; samples are taken from the experience replay pool in batches for training; s and a in (s, a, r, s') are input into the local critic network to obtain Q(s, a), then s' in (s, a, r, s') is input into the target actor network to obtain an action a'; s' and a' are input into the target critic network to obtain Q(s', a'), the local critic network is updated so that the output Q(s, a) is close to Q(s', a'), the local actor network outputs an action, and the local critic network gives a Q value, the parameters of the local actor network are updated to maximize the Q value; every fixed interval, the parameters of the local network are used to update the parameters of the target network.

Images