A method for resource allocation in mobile edge cloud computing
By modeling the mobile edge cloud computing network as a two-stage serial queue system and using a reinforcement learning network structure, the problem of end-to-end latency guarantee is solved, achieving efficient resource allocation and statistical latency guarantee, thus adapting to complex computing migration scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- YANGTZE DELTA REGION INST (QUZHOU) UNIV OF ELECTRONIC SCI & TECH OF CHINA
- Filing Date
- 2023-04-25
- Publication Date
- 2026-06-02
AI Technical Summary
Existing technologies cannot effectively guarantee end-to-end latency in mobile edge cloud computing, and existing methods lack generalization and have many limitations, making it difficult to find solutions, especially when the problem scale is large.
The mobile edge cloud computing network is modeled as a two-stage concatenated queue system. A reinforcement learning network structure is used, including an Actor network, a Critic network, and an experience replay buffer. The queue relationships are captured through gated recurrent units, and resource allocation is optimized by combining a deep deterministic policy gradient algorithm to build a predictive model to adapt to latency requirements.
It provides statistical latency guarantees for end-device compute migration while using as few resources as possible, adapting to complex dynamics, and is universally applicable, capable of handling random compute migration requests and service time.
Smart Images
Figure CN116489710B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of mobile edge computing technology, and specifically relates to a method for resource allocation in mobile edge cloud computing. Background Technology
[0002] With the rapid development and widespread application of cloud computing, IoT technology, and edge computing, more and more enterprises or individuals are starting to use edge or cloud servers in research projects to handle some computationally intensive tasks that end devices cannot complete. Because edge computing is in an emerging state, the deployment cost is very high. Cloud services usually have two charging models: time-based charging and resource usage-based charging. Improper use can lead to a huge waste of resources.
[0003] One of the most critical metrics for migrating computing tasks from mobile devices to cloud or edge servers is end-to-end latency. If this latency cannot be guaranteed, the migration is often meaningless and could even have serious consequences. Therefore, many researchers have proposed computation migration methods that guarantee quality of service. These methods typically employ multi-objective optimization to arrive at the final allocation strategy, considering metrics such as service time, energy consumption, and power allocation. Some researchers use traditional mathematical methods to solve multi-objective nonconvex optimization problems, such as mixed-integer linear programming, mixed-integer nonlinear programming, and game theory, and then use iterative, heuristic, and decomposition methods to solve them. However, these methods lack generalization and have many constraints, especially when the problem size is large, and may not find a solution. Summary of the Invention
[0004] To address the problems mentioned in the background art, the present invention provides a method for resource allocation in mobile edge cloud computing, which solves the problems of existing technologies that cannot guarantee end-to-end latency, lack of generalization, and numerous limitations.
[0005] To achieve the above objectives, the present invention provides the following technical solution:
[0006] A method for resource allocation in mobile edge cloud computing includes the following steps:
[0007] S1: The environment includes a serial queuing system, which includes an edge processing queue and a cloud processing queue. Compute migration requests are generated in two processes and are processed continuously. The first process is processed by the server in the edge processing queue, and the second process is processed by the server in the cloud processing queue.
[0008] S2: Initialize the controller;
[0009] S3: Construct a reinforcement learning network structure, which includes: an Actor network, a Critic network, and an experience replay buffer;
[0010] S4: In the Actor network, the sequence of length vectors of an edge processing queue and a cloud processing queue is converted into a two-dimensional tensor as input data. After introducing a gated recurrent unit to capture the correlation between the edge processing queue and the cloud processing queue in the input data, the two-dimensional tensor of the relationship is output. The two-dimensional tensor of the relationship is then reshaped into a one-dimensional tensor and fed into the first fully connected neural network DNN. The first fully connected neural network DNN is activated by the ReLU function and then outputs the action.
[0011] S5: Reshape the state tensor of the environment into a one-dimensional tensor in the Critic network;
[0012] S6: Feed the one-dimensional tensor of the environment's state into the second fully connected neural network (DNN) and activate it using the ReLU function to obtain the first output;
[0013] S7: The output action in the Actor network is fed into a third fully connected neural network (DNN) to extract features, and then activated using the ReLU function to obtain the second output;
[0014] S8: Use a connection layer to connect the first and second outputs side by side, feed them into the last fully connected neural network DNN, output a linear combination function that evaluates the action of the Actor network output and obtains the trained prediction model.
[0015] S9: After receiving the linear combination function and obtaining the trained prediction model, the controller updates its network parameters and allocation strategy based on the state, action, and reward, and automatically adjusts the allocation of computing resources according to the number of requests and available computing resources to adapt to latency requirements.
[0016] Preferably, the state, action, and reward in S9 are as follows:
[0017] State: The state is defined as the length of the edge processing queue and the cloud processing queue at the beginning of each time step in the serial queue system, i.e., state s = (q1, q2);
[0018] Action: An action is a numerical vector used to determine the computing speed of servers in the edge processing queue and the cloud processing queue, i.e., a = μ θ (s), and a deep deterministic strategy gradient algorithm is used to control the speed;
[0019] Reward: The reward is used to reflect the action a taken within the time interval [t, t+T]. t Assess the environmental impact and benefits of the action, using A. tLet A represent the set of computation migration requests that arrive within the time interval [t, t+T]. t ={co1, co2, ..., co n For each computation migration task, the end-to-end latency does not exceed the maximum tolerable latency, i.e., exceeding d. max This leads to the function for calculating the immediate reward:
[0020]
[0021] co n ∈A t ;
[0022] d n Let set A be t The end-to-end latency of computational migration task n in the process;
[0023] The total reward function is:
[0024]
[0025] The objective function is derived from the reward function:
[0026]
[0027] This expected value is used to limit end-to-end latency and the total amount of resources used.
[0028] Preferably, random noise is added to the output action of S4.
[0029] Preferably, the migration request and the corresponding service time data are generated using a gamma distribution, and this data will be used as input data for the serial queue system.
[0030] Preferably, in S4, the ReLU function is activated by a hyperbolic tangent function through a fully connected layer, and the output ranges from -1 to 1.
[0031] Compared with the prior art, the beneficial effects of the present invention are:
[0032] 1. The method proposed in this invention models the computational migration in the mobile edge cloud computing network as a two-stage serial queue system. In order to capture the linear relationship between the two queues, a gated recurrent unit is added to the reinforcement learning network, which reduces the state search space of reinforcement learning and improves the performance of the prediction model. The queue system also realizes the decoupling and asynchronous requests of multiple users to cloud services.
[0033] 2. The method proposed in this invention provides a statistical guarantee for the end-to-end latency of terminal device computation migration, while using as few resources as possible.
[0034] 3. The method proposed in this invention takes into account the complex dynamics of cloud computing-based computing migration networks. Computing migration requests can arrive at and be processed randomly in the serial system. Furthermore, the service time for each task can also be random. This makes the method universally applicable. Attached Figure Description
[0035] Figure 1 This is a schematic diagram of the computing resource allocation framework of the present invention;
[0036] Figure 2 This is a schematic diagram of the reinforcement learning network structure of the present invention;
[0037] Figure 3-1 This is a schematic diagram of the average end-to-end delay of the GRU model;
[0038] Figure 3-2 A schematic diagram of the average end-to-end delay of the FC model;
[0039] Figure 3-3 This is a schematic diagram of the average end-to-end delay of the LSTM model.
[0040] Figure 3-4 This is a schematic diagram illustrating the overall computational speed of the GRU model;
[0041] Figure 3-5 This is a schematic diagram illustrating the overall computational speed of the FC model;
[0042] Figure 3-6 This is a schematic diagram illustrating the overall computational speed of the LSTM model;
[0043] Figure 3-7 This is a schematic diagram of the average reward of the GRU model;
[0044] Figure 3-8 This is a schematic diagram of the average reward of the FC model;
[0045] Figure 3-9 This is a schematic diagram illustrating the average reward of the LSTM model.
[0046] Figure 4-1 For η a A diagram illustrating the total calculation speed of 0.95.
[0047] Figure 4-2 For η a A diagram illustrating the probability of violation of 0.95;
[0048] Figure 4-3 For η a =1.5 Schematic diagram of total calculation speed;
[0049] Figure 4-4 For η a A diagram illustrating the probability of violation of 1.5;
[0050] Figure 4-5 For η a A diagram illustrating the total calculation speed of 0.4.
[0051] Figure 4-6 For η a A diagram illustrating the probability of violation of 0.4. Detailed Implementation
[0052] To facilitate understanding of the technical content of this invention by those skilled in the art, the invention will be further described in detail below with reference to the accompanying drawings and specific examples. It should be understood that the specific examples described herein are merely illustrative and not intended to limit the scope of the invention.
[0053] Example 1
[0054] like Figure 1 , Figure 2 As shown, a method for resource allocation in mobile edge cloud computing includes the following steps:
[0055] S1: The environment includes a serial queuing system, which includes an edge processing queue and a cloud processing queue. The computation migration request is divided into two sub-processes and generated continuously. The first sub-process is processed by the server in the edge processing queue, and the second sub-process is processed by the server in the cloud processing queue.
[0056] S2: Initialize the controller and select the computing speed based on the length of the edge processing queue and the cloud processing queue. The initial length of the edge processing queue and the cloud processing queue is set to 0.
[0057] S3: Construct a reinforcement learning network structure, which includes: an Actor network, a Critic network, and an experience replay buffer;
[0058] S4: In the Actor network, the sequence of length vectors of an edge processing queue and a cloud processing queue is converted into a two-dimensional tensor as input data. A gated recurrent unit is introduced to capture the correlation between the edge processing queue and the cloud processing queue in the input data. The two-dimensional tensor of the relationship is output and then reshaped into a one-dimensional tensor and fed into the first fully connected neural network DNN. The ReLU function is passed through a fully connected layer and activated with the hyperbolic tangent function. The output range is between -1 and 1. After the first fully connected neural network DNN is activated by the ReLU function, it outputs an action and adds random noise to the output action so that the agent can explore more actions with potentially huge benefits.
[0059] S5: Reshape the state tensor of the environment into a one-dimensional tensor in the Critic network;
[0060] S6: Feed the one-dimensional tensor of the environment's state into the second fully connected neural network (DNN) and activate it using the ReLU function to obtain the first output;
[0061] S7: The output action in the Actor network is fed into a fully connected neural network (DNN) to extract features, and then activated using the ReLU function to obtain a second output;
[0062] S8: Use a connection layer to connect the first and second outputs side by side, feed them into the last fully connected neural network DNN, output a linear combination function that evaluates the action of the Actor network output and obtains the trained prediction model.
[0063] S9: After receiving the linear combination function and obtaining the trained prediction model, the controller updates its network parameters and allocation strategy based on the state, action, and reward, and automatically adjusts the allocation of computing resources according to the number of requests and available computing resources to adapt to latency requirements.
[0064] In this embodiment, the method proposed in this invention models computation migration in mobile edge cloud computing networks as a two-stage serial queue system. To capture the linear relationship between the two queues, a gated recurrent unit is added to the reinforcement learning network, reducing the state search space and improving model performance. The queue system also enables decoupling and asynchronous requests for cloud services from multiple users. The method provides statistical guarantees for end-to-end latency in edge device computation migration while using minimal resources. It considers the complex dynamics of cloud-based computation migration networks, allowing migration requests to arrive at and be processed randomly within the serial system. Furthermore, the service time for each task can also be random. This makes the method universally applicable.
[0065] Example 2
[0066] The difference between this embodiment and embodiment 1 is that the state, action, and reward in S9 are specifically as follows:
[0067] State: The state is defined as the length of the edge processing queue and the cloud processing queue at the beginning of each time step in the serial queue system, i.e., state s = (q1, q2);
[0068] Action: An action is a numerical vector used to determine the computing speed of servers in the edge processing queue and the cloud processing queue, i.e., a = μ θ (s), and a deep deterministic strategy gradient algorithm is used to control the speed;
[0069] Reward: The reward is used to reflect the action a taken within the time interval [t, t+T]. t Assess the environmental impact and benefits of the action, using A.t Let A represent the set of computation migration requests that arrive within the time interval [t, t+T]. t ={co1, co2, ..., co n For each computation migration task, the end-to-end latency does not exceed the maximum tolerable latency, i.e., exceeding d. max This leads to the function for calculating the immediate reward:
[0070]
[0071] co n ∈A t ;
[0072] d n Let set A be t The end-to-end latency of computational migration task n in the process;
[0073] The total reward function is:
[0074]
[0075] The objective function is derived from the reward function:
[0076]
[0077] This expected value is used to limit end-to-end latency and the total amount of resources used.
[0078] In this embodiment, the optimal resource allocation strategy is obtained by training a deep reinforcement learning model in this stage. Reinforcement learning methods are suitable for scenarios requiring real-time sequential decision-making. Furthermore, if the state space is large, deep reinforcement learning algorithms can be used to find the optimal solution. Reinforcement learning involves an agent learning knowledge through continuous interaction with the environment and then taking actions to maximize the final reward. Through timely feedback from the environment, i.e., immediate rewards, the agent gradually understands which actions will yield greater rewards and which will yield smaller rewards in certain states; ultimately, the agent forms an optimal strategy.
[0079] Example 3
[0080] The difference between this embodiment and embodiment 2 is that the migration request and the corresponding service time data are generated through gamma distribution, and this data will be used as the input data of the serial queue system.
[0081] In this embodiment, much of the current research in computational transfer studies is based on static simulation datasets, meaning that computational transfer requests arrive at the processing system at a fixed, uniform rate, or are assumed to follow a specific, simple probability distribution. However, this limits the applicability of the proposed method to only certain simple scenarios. In real-world IoT device computational transfer scenarios, the situation is much more complex. The process of multiple IoT devices generating computational transfer requests is completely random or follows a random distribution function. The arrival time and corresponding service time of computational transfer requests are generated using a gamma distribution, and this data serves as the input to a serial queue system. Referring to Embodiment 2, the following section compares the training process of this application with other deep learning models and the performance of baseline models.
[0082] Experiment 1: Comparison with training processes of other deep learning models
[0083] First, the method proposed in this invention, which uses a gated recurrent unit (GRU) network for reinforcement learning, is compared with methods using other neural networks. Other models include those using Long Short-Term Memory (LSTM) networks and fully connected neural networks (DNNs, FC). Each DRL model is trained multiple times using multiple different random seeds. After determining the network structure, multiple experiments are needed to find the optimal hyperparameter values. Experimental results are as follows... Figures 3-1 to 3-9 As shown, the GRU controller exhibited significant latency fluctuations in the first 400 rounds, subsequently stabilizing at around 2.5 except for a few spikes, reaching the predetermined target. The FC controller's end-to-end latency quickly converged to around 1 with minimal fluctuations. Finally, the LSTM controller generally showed larger fluctuations, eventually converging to approximately 5, and then fluctuating between 4 and 6.
[0084] The GRU controller initially used an unstable total amount of computing resources, ranging from 2 to 14. Due to continuous optimization of the allocation strategy, it first converged to around 7 and finally stabilized at around 6. Simultaneously, the average latency of the GRU controller met expectations, not exceeding 5. While the FC controller also met the requirement of latency not exceeding the maximum latency, its total computing resource usage reached 16, far exceeding the other two controllers, which significantly wasted computing resources on edge and cloud servers. The LSTM controller exhibited large fluctuations in the first 350 training epochs, with significant fluctuations around the 450th epoch, before converging to around 7. In summary, the FC-based controller converged quickly and met end-to-end latency requirements, but its resource usage increased rapidly. The other two RNN-based controllers were able to capture the dependencies between sequential input data, thus allocating resources more rationally. The GRU-based controller also met latency requirements. However, the LSTM-based controller did not meet the expected latency requirements. Furthermore, extensive training experiments demonstrated that the fully FC-based controller is highly prone to getting trapped in local optima and cannot escape them without human intervention.
[0085] The average reward values of GRU and FC controllers fluctuated significantly in the early stages, gradually converging as the number of training epochs increased. However, the reward value of the LSTM controller fluctuated considerably throughout the training process, ranging from -14 to -8. This is because the reward value is determined by latency and the amount of computational resources used. Although its resource usage was not high, the latency fluctuations were large, causing fluctuations in the average reward value. However, it can also be observed that its average reward value is mostly higher than that of the FC controller.
[0086] Experiment 2: Performance Comparison with Baseline Model
[0087] To demonstrate the correctness, effectiveness, and dynamic adjustment capabilities of the proposed model, two baseline models, AQL and MVN, were used in the testing phase for comparative experiments with a DRL-based controller. The AQL controller automatically adjusts the computing speed of the compute server based on changes in the length of edge and cloud processing queues. Specifically, if the queue length gradually increases, indicating a need for more computing resources to meet the computing migration requests from end devices, the AQL controller increases the server's computing speed to ensure that blocked tasks can be completed in a timely manner. The MVN controller outputs an average computing speed suitable for the current system based on available computing resources. This method is both simple and efficient when there is a thorough understanding of the operating environment. Furthermore, adding random noise to the final action value can slightly improve overall performance.
[0088] The DRL controller proposed in this invention can automatically allocate appropriate computing resources while ensuring good performance, that is, meeting the QoS requirements of computing migration tasks, thus achieving the expected goal. According to Figures 4-1 to 4-6 As shown, most probability values are below the dashed line, except for a few spikes. These spikes are due to a sudden increase in the number of computational migration tasks. Furthermore, the MVN controller consumes more computational resources than the other two controllers because it adjusts its allocation strategy based on the total available computational resources. Therefore, it cannot quickly correct for sudden situations.
[0089] The DRL controller proposed in this invention can automatically adjust the server's computing speed to meet the increased computing migration requirements as the request arrival rate increases. However, the other two controllers do not perform well as the average request arrival rate increases. Although they meet the end-to-end latency requirements, they require more computing resources than the DRL controller proposed in this invention. In particular, the AQL controller significantly increases the server's computing speed when the average request arrival rate increases slightly, which means it wastes a lot of computing resources. The controller proposed in this invention still performs well even when the average request arrival rate decreases. However, the AQL controller still cannot adapt to changes in the average arrival rate, so it cannot meet the end-to-end latency requirements. Furthermore, it was found that its latency violation probability fluctuated much more during testing than when the arrival rate increased, indicating that its performance is unstable.
Claims
1. A method for resource allocation in mobile edge cloud computing, characterized in that, Includes the following steps: S1: The environment includes a serial queuing system, which includes an edge processing queue and a cloud processing queue. Compute migration requests are generated in two processes and are processed continuously. The first process is processed by the server in the edge processing queue, and the second process is processed by the server in the cloud processing queue. S2: Initialize the controller; S3: Construct a reinforcement learning network structure, which includes: an Actor network, a Critic network, and an experience replay buffer; S4: In the Actor network, the sequence of length vectors of an edge processing queue and a cloud processing queue is converted into a two-dimensional tensor as input data. After introducing a gated recurrent unit to capture the correlation between the edge processing queue and the cloud processing queue in the input data, the two-dimensional tensor of the relationship is output. The two-dimensional tensor of the relationship is then reshaped into a one-dimensional tensor and fed into the first fully connected neural network DNN. The first fully connected neural network DNN is activated by the ReLU function and then outputs the action. In S4, the ReLU function is passed through a fully connected layer and activated by the hyperbolic tangent function, with the output ranging from -1 to 1; S5: Reshape the state tensor of the environment into a one-dimensional tensor in the Critic network; S6: Feed the one-dimensional tensor of the environment's state into the second fully connected neural network (DNN) and activate it using the ReLU function to obtain the first output; S7: The output action in the Actor network is fed into a third fully connected neural network (DNN) to extract features, and then activated using the ReLU function to obtain the second output; S8: Use a connection layer to connect the first and second outputs side by side, feed them into the last fully connected neural network DNN, output a linear combination function that evaluates the action of the Actor network output and obtains the trained prediction model. S9: After receiving the linear combination function and obtaining the trained prediction model, the controller updates its network parameters and allocation strategy based on the state, action and reward, and automatically adjusts the allocation of computing resources according to the number of requests and available computing resources to adapt to latency requirements. The specific states, actions, and rewards in S9 are as follows: State: The state is defined as the length of the edge processing queue and the cloud processing queue at the beginning of each time step in the concatenated queue system. ; Actions: Actions are numerical vectors used to determine the computing speed of servers in the edge processing queue and cloud processing queue. And a deep deterministic gradient algorithm is used to control the speed; Rewards: Rewards are used to reflect... Actions taken within the time interval The environmental impact and the benefits of the action are assessed. Indicates the time interval The set of computation migration requests reached within. For each computation migration task, the end-to-end latency does not exceed the maximum tolerable latency, i.e., exceeding... This leads to the function for calculating the immediate reward: ； Represents a set Computational migration tasks End-to-end delay; The total reward function is: ； The objective function is derived from the reward function: ； This expected value is used to limit end-to-end latency and the total amount of resources used.
2. The method for resource allocation in mobile edge computing according to claim 1, characterized in that, Add random noise to the output action of S4.
3. The method for resource allocation in mobile edge computing according to claim 1, characterized in that, The migration requests and corresponding service time data are generated using a gamma distribution, and this data will serve as the input data for the serial queue system.