Edge service migration method based on deep reinforcement learning
The deep neural network is trained through the deep deterministic policy gradient method to generate direct output service migration decisions, solving the problem of low service transfer learning efficiency in dynamic edge environments, significantly improving migration performance and user QoS.
Patent Information
- Application Number
- CN202411005631.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-25
AI Technical Summary
In dynamic edge environments, the existing service transfer method based on deep reinforcement learning is inefficient in learning, making it difficult to obtain the optimal transfer strategy in the huge action space, resulting in high task response delays and affecting QoS.
The service migration method based on deep deterministic strategy gradient is adopted, and the state space, action space and reward functions are defined through the deep reinforcement learning framework. The deep neural network is trained using the Markov decision-making process to generate direct output migration decisions to cope with complex dynamic environments.
In various scenarios, the performance of service migration has been significantly improved, faster convergence speed and better convergence effect have been achieved, and the QoS performance of users has been improved.
Smart Images

Figure CN118939400A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of communication network technology, and in particular to an edge service migration method based on deep reinforcement learning. Background Art
[0002] With the development of 5G communication technology and the popularization of mobile devices, various emerging intelligent applications continue to emerge, such as real-time video streaming, online games, augmented reality (AR) and virtual reality (VR). Such applications are usually time-sensitive and data-intensive, and rely on powerful computing power to ensure high quality of service (QoS), which poses a huge challenge to the performance of mobile devices. Cloud computing provides powerful processing capabilities by deploying server clusters remotely. However, since users are geographically far away from the remote cloud, it takes a lot of time to transmit data, which conflicts with the real-time requirements of intelligent applications. To solve this problem, Mobile Edge Computing (MEC) achieves the delivery of low-latency, high-bandwidth and high-reliability services by deploying computing and storage resources at the edge of the network. User devices can offload tasks to MEC servers for processing to improve the task response speed of intelligent applications. At the same time, the MEC architecture allows user data to be processed at the edge without uploading it to the remote cloud, effectively enhancing the security and privacy of the data. In addition, virtualization technology is regarded as a key resource allocation and management technology. When the user device offloads the task to the MEC server through the base station (BSs), the MEC server will use virtualization technology to create a dedicated service instance for the user and allocate certain computing resources to it.
[0003] Considering the user's mobility and the limited communication coverage of BSs, the user may be far away from the MEC server currently serving him, which will lead to a further degradation of QoS. Therefore, it is necessary to dynamically migrate the user's service instance to follow the user's mobility to better guarantee QoS. There is resource contention between service instances of different users. When the number of users in the system surges, it may lead to potential resource conflicts and performance degradation. In addition, the performance of service migration depends on multiple factors, such as user location, task data volume, task computing volume, etc. Inappropriate migration strategy will lead to excessive task response delay and seriously affect QoS.
[0004] Existing solutions usually rely on rules, heuristics or optimization theory. These methods usually rely on prior knowledge of the system to formulate strategies, but it is difficult to obtain the above information in a dynamic and changing environment. In addition, they usually require frequent adjustment of model parameters and involve a large number of iterative operations and a huge solution space search, which increases the computational overhead. Therefore, when facing the problem of service migration in a dynamic edge environment, it is necessary to explore more effective solutions to achieve efficient and adaptive service migration. As one of the important branches of machine learning, Deep Reinforcement Learning (DRL) is considered to be an effective solution to the problem of dynamic resource optimization. Through interaction with the environment, the agent can gradually adjust its strategy and obtain the maximum long-term cumulative reward in the environment. Most of the existing DRL-based service migration methods use value-based DRL. This type of method learns deterministic strategies by comparing the Q values of all candidate actions, but the huge action space in the edge system will cause this type of method to have low learning efficiency.
[0005] In order to solve the above problems, this paper proposes a deep deterministic policy gradient based service migration (DPSM) method to explore the optimal service migration strategy in edge environments. The DPSM method directly outputs migration decisions to cope with the huge action space and shows superior performance in various scenarios. Summary of the invention
[0006] The present invention proposes an edge service migration method based on deep reinforcement learning, which is used to explore the optimal service migration strategy in the edge environment. This method directly outputs migration decisions to cope with the huge action space and exhibits superior performance in various scenarios.
[0007] The present invention adopts the following technical solutions.
[0008] An edge service migration method based on deep reinforcement learning is proposed. The method uses a unified service migration model for the dynamic and changeable edge system environment, takes long-term QoS as the optimization goal, and uses migration, communication and computing delays for measurement. At the same time, based on the deep reinforcement learning DRL framework, the state space, action space and reward function of the service migration problem in the MEC environment are defined, and the above problem is formally represented as a Markov decision process MDP. At the same time, in the edge service migration method DPSM based on deep reinforcement learning, deep deterministic policy gradient training deep neural network is adopted to obtain the optimal migration strategy in the complex and dynamic edge environment.
[0009] The edge system consists of an MEC controller, M BS base stations and U users; each base station is equipped with a MEC server to form an edge node; the set of M edge nodes is recorded as M = {1, 2, ..., m, ..., M}, and the set of U users is recorded as U = {1, 2, ..., u, ..., U};
[0010] The edge system operates in discrete time slots, and the user's location changes at the beginning of each time slot T = {1, 2, ..., t, ..., T}. The smart application running on the mobile device of user u generates a computing task Task in each time slot. t (u);
[0011] Assuming that the local computing power of the user's mobile device is limited and its task requests need to be continuously offloaded to the MEC server for processing, then at the start time slot, the mobile device accesses the system through the BS and creates a service instance on the corresponding MEC server; the task requests of different mobile devices are processed by their respective service instances, and these service instances maintain parallel operation by occupying part of the computing resources of the MEC server.
[0012] The migration delay is calculated as follows: Defined as the service instance of user u, at the edge node where time slot t is located, Defined as the service instance of user u, at the edge node in time slot t-1, the system determines whether it can accept service migration by periodically querying the load of all edge nodes. Define a t (u) is the service migration decision of user u in time slot t, which includes all available edge nodes within the user’s access range, using d t To measure and The jump distance between Indicates the delay in unit migration. Indicates the amount of service data, Indicates the backhaul link bandwidth;
[0013] When t = 0, no service migration is required; otherwise, the service Relocation Therefore, the migration delay is expressed as follows:
[0014]
[0015] The communication delay is calculated as follows: Defined as the edge node to which the service instance of user u is connected in time slot t; mobile user u offloads the computing task to This process involves: users and The access delay, and and The return delay of the mobile device is p u , the unit channel gain is α, and the Gaussian noise power is σ 2 , user and The distance is The amount of task data is The signal-to-noise ratio is defined as follows:
[0016]
[0017] Define the total bandwidth of the BS as B, and evenly distribute the bandwidth to all mobile devices in the area in an orthogonal manner; then the access delay is defined as:
[0018]
[0019] Among them, B t (u) represents the available bandwidth of user u.
[0020] when When the service instance is transmitted via the backhaul link. represents the backhaul link bandwidth, Represents the single-hop transmission delay coefficient; similarly, using y t express and The backhaul link transmission delay is defined as:
[0021]
[0022] The total communication delay is defined as follows:
[0023] E t (u) = R t (u)+P t (u) Formula 5.
[0024] The calculation delay is calculated as follows: the task calculation density is denoted as k t (u), use measure The total load at time slot t, F represents the server computing capacity, and assuming that the computing resources allocated to user u are proportional to the resources required for its task, the computing delay is defined as follows:
[0025]
[0026] The optimization goal is specifically: within the time range T, the optimization goal of the proposed MEC system is to minimize the long-term system delay, which is formalized as the optimization problem P1 as follows:
[0027]
[0028] For the optimization problem P1, a service migration method based on deep deterministic policy gradient is used to form the optimal migration strategy in the dynamic edge environment.
[0029] In the service migration method based on deep deterministic policy gradient, during the optimization process of service migration, the DRL agent of the DRL framework is in the current state s according to the strategy π t Next select an action t , immediate reward r for environmental feedback t and transfer to the new state s t+1 , the process is expressed as MDP; at the same time, the sample (s t , a t , r t ,s t+1 ) will be stored in the experience pool, and the DRL agent will be trained by extracting small batch samples;
[0030] The state space, action space, and reward function are defined as follows:
[0031] State space: system state s at time slot t t It consists of relevant information of all users, as shown below:
[0032]
[0033] Among them, Lo t (u) represents the two-dimensional coordinates of user u at time slot t; Indicates the amount of task data; k t (u) represents the task computing density; Indicates the amount of service instance data; Indicates the edge node where the service instance of user u is located in the previous time slot;
[0034] Action space: Let action a t Including the service migration decisions of all users, then the action a in time slot t t It is expressed as:
[0035] a t ={a t (u)|u∈U} Formula 9;
[0036] Reward function: Assuming that the core goal of the service migration method is to reduce the long-term delay of the system, and assuming that the system's immediate reward is negatively correlated with the sum of the migration delay, communication delay, and calculation delay of all users, the reward function is defined as follows:
[0037]
[0038] The service migration method adopts deep deterministic policy gradient training of deep neural network to obtain the optimal migration strategy in complex and dynamic edge environments; in the actor-critic architecture, the actor network is responsible for generating corresponding service migration decisions under the current environmental state, and the critic network is responsible for evaluating the value of the decision.
[0039] In the actor-critic architecture, the algorithm used by the service migration method is expressed in pseudo code as follows:
[0040] The specific contents are:
[0041] Initialize the actor network μ and the critic network Q and their network parameters θ μ and θ Q (pseudo code line 1);
[0042] Initialize the target actor network and target critic network And assign the corresponding network parameters to the target network (pseudo code line 2);
[0043] Initialize the experience replay pool X, the number of training rounds E, and the maximum time slot T for each round (pseudo code line 3); in each training round, the user will create a service instance on the edge node closest to it (pseudo code lines 5-6);
[0044] The environmental state s at time slot t t Input actor network, which generates service migration decision a t And execute; Among them, the setting of exploration noise ε is used to prevent the algorithm from falling into the local optimum (pseudo code lines 8-9);
[0045] The system calculates the instant reward r t And transfer to the next state s t+1 (pseudo code line 10);
[0046] Sample(s) t ,a t ,r t ,s t+1 ) will be stored in X, and N samples will be randomly taken out from it for training network parameters (pseudo code lines 11-12);
[0047] Combine the target network with the current instant reward r t Calculate the discounted cumulative reward y t , and use the gradient descent algorithm to minimize the mean square error loss of the critic network (pseudo code lines 13-14);
[0048] Actor network is used to fit the state s t To action a t The goal is to maximize the value of the output action at the critic network; for the system state s t , adjust the network parameters θ μ , so that the output μ(s t |θ μ )Q(s) calculated based on the critic network t ,μ(s t |θ μ )|θ Q ) can be updated in the upward direction; the gradient ascent algorithm is used to approximate the optimal solution of the actor network strategy, and the target network parameters are updated using a soft update method (pseudocode lines 15-16).
[0049] The use process of the method includes:
[0050] Step A1: The DRL agent generates a service migration decision based on the relevant information of all users, including user location, task data volume, task computing density, service data size, and the edge node where the user service instance is located in the previous time slot;
[0051] Step A2: The MEC controller issues a service migration command to each MEC server based on the service migration decision to complete the migration of the service instance;
[0052] Step A3: The MEC server allocates computing resources to the service instances running on the existing machines based on the weighted resource allocation strategy to support their parallel operation;
[0053] Step A4: The MEC server receives or forwards the task request to the service instance corresponding to the user for processing according to the service migration decision;
[0054] Step A5: During the service migration process, record the status of each time slot, the actions taken, the rewards obtained, and the new status entered. Based on the above information, the DRL intelligent body will generate corresponding service migration decisions.
[0055] This paper proposes a deep deterministic policy gradient based service migration (DPSM) method to explore the optimal service migration strategy in edge environments. The DPSM method directly outputs migration decisions to cope with the huge action space and shows superior performance in various scenarios.
[0056] The DPSM method proposed in the present invention aims at the service migration problem in the edge environment. It can generate a suitable service migration plan according to the environmental status to maximize the system performance. A large number of simulation experiments have verified the effectiveness of the DPSM method of the present invention in improving user QoS. Compared with five benchmark methods (DQN, GA, AM, NM, PM), the DPSM method of the present invention shows more superior performance in test scenarios with different BS bandwidth, MEC server computing power, unit delay coefficient and number of users. Compared with the advanced DQN method, the DPSM method of the present invention also shows faster convergence speed and better convergence effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:
[0058] Attached Figure 1 is a schematic diagram of the edge system of the present invention;
[0059] Attached Figure 2 It is a schematic diagram of the principle of the DPSM method of the present invention. DETAILED DESCRIPTION
[0060] As shown in the figure, the edge service migration method based on deep reinforcement learning uses a unified service migration model for the dynamic and changeable edge system environment, takes long-term QoS as the optimization goal, and uses migration, communication and computing delays for measurement. At the same time, based on the deep reinforcement learning DRL framework, the state space, action space and reward function of the service migration problem in the MEC environment are defined, and the above problem is formally represented as a Markov decision process MDP. At the same time, in the edge service migration method DPSM based on deep reinforcement learning, deep deterministic policy gradient training deep neural network is adopted to obtain the optimal migration strategy in the complex and dynamic edge environment.
[0061] The edge system consists of an MEC controller, M BS base stations and U users; each base station is equipped with a MEC server to form an edge node; the set of M edge nodes is recorded as M = {1, 2, ..., m, ..., M}, and the set of U users is recorded as U = {1, 2, ..., u, ..., U};
[0062] The edge system operates in discrete time slots, and the user's location changes at the beginning of each time slot T = {1, 2, ..., t, ..., T}. The smart application running on the mobile device of user u generates a computing task Task in each time slot. t (u);
[0063] Assuming that the local computing power of the user's mobile device is limited and its task requests need to be continuously offloaded to the MEC server for processing, then at the start time slot, the mobile device accesses the system through the BS and creates a service instance on the corresponding MEC server; the task requests of different mobile devices are processed by their respective service instances, and these service instances maintain parallel operation by occupying part of the computing resources of the MEC server.
[0064] The migration delay is calculated as follows: Defined as the service instance of user u, at the edge node where time slot t is located, Defined as the service instance of user u, at the edge node in time slot t-1, the system determines whether it can accept service migration by periodically querying the load of all edge nodes. Define a t (u) is the service migration decision of user u in time slot t, which includes all available edge nodes within the user’s access range, using d t To measure and The jump distance between Indicates the delay in unit migration. Indicates the amount of service data, Indicates the backhaul link bandwidth;
[0065] When t = 0, no service migration is required; otherwise, the service Relocation Therefore, the migration delay is expressed as follows:
[0066]
[0067] The communication delay is calculated as follows: Defined as the edge node to which the service instance of user u is connected in time slot t; mobile user u offloads the computing task to This process involves: users and The access delay, and and The return delay of the mobile device is p u , the unit channel gain is α, and the Gaussian noise power is σ 2 , user and The distance is The amount of task data is The signal-to-noise ratio is defined as follows:
[0068]
[0069] Define the total bandwidth of the BS as B, and evenly distribute the bandwidth to all mobile devices in the area in an orthogonal manner; then the access delay is defined as:
[0070]
[0071] Among them, B t (u) represents the available bandwidth of user u.
[0072] when When the service instance is transmitted via the backhaul link. represents the backhaul link bandwidth, Represents the single-hop transmission delay coefficient; similarly, using y t express and The backhaul link transmission delay is defined as:
[0073]
[0074] The total communication delay is defined as follows:
[0075] E t (u) = R t (u)+P t (u) Formula 5.
[0076] The calculation delay is calculated as follows: the task calculation density is denoted as k t (u), use measure The total load at time slot t, F represents the server computing capacity, and assuming that the computing resources allocated to user u are proportional to the resources required for its task, the computing delay is defined as follows:
[0077]
[0078] The optimization goal is specifically: within the time range T, the optimization goal of the proposed MEC system is to minimize the long-term system delay, which is formalized as the optimization problem P1 as follows:
[0079]
[0080] For the optimization problem P1, a service migration method based on deep deterministic policy gradient is used to form the optimal migration strategy in the dynamic edge environment.
[0081] In the service migration method based on deep deterministic policy gradient, during the optimization process of service migration, the DRL agent of the DRL framework is in the current state s according to the strategy π t Next select an action t , immediate reward r for environmental feedback t and transfer to the new state s t+1 , the process is expressed as MDP; at the same time, the sample (s t, a t , r t ,s t+1 ) will be stored in the experience pool, and the DRL agent will be trained by extracting small batch samples;
[0082] The state space, action space, and reward function are defined as follows:
[0083] State space: system state s at time slot t t It consists of relevant information of all users, as shown below:
[0084]
[0085] Among them, Lo t (u) represents the two-dimensional coordinates of user u at time slot t; Indicates the amount of task data; k t (u) represents the task computing density; Indicates the amount of service instance data; Indicates the edge node where the service instance of user u is located in the previous time slot;
[0086] Action space: Let action a t Including the service migration decisions of all users, then the action a in time slot t t It is expressed as:
[0087] a t ={a t (u)|u∈U} Formula 9;
[0088] Reward function: Assuming that the core goal of the service migration method is to reduce the long-term delay of the system, and assuming that the system's immediate reward is negatively correlated with the sum of the migration delay, communication delay, and calculation delay of all users, the reward function is defined as follows:
[0089]
[0090] The service migration method adopts deep deterministic policy gradient training of deep neural network to obtain the optimal migration strategy in complex and dynamic edge environments; in the actor-critic architecture, the actor network is responsible for generating corresponding service migration decisions under the current environmental state, and the critic network is responsible for evaluating the value of the decision.
[0091] In the actor-critic architecture, the algorithm used by the service migration method is expressed in pseudo code as follows:
[0092] The specific contents are:
[0093] Initialize the actor network μ and the critic network Q and their network parameters θ μ and θ Q (pseudo code line 1);
[0094] Initialize the target actor network and target critic network And assign the corresponding network parameters to the target network (pseudo code line 2);
[0095] Initialize the experience replay pool X, the number of training rounds E, and the maximum time slot T for each round (pseudo code line 3); in each training round, the user will create a service instance on the edge node closest to it (pseudo code lines 5-6);
[0096] The environmental state s at time slot t t Input actor network, which generates service migration decision a t And execute; Among them, the setting of exploration noise ε is used to prevent the algorithm from falling into the local optimum (pseudo code lines 8-9);
[0097] The system calculates the instant reward r t And transfer to the next state s t+1 (pseudo code line 10);
[0098] Sample(s) t ,a t ,r t ,s t+1 ) will be stored in X, and N samples will be randomly taken out from it for training network parameters (pseudo code lines 11-12);
[0099] Combine the target network with the current instant reward r t Calculate the discounted cumulative reward y t , and use the gradient descent algorithm to minimize the mean square error loss of the critic network (pseudo code lines 13-14);
[0100] Actor network is used to fit the state s t To action a t The goal is to maximize the value of the output action at the critic network; for the system state s t , adjust the network parameters θ μ , so that the output μ(s t |θ μ )Q(s) calculated based on the critic network t ,μ(s t |θ μ )|θ Q) can be updated in the upward direction; the gradient ascent algorithm is used to approximate the optimal solution of the actor network strategy, and the target network parameters are updated using a soft update method (pseudocode lines 15-16).
[0101] The use process of the method includes:
[0102] Step A1: The DRL agent generates a service migration decision based on the relevant information of all users, including user location, task data volume, task computing density, service data size, and the edge node where the user service instance is located in the previous time slot;
[0103] Step A2: The MEC controller issues a service migration command to each MEC server based on the service migration decision to complete the migration of the service instance;
[0104] Step A3: The MEC server allocates computing resources to the service instances running on the existing machines based on the weighted resource allocation strategy to support their parallel operation;
[0105] Step A4: The MEC server receives or forwards the task request to the service instance corresponding to the user for processing according to the service migration decision;
[0106] Step A5: During the service migration process, record the status of each time slot, the actions taken, the rewards obtained, and the new status entered. Based on the above information, the DRL intelligent body will generate corresponding service migration decisions.
[0107] Example:
[0108] The DPSM method proposed in this example builds and trains a neural network based on Python 3.8 and the Pytorch framework; the simulation experiment for verification is carried out on a workstation equipped with an Intel(R) Xeon(R) Silver 4208 processor and an RTX 3090 graphics card.
[0109] In the simulation experiment, based on the vehicle trajectory dataset of Rome, Italy, an area in Rome is regarded as the experimental scenario. Four edge nodes are deployed in this area with a computing power of 20GHz and a bandwidth of 10MHz. At the t0 time slot, all mobile devices are connected to the system, and the corresponding MEC server will create a service instance for them. A training round consists of 60 time slots, and the mobile device continuously sends task requests to its service instance at a fixed time interval. The number of neurons in the hidden layer of the neural network is 256 and 128 respectively, the batch size N is 64, the soft update parameter ω is 0.01, and the learning rates of the actor network and critic are 10 -5 and 10 -4 , the discount factor γ is 0.90. After the model is trained, it can be applied to service migration optimization in different scenarios.
Claims
1. The edge service migration method based on deep reinforcement learning is characterized by: The method uses a unified service migration model for the dynamic and changeable edge system environment, takes long-term QoS as the optimization goal, and uses migration, communication and computing delays for measurement. At the same time, based on the deep reinforcement learning (DRL) framework, the state space, action space and reward function of the service migration problem in the MEC environment are defined, and the above problem is formally represented as a Markov decision process (MDP). At the same time, in the edge service migration method DPSM based on deep reinforcement learning, deep deterministic policy gradient training of deep neural networks is adopted to obtain the optimal migration strategy in the complex and dynamic edge environment.
2. The edge service migration method based on deep reinforcement learning according to claim 1, characterized in that: The edge system consists of an MEC controller, M BS base stations and U users; each base station is equipped with a MEC server to form an edge node; the set of M edge nodes is recorded as M = {1, 2, ..., m, ..., M}, and the set of U users is recorded as U = {1, 2, ..., u, ..., U}; The edge system operates in discrete time slots, and the user's location changes at the beginning of each time slot T = {1, 2, ..., t, ..., T}. The smart application running on the mobile device of user u generates a computing task Task in each time slot. t (u); Assuming that the local computing power of the user's mobile device is limited and its task requests need to be continuously offloaded to the MEC server for processing, then at the start time slot, the mobile device accesses the system through the BS and creates a service instance on the corresponding MEC server; the task requests of different mobile devices are processed by their respective service instances, and these service instances maintain parallel operation by occupying part of the computing resources of the MEC server.
3. The edge service migration method based on deep reinforcement learning according to claim 2 is characterized in that: The migration delay is calculated as follows: Defined as the service instance of user u, at the edge node where time slot t is located, Defined as the service instance of user u, at the edge node in time slot t-1, the system determines whether it can accept service migration by periodically querying the load of all edge nodes. Define a t (u) is the service migration decision of user u in time slot t, which includes all available edge nodes within the user’s access range, using d t To measure and The jump distance between Indicates the delay in unit migration. Indicates the amount of service data, Indicates the backhaul link bandwidth; When t = 0, no service migration is required; otherwise, the service Relocation Therefore, the migration delay is expressed as follows:
4. The edge service migration method based on deep reinforcement learning according to claim 2 is characterized in that: The communication delay is calculated as follows: Defined as the edge node to which the service instance of user u is connected in time slot t; mobile user u offloads the computing task to This process involves: users and The access delay, and and The return delay of the mobile device is p u , the unit channel gain is α, and the Gaussian noise power is σ 2 , user and The distance is The amount of task data is The signal-to-noise ratio is defined as follows: Define the total bandwidth of the BS as B, and evenly distribute the bandwidth to all mobile devices in the area in an orthogonal manner; then the access delay is defined as: Among them, B t (u) represents the available bandwidth of user u. when When the service instance is transmitted via the backhaul link. Indicates the backhaul link bandwidth, Represents the single-hop transmission delay coefficient; similarly, using y t express and The backhaul link transmission delay is defined as: The total communication delay is defined as follows: AND t (u)=R t (u)+P t (u) Owner。 5. The edge service migration method based on deep reinforcement learning according to claim 2, characterized in that: The calculation delay is calculated as follows: the task calculation density is denoted as k t (u), use measure The total load at time slot t, F represents the server computing capacity, and assuming that the computing resources allocated to user u are proportional to the resources required for its task, the computing delay is defined as follows:
6. The edge service migration method based on deep reinforcement learning according to claim 2, characterized in that: The optimization goal is specifically: within the time range T, the optimization goal of the proposed MEC system is to minimize the long-term system delay, which is formalized as the optimization problem P1 as follows: For the optimization problem P1, a service migration method based on deep deterministic policy gradient is used to form the optimal migration strategy in the dynamic edge environment.
7. The edge service migration method based on deep reinforcement learning according to claim 6 is characterized in that: In the service migration method based on deep deterministic policy gradient, during the optimization process of service migration, the DRL agent of the DRL framework is in the current state s according to the strategy π t Next select an action t , immediate reward r for environmental feedback t and transfer to the new state s t+1 , the process is expressed as MDP; at the same time, the sample (s t ,a t ,r t ,s t+1 ) will be stored in the experience pool, and the DRL agent will be trained by extracting small batch samples; The state space, action space, and reward function are defined as follows: State space: system state s at time slot t t It consists of relevant information of all users, as shown below: Among them, Lo t (u) represents the two-dimensional coordinates of user u at time slot t; Indicates the amount of task data; k t (u) represents the task computing density; Indicates the amount of service instance data; Indicates the edge node where the service instance of user u is located in the previous time slot; Action space: Let action a t Including the service migration decisions of all users, the action a in time slot t t It is expressed as: a t ={a t (u)|u∈U} Formula 9; Reward function: Assuming that the core goal of the service migration method is to reduce the long-term delay of the system, and assuming that the system's immediate reward is negatively correlated with the sum of the migration delay, communication delay, and calculation delay of all users, the reward function is defined as follows:
8. The edge service migration method based on deep reinforcement learning according to claim 7 is characterized in that: The service migration method adopts deep deterministic policy gradient training of deep neural network to obtain the optimal migration strategy in complex and dynamic edge environments; in the actor-critic architecture, the actor network is responsible for generating corresponding service migration decisions under the current environmental state, and the critic network is responsible for evaluating the value of the decision.
9. The edge service migration method based on deep reinforcement learning according to claim 8, characterized in that: In the actor-critic architecture, the specific contents of the algorithm adopted by the service migration method are: Initialize the actor network μ and the critic network Q and their network parameters θ μ and θ Q ; Initialize the target actor network and target critic network And assign the corresponding network parameters to the target network; initialize the experience replay pool X, the number of training rounds E and the maximum time slot of each round T; In each training round, users create service instances on the edge nodes closest to them; The environmental state s at time slot t t Input actor network, which generates service migration decision a t And execute; wherein, the setting of exploration noise ε is used to avoid the algorithm falling into local optimum; The system calculates the instant reward r t And transfer to the next state s t+1 ; Sample(s) t ,a t ,r t ,s t+1 ) will be stored in X, and N samples will be randomly taken out from it for training network parameters; Combine the target network with the current instant reward r t Calculate the discounted cumulative reward y t , and use the gradient descent algorithm to minimize the mean square error loss of the critic network; Actor network is used to fit the state s t To action a t The goal is to maximize the value of the output action at the critic network; for the system state s t , adjust the network parameters θ μ , so that the output μ(s t |θ μ )Q(s) calculated based on the critic network t ,μ(s t |θ μ )|θ Q ) can be updated in the upward direction; the gradient ascent algorithm is used to approach the optimal solution of the actor network strategy, and the target network parameters are updated using a soft update method.
10. The edge service migration method based on deep reinforcement learning according to claim 8, characterized in that: The use process of the method includes: Step A1: The DRL agent generates a service migration decision based on the relevant information of all users, including user location, task data volume, task computing density, service data size, and the edge node where the user service instance is located in the previous time slot; Step A2: The MEC controller issues a service migration command to each MEC server based on the service migration decision to complete the migration of the service instance; Step A3: The MEC server allocates computing resources to the service instances running on the existing machines based on the weighted resource allocation strategy to support their parallel operation; Step A4: The MEC server receives or forwards the task request to the service instance corresponding to the user for processing according to the service migration decision; Step A5: During the service migration process, record the status of each time slot, the actions taken, the rewards obtained, and the new status transferred to. Based on the above information, the DRL intelligent body will generate corresponding service migration decisions.
Citation Information
Patent Citations
Edge computing task unloading method based on deep reinforcement learning in ultra-dense network
CN115499441A
Service migration method based on DQN in edge environment
CN117202265A
Intelligent service migration method for edge Internet of Vehicles system
CN117880852A
KR20240072551A
Cited By
MAGRPO-based intelligent network connection automobile service migration decision-making method
CN121771236A
