Edge service migration method based on deep reinforcement learning

Abstract

The application provides an edge service migration method based on deep reinforcement learning, which uses a unified service migration model for a dynamic and changeable edge system environment, takes long-term QoS as an optimization target, and uses migration, communication and calculation delays for measurement, simultaneously defines a state space, an action space and a reward function of a service migration problem in a MEC environment based on a deep reinforcement learning DRL framework, and formulates the above problem as a Markov decision process MDP, and simultaneously adopts a deep deterministic policy gradient to train a deep neural network in the deep reinforcement learning-based edge service migration method DPSM, so as to obtain an optimal migration strategy in a complex and dynamic edge environment; the application is used for exploring an optimal service migration strategy in an edge environment, and directly outputs a migration decision, so as to cope with a huge action space, and exhibits more superior performance in various scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication network, and particularly to an edge service migration method based on deep reinforcement learning. BACKGROUND

[0002] With the development of 5G communication technology and the popularity of mobile devices, various emerging intelligent applications are emerging, such as real-time video streaming, online games, augmented reality (AR) and virtual reality (VR), etc. Such applications usually have time-sensitive and data-intensive characteristics and rely on powerful computing power to ensure high quality of service (QoS), which poses a great challenge to the performance of mobile devices. Cloud computing provides powerful processing power by deploying server clusters remotely. However, since users are far away from the remote cloud in geographical location, a large amount of time is needed to transmit data, which contradicts the real-time requirements of intelligent applications. To solve this problem, mobile edge computing (MEC) deploys computing and storage resources at the network edge to achieve low latency, high bandwidth and high reliability service delivery. User devices can offload tasks to MEC servers for processing to improve the task response speed of intelligent applications. At the same time, the MEC architecture allows user data to be processed at the edge without being uploaded to the remote cloud, effectively enhancing the security and privacy of data. In addition, virtualization technology is considered as a key resource allocation and management technology. When user devices offload tasks to MEC servers through base stations (BSs), MEC servers will create a dedicated service instance for users using virtualization technology and allocate certain computing resources to them.

[0003] Considering the mobility of users and the limited communication coverage of BSs, users can be far away from the MEC server currently serving them, which will further degrade the QoS. Therefore, the service instance of the user needs to be dynamically migrated following the mobility of the user to better guarantee the QoS. There is resource contention between the service instances of different users, and when the number of users in the system increases, it can lead to potential resource conflicts and performance degradation. In addition, the performance of service migration depends on multiple factors such as user location, task data volume, task computing volume, etc. Improper migration strategy will lead to high task response delay and seriously affect the QoS.

[0004] Existing solutions usually rely on rules, heuristics or optimization theory. These methods usually rely on system priori knowledge to make strategies, but it is difficult to obtain the above information in a dynamic and variable environment. In addition, they usually need to frequently adjust model parameters and involve a large number of iterative operations and large solution space search, increasing the computational overhead. Therefore, when facing the service migration problem in the dynamic edge environment, it is necessary to explore more effective solutions to achieve efficient adaptive service migration. As an important branch of machine learning, deep reinforcement learning (DRL) is considered an effective solution to dynamic resource optimization problems. Through interaction with the environment, the agent can gradually adjust its strategy and maximize the long-term cumulative reward in the environment. Existing DRL-based service migration methods mostly use value-based DRL, which learns deterministic strategies by comparing the Q values of all candidate actions, but the large action space in the edge system will result in low learning efficiency of this method.

[0005] To solve the above problems, the present application proposes a deep deterministic policy gradient based service migration (DPSM) method to explore the optimal service migration strategy in the edge environment. The DPSM method directly outputs the migration decision to cope with the large action space and exhibits superior performance in various scenarios. SUMMARY

[0006] The present application proposes an edge service migration method based on deep reinforcement learning for exploring the optimal service migration strategy in the edge environment. This method directly outputs the migration decision to cope with the large action space and exhibits superior performance in various scenarios.

[0007] The present application adopts the following technical solutions.

[0008] The edge service migration method based on deep reinforcement learning uses a unified service migration model for the dynamic and variable edge system environment, uses long-term QoS as the optimization target, and uses the delay of migration, communication and calculation for measurement. At the same time, the state space, action space and reward function of the service migration problem in the MEC environment are defined based on the deep reinforcement learning DRL framework, and the above problem is formalized as a Markov decision process MDP. At the same time, a deep deterministic policy gradient is used in the edge service migration method based on deep reinforcement learning DPSM to train a deep neural network to obtain the optimal migration strategy in the complex and dynamic edge environment.

[0009] The edge system consists of one MEC controller, M BS base stations, and U users; each base station is equipped with an MEC server to form an edge node; the set of M edge nodes is denoted as M = {1, 2, ..., m, ..., M}, and the set of U users is denoted as U = {1, 2, ..., u, ..., U};

[0010] The edge system operates in discrete time slots, with the user's location changing at the beginning of each time slot T = {1, 2, ..., t, ..., T}. The intelligent application running on user u's mobile device generates a computational task in each time slot. t (u);

[0011] Assuming that the local computing power of a user's mobile device is limited, and its task requests need to be continuously offloaded to the MEC server for processing, then at the beginning of the time slot, the mobile device accesses the system through the BS and creates a service instance on the corresponding MEC server; the task requests of different mobile devices are processed by their respective service instances, and these service instances maintain parallel operation by occupying part of the computing resources of the MEC server.

[0012] The migration delay is calculated as follows: Defined as a service instance for user u, at the edge node where time slot t is located, The service instance defined as user u, located at the edge node in time slot t-1, is determined by the system periodically querying the load of all edge nodes to determine whether it can accept service migration. Let a be the value of this instance. t (u) represents the service migration decision for user u in time slot t, which includes all available edge nodes within the user's access range, using d t To measure and The jump distance between them Indicates unit migration delay. Indicates the amount of service data. Indicates the backhaul link bandwidth;

[0013] When d t When the value is 0, no service migration is required; otherwise, the service needs to be migrated. relocation Therefore, the migration delay is expressed as follows:

[0014]

[0015] The communication delay is calculated as follows: Defined as the edge node connected to the service instance of user u in time slot t; moving user u offloads computing tasks to This process involves: the user and... Access latency, and and The return delay; where the transmission power of the mobile device is p u The unity channel gain is α, and the Gaussian noise power is σ. 2 Users and The distance is Task data volume is The signal-to-noise ratio is defined as follows:

[0016]

[0017] Define the total bandwidth of the BS as B, and distribute the bandwidth evenly to all mobile devices within the area in an orthogonal manner; then the access latency is defined as:

[0018]

[0019] Among them, B t (u) represents the available bandwidth for user u.

[0020] when At that time, service instances need to be transmitted via the backhaul link. Indicates the backhaul link bandwidth. This represents the single-hop transmission delay coefficient; similarly, y is used... t express and If the number of hops is given, then the backhaul link transmission delay is defined as:

[0021]

[0022] Total communication delay is defined as follows:

[0023] E t (u)=R t (u)+P t (u) Formula 5.

[0024] The computational latency is calculated as follows: the task computation density is denoted as k. t (u), using measure The total load in time slot t, F represents the server's computing power. Assuming that the computing resources allocated to user u are proportional to the resources required by its task, the computing latency is defined as follows:

[0025]

[0026] The specific optimization objective is as follows: Within the time range T, the optimization objective of the proposed MEC system is to minimize the long-term system delay, which is formalized as the optimization problem P1 as follows:

[0027]

[0028] For the optimization problem P1, a service migration method based on deep deterministic policy gradient is used to form the optimal migration strategy in the dynamic edge environment.

[0029] The service migration method based on deep deterministic policy gradient, during the service migration optimization process, involves the DRL agent of the DRL framework adjusting the current state s according to policy π. t Next select action a t Instant rewards for environmental feedback t And transition to the new state s t+1 This process is described as MDP; simultaneously, the sample (s) t a t r t s t+1 The data will be stored in the experience pool, and the DRL agent will be trained by extracting small batches of samples.

[0030] The state space, action space, and reward function are defined as follows:

[0031] State space: System state s in time slot t t It consists of relevant information for all users, represented as follows:

[0032]

[0033] Among them, Lo t (u) represents the two-dimensional coordinates of user u in time slot t; Indicates the amount of data for the task; k t (u) represents the task computation density; Indicates the amount of data in the service instance; This indicates the edge node where the service instance of user u was located in the previous time slot;

[0034] Action space: Let action a t If the service migration decision includes all users, then the action a in time slot t... t Represented as:

[0035] a t ={a t Formula Nine; (u)|u∈U}

[0036] Reward Function: Suppose the core objective of the service migration method is to reduce long-term system latency, and that the system's immediate reward is negatively correlated with the sum of migration latency, communication latency, and computation latency for all users. Then the reward function is defined as follows:

[0037]

[0038] The service migration method employs a deep deterministic policy gradient training deep neural network to obtain the optimal migration strategy in a complex and dynamic edge environment. In the actor-critic architecture, the actor network is responsible for generating the corresponding service migration decision under the current environment state, and the critic network is responsible for evaluating the value of the decision.

[0039] In the actor-critic architecture, the algorithm used in the service migration method is expressed in pseudocode as follows:

[0040] The specific content is as follows:

[0041] Initialize the actor network μ and the critic network Q and their network parameters θ μ and θ Q (Pseudocode line 1);

[0042] Initialize the target actor network and target critic network And assign the corresponding network parameters to the target network (pseudocode line 2);

[0043] Initialize the experience replay pool X, the number of training rounds E, and the maximum time slot T per round (pseudocode line 3); in each training round, the user will create a service instance on the nearest edge node (pseudocode lines 5-6);

[0044] The environmental state s of time slot t t Input actor network, its generated service migration decision a t And execute; where the setting of the noise ε is used to avoid the algorithm getting trapped in local optima (pseudocode lines 8-9);

[0045] The system calculates the instant reward r t And transition to the next state s t+1 (Pseudocode line 10);

[0046] Samples (s) t ,a t ,r t ,s t+1 The sample will be stored in X, and N samples will be randomly selected from it to train the network parameters (pseudocode lines 11-12);

[0047] Combining the target network with the current instant reward r t Calculate the cumulative discount reward y t And the gradient descent algorithm is used to minimize the mean squared error loss of the critic network (pseudocode lines 13-14);

[0048] Actor networks are used to fit the state s. t To action a t The mapping aims to maximize the value of the output action at the critic network; for system state s t Adjusting network parameters θ μ This makes the output μ(s) of the actor network t |θ μ Q(s) calculated based on the critic network t ,μ(s t |θ μ )|θ Q It can update in the ascending direction; it uses the gradient ascent algorithm to approximate the optimal solution of the actor network's strategy, and uses a soft update method to update the target network parameters (pseudocode lines 15-16).

[0049] The process of using the method includes:

[0050] Step A1: The DRL agent generates service migration decisions based on relevant information of all users, including user location, task data volume, task computation density, service data size, and the edge node where the user service instance was located in the previous time slot.

[0051] Step A2: Based on the service migration decision, the MEC controller issues service migration commands to each MEC server to complete the migration of service instances;

[0052] Step A3: The MEC server allocates computing resources to the service instances running on the existing machines based on a weighted resource allocation strategy to support their parallel operation;

[0053] Step A4: Based on the service migration decision, the MEC server receives or forwards task requests to the corresponding service instance for processing.

[0054] Step A5: During the service migration process, record the status of each time slot, the actions taken, the rewards obtained, and the new status transitioned to. Based on the above information, the DRL AI will generate the corresponding service migration decision.

[0055] This invention proposes a Deepdeterministic Policy Gradient Based Service Migration (DPSM) method to explore optimal service migration strategies in edge environments. The DPSM method directly outputs migration decisions to handle a large action space and exhibits superior performance across various scenarios.

[0056] The DPSM method proposed in this invention addresses the service migration problem in edge environments. It can generate appropriate service migration schemes based on environmental conditions to maximize system performance. Extensive simulation experiments have verified the effectiveness of the DPSM method in improving user QoS. Compared with five benchmark methods (DQN, GA, AM, NM, PM), the DPSM method demonstrates superior performance in test scenarios with different BS bandwidth, MEC server computing power, unit latency coefficient, and number of users. Compared with the advanced DQN method, the DPSM method of this invention also exhibits faster convergence speed and better convergence effect. Attached Figure Description

[0057] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0058] Appendix Figure 1 This is a schematic diagram of the edge system of the present invention;

[0059] Appendix Figure 2 This is a schematic diagram illustrating the principle of the DPSM method of the present invention. Detailed Implementation

[0060] As shown in the figure, the edge service migration method based on deep reinforcement learning uses a unified service migration model for the dynamic and ever-changing edge system environment. It takes long-term QoS as the optimization objective and uses the latency of migration, communication and computing as the metric. At the same time, it defines the state space, action space and reward function of the service migration problem in the MEC environment based on the deep reinforcement learning DRL framework, and formally represents the above problem as a Markov decision process (MDP). In the edge service migration method DPSM based on deep reinforcement learning, a deep deterministic policy gradient is used to train a deep neural network to obtain the optimal migration strategy in the complex and dynamic edge environment.

[0061] The edge system consists of one MEC controller, M BS base stations, and U users; each base station is equipped with an MEC server to form an edge node; the set of M edge nodes is denoted as M = {1, 2, ..., m, ..., M}, and the set of U users is denoted as U = {1, 2, ..., u, ..., U};

[0062] The edge system operates in discrete time slots, with the user's location changing at the beginning of each time slot T = {1, 2, ..., t, ..., T}. The intelligent application running on user u's mobile device generates a computational task in each time slot. t (u);

[0063] Assuming that the local computing power of a user's mobile device is limited, and its task requests need to be continuously offloaded to the MEC server for processing, then at the beginning of the time slot, the mobile device accesses the system through the BS and creates a service instance on the corresponding MEC server; the task requests of different mobile devices are processed by their respective service instances, and these service instances maintain parallel operation by occupying part of the computing resources of the MEC server.

[0064] The migration delay is calculated as follows: Defined as a service instance for user u, at the edge node where time slot t is located, The service instance defined as user u, located at the edge node in time slot t-1, is determined by the system periodically querying the load of all edge nodes to determine whether it can accept service migration. Let a be the value of this instance. t (u) represents the service migration decision for user u in time slot t, which includes all available edge nodes within the user's access range, using d t To measure and The jump distance between them Indicates unit migration delay. Indicates the amount of service data. Indicates the backhaul link bandwidth;

[0065] When d t When the value is 0, no service migration is required; otherwise, the service needs to be migrated. relocation Therefore, the migration delay is expressed as follows:

[0066]

[0067] The communication delay is calculated as follows: Defined as the edge node connected to the service instance of user u in time slot t; moving user u offloads computing tasks to This process involves: the user and... Access latency, and and The return delay; where the transmission power of the mobile device is p u The unity channel gain is α, and the Gaussian noise power is σ. 2 Users and The distance is Task data volume is The signal-to-noise ratio is defined as follows:

[0068]

[0069] Define the total bandwidth of the BS as B, and distribute the bandwidth evenly to all mobile devices within the area in an orthogonal manner; then the access latency is defined as:

[0070]

[0071] Among them, B t (u) represents the available bandwidth for user u.

[0072] when At that time, service instances need to be transmitted via the backhaul link. Indicates the backhaul link bandwidth. This represents the single-hop transmission delay coefficient; similarly, y is used... t express and If the number of hops is given, then the backhaul link transmission delay is defined as:

[0073]

[0074] Total communication delay is defined as follows:

[0075] E t (u)=R t (u)+P t (u) Formula 5.

[0076] The computational latency is calculated as follows: the task computation density is denoted as k. t (u), using measure The total load in time slot t, F represents the server's computing power. Assuming that the computing resources allocated to user u are proportional to the resources required by its task, the computing latency is defined as follows:

[0077]

[0078] The specific optimization objective is as follows: Within the time range T, the optimization objective of the proposed MEC system is to minimize the long-term system delay, which is formalized as the optimization problem P1 as follows:

[0079]

[0080] For the optimization problem P1, a service migration method based on deep deterministic policy gradient is used to form the optimal migration strategy in the dynamic edge environment.

[0081] The service migration method based on deep deterministic policy gradient, during the service migration optimization process, involves the DRL agent of the DRL framework adjusting the current state s according to policy π. t Next select action a t Instant rewards for environmental feedback t And transition to the new state s t+1 This process is described as MDP; simultaneously, the sample (s) ta t r t s t+1 The data will be stored in the experience pool, and the DRL agent will be trained by extracting small batches of samples.

[0082] The state space, action space, and reward function are defined as follows:

[0083] State space: System state s in time slot t t It consists of relevant information for all users, represented as follows:

[0084]

[0085] Among them, Lo t (u) represents the two-dimensional coordinates of user u in time slot t; Indicates the amount of data for the task; k t (u) represents the task computation density; Indicates the amount of data in the service instance; This indicates the edge node where the service instance of user u was located in the previous time slot;

[0086] Action space: Let action a t If the service migration decision includes all users, then the action a in time slot t... t Represented as:

[0087] a t ={a t Formula Nine; (u)|u∈U}

[0088] Reward Function: Suppose the core objective of the service migration method is to reduce long-term system latency, and that the system's immediate reward is negatively correlated with the sum of migration latency, communication latency, and computation latency for all users. Then the reward function is defined as follows:

[0089]

[0090] The service migration method employs a deep deterministic policy gradient training deep neural network to obtain the optimal migration strategy in a complex and dynamic edge environment. In the actor-critic architecture, the actor network is responsible for generating the corresponding service migration decision under the current environment state, and the critic network is responsible for evaluating the value of the decision.

[0091] In the actor-critic architecture, the algorithm used in the service migration method is expressed in pseudocode as follows:

[0092] The specific content is as follows:

[0093] Initialize the actor network μ and the critic network Q and their network parameters θ μ and θ Q (Pseudocode line 1);

[0094] Initialize the target actor network and target critic network And assign the corresponding network parameters to the target network (pseudocode line 2);

[0095] Initialize the experience replay pool X, the number of training rounds E, and the maximum time slot T per round (pseudocode line 3); in each training round, the user will create a service instance on the nearest edge node (pseudocode lines 5-6);

[0096] The environmental state s of time slot t t Input actor network, its generated service migration decision a t And execute; where the setting of the noise ε is used to avoid the algorithm getting trapped in local optima (pseudocode lines 8-9);

[0097] The system calculates the instant reward r t And transition to the next state s t+1 (Pseudocode line 10);

[0098] Samples (s) t ,a t ,r t ,s t+1 The sample will be stored in X, and N samples will be randomly selected from it to train the network parameters (pseudocode lines 11-12);

[0099] Combining the target network with the current instant reward r t Calculate the cumulative discount reward y t And the gradient descent algorithm is used to minimize the mean squared error loss of the critic network (pseudocode lines 13-14);

[0100] Actor networks are used to fit the state s. t To action a t The mapping aims to maximize the value of the output action at the critic network; for system state s t Adjusting network parameters θ μ This makes the output μ(s) of the actor network t |θ μ Q(s) calculated based on the critic network t ,μ(s t |θ μ )|θ QIt can update in the ascending direction; it uses the gradient ascent algorithm to approximate the optimal solution of the actor network's strategy, and uses a soft update method to update the target network parameters (pseudocode lines 15-16).

[0101] The process of using the method includes:

[0102] Step A1: The DRL agent generates service migration decisions based on relevant information of all users, including user location, task data volume, task computation density, service data size, and the edge node where the user service instance was located in the previous time slot.

[0103] Step A2: Based on the service migration decision, the MEC controller issues service migration commands to each MEC server to complete the migration of service instances;

[0104] Step A3: The MEC server allocates computing resources to the service instances running on the existing machines based on a weighted resource allocation strategy to support their parallel operation;

[0105] Step A4: Based on the service migration decision, the MEC server receives or forwards task requests to the corresponding service instance for processing.

[0106] Step A5: During the service migration process, record the status of each time slot, the actions taken, the rewards obtained, and the new status transitioned to. Based on the above information, the DRL AI will generate the corresponding service migration decision.

[0107] Example:

[0108] The DPSM method proposed in this example is based on Python 3.8 and the PyTorch framework to build and train neural networks; the simulation experiments used for verification were conducted on a workstation equipped with an Intel(R) Xeon(R) Silver 4208 processor and an RTX 3090 graphics card.

[0109] In the simulation experiment, based on the vehicle trajectory dataset from Rome, Italy, a region in Rome was considered as the experimental scenario. Four edge nodes were deployed within this region, with a computing power of 20 GHz and a bandwidth of 10 MHz. In time slot t0, all mobile devices connected to the system, and the corresponding MEC server created service instances for them. One training round contained 60 time slots, with mobile devices continuously sending task requests to their service instances at fixed time intervals. The neural network had 256 and 128 hidden layer neurons, a batch size N of 64, a soft update parameter ω of 0.01, and learning rates of 10 for both the actor and critic networks. -5 and 10 -4 The discount factor γ is 0.90. Once the model is trained, it can be applied to service migration optimization in different scenarios.

Claims

1. An edge service transfer method based on deep reinforcement learning, characterized in that: The method uses a unified service migration model for dynamic and ever-changing edge system environments, takes long-term QoS as the optimization objective, and uses migration, communication, and computation latency as metrics. It also defines the state space, action space, and reward function of the service migration problem in the MEC environment based on the deep reinforcement learning (DRL) framework, and formally represents the above problem as a Markov decision process (MDP). Furthermore, in the edge service migration method DPSM based on deep reinforcement learning, a deep deterministic policy gradient is used to train a deep neural network to obtain the optimal migration strategy in complex and dynamic edge environments. The edge system consists of one MEC controller, M BS base stations, and U users; each base station is equipped with an MEC server to form an edge node; the set of M edge nodes is denoted as M = {1, 2, ..., m, ..., M}, and the set of U users is denoted as U = {1, 2, ..., u, ..., U}; The edge system operates in discrete time slots, and the user's location changes at the beginning of each time slot T = {1, 2, ..., t, ..., T}; The smart application running on user u's mobile device generates a computing task in each time slot. t (u); Assuming that the local computing power of a user's mobile device is limited, and its task requests need to be continuously offloaded to the MEC server for processing, then at the beginning of the time slot, the mobile device accesses the system through the BS and creates a service instance on the corresponding MEC server; the task requests of different mobile devices are processed by their respective service instances, and these service instances maintain parallel operation by occupying part of the computing resources of the MEC server; The migration delay is calculated as follows: Defined as a service instance for user u, at the edge node where time slot t is located, The service instance defined as user u, located at the edge node in time slot t-1, is determined by the system periodically querying the load of all edge nodes to determine whether it can accept service migration. Let a be the value of this instance. t (u) represents the service migration decision for user u in time slot t, which includes all available edge nodes within the user's access range, using d t To measure and The jump distance between them Indicates unit migration delay. Indicates the amount of service data. Indicates the backhaul link bandwidth; When d t When the value is 0, no service migration is required; otherwise, the service needs to be migrated. relocation Therefore, the migration delay is expressed as follows: The communication delay is calculated as follows: Defined as the edge node connected to the service instance of user u in time slot t; moving user u offloads computing tasks to This process involves: the user and... Access latency, and and The return delay; where the transmission power of the mobile device is p u The unity channel gain is α, and the Gaussian noise power is σ. 2 Users and The distance is Task data volume is The signal-to-noise ratio is defined as follows: Define the total bandwidth of the BS as B, and distribute the bandwidth evenly to all mobile devices within the area in an orthogonal manner; then the access latency is defined as: Among them, B t (u) represents the available bandwidth for user u; when At this time, service instances need to be transmitted via the backhaul link; Indicates the backhaul link bandwidth. This represents the single-hop transmission delay coefficient; similarly, y is used... t express and If the number of hops is given, then the backhaul link transmission delay is defined as: Total communication delay is defined as follows: AND t (u)=R t (u)+P t (u) Where: The computational latency is calculated as follows: the task computation density is denoted as k. t (u), using measure The total load in time slot t, F represents the server's computing power. Assuming that the computing resources allocated to user u are proportional to the resources required by its task, the computing latency is defined as follows: The specific optimization objective is as follows: Within the time range T, the optimization objective of the proposed MEC system is to minimize the long-term system delay, which is formally expressed as optimization problem P1 as follows: For the optimization problem P1, a service migration method based on deep deterministic policy gradient is used to form the optimal migration strategy in the dynamic edge environment.

2. The edge service migration method based on deep reinforcement learning according to claim 1, characterized in that: The service migration method based on deep deterministic policy gradient, during the service migration optimization process, involves the DRL agent of the DRL framework adjusting the current state s according to policy π. t Next select action a t Instant rewards for environmental feedback t And transition to the new state s t+1 This process is described as MDP; simultaneously, the sample (s) t ,a t ,r t ,s t+1 The data will be stored in the experience pool, and the DRL agent will be trained by extracting small batches of samples. The state space, action space, and reward function are defined as follows: State space: System state s in time slot t t It consists of relevant information for all users, represented as follows: Among them, Lo t (u) represents the two-dimensional coordinates of user u in time slot t; Indicates the amount of data for the task; k t (u) represents the task computation density; Indicates the amount of data in the service instance; This indicates the edge node where the service instance of user u was located in the previous time slot; Action space: Let action a t If the service migration decision includes all users, then the action a in time slot t... t Represented as: a t ={a t Formula Nine; (u)|u∈U} Reward Function: Suppose the core objective of the service migration method is to reduce long-term system latency, and that the system's immediate reward is negatively correlated with the sum of migration latency, communication latency, and computation latency for all users. Then the reward function is defined as follows:

3. The edge service migration method based on deep reinforcement learning according to claim 2, characterized in that: The service migration method employs a deep deterministic policy gradient training deep neural network to obtain the optimal migration strategy in a complex and dynamic edge environment. In the actor-critic architecture, the actor network is responsible for generating the corresponding service migration decision under the current environment state, and the critic network is responsible for evaluating the value of the decision.

4. The edge service migration method based on deep reinforcement learning according to claim 3, characterized in that: In the actor-critic architecture, the specific algorithm used in the service migration method is as follows: Initialize the actor network μ and the critic network Q and their network parameters θ μ and θ Q ; Initialize the target actor network and target critic network Then assign the corresponding network parameters to the target network; initialize the experience replay pool X, the number of training rounds E, and the maximum time slot T per round; In each training round, the user creates a service instance on the nearest edge node; The environmental state s of time slot t t Input actor network, its generated service migration decision a t And execute; where the setting of the exploration noise ε is used to avoid the algorithm getting trapped in local optima; The system calculates the instant reward r t And transition to the next state s t+1 ; Samples (s) t ,a t ,r t ,s t+1 The data will be stored in X, and N samples will be randomly selected from it to train the network parameters. Combining the target network with the current instant reward r t Calculate the cumulative discount reward y t And the gradient descent algorithm is used to minimize the mean squared error loss of the critic network; Actor networks are used to fit the state s. t To action a t The mapping aims to maximize the value of the output action at the critic network; for system state s t Adjusting network parameters θ μ This makes the output μ(s) of the actor network... t |θ μ Q(s) calculated based on the critic network t ,μ(s t |θ μ )|θ Q It can update in the ascending direction; it uses the gradient ascent algorithm to approximate the optimal solution of the actor network strategy, and uses a soft update method to update the target network parameters.

5. The edge service migration method based on deep reinforcement learning according to claim 3, characterized in that: The process of using the method includes: Step A1: The DRL agent generates a service migration decision based on relevant information of all users, including user location, task data volume, task computation density, service data size, and the edge node where the user service instance was located in the previous time slot. Step A2: Based on the service migration decision, the MEC controller issues service migration commands to each MEC server to complete the migration of service instances; Step A3: The MEC server allocates computing resources to the service instances running on the existing machines based on a weighted resource allocation strategy to support their parallel operation; Step A4: Based on the service migration decision, the MEC server receives or forwards task requests to the corresponding service instance for processing. Step A5: During the service migration process, record the status of each time slot, the actions taken, the rewards obtained, and the new status transitioned to. Based on the above information, the DRL AI will generate the corresponding service migration decision.

Images