An adaptive edge service migration method based on DRL

Abstract

The present invention belongs to the technical field of mobile edge computing, and discloses a DRL-based adaptive edge service migration method, comprising: step 1, in each time slot, respectively obtaining the distance between a user terminal and its associated base station, the number of virtual machines started in each edge server, and the amount of available computing resources in each edge server; step 2, selecting an edge server connected to the base station associated with the user terminal as the associated edge server, and unloading the user task to the associated edge server; step 3, determining a target edge server with the minimum sum of delay cost and migration cost as an optimization target; step 4, migrating the user task from the associated edge server to the target edge server, and the target edge server processing the user task; wherein, if the determined target edge server is the associated edge server, there is no need to migrate the user task.

Description

Technical Field

[0001] The present invention belongs to the technical field of mobile edge computing, and in particular relates to a DRL-based adaptive edge service migration method. Background Art

[0002] With the rapid development of mobile Internet and Internet of Things, the generation and real-time processing of massive data are becoming more and more urgent, and Mobile Edge Computing (MEC) has emerged. MEC reduces latency, improves response speed, and enhances user experience by delegating computing and data storage capabilities to the edge of the network, close to the source of data. However, MEC faces many challenges in practical applications, one of which is the high mobility of users. User mobility brings the need for service migration. When a user moves from one network area to another, in order to ensure the quality of service (QoS), the service needs to be migrated from one edge server to another new server closer to the user. In this process, the computing resources of the edge server are limited. Migrating services not only takes time, but also occupies additional network and computing resources, resulting in migration costs. Therefore, whether it is necessary to migrate from the current edge server and where to migrate the service after making the migration decision have become two key issues that the MEC system needs to solve. To solve these two problems, it is necessary to fully consider the resource availability of each edge server in the MEC system and the distance between the user and the candidate server.

[0003] Researchers have proposed different solutions to the service migration problem. Recent studies have proposed a variety of methods for service migration in MEC environments. These methods mainly focus on optimizing the energy consumption and time cost during service migration. For example, researchers use evolutionary algorithms (such as SPEA2) to determine the optimal migration strategy to minimize energy consumption and migration time while ensuring service quality. In addition, some researchers model the problem as a Markov decision process and obtain service migration strategies through the DRL algorithm. However, these studies assume that the user's mobility pattern is uniform, which may not always be true in practice. The user's mobility pattern may change due to geographic location, time, and other external factors, which may affect the accuracy and effectiveness of the model. This is also a challenge that the service migration problem needs to face. Summary of the invention

[0004] The purpose of the present invention is to provide an adaptive edge service migration method based on DRL, which can adaptively determine the edge service migration strategy according to the real-time location changes of the user terminal to reduce the system delay cost and migration cost, and improve the accuracy and effectiveness of the migration strategy.

[0005] Another object of the present invention is to use an improved DQN network as a policy network, which can further improve the accuracy of the migration strategy and improve decision-making efficiency.

[0006] The technical solution provided by the present invention is:

[0007] A DRL-based adaptive edge service migration method includes the following steps:

[0008] Step 1: In each time slot, respectively obtain the distance between the user terminal and its associated base station, the number of virtual machines started in each edge server, and the amount of available computing resources in each edge server;

[0009] Step 2: Select an edge server connected to a base station associated with the user terminal as an associated edge server, and offload the user task to the associated edge server;

[0010] Step 3: Taking the minimum sum of delay cost and migration cost as the optimization goal, determine the target edge server;

[0011] Step 4: Migrate the user task from the associated edge server to the target edge server, and the target edge server processes the user task;

[0012] If the determined target edge server is the associated edge server, there is no need to migrate the user task;

[0013] The calculation model of the delay cost is:

[0014] ;

[0015] in, , , , They represent the communication delay, computation delay, migration delay, and return delay caused by processing user tasks.

[0016] Preferably, the calculation formula for processing the communication delay generated by the user task is:

[0017] ;

[0018] in, The uplink transmission rate at which the user terminal uploads the user task to the associated base station, is the user task amount in the current time slot.

[0019] Preferably, the uplink transmission rate at which the user terminal uploads the user task to the relevant base station is:

[0020] ;

[0021] in, is the number of user terminals associated with the base station, For the receive signal-to-noise ratio, The number of orthogonal resource blocks provided to the base station, The bandwidth of each resource block.

[0022] Preferably, the calculation formula for processing the calculation delay generated by the user task is:

[0023] ;

[0024] in, The number of CPU cycles required for edge servers to process user tasks, The number of virtual machines started for the edge servers, The computing power of the edge server.

[0025] Preferably, the calculation formula for processing the migration delay generated by the user task is:

[0026] ;

[0027] in, is the bandwidth of the wired link between base stations, The size of the user task.

[0028] Preferably, the calculation formula for the return delay generated by processing the user task is:

[0029] ;

[0030] in, To associate an edge server With the target edge server The number of hops between Is a positive coefficient.

[0031] Preferably, the calculation model of the migration cost is:

[0032] ;

[0033] in, is the migration energy consumption coefficient.

[0034] Preferably, in step 3, determining the target edge server based on a DQN network includes the following steps:

[0035] Step 1: Build an improved DQN network, which includes a Double DQN network and a Dueling DQN network; randomly generate network parameters, and add noise to the randomly generated network parameters to obtain initial network parameters;

[0036] Step 2: Integrate the distance between the user terminal and its associated base station, the number of virtual machines started in each edge server, and the amount of available computing resources in each edge server into a state space

[0037] Step 3: Transform the state space Input into the improved DQN network to get the action space

[0038] The action space This is the migration strategy;

[0039] Step 4: Determine the action space Reward value obtained And the new system status The experience sample Store in memory;

[0040] Among them, the reward function is set as:

[0041] ;

[0042] In the formula, is the maximum tolerable delay for the user, is the maximum tolerable migration energy consumption of the base station;

[0043] Step 5, iteratively optimize and train the improved DQN network with the goal of minimizing the Q value difference between the main network and the target network until the set number of iterations is reached to obtain the optimal improved DQN network; the optimal improved DQN network is used as the migration strategy network;

[0044] Step 6: Input the distance between the user terminal and its associated base station in the current time slot and the number of virtual machines started in each edge server into the migration strategy network, and the migration strategy network outputs the target edge server as the migration strategy.

[0045] Preferably, in said step 5, it further comprises:

[0046] In each training process, samples with large temporal difference errors in the memory are selected as training samples;

[0047] The calculation formula of the time difference error is:

[0048] ;

[0049] In the formula, is the time difference error, is the Q value of the main network, is the Q value of the target network.

[0050] The beneficial effects of the present invention are:

[0051] The DRL-based adaptive edge service migration method provided by the present invention can adaptively determine the edge service migration strategy according to the real-time location changes of the user terminal to reduce the system delay cost and migration cost, and improve the accuracy and effectiveness of the migration strategy.

[0052] The DRL-based adaptive edge service migration method provided by the present invention adopts an improved DQN network as a policy network, which can further improve the accuracy of the migration strategy and improve the decision-making efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a schematic diagram of service migration in the MEC system described in the present invention.

[0054] Figure 2 This is an overview diagram of the DRL-based adaptive edge service migration method (ASM) described in the present invention.

[0055] Figure 3 (a) is a schematic diagram of the cumulative rewards when the number of edge servers is 6 in the experimental example.

[0056] Figure 3 (b) is a schematic diagram of the cumulative rewards when the number of edge servers is 12 in the experimental example.

[0057] Figure 3 (c) is a schematic diagram of the cumulative rewards when the number of edge servers is 18 in the experimental example.

[0058] Figure 3 (d) is a schematic diagram of the cumulative rewards when the number of edge servers is 24 in the experimental example.

[0059] Figure 3 (e) is a schematic diagram of the cumulative rewards when the number of edge servers is 30 in the experimental example.

[0060] Figure 3 (f) is a schematic diagram of the cumulative rewards when the number of edge servers is 36 in the experimental example.

[0061] Figure 4 (a) is a graph showing the impact of the number of edge servers on service latency in the experimental example.

[0062] Figure 4 (b) is a graph showing the impact of the number of users on service delay in the experimental example.

[0063] Figure 5 (a) is a curve diagram showing the impact of the number of edge servers on migration energy consumption in the experimental example.

[0064] Figure 5 (b) is a curve diagram showing the impact of the number of users on migration energy consumption in the experimental example.

[0065] Figure 6 (a) is a graph showing the impact of the number of edge servers on the migration rejection rate in the experimental example.

[0066] Figure 6 (b) is a curve chart showing the impact of the number of users on the migration rejection rate in the experimental example. DETAILED DESCRIPTION

[0067] The present invention is further described in detail below in conjunction with the accompanying drawings so that those skilled in the art can implement the invention with reference to the description.

[0068] The present invention provides an adaptive edge service migration method based on DRL, which decomposes the optimization purpose into delay cost problem and migration cost problem to solve. The MEC system described in the present invention is composed of a group of edge servers (EdgeServer, ES) Composition, used to group highly mobile users Provide computing resources. Will be in the time slot Generate a task set Among them, the task generation speed is subject to the parameter Poisson distribution of edge servers Deployed at the same location as the base station to handle The edge servers at different locations are connected by wired links to facilitate service migration. Position in a time slot remains unchanged and is in position The user first sets the task set Transmitted to the base station BS via a wireless link, and then deployed in the edge server in the BS A virtual machine is created and computing resources are allocated to the virtual machine to process the task. In the next time slot middle, The location of the The task needs to make migration decisions based on the usage and location information of other servers.

[0069] like Figure 1 As shown in FIG. 1 , the MEC system proposed in the present invention is composed of multiple edge servers, base stations, and terminal users. The terminal users upload tasks to the base stations through wireless connections, and different base stations migrate tasks through wired connections. When a user offloads a task to the base station closest to him (the associated base station), the base station will create a task instance in the edge server connected to it. Among them, a user may create multiple task instances at the same time. When the user moves, the user's location may move. At this time, a migration strategy needs to be made based on the delay, migration cost and server resource usage, that is, whether to migrate and where to migrate. Once the migration decision is made, the original base station will migrate the task instance (not the task itself) to the target base station through a wired connection. The purpose of migration is to reduce service delay, and inappropriate migration, such as migrating tasks to servers with insufficient resources or servers in distant locations, will increase service delay and migration cost. The present invention will comprehensively consider various limiting factors and jointly optimize service delay and migration cost.

[0070] Cost of delay:

[0071] The present invention analyzes the delay cost from four perspectives: communication delay, computation delay, migration delay, and return delay. The delay in communicating with the BS is mainly determined by the task offloading delay. The speed of task offloading to the MEC system mainly depends on the wireless channel. The present invention stipulates that each base station uses an orthogonal frequency division multiple access (OFDMA) model to manage communication resources. Base station supply K Orthogonal resource blocks that do not interfere with each other, and the bandwidth of each resource block is W Each base station will allocate resource blocks equally to the terminals associated with it, and each resource block cannot be divided. The speed at which users upload tasks to the MEC system is mainly determined by the uplink. The calculation formula for the uplink transmission rate is:

[0072] (1)

[0073] in, E is the number of user terminals associated with the base station BS, The received signal-to-noise ratio is calculated as:

[0074] (2)

[0075] in, for A constant baseline transmit power, is the channel power gain, for and The distance between is the path loss exponent, is the fractional-order channel inversion power control component, is the noise power. Therefore, The communication delay caused by the offload task is:

[0076] (3)

[0077] in, For time slot Time User The amount of unloading tasks.

[0078] Computational latency refers to the time required for the edge server ES to process a task. The computing power of ES varies with the number of tasks processed at the same time. Specifically, ES starts a virtual machine for each task after it arrives and distributes computing resources evenly to these virtual machines. This article uses the number of CPU cycles per second to measure the computing power of ES. The computing power is , then the formula for calculating the delay is:

[0079] (4)

[0080] in, To handle tasks The number of CPU cycles required, for The number of virtual machines started.

[0081] Migration delay is the delay caused by service migration. Due to the high mobility of users, services need to be migrated between ESs to ensure low backhaul delay. Migration delay is mainly determined by the bandwidth of the wired link between base stations. The calculation formula is:

[0082] (5)

[0083] in, The size of the user task.

[0084] Backhaul latency is the time it takes for the edge server to return the computation result to the user (device). The delay caused by transmission, propagation, processing and queuing is negligible in the present invention because the amount of data of the calculation result is very small. In addition, the delay caused by the propagation, processing and queuing of the calculation result is similar to The number of hops between the associated edge server and the target edge server is determined by the following formula:

[0085] (6)

[0086] in, Is the associated edge server With the target edge server The number of hops between is a positive coefficient.

[0087] Migration costs:

[0088] When services are migrated, corresponding energy consumption will be generated, and its size depends on the distance between the source server and the target server. The calculation formula is:

[0089] (7)

[0090] in, is the migration energy consumption coefficient.

[0091] In order to improve the performance of the MEC system and the service quality of users at the same time, the present invention optimizes the service migration problem. As the user moves, the system will decide whether to migrate the service and find the optimal migration destination based on the usage of communication and computing resources, taking into account latency and cost.

[0092] Processing tasks The delay of is composed of communication delay, computation delay, migration delay and return delay, that is,

[0093] (8)

[0094] The total latency of the system is defined as the latency of processing all tasks. The sum of the delays generated is

[0095] (9)

[0096] The migration cost of the system is defined as the migration cost of all tasks The sum of the migration energy consumption generated is

[0097] (10)

[0098] In summary, the overall goal of the optimization of the present invention is to minimize the system delay cost and migration cost, that is,

[0099] (11)

[0100] To achieve this goal, the present invention will use DRL to learn an optimal migration strategy that can make migration decisions based on the real-time situation of the system to minimize system delay and cost. Specifically, the present invention first uses the Markov Decision Process (MDP) to model and solve the adaptive service migration problem.

[0101] Specifically, the state space, action space, and reward function are defined as follows.

[0102] State space: In ASM, the agent needs to make migration decisions based on the user's geographic location and the usage of the edge server. Therefore, the present invention defines the state space as follows:

[0103] (12)

[0104] in, Is a user The distance between the BS and the one it is associated with, It is an edge server The number of virtual machines started in yes Available computing resources, expressed as a percentage, where 0 represents No computing resources are available. The associated base station here refers to the base station closest to the user (device).

[0105] Action space: After knowing the state of the environment, the agent will take a series of actions to interact with the environment. In this paper, we define the action space as

[0106] (13)

[0107] in, is the target edge server. (associated edge server), it means that service migration will not be performed. When the agent makes an action, the state of the environment changes from Convert to .

[0108] Reward function: The reward is an evaluation based on the agent's current environment and actions. In this invention, the reward function is defined as

[0109] (14)

[0110] in, Indicates the user's service quality. is the maximum tolerable delay for the user, For the task The total delay, For the task The migration energy consumption, is the maximum tolerable migration energy consumption of the base station.

[0111] The algorithm of the present invention is based on the deep Q network (DQN). Q-Learning is a model-free reinforcement learning algorithm that learns an action-value function (Q function) to represent the expected return of taking a certain action in a given state. The update formula of the Q function is as follows:

[0112] (15)

[0113] in, is the learning rate, is the discount factor, It’s an instant reward. is the next state, is the next action. The goal of Q-Learning is to learn an optimal strategy , for this purpose, an optimal Q function can be given

[0114] (16)

[0115] When faced with high-dimensional state and action spaces, the convergence rate of Q-Learning will drop rapidly. In this case, neural networks can be used to learn , the neural network is called DQN. In DQN, the Q function is expressed as ,in is the network weight. In order to break the data correlation, DQN introduces an experience replay mechanism. When the agent explores the environment, it stores the experience (state, action, reward, next state) in the memory bank, and randomly extracts small batches of samples from it for learning during training. In order to stabilize the training process, DQN uses two neural networks: a current network and a target network. The parameters of the target network are updated to the parameters of the current network at regular intervals, thereby reducing the oscillation of the target value. During the training process, DQN updates the parameters through iterations. , so that the loss function is minimized. After the sample is passed into the current network, a Q value is obtained by forward propagation, and then the mean square error (MSE) loss between the Q value and the target Q value is calculated, which is defined as

[0116] (17)

[0117] in, is the target Q value. Then the network differentiates the loss function according to the weights, and the gradient is as follows

[0118] (18)

[0119] in, Represents any parameter in the network. Then, the gradient descent algorithm is used to update the network parameters:

[0120] (19)

[0121] in, is the learning rate.

[0122] The present invention improves the traditional DQN algorithm and applies it to the adaptive service migration method. Specifically, in order to solve the problem of overestimation of Q value and improve its accuracy, the present invention designs the ASM method based on DQN, Double Q-learning and Dueling network. At the same time, in order to improve its training efficiency and accelerate the convergence speed, priority experience playback and noise network are introduced.

[0123] The above improvements are described in detail below. In DQN, there is a problem of overestimation in the Q value update process, especially when the environment is more complex. This is because the DQN algorithm uses the same Q network when calculating the maximum Q value. And in standard DQN, the Q value function directly outputs the Q value corresponding to each action. However, not every state requires accurate distinction of all possible actions. In some states, the choice of certain actions may not be important. To this end, the present invention combines Double DQN and Dueling DQN to solve these two problems.

[0124] Double DQN solves the problem of overestimation of Q value by introducing two different Q networks: one for selecting actions and the other for evaluating the value of actions. The core idea is to separate action selection and Q value estimation. The specific formula is as follows:

[0125] (20)

[0126] in, is the current Q network used to select actions, is the target Q network used to evaluate the value of actions.

[0127] Dueling DQN decomposes the Q-value function into a state-value function And the action value function The two parts improve the estimation accuracy of Q value and accelerate the convergence of the model. The core idea is to divide the neural network into value branch and advantage branch, where the value branch is used to estimate the state value. , the advantage branch is used to estimate the action value function The final Q value is calculated by combining the outputs of these two branches:

[0128] (twenty one)

[0129] in, is the parameter of the dominant branch, is the parameter of the value branch, is the size of the action space.

[0130] By combining the above two methods, the performance of the DQN algorithm can be significantly improved in terms of accuracy, stability, convergence speed and adaptability.

[0131] Traditional DQN uses a -greedy-based exploration method. The basic idea is that when selecting an action, most of the time the current optimal action (exploitation) is selected, but there is a certain probability of selecting a random action (exploration), and this probability is controlled by the value. Although -greedy is simple and easy to implement, it also has some obvious disadvantages, such as insufficient exploration in the early stages; in the later stages, time may be wasted on unnecessary exploration. To this end, the -greedy strategy can be replaced by adding noise (Noisy Net) to the parameters of the neural network.

[0132] The basic idea of ​​Noisy Net is to express the weight parameters in the neural network as a combination of deterministic parameters and noise parameters. The specific formula is as follows:

[0133] (twenty two)

[0134] in, is the deterministic part of the weight. is the standard deviation part of the weight, which is used to control the size of the noise. is noise sampled from a standard normal distribution. In this way, the output of the neural network depends not only on the input, but also on the noise, which introduces randomness in the selection of actions. In this way, in each fully connected layer (or other applicable layer), for the input of the neural network x , output y for:

[0135] (twenty three)

[0136] During training, the noise parameters and deterministic parameters and deterministic parameters The model automatically adjusts the degree of exploration by updating the noise together. Through this adjustment, Noisy Net can explore more in the early stage of training, and as the training progresses, the influence of noise gradually decreases, allowing the model to make more use of the learned knowledge.

[0137] Secondly, the traditional experience replay used by DQN does not take into account the importance of different experiences when randomly sampling, which may lead to inefficient learning. Therefore, the experience samples can be prioritized, and the priority of each experience is determined by its temporal difference error Decide. It measures the difference between the current Q value and the target Q value, reflecting the importance of the experience to learning. The calculation formula is as follows:

[0138] (twenty four)

[0139] In this way, experiences with larger error losses are more likely to be selected because they contribute more to the updated strategy. As training progresses, the priorities and experiences in the experience library are constantly updated to keep the priorities of fresh and important experiences. The advantage of prioritized experience replay is that it can more effectively utilize experience, speed up convergence, and improve the performance of the final strategy.

[0140] The ASM method comprises the following steps:

[0141] First initialize the experience replay pool D and the number of trainings, and initialize the network and network parameters according to the state space in equation (12), the action space in equation (13), and the reward function set in equation (14), and copy the network parameters to the target network. At the beginning of each round of training, the user A service instance will be generated in the nearest ES and System Status Enter the network. After that, the network will make a migration decision based on the current state , and calculate the reward according to formula (13) , and transfer to the next state The network will then take this experience sample Deposit to D In the example above, the empirical samples are prioritized according to the time difference error. D A batch of samples are selected for network training. These samples are input into the network, and the mean square error loss is minimized using the gradient descent algorithm according to the loss function, with the goal of making the network output closer to the target network. C After the round, the parameters of the network are copied to the target network.

[0142] The final process of using the solution is:

[0143] (1) Use MDP to model the system model and set the corresponding state space, action space and reward function.

[0144] (2) Improve DQN and train a neural network to output user migration decisions.

[0145] (3) The MEC system migrates the user’s task instance to the corresponding edge server based on the migration decision.

[0146] Figure 2 This is an overview of the DRL-based adaptive edge service migration method (ASM) proposed in this invention. , the DRL agent will obtain status information from the MEC environment , and input it to the Noisy Q network. The network then outputs action information , and input it into the environment to obtain the action reward Status information for the next time slot Afterwards, the agent will collect , and put it into the experience replay pool. The experience replay pool will prioritize the samples according to their importance, and extract a batch of samples according to the priority and input them into the network for training. After multiple rounds of training, the network can make the best migration strategy in real time according to the MEC environment.

[0147] Test example

[0148] This experimental example uses two python modules to implement ASM. The first module consists of a MEC environment and randomly moving users, and the second module implements a DQN agent to make migration decisions. In the first module, the present invention uses a 6x6 grid environment to simulate the service migration framework. In this environment, each location has a BS, which is connected to the UE in the grid, and it is assumed that each BS covers all areas of its grid location. In order to be closer to the actual situation, this experimental example randomly deploys 6 to 36 edge servers in these BSs instead of deploying an ES in each BS. The present invention evaluates the CPU as an ES resource because in compute-intensive tasks, CPU performance directly affects the response time and processing power of the application, while the impact of memory and bandwidth is relatively small. Each server The total CPU resources are 70 billion instructions per second, the memory capacity is 16GB, the task queue capacity is 1200, and the size of each task is 5MB. During the experiment, the number of tasks processed by the server at any time should be less than 1200. Each server is configured with a maximum of ten virtual machines, and the memory size of each virtual machine is 0.5GB.

[0149] In this test example, the number of users is set between 20 and 120, and they move randomly in four directions in each time slot, and unload the service to ES. The task sending rate follows the Poisson distribution, and all tasks sent have the same priority and are uniformly distributed. The present invention assumes that each task can be sent in the time slot Therefore, this test example is set in the time slot Initially, user-configured virtual machines are migrated, not tasks.

[0150] In the second module, this experiment uses a single DQN agent to make migration decisions. The experiment consists of 500 episodes, each of which includes 80 steps. The agent stores each step as a sample in ReplayMemory for training the neural network. In order to compare with earlier studies, this experiment only makes migration decisions for a single user in the simulation, while the migration decisions of other users are randomly generated.

[0151] This test example compares the ASM method provided by the present invention with the D-ASM, Dist and AM methods, all of which are also composed of a single DQN agent. Among them, D-ASM uses the traditional DQN algorithm for service migration. Dist performs service migration based on the location of the terminal, always ensuring that the distance between the user and the ES is the shortest. AM always migrates the service to the server that the agent considers to be optimal. The present invention compares the performance of these four schemes based on the cumulative rewards during training, the average service delay, the average migration energy consumption, and the average migration rejection rate. Among them, the cumulative reward reflects the convergence of the model, the user's QoS is determined by the service delay, the restriction on migration energy consumption effectively reduces the operator's cost, and the migration rejection rate shows the model's ability to perform effective migration to reduce service interruptions.

[0152] like Figure 3 As shown, Figure 3 (a)-(f) compare the convergence of each algorithm under different numbers of servers when the number of users is 60. In the initial stage, the DRL agent is in the exploration stage and needs to continuously optimize the migration strategy through trial and error. The D-ASM agent can converge only after training for 100-150 episodes, while ASM can converge within 50 episodes and always maintain the highest cumulative reward. This is because the ASM agent can learn better strategies and therefore can better balance QoS and migration energy consumption. Specifically, ASM can make migration decisions based on the usage of ES and the distance between the user and ES, which can minimize processing delay, migration delay, and migration energy consumption, respectively, thereby maximizing long-term rewards. Therefore, under limited resources, the cumulative reward obtained by ASM is higher than other solutions.

[0153] One of the goals of service migration is to enhance user experience, so it is necessary to minimize service delay. Service delay includes communication delay, computing delay, migration delay, and backhaul delay. Figure 4 (a) and (b) show the average service delay of the four schemes with different numbers of ES and users, respectively. Figure 4 (a) shows that as the number of ES increases, the service delay tends to decrease. This is because more ES can generate more available resources, thereby speeding up the processing of tasks. In particular, compared with D-ASM, Dist and AM, the average service delay of ASM is reduced by 15.4%, 66.1% and 75.4% respectively. Figure 4 As shown in (b), as the number of terminals increases, the service delay tends to increase. This is because when the number of tasks increases, more virtual machines will be generated, resulting in higher processing delays. Therefore, proper migration is essential to reducing service delays. ASM generates the lowest service delay, which is reduced by 23%, 70% and 76.9% compared with D-ASM, Dist and AM respectively. This is because ASM can migrate services to the appropriate ES for processing. However, the learning efficiency of the agent of D-ASM is low, and the migration decisions made are poor. Dist only considers the distance between ES and users, and may not necessarily migrate services to ES with sufficient resources. AM has a high migration delay due to frequent migration.

[0154] In order to reduce the cost of operators, another goal of service migration is to reduce migration energy consumption. Migration energy consumption is related to the number of migrations and the migration distance. Frequent migrations and migrations over long distances will result in higher migration energy consumption. Figure 5 (a) and (b) show the average migration energy consumption of the four schemes when the number of ESs and the number of users are different. Figure 5 As shown in (a), as the number of ES increases, the migration energy consumption tends to increase. This is because when the number of ES is large, the DRL agent will perform more migrations to reduce service latency. Compared with the three solutions of D-ASM, Dist and AM, ASM generates lower migration energy, which is reduced by 54.3%, 92.5% and 95.2% respectively. Figure 5 (b) shows that as the number of terminals increases, the migration energy consumption tends to increase. This is because when the number of users is large, the number of tasks generated will also increase. Therefore, it is necessary to frequently migrate tasks to ES with more resources. ASM generates the least migration energy consumption, which is 63.8%, 92.9% and 94.9% less than D-ASM, Dist and AM respectively. This is because ASM will fully consider the migration distance and migrate as little as possible. However, the learning efficiency of D-ASM makes it unable to make the optimal migration decision. Dist only considers the distance between the user and ES, but does not consider the migration distance. AW has high migration energy consumption due to frequent migration.

[0155] Due to the limited resources in ES, service migration is not necessarily successful. When migration is rejected, the system needs to reconsider the migration target or abandon the migration, which will cause more waiting delays and thus reduce QoS. Figure 6 The service migration rejection rates of the four schemes are given. This test case assumes that if a server has hosted 10 services, it will not accept other services. Figure 6 As shown in (a), when the number of servers is small, the service rejection rate is high. Compared with the three schemes of D-ASM, Dist and AM, the proposed method (ASM) has a lower service rejection rate, which is reduced by 9.1%, 35.8% and 42.9% on average respectively. Figure 6 (b) shows that the migration rejection rate increases as the number of users increases. This is because the increase in the number of users will cause more computing resources to be occupied, causing most servers to refuse to receive services. The migration rejection rate of ASM is significantly lower than that of the other three solutions, which is reduced by 4.8%, 17.6% and 19% respectively. This is because ASM will fully consider the resource usage of the server to make migration decisions. However, D-ASM cannot learn better migration decisions. Dist and AM do not consider the usage of computing resources, so the migration rejection rate is higher.

[0156] Compared with the existing technology, the ASM method proposed in the present invention can formulate migration strategies in real time according to the situation of the MEC system to maximize QoS and minimize migration costs. At the same time, the improved DQN algorithm adopted by the present invention can converge faster and train a more accurate neural network. The present invention verifies the effectiveness and feasibility of the ASM method through a large number of simulation experiments. Compared with the other three benchmark methods (D-ASM, AM, Dist), the ASM method can make better migration decisions. Under different MEC environment parameter settings (the number of terminal users and the number of MEC servers), the ASM method can significantly reduce service delays, migration costs, and migration rejection rates.

[0157] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes, and they can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A DRL-based adaptive edge service migration method, characterized in that: The steps include: Step 1: In each time slot, respectively obtain the distance between the user terminal and its associated base station, the number of virtual machines started in each edge server, and the amount of available computing resources in each edge server; The associated base station is the base station closest to the user terminal; Step 2: Select an edge server connected to a base station associated with the user terminal as an associated edge server, and offload the user task to the associated edge server; Step 3: Taking the minimum sum of delay cost and migration cost as the optimization goal, determine the target edge server; Step 4: Migrate the user task from the associated edge server to the target edge server, and the target edge server processes the user task; If the determined target edge server is the associated edge server, there is no need to migrate the user task; The calculation model of the delay cost is: ； in, , , , They represent the communication delay, computation delay, migration delay, and return delay caused by processing user tasks; The calculation model of the migration cost is: ； in, is the migration energy consumption coefficient; To associate an edge server With the target edge server The number of hops between In step 3, the target edge server is determined based on the DQN network, including the following steps: Step 1: Build an improved DQN network, which includes a Double DQN network and a Dueling DQN network; randomly generate network parameters, and add noise to the randomly generated network parameters to obtain initial network parameters; Step 2: Integrate the distance between the user terminal and its associated base station, the number of virtual machines started in each edge server, and the amount of available computing resources in each edge server into a state space ; Step 3: Transform the state space Input into the improved DQN network to get the action space ; The action space This is the migration strategy; Step 4: Determine the action space Reward value obtained And the new system status The experience sample Store in memory; Among them, the reward function is set as: ； In the formula, is the maximum tolerable delay for the user, is the maximum tolerable migration energy consumption of the base station; Step 5, iteratively optimize and train the improved DQN network with the goal of minimizing the Q value difference between the main network and the target network until the set number of iterations is reached to obtain the optimal improved DQN network; the optimal improved DQN network is used as the migration strategy network; Step 6: input the distance between the user terminal and its associated base station in the current time slot and the number of virtual machines started in each edge server into the migration strategy network, and the migration strategy network outputs the target edge server as the migration strategy; In step 5, it also includes: In each training process, samples with large temporal difference errors in the memory are selected as training samples; The calculation formula of the time difference error is: ； In the formula, is the time difference error, is the Q value of the main network, is the Q value of the target network; is the network weight.

2. The DRL-based adaptive edge service migration method according to claim 1, characterized in that: The calculation formula for the communication delay caused by processing user tasks is: ； in, The uplink transmission rate at which the user terminal uploads the user task to the associated base station, is the user task amount in the current time slot.

3. The DRL-based adaptive edge service migration method according to claim 2, characterized in that: The uplink transmission rate at which the user terminal uploads the user task to the relevant base station is: ； in, is the number of user terminals associated with the base station, is the receive signal-to-noise ratio, The number of orthogonal resource blocks provided to the base station, The bandwidth of each resource block.

4. The DRL-based adaptive edge service migration method according to claim 3, characterized in that: The calculation formula for the computational delay caused by processing user tasks is: ； in, The number of CPU cycles required for edge servers to process user tasks, The number of virtual machines started for the edge servers, The computing power of the edge server.

5. The DRL-based adaptive edge service migration method according to claim 4, characterized in that: The calculation formula for the migration delay caused by processing user tasks is: ； in, is the bandwidth of the wired link between base stations, The size of the user's task.

6. The DRL-based adaptive edge service migration method according to any one of claims 1 to 5, characterized in that: The calculation formula for the return delay caused by processing user tasks is: ； in, To associate an edge server With the target edge server The number of hops between Is a positive coefficient.

Images