Service migration and resource allocation joint optimization method based on multi-agent reinforcement learning

Through multi-agent reinforcement learning and convex optimization methods, optimize service migration and resource allocation in IoT environments, the communication delay problem caused by insufficient computing power and mobility of user equipment is solved, real-time and fast service migration and resource allocation strategies are realized, and the system is reduced. System migration cost and response time.

CN118916159APending Publication Date: 2024-11-08YUNNAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410961573.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-17
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

When performing computing-intensive and latency-sensitive applications on mobile devices, user equipment (UE) cannot provide sufficient computing power, and user mobility causes increased communication delays during edge server migration, affecting service quality.

Method used

Adopt the joint optimization method of service migration and resource allocation based on multi-agent reinforcement learning. By obtaining user information in the IoT environment, judging service migration requirements, calculating the optimization response time and cost during migration, and solving it based on multi-agent reinforcement learning and convex optimization methods Joint optimization problem, obtain service migration strategy and optimal resource allocation strategy.

Benefits of technology

Provide appropriate service migration and resource allocation strategies in real time and quickly in real time in real environments, reducing user task response time and system migration costs, and ensuring service quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916159A_ABST
    Figure CN118916159A_ABST
Patent Text Reader

Abstract

The invention discloses a service migration and resource allocation joint optimization method based on multi-agent reinforcement learning, and the method comprises the steps: obtaining user information U in an IoT environment, and judging whether a user service context needs to be migrated or not; if migration is needed, calculating optimization response time and service migration cost when migration occurs; producing a joint optimization problem based on the optimization response time and the service migration cost; based on a multi-agent reinforcement learning and convex optimization method, solving a joint optimization problem to obtain a service migration strategy and an optimal resource allocation strategy; according to the method, service migration and resource allocation are combined, and meanwhile, the topological structure of the network and the isomerism of each edge server are considered; the joint optimization problem is solved through a multi-agent reinforcement learning and convex optimization method, and proper service migration and resource allocation strategies can be provided more timely and quickly in a real environment, so that the migration cost of a system is reduced while the task response time of a user is shortened.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of edge service technology, and specifically relates to a service migration and resource allocation joint optimization method based on multi-agent reinforcement learning. Background Art

[0002] In recent years, with the rapid development of 5G technology and the explosive growth of user equipment (UE), computationally intensive and delay-sensitive applications such as augmented reality and online interactive games have attracted widespread attention. These applications place high demands on the powerful computing power and intensive computing resources of UE. However, most current UEs are limited by battery capacity and size, making it impossible to provide the required computing power, making it impossible to provide satisfactory results when executing such applications on mobile devices.

[0003] Uploading tasks to a central cloud with sufficient computing resources for processing is a good solution, but due to the geographical isolation of cloud computing centers and UEs, they will generate high communication delays, which may exceed the delay requirements of the tasks. To meet this challenge, mobile edge computing MEC has become a highly anticipated new computing paradigm. Compared with traditional cloud computing, MEC transfers computing-intensive and delay-sensitive tasks from resource-constrained UEs to nearby edge servers ES, thereby reducing the load on terminals and networks and significantly reducing the end-to-end communication delay. At the same time, the edge server has abundant computing resources relative to the UE, which can effectively meet the delay requirements of different applications, thereby improving the quality of service QoS. However, the mobility of users in wireless networks brings new challenges to the computational offloading of long-term services. Due to the mobility of users, tasks cannot be simply offloaded to edge servers, because when users move to another edge node far away from the ES where their services are located, it will cause high communication delays. Therefore, in order to ensure QoS when moving between different edge nodes, an effective mobility management solution needs to be adopted at the edge of the network. Summary of the invention

[0004] In view of this, the object of the present invention is to provide a joint optimization method for service migration and resource allocation based on multi-agent reinforcement learning to solve the above technical problems.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A joint optimization method for service migration and resource allocation based on multi-agent reinforcement learning, comprising:

[0007] Obtain user information U in the IoT environment and determine whether the user service context needs to be migrated;

[0008] If migration is required, calculate the optimized response time and service migration cost when migration occurs;

[0009] Produce joint optimization problems based on optimizing response time and service migration costs;

[0010] Based on multi-agent reinforcement learning and convex optimization methods, the joint optimization problem is solved to obtain the service migration strategy and the optimal resource allocation strategy.

[0011] Furthermore, the user information U in the IoT environment is obtained to determine whether the user service context needs to be migrated, including:

[0012] Obtain IoT user information U = {u1, u2, ..., u n}; wherein m edge servers and the base station are closely deployed at the same location, and the set of the base station and the edge servers connected thereto is M = {1, 2, ..., m};

[0013] Get the user information U, which contains the user u at time slot t i The placement status of a specific service instance on each edge server in, The constraints are:

[0014]

[0015] according to Determine whether the user service context needs to be migrated; Indicates that user u in time slot t i A specific service instance is placed on edge server j. Indicates that user u in time slot t i A specific service instance of is not placed on edge server j, Indicates that user u in time slot t i If the edge server where the specific service instance is located This means that user u i If a specific service instance of the service is migrated, the user service context needs to be migrated.

[0016] Furthermore, the optimized response time and service migration cost when migration occurs are calculated, including:

[0017] Get user u i When the service context of user u is migrated i Service suspension delay Specific state context synchronization delay and service restoration delay Calculate user u iTotal migration delay in time slot t

[0018]

[0019] Get user u i The user tasks that need to be processed in each time slot generated by the service k used are transmitted to user u i When directly associated with the edge server, user u i Uplink communication delay Downlink communication delay for transmitting task results and user u i The transmission delay between the directly associated edge server and edge server j and the transmission task result return delay Calculate user u i The total communication delay at time slot t

[0020]

[0021] Get user u at time slot t i The computational latency of executing user tasks on the edge server where the specific service instance is located

[0022]

[0023] in, For edge server j in time slot t, i Allocated computing resources, F j is the maximum available computing resource of edge server j, The computational intensity required to process the tasks generated by service k in time slot t, The size of the task generated for service k in time slot t;

[0024] According to user i Total migration delay in time slot t User i The total communication delay at time slot t and the computational delay of edge server j executing the user task at time slot t Calculate the optimal response time for executing user tasks in time slot t

[0025]

[0026] Get the service migration cost of the user in time slot t

[0027]

[0028] Among them, L-1 is the service k from the edge server Migrate to edge servers The number of hops required, w E is the cost coefficient.

[0029] Further, get user u i When the service context of user u is migrated i Service suspension delay Specific state context synchronization delay and service restoration delay include:

[0030] Get the user u when performing migration i Service suspension delay:

[0031]

[0032] in, For user u i The service suspension delay, is the computational intensity required to suspend service k, is the size of the user-specific state context, is user u at time slot t-1 i The edge server where the specific service instance is located is user u i Allocated computing resources, The user u before the migration i The edge server where the specific service instance is located The placement status, The user u after the migration occurs i The edge server where the specific service instance is located The placement status;

[0033] Get the user u when performing migration i The specific state context synchronization delay is:

[0034]

[0035] in, For user u i The specific state context synchronization delay of L = {c, ..., j} represents the number of edge server hops required for the shortest path from edge server c to edge server j, B l,l+1 is the bandwidth between edge servers l and l+1;

[0036] Get the user u when performing migration i Service restoration delay:

[0037]

[0038] in, For user u i Service restoration delay represents the computational intensity required to restore service k, The user u after the migration occurs i The edge server where the specific service instance is located For user u i Allocated computing resources.

[0039] Further, get user u i The user tasks that need to be processed in each time slot generated by the service k used are transmitted to user u i When directly associated with the edge server, user u i Uplink communication delay Downlink communication delay for transmitting task results and user u i The transmission delay between the directly associated edge server and edge server j and the transmission task result return delay include:

[0040] Get user tasks and transfer them to user u i Uplink communication latency when directly connected to the edge server

[0041]

[0042] in, For user tasks, transfer to user u i Uplink communication latency when directly connected to the edge server For user u i Wireless transmission rate to edge server, For user u i The real-time location of the user u, W is the user u i and the wireless channel bandwidth between the edge servers directly associated with it, For user u i The transmission power, N0 is the Gaussian noise power spectrum density, g t For edge server j and user u i The wireless channel path loss between them, β0 represents the channel power at d0 = 1m, Represents the relationship between edge server j and user u i The distance between

[0043] Get user tasks and transfer them to user u iDownlink communication delay when directly associated with edge servers

[0044]

[0045] in, The size of the result returned by the task serving k in time slot t;

[0046] When the user task is transmitted to edge server j, user u i The transmission delay between the directly associated edge server and edge server j and the transmission task result return delay

[0047]

[0048] Among them, B l,l+1 is the bandwidth between edge servers l and l+1.

[0049] Furthermore, a joint optimization problem is generated based on optimizing response time and service migration cost, including:

[0050] Based on the cost-performance trade-off, according to the optimized response time and service migration costs Get the performance and cost trade-off formula:

[0051]

[0052] Among them, T * is the average service response delay constraint, μ is the basic reward for completing the service within the average response delay constraint in time slot t, and w Y is the weight of the response delay, Migration costs incurred due to service migration;

[0053] According to the performance and cost trade-off formula, the joint optimization problem is determined as follows:

[0054]

[0055] in, For edge server j in time slot t, i Allocated computing resources, F j is the maximum available computing resource of edge server j, is user u at time slot t i The placement status of a specific service instance on edge server j.

[0056] Furthermore, based on multi-agent reinforcement learning and convex optimization methods, the joint optimization problem is solved to obtain the service migration strategy and the optimal resource allocation strategy, including:

[0057] Based on the deployment data of edge servers in the IoT system in the network topology and the user data of IoT users, the total state of the environment in multi-agent reinforcement learning, as well as the state, action and reward of each agent, are constructed;

[0058] Overall state of the environment t is the system state at any time slot t;

[0059]

[0060] Among them, the system state s in any time slot t t By user u in IoT i The array is represented by the real-time state of service k in time slot t. The computational intensity required to process the tasks generated by service k The size of the user-related state context of service k User real-time location and user u i The edge server where the specific service instance is located

[0061] The state of each agent is the state that each agent can observe in time slot t

[0062] According to the currently observed state, the action generated by each agent in time slot t is

[0063] Among them, j represents user u i The specific edge server to which a specific service instance is migrated;

[0064] The reward for each agent is The total reward is

[0065] According to the overall state of the environment, as well as the state, action, and reward of each agent, a service migration strategy model is trained based on a multi-agent reinforcement learning method.

[0066] Determine user u according to the service migration strategy i service migration strategy for a specific service instance;

[0067] After determining the service migration strategy, the resource allocation strategy is determined based on the convex optimization method, including:

[0068] It is known that given the service migration strategy in time slot t, Only with Since resource allocation is performed in a distributed manner at each edge server j within time slot t, the transformation objective function is:

[0069]

[0070] Taking the first and second derivatives of the objective function, we get:

[0071]

[0072] in,

[0073] According to the first-order derivative results and the second-order derivative results of the objective function, it is determined that the objective function is a convex function;

[0074] Based on the objective function being a convex function, the resource allocation scheme under the migration strategy is determined by solving the Lagrangian function:

[0075]

[0076] Calculate the optimal resource allocation through KKT conditions The optimal solution is:

[0077]

[0078] in, is the optimal resource allocation strategy.

[0079] The beneficial effects of the present invention are:

[0080] The present invention combines service migration and resource allocation, while taking into account the topological structure of the network and the heterogeneity of each edge server; by solving the joint optimization problem through multi-agent reinforcement learning and convex optimization methods, it can provide appropriate service migration and resource allocation strategies in a more real-time and rapid manner in a real environment, so as to reduce the user task response time while reducing the migration cost of the system.

[0081] Other advantages, objectives and features of the present invention will be described in the following description and will be apparent to those skilled in the art to some extent, or they may be taught from the practice of the present invention. The purposes and other advantages of the present invention may be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0082] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0084] Figure 1A method flow chart of a method for joint optimization of service migration and resource allocation based on multi-agent reinforcement learning in an embodiment of the present invention;

[0085] Figure 2 This is a general implementation flow chart of a service migration and resource allocation joint optimization method based on multi-agent reinforcement learning in an embodiment of the present invention.

[0086] Figure 3 The diagram is a migration diagram of service migration caused by user movement in a joint optimization method for service migration and resource allocation based on multi-agent reinforcement learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0087] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0088] See also Figure 1-Figure 2 The present invention proposes a joint optimization method for service migration and resource allocation based on multi-agent reinforcement learning, comprising:

[0089] S1. Obtain user information U in the IoT environment and determine whether the user service context needs to be migrated;

[0090] Specifically:

[0091] Obtain IoT user information U = {u1, u2, ..., u n}; wherein m edge servers and the base station are closely deployed at the same location, and the set of the base station and the edge servers connected thereto is M = {1, 2, ..., m};

[0092] Get the user information U, which contains the user u at time slot t i The placement status of a specific service instance on each edge server in, The constraints are:

[0093]

[0094] according to Determine whether the user service context needs to be migrated; Indicates that user u in time slot t i A specific service instance is placed on edge server j. Indicates that user u in time slot t i A specific service instance of is not placed on edge server j, Indicates that user u in time slot t iIf the edge server where the specific service instance is located This means that user u i If a specific service instance is migrated, the user service context needs to be migrated;

[0095] S2. If migration is required, calculate the optimized response time and service migration cost when migration occurs; if migration is not required, the migration time and cost are 0;

[0096] Specifically:

[0097] Get user u i When the service context of user u is migrated i Service suspension delay Specific state context synchronization delay and service restoration delay

[0098] Get the user u when performing migration i Service suspension delay:

[0099]

[0100] in, For user u i The service suspension delay, is the computational intensity required to suspend service k, is the size of the user-specific state context, is user u at time slot t-1 i The edge server where the specific service instance is located is user u i Allocated computing resources, The user u before the migration i The edge server where the specific service instance is located The placement status, The user u after the migration occurs i The edge server where the specific service instance is located The placement status;

[0101] Get the user u when performing migration i The specific state context synchronization delay is:

[0102]

[0103] in, For user u i The specific state context synchronization delay of L = {c, ..., j} represents the number of edge server hops required for the shortest path from edge server c to edge server j, B l,l+1 is the bandwidth between edge servers l and l+1;

[0104] Get the user u when performing migration i Service restoration delay:

[0105]

[0106] in, For user u i Service restoration delay represents the computational intensity required to restore service k, The user u after the migration occurs i The edge server where the specific service instance is located For user u i Allocated computing resources;

[0107] Calculate user u i Total migration delay at time slot t

[0108]

[0109] Get user u i The user tasks that need to be processed in each time slot generated by the service k used are transmitted to user u i When directly associated with the edge server, user u i Uplink communication delay Downlink communication delay for transmitting task results and user u i The transmission delay between the directly associated edge server and edge server j and the transmission task result return delay

[0110] Get user tasks and transfer them to user u i Uplink communication latency when directly connected to the edge server

[0111]

[0112] in, For user tasks, transfer to user u i Uplink communication latency when directly connected to the edge server For user u i Wireless transmission rate to edge server, For user u i The real-time location of the user u, W is the user u i and the wireless channel bandwidth between the edge servers directly associated with it, For user u i The transmission power, N0 is the Gaussian noise power spectrum density, gt For edge server j and user u i The wireless channel path loss between them, β0 represents the channel power at d0 = 1m, Represents the relationship between edge server j and user u i The distance between

[0113] Get user tasks and transfer them to user u i Downlink communication delay when directly associated with edge servers

[0114]

[0115] in, The size of the result returned by the task serving k in time slot t;

[0116] When the user task is transmitted to edge server j, user u i The transmission delay between the directly associated edge server and edge server j and the transmission task result return delay

[0117]

[0118] Among them, B l,l+1 is the bandwidth between edge servers l and l+1;

[0119] Calculate user u i The total communication delay at time slot t

[0120]

[0121] Get user u at time slot t i The computational latency of executing user tasks on the edge server where the specific service instance is located

[0122]

[0123] in, For edge server j in time slot t, i Allocated computing resources, F j is the maximum available computing resource of edge server j, The computational intensity required to process the tasks generated by service k in time slot t, The size of the task generated for service k in time slot t;

[0124] According to user i Total migration delay in time slot t User iThe total communication delay at time slot t and the computational delay of edge server j executing the user task at time slot t Calculate the optimal response time for executing user tasks in time slot t

[0125]

[0126] Get the service migration cost of the user in time slot t

[0127]

[0128] Among them, L-1 is the service k from the edge server Migrate to edge servers The number of hops required, w E is the cost coefficient;

[0129] S3, production joint optimization problem based on optimizing response time and service migration cost;

[0130] Specifically:

[0131] Based on the cost-performance trade-off, according to the optimized response time and service migration costs Get the performance and cost trade-off formula:

[0132]

[0133] Among them, T * is the average service response delay constraint, μ is the basic reward for completing the service within the average response delay constraint in time slot t, and w Y is the weight of the response delay, Migration costs incurred due to service migration;

[0134] According to the performance and cost trade-off formula, the joint optimization problem is determined as follows:

[0135]

[0136] in, For edge server j in time slot t, i Allocated computing resources, F j is the maximum available computing resource of edge server j, is user u at time slot t i The placement status of a specific service instance on edge server j;

[0137] S4. Solve the joint optimization problem based on multi-agent reinforcement learning and convex optimization methods to obtain the service migration strategy and optimal resource allocation strategy;

[0138] Specifically:

[0139] Based on multi-agent reinforcement learning and convex optimization methods, the joint optimization problem is solved to obtain the service migration strategy and the optimal resource allocation strategy, including:

[0140] Based on the deployment data of edge servers in the IoT system in the network topology and the user data of IoT users, the total state of the environment in multi-agent reinforcement learning, as well as the state, action and reward of each agent, are constructed;

[0141] Overall state of the environment t is the system state at any time slot t;

[0142]

[0143] Among them, the system state s in any time slot t t By user u in IoT i The array is represented by the real-time state of service k in time slot t. The computational intensity required to process the tasks generated by service k The size of the user-related state context of service k User real-time location and user u i The edge server where the specific service instance is located

[0144] The state of each agent is the state that each agent can observe in time slot t

[0145] According to the currently observed state, the action generated by each agent in time slot t is

[0146] Among them, j represents user u i The specific edge server to which a specific service instance is migrated;

[0147] The reward for each agent is The total reward is

[0148] According to the overall state of the environment, as well as the state, action, and reward of each agent, a service migration strategy model is trained based on a multi-agent reinforcement learning method.

[0149] Determine user u according to the service migration strategy i service migration strategy for a specific service instance;

[0150] After determining the service migration strategy, the resource allocation strategy is determined based on the convex optimization method, including:

[0151] It is known that given the service migration strategy in time slot t, Only with Since resource allocation is performed in a distributed manner at each edge server j within time slot t, the transformation objective function is:

[0152]

[0153] Taking the first and second derivatives of the objective function, we get:

[0154]

[0155] in,

[0156] According to the first-order derivative results and the second-order derivative results of the objective function, it is determined that the objective function is a convex function;

[0157] Based on the objective function being a convex function, the resource allocation scheme under the migration strategy is determined by solving the Lagrangian function:

[0158]

[0159] Calculate the optimal resource allocation through KKT conditions The optimal solution is:

[0160]

[0161] in, for optimal resource allocation strategy;

[0162] The working principle of the above technical solution is as follows: the present invention proposes a service migration and resource allocation solution that adapts to user mobility in an MEC environment. Specifically, the present invention solves the trade-off between service performance and migration cost, and aims to reduce service response time to improve service performance while reducing system migration cost; in order to solve the optimization problem, the following steps are required: 1) modeling and analyzing the MEC environment; 2) establishing a reasonable mathematical model for the task response time of users in the MEC and the system migration cost; 3) solving the service migration optimization problem of cost-performance trade-off;

[0163] 1. MEC environment modeling and analysis

[0164] like Figure 3 As shown in the figure, we consider an IoT system with a network topology consisting of m edge servers, where the edge servers and base stations are closely deployed in the same location. We use M = {1, 2, ..., m} to represent the set of base stations and their connected edge servers, and we use U = {u1, u2, ..., un} represents a set of IoT users, each of which uses only one service. In a real environment, a user using multiple services can be virtualized as multiple users with the same mobile trajectory using different services;

[0165] Due to the mobility of users, the edge server will update user information and corresponding service migration decisions at fixed time intervals. Therefore, we assume that time is discretized into time slots of size t, where the service request of user u in time slot t can be expressed as in The computational intensity required to process the tasks generated by service k, is the user-related state context size of service k, The size of the task generated by service k in time slot t, The size of the result returned by the task serving k in time slot t;

[0166] At the beginning of each time slot, the user task is offloaded to the base station directly associated with the user by the wireless communication link; then, the task is further forwarded from the current base station to the edge server of the base station where the user-specific service instance (SI) is located for processing; when the edge server where the user-specific SI is located is different from the edge server directly associated with the user, additional transmission delay will be generated; the more transmission hops are required between the edge server where the user-specific SI is located and the edge server directly associated with the user, the greater the additional transmission delay will be; therefore, in order to maintain a satisfactory quality of service (QoS), that is, a low service response delay, for users with irregular mobility, the user SI needs to be dynamically migrated between multiple edge servers to follow the user's mobility; Indicates that user u in time slot t i The placement of SI on each edge server, for We have the following constraints

[0167]

[0168] in Indicates that user u in time slot t i The SI is placed on server j, and It means that it is not placed on edge server j. Indicates that user u in time slot t i If the edge server where the SI is located Then user u i SI of user u is migrated. i The SIs were not migrated;

[0169] 2. Modeling and Analysis of Task Delay and Migration Cost

[0170] 1) Migration time model

[0171] By comparing the current user information of each edge server with the latest generated user information, we can obtain the migration strategy of each user-specific SI. The migration of user-specific SI from one edge server to another will cause additional delay, which consists of three parts: service suspension delay, specific state context synchronization delay, and service recovery delay;

[0172] The service suspension delay is determined by the computing resources allocated by the source edge server to the user, the user-specific context size, and the corresponding required computing intensity; therefore, user u i The service suspension delay can be expressed as

[0173]

[0174] in represents the computational intensity required to suspend service k (i.e., the number of CPU cycles required to process each bit of data), is the size of the user-specific state context, The source edge server is for user u i Allocated computing resources;

[0175] Considering that edge servers may not be directly connected, we use the adjacency matrix to describe the network topology of edge servers;

[0176]

[0177] When H i,j =1 indicates that edge server i is directly connected to edge server j through a wired link, while H i,j = 0 means no direct connection. The wired bandwidth between edge servers can be expressed as

[0178]

[0179] If H i,j =0, the channel bandwidth between edge servers is 0, that is, B i,j =0; During the service migration process, the user's SI may need to pass through multiple base stations to migrate from the source edge server to the target edge server, so the specific state context synchronization delay is affected by the size of the user's specific state context, the number of hops between the source edge server and the target edge server, and the bandwidth; Therefore, the user u i The specific state context synchronization delay is expressed as

[0180]

[0181] Where L = {i, ..., j) represents the edge servers that the shortest path from edge server i to edge server j needs to pass through, B l,l+1 is the bandwidth between edge servers l and l+1;

[0182] The service recovery delay is similar to the service suspension delay, which is related to the computing resources allocated by the target edge server for the user, the user-specific context size, and the required computing intensity; therefore, the user u i The service recovery delay is expressed as

[0183]

[0184] in represents the computational intensity required to restore service k, that is, the number of CPU cycles required to process each bit. The target edge server is user u i Allocated computing resources;

[0185] Pause latency by combining services Specific state context synchronization delay and service restoration delay We will user u i The total migration delay at time slot t is expressed as

[0186]

[0187] 2) Communication and computation time model and analysis

[0188] The communication delay between the mobile user and the edge server usually includes propagation delay and data transmission delay, but since the propagation delay is very small, we ignore the uplink and downlink propagation delays; the wireless channel path loss between the edge server and the user device can be expressed as

[0189]

[0190] in represents the distance between the edge server and the user, β0 represents the channel power at d0 = 1m, and the wireless transmission rate of the user task transmitted to the edge server directly associated with the user can be obtained according to the Shannon formula, which can be expressed as

[0191]

[0192] Where w is user u i and the wireless channel bandwidth between the associated edge servers, For user u i The uplink communication delay of a mobile user transmitting a task to a directly associated edge server can be expressed as The downlink communication delay of the task result is expressed as

[0193] Since user u i The directly associated edge server may not be consistent with the edge server where the user's specific SI is located, so that the user task may need to be transmitted through multiple base stations to reach the target edge server. The task transmission delay and task result return delay between edge servers can be expressed as

[0194]

[0195] User i The total communication delay at time slot t can be expressed as

[0196]

[0197] Since the computing power of the edge server is limited, the total computing resources allocated by the edge server to the user cannot exceed the maximum available computing resources of the edge server, that is,

[0198]

[0199] in For edge server j in time slot t, i The allocated computing resources, so in time slot t, edge server j executes user u i The computational delay of a task can be expressed as

[0200]

[0201] Execute user u at time slot t i The total response time of the service can be expressed as

[0202]

[0203] 3) Migration cost model and analysis

[0204] Since users are always moving irregularly, by dynamically migrating SI between edge servers to follow the user's mobility, the communication delay can be significantly reduced, thereby improving the service quality; however, migration between edge services will incur additional operating costs. i The service migration cost can be calculated by the following formula:

[0205]

[0206] Where L-1 is the SI from the edge server Migrate to The number of hops required, wE is the cost coefficient;

[0207] 3. Technical solution for optimizing computational offloading strategy to achieve cost-performance trade-off in mobile edge computing

[0208] The main goal of this invention is to find a service migration strategy in the MEC environment to improve the service quality (performance) of user services and reduce the system migration cost. For better representation, we use the system utility of executing each user task to represent the trade-off between performance and cost, which can be expressed as

[0209]

[0210] Where T * is the average service response delay constraint, μ is the basic reward for completing the service within the average response delay constraint in time slot t, and w Y is the weight of the response delay, is the migration cost caused by service migration; in order to keep the migration cost low and obtain better QoS, we set the main goal of the joint optimization problem of service migration and resource allocation to maximize the system utility, that is,

[0211]

[0212] The specific solution steps are as follows:

[0213] 1. Obtain the best resource allocation strategy through convex optimization method

[0214] By analyzing We can conclude that given the service migration decision at time slot t In the case of Only with Since resource allocation is performed in a distributed manner at each edge server j within time slot t, the objective function can be transformed into:

[0215]

[0216] We can get the function The first and second order derivatives of are:

[0217]

[0218] in

[0219] therefore is a Convex function, from which we can construct the Lagrangian function to solve the resource allocation solution under the determined migration strategy

[0220]

[0221] Obtain optimal resource allocation through KKT conditions The optimal solution is:

[0222]

[0223] Optimal resource allocation Can be obtained:

[0224]

[0225] 2. Obtaining service migration strategy through multi-agent reinforcement learning method

[0226] We construct the state, action, and reward of each agent in multi-agent reinforcement learning separately

[0227] Total state of the environment: The system state at any time t is represented by an array consisting of the real-time states of the users in the IoT, including the task size, computational intensity, state context size, user real-time location, and the edge server where the user-specific SI is located, i.e.

[0228]

[0229] The state of each agent: The state that each agent can observe at time t can be expressed as

[0230]

[0231] Action of each agent: Based on the currently observed state, the action generated by each agent at time t is

[0232]

[0233] Where j represents user u i Which edge server should the SI be migrated to?

[0234] Reward: Our goal is to obtain the maximum system utility through service migration decisions and optimal resource allocation. Therefore, our reward function is set as follows, where is the reward obtained by each agent, and the total reward can be expressed as

[0235]

[0236] The beneficial effects of the above technical solution are: reducing the user task response time while reducing the migration cost of the system;

[0237] Among them, how to offload user tasks from mobile devices to appropriate edge servers and allocate computing resources is a core issue in the field of edge computing;

[0238] Prior art 1 proposes an online task offloading and resource allocation scheme based on Lyapunov optimization, which can find appropriate task offloading decisions and computing resource scheduling schemes based on the trade-off between energy efficiency and service delay. Prior art 2 proposes a computing load algorithm based on deep supervised learning, which is used in MEC systems of multiple wireless devices and a single edge server to minimize network utility. However, prior art 1 and prior art 2 only emphasize minimizing the user's service response delay by formulating adaptive task offloading and resource allocation strategies for resource allocation. The impact of user mobility has not been fully explored in the research. As the user moves away from the edge server, the task transmission delay may increase significantly, seriously affecting the user's service quality.

[0239] Prior art three proposes an online service migration algorithm based on Lyapunov, which decomposes a long-time slot optimization problem into multiple single-time slot optimization problems to maximize the system utility of the MEC network while ensuring the stability of the long-term service migration cost. In the actual implementation process, the service migration problem is modeled as a Markov process, and deep recursive Q learning is used to make service migration decisions to minimize user delay and system energy consumption. However, the research on migration strategies in prior art three usually ignores the impact of different service migrations and migration processes on resource allocation strategies. Comprehensive consideration of service migration and resource allocation strategies helps to obtain better user experience. At the same time, most studies obtain migration strategies through reinforcement learning or heuristic algorithms. Although relatively good results can be achieved, it is still difficult to solve the huge action space problem caused by multi-service and multi-user environments;

[0240] Based on the above technical problems, the present application studies the service migration problem in the mobile edge computing environment. In order to improve the flexibility of service migration and solve the trade-off between service performance and system migration cost in the MEC environment, the present application analyzes the coupling between service migration and resource allocation, and considers the impact of heterogeneous edge servers, heterogeneous services and network topology on service response time, aiming to quickly and in real time formulate appropriate service migration and resource allocation strategies to reduce service response time while reducing migration costs, and better adapt to the complexity and diversity of the real world; solves the trade-off between service performance and migration cost, aiming to reduce service response time and thus improve service performance while reducing system migration costs; through the above technical solution, service migration and resource allocation are combined, while considering the topological structure of the network and the heterogeneity of each edge server, and solving the joint optimization problem through multi-agent reinforcement learning and convex optimization methods, which can provide appropriate service migration and resource allocation strategies in a more real-time and rapid manner in a real environment, so as to reduce user task response time while reducing system migration costs.

[0241] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.

Claims

1. A joint optimization method for service migration and resource allocation based on multi-agent reinforcement learning, characterized in that: include: Obtain user information U in the IoT environment and determine whether the user service context needs to be migrated; If migration is required, calculate the optimized response time and service migration cost when migration occurs; Produce joint optimization problems based on optimizing response time and service migration costs; Based on multi-agent reinforcement learning and convex optimization methods, the joint optimization problem is solved to obtain the service migration strategy and the optimal resource allocation strategy.

2. The method for joint optimization of service migration and resource allocation based on multi-agent reinforcement learning according to claim 1, characterized in that: Obtain user information U in the IoT environment and determine whether the user service context needs to be migrated, including: Get IoT user information U = {u1, u2, ..., u n }; wherein m edge servers and the base station are closely deployed at the same location, and the set of the base station and the edge servers connected thereto is M = {1, 2, ..., m}; Get the user information U, which contains the user u at time slot t i The placement status of a specific service instance on each edge server in, The constraints are: according to Determine whether the user service context needs to be migrated; Indicates that user u in time slot t i A specific service instance is placed on edge server j. Indicates that user u in time slot t i A specific service instance of is not placed on edge server j, Indicates that user u in time slot t i If the edge server where the specific service instance is located This means that user u i If a specific service instance of the service is migrated, the user service context needs to be migrated.

3. The method for joint optimization of service migration and resource allocation based on multi-agent reinforcement learning according to claim 1, characterized in that: Calculate the optimized response time and service migration costs when migration occurs, including: Get user u i When the service context is migrated, the user UI service suspension delay Specific state context synchronization delay and service restoration delay Calculate user u i Total migration delay at time slot t Get user u i The user tasks that need to be processed in each time slot generated by the service k used are transmitted to user u i When directly associated with the edge server, user u i Uplink communication delay Downlink communication delay for transmitting task results and user u i The transmission delay between the directly associated edge server and edge server j and the transmission task result return delay Calculate user u i The total communication delay at time slot t Get user u at time slot t i The computational latency of executing user tasks on the edge server where the specific service instance is located in, For edge server j in time slot t, i Allocated computing resources, Fi is the maximum available computing resource of edge server j, The computational intensity required to process the tasks generated by service k in time slot t, The size of the task generated for service k in time slot t; According to user i Total migration delay in time slot t User i The total communication delay at time slot t and the computational delay of edge server j executing the user task at time slot t Calculate the optimal response time for executing user tasks in time slot t Get the service migration cost of the user in time slot t Among them, L-1 is the service k from the edge server Migrate to edge servers The number of hops required, w E is the cost coefficient.

4. The method for joint optimization of service migration and resource allocation based on multi-agent reinforcement learning according to claim 3 is characterized in that: Get user u i When the service context of user u is migrated i Service suspension delay Specific state context synchronization delay and service restoration delay include: Get the user u when performing migration i Service suspension delay: in, For user u i The service suspension delay, is the computational intensity required to suspend service k, is the size of the user-specific state context, is user u at time slot t-1 i The edge server where the specific service instance is located is user u i Allocated computing resources, The user u before the migration i The edge server where the specific service instance is located The placement status, The user u after the migration occurs i The edge server where the specific service instance is located The placement status; Get the user u when performing migration i The specific state context synchronization delay is: in, For user u i The specific state context synchronization delay of L = {c, ..., j} represents the number of edge server hops required for the shortest path from edge server c to edge server j, B 1,1+1 is the bandwidth between edge servers 1 and 1+1; Get the user u when performing migration i Service restoration delay: in, For user u i Service restoration delay, represents the computational intensity required to restore service k, is the user after migration ψ i The edge server where the specific service instance is located For user u i Allocated computing resources.

5. The method for joint optimization of service migration and resource allocation based on multi-agent reinforcement learning according to claim 3, characterized in that: Get user u i When the user tasks that need to be processed in each time slot generated by the service k are transmitted to the edge server directly associated with user ui, user u i Uplink communication delay Downlink communication delay for transmitting task results and user u i The transmission delay between the directly associated edge server and edge server j and the transmission task result return delay include: Get user tasks and transfer them to user u i Uplink communication latency when directly connected to the edge server in, For user tasks, transfer to user u i Uplink communication latency when directly connected to the edge server For user u i Wireless transmission rate to edge server, For user u i The real-time location of the user u, W is the user u i and the wireless channel bandwidth between the edge servers directly associated with it, For user u i The transmission power, N0 is the Gaussian noise power spectrum density, g t For edge server j and user u i The wireless channel path loss between them, β0 represents the channel power at d0 = 1m, Represents the relationship between edge server j and user u i The distance between Get user tasks and transfer them to user u i Downlink communication delay when directly associated with edge servers in, The size of the result returned by the task serving k in time slot t; When the user task is transmitted to edge server j, user u i The transmission delay between the directly associated edge server and edge server j and the transmission task result return delay Among them, B l,l+1 is the bandwidth between edge servers l and l+1.

6. The method for joint optimization of service migration and resource allocation based on multi-agent reinforcement learning according to claim 1, characterized in that: A joint optimization problem is created based on optimizing response time and service migration cost, including: Based on the cost-performance trade-off, according to the optimized response time and service migration costs Get the performance and cost trade-off formula: Where T* is the average service response delay constraint, μ is the basic reward for completing the service within the average response delay constraint in time slot t, and w Y is the weight of the response delay, Migration costs incurred due to service migration; According to the performance and cost trade-off formula, the joint optimization problem is determined as follows: in, For edge server j in time slot t, i Allocated computing resources, F j is the maximum available computing resource of edge server j, is user u at time slot t i The placement status of a specific service instance on edge server j.

7. The method for joint optimization of service migration and resource allocation based on multi-agent reinforcement learning according to claim 1, characterized in that: Based on multi-agent reinforcement learning and convex optimization methods, the joint optimization problem is solved to obtain the service migration strategy and the optimal resource allocation strategy, including: Based on the deployment data of edge servers in the IoT system in the network topology and the user data of IoT users, the total state of the environment in multi-agent reinforcement learning, as well as the state, action and reward of each agent, are constructed; Overall state of the environment t is the system state at any time slot t; Among them, the system state s in any time slot t t By user u in IoT i The array is represented by the real-time state of service k in time slot t. The computational intensity required to process the tasks generated by service k The size of the user-related state context of service k User real-time location and user u i The edge server where the specific service instance is located The state of each agent is the state that each agent can observe in time slot t According to the currently observed state, the action generated by each agent in time slot t is Among them, j represents user u i The specific edge server to which a specific service instance is migrated; The reward for each agent is The total reward is According to the overall state of the environment, as well as the state, action, and reward of each agent, a service migration strategy model is trained based on a multi-agent reinforcement learning method. Determine user u according to the service migration strategy i service migration strategy for a specific service instance; After determining the service migration strategy, the resource allocation strategy is determined based on the convex optimization method, including: It is known that given the service migration strategy in time slot t, Only with Since resource allocation is performed in a distributed manner at each edge server j within time slot t, the transformation objective function is: Taking the first and second derivatives of the objective function, we get: in, According to the first-order derivative results and the second-order derivative results of the objective function, it is determined that the objective function is a convex function; Based on the objective function being a convex function, the resource allocation scheme under the migration strategy is determined by solving the Lagrangian function: Calculate the optimal resource allocation through KKT conditions The optimal solution is: in, is the optimal resource allocation strategy.