A DAG micro-service migration method and device based on deep reinforcement learning

Abstract

The application discloses a DAG micro-service migration method and device based on deep reinforcement learning, and the method comprises the following steps: training a micro-service migration model by using a reinforcement learning micro-service migration algorithm based on action execution layering to obtain a trained micro-service migration model; deploying the trained micro-service migration model to a user end, and when the position of the user end moves, the trained micro-service migration model makes a migration decision of the micro-service according to obtained local information to obtain a target edge server; and migrating the micro-service based on the target edge server. Through the design of the hierarchical decision, the DAG micro-service migration problem in the edge environment where the global information is unknown, especially in the Internet of Vehicles scene, can be effectively solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mobile edge computing and artificial intelligence, and particularly relates to a DAG microservice migration method and device based on deep reinforcement learning. BACKGROUND

[0002] In actual scenarios, due to various reasons, the global information of the entire edge computing system cluster and the edge servers inside the cluster may also be unable to be obtained in time. For example, due to privacy, network instability or distance problems, the edge servers may not transmit their own resource state information to the scheduling system of the cluster in real time. Especially in the scenario of Internet of Vehicles, the high-speed movement of vehicles will cause the related microservices to need to be frequently migrated in a short time, and it is very difficult to obtain the global information in a short time and make reasonable migration decisions quickly. In this case, the microservice migration decision must rely on the local information fed back in real time and adapt to the dynamically changing network environment and resource status. Although in recent years, some service migration schemes have been proposed, there are still the following key problems to be solved:

[0003] 1) Lack of service migration scheme for DAG tasks: Current microservice migration schemes mostly focus on the migration of independent services and cannot effectively handle the complex dependency relationship in DAG tasks. In real scenarios, the tasks requested by users usually contain multiple subtasks with dependency relationship, following a clear DAG structure. However, most of the existing designs cannot cope with the service migration problem of DAG tasks. Although Ray et al. proposed the first microservice migration scheme for DAG tasks, this method assumes that all microservices corresponding to the DAG task will be migrated to the same edge server, which has great limitations in actual application and may cause resource overload of the target edge server, thereby increasing service delay. Allowing microservices to migrate to multiple edge servers to disperse the workload is more in line with actual needs.

[0004] 2) Unavailability of system global information: Most of the current service migration algorithms are designed to rely on prior knowledge of system global information to make migration decisions, but in actual application, due to unpredictable user movement patterns and dynamic edge server resources, the global information of the entire edge computing system is often not available in real time. This makes many theoretically effective service migration schemes ineffective in actual application.

[0005] 3) Service migration algorithms based on traditional mathematical theories are generally applicable to small-scale edge computing scenarios. If applied in large-scale dynamically changing edge computing scenarios, complex calculations are required to obtain reasonable migration decisions, and the decision time overhead is large.

[0006] Therefore, the prior art remains to be improved and enhanced. SUMMARY

[0007] The technical problem to be solved by the present application is to provide a DAG microservice migration method and device based on deep reinforcement learning in view of the above defects of the prior art.

[0008] In a first aspect, the present application provides a DAG microservice migration method based on deep reinforcement learning, wherein the method comprises:

[0009] The trained microservice migration model is deployed to the user end, and when the user end position moves, the trained microservice migration model makes a migration decision for the microservice according to the obtained local information to obtain a target edge server.

[0010] The trained microservice migration model is deployed to the user end, and when the user end position moves, the trained microservice migration model makes a migration decision for the microservice according to the obtained local information to obtain a target edge server.

[0011] The migration of the microservice is performed based on the target edge server.

[0012] In an implementation mode, the reinforcement learning microservice migration algorithm based on action execution layering is used to divide the microservice migration problem into two subtasks and interact with the environment to obtain rewards and the next state.

[0013] The subtasks include a high-level task and a low-level task, the high-level task is used to decide whether the DAG task needs to be migrated and select the location of the target edge cloud, and the low-level task is used to decide the target edge server in the target edge cloud to which the microservice is migrated.

[0014] In an implementation mode, when the user end position moves, the trained microservice migration model makes a migration decision for the microservice according to the obtained local information to obtain a target edge server, which includes:

[0015] The trained microservice migration model divides the execution actions into high-level actions and low-level actions to make a migration decision according to the local information obtained by the user end to obtain a target edge service, wherein the high-level actions make a migration decision to determine the target edge cloud to which the DAG microservice should be migrated, and the low-level actions make a migration decision to determine the target edge server to which the DAG microservice should be migrated.

[0016] In an implementation mode, when determining the target edge cloud to which the DAG microservice should be migrated, the trained microservice migration model determines the target edge cloud to which the DAG microservice should be migrated according to the overall computing demand of the task, the dependency relationship, the computing capacity and the load of the edge cloud.

[0017] In an implementation manner, when determining the target edge server to which the DAG microservice should be migrated, the trained microservice migration model determines the target edge server of each microservice according to the specific computing requirements of the microservice, the dependency relationship between the microservices, and the current load and computing capacity of each edge server.

[0018] In an implementation manner, the problem form of the high-level action is defined as follows:

[0019]

[0020] wherein, EC represents a set of edge clouds, ec represents an edge cloud in the set of edge clouds, T represents a set of time slots, t represents a time slot in the set of time slots, v u is a set of microservices, represents a microservice in the set of microservices, is the total service delay of processing the microservice on the time slot t, is a decision vector of the edge cloud in the time slot t, U is a set of users, and u is a user in the set of users.

[0021] In an implementation manner, the problem form of the low-level action is defined as follows:

[0022]

[0023] wherein, M represents the index of all edge servers in the edge cloud, m represents an edge server in all edge servers, N represents the index of all microservices in the DAG service, n represents a server in all microservices, is a decision vector of the low-level action.

[0024] In a second aspect, the embodiments of the present application also provide a DAG microservice migration device based on deep reinforcement learning, wherein the device comprises:

[0025] a model training module configured to train a microservice migration model by using a reinforcement learning microservice migration algorithm based on action execution layering, and obtain a trained microservice migration model;

[0026] a migration decision module configured to deploy the trained microservice migration model to a user end, and when the location of the user end moves, the trained microservice migration model makes a migration decision of the microservice according to obtained local information, and obtains a target edge server;

[0027] a migration execution module configured to execute the migration of the microservice based on the target edge server.

[0028] In a third aspect, the embodiments of the present application further provide a terminal, wherein the terminal comprises a memory, a processor, and a deep reinforcement learning-based DAG microservice migration program stored in the memory and executable on the processor, and when the processor executes the deep reinforcement learning-based DAG microservice migration program, the steps of the deep reinforcement learning-based DAG microservice migration method according to any one of the above solutions are implemented.

[0029] In a fourth aspect, the embodiments of the present application further provide a computer-readable storage medium, wherein the computer-readable storage medium stores a deep reinforcement learning-based DAG microservice migration program, and when the deep reinforcement learning-based DAG microservice migration program is executed by a processor, the steps of the deep reinforcement learning-based DAG microservice migration method according to any one of the above solutions are implemented.

[0030] Beneficial effects: Compared with the prior art, the present application provides a deep reinforcement learning-based DAG microservice migration method. The present application firstly adopts an action execution layer-based reinforcement learning microservice migration algorithm to train a microservice migration model, and obtains a trained microservice migration model. Then, the trained microservice migration model is deployed to a user end, and when the user end position moves, the trained microservice migration model makes a microservice migration decision according to obtained local information, and obtains a target edge server. Finally, the microservice migration is performed based on the target edge server. Through the design of layered decision-making, the present application can effectively solve the edge environment with unknown global information, especially the DAG microservice migration problem in the Internet of Vehicles. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 The flowchart of a preferred embodiment of the deep reinforcement learning-based DAG microservice migration method provided by the embodiments of the present application.

[0032] Figure 2 The architecture schematic diagram of the DAG service migration framework provided by the embodiments of the present application.

[0033] Figure 3 The system structure schematic diagram of the deep reinforcement learning-based DAG microservice migration system provided by the embodiments of the present application.

[0034] Figure 4 The network structure schematic diagram of the deep reinforcement learning algorithm of the present embodiment.

[0035] Figure 5 The training flowchart of the action execution layer-based reinforcement learning microservice migration algorithm.

[0036] Figure 6 The architecture schematic diagram of the deep reinforcement learning-based DAG microservice migration apparatus provided by the embodiments of the present application.

[0037] Figure 7 The principle block diagram of the terminal provided in the embodiment of the present application is shown. DETAILED DESCRIPTION

[0038] In order to make the objectives, technical solutions and effects of the present application clearer and more explicit, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0039] The flowchart shown in the accompanying drawings is only an example and does not necessarily include all the contents and operations or steps, nor does it necessarily be executed in the order described. For example, some operations or steps can be decomposed, combined or partially merged, so that the actual execution order can be changed according to the actual situation.

[0040] It should be understood that the terms used in the present application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the present application specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0041] It should be understood that, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the terms "first", "second" and the like are used to distinguish the same or similar items with basically the same function and effect. For example, the first control information and the second control information are only used to distinguish different control information and do not limit the order.

[0042] Those skilled in the art can understand that the terms "first", "second" and the like do not limit the quantity and execution order, and the terms "first", "second" and the like do not necessarily mean different.

[0043] It should also be understood that the term "and / or" used in the present application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0044] The DAG microservice migration method based on deep reinforcement learning of the present embodiment can be applied to a terminal, which can be a computer, a mobile phone or other intelligent product terminal. As shown in Figure 1 The DAG microservice migration method based on deep reinforcement learning of the present embodiment includes the following steps:

[0045] Step S100, training the microservice migration model by using the reinforcement learning microservice migration algorithm based on action execution layering to obtain the trained microservice migration model;

[0046] Step S200, deploy the trained micro-service migration model to the user end, when the user end position moves, the trained micro-service migration model makes a micro-service migration decision according to the obtained local information, and obtains a target edge server;

[0047] Step S300, perform micro-service migration based on the target edge server.

[0048] The present application explores the scene where global information is difficult to obtain in a mobile edge computing system, which is closer to reality, especially in the Internet of Vehicles scenario. Specifically, the present embodiment models the micro-service migration problem as a partially observable Markov decision process, and decomposes the migration target into high and low layer actions. Among them, the high layer action is used to determine whether the DAG service needs to be migrated and which edge cloud to migrate to, and the low layer action further determines the specific edge server to be migrated of the DAG micro-service based on the high layer action. Through the decoupling of the target, the action space of reinforcement learning can be effectively reduced, the training efficiency of the subsequent model can be improved, the waste of training time and cost caused by redundant action exploration can be reduced, and the rapid iteration and update of the subsequent model and the lightweight deployment of the model on the user end side are also facilitated. In order to overcome the problem that in the belief state updating process of POMDP (partially observable Markov decision process), as the state space increases, its dimension and complexity also increase significantly, resulting in impracticality of directly calculating or storing the belief state, the present embodiment introduces LSTM to capture long-term dependency information, and obtains an approximate representation of the belief state, so that the solution of the POMDP problem becomes efficient. Through the design of this hierarchical decision, the DAG micro-service migration problem in the edge environment where the global information is unknown, especially in the Internet of Vehicles, can be effectively solved.

[0049] Specifically, as shown in Figure 2 and Figure 3 , the architecture of the DAG micro-service migration framework is shown in Figure 2 , and the system structure diagram of the present application is shown in Figure 3 The DAG micro-service migration system of the present embodiment includes three entities: a user end 10, an edge cloud 20, and a remote cloud 30. The edge cloud 20 is located at the network edge and has limited resources. The remote cloud 30 has abundant resources, but its location is relatively far away from the user compared with the edge cloud 20. The user can move freely and decide to send tasks to the nearby edge cloud 20 for processing. The tasks offloaded by the user are composed of multiple dependent sub-tasks, which can be represented as a directed acyclic graph (DAG). In this graph, each node represents a sub-task, and each edge represents the dependency relationship between sub-tasks.

[0050] In order to reduce the size of the action space of the subsequent reinforcement learning, the present embodiment first divides the edge servers into a group of edge clouds EC={ec1, ec2, …, ecn} according to the location, and then divides the edge servers in each edge cloud into a group of edge servers ECi={ec1, ec2, …, ecn}.n}, wherein each edge cloud ec i EC contains a set of edge servers. Each edge cloud can be represented as ec1, ec2, …, ec n . Wherein, each ec i represents an independent edge cloud, i is an index from 1 to n, indicating different edge clouds. In each edge cloud, it can contain multiple edge servers. For the servers belonging to the edge cloud ec i , this embodiment uses s i1 , s i2 , …, s im to represent them, wherein m is the total number of servers in the edge cloud ec i . In this representation, s ij represents the jth server in the ith edge cloud. For example, s 23 represents the third server in the second edge cloud ec2.

[0051] Assuming that edge servers can be connected to each other through a wired network, mobile users can receive information from the edge server base station near them. Assuming that the continuous service migration decision is based on time slot t∈T={0, 1, 2, …, t}, and represents the migration decision made in the tth time slot, that is, the edge server to which the microservice should be migrated in the tth time slot. In the short term, the user always stays within the service coverage of the original edge server, and the edge network environment remains unchanged. Let be a set of microservices, wherein each microservice corresponds to a computing demand

[0052] The goal of the present invention is to make reasonable microservice migration decisions with the information available to the user end in the case of global information unawareness of the edge computing system, to reduce the total delay of the DAG microservice, and to provide better service experience for users.

[0053] Further, this embodiment defines migration delay, computing delay and communication delay. Migration delay mainly depends on the amount of migrated data, bandwidth and network conditions. This embodiment uses to represent the number of hops between the current server and the previous server. Assuming that user u needs to migrate a microservice container with a data volume of at time t, the bandwidth of the migration path is , then the migration delay can be defined as:

[0054]

[0055] wherein, represents the network condition coefficient, reflecting the influence of network conditions (such as delay and congestion) on migration time. is the distance impact factor, which indicates the impact of physical distance on migration delay. Indicates the migration bandwidth. When , it means that the user is already near the target server, so the migration delay is 0. When , the migration delay is determined by the transmission time, network condition coefficient and distance factor calculated based on bandwidth and data volume.

[0056] Computational latency is mainly affected by the computing power and workload of the edge cloud. Its computing latency depends not only on the computing power of the edge server and workloads It also depends on the time required to transfer the result data of all preceding microservices. set up for The computational delay can be defined as:

[0057]

[0058] in The calculation is as follows. For each preceding microservice Calculate the processing completion time T finish (v j ), and the time required to transmit the result data T trans (v j ). Determine the data transmission completion time of each preceding microservice: T complete (v j )=T finish (v j )+

[0059] T trans (v j ). Calculate all Middle T complete (v j ) minus the maximum value of T tinish (v j ) minimum value:

[0060]

[0061] In particular, if microservices Without the preceding microservice,

[0062] Communication delay and data transmission rate r t Related to the transmission distance, r t It can be calculated by Shannon's formula. The data transmission delay from the local edge server to the user can be defined as:

[0063]

[0064] If the sub-tasks and their related services are not in the local edge server, the data required by the user needs to be transmitted from other edge servers, and the bandwidth of the backhaul link is Let denote the distance between the edge server processing the microservice and the local edge server. Therefore, the backhaul link delay formula is:

[0065]

[0066] In this formula, is a coefficient that quantifies the impact of backhaul network conditions (such as delay and congestion) on transmission time. denotes the distance impact factor, which adjusts the delay according to the impact of physical distance. Considering the comprehensive impact of data transmission time, network conditions and physical distance on backhaul link delay, the calculation formula of communication delay can be obtained as:

[0067]

[0068] By and , the total service delay of processing microservice in time slot t is:

[0069]

[0070] The goal of this embodiment is to reduce the total service delay through reasonable service migration decisions and provide better service experience for users. Specifically, the goal is to find the optimal migration strategy for user services in all time slots to minimize the total service delay.

[0071] Based on this, this embodiment formalizes this problem as an optimization problem. Let the decision vector be where N represents the index of all microservices in the DAG service, and S represents the index of all edge servers in the edge computing system. The vector describes the mapping of microservices to edge servers in time slot t. For each element in the vector , it is defined as: if , it means that microservice n is migrated to edge server s in time slot t. If , it means that microservice n is not migrated to edge server s. The formal definition of the problem is as follows:

[0072]

[0073] The above formalized problem is an NP-hard problem. This embodiment divides the edge servers in the entire edge network into several edge clouds and processes the problem in two levels. The main benefit of this is to reduce the complexity of the problem and improve processing efficiency. By dividing into smaller edge clouds, local optimization problems can be more effectively managed and solved, while the overall solution is more suitable for actual application requirements. In addition, this method enhances the scalability and adaptability of the system, allowing for more flexible adjustment of strategies in the face of different environments and requirements.

[0074] Further, this embodiment designs a deep reinforcement learning service migration framework based on action execution hierarchy, named DAG service migration latency optimization framework (Optimization of DAG Service Migration Latency, OSML). The framework divides the action execution into two layers of actions (i.e., high-level actions and low-level actions) to make migration decisions. The OSML framework deploys the trained microservice migration model on the user side, so the trained microservice migration model only needs to make reasonable service migration decisions based on the information available to the user side.

[0075] The high-level action makes migration decisions to determine the target edge cloud to which the DAG microservice should be migrated. Specifically, in the first layer of the OSML framework, the focus is on determining which edge cloud the entire DAG task's services should be pre-migrated to. In this layer, the microservice migration model is responsible for determining the pre-migration location of the entire service based on the overall computing requirements of the task, dependency relationships, computing capabilities of edge clouds, and load. The goal of this layer is to optimize the service latency of the entire DAG service and ensure that the service is migrated to the most suitable edge cloud, thereby achieving overall optimization of service latency.

[0076] The low-level action makes migration decisions to determine the target edge server to which the DAG microservice should be migrated. Specifically, in the second layer of the framework, the focus is on determining which specific edge server in the edge cloud each microservice corresponding to a DAG subtask should be migrated to. In this layer, the microservice migration model is responsible for determining the optimal migration location of each microservice based on the specific computing requirements of the microservice, the dependency relationships between microservices, and the current load and computing capabilities of each edge server. The goal of this layer is to ensure that each microservice is migrated to the most suitable server, minimizing the latency of all microservices. With the OSML microservice migration framework, the microservice migration model of the present invention can effectively solve the complex DAG service migration problem in the vehicle networking environment without global system information.

[0077] Specifically, the embodiment considers only the pre-migration problem of the entire DAG service in the high-level action migration decision stage, specifically regards the DAG service as an indivisible whole, determines whether the entire service needs to be migrated and to which edge cloud it should be migrated. However, it should be noted that this pre-migration stage only helps to determine to which edge cloud the DAG service should be migrated, and the actual service migration operation is not performed in this stage.

[0078] The problem form of the high-level action migration decision stage is defined as follows:

[0079]

[0080] wherein, EC represents a set of edge clouds, ec represents an edge cloud in the set of edge clouds, T represents a set of time slots, t represents a time slot in the set of time slots, v u is a set of microservices, represents a microservice in the set of microservices, is the total service delay of processing the microservice in time slot t, is the decision vector of the edge cloud in time slot t, U is a set of users, and u is a user in the set of users.

[0081] After completing the high-level action migration decision stage, it is known whether the entire DAG service needs to be migrated and to which edge cloud it should be migrated. In the second stage, the specific migration scheme of the microservices in the DAG needs to be determined to ensure that each microservice can be migrated to the optimal edge server in the target edge cloud. Let the decision vector be wherein, N represents the index of all microservices in the DAG service, and M represents the index of all edge servers in the edge cloud. The vector describes the mapping of the microservice to the edge server in time slot t. For each element in the vector The present application has: If , it means that the microservice n is assigned to the edge server m in time slot t. If , it means that the microservice n is not assigned to the edge server m.

[0082] The problem form of the low-level action migration decision stage is defined as follows:

[0083]

[0084] In the actual application of the algorithm design, in combination with Figure 4 and Figure 5 , Figure 4 is a network structure diagram of the deep reinforcement learning algorithm of the embodiment, Figure 5A training flowchart of a hierarchical reinforcement learning microservice migration algorithm based on action execution. In order to solve the DAG service migration problem under the OMSL framework, the embodiment converts the original problem into a partially observable Markov decision process (S, O, A, R) composed of state space, observation space, action space and reward function. The specific definitions of the four elements are as follows:

[0085] 1) State space: The state space S of the system is composed of the global state s of the mobile edge computing system at time t. t

[0086] 2) Observation space: In the deep reinforcement learning problem defined in this embodiment, the observation space includes information essential for service migration decision. Specifically, the observation space is composed of the following parts: first, the location information P t (u) of the user's local edge server, which is used to describe the accurate location of the user's local edge server at time slot t. Next is the wireless transmission rate r t , which represents the transmission rate between the user and the wireless access network. In addition, the observation space also includes the overall size D t of the DAG task and the size of each subtask represented by a vector as These are usually represented in the form of a vector. The dependency relationship between DAG tasks is represented by the topological sorting sequence Seq t , which shows the order of executing each subtask. For example, if there are N subtasks, represents the order of task execution. Finally, the observation space also includes the number of CPU cycles required to execute each subtask which records the number of CPU cycles required to execute the nth subtask, represented in the form of a vector, Combining all this information, the observation space is represented as O t = [P t (u), r t , D t , Seq t , C t ], providing the key information required for the trained microservice migration model to make the best service migration decision at each time step.

[0087] ​3) Action space: The action space A defines all the actions that the trained microservice migration model can perform. In the service migration problem, the action refers to deciding which edge server to migrate the microservice to in each time slot. Since the embodiment divides the service migration problem into high-level goals and low-level goals for processing. The high-level goal represents whether the DAG needs to be migrated and which edge cloud to migrate to, and the low-level goal represents which edge server the microservice in the DAG service needs to be migrated to specifically. Therefore, the high-level action can be defined as action representing the decision of the DAG service to pre-migrate to a specific edge cloud. For example, if there are EC edge clouds, may be an EC-dimensional binary vector, where one element is set to 1 and the rest are set to 0, indicating the selected edge cloud. The low-level action can be defined as action representing the decision to migrate the microservice to a specific edge server. Similar to the high-level, this can also be a multi-dimensional binary vector. where N represents the index of all microservices in the DAG service, and M represents the index of all edge servers in the edge cloud. The vector describes the mapping of microservices to edge servers in time slot t. For each element in the vector The present invention has: If , it means that the microservice n is assigned to the edge server m in time slot t. If , it means that the microservice n is not assigned to the edge server m.

[0088] In summary, the joint action space A = (A H , A L ) of the high-level action and the low-level action can be obtained, and the joint action can be represented as

[0089] 4) Reward function: The reward function R is used to evaluate the state of the environment after performing a specific action. In the service migration problem, the goal of the problem is to minimize the total service delay, so the reward function can be designed as the negative total service delay r t = -L ′ (u, t, v u ).

[0090] Since there are four major pain points in the solution of POMDP, such as high-dimensional state space, partial observability, long-time dependence problem, and limited computing resources. In order to overcome these challenges, the embodiment applies the LSTM (Long Short-Term Memory) model to the solution of POMDP. LSTM can handle long-time sequence data and effectively estimate hidden states, thereby greatly reducing the complexity and uncertainty of solving POMDP.

[0091] In a partially observable Markov decision process, the state s t of the environment is not directly observable by the agent, and the agent can only infer the state of the environment through observations o t . Let the history of observations be O t = {o1, o2, …, o t}. The trained microservice transfer model processes the history of observations through an LSTM network to obtain the hidden state h t , i.e., h t = LSTM(o1, o2, …, o t ). Here, h t is the hidden state of the agent at time step t, which contains all the information about the history of observations. Then, h t is used to replace s t for state input training in the PPO reinforcement learning algorithm, and a trained microservice transfer model is obtained.

[0092] Parameter update: The reinforcement learning strategy optimization in this embodiment is implemented through the PPO algorithm, which improves the performance of the strategy by optimizing a specific objective function while constraining the magnitude of the strategy update. The PPO algorithm adjusts the strategy by calculating the probability ratio and optimizing the clipped objective function L(θ). The advantage function estimate A t is calculated using the TD residual and the GAE parameter.

[0093] The PPO algorithm first calculates a value called the probability ratio, which measures the ratio of the probability of taking a specific action under the current policy parameters to the probability of taking the same action under the old policy parameters:

[0094]

[0095] where π θ (a t | h t ) is the probability of selecting action a t given state h t under policy parameters θ. The PPO algorithm uses a clipped objective function to avoid the strategy update step being too large, thereby maintaining the stability of the training process. The objective function is defined as follows:

[0096]

[0097] where A is the advantage function estimate, and ∈ is the clipping parameter. The advantage function estimate is used to measure the superiority of selecting a specific action under a given state relative to the average policy. It is calculated as follows:

[0098]

[0099] where, δ t = r t + γV(h t+1 ; a) - V(h t ; φ) is the TD residual, γ is the discount factor, and λ is the GAE (Generalized Advantage Estimation) parameter.

[0100] Therefore, the embodiment uses reinforcement learning to solve the DAG microservice migration problem. The reinforcement learning microservice migration algorithm based on action execution layering designed based on the above process is used to divide the microservice migration problem into two subtasks; wherein the subtasks include a high-level task and a low-level task, the high-level task is used to decide whether the DAG task needs to be migrated and the location selection of the target edge cloud, and the low-level task is used to decide the target edge server to which the microservice is migrated in the target edge cloud. Through multiple rounds of training of the microservice migration model, a joint action (high-level action and low-level action) is generated based on the current state by the policy network at each time step, and the environment is interacted to obtain the reward and the next state. During the training process, the experience data is stored and used for iterative training of the subsequent model. The algorithm optimizes the policy network and the value function network using the PPO algorithm to ensure the stability and efficiency of the policy update, and finally generates the optimal policy π * , that is, the target edge server is obtained, which can guide the migration of the microservice in a dynamic network environment. Through the action layering manner, the trained microservice migration model divides the execution action into high-level action and low-level action according to the local information obtained by the user end to make migration decisions, which can reduce the calculation redundancy, also reduces the size of the action space, and improves the efficiency of model training and inference.

[0101] In summary, the present application explores the scene in which global information is difficult to obtain in a mobile edge computing system (especially in a vehicle networking scene), models the microservice migration problem as a partially observable Markov decision process, and decomposes the migration target into high-level action and low-level action. Among them, the high-level action is used to determine whether the DAG service needs to be migrated and which edge cloud to migrate to, and the low-level action further determines the specific edge server to which the DAG microservice is to be migrated based on the high-level action. Through the decoupling of the target, the action space of the reinforcement learning can be effectively reduced, the training efficiency of the subsequent model can be improved, the waste of training time and cost caused by redundant action exploration can be reduced, and the rapid iteration and update of the subsequent model and the lightweight deployment of the model on the user end can be facilitated. Moreover, in order to overcome the problem that in the belief state updating process of POMDP, as the state space increases, its dimension and complexity also increase significantly, making it impractical to directly calculate or store the belief state, the present application introduces LSTM to capture long-term dependency information to obtain an approximate representation of the belief state, so that the solution of the POMDP problem becomes efficient.

[0102] Based on the above embodiments, the present invention also provides a DAG microservice migration device based on deep reinforcement learning, such as Figure 6 As shown, the device includes: a model training module 100, a migration decision module 200 and a migration execution module 300. Specifically, the model training module 100 is used to train the microservice migration model using a reinforcement learning microservice migration algorithm based on action execution layering to obtain a trained microservice migration model. The migration decision module 200 is used to deploy the trained microservice migration model to the user end. When the user end moves, the trained microservice migration model makes a migration decision for the microservice based on the local information obtained to obtain a target edge server. The migration execution module 300 is used to execute the migration of the microservice based on the target edge server.

[0103] The working principles of each module in the deep reinforcement learning-based DAG microservice migration device of this embodiment are the same as the principles of each step in the above method embodiment, and will not be repeated here.

[0104] Each module in the aforementioned deep reinforcement learning-based DAG microservice migration device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a terminal in hardware form, or stored in a memory in the terminal in software form, allowing the processor to call and execute the corresponding operations of each module.

[0105] Based on the above embodiment, the present invention further provides a terminal, the principle block diagram of the terminal can be as follows: Figure 7 The terminal may include one or more processors 100 ( Figure 7 Only one is shown), a memory 101 and a computer program 102 stored in the memory 101 and executable on one or more processors 100.

[0106] In one embodiment, the processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0107] In one embodiment, the storage 101 can be an internal storage unit of the electronic device, such as a hard disk or a memory of the electronic device. The storage 101 can also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, or the like equipped on the electronic device. Further, the storage 101 can include both an internal storage unit and an external storage device of the electronic device. The storage 101 is used to store a computer program and other programs and data required by the terminal. The storage 101 can also be used to temporarily store data that has been output or will be output.

[0108] Those skilled in the art can understand that, Figure 7 The principle block diagram shown in the above embodiments is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the terminal to which the scheme of the present application is applied. The specific terminal can include more or less components than those shown in the diagram, or combine certain components, or have a different arrangement of components.

[0109] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. Any reference to memory, storage, operating database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0110] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A DAG microservice migration method based on deep reinforcement learning, characterized in that, The method comprises: training a microservice migration model by using a reinforcement learning microservice migration algorithm based on action execution layering to obtain a trained microservice migration model; deploying the trained microservice migration model to a user end, and when the location of the user end moves, the trained microservice migration model makes a migration decision for the microservice according to obtained local information to obtain a target edge server; performing migration of the microservice based on the target edge server; the reinforcement learning microservice migration algorithm based on action execution layering is used to divide the microservice migration problem into two subtasks and interact with the environment to obtain a reward and a next state; wherein the subtasks include a high-level task and a low-level task, the high-level task is used to decide whether the DAG task needs to be migrated and select a target edge cloud, and the low-level task is used to decide a target edge server in the target edge cloud to which the microservice is migrated; when the location of the user end moves, the trained microservice migration model makes a migration decision for the microservice according to obtained local information to obtain a target edge server, comprising: the trained microservice migration model divides execution actions into high-level actions and low-level actions to make a migration decision according to local information obtained by the user end to obtain a target edge service, wherein the high-level actions make a migration decision to determine a target edge cloud to which the DAG microservice should be migrated, and the low-level actions make a migration decision to determine a target edge server to which the DAG microservice should be migrated.

2. The deep reinforcement learning based DAG microservice migration method according to claim 1, wherein, when determining the target edge cloud to which the DAG microservice should be migrated, the trained microservice migration model determines the target edge cloud to which the DAG microservice should be migrated according to overall computing requirements of the task, dependency relationships, computing capabilities and loads of edge clouds. 3.The deep reinforcement learning based DAG microservice migration method of claim 1, wherein, when determining the target edge server to which the DAG microservice should be migrated, the trained microservice migration model determines the target edge server to which each microservice should be migrated according to specific computing requirements of the microservice, dependency relationships between the microservices, and current loads and computing capabilities of each edge server. 4.The DAG microservice migration method based on deep reinforcement learning according to claim 1, wherein, The problem form of the high-level actions is defined as follows: constrained to wherein, , denotes a set of edge clouds, ec denotes an edge cloud in the set of edge clouds, denotes a set of time slots, t denotes a time slot in the set of time slots, is a set of microservices, denotes a microservice in the set of microservices, is the total service delay of processing the microservice on the time slot t, is the decision vector of the edge cloud in the time slot t, U is a set of users, is a user in the set of users.

5. The deep reinforcement learning based DAG microservice migration method according to claim 4, characterized in that, The problem form of the low-level actions is defined as follows: constrained to wherein, an index representing all edge servers in the edge cloud, m representing one edge server in all edge servers, an index representing all micro services in the DAG service, n representing one server in all micro servers, a decision vector for low-level actions. 6.A DAG microservice migration apparatus based on deep reinforcement learning, characterized in that, The device is used to implement steps of the DAG microservice migration method based on deep reinforcement learning according to any one of claims 1-5, and the device comprises: a model training module configured to train a microservice migration model by using a reinforcement learning microservice migration algorithm based on action execution layering to obtain a trained microservice migration model; a migration decision module configured to deploy the trained microservice migration model to a user end, and when the location of the user end moves, the trained microservice migration model makes a migration decision for the microservice according to obtained local information to obtain a target edge server; a migration execution module configured to perform migration of the microservice based on the target edge server.

7. A terminal, characterized by comprising: The terminal comprises a memory, a processor, and a deep-reinforcement-learning-based DAG microservice migration program stored in the memory and executable on the processor, and when the deep-reinforcement-learning-based DAG microservice migration program is executed by the processor, the steps of the deep-reinforcement-learning-based DAG microservice migration method according to any one of claims 1-5 are implemented.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a deep-reinforcement-learning-based DAG microservice migration program, and when the deep-reinforcement-learning-based DAG microservice migration program is executed by the processor, the steps of the deep-reinforcement-learning-based DAG microservice migration method according to any one of claims 1-5 are implemented.

Images