Microservice deployment method based on intimacy perception and load balancing

Abstract

The present invention proposes a microservice deployment method based on intimacy perception and load balancing, which includes the following steps: S1, defining the microservice deployment problem as a Markov decision process; S2, inputting the directed weighted graph of the microservice into the AttnSAGE model to obtain the state s t , S3, change state s t Data is input into the Actor network to obtain the target sample {s t ,a t ,R t, s t+1}; S4, randomly sample several samples in the experience replay pool and put the target sample {s t ,a t ,R t, s t+1} into the experience replay pool; S5, the target sample and the several samples are input into the two critic networks to calculate the target Q value, and the target Q value is calculated by estimating the state s t and action a t The target Q value under [1] is used to guide the strategy update. The method of the present invention models the microservice dependency relationship as a directed weighted graph, and uses community partitioning and calculation of the eigenvector centrality of each microservice to generate a ranked list that comprehensively considers the dependency relationship between services.

Description

Technical Field

[0001] The present invention relates to the field of microservice technology, and in particular to a microservice deployment method based on intimacy perception and load balancing. Background Art

[0002] Deploying microservices in edge cloud environments faces three major challenges: First, complex dependencies exist between microservices. If closely dependent services are deployed across geographically dispersed edge nodes, synchronous calls and data sharing can significantly increase response latency. Second, unlike centralized cloud data centers, edge clouds are resource-constrained, heterogeneous, and decentralized. Efficiently utilizing limited edge cloud resources to meet quality of service requirements is a key issue. Third, the geographic distribution of user requests is often uneven and fluctuating. Especially during peak traffic periods, a sudden surge in requests in certain hotspots can overload edge nodes, degrading system performance. Existing approaches often focus on static deployment strategies or simple load balancing algorithms, but fail to fully consider the complex dependencies between microservices and the overhead of cross-node communication. Summary of the Invention

[0003] The present invention aims to at least solve the technical problems existing in the prior art, and in particular innovatively proposes a microservice deployment method based on intimacy perception and load balancing.

[0004] In order to achieve the above-mentioned object of the present invention, the present invention provides a microservice deployment method based on intimacy perception and load balancing, comprising the following steps:

[0005] S1, defines the microservice deployment problem as a Markov decision process, which is in the form of Where S, A, P, R, and γ represent environmental information, action space, state transition probability, reward, and discount factor, respectively;

[0006] S2, input the directed weighted graph of microservices into the AttnSAGE model to obtain the state s t ,The AttnSAGE model combines GraphSAGE and GAT to extract efficient feature representation from the ,dependencies between microservices through neighborhood feature aggregation and ,attention mechanism;

[0007] S3, the state s t Data is input into the Actor network to obtain the target sample {s t ,a t ,R t, s t+1}; where a t represents all the actions that the agent can take, R trepresents the direct reward obtained by taking an action, s t Indicates the current state, s t+1 Indicates the state of the next moment; through a t The specific deployment plan is obtained based on the value of .

[0008] S4, randomly sample several samples in the experience replay pool, and put the target sample {s t ,a t ,R t, s t+1}Put it into the experience replay pool;

[0009] S5, input the target sample and the several samples into two critic networks to calculate the target Q value, and estimate the state s t and action a t The target Q value under is used to guide the deployment strategy update.

[0010] Target Q value Defined as:

[0011]

[0012] Among them, R t Represents the direct reward obtained by taking action at the current time t;

[0013] γ represents the discount factor;

[0014] min means taking the minimum value;

[0015] and There are two target Critic networks in state s t+1 and actions Q value estimation under ;

[0016] θ1′ and θ2′ are the parameters of the target critic network.

[0017] Preferably, the Actor network and the two Critic networks approximate the target Actor network and the target Critic network respectively through loss functions.

[0018] The loss function of the Actor network is defined as:

[0019]

[0020] Where N is the number of samples randomly sampled from the experience replay pool;

[0021] It is the Q value of the action selected by the Actor network evaluated by the first Critic network;

[0022] The loss function of each Critic network is defined as follows:

[0023]

[0024] in, Is the current Critic network in s t and a t Q value estimation under ;

[0025] is the target Q value calculated by the target critic network.

[0026] Preferably, the reward includes a response time optimization reward R T and load usage optimization reward R L :

[0027] Response time optimization reward R T Defined as:

[0028]

[0029] in, They represent the average response time at the current time t and the average response time at the next time t+1 respectively;

[0030] k1, k2, k3, and k4 represent parameters used for the forward iteration of the reward function;

[0031] T min Indicates the minimum response time.

[0032] Load usage optimization reward R L Defined as the variance R of CPU and memory usage of all edge nodes L (s t ,a t ,s t+1 )=-V total , V total Represents the variance of comprehensive resource utilization. Such a reward is to ensure load balancing at the edge while reducing the long-term microservice response time.

[0033] Preferably, the minimum response time T min The latency model is obtained by refining four different response time models according to the deployment of microservices:

[0034] First response time model: User U k Within the coverage of the edge server and with normal load, all microservices S j Processed by the server, the response time T is:

[0035]

[0036] in, Indicates the transmission delay from the user uploading microservice request data to the edge server;

[0037] Indicates the transmission delay from the edge server to return the processing result to the user;

[0038] Indicates the processing latency of the edge server for microservice requests;

[0039] Second response time model: User U k The edge server is within its coverage and has a normal load, but the server cannot handle subsequent microservices, so they are handled by the cloud. The response time T is:

[0040]

[0041] in, Indicates the transmission delay of users uploading microservice request data to the central cloud;

[0042] Indicates the processing latency of the edge server for microservice requests;

[0043] It represents the transmission delay from the central cloud to the user for processing results;

[0044] The third response time model: User U k If the load exceeds the set threshold within the coverage area of ​​the edge server, the subsequent microservice instances are transferred to other edge servers whose load does not exceed the set threshold. The response time T is:

[0045]

[0046] in, Indicates the transmission delay from the user uploading microservice request data to the edge server;

[0047] Indicates the processing latency of the edge server for microservice requests;

[0048] T trans Indicates the transmission delay between edge servers;

[0049] Indicates the processing delay of subsequent microservices by other edge servers;

[0050] Indicates the transmission delay of other edge servers returning the processing results to the user;

[0051] Fourth response time model: User U k All microservices are handled by the cloud and are not within the reach of edge servers.

[0052]

[0053] Preferably, the comprehensive resource utilization variance V total Obtained through the load model:

[0054]

[0055] Where M represents the number of edge servers;

[0056] μ is the average value of the comprehensive resource utilization of all edge nodes;

[0057] Indicates the resource utilization of each edge node, η represents the weight coefficient of CPU, Represents the current edge node e M CPU and memory usage.

[0058]

[0059] in, and Represents the current edge node e M Available CPU and memory resources;

[0060] max(C) and max(M) are the maximum CPU and memory capacities of all edge nodes, respectively.

[0061] Preferably, the directed weighted graph is divided into communities using an intimacy-aware LCR algorithm, and microservices with high call frequencies are assigned to the same community;

[0062] First, we use the Leiden algorithm to divide tightly coupled microservices into the same community.

[0063] Then, the microservices within each community are ranked.

[0064] Preferably, the quality of community division is measured by modularity. When two microservices are divided into the same community, the modularity increases, and the two microservices are divided into the same community. The calculation formula of modularity is:

[0065]

[0066] Among them, Q represents the modularity used to measure the quality of community division;

[0067] W=∑ i,j wij represents the sum of edge weights in a directed weighted graph;

[0068] w ij Indicates the call frequency between microservices;

[0069] δ(c i ,c j ) is the indicator function, when service S i With S j The value is 1 if they belong to the same community, otherwise it is 0;

[0070] and Represents microservices S respectively i The out-degree and S j The in-degree.

[0071] Preferably, a cross-community communication penalty term is introduced into the modularity formula. The revised modularity formula is as follows:

[0072] Q′=Q-λ·∑ i,j w ij ·(1-δ(c i ,c j )), (9)

[0073] Where Q′ represents the corrected modularity;

[0074] λ represents the cross-community communication penalty coefficient.

[0075] Preferably, the ranking of microservices in each community is determined by calculating the importance of the services. The calculation formula for service importance is:

[0076]

[0077] Among them, λ1, λ2, and λ3 represent the weight coefficients of call centrality, dependency link priority, and pre-dependency, respectively;

[0078] PR(S i ) is service S i centrality;

[0079] BC(S i ) is used to measure service S i Criticality in the call path;

[0080] |pre(S i )| is the number of dependencies of the service, including out-degree and in-degree.

[0081] According to C(S i) Sort services within the community in descending order and deploy high-priority services first to ensure that critical services can respond quickly and reduce communication delays that depend on links.

[0082] In summary, due to the adoption of the above technical solution, the method of the present invention models the microservice dependency relationship as a directed weighted graph, and uses community division and calculation of the eigenvector centrality of each microservice to generate a ranked list that comprehensively considers the dependencies between services. At the same time, graph reinforcement learning is used for dynamic resource allocation and deployment, which significantly reduces the system response delay and achieves a balanced distribution of edge node loads. This method combines graph learning with graph reinforcement learning, and by perceiving the complex dependencies between microservices, it optimizes the resource utilization of multiple edge servers, effectively reduces the response time, and at the same time improves the quality of service (QoS) of mobile Internet of Things users and minimizes resource overhead, providing an efficient solution for the deployment of intelligent microservices in edge cloud environments. The specific beneficial effects are as follows:

[0083] (1) Reduced inter-microservice communication latency. Microservices often have complex dependencies. When frequently dependent microservices are deployed in distributed edge nodes at different locations, response latency increases significantly due to synchronous calls and data sharing. This invention senses the dependencies between microservices and preferentially allocates closely related microservices to the same or adjacent nodes, effectively reducing cross-node communication overhead.

[0084] (2) Optimized resource utilization. Resources in edge cloud environments are limited, heterogeneous, and dispersed. Traditional static deployment strategies make it difficult to fully utilize the computing, storage, and bandwidth resources of edge nodes. This paper combines graph learning and reinforcement learning to dynamically perceive the resource status of edge nodes, optimize resource allocation schemes, and achieve efficient utilization of limited resources.

[0085] (3) Load balancing is achieved. The geographical distribution of user requests is uneven and dynamically fluctuating. Especially during peak traffic periods, certain hotspots may experience a sudden surge in requests, leading to overloaded edge nodes. This invention monitors the load of edge nodes in real time and optimizes deployment strategies through PLMD-GRL to ensure load balancing across nodes and improve system stability.

[0086] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments with reference to the following drawings, in which:

[0088] Figure 1Schematic diagram of the edge cloud environment of the present invention.

[0089] Figure 2 Schematic diagram of the microservice model of the present invention.

[0090] Figure 3 Schematic diagram of the LCR algorithm framework of the present invention.

[0091] Figure 4 This is a diagram of the microservice deployment method based on graph reinforcement learning in the present invention. DETAILED DESCRIPTION

[0092] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0093] We will introduce model construction, including system model, microservice model, latency model, and load model, as well as the implementation of the intimacy-aware LCR algorithm, AttnSAGE model, and PLMD-GRL.

[0094] 1 Model construction

[0095] 1.1 System Model

[0096] like Figure 1 ,The system model is divided into user layer, edge layer and central cloud layer. The user layer consists of various mobile devices such as smartphones, IoT devices, sensors, etc. distributed in different geographical locations U={U1,U2,…,U k These devices continuously generate a large number of applications A={a1,a2,…,a i}, each application consists of a series of microservices S = {S1, S2, ..., S u The edge layer consists of M edge servers E = {e1, e2, ..., e M}, each edge server has computing resources Geographical location is The service radius is We assume that the central cloud layer has sufficient resources. When the edge server resources are insufficient or the user is out of service range, the central cloud is responsible for processing the task.

[0097] 1.2 Microservice Model

[0098] The execution of a microservice usually depends on the completion of one or more other microservices, so we use a directed weighted graph G = (V, E, W) to represent the calling relationship of microservices. u}, each node S i Corresponding to a microservice. Edge E = {e ij |i,j∈V},e ij Represents microservice S i Call microservice S j 。 W={w ij} indicates the calling frequency, between microservices (microservice S i and microservices j )’s call frequency w ij It is represented by the intimacy matrix AD, where the horizontal and vertical coordinates of the intimacy matrix AD represent the i-th microservice respectively. Taking the online retail application a1 as an example, user U1 accesses the application through a smartphone or computer to shop. The application consists of multiple microservices, including user authentication microservice S1, product recommendation microservice S2, shopping cart management microservice S3, payment microservice S4, order management microservice S5, and logistics management microservice S6. By tracking microservice S iu The number of requests and responses between them is used to construct the microservice intimacy matrix AD. The specific process is as follows Figure 2 In it, I. shows the composition of microservices in online retail applications; II. builds a microservice call graph based on the intimacy matrix; III. shows a deployment optimization solution based on microservice intimacy awareness.

[0099] 1.3 Delay Model

[0100] For different microservice deployment scenarios, we define task response latency models to provide theoretical support for subsequent latency optimization. When a user sends a service request to a nearby edge server, transmission latency occurs during the communication process. This latency is affected by multiple factors, including physical distance, network bandwidth, and edge server load. Four different response time models are refined based on the microservice deployment scenario. Response time T is composed of transmission latency and processing latency.

[0101] i. User U k Within the coverage of the edge server and with normal load, all microservices S j For example, all microservices (such as S1, S2, S3, S4, S5, and S6) are processed on the nearest edge server, so the response time only involves the two-way transmission delay between the user and the edge server, as well as the processing delay on the edge server.

[0102]

[0103] in, Indicates the transmission delay from the user uploading microservice request data to the edge server;

[0104] Indicates the transmission delay from the edge server to return the processing result to the user;

[0105] Indicates the processing latency of the edge server for microservice requests;

[0106] ii. User U k The edge server is within its coverage and has a normal load, but the server cannot process subsequent microservices and transfers them to the cloud. Assume that the edge server processes some microservices S edge (such as S1, S2, S3), but cannot handle subsequent microservices S cloud (such as S4, S5, S6), and subsequent microservices will be transferred to the cloud for processing.

[0107]

[0108] in, Indicates the transmission delay of users uploading microservice request data to the central cloud;

[0109] Indicates the processing latency of the edge server for microservice requests;

[0110] It represents the transmission delay from the central cloud to the user for processing results;

[0111] iii. User U k If an edge server is within its coverage but the load is too high, subsequent microservice instances are transferred to other edge servers with lower loads. Suppose some microservices (such as S1 and S2) are processed by the current edge server, but due to load issues, subsequent microservices (such as S3, S4, S5, and S6) are transferred to other edge servers for processing.

[0112]

[0113] in, Indicates the transmission delay from the user uploading microservice request data to the edge server;

[0114] Indicates the processing latency of the edge server for microservice requests;

[0115] T trans Indicates the transmission delay between edge servers;

[0116] Indicates the processing delay of subsequent microservices by other edge servers;

[0117] Indicates the transmission delay of other edge servers returning the processing results to the user;

[0118] iv. User U k All microservices are handled by the cloud and are not within the reach of edge servers.

[0119]

[0120] 1.4 Load Model

[0121] To avoid overloading edge node resources during peak traffic periods and thus degrading microservice performance, we propose a load balancing model based on resource utilization. This model monitors the CPU and memory usage of each edge node in real time to determine whether resource utilization is excessive. The specific calculation method is as follows:

[0122]

[0123] in, Represents the current edge node e M CPU and memory usage;

[0124] and Represents the current edge node e M Available CPU and memory resources;

[0125] max(C) and max(M) are the maximum CPU and memory capacities of all edge nodes, respectively.

[0126] Resource utilization of each edge node η represents the weight coefficient of CPU;

[0127] The variance of comprehensive resource utilization V total The calculation formula is:

[0128]

[0129] Where M represents the number of edge servers;

[0130] μ is the average value of the comprehensive resource utilization of all edge nodes. By calculating the variance V total , we can judge the uniformity of resource distribution in the system. The smaller the variance, the more balanced the resource allocation.

[0131] In this paper, our goal is to minimize the microservice response delay while ensuring load balancing. Therefore, the problem can be expressed as: Min{T, V total}, where T is the response time, V total is the comprehensive resource utilization variance.

[0132] 2 Intimacy-aware LCR algorithm

[0133] In order to reduce the cross-node communication overhead between microservices, the LCR algorithm uses the modular characteristics of the graph to divide the community and assign microservices with high call frequency to the same community. The process is as follows: Figure 3 As shown in Figure 1, where I represents the input directed weighted graph (DWG), II uses the Leiden algorithm to partition tightly coupled microservices into the same community, and III illustrates the method of prioritization within the community. For the directed weighted graph G of microservices, modularity is used to measure the quality of community partitioning, which is defined as:

[0134]

[0135] Among them, Q represents the modularity used to measure the quality of community division;

[0136] W=∑ i,j w ij represents the sum of edge weights in a directed weighted graph;

[0137] w ij Indicates the call frequency between microservices;

[0138] δ(c i ,c j ) is the indicator function, when service S i With S j The value is 1 if they belong to the same community, otherwise it is 0;

[0139] and Represents microservices S respectively i The out-degree and S j The in-degree.

[0140] By maximizing modularity Q, the LCR algorithm can effectively divide microservice communities, ensuring that closely dependent services are preferentially assigned to the same geographic location. However, when microservices are deployed on different nodes, the communication cost of cross-community calls increases significantly. To further optimize the geographical concentration of service distribution, the LCR algorithm introduces a cross-community communication penalty term into the modularity formula. The revised modularity formula is as follows:

[0141] Q′=Q-λ·∑ i,j w ij ·(1-δ(c i ,c j )), (9)

[0142] Where Q′ represents the corrected modularity;

[0143] λ represents the cross-community communication penalty coefficient;

[0144] After the community division is completed, the LCR algorithm sorts the services within each community to determine the deployment priority of the services. The sorting within the community comprehensively considers the call centrality of the service and the priority of the dependent link, and uses the following formula to calculate the service importance:

[0145]

[0146] Among them, λ1, λ2, and λ3 represent the weight coefficients of call centrality, dependency link priority, and pre-dependency, respectively;

[0147] PR(S i ) is service S i The centrality of can be obtained through the PageRank algorithm;

[0148] BC(S i )Measurement Service S i The criticality in the call path can be obtained by the Betweenness Criticality algorithm;

[0149] |pre(S i )| is the number of dependencies of the service, including out-degree and in-degree.

[0150] According to C(S i ) Sort services within the community in descending order and deploy high-priority services first to ensure that critical services can respond quickly and reduce communication delays that depend on links.

[0151] 3AttnSAGE model

[0152] In the dependency-aware deployment of microservices, the complex dependencies and communication patterns between microservices are crucial for optimizing deployment strategies. To effectively extract feature representations of complex dependencies between microservices, we propose a graph neural network-based model, AttnSAGE. This model combines GraphSAGE with the Graph Attention Network (GAT). By combining neighborhood feature aggregation and an attention mechanism, it extracts efficient feature representations from the dependencies between microservices, thereby supporting subsequent deployment strategy learning.

[0153] We use a directed weighted graph G = (V, E, W) to represent the microservice dependency and its deployment relationship. V, E, and W represent the node set, edge set, and weight set, respectively. In this graph, the node set V contains two types of nodes: microservice nodes and edge nodes. Each microservice S i Features Includes the computing requirements, storage requirements of the service and the dependencies with other microservices. Each edge node e j Features Including computing power, storage capacity, and network bandwidth. E is divided into two categories. If it is a microservice S i With edge node e j The edge between them (i.e., the deployment relationship) is recorded as E1 = {(i,j)|i∈S,j∈E}. If it is a microservice node S i and S j The edge between them (i.e., the call relationship) is recorded as E2 = {(i,j)|i,j∈S,i≠j}. The weight set W is expressed as follows:

[0154]

[0155] In edge cloud environments, due to the dynamic and complex distribution of user requests and edge servers, as well as the large scale of nodes and strong dependencies between neighborhoods, how to efficiently extract local features of nodes becomes a key issue. In order to quickly process the local dependencies of nodes and reduce the complexity of global computation, we use GraphSAGE as the initial aggregation stage of AttnSAGE. GraphSAGE is a graph neural network model based on local neighborhood aggregation. It updates the feature representation of each node by aggregating information from neighboring nodes, thereby achieving efficient local feature extraction. For a node, the feature update process is expressed as follows:

[0156]

[0157] in, represents the feature representation of node v in the l+1th layer, N(v) represents the set of neighbor nodes of node v, Agg represents the aggregation function, which adopts the mean aggregation operation, and ⊕ represents the feature splicing operation.

[0158] However, in microservice deployment scenarios, different neighboring nodes have significantly different impacts on the target node. This heterogeneous information is crucial for optimizing deployment strategies. For example, information about frequently called neighboring nodes is crucial for updating the target node's features, while weakly correlated neighboring nodes contribute noise to feature updates. To capture this heterogeneity, we further introduce GAT's attention mechanism on top of GraphSAGE aggregation to dynamically adjust the weights of neighboring node feature aggregation.

[0159] Calculate the attention coefficient e of each neighbor node to the target node through the self-attention mechanism vu , as shown below:

[0160]

[0161] Among them, W (l)is a linear transformation matrix, a is a learnable parameter vector, ∥ represents the connection of two vectors, and LeakyReLU(·) is an activation function used to introduce nonlinear transformation. vu is the edge weight, which reflects the strength of the association between the neighbor node and the target node, and refers to the call frequency or resource allocation relationship. In order to ensure the stability of the attention coefficient, softmax is used to calculate e vu Normalized to get α vu , represents the weight of neighbor nodes on the target node feature aggregation. The formula is:

[0162]

[0163] Through normalization operation, the attention coefficient α vu Satisfy∑ u∈N(v) α vu = 1. Finally, the features of the neighboring nodes are weighted and aggregated according to the attention coefficient, and the target node features are updated through the activation function σ:

[0164]

[0165] Through a dynamic weighted aggregation mechanism, the model focuses on neighbor information that has a greater impact on the target node, while deemphasizing neighbor information with weaker associations. To balance weight distribution and reduce potential noise interference, the AttnSAGE model further incorporates GraphSAGE for global fusion and enhancement, generating more robust feature representations with a global perspective, providing strong support for subsequent deployment strategy optimization.

[0166] Implementation of 4PLMD-GRL

[0167] 4.1DRL Design

[0168] We define the microservice deployment problem as a Markov Decision Process (MDP) in the form of Where S, A, P, R, and γ represent environmental information, action space, state transition probability, reward, and discount factor, respectively;

[0169] 1) State space. S defines all the environmental information that the agent can observe at each time slot t Among them, including (i) microservice request queue Records all microservices S waiting to be processed i And quantity. Based on the dependencies between microservices, the LCR algorithm generates queues And optimize the request allocation by adjusting the scheduling priority of the requests in the queue. (ii) Current scheduling request information, each microservice S i (t) includes its CPU usage requirements and memory requirements (iii) Edge Server Status The size of the edge node set E, |E|=M, represents the number of edge nodes currently involved. For each edge node e M , its status Including the remaining available CPU resources and memory resources Geographical location and service radius (iv) Cloud Server Status Here we assume that cloud resources are sufficient. (v) User device status The size of the user set U |U|=K represents the number of users. Contains the user's geographic location.

[0170] 2) Action space. A represents the action space in state s t All actions a that the agent can take t We assume that each proxy can only dispatch one request in a time slot to ensure the timeliness of scheduling. For each microservice, the proxy will select an action to deploy it to the specified edge node or cloud. If there are M edge nodes and 1 cloud node, then in each time slot, a t Includes M+1 discrete actions.

[0171] 3) State transition probability. We use P(s t+1 |s t ,a t ) describes the current state s t and take action a t Then the system transfers to the next state s t+1 The state transition is affected by the deployment adjustment of microservices and the changes in resource usage of edge nodes and the cloud.

[0172] 4) Reward function. The goal of the agent is to take action a t Get the maximum expected reward To minimize the total response delay and the load between edge nodes. γ∈(0,1] is a discount factor used to balance the weights of immediate rewards and future rewards. t It is the direct reward for taking action, mainly considering the response time to optimize the reward R T and load usage optimization reward R L Specifically, (i) the response time optimization reward R T Defined as

[0173]

[0174] (ii) Load usage optimization reward R L Defined as the variance R of CPU and memory usage of all edge nodes L (s t ,a t ,s t+1 )=-V total The reward is to ensure load balancing at the edge while reducing long-term microservice response time.

[0175] 4.2 Training Process

[0176] like Figure 4 , the PLMD-GRL framework demonstrates the process of optimizing microservice deployment in edge cloud environments. (1) Initialization phase: Initialize the parameters of the Actor network and the two Critic networks. Initialize the corresponding target network parameters and set their parameters to be the same as the main network. And create an experience replay buffer to store the experience samples generated by the interaction between the agent and the environment. (2) Interacting with the environment, the agent responds to the environment based on the current state s t Use the Actor network to generate actions and add some random noise (such as Gaussian noise) for exploration. t Act on the environment to obtain rewards and next state. t ,a t ,R t, s t+1} is stored in the experience replay buffer. (3) During the training phase, a mini-batch of samples is randomly sampled from the experience replay buffer every other time step to update the Actor and Critic networks. The Critic network is updated as follows: 1) Generate the target Q value from the target Actor and the target Critic network: First, the target Actor network generates the next action, and then uses the target Critic network to estimate the target Q value; 2) Calculate the TD error and update the Critic network parameters: First, the loss function of each Critic is the mean square error; Then, the parameters of the Critic network are updated through back propagation. The Actor network is updated as follows: 1) Update the Actor network every certain number of steps: First, use the gradient provided by the Critic network to update the parameters of the Actor network so that the action generated by the Actor can maximize the Q value; Then, calculate the loss function of the Actor; 2) Update the Actor network parameters through back propagation. Both the Critic network and the Actor network use the same target network parameter soft update. (4) Repeat training; Repeat the process of interacting with the environment, sampling data, and updating the network until convergence or the specified training step is reached. The framework uses the AttnSAGE model to capture the dependencies and deployment status between microservices and adopts an actor-critic based deep reinforcement learning (DRL) strategy for deployment optimization. The training process of PLMD-GRL is as follows:

[0177] 1) Dual Critic Networks. To reduce bias in value estimation, PLMD-GRL uses two independent Critic networks to calculate the target Q-value. The target Q-value is calculated using the Bellman equation, and the loss function of each Critic network is defined as follows:

[0178]

[0179] Where N is usually the number of samples randomly sampled from the experience replay pool;

[0180] Is the current Critic network in s t and a t The Q value is estimated under The target Q value is calculated by the target Critic network. Specifically, the target Q value is Defined as:

[0181]

[0182] Among them, R t Represents the direct reward obtained by taking action at the current time t;

[0183] γ represents the discount factor;

[0184] min means taking the minimum value;

[0185] and There are two target Critic networks in state s t+1 and actions Q value estimation under ;

[0186] θ1′ and θ2′ are the parameters of the target critic network.

[0187] is the noisy action output by the target Actor network at the next moment, that is where ∈ is the noise vector, and π φ′ is the target Actor network, represents a normal distribution with mean 0 and standard deviation σ, and c represents the noise clipping threshold.

[0188] 2) Actor Network. The goal of the Actor Network is to select actions that maximize the Q-value evaluated by the Critic Network, thereby maximizing the cumulative reward. To improve the stability of training, PLMD-GRL adopts a delayed update strategy, that is, the update delay of the Actor Network is not synchronized with the update of the Critic Network. The loss function of the Actor is defined as:

[0189]

[0190] Where N is usually the number of samples randomly sampled from the experience replay pool;

[0191] It is the Q value of the action selected by the Actor network evaluated by the first Critic network. In order to update the parameters of the Actor network, the following gradient calculation method is used to update the parameters.

[0192]

[0193] in, Represents the Actor network's parameters θ A gradient;

[0194] θ A Represents the parameters of the Actor network;

[0195] α A is the learning rate of the Actor network;

[0196] It’s about the gradient of action;

[0197] is the gradient of the Actor network.

[0198] 3) Target Network and Soft Update. To further improve the stability of training, PLMD-GRL uses a target network whose parameters are updated through a soft update mechanism. The update formula of the target network parameters is as follows:

[0199] θ′ i ←τθ i +(1-τ)θ′ i (twenty two)

[0200] Among them, θ i Represents the parameters of the current evaluation network, which can be the parameters of the Critic network or the Actor. θ′ i Represents the parameters of the target network, that is, the parameters of the target Critic network or the target Actor network. τ∈(0,1] is the soft update coefficient, which controls the update speed.

[0201] To verify the effectiveness of the proposed method PLMD-GRL, we conducted experimental verification based on the Australian EUA real dataset and synthetic data.

[0202] (1) Experimental setup

[0203] We conduct extensive experiments to evaluate the performance of PLMD-GRL. We use the real-world Australian EUA dataset and some synthetic data. The EUA dataset contains 125 edge servers and 817 user devices in Australia.

[0204] 1) Edge-cloud Environment: We selected servers from different geographical locations in the EUA dataset as edge nodes to construct a multi-region edge cloud environment. Each server was configured with different resource parameters, including processing power, storage capacity, and network bandwidth. Bandwidth constraints existed between edge nodes, simulating a realistic heterogeneous network environment. Furthermore, we set up a central cloud node with unlimited resources to handle tasks that the edge nodes could not complete.

[0205] 2) IoT Users: A certain number of IoT devices are randomly deployed in each area to simulate real-world user behavior. The geographic locations of the user devices are derived from the EUA real-world dataset. Each user sends service requests through the application, and the number and type of requests generated by different users vary. Furthermore, user requests exhibit specific patterns over time, simulating real-world peak traffic.

[0206] 3) Cross-region scenarios: We evaluate four typical experimental scenarios to test the performance of PLMD-GRL, as shown in Table 1.

[0207] 4) Microservice Requests: Each microservice has different resource requirements (such as compute, storage, and bandwidth). A typical application scale is approximately 20 applications, and each application calls 10-20 microservices. Regarding the dependencies between microservices, we simulate real-world microservice calls by generating an affinity matrix.

[0208] We implemented PLMD-GRL in PyTorch and combined it with the PyTorch Geometric library to accelerate graph neural network training and inference. We also used the reinforcement learning algorithm TD3 (Twin Delayed Deep Deterministic Policy Gradient) to train PLMD-GRL. The detailed setup is as follows: For the LCR implementation, we used Leiden for community detection and PageRank to calculate the importance ranking of nodes within a community. For the graph neural network, we constructed a hybrid graph network model using GraphSAGE and GAT, with a multi-head attention mechanism with 4 heads and 16 hidden layers. GATD consists of an actor network and two critic networks, all trained using the Adam optimizer with a fixed learning rate of 1×10^(-4). The actor network θ_A and the two critic networks Q_(θ_1) and Q_(θ_2) are parameterized by three layers of neural networks with Reluctant Unit (ReLU) activations. The output layer of θ_A uses the Tanh activation function to constrain the action range to [-1, 1]. The node sizes of the three layers of each critic network are 512, 512, and 1, respectively. The experiments were conducted on a Windows 11 desktop computer with the following specifications: 128GB of RAM, a 12th Gen i9-12900K CPU, and an RTX 4090 GPU. The programming language was Python 3.8, running the deep learning framework PyTorch 2.1.1 with CUDA 11.8.

[0209] Table 1 Cross-region scenarios

[0210] Scenario Number of servers Number of users Features Scenario 1 4 47 Two large areas and two small areas Scenario 2 5 60 Hotspot areas and non-hotspot areas have obvious characteristics Scenario 3 6 70 Request distribution is extremely uneven Scene 4 8 100 Large-scale and evenly distributed regions

[0211] (2) Performance evaluation and analysis

[0212] To evaluate the performance of PLMD-GRL, this study compares PLMD-GRL with eight baseline methods, including traditional reinforcement learning algorithms, deep reinforcement learning algorithms, and simple random and round-robin (RR) strategies. The comparison algorithms include A2C, DQN, SAC, REINFORCE, SARSA, TRPO, Random, and RR. We primarily consider two performance metrics: (i) average deployment response time; and (ii) edge server load.

[0213] (2.1) Analysis of experimental results with different weight coefficients

[0214] Comprehensive resource utilization variance V total It is used to measure the stability of resource utilization in the system. The comprehensive resource utilization is a weighted average of CPU and memory resource utilization, balanced by the weight coefficient η. Therefore, we analyze the performance of PLMD-GRL based on different weight coefficients.

[0215] Table 2 Comparison of experimental results with different weight coefficients

[0216] Weight coefficient Average response time (ms) load(%) η=0.3 576.63 0.21 η=0.4 595.94 0.24 η=0.5 602.66 0.25 η=0.6 628.33 0.32 η=0.7 586.06 0.27

[0217] In Table 2, when η = 0.3, the average response time and load are 576.63 ms and 0.21%, respectively. This deployment method performs optimally overall, with low resource utilization volatility. As η increases, system performance varies to varying degrees. When η = 0.4 and η = 0.5, the average response time and load both increase, reaching (595.94 ms, 0.24%) and (602.66 ms, 0.25%), respectively. This indicates that the increased CPU utilization makes the system more prone to processing compute-intensive tasks, but also leads to a gradual imbalance in resource utilization. When η = 0.6, the average response time and load reach their maximum values. Subsequently, when η = 0.7, although the average response time decreases, the load increases to 0.27%. This may be due to the increased resource contention caused by the excessive weighting of CPU utilization. Despite a slight improvement in load, the overall performance is still inferior to the results when η = 0.3.

[0218] (2.2) Experimental results analysis of different baseline methods

[0219] As shown in Table 3, the PLMD-GRL method performs best in the comparison of different methods. Specifically, PLMD-GRL achieves an average response time of 576.63ms and a load of 0.21%, representing improvements of 13.78% and 65.57%, respectively, compared to the best results among all other methods. In contrast, the REINFORCE and TRPO methods have average response times of 668.62ms and 677.44ms, respectively, and loads of 0.61% and 0.62%, respectively, showing significant increases in response time and load compared to PLMD-GRL. Despite their improvements in processing power for computationally intensive tasks, the A2C and DQN methods achieve reductions of 16.47% and 66.28%, respectively, compared to PLMD-GRL. The RR and SAC methods perform the worst, achieving reductions of 30.21% and 76.92%, respectively, compared to PLMD-GRL. Furthermore, while the Random and SARSA methods performed slightly better than RR and SAC, their average response time and load were still significantly higher than PLMD-GRL, decreasing by 16.69% and 67.57% respectively. Overall, the PLMD-GRL method demonstrated clear advantages in service deployment efficiency and stability.

[0220] Table 3 Comparison of experimental results of different baseline methods

[0221] method Average response time (ms) load(%) A2C 690.78 0.67 DQN 693.62 0.69 RR 826.19 0.91 Random 692.21 0.63 REINFORCE 668.62 0.61 SAC 801.98 0.67 SARSA 704.71 0.7 TRPO 677.44 0.62 PLMD-GRL 576.63 0.21

[0222] (2.3) Experimental results analysis of different numbers of applications

[0223] As shown in Table 4, the PLMD-GRL method demonstrates significant superiority across different application numbers (20, 30, 40, 50, and 60). Its average response times, 576.63ms, 626.02ms, 637.29ms, 634.52ms, and 635.55ms, respectively, are consistently significantly lower than those of the other methods. The load remains low, ranging from 0.21% to 0.23%, demonstrating excellent stability and resource allocation efficiency. In contrast, the reinforcement learning methods REINFORCE and TRPO perform suboptimally, with response times increasing from 668.62ms and 677.44ms to 733.75ms and 727.23ms, respectively. Their loads are slightly higher than PLMD-GRL, but relatively stable. The traditional method RR performs the worst, with the highest response time, increasing from 826.19ms to 923.84ms. Its load remains consistently between 0.91% and 0.92%, demonstrating extremely low resource utilization efficiency. Although random methods and other reinforcement learning methods (such as SAC and SARSA) slightly outperform RR in some scenarios, their response times and load are significantly higher than PLMD-GRL. Overall, the PLMD-GRL method demonstrates high adaptability and efficiency when handling applications of varying scales, significantly outperforming other compared methods in both average response time and load.

[0224] Table 4 Experimental results analysis of different numbers of applications

[0225]

[0226] (2.4) Experimental results analysis of different resource capacity settings

[0227] In this experiment, we investigated the impact of different capacity settings on the performance of various baseline methods by increasing server resource capacity by 0.5x, 1x, 1.5x, 2x, and 4x, assessing their adaptability and stability under both resource-constrained and resource-abundant conditions. As shown in Table 5, the PLMD-GRL method performs best under all resource capacity settings. Its average response time is 706.61ms at low resource capacity (0.5x), decreasing to 516.65ms at high resource capacity (4x), consistently significantly lower than the other methods. Its load steadily decreases from 0.1% (0.5x) to 0.12% (4x), demonstrating excellent resource utilization efficiency and stability. In contrast, the reinforcement learning methods TRPO and REINFORCE closely follow, with response times and load approaching those of PLMD-GRL under high-capacity scenarios. However, under low-capacity conditions (0.5x), their loads are higher, reaching 0.71% and 0.73%, respectively, affecting overall performance. The A2C and DQN methods show significant performance improvements when resource capacity increases, but suffer from high response times (over 900ms) and high load (over 0.76%) in low-capacity scenarios, making them less adaptable than PLMD-GRL. Traditional RR and random methods perform the worst, especially at low resource capacity, with response times exceeding 900ms and loads as high as 0.97%, resulting in resource utilization efficiency far lower than that of reinforcement learning methods. Overall, the PLMD-GRL method achieves efficient resource utilization with the lowest response time and load across various resource capacity settings, making it the optimal choice for multi-server environments.

[0228] Table 5 Experimental results analysis of different resource capacity settings

[0229]

[0230] (2.5) Analysis of experimental results in different cross-regional scenarios

[0231] In this experiment, we analyzed the performance of various baseline methods under different cross-region scenarios (varying numbers of servers and users), as shown in Table 6. The experimental results show that the PLMD-GRL method performs best in all scenarios, with the lowest average response time, ranging from 576.63ms (Scenario 1) to 905.17ms (Scenario 4). The load also remains at a minimum level (0.21%-0.25%), demonstrating extremely high deployment efficiency. Reinforcement learning methods TRPO and REINFORCE follow closely behind, showing some adaptability in complex scenarios (such as Scenario 3 with uneven request distribution and Scenario 4 with large-scale requests), but their response times and load are slightly higher than PLMD-GRL. In contrast, traditional methods RR and random methods perform worst in complex scenarios, especially in Scenario 3 (extremely uneven request distribution) and Scenario 4 (large-scale requests). The response time and load fluctuate significantly, making it difficult to effectively allocate resources. In summary, the PLMD-GRL method shows obvious advantages in complex cross-regional scenarios, and is especially suitable for complex environments with multiple servers and large-scale user distribution.

[0232] Table 6 Experimental results analysis under different cross-regional scenarios

[0233]

[0234] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A microservice deployment method based on intimacy perception and load balancing, characterized in that: The following steps are involved: S1, defines the microservice deployment problem as a Markov decision process, which is in the form of Where S, A, P, R, and γ represent environmental information, action space, state transition probability, reward, and discount factor, respectively; S2, input the directed weighted graph of microservices into the AttnSAGE model to obtain the state s t , the AttnSAGE model combines GraphSAGE and GAT; The directed weighted graph is divided into communities using the intimacy-aware LCR algorithm, assigning microservices with high call frequencies to the same community. First, we use the Leiden algorithm to divide tightly coupled microservices into the same community. Then, sort the microservices within each community; The quality of community division is measured by modularity. When two microservices are divided into the same community, the modularity increases, and the two microservices are divided into the same community. The calculation formula for modularity is: Among them, Q represents the modularity used to measure the quality of community division; W=∑ i,j w ij represents the sum of edge weights in a directed weighted graph; w ij Indicates the call frequency between microservices; δ(c i ,c j ) is the indicator function, when service S i With S j The value is 1 if they belong to the same community, otherwise it is 0; and Represents microservices S respectively i The out-degree and S j The in-degree of S3, the state s t Data is input into the Actor network to obtain the target sample {s t ,a t ,R t ,s t+1 }; where a t represents all the actions that the agent can take, R t represents the direct reward obtained by taking an action, s t Indicates the current state, s t+1 Indicates the state at the next moment; The loss function of the Actor network is defined as: Where N is the number of samples randomly sampled from the experience replay pool; It is the Q value of the action selected by the Actor network evaluated by the first Critic network; The loss function of each Critic network is defined as follows: in, Is the current Critic network in s t and a t Q value estimation under ; is the target Q value calculated by the target Critic network; The rewards include the response time optimization reward R T and load usage optimization reward R L : Response time optimization reward R T Defined as: in, They represent the average response time at the current time t and the average response time at the next time t+1 respectively; k1, k2, k3, k4 represent parameters; T min Indicates the minimum response time; Load usage optimization reward R L Defined as the variance R of CPU and memory usage of all edge nodes L (s t ,a t ,s t+1 )=-V total , V total represents the variance of comprehensive resource utilization; S4, randomly sample several samples in the experience replay pool, and put the target sample {s t ,a t ,R t, s t+1 }Put it into the experience replay pool; S5, input the target sample and the plurality of samples into two Critic networks to calculate the target Q value, the target Q value Defined as: Among them, R t Represents the direct reward obtained by taking action at the current time t; γ represents the discount factor; min means taking the minimum value; and There are two target Critic networks in state s t+1 and actions Q value estimation under ; θ1′ and θ2′ are the parameters of the target critic network.

2. A microservice deployment method based on intimacy perception and load balancing according to claim 1, characterized in that: The minimum response time T min The latency model is obtained by refining four different response time models according to the deployment of microservices: First response time model: User U k Within the coverage of the edge server and with normal load, all microservices S j Processed by the server, the response time T is: in, Indicates the transmission delay from the user uploading microservice request data to the edge server; Indicates the transmission delay from the edge server to return the processing result to the user; Indicates the processing latency of the edge server for microservice requests; Second response time model: User U k The edge server is within its coverage and has a normal load, but the server cannot handle subsequent microservices, so they are handled by the cloud. The response time T is: in, Indicates the transmission delay of users uploading microservice request data to the central cloud; Indicates the processing latency of the edge server for microservice requests; It represents the transmission delay from the central cloud to the user for processing results; The third response time model: User U k If the load exceeds the set threshold within the coverage area of ​​the edge server, the subsequent microservice instances are transferred to other edge servers whose load does not exceed the set threshold. The response time T is: in, Indicates the transmission delay from the user uploading microservice request data to the edge server; Indicates the processing latency of the edge server for microservice requests; T trans Indicates the transmission delay between edge servers; Indicates the processing delay of subsequent microservices by other edge servers; Indicates the transmission delay of other edge servers returning the processing results to the user; Fourth response time model: User U k All microservices are handled by the cloud and are not within the reach of edge servers.

3. A microservice deployment method based on intimacy perception and load balancing according to claim 1, characterized in that: The comprehensive resource utilization variance V total Obtained through the load model: Where M represents the number of edge servers; μ is the average value of the comprehensive resource utilization of all edge nodes; Indicates the resource utilization of each edge node, η represents the weight coefficient of CPU, Represents the current edge node e M CPU and memory usage.

4. A microservice deployment method based on intimacy perception and load balancing according to claim 1, characterized in that: It also includes the introduction of cross-community communication penalty terms in the modularity formula. The revised modularity formula is as follows: Q′=Q-λ·∑ i,j w ij ·(1-δ(c i ,c j )),(9) Where Q′ represents the corrected modularity; λ represents the cross-community communication penalty coefficient.

5. The microservice deployment method based on intimacy perception and load balancing according to claim 1 is characterized in that: The ranking of services in each community is determined by calculating the importance of the services. The calculation formula for service importance is: Among them, λ1, λ2, and λ3 represent the weight coefficients of call centrality, dependency link priority, and pre-dependency, respectively; PR(S i ) is service S i centrality; BC(S i ) is used to measure service S i Criticality in the call path; |pre( i )| is the number of dependencies of the service, including out-degree and in-degree.

Images