Cloud edge fusion environment micro-service deployment method based on multi-agent reinforcement learning

Abstract

The invention discloses a cloud edge fusion environment micro-service deployment method based on multi-agent reinforcement learning. Firstly, an edge layer is divided into a plurality of areas, and each area comprises a plurality of edge servers; constructing a spatial index tree according to the distance between the edge servers; then, according to the dependency relationship among the micro-services and historical requests of the micro-services, generating importance of the micro-services; selecting a plurality of micro-services with high importance as micro-services to be deployed; and finally, constructing a deployment strategy generation model based on multi-agent reinforcement learning, and deploying the micro-service to be deployed to the corresponding edge server according to a deployment strategy. By optimizing the time delay of the micro-service and the resource state of the edge server at the same time, it is avoided that too many micro-services are deployed to a few edge servers in a centralized mode, the micro-services are promoted to be distributed to different edge servers more evenly, the overload risk of a single edge server is reduced, and the service quality of the edge server is improved. And the resource utilization rate of the whole edge computing infrastructure is also improved.

Description

Technical Field

[0001] The present invention belongs to the field of edge computing technology, and specifically relates to a cloud-edge fusion environment microservice deployment method based on multi-agent reinforcement learning. Background Art

[0002] With the rapid development of the Internet of Things (IoT) technology, a large number of mobile IoT devices connected to fifth-generation mobile communication technology (5G) networks have generated massive amounts of data and brought heavy IoT application tasks. Traditional centralized cloud platforms are often far away from IoT devices, resulting in long task completion delays, which is unacceptable for latency-sensitive tasks such as real-time monitoring and traffic scheduling. Emerging new paradigms in industrial environments, such as edge computing, bring resources closer to customers, offering low latency and low energy consumption. Mobile Edge Computing (MEC) brings the high-performance capabilities of the cloud to the edge of the network, enabling tasks to be processed at edge nodes, significantly reducing the pressure of offloading traffic through the core network and reducing service response times.

[0003] With the massive influx of mobile devices accessing the internet, the number of services deployed to edge servers and user demand is increasing. However, edge server resources are limited, so properly deploying services and allocating resources within them is crucial. Furthermore, the latency requirements of different services vary, requiring consideration of the dynamic service demands from multiple IoT devices while simultaneously meeting latency constraints. This is a challenging process in a converged cloud-edge environment. In some cases, not directly estimating resources from edge servers can degrade the quality of service for IoT devices. Furthermore, the load and latency constraints of IoT devices cannot be ignored. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the technical problem that the present invention intends to solve is to provide a cloud-edge fusion environment microservice deployment method based on multi-agent reinforcement learning.

[0005] The present invention solves the technical problem by adopting the following technical solutions:

[0006] A method for deploying microservices in a cloud-edge fusion environment based on multi-agent reinforcement learning, characterized by comprising the following steps:

[0007] Step 1: Divide the edge layer into multiple regions, each containing multiple edge servers; construct a spatial index tree based on the distance between edge servers;

[0008] Step 2: Generate the importance of microservices based on the dependencies between microservices and the historical requests of microservices; select multiple microservices with high importance as the microservices to be deployed;

[0009] The call chain of the user requesting a microservice includes multiple microservices. Assume that the call chain l k Contains microservices u and w v , call chain l h Contains microservices p and w q ;

[0010] If the microservice w u With w v If there is data transmission between them, the call chain l k The values ​​of elements (u,v) and (v,u) in the importance matrix increase at the same time; if the microservice w u With w v If no data is transmitted between them, the call chain l k The values ​​of elements (u,v) and (v,u) in the importance matrix are 0; if the microservice w p and w q If there is data transmission between them, the call chain l h The values ​​of elements (p,q) and (q,p) in the importance matrix increase by r at the same time; if microservice w p and w q If no data is transmitted between them, the call chain l h The values ​​of elements (p,q) and (q,p) in the importance matrix are 0; where t is the call chain l k The number of users, r is the call chain l h the number of users;

[0011] The sum of the importance matrices of all call chains is the microservice importance matrix, and the importance of a microservice is the sum of the elements in the corresponding row of the microservice importance matrix;

[0012] Step 3: Build a deployment strategy generation model based on multi-agent reinforcement learning, and deploy the microservices to be deployed to the corresponding edge servers according to the deployment strategy;

[0013] The deployment strategy generation model is modeled using a Markov decision process and is defined by a tuple (S, A, R);

[0014] State space S: s t ∈S represents the state at time t;

[0015]

[0016] in, represents the state observed by agent n at time t, g represents the number of agents, represents the state of the state index node i observed by agent n at time t, C i(t) represents the list of child nodes accessible to the state index node i at time t, D i (t) represents the set of microservices deployed by the state index node i and its child nodes at time t, G i (t) represents the resource status of the edge server under the state index node i at time t, represents the position code of the index node at time t; X(t) represents the characteristics of the microservice to be deployed at time t, and L(t) represents the location information of the user at time t;

[0017] Action space A: a t ∈A represents the agent in state s t the actions taken next;

[0018]

[0019] Among them, r j (t) is the child index node, is the set of child index nodes of state index node i;

[0020] Reward function R: r t Indicates that the agent performs action a t Rewards received;

[0021]

[0022] Among them, K is the number of non-leaf nodes at the same level, T i is the number of microservices deployed on all edge servers under state index node i, d i,j It is the state index node i in all edge servers connected to microservice w j The number of direct dependencies, b j Is with microservices w j The total number of microservices with direct dependencies, e k It is the same as the microservice w on all edge servers under other nodes at the same level j The latency of dependent microservices, t max is the maximum delay allowed by the system, δ i,j It is a microservice j The minimum resource utilization deployed under state index node i, is the average resource utilization of all edge servers under state index node i, λ1 and λ2 are weight parameters, and c is a constant.

[0023] Furthermore, the deployment strategy generation model adopts the QMIX algorithm; the agent observes the current state, inputs the current state into the agent network, generates the Q value of all actions in the current state, selects an action to act on the environment and generate a reward; the maximum Q value generated by the agent network of group A is input into the hybrid network to generate the joint action value of all actions; the action corresponding to the maximum Q value generated by the agent network of group A is input into the central hybrid network, and the central hybrid network generates the optimal joint action value according to the current state observed by the agent network of group B, then the loss function of the agent network and the hybrid network is as follows:

[0024]

[0025] Where ω is the preference vector, is the joint action value, is the optimal joint action value, τ t , τ t+1 is the historical state observed by all agents at time t and t+1, u is the joint action of all agents; y b is the expected reward, δ(s,u) is the weight function, s t+1 is the state at time t+1, γ is the discount factor, and B is the number of experience replay groups.

[0026] Furthermore, the proxy network includes a VSN network, an LSTM network, a multi-head attention network and a multi-layer perceptron connected in series in sequence; the GRN network of the VSN network is replaced by a densely connected network. In the densely connected network, the input passes through the first dense layer and the ELU activation function, and then is processed by two parallel dense layers. The outputs of the two dense layers pass through gated linear units and are residually connected with the output of the first dense layer. The features obtained by the residual connection are normalized to obtain the output of the densely connected network.

[0027] Furthermore, the hybrid network and the central hybrid network both include multiple multilayer perceptrons, the inputs pass through multiple multilayer perceptrons respectively, the output features of these multilayer perceptrons are weighted fused, and the features obtained by weighted fusion pass through a multilayer perceptron to obtain the network output.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. Through multi-agent reinforcement learning, the dependencies of microservices and the resource status of edge servers are simultaneously optimized, and the deployment location of microservices is intelligently selected. This effectively reduces the waiting time for data transmission between microservices and the queuing calculation time of microservices on edge servers, thereby significantly reducing time latency.

[0030] 2. The present invention takes into account the resource status of the edge server and the delay of microservices, and trains the deployment strategy generation model so that the deployment strategy generation model can simultaneously optimize the resource status of the edge server and the overall response delay, avoiding the concentration of too many microservices on a few edge servers, and promoting a more balanced distribution of microservices on different edge servers, which not only reduces the overload risk of a single edge server, but also improves the resource utilization of the entire edge computing infrastructure.

[0031] 3. The present invention alleviates the problem of mutual influence between agents. The cloud-layer agent distributes the microservices to be deployed to each region, and the agents in the region deploy the microservices to the corresponding edge servers. This controls the impact of microservice dependencies on the agent deployment results when distributing microservices, enabling the multi-agent system to better adapt to the complex scenarios of a large number of service requests and numerous edge nodes in the cloud-edge fusion environment, and alleviates the problem that the optimal action of an agent depends on the optimal action of other agents at the same time. Agents can make decisions and collaborate more effectively, and the system as a whole shows greater robustness and scalability in the face of dynamic changes and high loads. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a framework diagram of the cloud-edge fusion environment;

[0033] Figure 2 This is a regional division diagram of the edge server in the embodiment;

[0034] Figure 3 Schematic diagram of the structure of the spatial index tree in the embodiment;

[0035] Figure 4 The following is a graph showing the change in load utility and the number of microservices for different methods;

[0036] Figure 5 The following is a graph showing the changes in the average response latency of different methods and the number of deployed microservices. DETAILED DESCRIPTION

[0037] Specific embodiments are given below in conjunction with the accompanying drawings. The specific embodiments are only used to introduce the technical solutions of the present invention in detail and are not intended to limit the scope of protection of the present application.

[0038] like Figure 1As shown, the cloud-edge fusion environment of the present invention includes a cloud layer, an edge layer, and a user layer; the user layer includes a large number of continuously mobile users, who initiate microservice requests to the edge layer through 5G / 6G wireless transmission technology, and the user location is modeled based on its nearest edge server; the edge layer includes a large number of edge servers, which usually have more communication and computing resources. The edge layer is divided into multiple regions based on the distance between edge servers, and an agent is deployed on the edge server with the best performance in each region. The agent collects the status of all edge servers in the region (computing / communication / storage resources, deployed microservices) through optical fiber Ethernet, and dispatches the services to be deployed assigned by the cloud layer to an edge server in the region. The cloud layer deploys agents, and the cloud layer collects information and user data from all edge servers in the edge layer through optical fiber Ethernet. The cloud layer agent coordinates the cross-regional microservice distribution based on user needs and edge server status, distributes microservices to each region, and finally the edge layer agent completes the deployment.

[0039] The present invention provides a cloud-edge fusion environment microservice deployment method based on multi-agent reinforcement learning, comprising the following steps:

[0040] Step 1: Divide the edge layer into multiple regions, each containing multiple edge servers; construct a spatial index tree based on the distance between edge servers;

[0041] (1.1) The root node of the spatial index tree stores the information of all edge servers. The number of child nodes of each node in the spatial index tree is between e0 and E0.

[0042] (1.2) Randomly select k (e0≤k≤E0) edge servers from all edge servers as the initial clustering centers, and use the K-means algorithm based on the Euclidean distance of edge servers to form k device clusters, which are respectively used as k child nodes of the root node;

[0043] There are three situations for child nodes: (a) If the number of devices contained in the node is less than e0, then merge this node with the adjacent node and re-judge the node situation; (b) If the number of devices contained in the node is between e0 and E0, then set this node as a leaf node; (c) If the number of devices contained in the node is greater than E0, then re-divide the device cluster corresponding to this node using the K-means algorithm until the number of devices contained in the node is between e0 and E0.

[0044] (1.3) After the spatial index tree is built, an inverted list needs to be assigned to each node. The inverted list of the leaf node is used to store the resource information corresponding to the edge server. The inverted list of the non-leaf node is the union of the inverted lists of all its child nodes, which is used to store the set of microservices deployed by its child nodes.

[0045] Step 2: Using microservices as the smallest unit, generate the importance of microservices based on the dependencies between them and their historical requests.

[0046] Given any user request microservice call chain l k and l h ; Among them, call chain l k Contains U microservices, F k Call chain l k The importance matrix of size (U,U), call chain l k The number of users is t; the call chain l h Contains U microservices, F h Call chain l h The importance matrix of size (U,U), call chain l h The number of users is r; assuming the call chain l k Contains microservices u and w v , call chain l h Contains microservices p and w q , u,v,p,q=1,2,...,U, then:

[0047] If the microservice w u With w v If there is data transmission between them, the call chain l k Importance matrix F k The values ​​of elements (u,v) and (v,u) in the same time increase by t; if the microservice w u With w v If no data is transmitted between them, the call chain l k Importance matrix F k The values ​​of elements (u,v) and (v,u) are 0; if the microservice w p and w q If there is data transmission between them, the call chain l h Importance matrix F h The values ​​of elements (p,q) and (q,p) increase by r at the same time; if microservice w p and w q If no data is transmitted between them, the call chain l h Importance matrix F h The values ​​of elements (p,q) and (q,p) are 0;

[0048] Call chain l i and l k The importance matrix F = F k +F hIt reflects the calling relationship between microservices and the information of users using microservices. The importance of each microservice is the sum of the elements of the corresponding row in the importance matrix F. For example, microservice w u The importance of is the sum of the elements in the u-th row of the importance matrix F; m microservices with high importance are deployed to the edge server.

[0049] Step 3: Build a deployment strategy generation model based on multi-agent reinforcement learning, and deploy multiple important microservices to edge servers according to the deployment strategy;

[0050] The deployment strategy generation model is modeled using a Markov decision process and is usually defined by a tuple (S, A, R);

[0051] State space S,s t ∈S represents the state at time t, that is, the environmental information observed by all agents at time t;

[0052]

[0053]

[0054] in, represents the state observed by agent n at time t, g represents the number of agents, represents the state of index node i (called state index node) observed by agent n at time t, C i (t) represents the list of child nodes accessible to the state index node i at time t, D i (t) represents the set of microservices deployed by the state index node i and its child nodes at time t, G i (t) represents the resource status of the edge server under the state index node i at time t (including the average load of the edge server, remaining resources (CPU / memory) and transmission rate), represents the location code of the index node at time t; X(t) represents the characteristics of the microservice to be deployed at time t, including resources (CPU / memory) and dependencies; L(t) represents the location information of the user at time t;

[0055] Action space A: a t ∈A represents the agent in state s t The action taken by the agent according to the state Select a child index node R from the state index node i j (t), expressed as:

[0056]

[0057] in, is the set of child index nodes of state index node i;

[0058] Reward function R: r t =R(s t ,a t ) indicates that the agent performs action a t The reward obtained after is defined according to the edge server load status and the possible delay of the deployed microservices, and is expressed as:

[0059]

[0060] Among them, K is the number of non-leaf nodes at the same level, T i is the number of microservices deployed on all edge servers under state index node i, d i,j It is the state index node i in all edge servers connected to microservice w j The number of direct dependencies, b j Is with microservices w j The total number of microservices with direct dependencies, e k It is the same as the microservice w on all edge servers under other nodes at the same level j The latency of dependent microservices; max Is the maximum delay allowed by the system. If the delay caused by the transmission distance between microservices is higher than the maximum delay, the microservice deployment is considered to have failed. i,j It is a microservice j The minimum resource utilization deployed under state index node i, is the average resource utilization of all edge servers under state index node i, λ1 and λ2 are weight parameters, and c is a constant as an additional penalty term.

[0061] The QMIX reinforcement learning model is used as the deployment strategy generation model, and the agent n observes the current state Will state Input into the agent network to generate state The Q value of all actions Used to guide action selection; select one of the actions to act on the environment and generate a reward r t , then the state s t Updated to s t+1 ; The agent continuously interacts with the index environment to generate experience tuples (s t ,a t ,r t ,s t+1 ) is used to train the agent network. The agents are divided into two groups, A and B, and the corresponding agent networks are also divided into two groups. The maximum Q value generated by the agent network in group A (the action corresponding to the maximum Q value is recorded as a t max ) is input into the hybrid network to generate the joint action value of all actions The action a corresponding to the maximum Q value generated by the agent network of group A t max Input into the central mixing network, the central mixing network observes the current state s according to the B group agent network t , generating the optimal joint action value These two joint action values ​​are used to calculate the loss of the proxy network and the hybrid network. The loss function of the proxy network and the hybrid network is as follows:

[0062]

[0063] Where ω is the preference vector, which represents the preference for different goals; is the joint action value, is the optimal joint action value, τ t is the historical state observed by all agents at time t, u is the joint action of all agents; y b is the expected reward, and δ(s,u) is the weight function, defined as follows:

[0064]

[0065] Where γ is the discount factor;

[0066] Starting from the root node of the spatial index tree, when selecting the child nodes of the current node, the optimal child node is greedily selected according to the Q value generated by the proxy network until a leaf node is found. The corresponding edge server set or edge server is the optimal region or edge server for deploying microservices.

[0067] The agent network includes a variable selection network (VSN), a long short-term memory recurrent neural network (LSTM), a multi-head attention network and a multilayer perceptron (MLP). The GRN network of the VSN network is replaced by a densely connected network. In the densely connected network, the state After passing through the first dense layer and the ELU activation function, it is processed by two parallel dense layers respectively. The outputs of the two dense layers are processed by gated linear units and then residually connected with the output of the first dense layer. The features obtained by the residual connection are normalized to obtain the output of the dense connection network; the output of the VSN network is further processed by LSTM and a multi-head attention network, and finally the Q value of all actions is generated by MLP.

[0068] The hybrid network and the central hybrid network both include multiple multilayer perceptrons, and the inputs pass through multiple multilayer perceptrons respectively. The output features of these multilayer perceptrons are weighted fused, and the features obtained by weighted fusion pass through a multilayer perceptron to obtain the network output.

[0069] Example

[0070] The EUA dataset and the tracking dataset provided by Alibaba were used to verify the microservice deployment method of this application. The EUA dataset selected the latitude and longitude information of 125 edge servers and the location information of 817 users. Each user request consists of a series of interdependent microservices. 30 applications were selected from Alibaba's tracking dataset. All applications contain approximately 200 microservices, and the number of microservice dependencies for each application ranges from 100 to 200. For example, when application A contains two microservices and microservice 1 depends on microservice 2, the dependency number of application A is 1. The microservice dependencies in the application are derived from Alibaba data tracking.

[0071] The cloud-edge fusion environment microservice deployment method based on multi-agent reinforcement learning in this embodiment includes the following steps:

[0072] Step 1: Divide the edge layer into multiple regions, each containing multiple edge servers; construct a spatial index tree based on the distance between edge servers;

[0073] like Figure 2 As shown in Figure 1, region N0 covers all edge servers, and the number of child nodes of each node in the spatial index tree is specified to be between 2 and 4. According to the K-means algorithm, two edge servers are first selected and clustered to form two regions N1 and N2. Since the number of devices in these two regions is greater than 4, the K-means algorithm is recursively used in these two regions to cluster again, and finally regions N3, N4, N5 and N6 are formed. The spatial index tree is shown in Figure 1. Figure 3 After the spatial index tree is built, an inverted list (InvFile) needs to be assigned to each node. The InvFile of a leaf node summarizes the resource information corresponding to the edge servers it contains. The InvFile of a non-leaf node is the summarized InvFile information of its child nodes, including the microservices deployed by its child nodes.

[0074] Step 2: Using microservices as the smallest unit, generate the importance of microservices based on the dependencies between them and their historical requests.

[0075] Based on 30 applications, generate the importance matrix F1, F2, ..., F of the call chain in each application 30 , and the importance of microservices;

[0076] Assume that application 1 is Application1 = {w1, w2, w3, ...}, each application consists of multiple microservices, and there are dependencies between microservices; for microservice w u and w v , if they have dependencies, when there are f users’ requests involving microservice w u and w v When , the values ​​of the elements (u,v) and (v,u) in the call chain importance matrix are f, and the same applies to other applications. The total call chain importance matrix F=F1+F2+...+F 30 The importance of each microservice is the sum of the elements of the corresponding row of all call chain importance matrices. For example, microservice w u The importance of is the sum of the elements in the u-th row of the importance matrix of all call chains. The top 60 microservices with the greatest importance are selected as the microservices to be deployed.

[0077] Step 3: Build a deployment strategy generation model based on multi-agent reinforcement learning, and deploy multiple important microservices to edge servers according to the deployment strategy;

[0078] (1) For the microservice to be deployed, agent n observes the current state Feed it into the proxy network to get the status The Q value of all actions Used to guide action selection; use ∈-greedy strategy to select an action a t ;

[0079]

[0080] in, is state s t The set of optional actions under ,∈ is used to control the trade-off between exploring new child nodes and utilizing known child nodes in action selection, Indicates the random number generation operation. When the random number is greater than ∈, it takes 1. When the random number is less than ∈, it takes Randomly select an action in

[0081] (2) Calculate the agent's execution action a according to formula (5) t The reward obtained; the edge server environment changes the observation state from s t Updated to s t+1 , the agent continuously interacts with the environment and collects experience tuples into the experience replay pool middle;

[0082] (3) During QMIX training, Sample a small batch of experience tuples to calculate the loss function

[0083] (4) Update the agent network, hybrid network and central hybrid network through the Adam optimizer, and soft update the agent network of group B by copying the parameters of the agent network of group A; when QMIX converges, obtain the convergence parameter θ of the deployment strategy * ;

[0084] (5) According to θ * The calculated Q value starts from the root node and greedily selects the optimal child index node during the search process until the optimal leaf node is found. The edge server contained in the optimal leaf node is the edge server used to deploy the corresponding microservice. For the cloud agent, it will start from the N0 node to select the optimal child index node until the four sub-areas N3, N4, N5, and N6 are selected and then return. The agent at the edge server layer will continue to select edge servers from the N3, N4, N5, and N6 sub-areas. When the edge server is selected, the deployment of a microservice is completed.

[0085] Using existing method RSDQL [1] LDLS [2] 、MobiEdge [3] Compared with the method of the present invention (denoted as TS-QMIX), Figure 4 、 5 The following table compares the load utility and average response latency of different methods. RSDQL considers the complex dependencies between microservices and the impact of aggregate server load on microservice execution performance. LDLS considers the combined impact of microservice dependencies on transmission latency and execution time. MobiEdge considers user request patterns.

[0086] The load utility measurement formula is: Y is the number of microservices deployed on all edge servers, δ j It is a microservice j Resource utilization rate deployed on the corresponding edge server, δ avg is the average resource utilization of all edge servers, c is the additional penalty term, when microservice s j When the resource utilization of the corresponding edge server exceeds 80%, a penalty item will be added to indicate the current system load imbalance.

[0087] Depend on Figure 4It can be seen that the total load utility of all methods increases with the increase in the number of microservices. This is because the number of edge servers remains unchanged, but the number of deployed microservices increases, resulting in an increase in the total load. Among all the methods, the load utility of the method of the present invention is significantly lower than that of the other three methods, achieving the effect of load balancing. The other three methods deploy too many microservices on the same edge server, resulting in a decrease in overall load utility. TS-QMIX performs better when processing more microservices and can effectively maintain system load balancing. On the one hand, the spatial index tree not only stores resource information, but also updates the load status of the edge server in real time. When more than 60 microservices are deployed, the load utility of TS-QMIX is significantly higher than that of other methods. This is because the agent can pay attention to the performance of latency and load at the same time through the hybrid network, actively avoid edge servers with overload risks and those that may cause high latency, and deploy microservices to low-load nodes, thereby avoiding performance degradation caused by local resource contention.

[0088] from Figure 5 It can be seen that as the number of deployed microservices increases, the average response delay shows an overall downward trend (mainly due to the reduction in transmission delay). The method of the present invention presents lower delay in most scenarios, especially when the scale of microservices is large, indicating that it has the ability to handle large-scale microservice deployment. MobiEdge and LDLS have a higher response delay due to the suboptimal edge layer deployment position. In order to ensure transmission delay, load balancing is sacrificed, resulting in increased computing delay, and ultimately a higher overall delay with the same number of microservices. When more than 40 microservices are deployed, the response delay changes slowly, which also proves that the microservice importance measurement mechanism proposed in the present invention is effective and correct. In summary, the method of the present invention exhibits excellent performance in terms of average response delay and load balancing.

[0089] The sources of the above existing methods are as follows:

[0090] [1]Lv W, Wang Q, Yang P, et al.Microservice deployment in edge computingbased on deep Q learning[J]. IEEE Transactions on Parallel and DistributedSystems, 2022, 33(11):2968-2978.

[0091] [2]Tang Z, Lou J, Jia W.Layer dependency-aware learning scheduling algorithms for containers in mobile edge computing[J]. IEEE Transactions onMobile Computing, 2022, 22(6):3444-3459.

[0092] [3]Wu T, Fan

[0093] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A cloud-edge fusion environment microservice deployment method based on multi-agent reinforcement learning, characterized in that: The following steps are involved: Step 1: Divide the edge layer into multiple zones, each containing multiple edge servers. Build a spatial index tree based on the distance between edge servers; Step 2: Generate the importance of microservices based on the dependencies between microservices and the historical requests of microservices; Select multiple microservices of high importance as the microservices to be deployed; The call chain of the user requesting a microservice includes multiple microservices. Assume that the call chain l k Contains microservices u and w v , call chain l h Contains microservices p and w q ; If the microservice w u With w v If there is data transmission between them, the call chain l k The values ​​of elements (u,v) and (v,u) in the importance matrix increase at the same time; if the microservice w u With w v If no data is transmitted between them, the call chain l k The values ​​of elements (u,v) and (v,u) in the importance matrix are 0; if the microservice w p and w q If there is data transmission between them, the call chain l h The values ​​of elements (p,q) and (q,p) in the importance matrix increase by r at the same time; if microservice w p and w q If no data is transmitted between them, the call chain l h The values ​​of elements (p,q) and (q,p) in the importance matrix are 0; where t is the call chain l k The number of users, r is the call chain l h the number of users; The sum of the importance matrices of all call chains is the microservice importance matrix, and the importance of a microservice is the sum of the elements in the corresponding row of the microservice importance matrix; Step 3: Build a deployment strategy generation model based on multi-agent reinforcement learning, and deploy the microservices to be deployed to the corresponding edge servers according to the deployment strategy; The deployment strategy generation model is modeled using a Markov decision process and is defined by a tuple (S, A, R); State space S: s t ∈S represents the state at time t; in, represents the state observed by agent n at time t, g represents the number of agents, represents the state of the state index node i observed by agent n at time t, C i (t) represents the list of child nodes accessible to the state index node i at time t, D i (t) represents the set of microservices deployed by the state index node i and its child nodes at time t, G i (t) represents the resource status of the edge server under the state index node i at time t, represents the position code of the index node at time t; X(t) represents the characteristics of the microservice to be deployed at time t, and L(t) represents the location information of the user at time t; Action space A: a t ∈A represents the agent in state s t the actions taken next; Among them, r j (t) is the child index node, is the set of child index nodes of state index node i; Reward function R: r t Indicates that the agent performs action a t Rewards received; Among them, K is the number of non-leaf nodes at the same level, T i is the number of microservices deployed on all edge servers under state index node i, d i,j It is the state index node i in all edge servers connected to microservice w j The number of direct dependencies, b j Is with microservices w j The total number of microservices with direct dependencies, e k It is the same as the microservice w on all edge servers under other nodes at the same level j The latency of dependent microservices, t max is the maximum delay allowed by the system, δ i,j It is a microservice j The minimum resource utilization deployed under state index node i, is the average resource utilization of all edge servers under state index node i, λ1 and λ2 are weight parameters, and c is a constant.

2. The cloud-edge fusion environment microservice deployment method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The deployment strategy generation model adopts the QMIX algorithm; The agent observes the current state, inputs the current state into the agent network, generates the Q value of all actions in the current state, selects an action to act on the environment and generate a reward; the maximum Q value generated by the agent network of group A is input into the hybrid network to generate the joint action value of all actions; the action corresponding to the maximum Q value generated by the agent network of group A is input into the central hybrid network, and the central hybrid network generates the optimal joint action value based on the current state observed by the agent network of group B. Then the loss function of the agent network and the hybrid network is as follows: Where ω is the preference vector, is the joint action value, is the optimal joint action value, τ t , τ t+1 is the historical state observed by all agents at time t and t+1, u is the joint action of all agents; y b is the expected reward, δ(s,u) is the weight function, s t+1 is the state at time t+1, γ is the discount factor, and B is the number of experience replay groups.

3. The cloud-edge fusion environment microservice deployment method based on multi-agent reinforcement learning according to claim 2 is characterized in that: The proxy network includes a VSN network, an LSTM network, a multi-head attention and a multi-layer perceptron connected in series in sequence; the GRN network of the VSN network is replaced by a dense connection network. In the dense connection network, the input passes through the first dense layer and the ELU activation function, and then is processed by two parallel dense layers. The outputs of the two dense layers pass through gated linear units and are residually connected with the output of the first dense layer. The features obtained by the residual connection are normalized to obtain the output of the dense connection network.

4. The cloud-edge fusion environment microservice deployment method based on multi-agent reinforcement learning according to claim 2 or 3 is characterized in that: The hybrid network and the central hybrid network both include multiple multilayer perceptrons, and the inputs pass through multiple multilayer perceptrons respectively. The output features of these multilayer perceptrons are weighted fused, and the features obtained by weighted fusion pass through a multilayer perceptron to obtain the network output.

Images