Micro-service automatic capacity expansion and contraction method based on graph neural network and reinforcement learning

Abstract

The invention discloses a micro-service automatic capacity expansion and contraction method based on a graph neural network and reinforcement learning. The method mainly comprises the steps of data acquisition and preprocessing, GNN model construction, online training mechanism implementation and continuous optimization of an automatic capacity expansion and contraction strategy. According to the method, the problem that a traditional automatic capacity expansion and contraction method is difficult to deal with complex micro-service architecture and dynamic load changes is solved, and challenges such as inaccurate performance bottleneck identification and resource waste caused by complex dependency relationships among micro-services in practical application are solved. According to the method, the resource utilization rate of the system is improved, unnecessary Pod overhead is reduced, the expandability and flexibility of the system are remarkably enhanced, the aim of maximizing the resource efficiency on the premise that the service quality is met is finally achieved, and powerful support is provided for intelligent operation and maintenance under the micro-service architecture.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology, and in particular to a method for realizing automatic scaling of microservices based on a graph neural network (GNN) and reinforcement learning (RL) BACKGROUND In modern cloud computing environments, as the business scale continues to expand and user demands become increasingly diverse, traditional monolithic architecture is gradually being replaced by flexible and scalable microservice architecture. Microservices achieve higher development efficiency, deployment flexibility, and system fault tolerance by splitting complex applications into multiple loosely coupled service modules. However, this architecture also presents new challenges, particularly in terms of resource management and elastic scaling. Due to the complex calling relationships and dependency chains between microservices, performance fluctuations in a single service can propagate through inter-service calls to the entire system, affecting overall Quality of Service (QoS) and Quality of Experience (QoE). At the same time, workloads in cloud environments are highly dynamic and uncertain, such as during e-commerce promotions, holiday traffic peaks, and other scenarios where sudden traffic can cause bottlenecks in some services, leading to increased latency, request timeouts, and even service crashes.

[0002] In modern cloud computing environments, as microservice architecture is widely used, efficient resource management and automatic scaling have become key challenges. Traditional methods mainly rely on static rules or simple threshold triggering mechanisms to adjust the number of service instances, such as making decisions based on changes in CPU usage or memory consumption. However, these methods often have response delays and lack a global perspective on the complex dependencies between various microservices in the entire system, leading to decreased service quality or wasted resources. In addition, overly simplified models are difficult to accurately predict future workload changes, and poorly designed reward functions can lead to excessive expansion of service instances, further exacerbating low resource utilization efficiency.

[0003] To address these issues, researchers have begun exploring methods that combine graph neural networks (GNN) and reinforcement learning (RL) to improve the automatic scaling strategy of microservices. GNN is used to capture the complex dependencies between microservices and make accurate latency predictions, while RL dynamically adjusts the service scale based on environmental feedback to optimize long-term performance. Although this approach shows great potential, it still faces challenges such as data fusion, trade-offs between model complexity and accuracy, and reward function design in practical applications. SUMMARY

[0004] The technical problem to be solved by the embodiments of the present application is to provide a method for realizing automatic scaling of microservices based on a graph neural network and reinforcement learning, which can effectively improve the overall system performance and resource utilization.

[0005] To address the aforementioned technical problems, this invention provides a method for automatic scaling of microservices based on graph neural networks and reinforcement learning, comprising the following steps: S1: Perform environmental preparation and data collection configuration, acquire historical data or real-time monitoring data within a specified time period, clean and store the collected raw data; S2: Construct the graph neural network input and embed it into the output of the resource-side GNN model: The GNN model output embedding on the system side: ,in, It is the number of layers in the GNN. It is the output embedding dimension. , It represents the total number of services; by fusing embeddings from the resource side and the system side through the encoder network, a global feature vector is generated. ; S3: Define the reinforcement learning environment, including the state space, action space, and reward function; input the global feature vector into the proximal policy algorithm. And during operation, it continuously optimizes decision-making strategies based on ongoing interactions with the actual microservice environment.

[0006] Furthermore, the data acquisition process in S1 includes inputting call graph data on both the system side and the resource side.

[0007] Furthermore, the graph data is formed through the following steps: S11: Identify each microservice deployed in the Kubernetes cluster and map each identified microservice instance to an independent node in the graph; S12: Extract metrics related to user experience and throughput for each microservice node to form a resource-side feature vector; S13: Extract indicators related to internal resource consumption and operating status for each microservice node to form a system-side feature vector; S14: Use the inter-service call chain captured by Istio service mesh on the resource side to construct directed edges; S15: On the system side, use service topology information captured based on the Istio service mesh or actual observed inter-service dependencies to construct undirected edges.

[0008] Furthermore, the step of constructing the graph neural network input in S2 includes: S211: Based on the collected data, construct an input call graph and set the nodes... Sum of edges Composition of resource side map , composed of node set Sum of edges Side diagram of the system ; S212: Resource-side input graph construction: For each microservice node, select metrics directly related to user experience as its node features. The user request chain between services is abstracted as directed edges, and the characteristics of the edges are... Reflecting call volume and path latency, an adjacency matrix is ​​formed on the resource side. ; S213: System-side input graph construction: For each microservice node, select indicators related to internal resource consumption and operating status as its node features. Undirected edges are constructed based on the service dependency topology of Istio to form an adjacency matrix on the system side. ; S214: Input node feature matrix in resource side graph Edge feature matrix Adjacency matrix ; S215: Input node feature matrix in system side graph Adjacency matrix ; S216: In the resource side map and system side diagram Once constructed, they are fed as independent inputs into their respective GNN models: In the resource-side GNN model: process the resource-side graph and output the node embeddings from the resource side. Node embedding It captures the complex characteristics of microservices in terms of user-oriented performance. In the system-side GNN model: process the system-side graph and output the system-side node embeddings. Node embedding It captures the complex representation of microservices in terms of internal resources and dependencies.

[0009] Furthermore, step S2 also includes: S221: The GNN models on the resource side and the system side process their respective graph inputs independently; S222: For any GNN model, the node feature representation H of the (l+1)th layer. (l+1) It is achieved by aggregating the node features H of the l-th layer (l) Information about it and its neighbors, and through the weight matrix W (l) and bias term B (l) Perform a transformation, then obtain the result through the activation function σ, using the formula... This describes the interlayer propagation process; S223: The input to the model is the original node feature matrix. For a resource-side GNN, For the system-side GNN, ; S224: After L layers of graph convolution calculation, the feature representation obtained by each node in the last layer L is the final embedding of that node; S225: Output of the resource-side GNN : is the node feature matrix of the last layer L of the resource-side GNN model. Its dimensions are ,in It refers to the number of microservice nodes. It is the dimension of the output embedding. Each line in the table represents a compressed, high-dimensional representation of a microservice on the resource side. S226: Output of the system-side GNN : is the feature matrix of the nodes in the last layer L of the system-side GNN model. Its dimensions are ,in It refers to the number of microservice nodes. It is the dimension of the output embedding. Each line in the table represents a compressed, high-dimensional representation of a microservice on the system side.

[0010] Furthermore, in S2, the encoder network uses a cross-diffusion attention mechanism and low-rank tensors to fuse resource-side and system-side embeddings to generate a global feature vector. .

[0011] Furthermore, the processing method of the cross-diffusion attention mechanism includes: Embed the resource side and system-side embedding Interacting through attention mechanisms:

[0012] in It is a projection matrix of requests, keys, and values; The low-rank tensor fusion processing method includes: Integrate the embeddings from the resource side and the system side: ; Generate global feature vectors The methods include: Map the fused embeddings to the global feature vector: ,in and These are the weights and biases of the fully connected layer.

[0013] Furthermore, in S3: The state space is used to represent the set of environmental information that the agent can perceive at each decision moment, and it uses the global embedding vector output by the GNN encoder. ; The action space defines the set of discrete operations that the agent can choose to execute at each decision step, and uses the following action constraints to ensure the stable operation of the service and the rational use of resources: The execution of any scaling up or down operation must ensure that the target microservice's Pod count is within a certain range. Maintain at the preset minimum boundary and maximum boundary Within; The reward function Used to evaluate the agent's state Execute action Later transitioned to a new state The immediate feedback signal obtained at that time uses the following comprehensive reward function structure:

[0014] in: This indicates the agent's decision-making time. Due to the execution of actions And observe the scalar reward value obtained after subsequent environmental feedback; The reward component is related to the achievement of service level targets; The reward component is related to the overall Pod resource consumption of the system; This is a hyperparameter used to weigh the importance of satisfying SLOs against the importance of resource conservation.

[0015] Furthermore, the application architecture of the near-end strategy algorithm in S3 includes: Policy Network: Input is the global state embedding vector generated by the GNN encoder. Based on the input state, the decision outputs scaling actions for the target microservice. ; Value network: Input global state embedding vector Assess the current status Its long-term value is used to assist in the optimization of policy networks.

[0016] Furthermore, step S3 also includes: S31: Agent parameter initialization: During initial deployment, the policy network of the reinforcement learning agent is initialized. and value network parameters and Perform random initialization; S32: Use a continuous perception-decision-execution-learning cycle, which includes steps of state perception and decision making, action execution and environmental feedback, and experience storage and parameter updating. The steps of state perception and decision-making include: At the preset decision-making time point First, the global state embedding vector of the current microservice system is obtained from the GNN encoder. = ; Embed the global state vector Input into the latest policy network The policy network outputs a list of all possible scaling actions. To balance exploration and utilization, random sampling is performed based on the probability distribution to select actions. ; The steps for action execution and environmental feedback include: Selected expansion / contraction action It is translated into a specific Kubernetes API call; in the execution of actions After a period of observation, the environment evolves to a new state, and the next decision-making observation point is reached. Actual performance metrics and resource usage are collected and used to calculate immediate rewards. The GNN encoder generates new global state embeddings. ; The steps of experience storage and parameter updating include: Experience tuples generated by interaction Stored in an experience replay buffer, which is used to store the interaction history within a recent period; Periodically sample a batch of empirical data from the experience replay buffer and optimize the process according to the near-end policy optimization algorithm to adjust the policy network parameters. and value network parameters Perform gradient updates.

[0017] Implementing the embodiments of this invention has the following beneficial effects: This invention overcomes the problem that traditional automatic scaling methods are unable to cope with complex microservice architectures and dynamic load changes, and solves the challenges of inaccurate performance bottleneck identification and resource waste caused by complex dependencies between microservices in practical applications. Experimental results show that this innovative combination not only improves system resource utilization and reduces unnecessary Pod overhead, but also significantly enhances the system's scalability and flexibility. Ultimately, it achieves the goal of maximizing resource efficiency while meeting service quality requirements, providing strong support for intelligent operation and maintenance under microservice architectures. Attached Figure Description

[0018] Figure 1 This is a flowchart of a method for automatically scaling up and down microservices based on graph neural networks (GNN) and reinforcement learning (RL). Figure 2 yes Figure 1 Detailed flowchart of the decision-making module of the RL agent. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

[0020] An embodiment of the present invention provides an automatic scaling method for microservices based on graph neural networks and reinforcement learning, which mainly includes the following steps.

[0021] S1: Data acquisition and preprocessing; S2: GNN model construction; S3: Implements an online training mechanism and continuously optimizes the automatic scaling strategy.

[0022] In S1, the following specific steps are followed for implementation.

[0023] S11: Environmental Preparation Deploy the Istio service mesh in the Kubernetes cluster to ensure that all target microservices are connected to the mesh.

[0024] Configure Prometheus to capture the required monitoring metrics from Istio. This includes, but is not limited to, key performance metrics such as P95 response time, QPS (queries per second), CPU utilization, memory usage, and the number of Pods.

[0025] S12: Data Acquisition Configuration: Define a standardized data collection template that covers information from both the system and resource sides.

[0026] The system-side input call graph data consists of nodes representing microservices, whose characteristics include CPU, memory, number of Pods, and cold start time; and edges representing the dependencies between microservices.

[0027] Resource-side input call graph data: Nodes are also microservices, but their characteristics focus more on user experience, such as P95 response time and QPS; edges take into account traffic volume and latency.

[0028] The data in the above graph was obtained by processing and transforming raw data collected in real time from the Istio service mesh and Prometheus monitoring tools in the Kubernetes cluster. The specific processing steps are as follows: S121: Node Identification and Mapping: Identify each microservice deployed in the Kubernetes cluster. Each identified microservice instance is mapped to an independent node in the graph.

[0029] S122: Feature Extraction Resource-side node features: For each microservice node, metrics related to user experience and throughput, such as P95 response time (P95 Latency) and QPS (queries per second), are extracted from Prometheus. These metrics constitute the resource-side feature vector of the node.

[0030] System-side node features: For each microservice node, metrics related to internal resource consumption and running status, such as CPU utilization, memory usage, and number of Pod instances, are extracted from Prometheus. These metrics constitute the system-side feature vector of the node.

[0031] S123: Edge Construction and Feature Assignment: Resource-side edge construction: Directed edges are built based on the inter-service call chains captured by the Istio service mesh (e.g., obtained through service tracing). If microservice A calls microservice B, a directed edge is built from A to B. The characteristics of this edge reflect the traffic (e.g., QPS) and latency information on that call chain. This information constitutes the edge characteristics and adjacency relationships on the resource side.

[0032] System-side edge construction: Undirected edges are constructed based on the service topology information provided by Istio or the actual observed service dependencies. An undirected edge is constructed between any two microservice nodes that have direct or indirect dependencies (e.g., scaling up or down one service may affect another), representing their association at the system level. Since the system side focuses more on internal state, these edges typically do not carry additional intrinsic characteristics; their existence itself represents a dependency and is used to construct adjacency relationships.

[0033] Through the above steps, the raw, scattered monitoring and tracing data is structured into nodes (microservices) and edges (dependencies), each with its own characteristics, thus forming graph data that can be input into the GNN model.

[0034] In the above, the system-side specifically refers to the focus on the operational status and resource consumption within the microservice cluster. It reflects the internal health of each microservice instance and its resource-level coupling with other microservices. Its core lies in the observation of system resource usage such as CPU, memory, and the number of Pods.

[0035] Post-input processing: System-side graph data (node ​​features and adjacency matrix) are fed as independent inputs into a separate GNN model (system-side GNN model). This GNN model is responsible for learning and extracting deep representations (embeddings) of microservices in terms of system resources and their internal dependencies. ).

[0036] Resource-side: Specifically, the resource side focuses on the service quality and user experience provided by microservices. It reflects the performance of microservices in handling user requests, as well as the latency and throughput of requests as they flow through different services. Its core lies in observing user-oriented performance metrics such as P95 response time and QPS.

[0037] Post-input processing: Resource-side graph data (node ​​features, edge features, and adjacency matrix) is fed as independent input into another independent GNN model (resource-side GNN model). This GNN model is responsible for learning and extracting deep representations (embeddings) of microservices in terms of external performance and request dependencies. ).

[0038] S13: Data Collection Mechanism Use PromQL, Prometheus' built-in query language, to write specific queries to retrieve historical or real-time monitoring data for a specified time period. For example, the P95 response time can be directly obtained using Prometheus' predefined statistical functions.

[0039] To ensure the accuracy and integrity of the data, a full data collection is performed periodically (e.g., every few minutes), and real-time alerts and additional sampling are conducted for abnormal fluctuations.

[0040] Specific query statements mainly utilize PromQL, the query language of Prometheus. The specific query statement needs to be defined according to the type and granularity of the metrics to be monitored.

[0041] S14: Data Processing and Storage The collected raw data is initially cleaned to remove invalid or extreme outliers.

[0042] Use time-series databases or other efficient storage solutions to save the processed data for quick access during subsequent GNN modeling.

[0043] The steps in S2 are as follows.

[0044] S21: Construction of graph neural network inputs, including the following steps: S211: Construct an input call graph based on data collected from the Kubernetes cluster via Istio and Prometheus.

[0045] These include: Composition of resource side map: resource side map It is a directed graph consisting of a set of nodes. Sum of edges composition: in: It is a set of user nodes. This indicates the number of user nodes. It is the set of edges in the user graph, representing the relationships between nodes. Each node... It has a feature vector ,in It is the node feature dimension.

[0046] Node feature matrix Represents the characteristics of all nodes. Edge feature matrix Represent the characteristics of all edges, Adjacency Matrix This indicates the connection relationships between nodes.

[0047] Specifically, nodes represent microservices, characterized by P95 latency and QPS; edges represent the dependencies between microservices, characterized by QPS and P95 latency.

[0048] The system side graph is an undirected graph consisting of a set of nodes. Sum of edges composition, ,in: It is a set of system nodes. , It is the set of edges in the system graph, representing the relationships between nodes. Each node... It has a feature vector ,in It is the node feature dimension.

[0049] Node feature matrix This represents the characteristics of all nodes.

[0050] Adjacency Matrix This represents the connections between nodes. Specifically, nodes represent microservices, characterized by CPU usage, memory usage, and pods; edges represent dependencies.

[0051] S212: Graphification of data: Source data: such as raw monitoring and tracing data continuously collected from Istio and Prometheus. This data includes: The call chain and topology information between services (such as which service calls which service, and the number of calls).

[0052] Performance metrics for each microservice (such as P95 latency, QPS).

[0053] Resource metrics for each microservice (such as CPU utilization, memory usage, and number of Pods).

[0054] Graph structure construction: The system structures these raw data into a graph form according to predefined mapping rules.

[0055] Node: Each monitored microservice instance is identified and mapped as a node in the graph.

[0056] Resource-side input graph construction: For each microservice node, select metrics directly related to user experience (P95 latency, QPS, traffic, etc.) as its node features. The user request chain between services (such as the actual call chain traced by Istio) is abstracted as directed edges, and the characteristics of these edges are... This reflects the call volume and path latency, forming an adjacency matrix on the resource side. .

[0057] System-side input graph construction: For each microservice node, select metrics related to internal resource consumption and operational status (CPU utilization, memory usage, number of Pods, etc.) as its node features. Based on Istio's service dependency topology (representing the inherent functional coupling or resource dependencies between services), undirected edges are constructed to form an adjacency matrix on the system side. .

[0058] S213: Enter the following in the resource side map: Node feature matrix ; Edge feature matrix ; Adjacency Matrix ; Enter the following in the system side diagram: Node feature matrix ; Adjacency Matrix .

[0059] S214: Once the resource side map ( , , ) and system side diagram ( , These are constructed and then fed as independent inputs into their respective GNN models: Resource-side GNN model: Processes the resource-side graph and outputs the node embeddings from the resource side. This embedding captures the complex representation of microservices in terms of user-oriented performance.

[0060] System-side GNN model: Processes the system-side graph and outputs the node embeddings on the system side. This embedding captures the complex representation of microservices in terms of internal resources and dependencies.

[0061] These two independent node embeddings and This data is then fed into the encoder network. The encoder network uses a cross-diffusion attention mechanism and low-rank tensor fusion technique to... and Deep fusion is performed to generate a unified, high-dimensional global feature vector. . It comprehensively reflects the internal and external states of the microservice system and serves as the state input for the reinforcement learning agent, providing comprehensive information for its decision-making.

[0062] S22: Steps for constructing inputs for a graph neural network This step details how to construct a resource-side call graph for the GNN model based on data collected from the Kubernetes cluster using Istio and Prometheus. ) and system side diagram ( This process is a key foundation for realizing the state awareness of intelligent agents.

[0063] Step Overview: The core of GNN input construction is to transform scattered and heterogeneous monitoring data into structured graph data (nodes, edges, features) for effective processing by the GNN model.

[0064] The specific construction steps are as follows: S221: Raw Data Acquisition and Preprocessing: Data sources: Continuously collect real-time and historical data from the Istio service mesh (which provides inter-call topology and tracing data) and the Prometheus monitoring system (which provides key metrics such as P95 latency, QPS, CPU utilization, memory usage, and number of Pods for each service) in the Kubernetes cluster.

[0065] Cleaning and Alignment: The collected raw data is initially cleaned to remove invalid and outlier values, and timestamps are aligned between data from different sources to ensure data consistency.

[0066] S222: Microservice Node Identification and Unique Mapping Identify all running and monitored microservice instances in the cluster.

[0067] Each unique microservice instance is mapped to a node in the graph. For example, if service A has 3 Pods, it can be considered a logical node characterized by the aggregate metrics of these Pods.

[0068] S223: Resource Side Map ( ) Build: Node features ( ) Build: For each microservice node The performance metrics directly related to user experience (such as P95 response time and QPS) are extracted from Prometheus as its feature vector. The nodes converge to form a feature matrix. . It is the total number of services. It is the feature dimension of the resource-side node.

[0069] Directed edge ( ) and edge features ( Construction: Analyze the inter-service call chain data captured by Istio. If microservice A calls microservice B, a directed edge is created between A and B. The features of this edge are derived from metrics along the call chain, such as QPS and P95 latency, which are aggregated to form an edge feature matrix. .

[0070] Adjacency matrix ( Construction: Based on the constructed set of directed edges, generate the directed adjacency matrix on the resource side. .

[0071] S224: System Side Diagram ( ) Build: Node features ( ) Build: For each microservice node We extract metrics related to internal resource consumption and runtime status (such as CPU utilization, memory usage, and number of Pod instances) from Prometheus as its feature vector. The nodes converge to form a feature matrix. . It is the total number of services (and) same), It is the feature dimension of the system side nodes.

[0072] Key dependencies Construction: Based on the service topology or configuration dependencies discovered by Istio, identify the inherent logical or resource dependencies between microservices (e.g., a database service is a dependency of many other services). These dependencies are represented as undirected edges, typically without carrying additional edge features.

[0073] Adjacency matrix ( Construction: Based on the constructed set of undirected edges, generate the undirected adjacency matrix on the system side. .

[0074] After completing the above steps, we obtain two parallel and interconnected graph structures: the resource side graph. and system side diagram These contain their respective node feature matrices, edge feature matrices (if applicable), and adjacency matrices, which serve as inputs to subsequent GNN models.

[0075] S23: Graph Convolution Calculation Process The computation of each layer of a GNN can be represented as:

[0076] in: It is the first The node feature matrix of the layer, with initial values ​​of ; It is an adjacency matrix with self-loops.

[0077] yes The degree matrix.

[0078] It is the first Layer weight matrix.

[0079] It is a bias term.

[0080] It is an activation function (such as ReLU).

[0081] Ultimately, the resource-side and system-side GNN models output embeddings respectively.

[0082] in It is the number of layers in the GNN. It is the output embedding dimension.

[0083] Output and The processing steps include: S231: The resource-side and system-side GNN models process their respective graph inputs independently.

[0084] S232: For any GNN model, the... Layer Node Feature Representation It is achieved by aggregating the node features of the l-th layer. Information about the neighboring property (defined by adjacency matrix A) and its neighbors, and through the weight matrix. and bias terms It is obtained by performing a transformation and then applying the activation function σ.

[0085] formula This process of interlayer propagation is described.

[0086] S233: Initialization: The input to the model is the original node feature matrix. For a resource-side GNN, = For the system-side GNN, = .

[0087] S234: After L layers of graph convolution calculation (where L is the number of layers in the GNN model), the feature representation obtained by each node in the last layer L is the final embedding of that node.

[0088] S235: Output of the resource-side GNN : is the node feature matrix of the last layer L of the resource-side GNN model. Its dimensions are ,in It refers to the number of microservice nodes. It is the dimension of the output embedding. Each line in the table represents a compressed, high-dimensional representation of a microservice in terms of resource-side dimensions (such as user-oriented performance and traffic dependencies).

[0089] S236: Output of the system-side GNN : is the feature matrix of the nodes in the last layer L of the system-side GNN model. Its dimensions are ,in It refers to the number of microservice nodes. It represents the dimension of the output embedding. Each row in Zs represents a compressed, high-dimensional representation of a microservice in system-side dimensions (such as internal resource consumption and structural dependencies).

[0090] and These are the final node embeddings learned by their respective GNN models, containing the microservice's own features and information about its neighbors in the graph. They are the direct inputs for information fusion in the downstream encoder network.

[0091] S24: Encoder Network The encoder network is responsible for fusing the embeddings from the resource side and the system side to generate a global feature vector. In this invention, the encoder network uses a cross-diffusion attention mechanism and low-rank tensor fusion. S241: Methods for handling cross-diffusion attention: Resource-side embedding and system-side embedding Interacting through attention mechanisms:

[0092] in , , It is a projection matrix of requests, keys, and values.

[0093] The result of this interactive computation, namely the attention score generated by the cross-diffusion attention mechanism, is... It plays a key role in guiding information fusion in the subsequent encoder network.

[0094] Specific applications are as follows: Information Alignment and Relevance Measurement: Attention Score Essentially Quantifies Resource-Side Embedding Each microservice in the system is embedded with the system side. This identifies the interrelationships or importance between each microservice in the model. It helps the model understand similar or related services from different perspectives (user experience and internal resources).

[0095] Weighted Information Fusion: The cross-diffusion attention mechanism uses these attention scores as a weight matrix and applies them to the system-side embedding. (or vice versa) This means that when the model tries to... and During fusion, it selectively and weights these attention scores. The information in the text. Attention mechanisms typically multiply by a projection matrix. The weighted Features and original Features are fused.

[0096] Generate fused embeddings: Through this weighted and aligned mechanism, information "diffused" and integrated from the two graph embeddings, thereby generating more representative and comprehensive fused embeddings. For example, the attention score of a high-latency microservice on the resource side might prompt the model to pay more attention to other related services on the system side that may have resource bottlenecks.

[0097] Therefore, this interactive computation result is the core mechanism for achieving deep integration of information from the resource side and the system side, ensuring the final global feature vector. It can comprehensively and selectively reflect the complex state of the system.

[0098] S242: Methods for handling low-rank tensor fusion include: Integrate the embeddings from the resource side and the system side: Low-rank tensor fusion can be efficiently computed through decomposition.

[0099] S243: Generate global feature vectors The methods include: Map the fused embeddings to the global feature vector: ,in and The weights and biases of the fully connected layer are used to obtain the values ​​for downstream tasks. Embedded.

[0100] exist In this equation, and These are the learnable parameters in the model.

[0101] (WeightMatrix): This is a linear transformation weight parameter with dimensions ( ),in yes Dimensions yes The dimension. It is responsible for fusing the features. Project (or transform) to the final global feature space.

[0102] (BiasVector): This is a bias parameter with dimensions ( It introduces an offset after feature transformation, increasing the model's representational power.

[0103] and It is the internal representation that the model itself learns from a large amount of data.

[0104] In S3, the main focus is on dynamically adjusting microservice resources based on predicted global embedded information and cluster status to meet service level objectives (SLOs) while optimizing resource utilization.

[0105] Furthermore, S3 includes the following: S31: Definition of Reinforcement Learning Environment To achieve intelligent scaling of microservices based on reinforcement learning, the reinforcement learning environment for agent interaction must first be precisely defined. This environment mainly consists of three elements: state space, action space, and reward function. These elements together define the boundaries and goal orientation of the agent's learning and decision-making. This will be implemented through the following steps.

[0106] S311: State space ( ) The state space S represents the set of environmental information that the agent can perceive at each decision-making moment. In this invention, to fully utilize the powerful representation capability of graph neural networks for the complex dependencies and multidimensional features of microservice systems, the state space is designed to directly adopt the global embedding vector output by the GNN encoder described in Chapter 2. .

[0107] At the moment of decision The state observed by the agent That is, the global embedding vector .

[0108] This vector is a comprehensive high-dimensional feature representation generated by the GNN model proposed in this invention (which integrates the internal GNN model and the external GNN model, and is processed by inter-layer fusion models such as Crossdiffusionattention and lowrankfusion) based on the current system snapshot. It integrates deeper information from the system call graph (containing system-side features such as CPU utilization, memory usage, and number of Pod instances for each microservice) and the user call graph (containing user-side features such as P95 response time, QPS, and inter-service dependencies for each microservice). As state input, the aim is to build an end-to-end decision-making system.

[0109] S312: Action Space ( ) Action space A defines the set of discrete operations that an agent can choose to execute at each decision step. For microservice scaling scenarios, actions primarily focus on adjusting the number of compute resource instances (i.e., the number of Pods) for a specific microservice.

[0110] Target microservice selection This invention employs a strategy of selecting a single target microservice for regulation at each decision step to simplify the complexity of the action space and ensure the interpretability of the decision.

[0111]

[0112] in This represents the total number of microservices in the system that can be scaled up or down.

[0113] In the definition of action space

[0114] and Their definitions and sources are as follows: (Target microservice): definition: Indicates the moment of decision-making A reinforcement learning agent selects specific target microservices for scaling operations. It is a component of the action space of the RL agent, specifically corresponding to a dimension in the Actor network output.

[0115] Source: This value is generated by the policy network (ActorNetwork) of the reinforcement learning agent upon receiving the current state. Then, based on the learned strategy, the agent makes a decision output. That is, the agent decides which microservice to scale based on its perception of the current overall system state. It starts from a predefined... Choose from the list.

[0116] (Collection of operable microservices): Definition: This is a collection containing unique identifiers for all microservices in the current Kubernetes cluster that are identified as being eligible for scaling. These identifiers are typically the names or service IDs of the microservices.

[0117] Source: This set is dynamically built and maintained during system initialization and operation through service discovery mechanisms (e.g., querying the list of registered services in the Istio service mesh) and configuration filtering (e.g., including only services that define scaling policies). It represents the target range of operations that an agent can choose to perform.

[0118] (Total number of operable microservices): definition: It is a collection of operable microservices The total number (cardinality of the set).

[0119] Source: Its value is directly determined by the size of the aforementioned set of operable microservices. As microservices are deployed or removed from the cluster, The value will also be updated dynamically.

[0120] These parameters collectively define the scope and specific choices that reinforcement learning agents make when selecting scaling targets, and they are closely related to the actual microservice deployment environment.

[0121] Determine the scaling operation type (Op) and magnitude (Mag) for the selected target microservice. The agent selects an operation type from a predefined set of discrete operations. The set includes: ScaleUp: Increases the number of Pod instances for the target microservice.

[0122] ScaleDown: Reduces the number of Pod instances for the target microservice.

[0123] NoOp: Keeps the current number of Pod instances of the target microservice unchanged.

[0124] To ensure precise and stable operation, this invention uses a fixed step size as the expansion / contraction range (Mag), that is, when When it is ScaleUp or ScaleDown, the amplitude It is fixed at 1 Pod instance.

[0125] Combine to form a complete action and apply constraints: Combine the target microservice q determined in S3121 with the operation type determined in S3122. Combined, they form a complete, executable discrete action. Before executing this action, the following action constraints must be applied for verification: The result of any scaling action must ensure that the number of Pods in the target microservice i is [not specified]. Maintain at the preset minimum boundary and maximum boundary Within, that is, the conditions are met. If a planned action would cause the number of Pods to violate this constraint, the action would be corrected to "NoOp" to ensure the stable operation of the service and compliance of resource usage.

[0126] S313: Reward Function ) reward function Used to evaluate the agent's state Execute action Later transitioned to a new state The function provides immediate feedback signals. Its design goal is to guide the agent to learn an optimal policy that maximizes resource utilization efficiency (i.e., using as few Pods as possible) while meeting the Service Level Objective (SLO). The integrated reward function structure used in this invention is as follows:

[0127] in: This indicates the agent's decision-making time. Due to the execution of actions And the scalar reward value obtained after subsequent environmental feedback was observed.

[0128] The reward component is related to the achievement of service level objectives (with end-to-end P95 response time as the core metric).

[0129] This is the reward component related to the overall Pod resource consumption of the system.

[0130] This is a hyperparameter used to weigh the importance of satisfying SLOs against the importance of resource conservation.

[0131] In S313, the following processing steps are also included: S3131: Service Level Target Incentive Component ( ): This component aims to incentivize agents to take actions that improve key system performance metrics (specifically, in this invention, the performance of actions). Then, in the next observation cycle Actual measured end-to-end P95 response time Conforms to preset (Target P95 delay threshold) action.

[0132] when Time (i.e., service performance meets or exceeds the target):

[0133] in, Basic positive rewards; This is the gain coefficient; To mitigate the excessive growth of rewards when P95 is significantly lower than SLO, a diminishing return factor is used, avoiding unnecessary resource investment in pursuing extreme but diminishing performance improvements. The logarithmic function ensures that rewards increase as P95 decreases, but at a slower rate.

[0134] when > When (i.e., service performance does not meet standards): =

[0135] in, This is the penalty coefficient;

[0136] This is the SLO penalty adjustment factor. The penalty term is proportional to the relative extent to which P95 exceeds the SLO, imposing a penalty on behaviors that violate the SLO, with the penalty being greater for more severe violations.

[0137] S3132: Pod resource consumption reward component ( ): This component aims to encourage agents to minimize the total number of Pod instances running in the system while meeting performance goals, thereby improving resource utilization efficiency.

[0138]

[0139] Refers to the execution of an action Finally, the sum of Pod instances of all controlled microservices within the system.

[0140] and These represent the system's preset global minimum and maximum total number of Pod instances, respectively.

[0141] This formula linearly maps the total number of Pods, making the total number of Pods as close as possible to... , The higher (maximum 1.0); the closer to The lower the value, the better (the lowest is 0).

[0142] S32: Reinforcement Learning Algorithm Selection and Ensemble: Proximal Policy Optimization (PPO), implemented through the following steps.

[0143] S321: Applicability analysis of the PPO algorithm.

[0144] The PPO was chosen as the core of the reinforcement learning decision in this invention because of its advantages in stability and sampling efficiency when dealing with high-dimensional states and complex decision tasks, which makes it well adaptable to the needs of dynamic scaling of microservices.

[0145] S322: Application Architecture of PPO Algorithm in this Invention Policy Network (Actor): Input: Global state embedding vector generated by the GNN encoder (Right now ).

[0146] Function: Based on the input state, decide and output scaling actions for the target microservice. .

[0147] Value Network (Critic): Input: Both are global state embedding vectors (Right now ).

[0148] Function: Evaluate the current state Its long-term value is used to assist in the optimization of policy networks.

[0149] S33: Online Training, Adaptation, and Decision Execution of Reinforcement Learning Agents The reinforcement learning agent designed in this invention not only performs scaling decisions after initial deployment, but more importantly, it possesses the ability to learn and adapt online continuously. This means that the agent can continuously optimize its decision-making strategy based on ongoing interaction with the actual microservice environment during its operation, adapting to dynamically changing workload patterns, system characteristic evolution, and potential unknown disturbances. This section will elaborate on the closed-loop process of online training, adaptation, and decision execution.

[0150] S33 specifically includes the following steps: S331: System Initialization and Agent Deployment Agent parameter initialization: During initial deployment, the policy network of the reinforcement learning agent is initialized. and value network parameters and Random initialization is possible.

[0151] Connection with the environment: After deployment, the agent establishes a communication connection with the target Kubernetes cluster to ensure that it can obtain the global state embedding generated in real time by the GNN encoder. It can also execute scaling commands via the Kubernetes API.

[0152] S332: The Core Loop of Online Interaction, Learning, and Adaptation The core operating mode of an intelligent agent is a continuous cycle of "perception-decision-execution-learning," which includes the following steps: S3321: Perception & Decision Making At the preset decision-making time point The agent first obtains the global state embedding of the current microservice system from the GNN encoder. = .

[0153] State Input into the latest policy network The policy network outputs a list of all possible scaling actions. The probability distribution.

[0154] Action selection and exploration: To balance exploration and exploitation, agents typically select actions by randomly sampling based on this probability distribution. In the PPO algorithm, the randomness of the strategy itself already includes an exploration mechanism.

[0155] S3322: Action Execution & Environment Feedback Selected expansion / contraction action It is translated into a specific Kubernetes API call.

[0156] The system is performing actions After an observation cycle (matching the decision-making cycle), the environment evolves to a new state.

[0157] At the next decision observation point The actual performance indicators and resource usage observed by the system are collected.

[0158] According to the reward function defined in Section 3.1.3 Calculate the instant reward .

[0159] The GNN encoder generates a new global state embedding. .

[0160] S3323: Experience Storage & Parameter Update The experience tuples generated from this interaction Stored in an ExperienceReplayBuffer. This buffer is used to store the interaction history over a recent period.

[0161] Online updates of model parameters: The agent periodically samples a mini-batch of empirical data from the experience replay buffer.

[0162] Using this sampled data, and following the PPO algorithm optimization process (including calculating the advantage function, constructing the PPO-Clip objective function, and calculating the value loss), the policy network parameters are adjusted. and value network parameters Perform gradient updates.

[0163] This online parameter update mechanism, based on actual interaction data, enables the agent's decision-making strategy to be continuously fine-tuned and optimized, thereby adapting to the dynamic changes in the microservice system environment, including periodic fluctuations in workload, sudden traffic surges, and performance characteristic changes introduced by microservice version iterations.

[0164] The above description is merely a preferred embodiment of the present invention and should not be construed as limiting the scope of the invention. Therefore, any equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for automatic scaling of microservices based on graph neural networks and reinforcement learning, characterized in that, Includes the following steps: S1: Perform environmental preparation and data collection configuration, acquire historical data or real-time monitoring data within a specified time period, clean and store the collected raw data; S2: Construct the graph neural network input and embed it into the output of the resource-side GNN model: The GNN model output embedding on the system side: ,in, It is the number of layers in the GNN. It is the output embedding dimension. , It represents the total number of services; by fusing embeddings from the resource side and the system side through the encoder network, a global feature vector is generated. ; S3: Define the reinforcement learning environment, including the state space, action space, and reward function; input the global feature vector into the proximal policy algorithm. And during operation, it continuously optimizes decision-making strategies based on ongoing interactions with the actual microservice environment.

2. The automatic scaling method for microservices based on graph neural networks and reinforcement learning according to claim 1, characterized in that, The data acquisition process of S1 includes inputting call graph data on both the system side and the resource side.

3. The automatic scaling method for microservices based on graph neural networks and reinforcement learning according to claim 2, characterized in that, The graph data is generated through the following steps: S11: Identify each microservice deployed in the Kubernetes cluster and map each identified microservice instance to an independent node in the graph; S12: Extract metrics related to user experience and throughput for each microservice node to form a resource-side feature vector; S13: Extract indicators related to internal resource consumption and operating status for each microservice node to form a system-side feature vector; S14: Use the inter-service call chain captured by Istio service mesh on the resource side to construct directed edges; S15: On the system side, use service topology information captured based on the Istio service mesh or actual observed inter-service dependencies to construct undirected edges.

4. The automatic scaling method for microservices based on graph neural networks and reinforcement learning according to claim 1, characterized in that, The steps in S2 for constructing the input to the graph neural network include: S211: Based on the collected data, construct an input call graph and set the nodes... Sum of edges Composition of resource side map , composed of node set Sum of edges Side diagram of the system ; S212: Resource-side input graph construction: For each microservice node, select metrics directly related to user experience as its node features. The user request chain between services is abstracted as directed edges, and the characteristics of the edges are... Reflecting call volume and path latency, an adjacency matrix is ​​formed on the resource side. ; S213: System-side input graph construction: For each microservice node, select indicators related to internal resource consumption and operating status as its node features. Undirected edges are constructed based on the service dependency topology of Istio to form an adjacency matrix on the system side. ; S214: Input node feature matrix in resource side graph Edge feature matrix Adjacency matrix ; S215: Input node feature matrix in system side graph Adjacency matrix ; S216: In the resource side map and system side diagram Once constructed, they are fed as independent inputs into their respective GNN models: In the resource-side GNN model: process the resource-side graph and output the node embeddings from the resource side. Node embedding It captures the complex characteristics of microservices in terms of user-oriented performance. In the system-side GNN model: process the system-side graph and output the system-side node embeddings. Node embedding It captures the complex representation of microservices in terms of internal resources and dependencies.

5. The automatic scaling method for microservices based on graph neural networks and reinforcement learning according to claim 4, characterized in that, S2 further includes the following steps: S221: The GNN models on the resource side and the system side process their respective graph inputs independently; S222: For any GNN model, the node feature representation H of the (l+1)th layer. (l+1) It is achieved by aggregating the node features H of the l-th layer (l) Information about it and its neighbors, and through the weight matrix W (l) and bias term B (l) Perform a transformation, then obtain the result through the activation function σ, using the formula... This describes the interlayer propagation process; S223: The input to the model is the original node feature matrix. For a resource-side GNN, ; For the system-side GNN ; S224: After L layers of graph convolution calculation, the feature representation obtained by each node in the last layer L is the final embedding of that node; S225: Output of the resource-side GNN : is the node feature matrix of the last layer L of the resource-side GNN model. Its dimensions are ,in It refers to the number of microservice nodes. It is the dimension of the output embedding. Each line in the table represents a compressed, high-dimensional representation of a microservice on the resource side. S226: Output of the system-side GNN : is the feature matrix of the nodes in the last layer L of the system-side GNN model. Its dimensions are ,in It refers to the number of microservice nodes. It is the dimension of the output embedding. Each line in the table represents a compressed, high-dimensional representation of a microservice on the system side.

6. The automatic scaling method for microservices based on graph neural networks and reinforcement learning according to claim 1, characterized in that, In step S2, the encoder network uses a cross-diffusion attention mechanism and low-rank tensors to fuse resource-side and system-side embeddings to generate a global feature vector. .

7. The automatic scaling method for microservices based on graph neural networks and reinforcement learning according to claim 6, characterized in that, The processing method of the cross-diffusion attention mechanism includes: Embed the resource side and system-side embedding Interacting through attention mechanisms: in It is a projection matrix of requests, keys, and values; The low-rank tensor fusion processing method includes: Integrate the embeddings from the resource side and the system side: ; Generate global feature vectors The methods include: Map the fused embeddings to the global feature vector: ,in and These are the weights and biases of the fully connected layer.

8. The automatic scaling method for microservices based on graph neural networks and reinforcement learning according to any one of claims 1-7, characterized in that, In S3: The state space is used to represent the set of environmental information that the agent can perceive at each decision moment, and it uses the global embedding vector output by the GNN encoder. ; The action space defines the set of discrete operations that the agent can choose to execute at each decision step, and uses the following action constraints to ensure the stable operation of the service and the rational use of resources: The execution of any scaling up or down operation must ensure that the target microservice's Pod count is within a certain range. Maintain at the preset minimum boundary and maximum boundary Within; The reward function Used to evaluate the agent's state Execute action Later transitioned to a new state The immediate feedback signal obtained at that time uses the following comprehensive reward function structure: in: This indicates the agent's decision-making time. Due to the execution of actions And observe the scalar reward value obtained after subsequent environmental feedback; The reward component is related to the achievement of service level targets; The reward component is related to the overall Pod resource consumption of the system; This is a hyperparameter used to weigh the importance of satisfying SLOs against the importance of resource conservation.

9. The automatic scaling method for microservices based on graph neural networks and reinforcement learning according to claim 8, characterized in that, The application architecture of the near-end strategy algorithm in S3 includes: Policy Network: Input is the global state embedding vector generated by the GNN encoder. Based on the input state, the decision outputs scaling actions for the target microservice. ; Value network: Input global state embedding vector Assess the current status Its long-term value is used to assist in the optimization of policy networks.

10. The automatic scaling method for microservices based on graph neural networks and reinforcement learning according to claim 9, characterized in that, S3 also includes the following steps: S31: Agent parameter initialization: During initial deployment, the policy network of the reinforcement learning agent is initialized. and value network parameters and Perform random initialization; S32: Use a continuous perception-decision-execution-learning cycle, which includes steps of state perception and decision making, action execution and environmental feedback, and experience storage and parameter updating. The steps of state perception and decision-making include: At the preset decision-making time point First, the global state embedding vector of the current microservice system is obtained from the GNN encoder. = ; Embed the global state vector Input into the latest policy network The policy network outputs a list of all possible scaling actions. To balance exploration and utilization, random sampling is performed based on the probability distribution to select actions. ; The steps for action execution and environmental feedback include: Selected expansion / contraction action It is translated into a specific Kubernetes API call; in the execution of actions After a period of observation, the environment evolves to a new state, and the next decision-making observation point is reached. Actual performance metrics and resource usage are collected and used to calculate immediate rewards. The GNN encoder generates new global state embeddings. ; The steps of experience storage and parameter updating include: Experience tuples generated by interaction Stored in an experience replay buffer, which is used to store the interaction history within a recent period; Periodically sample a batch of empirical data from the experience replay buffer and optimize the process according to the near-end policy optimization algorithm to adjust the policy network parameters. and value network parameters Perform gradient updates.

Images