An edge micro-service automatic scaling method fusing time series prediction and reinforcement learning

Abstract

The application relates to the field of service computing and discloses an edge microservice automatic expansion and reduction method combining time sequence prediction and reinforcement learning, which comprises the following steps: step 1) first, modeling the problem scene of an edge computing platform, a microservice application, a deployment scheme and a deployment cost; step 2) based on TimeMixer improvement, a time sequence prediction model ATimeMixer is proposed to obtain the future work load of the microservice; and step 3) designing a DDPG intelligent agent to make an optimized decision in combination with real-time information and predicted information; the original average pooling downsampling and moving average kernel time sequence decomposition in the TimeMixer are replaced by convolution-based downsampling and time sequence decomposition, so that the ability of ATimeMixer to capture diverse and complex feature patterns in the work load time sequence is improved.

Description

Technical Field

[0001] This invention relates to the field of service computing, and more specifically to an automatic scaling method for edge microservices that integrates temporal prediction and reinforcement learning. Background Technology

[0002] Over the past few years, the Internet of Things (IoT) industry has rapidly emerged, enabling connections between people and things, and between things themselves, thus promoting the digitization of the physical world. IoT applications have permeated all aspects of people's lives, aiming to provide high-quality services and security, such as smart elevators, smart healthcare, smart transportation, and even larger-scale smart cities. With the exponential growth in the number of IoT devices, IoT platforms aggregate and process massive amounts of device data daily. The applications they support are often computationally intensive and extremely sensitive to response latency. The emergence and development of cloud computing has provided a solution for IoT applications. Cloud data centers offer powerful computing and storage capabilities to efficiently process relevant IoT application requests and permanently store large amounts of data. However, cloud data centers are typically located far from IoT devices, and the high latency caused by distance has become a pain point for IoT applications under the cloud computing paradigm. Driven by these issues, a new, efficient, and flexible computing paradigm, edge computing, has gradually emerged and become the mainstream computing paradigm for IoT applications. As an extension of cloud computing, edge computing pushes services closer to users. Through distributed computing and storage infrastructure such as edge servers, latency-sensitive IoT applications are deployed nearby to accelerate access speeds. On the other hand, with the rapid development of the Internet of Things (IoT) and in order to better adapt to the characteristics of the edge computing environment, IoT application development has shifted from monolithic architecture to microservice architecture. This architecture decomposes IoT applications into multiple independent microservices, each responsible for specific module functions and independently developed, deployed, expanded and managed, which greatly improves the flexibility, scalability and maintainability of applications.

[0003] As system architecture evolves towards fine-grained, loosely coupled microservices, effectively coordinating complex request processes across multiple independent services becomes a key challenge. The implementation method of service coordination directly impacts the maintainability, scalability, and adaptability of the system. Microservice orchestration, as a declarative, metadata-driven coordination pattern, contrasts sharply with traditional hard-coded coordination. The latter embeds process logic directly into the code, which, while straightforward, reveals inherent drawbacks such as maintenance difficulties and poor scalability as system complexity increases. Microservice orchestration, on the other hand, explicitly manages business logic flows through centralized process definition and control, providing systematic support for distributed transaction processing, data consistency guarantees, and process visualization. However, during orchestration, with the arrival of time-varying requests, using a fixed number of microservice instances to handle requests lacks flexibility and efficiency, requiring the system to have the ability to dynamically adjust the number of microservice instances. On the one hand, when requests surge, the request processing demand exceeds the instance processing capacity, at which point the system needs to expand service instances to maintain service performance. On the other hand, after the peak of requests subsides, the decrease in requests can lead to an overabundance of service instances. At this point, the system also needs to scale down unnecessary service instances to save on operating costs. Therefore, achieving efficient and intelligent automatic scaling of microservice instances has become a significant challenge.

[0004] Based on the timing of scaling up and down, existing automatic scaling algorithms can be categorized into reactive, pre-testing, and hybrid approaches. Reactive algorithms typically make decisions based on the current system's actual performance and resource utilization, with scaling up occurring after workload changes. Therefore, they inherently suffer from scaling decisions lagging behind actual demand changes and the inability to predict periodic or sudden load fluctuations. Predictive algorithms primarily rely on time series analysis and machine learning techniques to predict workload or resource usage trends, thereby adjusting resource allocation in advance before demand changes. Hybrid algorithms combine the advantages of predictive and reactive methods, proactively adjusting resources by predicting future resource needs, while utilizing reactive mechanisms as a supplement to address inaccurate predictions. Furthermore, reinforcement learning's application in automatic scaling integrates the core characteristics of reactive and predictive methods, periodically collecting environmental data and responding to immediate conditions. This approach frames the problem as a Markov decision process, estimating the potential rewards of future states and then executing the expected optimal operation.

[0005] However, most existing automatic scaling algorithms based on reinforcement learning make decisions solely based on current information, and their predictions only reflect estimates of potential rewards for future states, without actively incorporating available future information into the predictive model. Secondly, they do not effectively handle the call dependencies between microservices and the resulting workload cascading effects. Summary of the Invention

[0006] This invention proposes an automatic scaling method for edge microservices that integrates temporal prediction and reinforcement learning. First, it models the problem scenario, including the edge computing platform, microservice applications, deployment scheme, and deployment costs. Next, it proposes the temporal prediction model ATimeMixer to obtain the future workload of microservices. Finally, it designs a Deep Deterministic Policy Gradient (DDPG) agent to make optimization decisions by combining real-time information and the prediction information generated by ATimeMixer.

[0007] The technical solution adopted in this invention is:

[0008] An automatic scaling method for edge microservices that integrates temporal prediction and reinforcement learning includes the following steps:

[0009] Step 1: First, model the problem scenarios for the edge computing platform, microservice applications, deployment schemes, and deployment costs.

[0010] Step 2: Based on the TimeMixer model, the ATimeMixer time series prediction model is improved. Convolution-based downsampling is used to replace the original average pooling downsampling in TimeMixer, and convolution-based time series decomposition is used to replace the original moving average kernel time series decomposition. At the same time, a graph attention mechanism is introduced to explicitly model the dependencies between the time series of workloads of each microservice. The future workload of microservices is obtained through multi-scale decomposition, trend-seasonal component separation and mixing, and multi-scale independent predictor fusion.

[0011] Step 3: Construct a topology-aware DDPG agent. The future workload prediction results output by the ATimeMixer model are incorporated into the decision-making process as future information. At the same time, the instance deployment history of the past few steps is introduced into the agent's observation space as state information to perceive the recent state. Construct a composite reward function that includes service performance, deployment cost, and scaling stability. Introduce a penalty term for potential scaling oscillations in the reward function to suppress frequent fluctuations in the decision-making process. Guide the agent to train offline through the composite reward function. After training, take the real-time system state and future workload prediction information as input, and output a microservice instance scaling decision that satisfies the edge node resource constraints and microservice instance deployment constraints through the policy network.

[0012] Furthermore, step 1 is specifically as follows:

[0013] Step (1.1) definition This is an edge computing platform represented by an undirected graph, where... It represents a finite set of edge computing nodes, each of which has an independent number of available CPU cores, memory size, and storage capacity; It is a finite set of edges defined by different node pairs, representing the network topology relationships between nodes;

[0014] Step (1.2) definition This is an IoT application developed based on a microservice architecture, in which... For all the microservices included in the application, each microservice is characterized by a set of tasks, an instance and its relative location on the edge computing platform, and the number of CPU cores, memory size and storage capacity required per instance. The application topology graph is modeled using a directed acyclic graph, and the set of directed edges is used to describe the call relationships between microservices.

[0015] Step (1.3) definition The deployment scheme for microservice instances is a matrix, where rows represent microservices and columns represent edge computing nodes. The matrix elements represent the number of deployment instances of the corresponding microservice on the corresponding edge node, and the deployment scheme satisfies the following conditions: each microservice has at least one instance deployed, and the resource usage of the edge node does not exceed its supply capacity.

[0016] Step (1.4) Calculate the total deployment cost of the microservice instances based on the unit resource cost and the number of microservice instances deployed.

[0017] Furthermore, step 2 is specifically as follows:

[0018] Step (2.1) defines the workload of a microservice as the number of calls received by all instances of the microservice per unit time, thereby obtaining the time-series characteristics of the microservice workload;

[0019] Step (2.2) uses a sliding window to divide along the temporal direction to obtain the input temporal features and the target temporal features;

[0020] Step (2.3) utilizes a graph attention network based on the microservice application topology graph. By aggregating the features of microservice nodes and their upstream nodes, enhanced temporal features with fused topological dependency information are obtained;

[0021] Step (2.4) performs multi-scale downsampling on the enhanced temporal features through one-dimensional convolution to obtain a multi-scale input temporal feature set;

[0022] Step (2.5) projects the multi-scale input temporal feature set onto the embedding layer, thereby obtaining the multi-scale deep feature set;

[0023] Step (2.6) uses stacked past decomposition mixed PDM blocks to decompose the multi-scale deep features into trend components and seasonal components. The trend components are mixed from top to bottom using TDM and the seasonal components are mixed from bottom to top using BUM to achieve multi-scale information fusion. After the components are re-merged, the residual connection is combined to output the processed multi-scale features.

[0024] Step (2.7) configures an independent predictor for each scale and outputs the scale-aligned multi-scale load prediction time series features;

[0025] Step (2.8) maps the multi-scale predicted temporal features to the final predicted temporal features through accumulation;

[0026] Step (2.9) defines the loss as the mean square error of the target time series feature and the predicted time series feature;

[0027] Step (2.10) train ATimMixer to obtain accurate future prediction time-series features of microservice workloads.

[0028] Furthermore, step 3 is specifically as follows:

[0029] Step (3.1) Construct the basic components of DDPG, including the policy network, evaluation network, target policy network, target evaluation network, and experience replay pool;

[0030] Step (3.2) Create a temporary buffer to record historical microservice instance deployment schemes for scaling stability assessment;

[0031] Step (3.3) Observe the environment and obtain the initial state. And the prediction results of ATimeMixer; initial state Includes graph structure data And the resource vectors of edge computing nodes;

[0032] Step (3.4) sets the initial state The input policy network, through a graph attention network, fuses information on microservice topology, current load, future load prediction, and historical deployment schemes. This information is then concatenated with edge node resource vectors, processed by a multilayer perceptron (MLP) and normalized, to output microservice instance scaling actions. ;

[0033] Step (3.5) Determine the expansion / contraction action If the corresponding deployment plan meets the constraints, a penalty or reward is given and the current decision is terminated, and the process restarts from step (3.2); if it meets the constraints, the scaling action is executed, and the process proceeds to the next time step.

[0034] Step (3.6) Calculate the reward for this round. The reward is a weighted sum of performance reward, cost reward and scaling stability reward. The performance reward corresponds to the request success rate, the cost reward is the negative value of the total deployment cost, and the scaling stability reward is based on the instance change direction of two consecutive deployment schemes to penalize oscillation behavior.

[0035] Step (3.7) stores the system interaction experience into the experience replay pool. The system interaction experience includes the current system status, the current load prediction result, the two most recent historical deployment schemes, the current scaling action, the current round of rewards, the next system status, and the load prediction result at the next moment.

[0036] Step (3.8) will involve expansion and contraction. Corresponding deployment scheme Add to the temporary buffer and update the deployment plan for the temporary buffer;

[0037] Step (3.9) sets the initial state Replace with the next state And iterate the decision-making process;

[0038] Step (3.10) When the experience in the experience replay pool reaches the preset sampling size Simultaneously, experience sampling and playback are initiated to train the evaluation network and policy network, and the target policy network and target evaluation network are updated using a soft update method.

[0039] Step (3.11) Repeat the training process of step (3.10) until the model converges;

[0040] After completing the above training in step (3.12), DDPG retains only the policy network, which is used to integrate the load prediction results of ATimeMixer for intelligent scaling decisions.

[0041] Furthermore, in step (2.3), the neighborhood of the graph attention network is defined as the microservice node itself and its upstream microservice nodes.

[0042] Furthermore, in step (2.6), the PDM block decomposes the trend component and the seasonal component through a learnable convolutional kernel, and both TDM and BUM achieve scale alignment and information interaction through a single-linear-layer multilayer perceptron (MLP).

[0043] Furthermore, the scaling action in step (3.4) corresponds to the flattening vector of the deployment scheme matrix, which directly represents the target number of microservices on each edge node.

[0044] Furthermore, in step (3.6), the scaling stability reward is used to quantify the penalty for oscillation behavior by judging whether the instance change directions of two consecutive deployment schemes are opposite.

[0045] Furthermore, both the policy network and the evaluation network encode the microservice application topology graph through a multi-layer graph attention network. The neighborhood of the graph attention network is defined as the microservice node itself and its downstream calling nodes. The encoded graph-level features are concatenated with the edge node resource vectors as network input.

[0046] The beneficial effects of this invention are mainly reflected in:

[0047] 1) The convolution-based downsampling and time series decomposition replace the original average pooling downsampling and moving average kernel time series decomposition in TimeMixer, improving ATimeMixer's ability to capture diverse and complex feature patterns in workload time series.

[0048] 2) To address the workload cascading effect caused by dependencies between microservices, ATimeMixer introduces a graph attention mechanism to explicitly and efficiently model the time-series dependencies between the workloads of each microservice, thereby improving its prediction accuracy.

[0049] 3) Integrate the prediction results of ATimeMixer as future information into the decision-making process of the DDPG reinforcement learning agent, thereby enhancing its decision-making ability by combining information from both the present and future perspectives.

[0050] 4) To improve the stability of the expansion and contraction process, DDPG incorporates the instance deployment history of the past few steps as state information into its observation space to perceive the recent state; and introduces a penalty term for potential expansion and contraction oscillations into the reward function to suppress frequent fluctuations in the decision-making process. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the architecture of the edge microservice automatic scaling method that integrates temporal prediction and reinforcement learning in this invention;

[0052] Figure 2 This is a schematic diagram of the results of the hyperparameter experiment in an embodiment of the present invention. Detailed Implementation

[0053] The present invention will now be further described with reference to the accompanying drawings.

[0054] Figure 1 This invention presents the overall architecture of the method, abbreviated as TDScaler, which includes an improved time-series forecasting model, ATimeMixer, based on TimeMixer, and a designed DDPG agent. The former is used to predict the future workload of microservices, while the latter fuses the prediction results with real-time information from the microservices to make optimization decisions. (See also...) Figure 1 An automatic scaling method for edge microservices that integrates temporal prediction and reinforcement learning includes the following steps:

[0055] The first step is to model the problem scenarios for the edge computing platform, microservice applications, deployment schemes, and deployment costs.

[0056] Step (1.1) definition This is an edge computing platform represented by an undirected graph, where... , representing a finite set of edge computing nodes, totaling There are 10 nodes. For a given edge computing node... Its available CPU cores, memory size, and storage capacity are expressed as follows: , , . It is a finite set of edges defined by different node pairs, representing the network topology relationships between nodes, totaling... Edges. Specifically, each edge computes a node. Each has a corresponding edge gateway. For example, a base station in a local area network or metropolitan area network, and uses an edge gateway link to establish communication with other nodes in order to jointly process IoT requests.

[0057] Step (1.2) definition This is an IoT application developed based on a microservice architecture, in which... For all the microservices included in the application, a total of indivual; This represents an application topology graph modeled by a directed acyclic graph, using a set of directed edges. To describe microservices Inter-call relationship, if there exists a directed edge such that point to , then it means Call Each microservice consists of a tuple. It means that, among them, express The base number of exposures is The task set; }express The set of instances and their relative positions on the edge computing platform, where, microservices Examples Placed on edge computing nodes superior; , , Each represents the deployment of one The number of CPU cores, memory size, and storage capacity required for the instance.

[0058] Step (1.3) definition The deployment scheme for microservice instances is a matrix, where rows represent microservices. The column represents the edge computing node. ,as follows:

[0059]

[0060] in, , indicating microservices of One instance is deployed on the edge computing node. superior. The basic range of values ​​is expressed as follows: its upper boundary Limited by resource consumption and The resource constraints are defined as follows:

[0061]

[0062] in, This indicates rounding down. In addition, the deployment plan must satisfy the following joint constraints: row constraints determine whether the number of microservice instances is greater than or equal to 1 by calculating the sum of each row; column constraints determine whether resource usage exceeds node supply based on the edge computing node instance deployment status represented by each column.

[0063]

[0064]

[0065] in, This indicates an accumulation operation. The first constraint ensures that each microservice has at least one instance deployed, while the latter represents the CPU, memory, and storage resource constraints of the edge computing node.

[0066] Step (1.4) defines the resource costs of 1 CPU core, 1GB memory, and 1GB storage as follows: , , The cost of deploying a microservice instance is... It is defined as the sum of the products of its required resources and corresponding costs:

[0067]

[0068] Therefore, the current total deployment cost Defined as the sum of the costs of all deployed instances:

[0069]

[0070] The second step is to propose a time-series prediction model, ATimeMixer, based on improvements to TimeMixer, in order to obtain the future workload of microservices.

[0071] Step (2.1) First, the workload of a microservice is defined as the number of calls received by all instances of the microservice per unit of time, thereby obtaining the time-series characteristics of the microservice workload. ,in, Indicates the length of the time series. This represents the number of variables, i.e., the total number of microservices. , indicating microservices The workload timing characteristics.

[0072] Step (2.2) uses a sliding window to perform the following steps along the time sequence: The process involves partitioning the data to obtain the input temporal features and the target temporal features, which are represented as follows: , ,in , The time series lengths of both are given.

[0073] Step (2.3) for a given application topology graph ,make The initial features are those of the microservice nodes in the graph. Then, the graph attention network described in steps (2.3.1-2.3.2) is used to update the node features to obtain enhanced features containing dependency information. .

[0074] Step (2.3.1) Based on the graph structure, the features of the central node are... Features of neighboring nodes Concatenate the data and combine it with the attention weight vector. The dot product is used to calculate the attention score. Then, following... Normalize the calculation to obtain attention weights. The formula is as follows:

[0075]

[0076] in, This represents the natural exponential function. Represents a non-linear activation function. This represents the attention weight vector, which is essentially a feedforward neural network. This indicates the transpose operation. This represents the learnable weight matrix. This represents the learned node feature vector. This indicates a splicing operation. Represents a node The neighborhood of a node is defined as a node. The set of itself and its upstream microservice nodes, where the latter represents the set of nodes if there exists a directed edge that makes the node... Pointing to node ,but It is used to sense the load transfer caused by load cascading effects.

[0077] Step (2.3.2) utilizes attention weights based on the graph structure. The features of neighboring nodes are weighted and summed to aggregate neighborhood information, and the node features are then updated. The formula is as follows:

[0078]

[0079] in, This represents the activation function.

[0080] Step (2.4) utilizes a one-dimensional convolution function Enhanced input time-series features representing past observation data Iterative process A downsampling step of 2 is performed to complete the multi-scale decomposition and obtain the multi-scale input temporal feature set. The sampling process is as follows:

[0081]

[0082] Wherein, convolution stride Setting it to 2 maximizes the number of scales obtained by recursive downsampling. , represents the lowest level of sequence, which contains the finest temporal variations; while the highest level of sequence It represents the most macroscopic changes over time.

[0083] Step (2.5) inputs the multi-scale temporal feature set Projected to the embedding layer This yields a multi-scale deep feature set. .

[0084] Step (2.6) utilizes stacked Past Decomposable Mixing (PDM) blocks to perform steps (2.6.1-2.6.5), for Performing multi-level past decomposition and mixing is formalized as follows:

[0085]

[0086] in, Indicates the total number of floors. This indicates the PDM block of the corresponding layer.

[0087] Step (2.6.1) uses a one-dimensional learnable convolutional integral to solve the kernel. Decomposed into trend components The formula is as follows:

[0088]

[0089] Convolution uses padding to maintain consistency in temporal length before and after convolution.

[0090] Step (2.6.2) is performed by... Subtract Obtain seasonal portions ,Right now .

[0091] Step (2.6.3) and These represent long-term and short-term changes, respectively. Therefore, the mixing operation is applied separately to each component to effectively analyze and mix information across different scales. A top-down-mixing (TDM) approach is employed to apply macroscopic insights from short to long scales to guide the modeling of trends at a more refined scale. A bottom-up mixing approach (BUM) is employed to supplement short-scale seasonal modeling with detailed information obtained from a long-scale microscopic perspective. The mixing process is shown below:

[0092]

[0093] in, , It is implemented as a single-linear-layer multi-layer perceptron (MLP) to achieve scale alignment.

[0094] Step (2.6.4) involves re-superimposing the components at each scale, performing information interaction using MLP, and then accumulating the results. Perform residual connections to output The formula is as follows:

[0095]

[0096] in, and This indicates the component mixing process described in step (2.6.3). It is an MLP consisting of two linear layers, using It is used as an activation function for information interaction.

[0097] Step (2.6.5) repeats steps (2.6.1-2.6.4), after which... After layer-by-layer decomposition and mixing, the final multi-scale past information is obtained. .

[0098] Step (2.7) Because time-series features at different scales exhibit different dominant changes, their prediction capabilities also differ. To fully utilize this multi-scale information, ATimeMixer configures a separate predictor for each scale. The prediction process is as follows:

[0099]

[0100] in, Implemented as a single-linear-layer MLP, directly input Mapping to prediction length ,Right now .

[0101] Step (2.8) accumulates the predicted temporal features at multiple scales. Mapped to the final predicted time-series features The formula is as follows:

[0102]

[0103] Step (2.9) defines the loss as the target temporal feature. and predicting time series features The mean square error is as follows:

[0104]

[0105] Step (2.10) train ATimMixer to obtain accurate time-series features for predicting future microservice workloads. .

[0106] The third step is to design a DDPG agent that combines real-time and predictive information to make optimized decisions.

[0107] Step (3.1) First, construct the basic components of DDPG, including the policy network. Evaluation network Target Policy Network Target evaluation network and experience replay pool .

[0108] in, Acceptance status The input is a neural network that outputs a unique action. . Accepting status and actions The input is the Q-value of the action, which is used to evaluate the currently generated action, thereby assisting in... Training. and Indicates and and A target network with the same structure but delayed updates is used to compute the corresponding target value to stabilize the learning process. Experience gained during storage interaction It is used for sampling and playback to train the DDPG agent.

[0109] Step (3.2) Create a temporary buffer And record the current microservice instance deployment scheme. .

[0110] Step (3.3) Observe the environment and obtain the initial state. and the prediction results of ATimeMixer. .in, Includes graph structure data and the resource vectors of edge computing nodes. .

[0111] Step (3.4) sets the initial state Input Policy Network Proceed to steps (3.4.1-3.4.9) to output the action. Among them, actions It is A dimensional vector representing the common output. Each action. Elements corresponding to the deployment scheme , indicating microservices At edge computing nodes The number of pre-deployed instances is compared with the current instance deployment plan of the system to determine whether the microservice should be scaled up or down, and is defined as follows:

[0112]

[0113] Step (3.4.1) First, for the graph structure data existing in the state space, that is, a given application topology graph. Its node information and structural information can be represented as .in The graph adjacency matrix records the call dependencies between microservices; This is a microservice node feature matrix, containing relevant information about microservices, where... This represents the current microservice workload vector. , , , representing the CPU, memory, and storage resource requirements vectors of a microservice, respectively. Furthermore, this feature incorporates the predicted future microservice workload. Finally, in order to perform scaling stability training, this invention uses a temporary buffer. Take the two most recent deployment schemes and formalize them as follows: Because it was just initialized Only the current deployment scheme exists, therefore, it is retrieved twice. Perform repeated assignments. Subsequently, when If there are sufficient historical deployment plans available, the selection can be made normally.

[0114] Step (3.4.2) is based on the graph structure, and the features of the central node are... Features of neighboring nodes Concatenate the data and combine it with the attention weight vector. The dot product is used to calculate the attention score. Then, following... Normalize the calculation to obtain attention weights. The formula is as follows:

[0115]

[0116] in, This represents the learnable weight matrix. This represents the learned node feature vector. This represents the attention weight vector, which is essentially a feedforward neural network. Represents a node The neighborhood of a node is defined as a node. itself and the node The node being called, i.e., the node has a directed edge such that the node... Pointing to node ,but .

[0117] Step (3.4.3) utilizes attention weights based on the graph structure. The features of neighboring nodes are weighted and summed to aggregate neighborhood information, and the node features are then updated. The formula is as follows:

[0118]

[0119] Step (3.4.4) repeats steps (3.4.2-3.4.3) until the desired result is achieved. Layered graph convolutions are used to obtain the final node feature vectors. .

[0120] Step (3.4.5) uses the global graph pooling function Transform node-level features into graph-level features The formula is as follows:

[0121]

[0122] Step (3.4.6) In order to further perceive resource constraints, Resource vectors of edge computing nodes Concatenation to obtain the final state feature vector :

[0123]

[0124] Step (3.4.7) uses MLP to map the state feature vectors to actions. .

[0125] Step (3.4.8) samples a random noise from a Gaussian distribution. and add to the action This introduces randomness into the selection of movements, ensuring movement exploration during the training phase.

[0126] Step (3.4.9) utilizes The activation function and the independent scaling method perform normalization, making the action... It meets its basic value range.

[0127] Step (3.5) If the action If the corresponding deployment plan does not meet the joint constraints, a reward will be given immediately. and the next state And the next prediction Setting it to "done" indicates the end of this round. Next, clear the temporary buffer. and in storage experience Then, immediately restart from step (3.2).

[0128] Step (3.6) Otherwise, proceed according to the action. The corresponding deployment plan adjusts the microservice instances and proceeds to the next time step to obtain the next state. Next prediction and rewards Among them, rewards Defined as a performance bonus Cost incentives and scaling stability rewards The composite function is as follows:

[0129]

[0130] in, Equals the request success rate. , The reward for expanding and contracting oscillations is defined as follows:

[0131]

[0132] It is based on the historical deployment scheme of storage. and current action Corresponding deployment scheme The direction of change is used to determine whether the strategy has potential oscillations. It is the penalty coefficient. The instance change matrix represents the deployment scheme. It is a sign function that returns whether each element is positive, negative, or zero. Represents the Hadamard product of matrices. Indicates a pair with A matrix of the same size containing only -1s. This indicates that the indicator function returns 1 if the condition is true, and 0 otherwise. Here, it is used to determine whether a matrix element is -1. This is used to count the number of elements with a value of -1 in the matrix. -1 indicates that the update directions of two consecutive updates are opposite. For example, if expansion occurs immediately after shrinking, there is a potential for oscillation, which needs to be penalized. The larger this value is, the more services experience "back and forth" changes in the number of instances. Finally, , , This represents the weights that sum to 1, and can be adjusted according to the focus.

[0133] Step (3.7) organizes the relevant information into experience. And store it in the experience replay pool. .

[0134] Step (3.8) will Corresponding deployment scheme Put into temporary buffer Thus obtain .

[0135] Step (3.9) sets the initial state Replace with the next state And repeat steps (3.4-3.8).

[0136] Step (3.10) When the stored experience reaches the sample size At the same time, based on the interaction in step (3.9), steps (3.10.1-3.10.10) are performed simultaneously to train the model and update the relevant network parameters by sampling and replaying.

[0137] Step (3.10.1) Randomly sample a batch of experiences from the experience replay pool. , used for training.

[0138] Step (3.10.2) executes steps (3.10.2.1-3.10.2.3) to calculate the Q-value using the evaluation network, i.e.

[0139] Step (3.10.2.1) repeats steps (3.4.1-3.4.6) to obtain the processed state feature vector. The two most recent deployment schemes in step (3.4.1) are taken from The predicted future microservice workloads are derived from .

[0140] Step (3.10.2.2) will and splicing, i.e.

[0141] Step (3.10.2.3) will Input MLP to obtain Q value

[0142] Step (3.10.3) if If the target Q value does not exist, then...

[0143] Step (3.10.4) Otherwise, based on Utilize the target policy network to output actions ,Right now This process is consistent with step (3.4), where the two most recent deployment schemes in step (3.4.1) are taken from... The predicted future microservice workloads are derived from Furthermore, the target network does not need to add random noise in step (3.4.8) when outputting actions.

[0144] Step (3.10.5) calculates the target Q-value using the target evaluation network. This process is consistent with step (3.10.2).

[0145] Step (3.10.6) utilizes temporal difference learning, combined with rewards. Calculate the learning objective. ,in, This represents the discount factor.

[0146] Step (3.10.7) defines the evaluation network loss as... ,in This represents the expected loss of this sampling.

[0147] Step (3.10.8) uses stochastic gradient descent to update the evaluation network parameters. ,in Indicates the network learning rate. This represents the gradient generated by the loss.

[0148] Step (3.10.9) uses the policy gradient algorithm to update the policy network parameters, as shown in the following formula:

[0149]

[0150] Step (3.10.10) In order to gradually update the target network and ensure the stability of training, a soft update method is used to simultaneously update the target policy and the evaluation network. The parameters of the target network will accept a small portion of the changes in the corresponding network parameters, as shown in the following formula:

[0151]

[0152] in, It is the update coefficient.

[0153] Step (3.11) Repeat the training process of step (3.10) until the model converges.

[0154] After completing the above training in step (3.12), DDPG retains only the policy network, which is used to integrate the load prediction results of ATimeMixer for intelligent scaling decisions.

[0155] This embodiment analyzes the actual effects of the present invention using specific data, as follows:

[0156] 1) Experimental setup

[0157] First, this invention uses IFogSim2 to build an edge computing simulation platform. IFogSim2 is a Java-based toolkit for modeling and simulating resource management technologies in IoT, edge, and fog computing environments. Building upon the existing IFogSim simulation toolkit, IFogSim2 further integrates multiple simulation models related to service migration, dynamic distributed cluster formation, and microservice orchestration in edge / fog computing.

[0158] Secondly, IFogSim2 supports the integration of real-world datasets, such as the EUA dataset. The EUA dataset contains a large amount of location information, such as latitude and longitude, of edge servers and users in major Australian cities, including Melbourne and Sydney, and their central business districts. This invention selected the location information of 16 edge servers and nearby users located in Melbourne's central business district to provide data support for the simulation platform and experiments.

[0159] Inspired by Alibaba's work centers, this invention organizes seven different request structures and sizes, encompassing eight microservices, to simulate IoT applications. In request simulation, this invention follows the arrival rate... The Poisson distribution is observed. Secondly, to establish appropriate deadlines for various requests within the edge computing platform, this invention examines the average service latency for each request. and multiply it by the appropriate deadline factor. To establish a reasonable deadline .

[0160] Finally, this invention implements TDScaler using PyTorch and integrates it into the simulation platform. All experiments were conducted on a server running Windows 11, equipped with an AMD R9-7945HX CPU workstation with a frequency of 2.50GHz, 16 GB of memory, and an NVIDIA GeForce RTX 4060 GPU.

[0161] 2) Compare to baseline

[0162] First, to evaluate the performance of ATimeMixer in predicting future workloads of microservices, this invention provides the following baselines, including TimeMixer, DLinear based on MLP decomposition, Informer and Autoformer based on Transformer architecture; and Long Short-Term Memory (LSTM) and Gate Recurrent Unit (GRU) based on Recurrent Neural Network (RNN).

[0163] Secondly, to further verify the scaling performance of TDScaler, this invention selected the following baseline methods for comparison, including:

[0164] HPA: A reactive autoscaling technology supported by Kubernetes that adjusts the number of container instances based on resource utilization thresholds and cooldown times.

[0165] PCA: This method uses a bidirectional long short-term memory (Bi-LSTM) network to predict future workloads and relies on a single threshold to adjust the number of microservice instances.

[0166] ProScale: This algorithm uses the average workload over the most recent N time slots as the prediction result for the next time slot, and employs a corresponding heuristic online algorithm to determine the number of instances of each microservice and its placement decision in the edge computing environment. Furthermore, the algorithm also incorporates an emergency processor to handle unpredictable request spikes.

[0167] GeoScale: This algorithm is used for automatic scaling of microservices in edge environments with a focus on Quality of Service (QoS), aiming to optimize average request response time under long-term cost budget constraints. It first uses the Lyapunov optimization framework to decompose the long-term optimization problem into a series of subproblems within each time slot, and then applies an algorithm based on symbolic geometric programming to obtain near-optimal solutions to each NP-hard subproblem.

[0168] IReS: This method uses Dlinear to predict the workload of microservices and dynamically adjusts the number of microservice instances based on a threshold.

[0169] ARAScaler: This algorithm uses an improved TimeMixer model, ETimeMixer, to predict future workloads. Furthermore, it segments the predicted workloads to identify bursty, non-burst, dynamic, and static states, and scales by calculating the optimal number of container instances for each identified state.

[0170] 3) Evaluation Indicators

[0171] First, to evaluate the performance of ATimeMixer in predicting future workloads of microservices, this invention provides the following metrics, including Mean Squared Error (MSE) and Mean Absolute Error (MAE):

[0172]

[0173]

[0174] in, Indicates the predicted value. Represents the actual value.

[0175] Secondly, to evaluate the scaling performance of TDScaler, this invention provides the following metrics, including request success rate and deployment cost. The former is based on the following definition of statistics:

[0176]

[0177] in, It is a binary variable, and the delay depends on the actual request. Does the deadline meet? Constraints are used to determine whether a request is successful or not.

[0178] 4) Experimental Results

[0179] Table 1 presents the experimental comparison results of the time series prediction performance between the ATimeMixer method proposed in this invention and the baseline method. The evaluation metrics used are MSE and MAE. The best results for each metric are indicated in bold, and the second-best results are indicated by underline.

[0180] Table 1

[0181]

[0182] The data in the table shows that ATimeMixer achieves the best results in both MSE and MAE metrics compared to other deep learning models. Specifically, LSTM and GRU are both models based on improvements to RNNs. These models, due to their inherent recurrent structure, provide an intuitive and natural framework for sequence modeling; however, their inherent limitations in sequential computation and long-range dependency decay severely restrict their performance. Therefore, compared to other advanced models, LSTM and GRU have the worst prediction performance. Informer and Autoformer are both models based on improvements to the Transformer architecture. Their core attention mechanism can directly capture the global dependencies between any two positions in a sequence. These features overcome the inherent limitations of RNNs, achieving further performance breakthroughs. However, while the development of the above models improves performance, it also increases model complexity. Dlinear, however, simplifies the time series prediction algorithm, achieving better performance based on a linear model by leveraging the core mechanism of time series decomposition. TimeMixer, in addition to inheriting the core mechanism of Dlinear, adds linear multi-scale time series analysis, further improving its prediction performance. Finally, ATimeMixer enhances its ability to capture temporal features by utilizing convolution-based downsampling and time-series decomposition. Secondly, by employing a graph attention network, ATimeMixer explicitly models microservice dependencies, enabling it to detect load shifts caused by potential cascading effects. This invention argues that these two core improvements give ATimeMixer a performance advantage, reducing MSE and MAE metrics by 8.61% and 3.87% respectively compared to the best baseline, TimeMixer.

[0183] As can be seen from the definition of reward, the weight parameters To adjust the priority of each objective, a grid search strategy was employed in the composite reward weight experiment to find the optimal combination. First, a coarse-grained search with a step size of 0.1 was performed within the range of 0-1 to determine an initial optimal parameter range. Then, a fine-grained search with a step size of 0.05 was performed within this range to determine the optimal parameters. Table 2 shows some weight parameter combinations during the grid search process and illustrates the system performance metrics for each parameter combination, including request success rate and deployment cost.

[0184] Table 2

[0185]

[0186] The data in the table shows that when the weight combination is... At that time, it is most appropriate. First, fix it. It can be observed that when At times, the cost-effectiveness of improving request success rate is not high; while when decline, When costs rise, the impact on request success rate is greater than on deployment cost. Secondly, fixed... It can be found that when Continued decline While costs continue to rise, the impact is minimal, but request success rates continue to decline. Finally, fixed... Similarly, it can be observed that costs will increase with... The reduction in size significantly increases the performance. Therefore, based on the above analysis and the emphasis on performance, this invention selects... As an experimental setup.

[0187] Table 3 presents the scaling performance comparison results of TDScaler and the baseline from the perspectives of request success rate and deployment cost. Compared to TDScaler, HPA is a reactive algorithm, and its scaling behavior has a delay. This inherent defect is further amplified in the complex IoT environment, resulting in a significant impact on both its request success rate and deployment cost. Secondly, as predictive algorithms, ARAScaler and IReS both exhibit an oversupply trend, leading to a gradual increase in deployment costs without a significant improvement in request success rate. PCA, on the other hand, shows a clear undersupply of predictions. While this reduces costs, it also harms system performance, leading to a decrease in request success rate. ProScale's mean-based prediction, although simple and fast, is insufficient to handle the complex and variable workloads in the IoT environment, thus its scaling performance is inferior to TDScaler. GeoScale emphasizes long-term cost budget constraints, which brings it a cost advantage but also leads to conservative scaling decisions, ultimately affecting performance. Finally, thanks to the fusion of time-series prediction and reinforcement learning, as well as targeted improvements, TDScaler achieves scaling performance superior to the baseline. Compared to the best baseline ARAScaler, TDScaler further reduces deployment costs by 3.26% while maintaining a similar request success rate.

[0188] Table 3

[0189]

[0190] Figure 2 These are the hyperparameter experimental results for TDScaler. This experiment explored the impact of the number of graph convolutional layers. The graph convolutional preprocessing layer in ATimeMixer is used to detect potential workload shifts from upstream microservices; therefore, this invention only applies a single convolutional layer to it. The graph convolution in the DDPG agent part is the focus of this experiment because the representation of the state feature vector greatly influences the agent's decision-making. Its impact on request success rate and deployment cost is as follows: Figure 2 As shown in (a) and (b), the request success rate initially increases and then fluctuates with the increase in the number of convolutional layers, while the deployment cost initially decreases and then increases. Specifically, when the number of convolutional layers increases from 2 to 3, although the deployment cost increases slightly, the improvement in the request success rate is relatively significant. Therefore, based on the above analysis, this invention selected 3 convolutional layers as the final experimental setting.

[0191] Table 4 presents the ablation experiment results for TDScaler. Specifically, TDScaler-1, which removes ATimeMixer, eliminates the fusion mechanism, allowing the agent to make decisions solely based on the current state. TDScaler-2, which removes the graph attention network from ATimeMixer, no longer explicitly models microservice dependencies, treating variables as independent. TDScaler-3, which removes the convolutional downsampling from ATimeMixer, reuses average pooling-based downsampling. TDScaler-4, which removes convolution-based time series decomposition, reuses moving average kernel-based time series decomposition. TDScaler-5, which removes the graph attention mechanism, treats defined neighbor nodes equally based on their degree.

[0192] Table 4

[0193]

[0194] Analysis of the data in the table reveals that TDScaler-1 experienced the most severe performance degradation. This is because the agent lacked a future-oriented perspective, resulting in a lack of forward-looking decision-making. TDScaler-2 lacked correlation modeling between variables and failed to perceive the potential load cascading effects brought about by microservice dependencies, leading to a decrease in the accuracy of ATimeMixer's time-series prediction results, which in turn affected the agent's decision-making. Compared to convolution operations, TDScaler-3 and TDScaler-4's time-series decomposition based on average pooling downsampling and moving average kernels can only average out features. This makes it unable to consider non-stationary features, weakening ATimeMixer's ability to capture time-series features, and thus affecting the agent's performance after fusing future information. Finally, TDScaler-5 removed the graph attention mechanism, causing nodes to treat all neighbors equally when aggregating neighborhood information. This limits the flexibility of aggregation and lacks certain accuracy, ultimately affecting the model's scaling performance. In summary, the variants, due to the lack of optimization components, suffer from decreased prediction or learning performance. This ultimately leads to either an oversupply of instances resulting in decisions, causing increased costs without affecting the request rate, or an undersupply, which, while reducing costs, significantly impacts the request success rate. The TDScaler model, however, integrates the advantages of each component, achieving superior prediction and decision-making performance and enabling better automatic scaling.

[0195] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Various equivalent substitutions or modifications made by those skilled in the art without departing from the spirit and substance of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for automatic scaling of edge microservices that integrates temporal prediction and reinforcement learning, characterized in that, Includes the following steps: Step 1: First, model the problem scenarios for the edge computing platform, microservice applications, deployment schemes, and deployment costs. Step 2: Based on the TimeMixer model, the ATimeMixer time series prediction model is improved. Convolution-based downsampling is used to replace the original average pooling downsampling in TimeMixer, and convolution-based time series decomposition is used to replace the original moving average kernel time series decomposition. At the same time, a graph attention mechanism is introduced to explicitly model the dependencies between the time series of workloads of each microservice. The future workload of microservices is obtained through multi-scale decomposition, trend-seasonal component separation and mixing, and multi-scale independent predictor fusion. Step 3: Construct a topology-aware DDPG agent, incorporating future workload predictions from the ATimeMixer model as future information into the decision-making process, while introducing the instance deployment history of the past few steps as state information into the agent's observation space to perceive the recent state; construct a composite reward function that includes service performance, deployment cost, and scaling stability, and introduce a penalty term for potential scaling oscillations into the reward function to suppress frequent fluctuations in the decision-making process, and guide the agent to train offline through the composite reward function; After training, the real-time system status and future workload prediction information are used as inputs, and the policy network outputs a microservice instance scaling decision that satisfies the resource constraints of edge nodes and the deployment constraints of microservice instances.

2. The automatic scaling method for edge microservices that integrates temporal prediction and reinforcement learning according to claim 1, characterized in that, Step 1 is described in detail as follows: Step (1.1) definition This is an edge computing platform represented by an undirected graph, where... It represents a finite set of edge computing nodes, each of which has an independent number of available CPU cores, memory size, and storage capacity; It is a finite set of edges defined by different node pairs, representing the network topology relationships between nodes; Step (1.2) definition This is an IoT application developed based on a microservice architecture, in which... For all the microservices included in the application, each microservice is characterized by a set of tasks, an instance and its relative location on the edge computing platform, and the number of CPU cores, memory size and storage capacity required per instance. The application topology graph is modeled using a directed acyclic graph, and the set of directed edges is used to describe the call relationships between microservices. Step (1.3) definition The deployment scheme for microservice instances is a matrix, where rows represent microservices and columns represent edge computing nodes. The matrix elements represent the number of deployment instances of the corresponding microservice on the corresponding edge node, and the deployment scheme satisfies the following conditions: each microservice has at least one instance deployed, and the resource usage of the edge node does not exceed its supply capacity. Step (1.4) Calculate the total deployment cost of the microservice instances based on the unit resource cost and the number of microservice instances deployed.

3. The automatic scaling method for edge microservices that integrates temporal prediction and reinforcement learning according to claim 1, characterized in that, Step 2 is described in detail below: Step (2.1) defines the workload of a microservice as the number of calls received by all instances of the microservice per unit time, thereby obtaining the time-series characteristics of the microservice workload; Step (2.2) uses a sliding window to divide along the temporal direction to obtain the input temporal features and the target temporal features; Step (2.3) utilizes a graph attention network based on the microservice application topology graph. By aggregating the features of microservice nodes and their upstream nodes, enhanced temporal features with fused topological dependency information are obtained; Step (2.4) performs multi-scale downsampling on the enhanced temporal features through one-dimensional convolution to obtain a multi-scale input temporal feature set; Step (2.5) projects the multi-scale input temporal feature set onto the embedding layer, thereby obtaining the multi-scale deep feature set; Step (2.6) uses stacked past decomposition mixed PDM blocks to decompose the multi-scale deep features into trend components and seasonal components. The trend components are mixed from top to bottom using TDM and the seasonal components are mixed from bottom to top using BUM to achieve multi-scale information fusion. After the components are re-merged, the residual connection is combined to output the processed multi-scale features. Step (2.7) configures an independent predictor for each scale and outputs the scale-aligned multi-scale load prediction time series features; Step (2.8) maps the multi-scale predicted temporal features to the final predicted temporal features through accumulation; Step (2.9) defines the loss as the mean square error of the target time series feature and the predicted time series feature; Step (2.10) train ATimMixer to obtain accurate future prediction time-series features of microservice workloads.

4. The automatic scaling method for edge microservices that integrates temporal prediction and reinforcement learning according to claim 1, characterized in that, Step 3 is described in detail below: Step (3.1) Construct the basic components of DDPG, including the policy network, evaluation network, target policy network, target evaluation network, and experience replay pool; Step (3.2) Create a temporary buffer to record historical microservice instance deployment schemes for scaling stability assessment; Step (3.3) Observe the environment and obtain the initial state. And the prediction results of ATimeMixer; initial state Includes graph structure data And the resource vectors of edge computing nodes; Step (3.4) sets the initial state The input policy network, through a graph attention network, fuses information on microservice topology, current load, future load prediction, and historical deployment schemes. This information is then concatenated with edge node resource vectors, processed by a multilayer perceptron (MLP) and normalized, to output microservice instance scaling actions. ; Step (3.5) Determine the expansion / contraction action If the corresponding deployment plan meets the constraints, a penalty or reward is given and the current decision is terminated, and the process restarts from step (3.2); if it meets the constraints, the scaling action is executed, and the process proceeds to the next time step. Step (3.6) Calculate the reward for this round. The reward is a weighted sum of performance reward, cost reward and scaling stability reward. The performance reward corresponds to the request success rate, the cost reward is the negative value of the total deployment cost, and the scaling stability reward is based on the instance change direction of two consecutive deployment schemes to penalize oscillation behavior. Step (3.7) stores the system interaction experience into the experience replay pool. The system interaction experience includes the current system status, the current load prediction result, the two most recent historical deployment schemes, the current scaling action, the current round of rewards, the next system status, and the load prediction result at the next moment. Step (3.8) will involve expansion and contraction. Corresponding deployment scheme Add to the temporary buffer and update the deployment plan for the temporary buffer; Step (3.9) sets the initial state Replace with the next state And iterate the decision-making process; Step (3.10) When the experience in the experience replay pool reaches the preset sampling size Simultaneously, experience sampling and playback are initiated to train the evaluation network and policy network, and the target policy network and target evaluation network are updated using a soft update method. Step (3.11) Repeat the training process of step (3.10) until the model converges; After completing the above training in step (3.12), DDPG retains only the policy network, which is used to integrate the load prediction results of ATimeMixer for intelligent scaling decisions.

5. The automatic scaling method for edge microservices that integrates temporal prediction and reinforcement learning according to claim 1, characterized in that, In step (2.3), the neighborhood of the graph attention network is defined as the microservice node itself and its upstream microservice nodes.

6. The automatic scaling method for edge microservices that integrates temporal prediction and reinforcement learning according to claim 1, characterized in that, In step (2.6), the PDM block decomposes the trend component and the seasonal component through a learnable convolutional kernel, and both TDM and BUM achieve scale alignment and information interaction through a single-linear-layer multilayer perceptron (MLP).

7. The automatic scaling method for edge microservices that integrates temporal prediction and reinforcement learning according to claim 1, characterized in that, The scaling action in step (3.4) corresponds to the flattening vector of the deployment scheme matrix, which directly represents the target number of microservices on each edge node.

8. The automatic scaling method for edge microservices that integrates temporal prediction and reinforcement learning according to claim 1, characterized in that, In step (3.6), the scaling stability reward is determined by judging whether the instance change direction of two consecutive deployment schemes is opposite, thus quantifying the penalty for oscillation behavior.

9. The automatic scaling method for edge microservices that integrates temporal prediction and reinforcement learning according to claim 1, characterized in that, Both the policy network and the evaluation network encode the microservice application topology graph through a multi-layer graph attention network. The neighborhood of the graph attention network is defined as the microservice node itself and its downstream calling nodes. The encoded graph-level features are concatenated with the edge node resource vectors as network input.

Images