Server non-perception resource scheduling method and system based on cluster telescopic adaptive reinforcement learning

Abstract

The invention discloses a server non-perception resource scheduling method and system based on cluster telescopic self-adaptive reinforcement learning. The method comprises the following steps: step 1, using a monitoring assembly based on a time sequence database to collect various operation load indexes of a cluster server; and step 2, constructing a function resource scheduler by using a Kubernetes scheduling framework. And step 3, constructing a scheduling decision agent by using a cluster telescopic adaptive reinforcement learning algorithm, inputting a variable-length observation sequence, and outputting a node deployment action. And step 4, using a historical cloud service load data set to construct a load generator to simulate a production environment to train and strengthen an intelligent agent deep neural network. And 5, using the trained scheduling decision agent to make a production environment scheduling decision. According to the method, the existing reinforcement learning algorithm is subjected to elastic cluster-oriented algorithm optimization, the expansibility and robustness of the scheduling algorithm when the cluster executes node scaling are improved, and the method has practical application value.

Description

Technical Field

[0001] The present invention belongs to the technical field of Internet cloud computing, and particularly relates to a serverless resource scheduling method and system for cluster scaling adaptive reinforcement learning. Background Art

[0002] Serverless computing is a new generation of cloud computing model, in which Function as a Service (FaaS) is an important product form in serverless. Different from the traditional microservices architecture, FaaS shrinks the concept of service in the microservices framework to the function level. Each function is responsible for a specific function, runs in a resource-constrained instance, and uses an event-triggered method for remote calls. This model can provide more powerful concurrency capabilities, elastic scheduling capabilities, and error tolerance capabilities, hide the tenant's perception of cloud resources, and liberate developers from cumbersome infrastructure management, so as to focus on the logical development of business functions. In recent years, serverless computing has been applied in the fields of Internet of Things, streaming media processing, etc. In a distributed system, resource scheduling plays a crucial role in exerting the performance and elastic scheduling of the computing cluster.

[0003] A resource scheduler refers to the process of reasonably allocating these resources to jobs or tasks according to the available situation of computing resources and the requirements of jobs. This process involves aspects such as resource management, load balancing, and performance optimization. Compared with traditional cloud computing, the resource granularity managed by serverless computing is finer, and there are challenges such as frequent scheduling, cold start latency, and complex dependency state management. Therefore, in recent years, function scheduling methods based on reinforcement learning have received more attention. This method exhibits stronger adaptive capabilities than traditional heuristic resource scheduling methods, can learn heterogeneous resource environments and computing requirements, dynamically adjust scheduling strategies, reduce the service level objective violation rate, improve the utilization rate of computing resources, and reduce the computing costs of cloud service providers.

[0004] Cluster scaling refers to increasing the capacity and performance of the system and improving the system processing ability by increasing the number of nodes in the cluster. When the required request concurrency for deploying services increases, the operator provides sufficient computing resources by increasing the number of service processing nodes to provide users with concurrent processing capabilities that meet their needs.

[0005] However, the existing function resource scheduling methods based on reinforcement learning lack consideration of the situation of cluster scaling. The action space of its agent is usually represented as the number of currently schedulable nodes, without considering the coping strategies when the cluster makes scaling, that is, when the cluster nodes are added or reduced, the existing reinforcement learning methods cannot well adapt to the changes, and problems such as the inability to place function instances on newly added nodes and the inability to include removed nodes in the scheduling target will occur. Summary of the Invention

[0006] To overcome the deficiencies of the above-mentioned prior art, the purpose of the present invention is to provide a server-unaware resource scheduling method and system based on cluster scaling adaptive reinforcement learning. When dealing with triggering cluster scaling, this method can sense the addition and deletion of nodes, reduce the convergence time of the learning reward value of the scheduling agent, and improve the utilization rate of cloud resources.

[0007] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0008] A server-unaware function scheduling method and system based on cluster scaling adaptive reinforcement learning, including the following steps:

[0009] Step 1: Use a monitoring component based on a time series database to collect various operating load indicators of cluster servers.

[0010] Step 2: Use the Kubernetes scheduling framework to build a function resource scheduler.

[0011] Step 3: Use a cluster scaling adaptive reinforcement learning algorithm to build a scheduling decision agent, input a variable-length observation sequence, and output a deployment node action.

[0012] Step 4: Use the historical cloud service load data set to build a load generator to simulate the production environment and train the deep neural network of the reinforcement agent.

[0013] Step 5: Use the trained scheduling decision agent to make scheduling decisions in the production environment.

[0014] The said Step 1 includes: using Prometheus to realize the collection and storage of load indicators. Install the Prometheus server on the cluster master node, and install the node-exporter client on each node of the cluster to collect data, and use the active push method of the client for data collection.

[0015] Furthermore, configure the data source, push address, and data types collected by node-exporter. By collecting various indicators such as the system CPU usage rate, user CPU usage rate, network card in and out traffic, memory usage rate, and system load provided by the local operating system of the collection client, submit them to the Prometheus server and persist them in the time series database. External components obtain the required indicator data through the open external interface provided by the Prometheus server.

[0016] Step 2 includes: using the Kubernetes distribution Microk8s to build and manage the cluster. Taking OpenFaaS as the FaaS application platform, the function interacts with the Kubernetes API through the faas-netes component to complete the selection of resource nodes. When the OpenFaaS platform and the corresponding service functions are deployed, function instances will not be generated in the Kubernetes cluster before being called by the user; when the user initiates a call, first, OpenFaaS notifies the Kubernetes cluster through the faas-netes component, calls the resource scheduler registered in the cluster, and outputs the scheduling result according to the algorithm corresponding to the scheduler. Then, resources are allocated to the function container instance according to the function resource configuration when registering to the OpenFaaS platform. Finally, when the resource allocation is completed, the user's request is redirected by the gateway to the function container instance for calculation and finally the result is returned.

[0017] Furthermore, the present invention customizes the scheduling plugin through the Kubernetes scheduling framework and binds it to the function namespace of OpenFaaS to achieve custom resource scheduling. In the Kubernetes scheduling framework, the scheduler mainly selects the resource deployment node through the pipeline process of node filtering (Filter) and node score ranking (Score), and rewrites the Score interface of the scheduling framework to implement the external hot-plugging and mounting method of the custom scheduling agent.

[0018] Furthermore, the scheduler sends a request to the separately deployed reinforcement learning agent, and the agent returns the scheduling node result to the scheduler. The scheduler ranks the nodes according to the result. Among them, the node score corresponding to the result returned by the agent is set to the maximum value, and the scores of the remaining nodes are ranked according to the default method. Finally, Kubernetes selects the node with the maximum score (i.e., the decision result of the agent) according to the score ranking for resource scheduling.

[0019] Step 3 includes: using a reinforcement learning algorithm with cluster scaling adaptability to build a deep reinforcement learning scheduling decision agent.

[0020] Furthermore, the reinforcement learning algorithm with cluster scaling adaptability uses a perception neuron based on the attention mechanism to process variable-length observation sequences in the input layer, and uses a node mapping method based on hardware capability quantization to complete action encoding in the output layer, and finally obtains the decision action of the agent.

[0021] Furthermore, in the specific reinforcement learning algorithm, the proximal policy optimization algorithm (PPO) is used. Its core mainly consists of two neural networks, namely the policy network (Actor) and the value network (Critic). The two networks are composed of a layer of attention neurons (AttentionNeuronLayer) and three layers of MLP. Each layer consists of 64 neurons. The attention neurons are used as the input layer, and the input dimension is set to d input =M×3+2, where M is the maximum number of nodes that the cluster can accommodate.

[0022] The learning process of reinforcement learning is further explained. First, parameters are initialized. By executing a policy in the environment, a series of data on states, actions, rewards, and next states is collected. This process is repeated over multiple rounds to complete network parameter training. Each round of training calculates an advantage function to evaluate the quality of actions. The collected empirical data is used to update the critic network, and the restricted policy gradient is calculated. After several rounds of training, the actor network parameters are updated. This process is repeated until the two neural networks converge.

[0023] Furthermore, the three elements of reinforcement learning are described:

[0024] Environmental state: O={F,RT,o1,o2,…,o N}, where N represents the number of currently schedulable nodes, Represents the Pod resource utilization of the selected function instance. RT represents the difference between the current function SLO response time and the actual average response time. i ={RU mem ,RU net ,RU cpu}, RU r Represents the average utilization of resource r at the current node i. The status values are all normalized.

[0025] Action Set: A c =[-1,1], by setting the Gaussian strategy activation function at the output of the agent strategy network, the mean μ and variance σ of the Gaussian distribution in the continuous action space are output, and the action noise is added by the standard deviation as a parameter to complete the random sampling of the action to obtain the continuous action value A c .

[0026] Reward function: The reward value calculation formula is:

[0027]

[0028] Where L is the user-set function calculation delay threshold (also known as the SLO violation threshold), L a$\overline {L}$ is the average execution delay of the function to be migrated at the current moment, $R$ represents the resource set (CPU, memory, network, and disk), $RU_{ij}$ represents the average utilization rate of resource $i$ in node $j$, and $N$ is the number of resources of the currently schedulable nodes.

[0029] Further, a heuristic computing power weighted evaluation method is used to process the continuous action space output by the reinforcement learning agent, and the result is mapped to the corresponding discrete value of the specific deployment node to obtain the decision deployment node result. The specific description is as follows: Find a one-to-one mapping relationship $f:A$ c $\to A$ such that the discrete set $A = \{1, 2, 3, \ldots, N\}$ selected by the deployment node is mapped to the continuous action interval $A$ c $\in [-1, 1]$. For the construction problem of the mapping $f$, in this study, the $N$ nodes are intuitively statically numbered and evenly distributed within the interval, so that each node has the same probability of being selected and gradually converges to the optimal decision value during iterative training. In the discrete action design of reinforcement learning based on value function approximation (such as DQN), the action values are unordered, and the strategy for the agent to select actions remains unchanged (such as the $\epsilon$-greedy method used in the DQN method), and the designer does not need to consider the orderliness of each action value. However, in the reinforcement learning algorithm based on policy gradient, when dealing with the continuous action space, an unreasonable action distribution is likely to cause the result to fall into a local optimal solution.

[0030] Further, based on the action coding method of hardware capability quantization of the present invention, the deployment nodes are sorted and encoded. Through static analysis and quantitative evaluation of the task capacity that the node resources can carry, as shown in the following formula, the nodes are sorted according to the magnitude relationship of the quantitative evaluation values and then evenly distributed into the action interval.

[0031] $C$ i $= \alpha c$ i cpu $+ \beta c$ i mem $+ \gamma c$ i net $+ \eta (c$ i gpu $+ c$ i ssd $), i \in \{1, 2, \ldots, N\}$

[0032] where $c$ that describes the general common resource capabilities i cpu is the number of CPU cores of the node, $c$ i mem is the memory capacity of the node, $c$ i net is the total bandwidth of the node. In addition, it includes the heterogeneous hardware resource capability index of the node, $c$ i gpuIndicates whether the node is equipped with a GPU, c i ssd Indicates whether the node is equipped with an SSD. If so, the value is 1; otherwise, it is 0. The variables α, β, γ, and η represent the parameters controlling the weights of various indicators. Through the function f constructed by the above method, the discrete action A = {a i |i = 1, 2, 3, …, N} is obtained, where a i Indicates deploying the function instance to the computing node corresponding to the action value i.

[0033] Step 4 includes: pre-training the deep neural network of the reinforcement agent to enable the decision-making agent to have a good initial decision-making effect when deployed to the real production environment.

[0034] Furthermore, on the training dataset, the "2021 Azure Functions InvocationTrace" dataset is selected. This dataset records the function trigger records of the cluster for two weeks.

[0035] Furthermore, sampling and replay are performed on the dataset. Sampling means selecting several functions to replace the benchmark function, tracking their invocation times in the cluster, and obtaining a schedule of function trigger invocations. Replay means simulating the request program to send requests to the FaaS gateway built in the experimental environment according to the trigger schedule obtained by sampling.

[0036] Furthermore, in the invocation functions specifically used for training, the present invention uses benchmark functions in the field of serverless computing academic research: Base64: Encodes and decodes strings using the Base64 algorithm. Markdown2HTML: Renders Markdown text as HTML. Image-Resize: Scales the resolution of images. File-IO: Writes and uploads files to the local.

[0037] Furthermore, the training process of the deep reinforcement learning agent is described as follows: The monitoring component located at the cloud master node obtains the average resource utilization rate, average function execution delay, and other indicator information within a 15s time window from each node every 15s and stores it in the local database. Each time the scheduling agent receives a resource scheduling request, it simultaneously extracts the observation indicators at the previous moment from the historical indicator data, calculates the reward for the previous action, and updates the policy network parameters.

[0038] Another object of the present invention is to provide a system based on a serverless resource scheduling method of cluster scaling adaptive reinforcement learning. The system includes a function-as-a-service platform module, an indicator collection module, a scheduling decision module, and a scheduling execution module.

[0039] The function-as-a-service platform module implements the FaaS basic function platform, is hosted on the Kubernetes cluster management tool, carries the function functions required by users, receives user requests, and tries to respond within the scope of the user service quality.

[0040] The cluster metric collection module continuously monitors and records the usage of CPU, memory, network, and disk on the nodes, and returns the current environment status of the cluster when receiving a status acquisition request.

[0041] The resource scheduling decision-making module is the core component of the system. It is connected to two modules, namely the cluster metric collection module and the resource scheduling execution module. It is equipped with a reinforcement learning agent that integrates the node scaling adaptive mapping method and the proximal policy optimization algorithm, and undertakes the responsibility of resource scheduling decision-making.

[0042] The resource scheduling execution module receives the decision result of the resource scheduling decision-making module, implements the scheduling interface of the Kubernetes scheduling framework, and replaces the cluster default resource scheduling method with the method of the present invention.

[0043] Compared with the existing method, the beneficial effects of the present invention are as follows:

[0044] The present invention realizes serverless resource scheduling based on the reinforcement learning algorithm. Compared with traditional heuristic learning, it can optimize resource utilization and reduce the computing cost of the services provided by operators; the trained reinforcement learning model can make scheduling decisions in a short time and reduce the cold start delay of function instance resources; the reinforcement learning algorithm has the ability of learning and generalization, and can handle complex problems with dynamic changes, such as the cold start delay of serverless application resource scheduling and deployment, and the computing characteristics of different functions.

[0045] The present invention solves the action space, that is, the scheduling problem of the dynamically changing number of cluster nodes, by using the continuous action space for the output of reinforcement learning, provides the adaptive resource scheduling ability for the elastic cloud cluster, and improves the existing serverless resource scheduling method based on reinforcement learning. When the cluster undergoes horizontal elastic expansion and new nodes are added to the action space, by using the heuristic computing power weighted evaluation method to select a reasonable mapping strategy for the output action result of the deep neural network, the speed of the agent policy re-convergence can be improved, and the algorithm robustness can be further improved. It has a wide range of application prospects. Brief Description of the Drawings

[0046] Figure 1 It is a schematic diagram of the reinforcement learning server unaware resource scheduling method based on cluster scaling adaption provided by the embodiment of the present invention.

[0047] Figure 2It is the architecture diagram of the reinforcement learning serverless oblivious resource scheduling method based on cluster scaling adaptability provided by an embodiment of the present invention.

[0048] Figure 3 It is the schematic diagram of node scaling adaptability mapping provided by an embodiment of the present invention.

[0049] Figure 4 It is the schematic diagram of the scheduling executor scheduling process provided by an embodiment of the present invention. Detailed implementation manners

[0050] The following further elaborates on the specific implementation manners of the present invention in conjunction with the specification drawings and specific implementation manners. The following embodiments or drawings are used to illustrate the present invention, but not to limit the scope of the present invention.

[0051] In this system, the processing of signal data starts from the cluster metric collection module. The metric collection clients deployed on each node collect hardware metrics such as CPU, memory, network, and disk in real time. The collected raw data undergoes preliminary data cleaning and preprocessing to remove outliers and noise, ensuring the accuracy and consistency of the data. After standardization processing, each metric is converted into a unified dimension format, preparing for subsequent feature extraction and encoding.

[0052] The preprocessed data is further input into the node computing power evaluation encoding formula, which comprehensively evaluates the hardware information of each node. After the signal data undergoes feature extraction, each hardware metric is sorted according to a preset weight to form a node computing power evaluation value, and the sorting result is mapped to the discrete action space. This mapping process not only extracts key features but also realizes the conversion from continuous signals to discrete decision-making actions, providing a clear input signal for the decision-making of the reinforcement learning agent.

[0053] In the resource scheduling decision module, the processed signal data is fed into both the policy network (Actor) and the value network (Critic) simultaneously, and is screened by the attention input neurons to highlight key metric signals. The reinforcement learning agent is based on the proximal policy optimization algorithm (PPO), and through continuous training, it performs feature association and policy optimization on the input signal data, and generates control instructions applicable to cluster resource scheduling in real time. Throughout the process, the signal data not only serves as static features for model training but also participates in the feedback loop of dynamic decision-making, continuously improving the model's cognitive ability of the environment.

[0054] Finally, the processed and mapped signal data results are sent to the resource scheduling execution module by rewriting the Score interface in the Kubernetes scheduling framework, driving the scaling adaptability of each node in the cluster. At the same time, the pre-training load generation module uses the function call records in the predefined dataset to replay and simulate the load of the signal data to verify the effectiveness of the decision instructions and provide continuous training samples for the reinforcement learning agent. This closed-loop feedback mechanism ensures the efficient coordination of the entire process of signal data from collection, preprocessing, feature extraction to decision execution, realizing the seamless intelligent scheduling of cluster resources.

[0055] The system first deploys the FaaS platform module in the Kubernetes cluster to host various user functions. When a user sends a request, the platform automatically responds to the request according to the pre-configured function instances and completes the function call under the premise of meeting the service level objective (SLO). Through containerization technology, the platform realizes the rapid deployment and dynamic scaling of functions, thus meeting the high concurrency and resource elasticity requirements in the Internet cloud computing environment.

[0056] To monitor the cluster resource usage in real time, the system introduces a Prometheus server and metric collection clients deployed on each node. The clients regularly collect key metrics such as CPU, memory, network traffic, disk I / O, and system load, and push the data to Prometheus. When receiving a status acquisition request, the monitoring module can quickly return the current running status of the entire cluster, providing data support for subsequent resource scheduling.

[0057] The resource scheduling decision module is constructed based on a reinforcement learning agent that integrates the proximal policy optimization (PPO) algorithm. The agent has a policy network (Actor) and a value network (Critic) built in. Both of them use attention layers to extract redundant input information and are processed by multi-layer perceptrons to output the Gaussian distribution parameters (mean μ and variance σ) in the continuous action space. Subsequently, through the node scaling adaptability mapping method, the continuous action values are mapped to discrete deployment nodes. This mapping process first scores and sorts the hardware capabilities (CPU, memory, network, GPU, SSD) of each schedulable node according to the computing power evaluation coding formula, and then evenly distributes the sorted nodes in the interval (-1, 1), thus realizing efficient and accurate node selection.

[0058] The system realizes the external hot pluggability of the custom scheduling agent by rewriting the Score interface in the Kubernetes scheduling framework. When a scheduling request enters the Score interface, the interface will actively send a request to the resource scheduling decision module to obtain the decision result of the optimal deployment node. Subsequently, the scheduling execution module sets the score of this node to the maximum, and other nodes are arranged according to the default logic. Finally, the Kubernetes scheduler selects the best node based on the node scores to complete the function instance deployment, replacing the traditional default scheduling method, thereby improving the resource utilization rate and task execution efficiency.

[0059] To improve the initial effect of the scheduling model before deployment in a real production environment, the system is equipped with a pre-training load generation module. Based on the "2021 Azure Functions Invocation Trace" dataset, this module samples and replays function call records in a real environment to construct a function call schedule. By simulating calls to multiple benchmark functions (such as Base64, Markdown2HTML, Image-Resize, File-IO), the system regularly collects metrics such as resource utilization rate and function execution latency within a 15-second time window on the cloud master node. The agent calculates the reward based on historical data during each scheduling and updates the policy network parameters in real time, thus forming a good scheduling policy during the offline training phase.

[0060] Each module of this system works in coordination: the FaaS platform efficiently responds to user requests, the monitoring module provides real-time feedback on resource status, the reinforcement learning agent performs dynamic mapping of nodes based on environmental data and computing power evaluation, and the custom scheduling interface executes scheduling decisions; at the same time, the pre-training load generation module provides sufficient training data for the model to ensure that the scheduling policy has good adaptability during actual deployment. Overall, this method can not only significantly improve the resource utilization rate and reduce the cold start latency of function instances, but also flexibly handle complex dynamic environments, and is widely applicable to the fields of Internet cloud computing and distributed cluster technologies.

[0061] See Figure 2 It is a schematic diagram of the system architecture of the OpenFaaS platform for a reinforcement learning resource scheduling method based on cluster scaling adaptability for an elastic cluster provided by an embodiment of the present invention, showing the modules included in the system of the present invention and the connection relationships between the modules. The specific steps are as follows:

[0062] Step 1, use a monitoring component based on a time series database to collect various running load metrics of the cluster servers

[0063] Specifically, we use the Prometheus component in OpenFaaS as a monitoring component and time series database. By deploying node-exporter on each cluster node, we collect server node status information in real time. Furthermore, through the collection configuration, node-exporter captures the environmental monitoring data set O and sends it to the Prometheus server deployed on the master node, where it is stored in the master node's database.

[0064] Step 2: Use the Kubernetes scheduling framework to build a function resource scheduler.

[0065] Specifically, in the Kubernetes scheduling framework, the scheduling framework Score interface is rewritten. The implementation logic of this interface is as follows: when the scheduling executor receives a pod scheduling request, it initiates a request to the scheduling decision module deployed on the master node. The decision module obtains the result through the reinforcement learning network calculation and returns the scheduling node result to the scheduler. The scheduler scores and sorts the nodes according to the result. Among them, the node score corresponding to the result returned by the agent is set to the maximum value, and the scores of the remaining nodes are ranked according to the default method. Finally, Kubernetes selects the node with the largest score (that is, the agent decision result) for resource scheduling based on the score ranking.

[0066] Step 3: Use the cluster scaling adaptive reinforcement learning algorithm to build a deep reinforcement learning scheduling decision-making agent, and use the node scaling adaptive mapping method based on quantified job processing capabilities to process the agent's decision results.

[0067] First, the proximal policy optimization algorithm used includes two neural networks: the policy network (Actor) and the value network (Critic). Each network consists of two layers of MLP, each layer consists of 64 neurons, and the output layer outputs a one-dimensional Gaussian probability distribution, specifically the mean μ and the mean square error δ 2 .

[0068] Then, a function transformation is performed on the Gaussian distribution output by the neural network to confine the unbounded Gaussian distribution to the bounded output interval, and the transformation function is implemented using tanh.

[0069] Secondly, through the distribution sampling method provided by pytorch, the specific floating-point sampling action value A is obtained from the distribution c , where A c ∈(-1,1).

[0070] Finally, through the node scaling adaptive mapping method of the present invention, the mapping process is summarized as follows: Figure 3 As shown, the action value is mapped to the deployment node. The mapping is described as follows:

[0071] First, collect the hardware information of the schedulable nodes in the current cluster. It includes c for describing the general common resource capabilities. i cpu is the number of CPU cores of the node, c i mem is the memory capacity of the node, c i net is the total bandwidth of the node. Additionally, it includes indicators for describing the heterogeneous hardware resource capabilities of the node, c i gpu indicates whether the node is equipped with a GPU, c i ssd indicates whether the node is equipped with an SSD. If so, the value is 1; otherwise, it is 0.

[0072] Second, evaluate the computing power of the schedulable nodes through the following formula to obtain the computing power evaluation value C.

[0073] C i = αc i cpu + βc i mem + γc i net + η(c i gpu + c i ssd ), i ∈ {1, 2, …, N}

[0074] Among them, the variables α, β, γ, η represent the parameters for controlling the weights of each indicator.

[0075] Third, sort the schedulable nodes with C as the sorting value to obtain the sorted node set:

[0076]

[0077] Finally, let the sorted nodes be distributed in the interval (-1, 1) in a uniform distribution, as shown below.

[0078]

[0079] Through the function f constructed by the above method, the discrete action f(A c ) = {a i | i = 1, 2, 3, …, N} is obtained, where a i represents deploying the function instance to the computing node corresponding to the action value i. Finally, the decision-making problem of the deployment node is completed.

[0080] Step 4, use the historical cloud service load data set to construct a load generator to simulate the production environment and train the deep neural network of the reinforcement agent.

[0081] The function trigger sequence used for training is generated with reference to the "2021 Azure Functions Invocation Trace" dataset, which records the function trigger records of the cluster for two weeks. Select several functions with the highest statistical call volume from the records as the criteria, and generate a trigger vector sequence according to the given function trigger rate. This sequence determines the duration of the load generation process, the timestamp of the gateway request sent after the process starts, and the maximum tolerable response latency of the user for this function.

[0082] Furthermore, sample and replay the dataset. Sampling means selecting several functions to replace the benchmark function, tracking their call times in the cluster, and obtaining a schedule of function trigger calls. Replaying means simulating the request program to send requests to the FaaS gateway built in the experimental environment according to the trigger schedule obtained by sampling.

[0083] Furthermore, for the calling functions specifically used in training, the present invention uses benchmark functions: Base64: Encode and decode strings using the Base64 algorithm. Markdown2HTML: Render Markdown text as HTML. Image-Resize: Scale the resolution of images. File-IO: Write uploaded files to the local.

[0084] Furthermore, the training process of the deep reinforcement learning agent is described as follows: The monitoring component located at the cloud master node obtains metrics such as the average resource utilization rate and the average function execution latency within a 15s time window from each node every 15s and stores them in the local database. Every time the scheduling agent receives a resource scheduling request, it simultaneously extracts the observed metrics at the previous moment from the historical metric data, calculates the reward for the previous action, and updates the policy network parameters.

[0085] Step 5, use the trained scheduling decision agent to make scheduling decisions in the production environment.

[0086] Evidence of the relevant effects of the embodiment. The embodiment of the present invention has achieved some positive effects during the research and development or use process, and indeed has advantages in the cluster scaling scenario compared with the prior art. The following content is described in combination with the data, charts, etc. of the experimental process.

[0087] The results of the experiment compared with the PPO algorithm with a fixed number of clusters modeled are shown in Table 1. According to the data in Table 1, it can be seen that under the same parameter configuration, the convergence speed of this algorithm is slower than that of the general method of modeling with fixed nodes, but the average result of the reward is more stable after the learning strategy stabilizes. When training and iterating the two algorithms with the number of cluster nodes being 5, after the two agents converge and 2 nodes are added to the cluster, this algorithm can adapt to the newly added nodes and perform scheduling operations; after reducing the number of cluster nodes, this algorithm can also cover all-node actions for the reduced cluster scale.

[0088] Table 1 Training results under load simulation

[0089]

[0090] Through the above steps, the present invention solves the action space, that is, the scheduling problem of the dynamically changing number of cluster nodes, by using a continuous action space for the output of reinforcement learning, provides an adaptive resource scheduling ability for elastic cloud clusters, and improves the existing serverless resource scheduling method based on reinforcement learning. When the cluster undergoes horizontal elastic expansion and new nodes are added to the action space, by using a heuristic computing power weighted evaluation method to select a reasonable mapping strategy for the output action results of the deep neural network, the speed of the agent strategy to re-converge can be improved, and the robustness of the algorithm can be further enhanced. It has a wide range of application prospects.

[0091] The above is only the preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modification or change made by those of ordinary skill in the art according to the disclosure of the present invention shall be included in the protection scope recorded in the claims.

Claims

1. A server-unaware resource scheduling system for cluster scaling adaptive reinforcement learning, characterized in that The system includes: A FaaS platform module, which is placed within a Kubernetes cluster and hosts multiple user functional functions; A cluster metric collection module, including a Prometheus server deployed on the cluster master node and metric collection clients deployed on each node, where the clients are used to collect hardware metrics such as CPU, memory, network, and disk; A resource scheduling decision-making module, which contains a reinforcement learning agent based on the proximal policy optimization algorithm. The agent includes a policy network (Actor), a value network (Critic), attention input neurons, and a node scaling adaptive mapping module. The mapping module sorts the hardware information of each node using a node computing power evaluation coding formula and maps the sorting result to a discrete action space; A resource scheduling execution module, which issues scheduling instructions by rewriting the Score interface in the Kubernetes scheduling framework; and A pre-training load generation module, which samples and replays function call records in a predefined dataset to generate load tasks.

2. The system according to claim 1, wherein In the cluster metric collection module, the Prometheus server is used to receive and store the operation data collected by the metric collection clients of each node.

3. The system according to claim 1, wherein In the resource scheduling decision-making module, the policy network (Actor) and value network (Critic) of the reinforcement learning agent are each composed of one layer of attention input neurons and three multi-layer perceptrons (MLPs) with a fixed number of neurons. Its input dimension is d_input = M × 3 + 2, where M is the maximum number of nodes in the cluster.

4. The system according to claim 1, wherein The node scaling adaptive mapping module includes: Collecting hardware information of the currently schedulable nodes, including the number of CPU cores, memory capacity, total bandwidth, and whether it is equipped with a GPU and an SSD; Scoring the nodes according to the computing power evaluation coding formula \(C^i=\alpha c^i_{cpu}+\beta c^i_{mem}+\gamma c^i_{net}+\eta(c^i_{gpu}+c^i_{ssd})\); Sorting the scoring results and evenly distributing the sorted nodes within the interval (-1, 1) to form a discrete action set.

5. The system according to claim 1, wherein The resource scheduling execution module rewrites the Score interface of the Kubernetes scheduling framework. When receiving a scheduling request, it sets the node score corresponding to the scheduling node returned by the reinforcement learning agent to a preset maximum value and sorts according to the node scores.

6. The system according to claim 1, wherein In the pre-training load generation module, the dataset is a predefined dataset containing function call trigger records. The module generates load tasks by sampling and replaying the collected call records and stores the metric data collected from the cluster nodes according to a fixed time window.

7. A server-unaware resource scheduling method for cluster scaling adaptive reinforcement learning, characterized in that, The method includes the following steps: (1) Deploy the FaaS platform module in the Kubernetes cluster and host user functional functions on this platform; (2) Deploy the cluster metric collection module, including deploying the Prometheus server on the cluster master node and deploying metric collection clients on each node. The clients collect hardware metrics such as CPU, memory, network, and disk of each node. (3) Use the resource scheduling decision module to execute scheduling decisions. The decision module contains a reinforcement learning agent based on the Proximal Policy Optimization algorithm. The agent includes a policy network (Actor), a value network (Critic), attention input neurons, and a node scaling adaptive mapping module. The mapping module sorts each node using the computing power evaluation coding formula of node hardware information and maps the sorting result to a discrete action space. (4) Implement the issuance of scheduling instructions through the resource scheduling execution module. The execution module sets scores for scheduling nodes by rewriting the Score interface in the Kubernetes scheduling framework. (5) Based on the sampling records of the predefined dataset, perform load sampling and replay on the pre-trained load generation module.

8. The method according to claim 1, characterized in that, In step (2), the Prometheus server is deployed on the cluster master node to receive and store the hardware operation metrics collected by the metric collection clients of each node.

9. The method according to claim 1, wherein In step (3), the policy network (Actor) and value network (Critic) of the reinforcement learning agent each include one layer of attention input neurons and three-layer multi-layer perceptrons (MLPs) with a fixed number of neurons. The input dimension is defined as \(d_{input}=M×3 + 2\), where \(M\) is the maximum number of nodes in the cluster.

10. The method according to claim 1, characterized in that, In step (4), the resource scheduling execution module, by rewriting the Score interface of the Kubernetes scheduling framework, sets the score corresponding to the scheduling node returned by the reinforcement learning agent to the preset maximum value during the processing of scheduling requests, and determines the final scheduling node according to the node score sorting.

Images