Serverless computing load automatic scaling method and system based on reinforcement learning

Abstract

The present invention discloses a method and system for automatically scaling serverless computing loads based on reinforcement learning. The method includes using a reinforcement learning algorithm to modify specified elastic scaling parameters as an action, calculating a reward value after executing the action, and implementing automatic scaling of the serverless computing load. The method also uses the following steps to calculate an average mixed reward value as the reward value: obtaining the system health utilization rate and CPU usage rate after executing the action; calculating a mixed reward value composed of the two; using a specified anomaly detection algorithm to detect anomalies in the mixed reward value; removing anomalies from the time series of the mixed reward value to obtain a filtered mixed reward value time series; and calculating the average of the mixed reward value time series to obtain the average mixed reward value used as the reward value. The present invention aims to achieve fast and robust online parameter adjustment of serverless computing workloads, thereby achieving automatic scaling of the load scale.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud computing, and in particular to a method and system for automatic scaling of serverless computing loads based on reinforcement learning. Background Art

[0002] With the rapid development of cloud computing technology, serverless computing, as an emerging computing paradigm, has attracted widespread attention from both academia and industry. Serverless computing demonstrates unique advantages in fields such as artificial intelligence, data processing, and the Internet of Things, particularly its remarkable flexibility and efficiency in dynamic resource scheduling and automatic scaling. Currently, Function as a Service (FaaS) is the predominant application model for serverless computing. Compared to traditional architectures based on virtual machines or containers, serverless computing applications experience more pronounced and unpredictable workload fluctuations, particularly with high request concurrency and uncertain task execution times. For example, a sudden surge in user requests can cause the execution frequency of serverless functions to increase or decrease significantly within a short period of time, leading to drastic fluctuations in resource demand. Furthermore, Kubernetes, the most mature container orchestration engine currently deployed in data centers, employs a static threshold-driven elastic scaling algorithm that automatically increases or decreases the number of pods hosting applications based on current resource utilization and predefined thresholds. Load scheduling ensures the balanced distribution of newly created pods within the cluster. These two processes work together to enhance rapid response to sudden workload demands. However, traditional static threshold-driven elastic scaling algorithms often struggle to adapt flexibly to sudden resource changes, potentially leading to resource waste or service quality degradation. Furthermore, adjusting system parameters based solely on experience for varying workloads can easily lead to system instability, increased latency, and higher costs. In recent years, applying reinforcement learning (RL) to online tuning of serverless computing systems has become a new research trend. However, its applicability is limited by the long training time and large amounts of training data required, as well as its poor adaptability to new workload patterns. Therefore, designing and implementing lightweight online parameter threshold tuning algorithms that can handle the highly dynamic workloads of serverless scenarios has become a core research focus. A similar approach has recently been proposed, relying on single-point feedback from reinforcement learning and using reward information only at the parameter point being adjusted, without modeling complex unknown reward functions. Compared to global modeling techniques, this algorithm's lightweightness stems from eliminating the additional computation required to estimate rewards at unexplored parameter states, which reduces model complexity and makes it particularly suitable for online tuning during deployment. However, the single reward value used does not provide sufficient information to guide rapid exploration, and the gradient ascent method is sensitive to noise, making it prone to getting stuck in local minima. Summary of the Invention

[0003] Technical problem to be solved by the present invention: In response to the above-mentioned problems in the prior art, a method and system for automatic scaling of serverless computing loads based on reinforcement learning are provided. The present invention aims to achieve fast and robust online parameter adjustment of serverless computing workloads, thereby achieving automatic scaling of load scale.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0005] A method for automatically scaling a serverless computing load based on reinforcement learning includes adopting a reinforcement learning algorithm to modify a specified elastic scaling parameter as an action, calculating a reward value after executing the action, and implementing automatic scaling of the serverless computing load. The method also uses the following steps to calculate an average mixed reward value as the reward value:

[0006] S101, obtaining the system health utilization and CPU usage after the action is executed;

[0007] S102, calculating a mixed reward value composed of a mixture of system health utilization and CPU usage;

[0008] S103, using a specified anomaly detection algorithm to detect anomalies of the mixed reward value;

[0009] S104, removing abnormal values ​​from the time series of the mixed reward value to obtain a filtered time series of the mixed reward value;

[0010] S105 , calculating an average value of the mixed reward value time series to obtain an average mixed reward value used as a reward value.

[0011] Optionally, the specified elastic scaling parameters include part or all of the CPU usage lower limit cpuLow, the CPU usage upper limit cpuHigh, the memory usage lower limit memLow, the memory usage upper limit memHigh, the expected number of application copies to run replica, and the CPU threshold containerCpuThreshold of the current container; if the CPU usage is less than the CPU usage lower limit cpuLow, the CPU number is reduced; if the CPU usage is greater than the CPU usage upper limit cpuHigh, the CPU number is expanded; if the memory usage is less than the memory usage lower limit memLow, the memory number is reduced; if the memory usage is greater than the memory usage upper limit memHigh, the memory number is expanded; if the number of running application copies is less than the expected number of running application copies replica, the number of running application copies is increased; if the number of running application copies is greater than the expected number of running application copies replica, the number of running application copies is reduced; if the CPU threshold of the container is less than the CPU threshold containerCpuThreshold of the current container, the CPU number is reduced; if the CPU threshold of the container is greater than the CPU threshold containerCpuThreshold of the current container, the CPU number is expanded.

[0012] Optionally, the calculation function expression of the system health utilization rate in step S101 is:

[0013] ,

[0014] in, For system health utilization, For a given time period The time when the system is in a healthy state is the time when the CPU usage is healthy and the memory usage is healthy. For logical operations, CPU usage is healthy Indicates that the CPU usage is high (CPU usage lower limit cpuLow) and lower than the CPU usage upper limit cpuHigh, and the memory usage is healthy. Indicates that the memory usage is greater than the lower limit of memory usage memLow and less than the upper limit of memory usage memHigh.

[0015] Optionally, the calculation function expression of the mixed reward value in step S102 is:

[0016] ,

[0017] in, is the mixed reward value, is the weight coefficient, For system health utilization, Indicates the CPU usage.

[0018] Optionally, in step S103, the use of a specified anomaly detection algorithm to detect anomalies of the mixed reward value includes: using the autoregressive moving average model ARMA to predict the current mixed reward value using the time series of the mixed reward value, and then taking the difference between the predicted mixed reward value and the calculated mixed reward value; if the difference exceeds a preset threshold, the currently calculated mixed reward value is determined to be an anomaly; otherwise, the currently calculated mixed reward value is determined to be a normal value; or, based on combining historical data of the mixed reward value with the 3𝜎 principle, determining whether the currently calculated mixed reward value is an anomaly.

[0019] Optionally, the adopting of a reinforcement learning algorithm to modify a specified elastic scaling parameter as an action and calculating a reward value after executing the action to achieve automatic scaling of the serverless computing load includes:

[0020] S201: Initialize and explore the radius of each elastic scaling parameter and the values ​​of various elastic scaling parameters;

[0021] S202, in radius Generate a random perturbation vector and calculate the values ​​of each elastic scaling parameter after random perturbation;

[0022] S203, executing an action of modifying elastic scaling parameters to achieve automatic scaling of serverless computing load;

[0023] S204, calculating the reward value after executing the action;

[0024] S205: Update the values ​​of each elastic scaling parameter using a gradient descent algorithm in combination with the reward value. If further iteration is required, jump to step S202; otherwise, terminate and exit.

[0025] Optionally, in step S205, when the gradient descent algorithm is used to update the values ​​of each elastic scaling parameter in combination with the reward value, the current gradient of each elastic scaling parameter is included. The dynamic window gradient is calculated as the adjusted gradient using the following formula:

[0026] ,

[0027] in, is the dynamic window gradient, is a dynamically adjustable coefficient. To take the average value, is the average value of the historical gradient at each time point in the most recent historical window, The calculation function expression of the size of the recent history window is the current gradient:

[0028] ,

[0029] ,

[0030] in, is the size of the recent history window, 、 are constants used to control the initial and maximum values ​​of the history window, is the current reward value, is the average reward value in the most recent historical window, A dynamically adjustable coefficient The amount of change, is the median reward value in the most recent historical window, is a constant, To take the maximum value.

[0031] In addition, the present invention also provides a serverless computing load automatic scaling system based on reinforcement learning, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the serverless computing load automatic scaling method based on reinforcement learning.

[0032] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the reinforcement learning-based serverless computing load automatic scaling method through a processor.

[0033] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the reinforcement learning-based serverless computing load automatic scaling method through a processor.

[0034] Compared with the prior art, the present invention can mainly achieve the following beneficial effects:

[0035] 1. When adjusting elastic scaling-related parameters through online reinforcement learning, the present invention integrates an average mixed reward (AMR) mechanism to calculate the average mixed reward value, including the intrinsic learning reward structure of the policy gradient (obtaining the system health utilization rate and CPU usage rate after executing the action and calculating the mixed reward value composed of the two), and the robustness reward structure (using a specified anomaly detection algorithm to detect anomalies in the mixed reward value; removing anomalies from the time series of the mixed reward value to obtain a filtered mixed reward value time series; and calculating the average of the mixed reward value time series to obtain the average mixed reward value used as the reward value). By utilizing two different reward structures to accelerate convergence, it is possible to achieve fast and robust online parameter adjustment of serverless computing workloads, thereby realizing automatic scaling of the load scale, thereby enhancing the robustness of the system.

[0036] 2. The present invention can further integrate a dynamic window gradient (DG) mechanism when adjusting elastic scaling related parameters through online reinforcement learning. The dynamic window gradient (DG) mechanism is used to smooth gradient calculations by including historical values, thereby further enhancing the robustness of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Schematic diagram of the process of calculating the average mixed reward value in an embodiment of the present invention.

[0038] Figure 2 Schematic diagram of the process of the reinforcement learning algorithm in an embodiment of the present invention.

[0039] Figure 3 Schematic diagram of the system structure in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention will be further described in detail below with reference to the accompanying drawings in the embodiments of the present invention.

[0041] The serverless computing load automatic scaling method based on reinforcement learning in this embodiment includes using a reinforcement learning algorithm to modify a specified elastic scaling parameter as an action and calculate a reward value after executing the action to achieve automatic scaling of the serverless computing load, such as Figure 1 As shown, the following steps are used to calculate the average mixed reward value as the reward value:

[0042] S101, obtaining the system health utilization and CPU usage after the action is executed;

[0043] S102, calculating a mixed reward value composed of a mixture of system health utilization and CPU usage;

[0044] S103, using a specified anomaly detection algorithm to detect anomalies of the mixed reward value (X abnormal );

[0045] S104: Remove the outliers X from the time series of the mixed reward value. abnormal Get the filtered mixed reward value time series X;

[0046] S105 , calculating an average value of the mixed reward value time series to obtain an average mixed reward value used as a reward value.

[0047] Traditional reinforcement learning algorithms often use CPU utilization, memory occupancy or SLO (service level objective) as reward values ​​because these data are easy to obtain in the system. This embodiment designs a reward value that takes into account both internal and external rewards. The internal reward value is CPU utilization, and the external reward value is the healthy utilization percentage (HUP) of system resources. The external reward value HUP is an indicator used in existing work to measure the overall state of the system. It is defined as the ratio of the time that all nodes are in a healthy state in a given time period to the total duration of the time period. Both CPU and memory have upper and lower bounds, and the current time period is considered to be healthy only when the CPU and memory utilization fall within these bounds. HUP faces two limitations. One is that it introduces noise, which hinders the exploration of optimal parameters and may lead to suboptimal solutions, especially in dynamic or multi-objective tasks. The other is that it may cause instability, slow convergence, or failure to converge to the optimal strategy. To this end, this embodiment proposes the following Figure 1 The Average Mixed Reward (AMR) shown in Figure 1 integrates the intrinsic learning reward structure of policy gradients (LIRPG) and the robustness reward structure, overcoming these two limitations. CPU utilization was chosen as the internal reward because, compared to increasing memory, increasing CPU usage can improve scheduling response time. Using CPU utilization as an internal reward is intended to encourage optimizing system resource health while optimizing resource utilization as much as possible.

[0048] To assess the health of the system across time, the elastic scaling parameters specified in this embodiment include the CPU usage lower limit cpuLow, the CPU usage upper limit cpuHigh, the memory usage lower limit memLow, the memory usage upper limit memHigh, the expected number of application replicas to run, and the CPU threshold containerCpuThreshold of the current container (some of them can also be selected according to the needs of the scenario). The above six elastic scaling parameters can establish the elastic boundaries of the system. By adjusting the above six parameters (i.e. k=6) to implement dynamic threshold driven elastic scaling strategy. In each iteration, the cluster manager receives a set of the above 6 elastic scaling parameters. k dimensional parameter vector a(t) , which is then used to set system parameters.

[0049] The above six elastic scaling parameters can establish the elastic boundary of the system. If this threshold is exceeded, the number of CPUs and memory will be automatically scaled, including: (1) If the CPU usage is less than the CPU usage lower limit cpuLow, the number of CPUs will be shrunk; if the CPU usage is greater than the CPU usage upper limit cpuHigh, the number of CPUs will be expanded; (2) If the memory usage is less than the memory usage lower limit memLow, the number of memory will be shrunk; if the memory usage is greater than the memory usage upper limit memHigh, the number of memory will be expanded; (3) If the number of running application copies is less than the expected number of running application copies replica, the number of running application copies will be increased; if the number of running application copies is greater than the expected number of running application copies replica, the number of running application copies will be reduced; (4) If the container's CPU threshold is less than the current container's CPU threshold containerCpuThreshold, the number of CPUs will be shrunk; if the container's CPU threshold is greater than the current container's CPU threshold containerCpuThreshold, the number of CPUs will be expanded.

[0050] System health utilization is defined as the ratio of the time all nodes are healthy in a given time period to the total duration of that time period. Both CPU and memory utilization have upper and lower bounds, and the current time period is considered healthy only when both CPU and memory utilization fall within these bounds. The calculation function for system health utilization in this embodiment is:

[0051] ,

[0052] in, For system health utilization, For a given time period The time when the system is in a healthy state is the time when the CPU usage is healthy and the memory usage is healthy. For logical operations, CPU usage is healthy Indicates that the CPU usage is high (CPU usage lower limit cpuLow) and lower than the CPU usage upper limit cpuHigh, and the memory usage is healthy. Indicates that the memory usage is greater than the lower limit of memory usage memLow and less than the upper limit of memory usage memHigh.

[0053] The combination of internal and external rewards forms a mixed reward value (MixReward), which is defined as the weighted sum of CPU and HUP, where CPU represents the CPU utilization rate in the cluster. This combination of the two accelerates the optimization of system parameters. The calculation function expression of the mixed reward value in step S102 of this embodiment is:

[0054] ,

[0055] in, is the mixed reward value, is the weight coefficient, For system health utilization, Indicates the CPU usage.

[0056] To minimize the impact of noise on reward values, this embodiment introduces a robust reward structure to prevent overly rapid exploration of the parameter space. For most hybrid rewards, historical average window backpropagation is used to smooth historical data and reduce the impact of noise. Furthermore, to prevent oversmoothing from potentially missing critical data, this embodiment employs anomaly detection to filter out hybrid reward values ​​that are important to the system. As an optional implementation, the use of a specified anomaly detection algorithm to detect outliers in the hybrid reward values ​​in step S103 includes: using the autoregressive moving average (ARMA) model to predict the current hybrid reward value based on the time series of the hybrid reward values; then, taking the difference between the predicted hybrid reward value and the calculated hybrid reward value; if the difference exceeds a preset threshold, the current calculated hybrid reward value is determined to be an outlier; otherwise, the current calculated hybrid reward value is determined to be normal. The anomaly detection algorithm used in this embodiment is the autoregressive moving average (ARMA) model, which analyzes the time series data of the hybrid rewards generated by the workload and predicts the current value. The ARMA model consists of two main components: autoregressive (AR) and moving average (MA). AR represents the linear relationship between the current value and observations at several previous time points. If the current value is predicted using observations from the past four time points, this constitutes a fourth-order autoregressive model. MA, on the other hand, represents a linear combination of the current value and the previous error terms. It assumes that the current value is influenced not only by past observations but also by the previous error terms. MA modifies the prediction using residual terms from several previous time points, thereby modeling random volatility. As an alternative implementation, historical data on mixed reward values ​​can be combined with the 3𝜎 principle to determine whether the currently calculated mixed reward value is an outlier.

[0057] like Figure 2 As shown, in this embodiment, a reinforcement learning algorithm is used to modify the specified elastic scaling parameters as an action and calculate the reward value after executing the action to achieve automatic scaling of the serverless computing load, including:

[0058] S201: Initialize and explore the radius of each elastic scaling parameter and the values ​​of various elastic scaling parameters;

[0059] S202, in radius Generate a random perturbation vector and calculate the values ​​of each elastic scaling parameter after random perturbation;

[0060] S203, executing an action of modifying elastic scaling parameters to achieve automatic scaling of serverless computing load;

[0061] S204, calculating the reward value after executing the action;

[0062] S205: Update the values ​​of each elastic scaling parameter using a gradient descent algorithm in combination with the reward value. If further iteration is required, jump to step S202; otherwise, terminate and exit.

[0063] In the parameter optimization process of reinforcement learning algorithms, traditional methods often rely directly on gradients and reward values ​​to update parameters. This approach can easily cause the algorithm to fall into local extremes, thereby affecting the overall performance of the model. To solve this problem, this embodiment adopts a dynamic window gradient method (Dynamic Window Gradient, DG) based on historical gradient information to comprehensively consider historical reward values ​​and changes, and dynamically adjust the window size to improve scheduling efficiency. Figure 2 As shown,

[0064] In step S205 of this embodiment, when the gradient descent algorithm is used to update the values ​​of each elastic scaling parameter in combination with the reward value, the current gradient of each elastic scaling parameter is included. The dynamic window gradient is calculated as the adjusted gradient using the following formula:

[0065] ,

[0066] in, is the dynamic window gradient, is a dynamically adjustable coefficient. To take the average value, is the average value of the historical gradient at each time point in the most recent historical window, The calculation function expression of the size of the recent history window is the current gradient:

[0067] ,

[0068] ,

[0069] in, is the size of the recent history window, 、 are constants used to control the initial and maximum values ​​of the history window, is the current reward value, is the average reward value in the most recent historical window, A dynamically adjustable coefficient The amount of change, is the median reward value in the most recent historical window, is a constant, The maximum value is taken. This dynamic adjustment mechanism enables the algorithm to flexibly adapt to different environmental conditions, especially in rapidly changing tasks, effectively collecting more relevant data and improving learning outcomes. Upon receiving the new values ​​of the six elastic scaling parameters—cpuLow, cpuHigh, memLow, memHigh, replica, and containerCpuThreshold—the cluster manager updates the corresponding parameter configurations in the system and invokes existing cloud platform functions to dynamically adjust the load scale. Simultaneously, it calculates a reward value based on the new system state after scheduling execution and sends it to the server.

[0070] Figure 3 The system structure diagram of this embodiment is as follows. The system consists of three parts: a serverless cluster manager, a server, and a client API. The serverless cluster manager receives the adjustment parameters selected by the server and completes the elastic scaling work for the serverless workload. The cluster manager passes the elastic scaling parameters that affect the load scale to the server by calling the client API (e.g. Figure 3 As shown in ① in the figure, the server creates a service instance, which runs an online reinforcement learning algorithm that integrates average mixed reward (AMR) and dynamic window gradient (DG). The cluster manager then calls the service instance to perform a round of exploration in the parameter space to predict the next set of optimal parameters (such as Figure 3 Next, the cluster manager receives the predicted parameters through the client and updates the current system parameters (as shown in ② in the figure). Figure 3 Then, the cluster manager uses the received parameters to perform elastic expansion and workload scheduling, and calculates a new reward value based on the newly collected cluster state and sends it to the service instance (as shown in ③). Figure 3 ④ in the figure).

[0071] The advantages of the serverless computing load automatic scaling method based on reinforcement learning in this embodiment are as follows: (1) In terms of system architecture, a lightweight, online reinforcement learning method is used to meet the challenge of serverless computing automatic scaling parameters. Compared with deep reinforcement learning methods that require exploring a huge state space and thus require high training costs, this embodiment can reduce the cost requirements brought by a large amount of training, while enhancing adaptability to various complex business scenarios. (2) In the reward value calculation stage, the concept of reward is expanded and a mixed reward value is adopted, enabling the system to adjust its behavior based on internal attributes while receiving external attributes from the environment. External attributes are defined as rewards that represent the overall system goals, while internal attributes are defined as rewards that guide the system to optimize in a specific direction and are also related to the overall system goals, thereby accelerating the convergence speed of reinforcement learning. (3) In the parameter update stage, the reward value, current gradient and historical gradient are comprehensively considered to determine the amplitude of parameter update, thereby generating new parameter values. By integrating historical gradient information, this method can smooth the current gradient value and enhance the robustness of parameter update, effectively reducing the convergence difficulty caused by local extreme values. In order to determine the degree of dependence on historical gradients, a historical dynamic window is constructed to calculate the coefficients. In addition, the window size is dynamically adjusted by calculating the reward value and the current change value of the coefficient, and the window size in turn affects the size of the coefficient. The above process enhances the robustness of the method and enables efficient parameter updates even under complex workload patterns. In summary, the method of this embodiment implements the adjustment of elastic scaling related parameters through online reinforcement learning, and integrates the average mixed reward (AMR) and dynamic window gradient (DG) mechanisms. The average mixed reward value (AMR) mechanism accelerates convergence by utilizing two different reward structures, while the dynamic window gradient (DG) mechanism smoothes the gradient calculation by including historical values, thereby enhancing the robustness of the system.

[0072] In addition, this embodiment also provides a serverless computing load automatic scaling system based on reinforcement learning, including an interconnected microprocessor and a memory, wherein the microprocessor is programmed or configured to execute the serverless computing load automatic scaling method based on reinforcement learning.

[0073] In addition, this embodiment also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the reinforcement learning-based serverless computing load automatic scaling method through a processor.

[0074] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the reinforcement learning-based serverless computing load automatic scaling method through a processor.

[0075] Those skilled in the art should understand that the technical solution provided by the present invention may be in the form of a method, a system, or a computer program product. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions described in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0076] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A serverless computing load automatic scaling method based on reinforcement learning, characterized in that: This includes using a reinforcement learning algorithm to modify specified elastic scaling parameters as actions, calculating reward values ​​after executing the actions, and implementing automatic scaling of serverless computing loads. The following steps are used to calculate an average mixed reward value as the reward value: S101, obtaining the system health utilization and CPU usage after the action is executed; S102, calculating a mixed reward value composed of a mixture of system health utilization and CPU usage; S103, using a specified anomaly detection algorithm to detect anomalies of the mixed reward value; S104, removing abnormal values ​​from the time series of the mixed reward value to obtain a filtered time series of the mixed reward value; S105, calculating an average value of the mixed reward value time series to obtain an average mixed reward value used as a reward value; The specified elastic scaling parameters include part or all of the CPU usage lower limit cpuLow, the CPU usage upper limit cpuHigh, the memory usage lower limit memLow, the memory usage upper limit memHigh, the expected number of running application copies replica, and the CPU threshold containerCpuThreshold of the current container; if the CPU usage is less than the CPU usage lower limit cpuLow, the number of CPUs is reduced; if the CPU usage is greater than the CPU usage upper limit cpuHigh, the number of CPUs is expanded; if the memory usage is less than the memory usage lower limit memLow, the memory amount is reduced; if the memory usage is greater than the memory usage upper limit memHigh, the memory amount is expanded; if the number of running application copies is less than the expected number of running application copies replica, the number of running application copies is increased; if the number of running application copies is greater than the expected number of running application copies replica, the number of running application copies is reduced; if the container's CPU threshold is less than the current container's CPU threshold containerCpuThreshold, the number of CPUs is reduced; if the container's CPU threshold is greater than the current container's CPU threshold containerCpuThreshold, the number of CPUs is expanded; The calculation function expression of the system health utilization rate in step S101 is: ， in, For system health utilization, For a given time period The time when the system is in a healthy state is the time when the CPU usage is healthy and the memory usage is healthy. For logical operations, CPU usage is healthy Indicates that the CPU usage is high (CPU usage lower limit cpuLow) and lower than the CPU usage upper limit cpuHigh, and the memory usage is healthy. Indicates that the memory usage is greater than the lower limit of memory usage memLow and less than the upper limit of memory usage memHigh.

2. The serverless computing load automatic scaling method based on reinforcement learning according to claim 1 is characterized in that: The calculation function expression of the mixed reward value in step S102 is: ， in, is the mixed reward value, is the weight coefficient, For system health utilization, Indicates the CPU usage.

3. The serverless computing load automatic scaling method based on reinforcement learning according to claim 1 is characterized in that: In step S103, the specified anomaly detection algorithm is used to detect anomalies of the mixed reward value, including: using the autoregressive moving average model ARMA to predict the current mixed reward value based on the time series of the mixed reward value, and then taking the difference between the predicted mixed reward value and the calculated mixed reward value. If the difference exceeds a preset threshold, the currently calculated mixed reward value is determined to be an anomaly; otherwise, the currently calculated mixed reward value is determined to be a normal value; or, whether the currently calculated mixed reward value is an anomaly is determined based on combining historical data of the mixed reward value with the 3𝜎 principle.

4. The method for automatic scaling of serverless computing load based on reinforcement learning according to claim 1, characterized in that: The method of using a reinforcement learning algorithm to modify a specified elastic scaling parameter as an action and calculate a reward value after executing the action to achieve automatic scaling of a serverless computing load includes: S201: Initialize and explore the radius of each elastic scaling parameter and the values ​​of various elastic scaling parameters; S202, in radius Generate a random perturbation vector and calculate the values ​​of each elastic scaling parameter after random perturbation; S203, executing an action of modifying elastic scaling parameters to achieve automatic scaling of serverless computing load; S204, calculating the reward value after executing the action; S205: Update the values ​​of each elastic scaling parameter using a gradient descent algorithm in combination with the reward value. If further iteration is required, jump to step S202; otherwise, terminate and exit.

5. The method for automatic scaling of serverless computing load based on reinforcement learning according to claim 4, characterized in that: In step S205, when the gradient descent algorithm is used to update the values ​​of each elastic scaling parameter in combination with the reward value, the current gradient of each elastic scaling parameter is included. The dynamic window gradient is calculated as the adjusted gradient using the following formula: ， in, is the dynamic window gradient, is a dynamically adjustable coefficient. To take the average value, is the average value of the historical gradient at each time point in the most recent historical window, The calculation function expression of the size of the recent history window is the current gradient: ， ， in, is the size of the recent history window, 、 are constants used to control the initial and maximum values ​​of the history window, is the current reward value, is the average reward value in the most recent historical window, A dynamically adjustable coefficient The amount of change, is the median reward value in the most recent historical window, is a constant, To take the maximum value.

6. A serverless computing load automatic scaling system based on reinforcement learning, comprising an interconnected microprocessor and a memory, characterized in that: The microprocessor is programmed or configured to execute the serverless computing load automatic scaling method based on reinforcement learning as described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instruction is programmed or configured to execute the serverless computing load automatic scaling method based on reinforcement learning as described in any one of claims 1 to 5 through a processor.

8. A computer program product comprising a computer program or instructions, characterized in that The computer program or instruction is programmed or configured to execute the serverless computing load automatic scaling method based on reinforcement learning as described in any one of claims 1 to 5 through a processor.

Images