Method, system, and computer program product for microservice traffic limiting

Abstract

The invention relates to a method for flow limiting of a micro-service. The method comprises the following steps: generating a control plane configuration of a service grid for flow limiting of the micro-service based on a flow limiting demand for the micro-service and according to the flow of the micro-service and measurement of system resources; a control plane component of the service grid converts the control plane configuration of the micro-service flow limitation into data plane configuration of the service grid for the micro-service flow limitation of the micro-service and pushes the data plane configuration to an agent of the service grid; and performing traffic limiting by an agent of the service grid on traffic sent to the micro-service based on the data plane configuration of the micro-service traffic limiting. The disclosure also relates to a system and a computer program product for microservice traffic limiting.

Description

Technical Field

[0001] This disclosure relates to the field of distributed computing, and more particularly to a method, system, and computer program product for rate limiting of microservices. Background Technology

[0002] Microservices architecture involves designing an application as a set of small, modular services that can be deployed independently; these services are often referred to as "microservices." Each microservice performs a unique process and communicates through well-defined, lightweight mechanisms to serve business objectives using communication protocols. Microservices architecture has become the preferred approach for developing software systems, especially those operating in cloud-based environments.

[0003] Microservice architectures are susceptible to sudden traffic surges (e.g., instantaneous high concurrency), which can cause the CPU utilization of microservices to reach its limit, leading to a spike in microservice response time. This can result in upstream thread backlog, ultimately causing a complete denial-of-service attack and system crash. Rate limiting is the primary means of preventing service overload. By controlling the amount of concurrent access or the request rate, rate limiting can effectively protect the system from being overwhelmed by excessive requests. Traditional microservices use a Software Development Kit (SDK) to embed rate limiting modules into the microservice business code, and then dynamically adjust the rate limiting threshold by combining it with monitoring agents in the microservice pod to collect system metrics. This approach requires intrusive modifications to the microservices, resulting in complex adaptation, testing, and deployment processes, leading to low deployment efficiency. Furthermore, this approach requires customizing functional modules in different programming languages ​​for microservices using different programming languages, such as rate limiting modules, monitoring and data reporting modules, resulting in high implementation costs, long deployment cycles, and heavy operational burdens. Summary of the Invention

[0004] This disclosure provides methods, systems, and computer program products for rate limiting in microservices.

[0005] According to an embodiment of this disclosure, a method for microservice rate limiting is provided, the method comprising: generating a control plane configuration for microservice rate limiting of a service mesh based on rate limiting requirements for the microservice and traffic and system resource metrics for the microservice; converting the control plane configuration for microservice rate limiting into a data plane configuration for microservice rate limiting of the service mesh by a control plane component of the service mesh and pushing it to a proxy of the service mesh; and implementing rate limiting on traffic sent to the microservice by the proxy of the service mesh based on the data plane configuration for microservice rate limiting.

[0006] According to another embodiment of this disclosure, a system for microservice rate limiting is provided. The system includes: one or more processors; a memory coupled to the one or more processors and storing computer program instructions; wherein, when executed by the one or more processors, the computer program instructions cause the one or more processors to perform the following operations: generating a control plane configuration for microservice rate limiting of the microservice based on rate limiting requirements for the microservice and according to traffic metrics and system resource metrics for the microservice; the control plane component of the service mesh converting the control plane configuration for microservice rate limiting into a data plane configuration for microservice rate limiting of the service mesh and pushing it to a proxy of the service mesh; and the proxy of the service mesh applying rate limiting to traffic sent to the microservice based on the data plane configuration for microservice rate limiting.

[0007] According to another embodiment of this disclosure, a computer program product is provided. The computer program product includes computer-readable instructions. When executed by a processor, these instructions cause the processor to perform the method for rate limiting in microservices described in the above embodiment.

[0008] According to the implementation scheme of this disclosure, the improved microservice rate limiting mechanism utilizes the underlying traffic control capabilities of the service mesh to push the business application layer that requires code intrusion down to the infrastructure layer, fundamentally changing the implementation level of the rate limiting function and realizing a microservice rate limiting solution that is decoupled from the microservice business layer. Attached Figure Description

[0009] The above and other objects, features, and advantages of this disclosure will become more apparent from a more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to provide a further understanding of the embodiments of this disclosure and form part of the specification. The drawings, together with the embodiments of this disclosure, are used to interpret this disclosure and do not constitute a limitation thereof. In the drawings, unless expressly indicated, like reference numerals generally represent like parts, steps, or elements.

[0010] Figure 1 A flowchart illustrating a method 100 for rate limiting of microservices according to an embodiment of this disclosure is shown;

[0011] Figure 2 A schematic diagram of a microservice rate limiting architecture 200 according to one embodiment of this disclosure is illustrated;

[0012] Figure 3 A schematic diagram illustrating an Istio mesh architecture 300 according to an embodiment of the present disclosure is shown;

[0013] Figure 4 Examples Figure 2A schematic diagram of the example architecture 400 for the controller component in the diagram;

[0014] Figure 5 This is a flowchart illustrating the process of parsing a custom rate-limiting resource according to an implementation scheme of this disclosure 500;

[0015] Figure 6 A flowchart illustrating the resource flow limiting protection process 600 according to an implementation scheme of this disclosure is shown;

[0016] Figure 7 A schematic diagram illustrating an Istio local rate limiting method 700 according to an embodiment of the present disclosure is shown;

[0017] Figure 8 A schematic diagram illustrating the flow 800 of a rate limiting threshold algorithm according to an embodiment of the present disclosure is shown;

[0018] Figure 9 This is a schematic diagram illustrating an example flow 900 of a current limiting threshold algorithm according to one embodiment of the present disclosure;

[0019] Figure 10 This is a schematic diagram illustrating an example flow 1000 of a current limiting threshold algorithm according to another embodiment of the present disclosure;

[0020] Figure 11 This is an exemplary block diagram illustrating a computing device 1100 according to some embodiments of the present disclosure.

[0021] Those skilled in the art will understand that the elements in the accompanying drawings are illustrated for simplicity and clarity and are not necessarily drawn to scale. For example, the dimensions of some elements in the illustrations, block diagrams, or flowcharts may be exaggerated relative to other elements to aid in the accurate understanding of this embodiment. Detailed Implementation

[0022] The following detailed description is illustrated in the accompanying drawings. While several exemplary embodiments are described herein, modifications, adaptations, and other implementations are possible. For example, components and steps illustrated in the drawings may be replaced, added, or modified, and the exemplary methods described herein may be modified by replacing, reordering, deleting, or adding steps to the disclosed methods. Therefore, the following detailed description is not limited to the disclosed embodiments and examples. Rather, the appropriate scope of the invention is determined by the appended claims.

[0023] In the detailed description below, numerous specific details are set forth in order to provide a thorough understanding of certain aspects. However, those skilled in the art will understand that some aspects can be practiced without these specific details. In other instances, well-known methods, procedures, components, units, and / or circuits have not been described in detail to avoid obscuring the discussion.

[0024] As used herein, discussions of terms such as “generate,” “push,” “implement,” “aggregate,” “compare,” “compute,” “store,” “acquire,” “set,” or similar terms may refer to operations and / or processes of a computer, computing platform, computing system, or other electronic computing device that manipulate and / or convert physical (e.g., electronic) quantities represented in computer registers and / or memory into physical quantities in computer registers and / or memory or other information storage media that may store instructions for performing the operations and / or processes. The word “such as” is not used in a limiting sense but for interpretive purposes.

[0025] The use of terms such as “on one aspect,” “an aspect,” “example aspect,” and “various aspects” indicates that an aspect described in this way may include a specific feature, structure, or characteristic, but not every aspect necessarily includes the implementation of that specific feature, structure, or characteristic. Furthermore, the repeated use of the phrase “on one aspect” does not necessarily refer to the same aspect, although it may.

[0026] As used herein, unless otherwise stated, ordinal adjectives such as “first,” “second,” etc., are used to describe general objects only to indicate different instances of similar objects mentioned, and are not intended to imply that the objects so described must have a given order in time, space, sequence, or any other way. Similarly, articles such as “a,” “an,” or “the” do not indicate a quantity limitation, but rather that at least one exists. Words such as “connection” and “coupling” are not limited to physical or mechanical connections, but also include direct or indirect electrical or communication connections.

[0027] Furthermore, the technical features involved in the different embodiments of this disclosure described below can be combined with each other, provided that there is no conflict between them.

[0028] Figure 1 A flowchart illustrating a method 100 for microservice rate limiting according to an embodiment of this disclosure is shown. Figure 1 As shown, the method 100 for rate limiting of microservices includes the following steps 102-106.

[0029] In step 102, a control plane configuration for microservice rate limiting is generated within the service mesh based on the rate limiting requirements for the microservice and according to the traffic metrics and system resource metrics for the microservice. The control plane configuration follows a service mesh control plane format. The control plane configuration can be generated by a control module, which is a program module that performs calculations and control. The traffic metrics for the microservice are historical data on the traffic (e.g., requests per second, QPS) delivered to the microservice over a period of time. In some implementations, the system resource metrics for the microservice may include the utilization of the central processing unit (CPU) allocated to the microservice. This server may be a cloud server running the microservice. Additionally or alternatively, the system resource metrics for the microservice may include the server's memory utilization. In some implementations, the control plane configuration for microservice rate limiting includes a rate limiting threshold for the microservice.

[0030] In step 104, the service mesh's control plane converts the microservice rate limiting control plane configuration into the service mesh's microservice rate limiting data plane configuration and pushes it to the service mesh's proxy. The service mesh's control plane listens for the generated microservice rate limiting control plane configuration. Once the microservice rate limiting control plane configuration changes (e.g., adding, modifying, or deleting a control plane configuration), the service mesh's control plane converts the microservice rate limiting control plane configuration into the service mesh's data plane configuration and sends it to the corresponding service mesh proxy. In step 106, the service mesh's proxy applies rate limiting to traffic sent to the microservice based on the microservice rate limiting data plane configuration. The service mesh's proxy has built-in rate limiting and circuit breaking capabilities. When traffic flows through the service mesh's proxy, the proxy can apply rate limiting based on the microservice rate limiting data plane configuration (especially the microservice's rate limiting threshold), thereby limiting the traffic sent to the microservice.

[0031] Method 100 for microservice rate limiting fundamentally changes the implementation level of rate limiting functionality, moving it down from the business application layer (which requires code intrusion) to the infrastructure layer. It fully leverages the service mesh, achieving "zero-development, seamless integration" in microservice architecture. By utilizing the service mesh, Method 100 builds an automated rate limiting closed-loop system, solving the pain point of low implementation efficiency in traditional rate limiting solutions.

[0032] For reference Figure 2 Further explanation of the methods for rate limiting in microservices 100. Figure 2 A schematic diagram of a microservice rate limiting architecture 200 according to one embodiment of this disclosure is illustrated. For example... Figure 2As shown, the development and operations personnel 202 input the custom rate-limiting resource 204 into the application programming interface (API) server 206. The custom rate-limiting resource 204 is a declarative API object, including fixed values ​​and internal values. The fixed values ​​define which services(s) are enabled for rate limiting, while the internal values ​​define the internal associations and flags of the rate-limiting function. The internal values ​​are independent of the final surface of rate limiting (e.g., the returned error code, the rate-limiting threshold). The custom rate-limiting resource 204 can contain the rate-limiting requirements for microservices mentioned in step 102. After receiving the custom rate-limiting resource 204, the API server 206 pushes it to the controller component 208 in the form of an event stream. Meanwhile, the monitoring component 210 collects and stores metrics for the microservices, including but not limited to traffic, response latency, error rate, CPU metrics, and memory metrics. The monitoring component 210 can be Prometheus, Jaeger, Zipkin, Datadog, Grafana Mimir, or Kubernetes Metrics Server, etc.

[0033] Controller component 208 is a control program developed by the developers, which corresponds to the control module mentioned above. Controller component 208 parses the received custom rate-limiting resource 204, pulls relevant data from monitoring component 210 for calculating the rate-limiting threshold, and calculates the rate-limiting threshold based on the pulled data, dynamically generating the Istio rate-limiting EnvoyFilter resource 212. The rate-limiting EnvoyFilter resource records a complete set of instructions, including "which configuration segment to target, which algorithm to use, the allowed traffic threshold, and how to respond after exceeding the limit." Istio is a service mesh, and the Istio rate-limiting EnvoyFilter resource 212 can correspond to the microservice rate-limiting control plane configuration mentioned in step 102. Controller component 208 submits the generated rate-limiting EnvoyFilter resource 212 to the API server. The Istio control plane component Istiod 214 monitors the changes in the rate-limiting EnvoyFilter resource 212 in real time. After detecting the change in the rate-limiting EnvoyFilter resource 212, Issue 214 merges the new rate-limiting EnvoyFilter resource 212 with its own generated default configuration into an Extensible Discovery Service (xDS) configuration, and distributes it to the Istio proxy 217 in the microservice Pod 216. The xDS configuration can correspond to the microservice rate-limiting data plane configuration mentioned in step 104 of the data plane configuration.

[0034] Istio proxy 217 and microservice container 218 are set up in the same microservice Pod 216. Traffic first passes through Istio proxy 217 and then is sent to microservice container 218. Istio proxy 217 automatically starts rate limiting based on xDS configuration. Figure 2 The microservice rate limiting architecture 200 can be implemented based on the Kubernetes (k8s) platform.

[0035] Figure 3 A schematic diagram of an Istio mesh architecture 300 according to an embodiment of this disclosure is illustrated. For example... Figure 3 As shown, the Istio mesh primarily consists of a data plane and a control plane. Traffic in Istio is divided into data plane traffic and control plane traffic. Data plane traffic refers to the messages sent and received by the business logic of the workload, while control plane traffic refers to the configuration and control messages sent between Istio components to orchestrate the behavior of the mesh. Traffic management in Istio specifically refers to data plane traffic. The Envoy proxy (which can also be called the Istio proxy) is a high-performance proxy used to coordinate inbound and outbound traffic for all services in the service mesh. The Envoy proxy is the only Istio component that interacts with data plane traffic. The Envoy proxy is deployed as a sidecar proxy for services, logically adding many of its built-in features, such as dynamic service discovery, load balancing, TLS endpoints, HTTP / 2 and gRPC proxying, and fault injection. This sidecar proxy deployment allows Istio to extract a large amount of signals about traffic behavior as attributes. Istio can use these attributes to implement policy decisions and send them to the monitoring system to provide information about the behavior of the entire mesh.

[0036] The Istio mesh control plane consists of the Istiod binary, responsible for translating high-level routing rules and traffic control behaviors into Envoy-specific configurations and then sending them to the sidecar agents at runtime. The Pilot component in the sidecar agent control plane provides Envoy sidecar agents with service discovery, intelligent routing traffic management, and resilience features (e.g., timeouts, retries, circuit breakers, etc.). The Pilot component translates high-level routing rules controlling traffic behavior into environment-specific configurations and distributes them to the sidecar agents at runtime. The Pilot component abstracts platform-specific service discovery mechanisms and integrates them into a standard format that any sidecar agent conforming to the Envoy API can use. The Citadel component in the control plane provides security measures, supporting robust service-to-service and end-user authentication through built-in identity and certificate management. The Galley component in the control plane is used for configuration validation, extraction, processing, and distribution validation for the Istio API. The Galley component is responsible for isolating other Istio components from the details of retrieving user configurations from the underlying platform (e.g., Kubernetes). The data plane consists of a collection of proxy services that utilize an extended Envoy proxy server, manifested as sidecar proxy containers within each Pod. These sidecar proxies coordinate and control network communication between microservices, while simultaneously collecting and reporting useful telemetry data. Sidecar proxies can add additional functionality to network traffic management, including providing security, privacy protection, and policies. In inter-service call scenarios, proxies can hide the topology details of the service backend for clients, simplifying the complexity of interactions and protecting backend services from overload. For example, when creating a Pod on the Kubernetes platform, the Kubernetes API server calls the sidecar proxy injection service of the Istio control plane component to automatically modify the microservice description information and inject the sidecar proxy. When the Pod is actually created, the sidecar proxy container is created simultaneously within the Pod that creates the microservice container. Sidecar proxies allow adding Istio functionality to existing deployments without requiring re-architecting or code rewriting.

[0037] Figure 4 Examples Figure 2 A schematic diagram of an example architecture 400 for the controller component. (See diagram for example.) Figure 4As shown, the controller component includes a main module 410, a controller module 412, an EnvoyFilter module 414, and a refresh module 420. The main module 410 is responsible for initializing each module, enabling listening for CR resources 402 and Kubernetes events 404, gRPC requests 406, and timed triggers 408, and starting the port, etc. The controller module 412 is responsible for listening to and parsing CR resources 402. The refresh module 420 is responsible for listening to various events that trigger rate limiting updates (such as Kubernetes events 404, gRPC requests 406, and timed triggers 408) and calculating the rate limiting threshold. The EnvoyFilter module 414 is responsible for generating and updating the EnvoyFilter configuration 430 (which can also be called the EnvoyFilter resource), which contains the rate limiting thresholds for the microservices.

[0038] The refresh module 420 includes a notification module 421, a timing module 422, a gRPC module 423, a management module 424, a calculation module 425, and a data acquisition module 426. The notification module 421 listens for Kubernetes events 404 (e.g., deployment update) and triggers a refresh upon receiving the event. The timing module 422 starts a timer, which triggers a refresh upon expiration. The gRPC module 423 receives external requests to update the rate limiting threshold (gRPC request 406) and triggers a refresh upon receiving the request. The data acquisition module 426, for example, extracts data from Prometheus 440 to calculate the rate limiting threshold. The calculation module 425 executes the algorithm for calculating the rate limiting threshold. The management module 424 manages the various sub-modules within the refresh module 420 and interacts with other modules outside the refresh module 420, such as registering components upon receiving a notification from the controller module 412 and calling the EnvoyFilter module 414 to generate the EnvoyFilter configuration 430.

[0039] For reference Figure 2 As described, the controller component parses the received custom rate-limiting resources. Figure 5 This is a flowchart illustrating the process of parsing a custom rate-limiting resource according to an implementation scheme of this disclosure. For example... Figure 5As shown, in step 502, the controller component listens for an event related to CR configuration (i.e., the custom rate-limiting resource). In response to the CR configuration event, in step 504, it checks if an EnvoyFilter exists; specifically, it searches for an EnvoyFilter with the corresponding name based on the CR name. If an EnvoyFilter with the corresponding name exists, process 500 proceeds to step 505 to determine if rate limiting is enabled, for example, by checking if the localRateLimit.enbale field in the EnvoyFilter is false ("No"). If rate limiting is disabled, process 500 proceeds to step 506 to disable the rate limiting configuration; the filter_enforced field for the corresponding port in the EnvoyFilter is set to 0, and finally, the CR status is updated in step 530. If rate limiting is not disabled, process 500 proceeds to step 507 to update the rate limiting configuration; the rate limiting configuration for the corresponding port in the EnvoyFilter is updated, and finally, the CR status is updated in step 530. If it is determined in step 504 that an EnvoyFilter with the corresponding name does not exist, then process 500 proceeds to step 508 to determine whether the rate limiting function is disabled, for example, by determining whether the localRateLimit.enbale field in the EnvoyFilter is false ("No"). If the rate limiting function is disabled, then process 500 proceeds to step 509, terminates the process, does not create an EnvoyFilter, and finally updates the CR status in step 530. If the rate limiting function is disabled, then an EnvoyFilter is created, and process 500 proceeds to step 510. In step 510, the matching selector tag is parsed. Then, in step 511, the matching method is confirmed. If it is a tag match, then process 500 proceeds to step 512 to record the matching tag. If it is a service match, then process 500 proceeds to step 513 to find the service, and then records the service's selector tag in step 514. After recording the selector tag, the rate limiting parameters corresponding to the selector tag are parsed sequentially, step 515. Then, the EnvoyFilter configuration is generated based on the parsed rate limiting parameters, step 516. The refresh policy can be parsed from the EnvoyFilter configuration (step 517). Based on the parsed refresh policy, scheduled refresh can be enabled. First, in step 518, it is determined whether scheduled refresh should be started, for example, by checking if the refreshPolicy.enbale field is false ("No"). If scheduled refresh is not enabled, process 500 proceeds to step 519, terminates the process, and finally updates the CR status in step 530. If scheduled refresh is enabled, process 500 proceeds to step 520, starts the scheduled refresh task, notifies the module to listen for Kubernetes events, and finally updates the CR status in step 530.

[0040] Figure 6 A flowchart illustrating a resource flow limiting protection process 600 according to an embodiment of this disclosure is provided. Figure 6 As shown, process 600 begins at step 602, calling external resources such as API server resources or monitoring component (Prometheus) resources. After calling the external resources, it is determined whether the traffic exceeds the rate limiting threshold, step 604. If the traffic does not exceed the rate limiting threshold, process 600 proceeds to step 605, calling the target resource to process the business request. If the traffic exceeds the rate limiting threshold, process 600 proceeds to step 606, determining whether the CR parsing process has encountered an error. In response to an error in the CR parsing process, the CR parsing process is entered, and the CR task is re-added to the CR processing queue, step 608. In response to no error in the CR parsing process, the threshold update process is entered, and the CR task is re-added to the threshold refresh delay queue according to the configured time in the CR, step 610.

[0041] As described above, an Istio proxy and a microservice container are created within the same microservice Pod. The Istio proxy has local rate limiting capabilities, and traffic is sent to the microservice container after passing through the Istio proxy. Specifically, when an external call is made to the microservice Pod, the incoming business request undergoes local rate limiting judgment at the Istio proxy. Business requests that are not rate-limited are sent to the microservice container, while rate-limited business requests return the corresponding error code. Figure 7 This diagram illustrates the token bucket rate limiting method 700 for Istio proxies. Figure 7 As shown, in step 702, tokens are generated at a timed interval, and then in step 704, it is determined whether the token bucket 712 is full. If the token bucket 712 is full ( Figure 7 If the token is "yes" in the code, then method 700 proceeds to step 706 and discards the token; if the token bucket is not full ( Figure 7 If the request is "No" in the token bucket 710, then the token is injected into the token bucket 712. When the service request 710 arrives, a token T is taken from the token bucket 712, and the request is successful. If there is no token available in the token bucket 712, the service request 710 is rejected, the request fails, and an error code is returned. Those skilled in the art should understand that the token bucket rate limiting method 700 is an example implementation of the local rate limiting algorithm of the Istio proxy, and other rate limiting algorithms can be used for the local rate limiting method of the Istio proxy.

[0042] Figure 8 A schematic diagram illustrating flow 800 of a rate limiting threshold algorithm according to an embodiment of this disclosure is shown. For example... Figure 8As shown, process 800 includes steps 802-808. In step 802, the maximum traffic of the microservice when the CPU utilization allocated to the microservice is less than or equal to the CPU utilization expansion threshold is multiplied by a first coefficient to obtain the initial rate limiting value, where the first coefficient is greater than 1 and less than the ratio of the target CPU utilization to the expansion threshold. The goal of the algorithm is to stabilize the CPU utilization allocated to the microservice at the traffic value corresponding to the target utilization (e.g., 85%). The target utilization is a balance point selected based on common empirical values, aiming to maximize resource utilization while ensuring service performance. According to the barber model, when resource utilization increases to a certain level, the business will experience increased latency, failures, etc. The utilization at this point is the maximum value of resource utilization. For example, when CPU utilization increases, latency increases. Different businesses have different tolerances for latency, and therefore different tolerances for CPU utilization. Empirical thresholds for CPU utilization include the horizontal Pod auto-scaler (HPA) expansion line, the test baseline, the CPU alarm line, and the CPU full load line. In one embodiment, the HPA expansion threshold is 60%, the test baseline is 70%, the CPU alarm threshold is 90%, and the CPU full load threshold is 100%. To improve resource utilization, the average of the test baseline (70%) and the CPU full load threshold (100%), 85%, is selected as the target utilization rate. When the CPU demand of a Pod exceeds the upper limit, the Pod may be rate-limited by the node, and the Istio proxy in the Pod will also be rate-limited. Therefore, the target CPU utilization rate is selected based on the Pod's requests. As mentioned above, the traffic value corresponding to the CPU utilization rate equaling the target utilization rate is used as the rate-limiting threshold. Rate-limiting traffic through the rate-limiting threshold can prevent the CPU utilization rate of the service from continuing to increase and stabilize it at the target utilization rate (e.g., 85%). However, since the service is configured with HPA, when the average CPU utilization rate reaches the HPA expansion threshold (e.g., 60%), the CPU allocated to the service will be expanded. Therefore, it is not possible to directly obtain the traffic value corresponding to the CPU utilization rate at the target utilization rate, but it needs to be estimated based on the actual traffic reached by the service. In theory, the rate limiting threshold can be obtained by measuring the traffic volume when the current CPU utilization equals the target utilization. However, CPU data is obtained through monitoring components (e.g., Prometheus), so when traffic surges instantaneously, the CPU metric may not be displayed in real time, leading to an inflated traffic value. Therefore, it is necessary to reduce the impact of data collection on the CPU metric. Since the average CPU utilization of a normal service is at most the hpa expansion threshold, the rate limiting threshold algorithm uses the maximum traffic volume under normal service conditions (i.e., the maximum traffic volume of the service when CPU utilization is less than or equal to the expansion threshold) as the traffic value corresponding to the CPU utilization equaling the hpa expansion threshold.Multiplying the maximum traffic under normal service conditions by a magnification factor (i.e., the first factor) yields the traffic value corresponding to CPU utilization equal to the target utilization. This magnification factor should be less than the ratio of the CPU's target utilization to the HPA expansion limit. In one embodiment, the CPU's target utilization is 85%, the HPA expansion limit is 60%, and this magnification factor is 1.3.

[0043] The deployment of certain services results in their CPU utilization being below the HPA expansion threshold under normal circumstances. For example, some services may have CPU utilization at half the HPA expansion threshold under normal circumstances, leading to a lower calculated rate limiting value and underutilization of service resources. Therefore, to prevent services with low CPU utilization under normal circumstances from calculating overly conservative rate limiting thresholds, an additional amplification factor (i.e., the second factor) can be introduced for correction. Since the HPA expansion threshold is anchored by the average CPU utilization of the service under normal circumstances, the average CPU utilization should be selected when considering CPU utilization. Considering the peak and off-peak traffic, a 24-hour period is selected. During off-peak hours, the service processes less traffic and has lower CPU utilization, which may result in a higher rate limiting threshold when calculating the amplification factor. Therefore, the peak hours are selected. Furthermore, since the purpose of calculating this amplification factor is to identify traffic with CPU utilization below the HPA expansion threshold, traffic with CPU utilization above the expansion threshold should be excluded. In step 804, the expansion threshold is divided by the maximum CPU utilization to obtain a second coefficient, where the maximum utilization is the smaller of the maximum average utilization of the CPU per unit time over the most recent predetermined period (e.g., 24 hours) and the expansion threshold. In step 806, the initial current limit value is multiplied by the second coefficient to obtain the corrected current limit value.

[0044] If increased traffic causes CPU utilization to rise to the target utilization level before HPA expansion is completed, the traffic value at this point represents the actual traffic the service can handle when CPU utilization reaches the target level. Therefore, this actual traffic also needs to be considered. To absorb effective information under real high load, the algorithm synchronously records CPU utilization in step 808 and uses the larger of the microservice's maximum traffic when CPU utilization is less than or equal to the target utilization and the corrected rate limiting value as the microservice's rate limiting threshold. Figure 8 The rate limiting threshold algorithm illustrated in the example can make intelligent decisions based on historical and real-time data, ensuring high accuracy of the rate limiting threshold, thereby significantly improving the inherent resilience and reliability of microservices in the face of sudden traffic surges.

[0045] Furthermore, when traffic spikes only momentarily (on the order of seconds) and then immediately recovers, it's impossible to collect CPU data at that instant, so such values ​​need to be excluded. To eliminate noise from instantaneous traffic spikes or data collection anomalies, outliers can be eliminated statistically before calculation. For example, the Interquartile Range (IQR) algorithm can be used to identify outliers, ensuring the robustness and reliability of the dataset used in the calculation. Additionally, traffic has peaks and troughs; calculating outliers together will lead to underestimating the results. Therefore, to eliminate the impact of intraday traffic fluctuations (peaks / troughs), a time-segmented calculation strategy is adopted. Historical data for each day is first aggregated in hourly windows, and then compared horizontally with data from the same time period each day (e.g., all from 09:00-10:00), using IQR to eliminate outliers. This method, compared to clustering algorithms, is more engineering-simple and interpretable, accurately capturing the cyclical patterns of business operations. In some implementations, traffic metrics for microservices are aggregated according to unit time windows; and outlier filtering is performed on traffic metrics within each unit time window by comparing data from the same unit time window each day.

[0046] In some implementations, in response to a difference greater than a first predetermined value between the calculated rate limiting threshold and the current rate limiting threshold, the rate limiting threshold for the microservice is calculated based on historical data of traffic and system resource metrics for the microservice with an added predetermined time. Furthermore, in response to a difference less than or equal to a second predetermined value between the rate limiting threshold calculated based on historical data with the added predetermined time and the rate limiting threshold calculated without the added predetermined time, the rate limiting threshold calculated based on historical data without the added predetermined time is stored in the control plane configuration for microservice rate limiting. If the calculated new threshold differs significantly from the currently effective threshold, the system will automatically expand the data collection time range for recalculation and verification to avoid drastic threshold fluctuations due to data anomalies within a single time window. Only after the new threshold passes stability verification will it be safely issued and take effect, thus ensuring a smooth system transition.

[0047] As mentioned above, the rate limiting threshold can be refreshed either periodically or via an event. Typically, upon receiving a trigger event to refresh the rate limiting threshold, the type of the trigger event is first determined, and different processing procedures apply to different types of trigger events. The following section combines... Figures 9-10 This will illustrate the process of triggering and refreshing the rate limiting threshold in different ways.

[0048] The refresh response to the rate limiting threshold is triggered periodically, using methods such as... Figure 9 The algorithm shown is a rate limiting threshold algorithm for timed triggering. Figure 9 This is a schematic diagram illustrating an example flow 900 of a rate-limiting threshold refresh algorithm according to one embodiment of the present disclosure. Figure 9As shown, process 900 includes steps 902-906. In step 902, traffic metrics and system resource metrics for the microservice are aggregated over the most recent several days (e.g., 7 days) according to a unit time window (e.g., 1 hour). Then, in step 904, outlier filtering of traffic metrics is performed by comparing traffic metrics data from the same unit time window over the several days, for example, using an IQR algorithm. In step 906, a rate limiting threshold for the microservice is calculated based on the filtered traffic metrics and system resource metrics over the several days. The calculation of the rate limiting threshold for the microservice can be found in [reference needed]. Figure 8 The described process is 800. In some implementations, the refresh period for the rate limiting threshold can be set. In some embodiments, the refresh period for the rate limiting threshold can be 1 day.

[0049] The refresh response to the rate limiting threshold is triggered by an external event, using, for example... Figure 10 The algorithm shown is a rate limiting threshold algorithm triggered by external events. External triggering events can include at least one of deployment update events (Kubernetes events) and rate limiting alarm events (GRPC requests). Figure 10 This is a schematic diagram illustrating an example flow 1000 of a current limiting threshold algorithm according to another embodiment of this disclosure. For example... Figure 10 As shown, after receiving an external trigger event in step 1002, the traffic metric and system resource metric for the microservice within the most recent first time window (e.g., 10 minutes) are obtained (step 1004). Then, in step 1006, the rate limiting threshold for the microservice is calculated based on the traffic metric and system resource metric for the microservice within the first time window. In some embodiments, calculating the rate limiting threshold for the microservice in step 1006 may include: multiplying the maximum traffic of the microservice when the CPU utilization of the CPU allocated to the microservice within the first time window is less than or equal to a scaling threshold of CPU utilization by a first coefficient to obtain an initial rate limiting value, wherein the first coefficient is greater than 1 and less than the ratio of the target CPU utilization to the scaling threshold; and using the larger of the maximum traffic of the microservice when the CPU utilization is less than or equal to the target utilization within the first time window and the initial rate limiting value as the rate limiting threshold for the microservice. Subsequently, in step 1008, the timed refresh period of the rate limiting threshold for the microservice is set to a first period value (e.g., 30 minutes), and the time window on which the data for subsequently refreshing the rate limiting threshold for the microservice is based is doubled. In other words, the rate limiting threshold for the microservice will be refreshed periodically. For periodic updates of microservice rate limiting thresholds, please refer to... Figure 8The described process is 800. In some implementations, after refreshing the microservice's rate limiting threshold in response to an external trigger event, the refresh cycle for the next scheduled refresh of the microservice's rate limiting threshold can be double the previous refresh cycle, and the time window of the data on which the next scheduled refresh of the microservice's rate limiting threshold is based increases sequentially. For example, if the time window of the data on which the microservice's rate limiting threshold is based in response to an external trigger event is 10 minutes, the time windows of the data on which the microservice's rate limiting threshold is based for subsequent scheduled refreshes will be 20 minutes, 40 minutes, 80 minutes, and so on, up to a preset upper limit (e.g., a 7-day time window). Furthermore, in one embodiment, after refreshing the microservice's rate limiting threshold in response to an external trigger event, the next scheduled refresh cycle for the microservice's rate limiting threshold is 30 minutes, and the subsequent scheduled refresh cycles are 1 hour, 2 hours, 4 hours, 12 hours, 1 day, 2 days, and so on. During the scheduled refresh of the microservice's rate limiting threshold, if an external trigger event is received, the time window of the data on which the microservice's rate limiting threshold is based and the refresh cycle of the microservice's rate limiting threshold will be reset.

[0050] In some implementations, in response to the difference between the calculated rate limiting threshold and the current rate limiting threshold being greater than a third predetermined value, the refresh cycle of the microservice's rate limiting threshold is set to twice the current refresh cycle; and in response to the difference between the calculated rate limiting threshold and the current rate limiting threshold being less than a third threshold, the refresh cycle of the microservice's rate limiting threshold is maintained at the current refresh cycle. In other words, if the rate limiting threshold calculated each time fluctuates significantly from the previous one, the refresh rate is increased; if the rate limiting threshold calculated each time is essentially the same, indicating that the rate limiting threshold is stable, the refresh rate is maintained. In another implementation, in response to the difference between the calculated rate limiting threshold and the current rate limiting threshold being less than a fourth threshold, the refresh cycle of the microservice's rate limiting threshold is set to half the current refresh cycle. In other words, if the rate limiting threshold calculated each time is essentially the same, indicating that the rate limiting threshold is stable, the refresh rate is decreased.

[0051] Figure 11 This is an exemplary block diagram illustrating a computing device according to some embodiments of the present disclosure.

[0052] like Figure 11 As shown, computing device 1100 may include one or more processors 1110 and memory 1120. The one or more processors 1110 are communicatively coupled to memory 1120 and configured to perform the methods discussed above.

[0053] Examples of processor 1110 include microprocessors, microcontrollers, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functions described in this disclosure.

[0054] Processor 1110 can execute software. Software should be broadly interpreted as instructions, instruction sets, code, code segments, program code, programs, subroutines, software modules, application programs, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., regardless of whether it is referred to as software, firmware, middleware, microcode, hardware description languages, or otherwise. The software may reside on memory 1120.

[0055] Memory 1120 may be a non-transitory computer-readable medium. Non-transitory computer-readable media include, for example, magnetic storage devices (e.g., hard disks, floppy disks, magnetic stripes), optical disks (e.g., optical discs (CDs) or digital versatile optical discs (DVDs)), smart cards, flash memory devices (e.g., cards, memory cards, or key drives), random access memory (RAM), read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), registers, removable disks, and any other suitable medium for storing software and / or instructions that can be accessed and read by a computer. Memory 1120 may reside in processor 1110, be external to processor 1110, or be distributed across multiple entities including processor 1110. Memory 1120 may be embodied in a computer program product. For example, a computer program product may include a computer-readable medium in packaging material. Those skilled in the art will recognize how the functions described throughout this disclosure can be implemented based on the specific application and overall design constraints imposed on the overall system.

[0056] Furthermore, according to another embodiment of this disclosure, a computer program product for wireless communication is disclosed. As an example, the computer program product includes a non-transitory computer-readable storage medium containing program instructions executable by a processor. When executed, the program instructions cause the processor to perform one or more of the processes described above; details are omitted here for brevity.

[0057] This invention can be a system, method, and / or computer program product at any possible level of integration technical detail. The computer program product may include computer-readable program instructions for causing a processor to perform various aspects of this disclosure.

[0058] Unless otherwise stated, terms such as “if,” “when,” and “although” should be interpreted as “under the condition of,” rather than implying an immediate temporal relationship or response. That is, these phrases, such as “when,” do not imply an immediate action in response to an action occurring or during an action, but merely suggest that an action will occur if the condition is met, but does not require a specific or immediate time constraint for the action to occur. Combinations such as “at least one of A, B, or C,” “one or more of A, B, or C,” and “at least one of A, B, and C” include any combination of A, B, and / or C, and may include multiple A, multiple B, or multiple C. Combinations such as “at least one of A, B, or C” and “at least one of A, B, and C” can be only A, only B, only C, A and B, A and C, B and C, or A, B, and C, wherein any such combination may contain one or more members of A, B, or C.

[0059] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible structure, function, and operation of the methods and apparatus according to various embodiments of this application. In this respect, each block in the flowchart or block diagram may represent a module, a program segment, or a portion of code containing at least one executable instruction for implementing a specified logical function. It should also be noted that in some alternative embodiments, the functions described in a block may occur in a different order than those described in the accompanying drawings. For example, two blocks shown consecutively may actually be executed in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware system that performs the specified function or operation, or by a combination of dedicated hardware and computer instructions.

[0060] The various embodiments described in this disclosure are for illustrative purposes and are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein is chosen to best explain the principles of the embodiments, their practical application, or improvements to techniques found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for rate limiting in microservices, the method comprising: A control plane configuration for microservice rate limiting is generated based on the rate limiting requirements of the microservice and according to the traffic and system resource metrics of the microservice; The control plane component of the service mesh converts the control plane configuration for microservice rate limiting into the data plane configuration for microservice rate limiting of the service mesh and pushes it to the service mesh proxy; as well as The service mesh proxy implements rate limiting on traffic sent to the microservice based on the data plane configuration of the microservice rate limiting.

2. The method according to claim 1, wherein, The control plane configuration for the microservice rate limiting includes the rate limiting threshold for the microservice.

3. The method according to claim 1, wherein, The system resource metrics include CPU utilization of the central processing unit CPU allocated to the microservice, and the traffic metrics and system resource metrics for the microservice are periodically obtained by a monitoring component outside the microservice pod of the microservice.

4. The method according to claim 2, wherein, The control plane configuration for microservice rate limiting, based on the rate limiting requirements of the microservices and generating a service mesh according to the traffic metrics and system resource metrics of the microservices, includes: The rate limiting threshold for the microservice is calculated using the following steps: When the CPU utilization of the central processing unit CPU allocated to the microservice is less than or equal to the expansion threshold of the CPU utilization, the maximum traffic of the microservice will be multiplied by a first coefficient to obtain an initial rate limit value, wherein the first coefficient is greater than 1 and less than the ratio of the target CPU utilization to the expansion threshold. The second coefficient is obtained by dividing the expansion threshold by the maximum utilization of the CPU, wherein the maximum utilization is the smaller of the maximum value of the average utilization of the CPU per unit time in the most recent predetermined time and the expansion threshold. Multiply the initial current limit value by the second coefficient to obtain the corrected current limit value; and The larger of the microservice's maximum traffic when the CPU utilization is less than or equal to the target utilization and the corrected rate limiting value is used as the rate limiting threshold for the microservice.

5. The method according to claim 1, wherein, The control plane configuration for microservice rate limiting, based on the rate limiting requirements of the microservices and generating a service mesh according to the traffic metrics and system resource metrics of the microservices, includes: Aggregate traffic metric data for the microservice according to a unit time window; and Outlier filtering is performed on traffic metrics for each time window by comparing data from the same time window each day.

6. The method according to claim 4, wherein, The control plane configuration for microservice rate limiting, which generates a service mesh based on the rate limiting requirements of the microservice and according to the traffic and system resource metrics of the microservice, also includes: In response to a situation where the difference between the calculated rate limiting threshold and the current rate limiting threshold exceeds a first predetermined value, the rate limiting threshold for the microservice is calculated based on historical data of traffic metrics and system resource metrics for the microservice, extended by the predetermined time; and In response to the fact that the difference between the rate limiting threshold calculated based on historical data with the predetermined time added and the rate limiting threshold calculated based on historical data without the predetermined time added is less than or equal to a second predetermined value, the rate limiting threshold calculated based on historical data without the predetermined time added is stored in the control plane configuration of the microservice rate limiting.

7. The method according to claim 1, wherein, The control plane configuration for the microservice rate limiting is generated periodically or in response to externally triggered events.

8. The method according to claim 7, wherein, The external triggering events include at least one of deployment update events and rate limiting alarm events.

9. The method according to claim 7, wherein, The control module generates the control plane configuration for microservice rate limiting in response to externally triggered events, including: Obtain the most recent traffic and system resource metrics for the microservice within the first time window; and The rate limiting threshold for the microservice is calculated based on the traffic metric and system resource metric for the microservice within the first time window.

10. The method according to claim 9, wherein, The rate limiting threshold for the microservice is calculated based on the traffic and system resource metrics for the microservice within the first time window, including: When the CPU utilization of the central processing unit (CPU) allocated to the microservice within the first time window is less than or equal to a scaling threshold of the CPU utilization, the maximum traffic of the microservice is multiplied by a first coefficient to obtain an initial rate limit value, wherein the first coefficient is greater than 1 and less than the ratio of the target CPU utilization to the scaling threshold; and The larger of the microservice's maximum traffic and the initial rate limiting value when the CPU utilization is less than or equal to the target utilization within the first time window will be used as the rate limiting threshold for the microservice.

11. The method according to claim 9, wherein, The control plane configuration for generating the microservice rate limiting in response to externally triggered events also includes: Set the timed refresh period of the control plane configuration for the microservice rate limiting to a first period value; and The time window on which the control plane configuration for subsequent updates to the microservice rate limiting is based will be doubled.

12. The method according to claim 7, wherein, The control module periodically generates the control plane configuration for the microservice rate limiting, including: Aggregate the traffic metrics and system resource metrics for the microservice over the most recent days according to a unit time window; Outlier filtering is performed on the traffic metrics by comparing data from the same unit time window over the multiple days; and The rate limiting threshold for the microservice is calculated based on the filtered traffic metrics over the specified number of days and the system resource metrics.

13. The method according to claim 12, wherein, In response to a situation where the difference between the calculated rate limiting threshold and the current rate limiting threshold exceeds a third predetermined value, the refresh cycle of the control plane configuration for the microservice rate limiting is set to twice the current refresh cycle; and If the difference between the calculated rate limiting threshold and the current rate limiting threshold is less than the third threshold, the refresh cycle of the control plane configuration for the microservice rate limiting is maintained at the current refresh cycle.

14. The method according to claim 7, wherein, The control module periodically generates the control plane configuration for the microservice rate limiting, including: After calculating the rate limiting threshold, the refresh cycle of the control plane configuration for the microservice rate limiting is set to twice the current refresh cycle, and the time window on which the data for refreshing the control plane configuration for the microservice rate limiting is based is doubled.

15. A system for rate limiting in microservices, the system comprising: One or more processors; A memory coupled to the one or more processors and storing computer program instructions; Wherein, when the computer program instructions are executed by the one or more processors, the one or more processors cause the one or more processors to perform the following operations: A control plane configuration for microservice rate limiting is generated based on the rate limiting requirements of the microservice and according to the traffic and system resource metrics of the microservice; The service mesh's control plane component converts the microservice rate limiting control plane configuration into the service mesh's microservice rate limiting data plane configuration for the microservice and pushes it to the service mesh's proxy; and The service mesh proxy applies rate limiting to traffic sent to the microservice based on the data plane configuration of the microservice rate limiting.

16. The system according to claim 15, wherein, The control plane configuration for the microservice rate limiting includes the rate limiting threshold for the microservice.

17. The system according to claim 15, wherein, The system resource metrics include CPU utilization of the central processing unit CPU allocated to the microservice, and the traffic metrics and system resource metrics for the microservice are periodically obtained by a monitoring component outside the microservice pod of the microservice.

18. The system according to claim 16, wherein, The control plane configuration for microservice rate limiting, based on the rate limiting requirements of the microservices and generating a service mesh according to the traffic metrics and system resource metrics of the microservices, includes: The rate limiting threshold for the microservice is calculated using the following steps: When the CPU utilization of the central processing unit CPU allocated to the microservice is less than or equal to the expansion threshold of the CPU utilization, the maximum traffic of the microservice will be multiplied by a first coefficient to obtain an initial rate limit value, wherein the first coefficient is greater than 1 and less than the ratio of the target CPU utilization to the expansion threshold. The second coefficient is obtained by dividing the expansion threshold by the maximum utilization of the CPU, wherein the maximum utilization is the smaller of the maximum value of the average utilization of the CPU per unit time in the most recent predetermined time and the expansion threshold. Multiply the initial current limit value by the second coefficient to obtain the corrected current limit value; and The maximum value between the maximum traffic of the microservice when the CPU utilization is less than or equal to the target utilization and the corrected rate limiting value is used as the rate limiting threshold of the microservice.

19. The system according to claim 15, wherein, The control plane configuration for the microservice rate limiting is generated periodically or in response to externally triggered events.

20. A computer program product comprising computer-readable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 14.

Images