Micro-service request overload control device and method with request service priority sensing capability

Abstract

The invention discloses a micro-service request overload control device and method with request service priority sensing capability, and relates to the technical field of micro-services, and the device comprises a token generation container module which is used for generating a preset number of tokens; the micro-service price setting module is used for setting an initial admission price for each micro-service and updating the admission price according to the CPU utilization rate; the token distribution module is used for determining the micro-service required by the request and distributing the required token for the request according to the micro-service; when the CPU utilization rate is higher than a preset threshold value, the number of tokens needing to be distributed to each request in each token generation period is calculated through an integer linear programming method according to the priority of the request; and the gateway module is used for judging whether the number of tokens allocated to the request reaches the access price or not, and receiving the request of reaching the access price. According to the application, the admission price can be set in combination with the CPU utilization rate, and the request rate of high service value is prevented from being limited by mistaken discarding in combination with the priority.

Description

Technical Field

[0001] The present application relates to the field of microservice technology, and in particular to a microservice request overload control device and method with request service priority perception capability. Background Art

[0002] Traditional overload control methods, such as rate limiting and active queue management (e.g., limiting the rate of all requests when overloaded, and directly denying incoming requests when the server is overloaded), can alleviate overload problems to a certain extent. However, these methods are often based on simple rules and cannot adapt to the dynamically changing load patterns and complex dependencies in microservice architectures. Currently, academia and industry have proposed a variety of technical solutions for overload control, including rate limiting on the sender side, active queue management on the receiver side, and signal-driven adaptive control strategies. While these methods can alleviate local overload problems to a certain extent, they generally suffer from several drawbacks: First, they lack adaptability to multi-tier service topologies and have difficulty capturing the load propagation patterns in cross-service dependencies common in microservice architectures, making it difficult to understand the overall load situation of microservices. Second, they lack effective discrimination between request priority and business criticality. When rate limiting is required due to overload, they may mistakenly discard requests with high business value (such as user access and payment transaction requests). Summary of the Invention

[0003] The purpose of this application is to provide a microservice request overload control device and method with the ability to perceive the priority of request business, which can set the access price in combination with CPU utilization, and limit the request rate of high business value in combination with priority to avoid erroneous discarding.

[0004] To achieve the above objectives, this application provides the following solutions:

[0005] In a first aspect, the present application provides a microservice request overload control device with request service priority perception capability, comprising:

[0006] A token generation container module, configured to generate a preset number of tokens every other token generation cycle and store unallocated tokens;

[0007] A microservice price setting module is used to set an initial admission price for each microservice and update the admission price based on CPU utilization; wherein the price is the number of tokens required;

[0008] Token allocation module, used to:

[0009] Determine the microservices required by the request and assign the required token to the request based on the required microservices;

[0010] When the CPU utilization rate is higher than a preset threshold, the number of tokens to be allocated to each request in each token generation cycle is calculated using an integer linear programming method based on the priority of the request; the priority of the request is a preset priority;

[0011] Allocate tokens to each request in each token generation cycle according to the number of tokens;

[0012] The gateway module is used to determine whether the number of tokens allocated to the request reaches the admission price, and receive the request that reaches the admission price.

[0013] In a second aspect, the present application provides a microservice request overload control method with request business priority awareness capability, comprising:

[0014] Generate a preset number of tokens every token generation cycle through a token generation container module, and save unallocated tokens;

[0015] Through the microservice price setting module, an initial admission price is set for each microservice, and the admission price is updated according to CPU utilization; wherein the price is the number of tokens required;

[0016] The token allocation module determines the microservices required by the request and allocates the required tokens to the request based on the required microservices. When the CPU utilization rate is higher than a preset threshold, the number of tokens required to be allocated to each request in each token generation cycle is calculated using an integer linear programming method based on the priority of the request. The priority of the request is a preset priority. Tokens are allocated to each request in each token generation cycle based on the number of tokens.

[0017] Through the gateway module, it is determined whether the number of tokens allocated to the request reaches the admission price, and the request that reaches the admission price is received.

[0018] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0019] The present application provides a microservice request overload control device and method with the ability to perceive request business priority, the device comprising: a token generation container module, used to generate a preset number of tokens every token generation cycle and save unallocated tokens; a microservice price setting module, used to set an initial access price for each microservice and update the access price according to CPU utilization; wherein the price is the required number of tokens; a token allocation module, used to: determine the microservice required for the request, and allocate the required tokens to the request according to the required microservice; when the CPU utilization is higher than a preset threshold, the number of tokens required to be allocated to each request in each token generation cycle is calculated by an integer linear programming method according to the priority of the request; the priority of the request is a pre-set priority; tokens are allocated to each request in each token generation cycle according to the number of tokens; a gateway module, used to determine whether the number of tokens allocated to the request reaches the access price, and receive requests that reach the access price. This application introduces the concept of microservice access price, establishes a dynamic mapping relationship between price and resources, uses the access price of microservices to represent the current resource usage, and also designs a price transmission mechanism, so that the overall load of the microservice architecture can be captured in the complex topological dependencies of the microservices, helping to make a judgment on request rate limiting. In addition, in order to avoid the rate of critical business requests being limited when the pressure on microservices is too high, the system divides the requests into different priorities. In each time period, through the linear programming method, the token generation and microservice price are comprehensively considered to limit the number of requests sent in different categories to ensure that high-priority requests are processed first. Therefore, this application can set the access price in combination with CPU utilization, and limit the rate of requests with high business value in combination with priority to avoid erroneous discarding. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 This is a general design diagram of a microservice request overload control device with request service priority awareness provided by an embodiment of the present application;

[0022] Figure 2 A schematic diagram of a request transmission path sent by a gateway to a downstream microservice via RPC provided in one embodiment of the present application;

[0023] Figure 3 A schematic diagram of the RPC response return path provided in one embodiment of the present application;

[0024] Figure 4 A schematic diagram of the overall design of the solution provided for one embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0026] In recent years, with the continuous expansion of software design and the increasing complexity of business requirements, higher system requirements have been placed on performance, high throughput, high stability, and high scalability. Traditional monolithic architectures, due to their high degree of coupling, cumbersome deployment, and complex troubleshooting, have become unable to meet the demands of continuously expanding business needs. Against this backdrop, the microservices architecture has emerged as a key software development framework in the cloud-native field. The microservices architecture partitions applications into a set of independent, self-contained small services, enabling each service to be independently deployed and managed. These services are typically built around specific business capabilities, running in independent processes and interacting through lightweight communication mechanisms such as RPC or RESTful interfaces. Microservices not only support automated deployment but can also be developed in different programming languages ​​and use a variety of data storage technologies, while minimizing centralized management. Currently, many globally renowned companies (such as Netflix, Twitter, Facebook, and Airbnb) are using the microservices architecture to build cloud-based applications. Compared to traditional monolithic architectures, microservices offer greater resource scalability and are better able to address complex and changing business needs and the performance challenges of large-scale systems.

[0027] However, modern cloud applications are becoming increasingly complex. Internet service providers such as Twitter, Netflix, and Amazon deploy complex service graphs comprised of hundreds of microservices. Detecting, diagnosing, and mitigating performance impairments in these applications is challenging. Detection is often delayed because performance impairments from slow microservices often only become apparent after the impact of slow microservices propagates through the service graph. Diagnosis becomes even more complex when application latency is affected by multiple dependent microservices. The decision-making process is further complicated by the need to simultaneously control load and allocate resources to each microservice. Microservice applications are susceptible to unexpected loads caused by a variety of factors, such as sudden traffic surges, failures, retry storms triggered by erroneous clients, underscaling of microservices, instance failures, and issues with the cloud service provider. These situations often lead to breaches of service level objectives (SLOs) and, in extreme cases, service outages. For example, Amazon experienced a service outage due to a massive traffic surge during Prime Day, while Zoom and Microsoft also experienced outages due to increased demand. Traffic fluctuations, both large and small, are not uncommon, so services must have built-in mechanisms to gracefully handle them. Overload is a particularly challenging issue that can cause performance impairments in a service graph. Load fluctuations and traffic bursts often cause an application's computing demands to exceed allocated system resources, resulting in requests and tasks waiting in queues for extended periods and ultimately violating service level objectives (SLOs). Therefore, overload control is crucial for ensuring application performance. Overload limiting is a key component for achieving high availability in container-based architectures. When the number of requests exceeds the container's resource capacity, service quality inevitably degrades, impacting user experience. Furthermore, performance degradation in one container can have a ripple effect on other containers: microservices that depend on the overloaded container will also suffer performance issues. To address this, containers typically use an overload controller. The overload controller is configured with a threshold that specifies the maximum number of requests allowed within a specific time interval. When too many requests arrive, a rate limiter rejects requests exceeding the limit to prevent the container from overloading.

[0028] Overload control frameworks typically communicate performance signals between senders and receivers. These signals trigger corresponding actions, such as rate limiting on the sender or request dropping on the receiver, to manage application performance and system resource utilization. Despite recent progress, designing an effective signal and action mechanism to meet the unique requirements of microservice graphs remains challenging. Several key requirements are crucial for an effective overload control scheme. In a microservices architecture, effective overload control requires topology awareness, request awareness, and target flexibility to ensure system stability and efficient resource utilization. Topology awareness requires that microservices can anticipate the impact of actions based on local system state on the overall application and system. To effectively mitigate overload, control measures should consider the computational path of requests within the microservices graph, thereby achieving application fairness and efficient resource utilization. Request awareness, meanwhile, enables the system to differentiate processing based on request complexity and computational intensity. Because data center workloads often exhibit a heavy-tailed distribution, where a small number of complex queries often require more time or resources than more common, simple queries, request awareness facilitates appropriate resource allocation to address diverse load profiles. Target flexibility is also crucial. When optimizing applications and systems, data center operators must strike a balance between fairness, effective throughput (goodput), and resource efficiency. To this end, overload control mechanisms must be flexible enough to support diverse control strategies through well-designed signals and actions to meet the needs of different application scenarios.

[0029] Previous work on microservice traffic overload control has primarily relied on rate limiting and active queue management (AQM) on senders. These two mechanisms are widely used in networks to prevent congestion. Rate limiting aims to control the request sending rate, preventing an influx of requests that exceeds the system's processing capacity. AQM proactively controls queue lengths before congestion occurs. By preemptively dropping or marking some packets, it prompts senders to reduce their sending rates, thereby avoiding queue overflow and high latency. These two mechanisms work together to prevent system overload and ensure service stability and responsiveness. However, overload control in large-scale microservice systems must address the complexity and dynamics of the system, which can be challenging in practice. First, all microservices must be monitored. If a microservice is not monitored, potential overload may occur at that node and further spread to related upstream microservices, leading to cascading overload in the system. Furthermore, as the number of microservices increases, assessing the load between senders and receivers becomes increasingly difficult. Clients initiating requests may face limited network visibility and inaccurate rate limits. Overloaded queues may drop requests, resulting in a collapse in effective throughput (goodput). Finally, in real-world microservice applications, requests often have specific business priorities. Failure to assess the business importance of requests can result in important requests being discarded while less important ones being admitted. This not only affects service quality but can also prevent key business functions from executing properly, negatively impacting user experience and overall system performance. To address these issues, recent research has proposed receiver-driven overload control at the network and application layers. These techniques manage their own utilization by determining the rate at which senders can request services and consume resources. Both techniques employ basic currency concepts. However, existing approaches fail to fully address the complex interactions between multiple layers or layers of a service graph. Furthermore, few approaches prioritize requests when making admission decisions during traffic overload control, potentially resulting in important requests being discarded while less important ones are admitted.

[0030] The following are application scenarios of microservice overload control technology:

[0031] Container resource management and scheduling optimization: In large-scale cloud computing platforms, the overload control technology of the microservices architecture effectively manages the resource utilization of each container. By monitoring the load status of microservices in real time, especially in scenarios with large traffic fluctuations, the system can optimize container resource allocation through overload control mechanisms to avoid resource conflicts and system crashes.

[0032] Service Quality Assurance and SLA Management: In industries with stringent service quality requirements, such as finance and e-commerce, microservice overload control technology can help ensure that each microservice's response time remains within acceptable limits, avoiding service degradation or interruptions due to overload. By assessing request priority and business importance, the system can intelligently prioritize requests, ensuring the normal operation of critical business functions and meeting predetermined service level agreements (SLAs).

[0033] Overload control for multi-layer microservice dependencies: In a microservice architecture with complex service dependencies, overload control technology effectively manages the load of different microservices based on the topology of the microservice graph. When a request requires multiple microservices to complete, the system intelligently determines the request priority and the load of each service layer, preventing performance degradation of the entire request chain due to overload at a single layer. This allows for more precise control of system load and prevents performance degradation of critical services.

[0034] Cross-service resource coordination and fairness: In large microservice systems, each microservice may face varying loads and resource demands. Overload control technology can coordinate resource competition among these services, ensuring that when the system is under high pressure, some microservices do not overuse resources, preventing others from receiving sufficient resources. Through fair resource allocation, the system can globally balance the load across microservices, improving overall system stability and throughput.

[0035] Currently, only a few works can consider the importance of request admission schemes and request business priorities in traffic control strategies. Designing a microservice traffic overload control system with better performance is an inevitable trend.

[0036] The present application designs a price-based microservice request overload control device with the ability to perceive request business priority. The device introduces the concept of microservice access price, establishes a dynamic mapping relationship between price and resources, uses the access price of microservices to represent the current resource usage, and also designs a price transmission mechanism, so that the load of the entire microservice architecture can be captured in the complex topological dependencies of microservices, helping to make a judgment on request rate limiting. In addition, in order to avoid the rate limiting of key business requests (such as login and payment requests) when the pressure on microservices is too high, the system divides the requests into different priorities. In each time period, through the linear programming method, the token generation and microservice price conditions are comprehensively considered, and the number of requests sent of different categories is limited to ensure that high-priority requests are processed first. The method designed by the present invention combines request rate limiting with request priority access. This method can not only ensure the high availability of the microservice cluster under high concurrent requests, but also give priority to important requests and improve user experience.

[0037] This application designs a microservice traffic overload control system based on traffic overload control in a microservice architecture. It introduces the concept of request execution pricing into microservices and considers the business priority of requests. This system captures the cascading dependencies of microservices to achieve reliable request admission while ensuring the normal operation of key business functions and meeting predetermined service levels.

[0038] This device deploys microservice applications on Kubernetes, using the Istio mesh and its EnvoyProxy to perform request filtering and overload control based on a price specified for the microservice system. It also prioritizes requests based on their priority using an integer linear programming model, resulting in a complete microservice traffic overload control system. This system primarily includes a price-based request filtering device and a priority-based request routing device.

[0039] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0040] In an exemplary embodiment, a microservice request overload control device with request service priority awareness capability is provided, including: a token generation container module, a microservice price setting module, a token allocation module, and a gateway module.

[0041] The token generation container module is used to generate a preset number of tokens every other token generation cycle and store unallocated tokens.

[0042] The microservice price setting module is used to set an initial admission price for each microservice and update the admission price based on CPU utilization; wherein the price is the required number of tokens.

[0043] The formula for updating the admission price based on CPU utilization is:

[0044]

[0045] Among them, price is the latest adjusted admission price, base_price is the initial admission price of each microservice; i is an adjustable elasticity coefficient, λ1 is the price growth rate when the CPU utilization is in the range of 70%-80%, λ2 is the price growth rate when the CPU utilization is in the range of 80\%-90\%. When the resource utilization is higher than 90%, the infinite price is triggered.

[0046] Token allocation module, used to:

[0047] Determine the microservices required by the request and allocate the required tokens to the request based on the required microservices; wherein the tokens required by the request refer to the sum of the tokens required by the microservices required by the request.

[0048] When the CPU utilization is higher than a preset threshold, the number of tokens that need to be allocated to each request in each token generation cycle is calculated based on the priority of the request using the integer linear programming method; the priority of the request is a pre-set priority; and tokens are allocated to each request in each token generation cycle based on the number of tokens.

[0049] The number of tokens that need to be allocated to each request in each token generation cycle is calculated using an integer linear programming method, specifically including:

[0050] Using a weighted objective function, the number of tokens that need to be allocated to each request in each token generation cycle is calculated;

[0051] Among them, the weighted objective function is:

[0052]

[0053] V1×X1+V2×X2+V3×X3+V4×X4+V5×X5+V6×X6≤T;

[0054] Among them, k1, k2 and k3 represent the values ​​of three priorities respectively; X1, X2, X3, X4, X5 and X6 represent the number of tokens that need to be allocated to the six requests in each token generation cycle respectively; V1, V2, V3, V4, V5 and V6 represent the number of tokens required to send the six requests respectively; T represents the total number of tokens generated in each token generation cycle.

[0055] The gateway module is used to determine whether the number of tokens allocated to the request reaches the entry price and receive the request that reaches the entry price. The entry price required for the request is the sum of the entry prices of the microservices required by the request.

[0056] The gateway module is further configured to control the number of requests sent during the current cycle to be maintained between 60% and 120% of the upper limit of the number of requests sent during the previous cycle.

[0057] Specifically, see Figure 1The overall technical solution process begins with the gateway service (gateway module). A token generation module is deployed within the gateway service. This module generates tokens at a fixed rate and allocates an appropriate number of tokens based on the microservice prices along the request path. (When allocating tokens to requests, the business priority of the request is taken into account, and integer linear programming is used to dynamically adjust the number of requests sent for different categories to ensure that high-priority requests are processed first.)

[0058] Each subsequent request carries the corresponding token as it flows to downstream microservices, which then decide whether to admit the request based on their own dynamic pricing. Meanwhile, on the return path, each microservice includes the current service price in its response, enabling the gateway to adjust token allocation strategies in real time.

[0059] As a specific implementation method, the specific implementation method of this embodiment includes the following steps:

[0060] Step 1: A token generation module is deployed inside the gateway. This module uses a fixed-rate fixed window counting algorithm to continuously generate tokens to ensure that the system can stably control request traffic and prevent sudden traffic shocks from causing service unavailability.

[0061] When each request enters the gateway, it will obtain an appropriate number of tokens based on the microservice price corresponding to its request path, thereby achieving refined traffic management for microservices with different resource consumption levels.

[0062] For example, for compute-intensive microservices, their prices are higher, so the gateway needs to allocate more tokens to such requests. For lightweight services with low resource consumption, fewer tokens are needed to meet the admission conditions, thereby optimizing overall resource utilization.

[0063] When allocating tokens to requests, the business priority of the requests is taken into consideration, and the number of requests of different categories sent is dynamically adjusted through integer linear programming to ensure that high-priority requests are processed first.

[0064] Specifically:

[0065] Each microservice node resides in a pod. Each pod contains multiple containers, each performing different functions. The gateway microservice pod includes a main container, an Envoy container (token allocation module), and a token generation container. The Envoy container can be understood as the token allocation module mentioned above, and the remaining modules can also correspond to the above based on their specific functions.

[0066] The main container primarily runs the gateway's core logic, including parsing user requests, identifying target microservices, and forwarding requests. However, to implement token-based request scheduling and traffic control, the main container's request traffic is not sent directly to the target microservices but must be intercepted and processed by the Envoy container. The Envoy container uses iptables rules to redirect all external requests from the main container to itself and determines whether a request is allowed based on the number of tokens provided by the token generation container.

[0067] The token generation container is a key component in the gateway pod. It is responsible for generating tokens at a fixed rate and dynamically allocating the number of available tokens for different types of requests using an integer linear programming (ILP) optimization strategy.

[0068] Its core algorithm is based on a fixed window counting mechanism, where the system periodically issues a fixed number of tokens and uses integer linear programming to determine the number of tokens that should be sent for each request. It fully considers the business priority of requests, classifying them and assigning a priority to each category. Finally, using integer linear programming, it determines the limit on the number of requests that can be sent, thus ensuring that high-priority requests can be sent in overloaded situations.

[0069] This embodiment divides all business requests of microservice applications into three types, namely critical, guaranteed and best-effort. For example, suppose there are six types of requests (Q1 to Q6) in the system, and these six requests correspond to different business functions. Among them, Q1 is the highest priority request with the highest business value, which is critical and assigned a priority of 1; Q2 and Q3 are guaranteed requests with a priority of 2; Q4, Q5 and Q6 are the lowest priority requests, which are best-effort and assigned a priority of 3. This classification is based on the functional type and business importance of the request, which can be obtained when the microservice application is offline with prior knowledge of the system design. In addition, we have obtained the access price of all microservices at the gateway microservice by responding to the price-carrying strategy. Suppose there are microservice instances A, B, and C, and their access prices are P, B, and C, respectively. A 、P B 、P C For each request Q i , the request price V is calculated according to the path accumulation principle:

[0070]

[0071] Where Path(Q i ) indicates the request type Q i The set of microservice paths that need to be called. For example, if the call path of request type Q1 is A→B→C, then its price V1=P A +P B +P C, which is also the total number of tokens required to make this request. From this we get the number of tokens required to send each request.

[0072] Next, assuming that the number of tokens generated in each token generation cycle is T, and the number of tokens consumed to send each request from Q1 to Q6 is V1 to V6, we want to obtain the upper limit of the number of requests sent for each request from Q1 to Q6, corresponding to X1 to X6. To obtain the upper limit of the number of requests sent for each request while fully reflecting the importance of each type of request, a weighted objective function is designed in the form of:

[0073] The objective function maximizes the total number of weighted requests sent while ensuring that the total token consumption satisfies: V1×X1+V2×X2+V3×X3+V4×X4+V5×X5+V6×X6≤T#(2). In this way, when token resources are limited, high-priority login and payment requests will be prioritized, while low-priority query requests may be appropriately restricted to ensure the continuity and high responsiveness of key business functions. In addition, in order to reflect the smooth changes in business requests in continuous time periods and the stability of the system, the scheme imposes dynamic constraints on the number of requests of each type in each cycle. Specifically, the upper limit X on the number of requests sent for each request i at the current time t is i All must maintain the upper limit of the number of requests i sent at the last moment between 60% and 120%, i.e.

[0074]

[0075] This design not only prioritizes high-priority requests when token resources are limited, but also ensures the consistency of the number of requests sent over time, thereby improving overall scheduling stability and response efficiency. The entire optimization problem is a multi-stage linear programming problem that can be solved using standard methods. Therefore, within each time period, the token generation container generates an upper limit on the number of requests of each type that can be sent within that period.

[0076] Next, when the request is processed by the main container and sent to the Envoy container, the Envoy container obtains the upper limit of the number of requests that can be sent in the current cycle from the token generation container and adds a corresponding number of tokens to the request based on the target microservice price corresponding to the request path. The overall process is shown in Algorithm 1.

[0077] In practice, the token generation rate depends on the performance of the server. If the server has excellent performance and can handle a large number of requests simultaneously, then a faster token generation rate of 1,000 per second can be set. However, if the server has average performance, only 500 per second can be generated. This depends on the actual situation. Furthermore, when using an integer linear programming strategy to calculate the number of tokens to send for each request, since we know the total number of tokens required for each request's access path, we only need to add the corresponding tokens to each request when sending it.

[0078] Step 2: After the request obtains the token, the gateway will append the allocated token number to the request metadata and pass it along with the request to the downstream microservice.

[0079] Step 3: After receiving a request, each microservice decides whether to accept it based on its own dynamic pricing strategy. The dynamic pricing of a microservice is determined by its current load, resource utilization, and service pricing strategy. Specifically, if the number of tokens carried by the request is greater than or equal to the current price, the request is accepted and processed. Otherwise, the microservice rejects the request and returns an appropriate error message.

[0080] Specifically, when it comes to the Pod of a downstream microservice, the Pod is also composed of three containers working together: the main container, the Envoy container, and the price update container. The Envoy container is responsible for traffic management and token admission control. It intercepts the request when it flows in and determines whether the request can be admitted based on the latest price information provided by the price update container. The core function of the price update container is to calculate the dynamic price of the microservice. Its algorithm is adjusted based on the current microservice load, CPU usage, memory occupancy, and historical request processing. For example, if the current load of the microservice is low, the price update container will appropriately lower the service price, so that the gateway can allocate more requests to the microservice with lower token consumption; conversely, if the load is high, the price will increase to reduce the number of requests entering the microservice and prevent service degradation or unavailability due to traffic overload.

[0081] The price update process for the price update container is shown in Algorithm 2. The system first obtains the current timestamp and determines whether the difference from the last update timestamp exceeds the preset time interval threshold. If not, the price update is not performed. If it exceeds, the current CPU utilization is obtained through Prometheus and the microservice access price is readjusted based on the utilization. Prometheus is a monitoring platform that collects metrics from monitored targets by scraping the target's metrics HTTP endpoint.

[0082] When the Envoy container intercepts a request from an upstream microservice, it retrieves the microservice's current admission price from the price update container and determines whether the request should be accepted. The process is shown in Algorithm 3. First, the system obtains the current timestamp and determines whether the microservice price needs to be updated. When the current time exceeds a certain interval from the last update time, it retrieves the latest price from the price update container via HTTP and updates the local cache. Next, it extracts the token count from the metadata in the request header. If no token is included, the request is rejected. Finally, the extracted token count is compared with the microservice admission price. If the token count is not less than the service price, the request is accepted and forwarded to the main container; otherwise, the request is rejected due to insufficient tokens. After the Envoy container determines that the request is acceptable, it forwards it to the main container, which executes the specific business logic and generates the processing result. When the request is processed and is about to be sent to the downstream microservice of the current microservice, the Envoy container will also intercept all requests, deduct the token count required by the microservice, and then add the token count to the requests flowing to the downstream microservice of the current microservice. When the downstream microservice of the current microservice receives a request, it will also determine whether the request can be admitted based on the number of tokens carried in the request. If there are sufficient tokens, the request will be allowed to enter; if there are insufficient tokens, the request will be rejected. When the request continues to be sent downstream, the number of tokens corresponding to the current microservice price will be deducted and the remaining tokens will be added to the request. Each downstream microservice will execute this process until the request is fully executed in the microservice. It effectively implements request admission control based on dynamic prices, ensuring that the system can reasonably limit the entry of requests when the load is high. When the response is returned, each microservice will carry the latest price of itself and the downstream microservices in the response, so that the upstream microservice has an understanding of the prices of all downstream microservices, thereby achieving accurate request admission.

[0083] The request transmission path sent by the gateway to the downstream microservice through RPC is as follows Figure 2 shown.

[0084] On the response return path after the request is processed (the return path is actually the response, which is completely the reverse process of the request sending path), each microservice will attach its current price and the prices of all its downstream microservices it knows in the response (for example, if the microservice request sending order is A to B to C, then when C responds to B, it will carry the price of C, and when B responds to A, it will carry the prices of B and C), so that its upstream microservice can know the latest price information of the microservice and all downstream microservices and store it in the price update container at the same time. These prices will be returned to the gateway microservice step by step with the response, so that the gateway microservice can know the latest prices of all downstream microservices and use the integer linear programming strategy in the token generation container based on these feedback data to adjust the token allocation strategy in real time (the adjustment process here actually refers to using integer linear programming to calculate the maximum number of each request to be sent in each token generation cycle, and then attaching tokens when sending these requests. This is the token allocation strategy). The RPC response return path is as follows: Figure 3 shown.

[0085] When the price of a microservice increases, it indicates that its resources are becoming scarce. In this case, the gateway may reduce traffic allocation to that microservice or increase the token cost required for requests to curb excessive requests. Conversely, when the price of a microservice decreases, it indicates that its load is light. The gateway can appropriately increase traffic scheduling to that microservice to improve overall resource utilization. This closed-loop control mechanism based on real-time feedback enables the system to adapt to dynamic changes in load, ensuring that resources are properly allocated across different microservices and achieving optimal traffic scheduling overall.

[0086] Each microservice will have a default initial price, which is related to the request processing capability of the microservice, and the range of the initial price is between 1 and 10. The initial price of nodes that process lightweight tasks is lower, while nodes that carry complex calculations or high-concurrency processing have a preset higher baseline value. The heterogeneity of microservice nodes makes it necessary to set different prices for different nodes. In order to determine the initial price of each microservice, a large number of probe requests can be input when the microservice application is not yet online to establish an initial price table. The price can be determined by measuring how much the CPU utilization of the microservice will increase when each request is processed. The greater the increase in CPU utilization, the higher the initial price of the microservice should be set. In addition, a step-by-step pricing strategy is adopted in this system, and the dynamic price of microservices will change with resource usage. The dynamic pricing strategy of each microservice is implemented based on periodic resource monitoring. The system uses a fixed time window t interval Query the deployed performance monitoring tool Prometheus and collect CPU utilization U cpu , and updates the admission price based on CPU utilization, as shown in the following formula:

[0087]

[0088] Among them, price is the latest price after adjustment, and base_price is the base price of each microservice. i The elasticity coefficient is adjustable. λ1 is the price increase when CPU utilization is between 70% and 80%. A smaller value ensures gentle price adjustments under light loads. λ2 is the price increase when CPU utilization is between 80% and 90%. A larger value can quickly suppress the request rate. When resource utilization exceeds 90%, an infinite price is triggered. In actual system implementations, this results in the microservice rejecting all requests.

[0089] This process achieves intelligent traffic management through the synergy of a token mechanism, dynamic pricing, and integer linear programming. The gateway not only allocates tokens according to the pricing strategies of different microservices, but also optimizes request scheduling using integer linear programming to ensure that high-priority requests are prioritized. Tokens are transferred with requests, and the microservices dynamically determine admission policies, allowing each microservice to operate within its own load capacity. During the request return process, price feedback from the microservices allows the gateway to adjust the token allocation scheme, thus forming an adaptive traffic control system. This mechanism not only effectively ensures system stability but also maximizes resource utilization, enabling the entire microservice architecture to maintain efficient and reliable operation even in complex and changing load environments.

[0090]

[0091] When a request reaches the Pod of a downstream microservice, the Pod is also composed of three containers working together: the main container, the Envoy container, and the price update container. The Envoy container is responsible for traffic management and token admission control. It intercepts the request when it flows in and determines whether the request can be admitted based on the latest price information provided by the price update container. The core function of the price update container is to calculate the dynamic price of the microservice. Its algorithm is adjusted based on the current microservice load, CPU usage, memory occupancy, and historical request processing. For example, if the current load of the microservice is low, the price update container will appropriately lower the service price, so that the gateway can allocate more requests to the microservice with lower token consumption; conversely, if the load is high, the price will increase to reduce the number of requests entering the microservice and prevent service degradation or unavailability due to traffic overload.

[0092] The price update process for the price update container is shown in Algorithm 2. The system first obtains the current timestamp and determines whether the difference from the last update timestamp exceeds the preset time interval threshold. If not, the price update is not performed. If it exceeds, the current CPU utilization is obtained through Prometheus and the microservice access price is readjusted based on the utilization. Prometheus is a monitoring platform that collects metrics from monitored targets by scraping the target's metrics HTTP endpoint.

[0093]

[0094]

[0095] The entire process forms a complete closed loop: the gateway's token generation container is responsible for token generation and allocation, the Envoy container intercepts and releases requests, and controls traffic to downstream microservices based on the number of tokens. The price update container within the downstream microservice pod dynamically calculates prices, influences request admission rules, and in turn adjusts the gateway's token allocation policy. This architecture not only enables adaptive load control at the microservice level but also ensures optimal resource scheduling globally, enabling the system to maintain efficient and stable operation under varying load conditions.

[0096] Generally speaking, if Figure 4 As shown, this embodiment provides a price-based microservice request overload control device with request business priority perception capability in a microservice architecture scenario. The device is mainly deployed on a server or container through Kubernetes and a microservice architecture application, and mainly includes a price-based request filtering device and a priority-based request sending device.

[0097] The microservice architecture application is deployed on a server or container, along with Jaeger and Prometheus for acquiring monitoring data. The price-based request filtering device is used to implement a request-carrying token at the gateway microservice and control request admission based on server price at the entry point of each downstream microservice. The priority-based request sending device specifies the number of requests that can be sent within a cycle for each request based on the price of each request and an integer linear programming model, thus implementing request sending based on request priority.

[0098] Based on the same inventive concept, an embodiment of the present application further provides a microservice request overload control method with request service priority awareness capability, comprising:

[0099] S1. Generate a preset number of tokens every token generation cycle through the token generation container module, and save unallocated tokens.

[0100] S2. Set an initial admission price for each microservice through the microservice price setting module, and update the admission price based on CPU utilization; wherein the price is the required number of tokens.

[0101] S3. Determine the microservices required for the request through the token allocation module, and allocate the required tokens to the request based on the required microservices; when the CPU utilization is higher than the preset threshold, calculate the number of tokens required to be allocated to each request in each token generation cycle through the integer linear programming method based on the priority of the request; the priority of the request is a pre-set priority; allocate tokens to each request in each token generation cycle based on the number of tokens.

[0102] S4. Determine, through the gateway module, whether the number of tokens allocated to the request reaches the admission price, and receive the request that reaches the admission price.

[0103] The entire microservice request processing process revolves around the collaborative work of the gateway service and downstream microservices. This solution uses Kubernetes-based deployment of microservice applications, with each microservice node located in a Pod. Each Pod contains multiple containers to perform different functions. The Pod of the gateway microservice contains a main container, an Envoy container, and a token generation container, while the Pod of each downstream microservice contains a main container, an Envoy container, and a price update container. Each container plays a key role in traffic management, request scheduling, and resource allocation, enabling the entire system to efficiently and dynamically adapt to load changes and reasonably control the flow of requests.

[0104] The gateway microservice pod consists of three containers: a main container, an Envoy container, and a token generation container. The main container primarily runs the core logic of the gateway, including parsing user requests, determining the target microservice, and forwarding requests. However, to implement token-based request scheduling and traffic control, the main container's request traffic is not sent directly to the target microservice. Instead, it must be intercepted and processed by the Envoy container. The Envoy container redirects all external requests from the main container to itself through iptables rules and determines whether the request can be passed based on the number of tokens provided by the token generation container.

[0105] The token generation container is a key component of the gateway pod, responsible for generating tokens at a fixed rate and dynamically allocating the number of available tokens to different request categories using an integer linear programming (ILP) optimization strategy. Its core algorithm is based on a fixed window counting mechanism, where the system periodically issues a fixed number of tokens and uses integer linear programming to determine the number of tokens to be sent for each request. This algorithm fully considers the business priority of requests, classifying them and assigning a priority to each category. Finally, using integer linear programming, it determines the limit on the number of requests that can be sent, thereby ensuring that high-priority requests can be sent even in overloaded situations.

[0106] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0107] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A microservice request overload control device with request service priority perception capability, characterized in that: include: A token generation container module, configured to generate a preset number of tokens every other token generation cycle and store unallocated tokens; A microservice price setting module is used to set an initial admission price for each microservice and update the admission price based on CPU utilization; wherein the price is the number of tokens required; Token allocation module, used to: Determine the microservices required by the request and assign the required token to the request based on the required microservices; When the CPU utilization rate is higher than a preset threshold, the number of tokens to be allocated to each request in each token generation cycle is calculated using an integer linear programming method based on the priority of the request; the priority of the request is a preset priority; Allocate tokens to each request in each token generation cycle according to the number of tokens; The gateway module is used to determine whether the number of tokens allocated to the request reaches the admission price, and receive the request that reaches the admission price.

2. The microservice request overload control device with request service priority perception capability according to claim 1 is characterized in that: The number of tokens that need to be allocated to each request in each token generation cycle is calculated using integer linear programming, specifically including: Using a weighted objective function, the number of tokens that need to be allocated to each request in each token generation cycle is calculated; Among them, the weighted objective function is: V1×X1+V2×X2+V3×X3+V4×X4+V5×X5+V6×X6≤T; Among them, k1, k2 and k3 represent the values ​​of three priorities respectively; X1, X2, X3, X4, X5 and X6 represent the number of tokens that need to be allocated to the six requests in each token generation cycle respectively; V1, V2, V3, V4, V5 and V6 represent the number of tokens required to send the six requests respectively; T represents the total number of tokens generated in each token generation cycle.

3. The microservice request overload control device with request service priority perception capability according to claim 1 is characterized in that: The gateway module is further configured to control the number of requests sent during the current cycle to be maintained between 60% and 120% of the upper limit of the number of requests sent during the previous cycle.

4. The microservice request overload control device with request service priority perception capability according to claim 1 is characterized in that: The formula for updating the admission price based on CPU utilization is: Among them, price is the latest adjusted admission price, base_price is the initial admission price of each microservice; i is an adjustable elasticity coefficient, λ1 is the price growth rate when the CPU utilization is in the range of 70%-80%, λ2 is the price growth rate when the CPU utilization is in the range of 80\%-90\%. When the resource utilization is higher than 90%, the infinite price is triggered.

5. The microservice request overload control device with request service priority perception capability according to claim 1 is characterized in that: The tokens required for a request refer to the sum of the tokens required by the microservices required by the request; The admission price that a request needs to reach refers to the sum of the admission prices of the microservices required by the request.

6. A microservice request overload control method with request business priority perception capability, characterized in that: include: Generate a preset number of tokens every token generation cycle through a token generation container module, and save unallocated tokens; Through the microservice price setting module, an initial admission price is set for each microservice, and the admission price is updated according to CPU utilization; wherein the price is the number of tokens required; The token allocation module determines the microservices required by the request and allocates the required tokens to the request based on the required microservices. When the CPU utilization rate is higher than a preset threshold, the number of tokens required to be allocated to each request in each token generation cycle is calculated using an integer linear programming method based on the priority of the request. The priority of the request is a preset priority. Tokens are allocated to each request in each token generation cycle based on the number of tokens. Through the gateway module, it is determined whether the number of tokens allocated to the request reaches the admission price, and the request that reaches the admission price is received.

7. The microservice request overload control method with request service priority awareness according to claim 6 is characterized in that: The number of tokens that need to be allocated to each request in each token generation cycle is calculated using integer linear programming, specifically including: Using a weighted objective function, the number of tokens that need to be allocated to each request in each token generation cycle is calculated; Among them, the weighted objective function is: V1×X1+V2×X2+V3×X3+V4×X4+V5×X5+V6×X6≤T; Among them, k1, k2 and k3 represent the values ​​of three priorities respectively; X1, X2, X3, X4, X5 and X6 represent the number of tokens that need to be allocated to the six requests in each token generation cycle respectively; V1, V2, V3, V4, V5 and V6 represent the number of tokens required to send the six requests respectively; T represents the total number of tokens generated in each token generation cycle.

8. The microservice request overload control method with request service priority awareness according to claim 6 is characterized in that: The gateway module is further configured to control the number of requests sent during the current cycle to be maintained between 60% and 120% of the upper limit of the number of requests sent during the previous cycle.

9. The microservice request overload control method with request service priority awareness according to claim 6 is characterized in that: The formula for updating the admission price based on CPU utilization is: Among them, price is the latest adjusted admission price, base_price is the initial admission price of each microservice; i is an adjustable elasticity coefficient, λ1 is the price growth rate when the CPU utilization is in the range of 70%-80%, λ2 is the price growth rate when the CPU utilization is in the range of 80\%-90\%. When the resource utilization is higher than 90%, the infinite price is triggered.

10. The microservice request overload control method with request service priority awareness capability according to claim 6 is characterized in that: When the gateway module determines that the number of tokens allocated to the request does not reach the admission price, it refuses to accept the request that does not reach the admission price; The request that does not reach the admission price is allocated tokens in each token generation cycle until the number of tokens allocated to the request reaches the admission price, and the gateway module is used to receive the request that reaches the admission price.

Images