Current limiting method, system and device based on AI gateway and microservice gateway and medium

Abstract

The invention relates to the technical field of computers, in particular to a flow limiting method and system based on an AI gateway and a micro-service gateway, a medium and equipment, and the method comprises the steps: receiving service flow data corresponding to the micro-service gateway, AI flow data corresponding to the AI gateway, and a system resource utilization rate, building a quota negotiation channel between the micro-service gateway and the AI gateway, and transmitting the quota negotiation channel to the micro-service gateway; constructing a data set; calculating the flow quota of the micro-service gateway and the AI gateway through a dynamic quota algorithm based on the data set; receiving and identifying a service scene priority corresponding to the target request, and reserving an elastic burst quota for the target request based on the service scene priority; and judging the relationship between the total number of the current effective requests and the traffic quota of the corresponding gateway, and triggering a corresponding process. By establishing a dynamic quota mechanism and constructing a double-gateway cooperation mode of an AI gateway and a micro-service gateway, insufficient overall resource utilization efficiency caused by local current limiting is avoided, optimal global resource allocation is realized, and system stability is improved through cascade backoff.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of flow control, in particular to a flow limiting method and system based on an AI gateway and a micro-service gateway, a device and a medium. BACKGROUND

[0002] With the deep combination of AI technology and micro-service architecture in the system, the system needs to process both AI computing-intensive traffic and personnel, salary, scheduling and other business traffic, and the two types of traffic are transmitted through AI gateways and micro-service gateways respectively. In order to avoid traffic overload, flow limiting needs to be used to ensure system stability.

[0003] The prior art usually completely isolates AI traffic and business traffic, and independently performs flow limiting operations in AI gateways and micro-service gateways. However, after AI traffic and business traffic are isolated, the system cannot dynamically adjust the quota proportion of the two types of traffic according to the overall load, which is prone to local flow limiting and local idle resource conflicts, resulting in low overall resource utilization efficiency; and the existing token bucket and leaky bucket algorithms use fixed thresholds, which cannot adapt to sudden traffic and changes in business priority, which may mislimit valid requests or fail to prioritize high-importance businesses; in addition, AI computing-intensive requests and high-priority business requests compete for underlying resources, but there is no coordination mechanism at the gateway level, which may cause high-priority businesses to fail due to resource occupation; in addition, the existing flow limiting method cannot identify business scenario priority, and indiscriminately limits all requests under system pressure, affecting the continuity of high-priority businesses. SUMMARY

[0004] In order to solve the problem of low overall resource utilization efficiency caused by isolating AI traffic and business traffic in the existing flow limiting technology, the present application provides a flow limiting method and system based on an AI gateway and a micro-service gateway, a device and a medium.

[0005] In a first aspect, the present application provides a flow limiting method based on an AI gateway and a micro-service gateway, comprising: receiving business traffic data corresponding to a micro-service gateway, AI traffic data corresponding to an AI gateway, and system resource utilization, establishing a quota negotiation channel between the micro-service gateway and the AI gateway, and constructing a data set; based on the data set, calculating the traffic quota of the micro-service gateway and the AI gateway through a dynamic quota algorithm; receiving and identifying the business scenario priority corresponding to the target request, and reserving an elastic burst quota for the target request based on the business scenario priority; in response to the total number of current valid requests being less than or equal to the traffic quota of the corresponding gateway, sending the target request to the corresponding gateway for processing; In response to the current total number of active requests being greater than the traffic quota of the corresponding gateway, a cross-gateway quota coordination process is triggered.

[0006] By adopting the technical solution, the double-gateway quota negotiation channel is established to realize the collaborative collection of traffic and resource data. In combination with the dynamic quota algorithm, the business priority awareness, and the cross-gateway coordination process, the defects of the traditional traffic limiting, such as the complete isolation of AI and business traffic, the incapability of static threshold to adapt to fluctuations, and the lack of coordination in resource competition, are solved. The collaborative management of double-gateway traffic and underlying resources is realized, and the overall resource utilization and business continuity of the system are improved.

[0007] In a specific implementable solution, the dynamic quota algorithm comprises: Based on the data set, the total system resource capacity, the current resource occupation of the micro-service gateway, and the current resource occupation of the AI gateway are obtained, and a preset proportion of buffer resources is reserved, the resources including CPU resources and memory resources; In combination with the buffer resources and the business period characteristics, the remaining allocable resources are allocated to the micro-service gateway and the AI gateway according to the double-gateway resource occupation matching logic, and the traffic quota is generated; A real-time resource monitoring feedback loop is built, and real-time resource occupation data and system overall resource load data of the micro-service gateway and the AI gateway are collected every preset time interval. Based on the collected resource data, the traffic quota is adjusted; In response to a preset business event, the traffic quota proportion of the micro-service gateway is adjusted based on the business resource demand corresponding to the target event.

[0008] By adopting the technical solution, the dynamic quota algorithm avoids the risk of resource overload by reserving buffer resources, allocates resources in combination with business period characteristics, builds a real-time resource monitoring feedback loop to realize dynamic adjustment of the quota, further optimizes the system resource utilization, and makes the traffic quota real-time match the business demand and resource load.

[0009] In a specific implementable solution, the receiving and identifying the business scenario priority corresponding to the target request, and reserving the elastic burst quota for the target request based on the business scenario priority comprises: A request classification matrix is established, a time-sensitive evaluation model is constructed in combination with business criticality and time-sensitive sensitivity, and the target request is divided according to different business priorities; A dynamic weight adjustment algorithm is adopted to adjust the business weight of each business priority request in real time according to the business cycle, and the elastic burst quota of the target request is generated based on the business weight.

[0010] By adopting the technical scheme, the request classification matrix and the time efficiency evaluation model are constructed, and the elastic burst quota is generated by combining the dynamic weight adjustment algorithm, so that the processing success rate of core businesses under traffic fluctuation is guaranteed, and the accuracy and reliability of business processing are improved.

[0011] In a specific implementable scheme, the AI gateway and micro-service gateway based traffic limiting method further includes: The historical resource occupation data is collected and combined with the data set to construct a resource competition prediction model; Based on the resource competition prediction model, resource competition prediction is carried out for multiple time dimension periods of system operation, including a short-term time period based on real-time operation data, a medium-term time period combined with historical data of the same period, and a long-term time period associated with business calendar events; In response to the resource competition prediction model predicting that there is resource shortage in any time dimension period, and the system resource utilization rate is less than or equal to a first safety threshold, the traffic quota is adjusted, and the access amount of high resource consumption type requests is reduced, and the resource shortage includes that the current total number of valid requests is greater than the traffic quota of the corresponding gateway.

[0012] By adopting the technical scheme, the resource competition prediction model is constructed by fusing historical resource data and real-time data set, resource prediction is carried out for multiple time dimensions, the quota is actively adjusted, and the access amount of high resource consumption requests is reduced, so as to avoid cascading failures caused by resource shortage, and further improve the system operation stability and the initiative of resource scheduling.

[0013] In a specific implementable scheme, the cross-gateway quota coordination process includes: In response to the micro-service gateway triggering traffic limiting, and the resource utilization rate of the AI gateway being less than or equal to a second safety threshold, and the business weight of the micro-service gateway pending request being greater than a preset business threshold, a cross-gateway quota borrowing mechanism is started; The quota is borrowed from the AI gateway according to a hierarchical borrowing strategy, and the borrowed quota is allocated to the target pending request of the micro-service gateway; In response to the resource utilization rate of the AI gateway being greater than the second safety threshold during the borrowing, the cross-gateway quota borrowing mechanism is terminated.

[0014] By adopting the technical scheme, the traffic limiting state of the micro-service gateway, the resource redundancy of the AI gateway, and the priority of the pending request are accurately determined, the hierarchical borrowing mechanism is started and the borrowing is dynamically terminated, the elastic regulation of the resources of the two gateways is realized, and the continuity of core businesses and the stability of AI services are balanced.

[0015] In a specific implementable scheme, the AI gateway and micro-service gateway based traffic limiting method further includes: establish a cross-gateway cascading backoff mechanism, and monitor a flow limiting trigger state of the micro-service gateway and the AI gateway in real time; In response to triggering flow limiting in any of the micro-service gateway and the AI gateway, a flow limiting coordination signal is sent to the other gateway; Based on the flow limiting coordination signal, a gradual service degradation strategy is executed to adjust the traffic quota; In response to neither of the micro-service gateway and the AI gateway triggering flow limiting, a dual-gateway coordinated recovery process is started to gradually recover the traffic quota according to a preset gradient.

[0016] By adopting the above technical solution, the cross-gateway cascading backoff mechanism monitors the dual-gateway flow limiting state in real time, triggers a coordination signal and a gradual degradation when any gateway triggers flow limiting, and recovers the quota according to a gradient when there is no flow limiting, thereby preferentially guaranteeing core services by using flexible degradation, preventing secondary load fluctuations caused by sudden quota increase through a coordinated recovery process, achieving system fault isolation and smooth recovery, and improving system risk resistance and service availability.

[0017] In one specific implementation, the flow limiting method based on the AI gateway and the micro-service gateway further includes: Based on the historical resource occupation data and the data set, an LSTM neural network model outputs a token bucket generation rate coefficient; In combination with a preset token bucket base rate and the token bucket generation rate coefficient, the AI gateway calculates a token bucket dynamic rate, and the micro-service gateway configures a token bucket parameter set.

[0018] By adopting the above technical solution, the LSTM neural network model combines historical and real-time data to output a token bucket generation rate coefficient, dynamically calculates a token bucket rate, and configures dual-gateway parameters, thereby solving the defect of traditional token bucket algorithms using fixed thresholds and being unable to adapt to changes in traffic and resource trends. The token bucket parameters are intelligently adjusted according to resource load and traffic trends, so that the flow limiting strategy and the dual-gateway dynamic quota algorithm are coordinated, further improving the flexibility and accuracy of flow limiting, and guaranteeing real-time matching of dual-gateway request processing rate and resource load.

[0019] In a second aspect, the application also provides a flow limiting system based on an AI gateway and a micro-service gateway, including: A quota negotiation data module is configured to establish a quota negotiation channel between the micro-service gateway and the AI gateway, collect business traffic, AI traffic data, and system resource utilization, and build a data set; A dynamic quota scheduling module is configured to execute a dynamic quota algorithm, analyze system resource capacity and dual-gateway resource occupation, and allocate and adjust traffic quota in combination with business time period characteristics; The resource competition prediction module is configured to fuse historical and real-time data, and perform resource competition prediction on multiple time dimension periods to adjust the traffic quota. The cross-gateway quota adjustment module is configured to monitor the flow limiting state, trigger a quota lending and borrowing mechanism, and adjust the quota in stages.

[0020] By adopting the above technical solutions, the quota negotiation data module, the dynamic quota scheduling module, the resource competition prediction module, and the cross-gateway quota adjustment module are coordinated to solve the problems of no coordination between the two gateways and scattered flow limiting strategies in the traditional system. The modules correspond to the functions of data collection and negotiation between the two gateways, dynamic quota allocation, resource competition prediction, and cross-gateway quota adjustment, respectively, to realize full-link coordination from data collection, algorithm execution to strategy landing, support the complete process of dynamic coordination flow limiting between the two gateways, improve the system throughput, guarantee the success rate of high-priority requests, and reduce the abnormal recovery time.

[0021] In a third aspect, the present application also provides an electronic device including a processor, a memory, a user interface, and a network interface. The memory is configured to store instructions, the user interface and the network interface are configured to communicate with other devices, and the processor is configured to execute the instructions stored in the memory to enable the electronic device to perform the flow limiting method based on the AI gateway and the micro-service gateway.

[0022] In a fourth aspect, the present application also provides a computer-readable storage medium storing a plurality of instructions. The instructions are adapted to be loaded and executed by a processor to perform the flow limiting method based on the AI gateway and the micro-service gateway.

[0023] In summary, the present application includes at least one of the following beneficial effects: 1. The present application establishes a dynamic quota mechanism to build a dual-gateway coordination mode of the AI gateway and the micro-service gateway, avoids insufficient overall resource utilization efficiency caused by local flow limiting, and realizes optimal global resource allocation. Through the establishment of a dual-gateway real-time communication channel, the combination of a dynamic quota algorithm and cross-gateway quota lending and borrowing, and the breaking of complete isolation between AI traffic and business traffic, dynamic balance is achieved in terms of elastic adjustment in resource redundancy and priority to the core in resource shortage. At the same time, the business priority perception flow limiting reserves an elastic burst quota for high-priority requests to guarantee the high-priority requests.

[0024] 2. The present application uses a resource competition prediction model to predict multiple time dimension resources in combination with historical and real-time data, and uses a dual-gateway dynamic quota negotiation and cross-gateway cascading backoff mechanism to break through the limitations of traditional flow limiting, which only focuses on a single gateway or a static threshold, and integrates traffic, resources, and business priorities from a global system perspective to solve the problems of resource fragmentation and cascading failure risk.

[0025] 3. The application outputs dynamic token bucket coefficients by means of an LSTM neural network model, combines a preset basic rate to adjust the double-gateway token bucket parameters in real time, solves the defect that the static threshold of the traditional token bucket algorithm cannot adapt to burst traffic; at the same time, the multi-module adaptive capability reduces the dependence of manual threshold adjustment, reduces the operation and maintenance cost, prevents global failure caused by single gateway overload through cascading backoff and quota lending termination mechanism, and improves system stability and expansibility. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is a flowchart of a flow limiting method based on an AI gateway and a micro-service gateway provided by an embodiment of the application; Figure 2 is an architecture diagram of a flow limiting system based on an AI gateway and a micro-service gateway provided by an embodiment of the application; Figure 3 is a timing diagram of a flow limiting system based on an AI gateway and a micro-service gateway provided by an embodiment of the application. DETAILED DESCRIPTION

[0027] The terms used in the following embodiments of the application are only for the purpose of describing specific embodiments and are not intended to be limiting on the application. As used in the specification, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "and / or" as used herein refer to any or all possible combinations of one or more of the associated listed items.

[0028] Hereinafter, the terms "first" and "second" are only for the purpose of description, and cannot be understood as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first" and "second" can explicitly or implicitly include one or more of the features, and in the description of the embodiments of the application, the meaning of "multiple" is two or more, unless otherwise specified.

[0029] First, the AI gateway and the micro-service gateway are introduced. The AI gateway includes a model abstraction layer, a control plane, a data plane, and a security enhancement layer, which is used to uniformly manage multi-source AI model access, Token-level flow limiting, content security auditing, and multi-modal traffic scheduling, and solves the problems of heterogeneous access of large models, resource waste, and compliance risks; the micro-service gateway includes a control plane, a data plane, and an extension plane, which is used as a micro-service gateway to realize dynamic routing, load balancing, protocol conversion, and service governance, and is used to support unified traffic management and service decoupling of micro-service architecture.

[0030] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application but not all the embodiments of the present application.

[0031] Reference Figure 1 , Figure 1 A flowchart of a throttling method based on an AI gateway and a micro-service gateway is shown, which can be implemented by relying on a computer program, can be implemented by relying on a single-chip microcomputer, and can run on a throttling system based on an AI gateway and a micro-service gateway. The computer program can be integrated in a computer device or can run as an independent tool application. Specifically, the method comprises S100 to S500, and the steps are as follows: S100, receiving service traffic data corresponding to a micro-service gateway, AI traffic data corresponding to an AI gateway, and system resource utilization, establishing a quota negotiation channel between the micro-service gateway and the AI gateway, and constructing a data set; Reference Figure 2 , Figure 2 An architecture diagram of a throttling system based on an AI gateway and a micro-service gateway is shown.

[0032] In some embodiments, the architecture of the throttling system can comprise: A client as an initiator of service traffic and AI traffic, outputs a service request or an AI request to be processed; A traffic classifier deployed between the client and the gateway layer, used to divide the client request into two categories of service traffic and AI traffic according to the request characteristics, and routed to the downstream gateway respectively; A dynamic quota manager as a core scheduling unit of the double-gateway collaborative throttling, receives service traffic and AI traffic, and request characteristic data; A resource monitoring system, which collects system resource data in real time and transmits the resource data to the dynamic quota manager; A prediction model, which generates resource adjustment suggestions based on historical and real-time resource data output by the resource monitoring system, and feeds back to the dynamic quota manager in combination with the business cycle law.

[0033] A business throttling controller, which cooperates with the micro-service gateway to execute a throttling strategy on the service traffic, and is associated with a business priority evaluation module for analyzing the priority of the service request to provide a basis for throttling decision; An AI throttling controller, which cooperates with the AI gateway to execute a throttling strategy on the AI traffic, and is associated with an AI task complexity evaluation module for analyzing the resource consumption characteristics of the AI request to assist dynamic adjustment of the throttling strategy; Microservice gateway, connecting business services such as salary service, personnel service, attendance service, and scheduling service, is responsible for analyzing, forwarding business traffic, and executing business logic; AI gateway, connecting AI model services such as AI model service 1 and AI model service 2, is responsible for analyzing, forwarding AI traffic, and scheduling model inference tasks.

[0034] Specifically, the hybrid request initiated by the client is sent to the dynamic quota manager after being classified by the traffic classifier according to the request protocol and service identifier. The dynamic quota manager aggregates the real-time state of the business traffic control and AI traffic control, and the resource load data of the resource monitoring system. The prediction model fuses historical resource data and business cycle characteristics to output quota adjustment suggestions. The dynamic quota manager performs cross-gateway collaborative allocation of business traffic and AI traffic quotas based on the adjustment suggestions, realizes dynamic adaptation of system resources and business demand, and the business traffic control implements differentiated traffic limiting of the business traffic of the microservice gateway based on the business priority evaluation result. The AI traffic control implements hierarchical traffic limiting of the AI traffic of the AI gateway based on the AI task complexity evaluation result. The limited business traffic is forwarded to the microservice gateway, and the microservice gateway schedules the salary, personnel, and other business services for execution. The AI traffic is forwarded to the AI gateway, and the AI gateway schedules AI model service 1 and 2 for inference.

[0035] Reference Figure 2 and Figure 3 In some embodiments, the system deploys a dynamic quota manager as a relay core for double-gateway communication. The microservice gateway and the AI gateway establish a bidirectional long connection with the dynamic quota manager through the built-in communication submodule, forming a quota negotiation channel among the microservice gateway, the dynamic quota manager, and the AI gateway. The channel uses the high-performance remote procedure call (g-Remote Procedure Call, gRPC) protocol to realize high real-time data transmission, meeting the real-time interaction needs of quota adjustment instructions and resource state data between the double gateways.

[0036] In some embodiments, the traffic access layer of the microservice gateway deploys a microservice data acquisition module to collect real-time business request data. The business request data can include queries per second (Queries Per Second, QPS), request business type, request timeout number, and single request average processing time. The collection frequency can be set to 100 ms / time, and the collected business request data is uploaded to the dynamic quota manager in real time through the quota negotiation channel. The business type can include salary calculation, personnel query, and scheduling management.

[0037] In some embodiments, the model calling layer of the AI gateway deploys an AI data collection module to collect AI request data in real time. The AI request data can include AI model calling volume, CPU occupation time of a single AI request, memory occupation peak, and model inference time consumption. The collection frequency of the AI request data can be consistent with the collection frequency of the business request data in the micro-service gateway.

[0038] In some embodiments, the operating system layer of the system server deploys a system resource data collection module to collect CPU utilization and memory utilization in real time. The collection frequency can be set to 50 ms / time. After aggregation processing, the collected data is uploaded to the dynamic quota manager through the quota negotiation channel.

[0039] In some embodiments, the data set refers to the association and integration of the micro-service gateway business traffic data, the AI gateway AI traffic data, and the system resource utilization data in each time window according to a fixed timestamp after the dynamic quota manager receives the double-gateway traffic data and the system resource data, forming a multi-dimensional data unit set of timestamp, business traffic, AI traffic, and resource utilization.

[0040] In some embodiments, a preprocessing operation is performed on the multi-dimensional data unit set. The preprocessing operation can include outlier rejection and data standardization. The preprocessed data is encapsulated in JSON format and stored in the time series database built-in the dynamic quota manager, forming a standardized data set, and the data set retains historical data. The outlier rejection can include rejecting abnormal data such as QPS data exceeding the preset normal range and negative resource utilization data.

[0041] S200, based on the data set, calculating the traffic quota of the micro-service gateway and the AI gateway through a dynamic quota algorithm; Based on the above embodiments, as another optional embodiment, the dynamic quota algorithm includes: S201, based on the data set, obtaining the total system resource capacity, the current resource occupation of the micro-service gateway, and the current resource occupation of the AI gateway, and reserving a preset proportion of buffer resources, including CPU resources and memory resources; In some embodiments, the CPU and memory resources of the micro-service gateway and the AI gateway are collected in real time, and the total system resource capacity and the current resource occupation of the double gateway are obtained. The calculation formula of the buffer resources and the remaining resources can include: Buffer resource amount = total system resource capacity * buffer resource proportion; Remaining resources = total system resource capacity - buffer resource amount - current resource occupation of double gateway.

[0042] In some embodiments, a sliding window algorithm is used to analyze historical resource fluctuations, and the preset buffer resource ratio is adjusted based on a preset first step. For example, the preset buffer resource ratio is 20%, the first step is 5%, and in response to historical data showing that the resource fluctuation is greater than or equal to 30%, the buffer resource ratio is increased to 25%.

[0043] S202, in combination with the buffer resource and the service period feature, the remaining allocable resource is allocated to the micro service gateway and the AI gateway according to the double gateway resource occupation matching logic, and the traffic quota is generated; The service period feature in the embodiments of the present application refers to the service period divided based on historical traffic data through a clustering algorithm, which is used to represent the service load condition in different periods. The service period feature can include weekdays, weekends, early peak, and late peak. For example, 9:00-18:00 on weekdays is defined as a high service load period. A period resource demand mapping table is established to record the average resource occupation ratio of the micro service and the AI gateway in each period.

[0044] In some embodiments, the double gateway resource occupation matching logic can use a dynamic weight distribution algorithm: Target gateway quota = remaining resource * target gateway historical weight * target gateway current load coefficient / total weight.

[0045] Wherein, the current load coefficient is dynamically adjusted according to real-time CPU and memory usage; total weight = micro service gateway historical weight * micro service gateway current load coefficient + AI gateway historical weight * AI gateway current load coefficient. For example, the remaining resource is 100 cores of CPU, the micro service gateway historical weight = 0.6, and the AI gateway historical weight = 0.4. The micro service gateway current load coefficient = 1.2, and the AI gateway current load coefficient = 0.8. Then the micro service gateway quota ≈ 69 cores, and the AI gateway quota ≈ 31 cores.

[0046] S203, build a real-time resource monitoring feedback loop, collect real-time resource occupation data and system overall resource load data of the micro service gateway and the AI gateway every preset time interval, and adjust the traffic quota based on the collected resource data; In some embodiments, Prometheus and Grafana are used to build a real-time resource monitoring feedback loop. The real-time resource monitoring feedback loop can collect resource data every 10 seconds. The real-time resource monitoring feedback loop can use Influx DB to store historical data, and the historical data supports time series query.

[0047] In some embodiments, a PID controller is configured to adjust the traffic quota according to the resource utilization deviation, and the calculation formula includes: ΔQ = Kp * e(t) + Ki * ∫e(t)dt + Kd * de(t) / dt Wherein, AQ is the quota adjustment amount, representing the final adjustment value of the traffic quota of the micro-service gateway and the AI gateway. If AQ is positive, it means that the traffic quota of the corresponding gateway needs to be increased. If AQ is negative, it means that the quota needs to be reduced. Kp is the proportional coefficient. Ki is the integral coefficient. Kd is the differential coefficient. e(t) is the deviation of the current resource utilization rate from the target resource utilization rate.

[0048] In some embodiments, in response to the overall resource utilization rate of the system being greater than the resource utilization rate threshold (such as 85%), a global quota reduction is triggered, and an alert is sent through the Alert manager. Wherein, the global quota reduction includes reducing the quota of each gateway based on a preset second step (such as 20%).

[0049] S204, in response to a preset business event, adjusting the traffic quota proportion of the micro-service gateway based on the business resource demand corresponding to the target event.

[0050] The business event in the embodiments of the present application can include a business peak event, such as the start of a promotion activity, system upgrade. The triggering method of the business peak event includes API call manual triggering and log detection triggering. Wherein, the log detection triggering includes analyzing the log through the ELK Stack, and automatically triggering when a specific keyword (such as "promotion activity") is detected.

[0051] In some embodiments, based on historical business event data, a resource demand evaluation model is trained to predict the impact of business events on resources. For example, through the resource demand evaluation model, it is predicted that a business peak event such as a promotion activity will cause the resource demand of the micro-service gateway to increase by a first gradient value (such as 30%) and the resource demand of the AI gateway to increase by a second gradient value (such as 10%).

[0052] In some embodiments, the quota adjustment formula includes: New quota proportion = original proportion * (1 + event impact coefficient).

[0053] For example, the promotion activity event impact coefficient = 0.3, the original quota proportion of the micro-service gateway = 60%, then the new quota proportion of the micro-service gateway = 78%, and correspondingly, the new quota proportion of the AI gateway = 22%. After the quota adjustment, a quota adjustment instruction is generated and pushed to the dual gateway through the configuration center (such as Nacos).

[0054] S300, receiving and identifying the business scenario priority corresponding to the target request, and reserving an elastic burst quota for the target request based on the business scenario priority; On the basis of the above-mentioned embodiments, as another optional embodiment, receiving and identifying the business scenario priority corresponding to the target request, and reserving an elastic burst quota for the target request based on the business scenario priority includes: S301, a request classification matrix is established, a time-sensitive evaluation model is constructed in combination with business criticality and time-sensitive sensitivity, and target requests are divided according to different business priorities; A time-sensitive evaluation model is constructed with business criticality and time-sensitive sensitivity as double core dimensions, and a 3*3 request classification matrix is established.

[0055] Among them, the business criticality is used to represent the influence degree of the business on the system core target in the embodiments of the present application, and can be divided into three levels, and is set in combination with historical business interruption loss data and service level agreement (Service Level Agreement, SLA) priority.

[0056] Specifically, the business criticality can include core level C1, regular level C2, and non-core level C3. The core level C1 includes requests that directly affect business revenue or system stability, such as real-time risk control requests and order payment business requests, and interruption will cause great business loss. The regular level C2 includes requests that support daily operation but are not core, such as user information query requests and log query requests, and interruption affects a limited range of single function modules. The non-core level C3 includes auxiliary or low-value requests, such as system operation and maintenance statistics requests and test requests, and interruption has no direct business loss.

[0057] The time-sensitive sensitivity is used to represent the maximum allowable delay time and data timeliness requirement of the target request in the embodiments of the present application, and can be divided into three levels, and is set in combination with historical request timeout failure rate data.

[0058] Specifically, the time-sensitive sensitivity can include high timeliness T1, medium timeliness T2, and low timeliness T3. The maximum allowable delay time of high timeliness T1 is less than or equal to 100ms, and the data needs to be updated in real time, such as real-time recommendation requests and real-time inventory deduction requests, and the timeout failure rate needs to be less than or equal to 0.1%. The maximum allowable delay time of medium timeliness T2 is between 100ms and 1s, and the data update period is less than or equal to 10 minutes, such as user order history query requests and cache requests, and the timeout failure rate needs to be less than or equal to 1%. The maximum allowable delay time of low timeliness T3 is greater than 1s, and the data update period is greater than or equal to 1 hour, such as monthly business report generation requests and data archiving requests, and the timeout failure rate needs to be less than or equal to 5%.

[0059] Based on the double core dimensions of business criticality and time-sensitive sensitivity, the nine cells of the request classification matrix correspond to nine levels of business priority, and the priority from high to low is C1T1, C1T2, C2T1, C1T3, C2T2, C3T1, C2T3, C3T2, and C3T3.

[0060] For example, the business priority of the real-time risk control request is C1T1, which is higher than the business priority C2T2 of the order history query request.

[0061] In some embodiments, the requests of the AI gateway can also be divided in combination with the task complexity.

[0062] S302, using a dynamic weight adjustment algorithm, adjusting the service weight of each service priority request in real time according to the service period, and generating an elastic burst quota of the target request based on the service weight.

[0063] Based on the 9-level priority of the request classification matrix, preset the basic service weight of each priority. Among them, ensure that the service weight of the high priority request is higher than that of the low priority request, and the sum of the basic service weight is 1.0.

[0064] The service period is used to represent the traffic data of different periods in the embodiments of the present application. Based on the historical traffic data of the double gateway, the service period can be divided into 3 categories by K-means clustering algorithm.

[0065] Specifically, the service period can include peak period, flat peak period, and trough period. Among them, the peak period includes 9:00-12:00 and 14:00-18:00 on weekdays, the QPS of the micro service gateway is greater than or equal to 2000, the QPS of the AI gateway is greater than or equal to 500, the weight of the core priority service (C1T1, C1T2, C2T1) is up-regulated, and the weight of the non-core priority service (C2T3, C3T2, C3T3) is down-regulated; the flat peak period includes 12:00-14:00 and 18:00-22:00 on weekdays, the QPS of the micro service gateway is between 500 and 2000, the QPS of the AI gateway is between 100 and 500, and the weight of each level of service remains unchanged; the trough period includes, for example, 22:00-9:00 the next day and holidays, the QPS of the micro service gateway is less than 500, the QPS of the AI gateway is less than 100, the weight of the core priority is down-regulated, and the weight of the non-core priority is up-regulated.

[0066] In some embodiments, the dynamic quota manager collects double gateway QPS data every 5 minutes, determines the corresponding service period based on the QPS data, and triggers the weight adjustment of the corresponding service period.

[0067] S400, in response to the total number of current valid requests being less than or equal to the traffic quota of the corresponding gateway, sending the target request to the corresponding gateway for processing; The valid request in the embodiments of the present application refers to the request whose priority identification passes, format verification is compliant, and is not timed out. The valid request includes the business request of the micro service gateway and the AI request of the AI gateway. Among them, the business request can include salary calculation and personnel query; the AI request can include model reasoning and data classification.

[0068] Specifically, in response to each request entering the quota negotiation channel, invalid requests are removed by the pre-filtering module, and the filtering result is synchronized to the dynamic quota manager and is not included in the total number of valid requests. The total number of current valid requests is counted, and if it is less than or equal to the traffic quota of the corresponding gateway, the target request or target request set is sent to the corresponding gateway for processing. Invalid requests include query requests with missing parameters, repeated requests with timeout retransmission.

[0069] S500, in response to the total number of current valid requests being greater than the traffic quota of the corresponding gateway, a cross-gateway quota coordination process is triggered.

[0070] Based on the above embodiment, as another optional embodiment, the cross-gateway quota coordination process includes: S501, in response to the microservice gateway triggering flow limiting, and the resource utilization rate of the AI gateway being less than or equal to a second safety threshold, and the business weight of the pending request of the microservice gateway being greater than a preset business threshold, a cross-gateway quota borrowing mechanism is started; In some embodiments, in response to the total number of valid requests of the microservice gateway being greater than or equal to the available quota of the microservice gateway for 3 consecutive statistical periods (such as 10 ms per statistical period), or the CPU utilization rate of the microservice gateway being greater than or equal to a CPU threshold for 3 consecutive statistical periods, or the memory utilization rate being greater than or equal to a memory threshold, it is determined that the microservice gateway triggers flow limiting.

[0071] In some embodiments, after the microservice gateway triggers flow limiting, a flow limiting trigger signal is sent to the dynamic quota manager through a gRPC channel. The flow limiting trigger signal includes the flow limiting type, the current number of valid requests, and the priority distribution of the pending request. The flow limiting type can include request quantity limiting, CPU limiting, and memory limiting.

[0072] The resource utilization rate of the AI gateway is set based on the CPU utilization rate and the memory utilization rate of the AI gateway by a sliding average method; and the second safety threshold is set based on the CPU threshold coefficient and the memory threshold coefficient of the AI gateway by a percentile method.

[0073] In some embodiments, the preset business threshold can be set to the maximum value in the business weight of non-core priority services (C2T3, C3T2, C3T3). At this time, the pending request with a business weight greater than the preset business threshold is a core priority service request.

[0074] S502, borrow quota to the AI gateway according to the hierarchical borrowing strategy, and allocate the borrowed quota to the target pending request of the microservice gateway; In some embodiments, the hierarchical borrowing strategy includes: dividing the borrowing quota into multiple levels of borrowing proportions according to the business priority of the microservice gateway pending request. Among them, the borrowing proportion refers to the proportion of the borrowed quota in the idle quota of the AI gateway, and affects the maximum borrowing amount at a time; the sum of the borrowing proportions of each level is less than or equal to 100%, and the remaining available quota of the AI gateway after borrowing is greater than or equal to 50% of the total available quota, so as to avoid resource shortage of the AI gateway due to borrowing. For example, if the idle quota of the AI gateway = 200 requests / second, the highest level of borrowing proportion can borrow at most 100 requests / second, and the remaining 100 requests / second are used to guarantee the basic services of the AI gateway.

[0075] The microservice gateway receives a borrowing mechanism start instruction, and submits a quota borrowing request to the dynamic quota manager according to the business priority order. The quota borrowing request carries the request quantity of the pending request and the required borrowed quota.

[0076] The dynamic quota manager checks the request, and responds to the case that the total borrowing amount is less than or equal to the idle quota of the AI gateway, and the single borrowing amount is less than or equal to the single maximum borrowing amount, and the check is passed. After the target request check is passed, a quota borrowing voucher is generated, and a quota deduction instruction is sent to the AI gateway, and the AI gateway deducts the corresponding quota. Among them, the quota borrowing voucher includes the borrowed quota amount, the borrowing validity period, and the priority association.

[0077] After the microservice gateway receives the quota borrowing voucher, the borrowed quota is allocated to each target request according to the business priority from high to low. The allocation result is synchronized to the Redis effective request statistics, and the available quota of the microservice gateway is updated. At this time, the available quota = original quota + borrowed quota.

[0078] S503, in response to the resource utilization rate of the AI gateway during borrowing being greater than the second safety threshold, terminating the cross-gateway quota borrowing mechanism.

[0079] Specifically, in response to the resource utilization rate of the AI gateway during borrowing being greater than the second safety threshold, the dynamic quota manager generates a borrowing termination instruction and sends it to the microservice gateway and the AI gateway respectively.

[0080] After the microservice gateway receives the borrowing termination instruction, it stops using the borrowed quota to receive new requests. For the requests that have been started by the borrowed quota, it is allowed to continue to be completed, and the borrowed quota is released after the processing is completed.

[0081] After the AI gateway receives the borrowing termination instruction, it stops providing new quota to the microservice gateway, and calculates the borrowed but unused quota. The borrowed but unused quota is supplemented back to the available quota of the AI gateway itself, and is synchronized to the dynamic quota manager to update the quota state.

[0082] In some embodiments, the dynamic quota manager records the loan data of this time, stores it into the time series database, and is used for subsequent optimization of the hierarchical loan ratio and the safety threshold. Wherein, the loan data can include loan duration, loan quota, termination reason, and request processing success rate.

[0083] On the basis of the above-mentioned embodiments, as another optional embodiment, the traffic limiting method based on the AI gateway and the micro-service gateway further comprises: S601, collect historical resource occupation data and combine data sets to build a resource competition prediction model; Through the time series database, the resource occupation data of a preset historical period (such as the past 6 months) is extracted, including the resource utilization rate of the double gateway, various request data, and system total resource capacity data.

[0084] The resource occupation data of the preset historical period is associated with the data set, aligned according to the minimum time window, and the abnormal values are removed to form a model training set, and a resource competition prediction model is built.

[0085] S602, based on the resource competition prediction model, the resource competition prediction is carried out for multiple time dimension periods of system operation, including short-term time period based on real-time running data, medium-term time period combined with historical same period data, and long-term time period associated with business calendar events; In some embodiments, the prediction of the short-term time period is based on real-time running data, which can be used to predict the next short-term time period (such as 5 minutes). Specifically, the double gateway QPS, CPU and memory utilization rate in the last short-term time period are collected in real time, and the QPS growth rate and resource utilization rate change rate are calculated by sliding window algorithm.

[0086] The prediction of the medium-term time period is based on historical same period data, which can be used to predict the next medium-term time period (such as 1-24 hours). Specifically, the resource utilization rate mean and traffic distribution characteristics of the historical same period data are extracted, and the current period business traffic and AI traffic data are combined. Wherein, the historical same period data can include data of the same day and time period of the last week, and the same day and time period of the last month.

[0087] The prediction of the long-term time period is based on the associated business calendar events, which can be used to predict the next long-term time period (such as the next 1-30 days). Specifically, the associated business calendar events are extracted, the resource consumption data during the historical similar events are extracted, and the current system resource capacity is combined. Wherein, the associated business calendar events can include end-of-month salary calculation, quarterly report generation, and holiday promotion.

[0088] S603, in response to the resource competition prediction model predicting that resource shortage occurs in any time dimension period, and the system resource utilization rate is less than or equal to the first safety threshold, adjusting the traffic quota and reducing the access amount of high resource consumption type requests. The resource shortage includes that the total number of current valid requests is greater than the traffic quota of the corresponding gateway.

[0089] The first safety threshold is set by the percentile method based on the CPU threshold coefficient and the memory threshold coefficient of the system in the embodiments of the present application.

[0090] After the resource competition prediction module predicts that resource shortage occurs in any time dimension period, the dynamic quota manager immediately collects the current system resource utilization rate. If the system resource utilization rate is less than or equal to the first safety threshold, it is determined that the traffic quota has adjustment space, and quota adjustment and request control are performed.

[0091] In some embodiments, the traffic quota of the double gateway is adjusted according to the time dimension adaptation principle. For example, for short-term resource shortage, the quota of high resource consumption type requests is temporarily reduced. The AI model training request quota is reduced, the complex inference request quota is reduced, the lightweight AI request is preferentially guaranteed, the non-core business quota of the micro service gateway is reduced, and the released quota is distributed to the core business. After adjustment, it is pushed to the double gateway in real time through the gRPC channel; For medium-term resource shortage, based on historical data of the same period, the core business quota baseline of the micro service gateway in the high-risk period is increased, and the non-core business quota baseline of the AI gateway is reduced. A quota buffer pool is built, and the dynamic quota manager flexibly schedules according to real-time resource load, and preferentially allocates to sudden high-priority requests; For long-term resource shortage, the quota structure is adjusted, the proportion of event-related business quota of the micro service gateway is increased, and the proportion of non-event-related business quota of the AI gateway is reduced. The quota is recalculated every hour, and the real-time resource utilization rate is dynamically adjusted to ensure that the resource utilization and business demand match.

[0092] The high resource consumption type request in the embodiments of the present application includes AI requests with high CPU occupation and high memory occupation peak in the AI gateway received requests, such as large model training and multi-modal inference. The high resource consumption type request also includes core business requests in the micro service gateway received requests.

[0093] On the basis of the above-mentioned embodiments, as another optional embodiment, the traffic limiting method based on the AI gateway and the micro service gateway further includes: S701, a cross-gateway cascading backoff mechanism is established, and the traffic limiting trigger state of the micro service gateway and the AI gateway is monitored in real time; Specifically, a cross-gateway cascading backoff control center is built, and the control center integrates a state monitoring submodule, a rule management submodule, and a data synchronization submodule. The state monitoring submodule collects resource dimension data of the micro-service gateway and the AI gateway in real time through a multiplexing quota negotiation channel. The resource dimension data includes CPU and memory utilization, and resource occupation growth rate.

[0094] A dynamic decision rule library is constructed, and the rule library includes a decision threshold for flow limiting triggering. The rule library supports iterative optimization based on historical flow limiting event data, and adjusts the decision threshold through a machine learning algorithm. The historical flow limiting event data can include flow limiting triggering records and business loss statistics in a historical period.

[0095] S702, in response to any of the micro-service gateway and the AI gateway triggering flow limiting, sending a flow limiting coordination signal to the other gateway; When the micro-service gateway or the AI gateway triggers a flow limiting condition, the gateway generates a flow limiting coordination signal. The flow limiting coordination signal includes the flow limiting gateway identifier, the priority distribution of the request to be processed, and the current effective request number, resource utilization, and core request proportion at the flow limiting triggering time.

[0096] S703, based on the flow limiting coordination signal, executing a gradual service degradation strategy to adjust the traffic quota; The target gateway receives the flow limiting coordination signal, starts the business priority mapping, and executes the hierarchical degradation strategy. Specifically, the traffic quota of non-core business is reduced, the queue timeout control of non-core business requests is executed, and the queue timeout time is shortened; the traffic quota of regular business requests is maintained unchanged, but the use permission of the elastic burst quota of the regular business requests is suspended, and the regular business requests are allocated to service instances with low resource occupation to avoid intensifying resource competition; the traffic quota of core business requests is maintained unchanged, and the quota released by non-core business requests is supplemented to the quota of core business requests according to the core business request processing amount of the triggering gateway, to ensure the stable resource supply of core business requests.

[0097] In some embodiments, the dynamic quota manager synchronously adjusts the overall quota proportion in real time in linkage with the two gateways. For example, in response to the micro-service gateway triggering coordination due to core business request flow limiting, the AI gateway releases the quota to the core business requests of the micro-service gateway while degrading its own non-core AI requests, to realize the directional flow of resources between the two gateways.

[0098] S704, in response to neither the micro-service gateway nor the AI gateway triggering flow limiting, starting a double-gateway coordination recovery process to gradually restore the traffic quota according to a preset gradient.

[0099] In some embodiments, the dynamic quota manager checks the flow limiting state of the dual-gateway at a preset fixed period (e.g., 100 ms). When the gradient recovery strategy is executed in a hierarchical manner, the recovery process is pushed back in the order of the priority of non-core service requests, regular service requests, and core service requests, and each level sets an independent recovery period and step size.

[0100] On the basis of the above-mentioned embodiments, as another optional embodiment, the flow limiting method based on the AI gateway and the micro-service gateway further comprises: S801, based on the historical resource occupation data and the data set, outputting a token bucket generation rate coefficient through an LSTM neural network model; In combination with the historical resource occupation data and the data set, a training set is formed, the training of the LSTM neural network model is performed, and the token bucket generation rate coefficient is outputted through the trained LSTM neural network model.

[0101] S802, in combination with the preset token bucket basic rate and the token bucket generation rate coefficient, calculating a token bucket dynamic rate through the AI gateway and configuring a token bucket parameter set through the micro-service gateway.

[0102] In some embodiments, the preset token bucket basic rate is set based on the comprehensive consideration of the system load and the service demand. Specifically, based on the historical running data of the dual-gateway in different service periods, the average processing time of each type of request is calculated. The preset token bucket basic rate calculation formula can include: Token bucket basic rate = gateway available resources * resource utilization rate / average processing time.

[0103] For example, the available CPU of the micro-service gateway is 60 cores, the resource utilization rate is 70%, and the average processing time of a single request is 0.2 core·second, so the basic rate is 210 requests / second; the available CPU of the AI gateway is 40 cores, the resource utilization rate is 65%, and the average processing time of a single request is 0.5 core·second, so the basic rate is 52 requests / second.

[0104] After the AI gateway receives the token bucket basic rate, the token bucket dynamic rate is calculated, and the calculation formula includes: Token bucket dynamic rate = token bucket basic rate * generation rate coefficient.

[0105] For example, the basic rate of the micro-service gateway is 210 requests / second, and the generation rate coefficient is 1.2, so the dynamic rate is 252 requests / second; the basic rate of the AI gateway is 52 requests / second, and the generation rate coefficient is 0.8, so the dynamic rate is 41.6 requests / second.

[0106] At the same time, based on the token bucket dynamic rate, the bucket capacity and the token generation interval of the token bucket are derived. The calculation formula of the bucket capacity includes: Bucket capacity = token bucket dynamic rate * empirical coefficient.

[0107] wherein the experience coefficient is used to ensure that the token bucket can buffer the instantaneous burst traffic.

[0108] In some embodiments, the token generation interval is equal to 1 / token bucket dynamic rate.

[0109] The micro-service gateway receives the token bucket related parameters and converts them into executable configurations of its own token bucket, including the scheduling interval of the token generation thread, the memory allocation of the bucket capacity, the buffer queue length of the burst request, and updates them in real time through the local configuration synchronization interface.

[0110] The application also provides a throttling system based on an AI gateway and a micro-service gateway, which implements the throttling method based on the AI gateway and the micro-service gateway.

[0111] In some embodiments, the throttling system can include: A quota negotiation data module for establishing a quota negotiation channel between the micro-service gateway and the AI gateway, collecting business traffic, AI traffic data and system resource utilization, and constructing a data set; A dynamic quota scheduling module for executing a dynamic quota algorithm, analyzing system resource capacity and double-gateway resource occupation, and adjusting traffic quota in combination with business time period characteristics; A resource competition prediction module for fusing historical and real-time data, conducting resource competition prediction in multiple time dimension periods, and adjusting traffic quota; A cross-gateway quota adjustment module for monitoring throttling state, triggering quota borrowing mechanism, and adjusting quota in stages.

[0112] In some embodiments, the throttling system based on the AI gateway and the micro-service gateway can also include a cascading backoff recovery module. The cascading backoff recovery module is used to monitor the double-gateway throttling trigger state in real time, send throttling signals in coordination and execute gradual degradation and gradient recovery processes, and guarantee policy smooth convergence.

[0113] In some embodiments, the throttling system based on the AI gateway and the micro-service gateway can also include an LSTM traffic driving module. The LSTM traffic driving module outputs token bucket parameters based on an LSTM model.

[0114] On the basis of the above-mentioned embodiments, as another optional embodiment, the application can also include a computer storage medium, which can store a plurality of instructions, the instructions being suitable for being loaded and executed by a processor to perform the throttling method based on the AI gateway and the micro-service gateway of the above-mentioned embodiments. The specific execution process can be referred to the specific description of the above-mentioned embodiments, which will not be described here.

[0115] On the basis of the above-mentioned embodiments, as another optional embodiment, the embodiments of the present application can further include an electronic device. The electronic device can include at least one processor, at least one communication bus, a user interface, at least one network interface, and a memory.

[0116] The communication bus is used to realize the connection and communication between the components.

[0117] The user interface can include a display screen (Display) and a camera (Camera). Optionally, the user interface can further include a standard wired interface and a wireless interface.

[0118] The network interface can include a standard wired interface and a wireless interface (such as a WI-FI interface).

[0119] The processor can include one or more processing cores. The processor connects various parts in the server through various interfaces and lines, executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one of the following hardware forms: Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor can be integrated with one or a combination of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU is mainly used to process operating systems, user interfaces, and application programs. The GPU is used to render and draw the content to be displayed on the display screen. The modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor, but can be realized by a separate chip.

[0120] The memory can include a random access memory (RAM) and a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, codes, code sets, or instruction sets. The memory can include a program storage area and a data storage area, where the program storage area can store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; and the data storage area can store data involved in the above-mentioned various method embodiments, etc. The memory can also be at least one storage device located away from the processor. As a computer storage medium, the memory can include an operating system, a network communication module, a user interface module, and an application program of the AI gateway and micro-service gateway based flow limiting method.

[0121] In the electronic device, the user interface is mainly used to provide an interface for user input and obtain user input data; and the processor can be used to call the application program of the AI gateway and micro-service gateway based flow limiting method stored in the memory, and when executed by one or more processors, the electronic device performs the method of one or more of the above embodiments. It should be noted that for the above-mentioned various method embodiments, in order to simply describe, they are all expressed as a combination of a series of actions, but those skilled in the art should know that the application is not limited by the described action sequence, because according to the application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily required by the application.

[0122] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0123] In the several embodiments provided by the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of units is only a logical function division. There can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some other services, devices or units, and can be electrical or other forms.

[0124] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e. may be located in one place, or may be distributed to multiple network units. Part or all of the units may be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0125] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0126] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for making a computer device (which can be a personal computer, a server or a network device, etc.) execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, magnetic disk or optical disk and various program code storage media.

[0127] The above are only exemplary embodiments of the present disclosure, and cannot limit the scope of the present disclosure. That is, any equivalent changes and modifications made in accordance with the teachings of the present disclosure are still within the scope of the present disclosure. Other embodiments of the present disclosure will be readily apparent to those skilled in the art upon considering the specification and practicing the true principles of the present disclosure.

[0128] The present application is intended to cover any variations, uses or adaptive changes of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or conventional technical means in the technical field not described in the present disclosure. The specification and examples are only considered as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. An AI gateway and microservice gateway-based traffic limiting method, characterized in that, The method comprises: receiving service traffic data corresponding to a micro-service gateway, AI traffic data corresponding to an AI gateway, and system resource utilization, establishing a quota negotiation channel between the micro-service gateway and the AI gateway, and constructing a data set; based on the data set, calculating the traffic quota of the micro-service gateway and the AI gateway through a dynamic quota algorithm; receiving and identifying the business scenario priority corresponding to the target request, and reserving an elastic burst quota for the target request based on the business scenario priority; in response to the current total number of active requests being less than or equal to the traffic quota of the corresponding gateway, sending the target request to the corresponding gateway for processing; in response to the current total number of active requests being greater than the traffic quota of the corresponding gateway, triggering a cross-gateway quota coordination process.

2. The AI gateway and microservice gateway-based throttling method of claim 1, wherein, The dynamic quota algorithm comprises: based on the data set, obtaining the total system resource capacity, the current resource occupation of the micro-service gateway, and the current resource occupation of the AI gateway, reserving a preset proportion of buffer resources, the resources including CPU resources and memory resources; combining the buffer resources and business time period characteristics, allocating the remaining allocable resources to the micro-service gateway and the AI gateway according to a double-gateway resource occupation matching logic to generate the traffic quota; building a real-time resource monitoring feedback loop, collecting real-time resource occupation data of the micro-service gateway and the AI gateway and system overall resource load data every preset time interval, and adjusting the traffic quota based on the collected resource data; in response to a preset business event, adjusting the traffic quota proportion of the micro-service gateway based on the business resource demand corresponding to the target event.

3. The AI gateway and microservice gateway based throttling method of claim 1, wherein, The receiving and identifying the business scenario priority corresponding to the target request, and reserving an elastic burst quota for the target request based on the business scenario priority comprises: establishing a request classification matrix, constructing a time-sensitive evaluation model combining business criticality and time-sensitive, and dividing the target request according to different business priorities; using a dynamic weight adjustment algorithm to adjust the business weight of each business priority request according to the business cycle in real time, and generating the elastic burst quota of the target request based on the business weight.

4. The AI gateway and microservice gateway-based throttling method of claim 1, wherein, Further comprising: collecting historical resource occupation data and combining the data set to construct a resource competition prediction model; based on the resource competition prediction model, carrying out resource competition prediction for multiple time dimension periods of system operation, the multiple time dimension periods including a short-term time period based on real-time operation data, a medium-term time period combining historical same period data, and a long-term time period associated with business calendar events; in response to the resource competition prediction model predicting that there is resource shortage in any time dimension period, and the system resource utilization is less than or equal to a first safety threshold, adjusting the traffic quota and reducing the access amount of high resource consumption type requests, the resource shortage including the current total number of active requests being greater than the traffic quota of the corresponding gateway.

5. The AI gateway and microservice gateway based throttling method of claim 1, wherein, The cross-gateway quota coordination process comprises: In response to the micro-service gateway triggering flow control, and the resource utilization of the AI gateway being less than or equal to a second safety threshold, and the business weight of the target pending request of the micro-service gateway being greater than a preset business threshold, starting a cross-gateway quota borrowing and lending mechanism; Borrowing quota from the AI gateway according to a hierarchical borrowing and lending strategy, and distributing the borrowed quota to the target pending request of the micro-service gateway; In response to the resource utilization of the AI gateway being greater than the second safety threshold during the borrowing and lending, terminating the cross-gateway quota borrowing and lending mechanism.

6. The AI gateway and microservice gateway-based throttling method of claim 1, wherein, Further comprising: Establishing a cross-gateway cascading backoff mechanism, and monitoring the flow control triggering state of the micro-service gateway and the AI gateway in real time; In response to any of the micro-service gateway and the AI gateway triggering flow control, sending a flow control coordination signal to the other gateway; Based on the flow control coordination signal, executing a gradual service degradation strategy, and adjusting the traffic quota; In response to neither of the micro-service gateway and the AI gateway triggering flow control, starting a dual-gateway coordinated recovery process, and gradually recovering the traffic quota according to a preset gradient.

7. The AI gateway and microservice gateway based throttling method of claim 4, wherein, Further comprising: Based on the historical resource occupation data and the data set, outputting a token bucket generation rate coefficient through an LSTM neural network model; Combining a preset token bucket base rate and the token bucket generation rate coefficient, calculating a token bucket dynamic rate through the AI gateway, and configuring a token bucket parameter set through the micro-service gateway.

8. An AI gateway and microservice gateway-based traffic limiting system based on the AI gateway and microservice gateway-based traffic limiting method of claims 1-7, characterized in that, Comprising: A quota negotiation data module for establishing a quota negotiation channel between the micro-service gateway and the AI gateway, collecting business traffic, AI traffic data and system resource utilization, and constructing a data set; A dynamic quota scheduling module for executing a dynamic quota algorithm, analyzing system resource capacity and dual-gateway resource occupation, and distributing and adjusting traffic quota in combination with business time period characteristics; A resource competition prediction module for fusing historical and real-time data, conducting resource competition prediction in multiple time dimension periods, and adjusting traffic quota; A cross-gateway quota adjustment module for monitoring flow control state, triggering a quota borrowing and lending mechanism, and adjusting quota in stages.

9. An electronic device, comprising: The electronic device comprises a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory, so that the electronic device executes the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded and executed by the processor to execute the method of any one of claims 1-7.

Images