Interface response processing method and device of AI large model, equipment and medium

Abstract

The invention discloses an interface response processing method and device for an AI large model, equipment and a medium. The method comprises the steps that an annular buffer area is constructed, standard processing duration is divided into a plurality of time units, a request queue is established, and a time section mapping mechanism is formed; when an interface calling request is received, intelligently predicting the processing feasibility of the request within the maximum waiting duration according to the throughput completion condition of the current time section, the standard throughput index and the real-time load state of each queue in a buffer area; for the processable requests, the system automatically allocates the processable requests to the request queue of the optimal time unit, otherwise, immediately returns overtime response; and the system accurately schedules each queue request to the AI model for processing through a time triggering mechanism. Through annular buffer management, throughput analysis and dynamic scheduling, the AI interface processing efficiency is remarkably improved, the timeout rate is reduced, the resource utilization rate is improved, and a stable and reliable flow control scheme is provided for high-concurrency AI services.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an interface response processing method, device, equipment and medium for a large AI model. Background Art

[0002] With the rapid development of AI (Artificial Intelligence) technology, the deployment scale of large AI models in various application scenarios continues to expand, and the service response problem under high-concurrency requests has become increasingly prominent.

[0003] When AI computing resources are limited, when users initiate session requests to large AI models through a port, the surge in concurrent users or the excessive number of response tokens required for some complex queries often lead to call failures due to exceeding the provider's computing capacity. Specifically, after a user sends a service request and waits for a certain period of time, they may receive an interface response message indicating that the request failed due to service overload, stating "Server busy, please try again later."

[0004] In the process of implementing the present invention, the inventors discovered that the interface response processing mechanism of the existing AI large model affects the response reliability and user experience of the large model service, and has become a key bottleneck restricting the large-scale deployment of AI services. Summary of the Invention

[0005] The embodiments of the present invention provide an interface response processing method, device, equipment and medium for an AI large model, which can predict the user response of the AI ​​large model in advance, thereby significantly shortening the user's ineffective waiting time for the response results of the AI ​​large model.

[0006] According to one aspect of an embodiment of the present invention, a method for processing an interface response of an AI large model is provided, the method comprising:

[0007] Obtain the standard throughput value of the large AI model within the standard duration, split the standard duration into multiple time units, and construct a ring buffer corresponding to each time unit in the standard duration;

[0008] After dividing the system time into multiple time segments according to the standard duration, a mapping relationship between each time segment and the ring buffer is established, and a request queue is allocated for each time unit of the ring buffer;

[0009] When a call request to the AI ​​large model is detected, the AI ​​large model is predicted to be able to respond to the call request within the maximum waiting time based on the completed throughput value, standard throughput value, and the number of requests in each request queue in the ring buffer during the time period of the call request.

[0010] If yes, the call request is added to the matching request queue in the ring buffer; otherwise, a timeout interrupt feedback message is directly generated as a response result to the call request;

[0011] When the current system time hits any time unit in the ring buffer, each request in the request queue corresponding to the hit time unit is provided to the AI ​​big model for model calculation.

[0012] According to another aspect of an embodiment of the present invention, there is also provided an interface response processing device for an AI large model, the device comprising:

[0013] The throughput benchmark acquisition module is used to obtain the standard throughput value of the AI ​​large model within the standard duration, split the standard duration into multiple time units, and construct a ring buffer corresponding to each time unit in the standard duration;

[0014] The time partition mapping module is used to divide the system time into multiple time segments according to the standard time length, establish a mapping relationship between each time segment and the ring buffer, and allocate a request queue for each time unit in the ring buffer;

[0015] The request response prediction module is used to predict whether the AI ​​large model can respond to the call request within the maximum waiting time when a call request to the AI ​​large model is detected, based on the completed throughput value of the AI ​​large model in the time segment to which the call request belongs, the standard throughput value, and the number of requests in each request queue in the ring buffer;

[0016] The request scheduling decision module is used to add the call request to the matching request queue in the ring buffer if yes; otherwise, directly generate timeout interrupt feedback information as a response result to the call request;

[0017] The time-triggered execution module is used to provide each request in the request queue corresponding to the hit time unit to the AI ​​big model for model calculation when the current system time hits any time unit in the ring buffer.

[0018] According to another aspect of an embodiment of the present invention, an electronic device is provided, the electronic device including:

[0019] at least one processor; and

[0020] a memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the interface response method of an AI large model described in any embodiment of the present invention.

[0022] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement an interface response method of an AI large model described in any embodiment of the present invention when executed.

[0023] According to another aspect of an embodiment of the present invention, a computer program product is provided, comprising computer instructions, which implement the steps of the method according to any embodiment of the present invention when executed by a processor.

[0024] The technical solution of the embodiment of the present invention obtains the standard throughput value of the AI ​​large model within the standard time length and constructs a circular buffer, splits the standard time length into multiple time units, establishes a mapping relationship between time segments and buffers, and allocates an independent request queue for each time unit, thereby building a refined time management and capacity planning system, effectively solving the resource waste problem caused by traditional coarse-grained scheduling. In the request processing stage, the system makes intelligent predictions based on real-time throughput data and queue status, accurately judges the feasibility of request processing and implements a dual-path response strategy, that is, it adds processable requests to the matching queue or immediately returns a timeout response, thereby significantly improving decision-making accuracy and avoiding invalid waiting. Finally, through the time-driven mechanism, the processing of the corresponding queue request is automatically triggered when the current system time hits the buffer time unit, thereby realizing accurate scheduling of requests. This solution deeply integrates circular buffer management with time wheel scheduling technology to form a complete intelligent flow control closed loop, which optimizes resource utilization while improving the success rate of interface response, and provides highly reliable technical guarantees for AI large model services.

[0025] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1This is a flowchart of an interface response processing method for an AI large model provided in accordance with the first embodiment of the present invention;

[0028] Figure 2 This is a flowchart of another method for processing an interface response of an AI large model provided in accordance with the second embodiment of the present invention;

[0029] Figure 3 2 is a schematic diagram of an interface response processing method of an AI large model in a specific application scenario applicable to an embodiment of the present invention;

[0030] Figure 4 2 is a schematic diagram of a method and apparatus for processing an interface response of an AI large model according to a third embodiment of the present invention;

[0031] Figure 5 It is a structural diagram of an electronic device that implements an interface response processing method of an AI large model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] Example 1

[0035] Figure 1This is a flowchart of an interface response processing method for an AI big model provided in the first embodiment of the present invention. This embodiment is applicable to the situation where intelligent scheduling and flow control of AI big model interface call requests are performed in high-concurrency scenarios. The method can be executed by an interface response processing device for an AI big model. The device can be implemented in the form of hardware and / or software and can generally be configured in an electronic device, such as a big model server deployed with an AI big model, or other servers that communicate with the big model server, etc. The server is used in conjunction with the client that sends the AI ​​big model call request.

[0036] Correspondingly, such as Figure 1 As shown, the method includes:

[0037] S110. Obtain the standard throughput value of the AI ​​large model within the standard duration, split the standard duration into multiple time units, and construct a ring buffer corresponding to each time unit in the standard duration.

[0038] Among them, AI large models can be understood as pre-trained large language models (LLMs) based on deep neural network architectures (e.g., Transformers) with a large number of model parameters (e.g., more than 1 billion). This AI large model can be a generative AI system, that is, after training with massive data, it can achieve natural language understanding, generation, and reasoning to obtain new content for multi-source input data such as text, images, or audio and video. Typically, this AI large model can be an autoregressive language model that supports setting context size.

[0039] In an embodiment of the present invention, the standard throughput value of the AI ​​large model within the standard time length can be specifically understood as: the maximum value of the standard throughput that the AI ​​large model can stably process within a specific time window (for example, 1 minute).

[0040] Among them, the standard throughput can be understood as a service capacity indicator. This indicator can be the interface call throughput, which indicates the number of API (Application Programming Interface) requests that can be processed per unit time, or the indicator can also be the computing resource throughput, which indicates the total amount of tokens that can be consumed per unit time. Token is the smallest unit for AI large models to process text. It contains semantic information (such as words or characters) and serves as the basis for calculation, directly affecting the calculation time and resource consumption. Specifically, Chinese and English use characters and words as the basis for measuring tokens, respectively. One token is approximately equal to one English word or 2-3 Chinese characters.

[0041] A ring buffer can be understood as a circular queue architecture. By dividing the standard duration into multiple time units, for example, if a standard 1-minute duration is divided into 60 1-second time units, a ring buffer can be divided into 60 time units accordingly. Furthermore, in each historical minute that has passed or the next minute that will pass, the resource allocation of the AI ​​model in each second of the corresponding minute can be recorded through each time unit in the ring buffer.

[0042] S120 , after dividing the system time into multiple time segments according to the standard duration, establishing a mapping relationship between each time segment and the ring buffer, and allocating a request queue to each time unit of the ring buffer.

[0043] In an embodiment of the present invention, the mapping relationship between time segments and the ring buffer can be understood as follows: based on the standard throughput value of the AI ​​large model within the standard time length obtained earlier, the entire system time is divided according to this standard time length and cut into multiple continuous and equal-length time segments. These time segments are like grids on the timeline, which facilitate the subsequent classification and processing of requests in different time periods. Next, a mapping relationship is established between these time segments and the previously constructed ring buffer, which means that each time segment can accurately correspond to a certain part of the ring buffer, just like finding a dedicated "storage cabinet" for each time grid.

[0044] In a specific example, the system time can be divided into the following time segments: time segment 1 (April 18, 2025, 16:15:00-April 18, 2025, 16:15:59), time segment 2 (April 18, 2025, 16:16:00-April 18, 2025, 16:16:59), etc. Furthermore, only one ring buffer can be used to establish a mapping relationship with each time segment.

[0045] Finally, a request queue can be allocated to each time unit in the ring buffer to store the call requests that the AI ​​big model needs to process within that time unit. This request queue can be thought of as a "waiting room" for the call requests to the AI ​​big model received within the corresponding time unit, ensuring that the requests for each time unit are arranged in an orderly manner, facilitating the subsequent processing and scheduling of the AI ​​big model.

[0046] Specifically, when a call request A is received, the call request A can be first assigned to a request queue. When the call request A needs to be processed, the call request A is taken out of the request queue and sent to the AI ​​big model for processing. For example, the sending of the call request A is completed by calling the call interface provided by the third-party AI big model.

[0047] It's understandable that dividing system time into segments based on standard durations, mapping them to a ring buffer, and assigning independent request queues to each time unit enables refined management and orderly storage of request processing. This time-space mapping mechanism ensures the isolation and controllability of request scheduling, effectively preventing interference between requests in different time periods. It also improves the system's efficiency in tracking and managing requests, laying the foundation for subsequent intelligent prediction and dynamic scheduling, enabling the system to maintain stable processing capabilities and efficient resource utilization in high-concurrency scenarios.

[0048] S130. When a call request to the AI ​​big model is detected, the completed throughput value of the AI ​​big model in the time segment to which the call request belongs, the standard throughput value and the number of requests in each request queue in the ring buffer are predicted to determine whether the AI ​​big model can respond to the call request within the maximum waiting time.

[0049] The AI ​​large-scale model interface response prediction mechanism proposed in the embodiments of this invention enables intelligent request scheduling through multi-dimensional real-time data analysis. This mechanism makes decisions based on three key metrics: the throughput value completed by the AI ​​large-scale model during the time period of the call request, the preset standard throughput value, and the number of pending requests in each request queue in the ring buffer. These three metrics together constitute the basic data for dynamic prediction.

[0050] Specifically, whenever a call request is received, the time segment to which the request belongs can be determined first. Taking 1 minute as the standard duration, if the call request is made at 12:05:23, then the call request belongs to the time segment 12:05:00-12:05:59. At this point, the throughput value completed within the time segment can be obtained accordingly, for example, 80 calls have been processed or 40k tokens have been consumed. At the same time, it is necessary to further query the preset standard processing capacity, assuming that the large AI model has the ability to process 100 calls per minute or consume 50k tokens. In addition, the status of pending requests in the request queue corresponding to one or more unexpired future time units in the ring buffer within this time segment will be further checked.

[0051] The maximum waiting time is an important parameter of this solution. It can be understood as the maximum response time that the AI ​​large model can tolerate. The response time can be understood as the time it takes from the user submitting a call request to the time the AI ​​large model starts processing the call request (that is, starts outputting a token to the user). Generally speaking, from the time the user submits a call request, the client or server starts timing the user. If the AI ​​large model has not started to feedback the token when the maximum waiting time is reached, the call request to the AI ​​large model's calling interface will be directly disconnected to release the AI ​​large model's computing resources. Among them, the maximum waiting time is usually set by the server, such as 5 seconds or 10 seconds.

[0052] In this embodiment, the AI ​​large model's scheduled load for a future period can be obtained from the real-time data maintained in the ring buffer. This allows the estimated wait time from the time a new call request is issued until it can be processed by the AI ​​large model to be calculated. Only if the estimated wait time does not exceed the maximum wait time will the call request be accepted and added to the request queue. Otherwise, a timeout response will be immediately returned for the call request.

[0053] Understandably, this prediction mechanism demonstrates strong adaptability in practical applications. Under low-load conditions, for example, if the large AI model has consumed only 30k / 50k tokens in the current time period, and each request queue for one or more future time units matching the current time period has only two pending requests, new call requests will typically be accepted and stored in the corresponding request queue. Because these new call requests will theoretically be processed by the large AI model within the maximum waiting time, users only need to wait within this maximum waiting time.

[0054] Under high load conditions, such as when 48k / 50k tokens are in use and there are already 1,000 pending call requests in the request queues corresponding to future time units, new call requests will typically be rejected. Users no longer need to wait for a response message that is bound to fail within the maximum waiting time. They can simply try again later or resend a new call request to re-queue for the AI ​​service appointment at a new future time interval. This intelligent flow control mechanism ensures the stability of the AI ​​service while significantly optimizing the user experience and effectively avoiding unnecessary waiting times.

[0055] S140: If yes, add the call request to the matching request queue in the ring buffer; otherwise, directly generate timeout interrupt feedback information as a response result to the call request.

[0056] The request processing decision mechanism proposed in the embodiments of the present invention implements differentiated processing of AI large model call requests through intelligent judgment. When the system predicts that a call request can be processed within the maximum waiting time, it will route the request to the corresponding request queue in the ring buffer.

[0057] Specifically, while predicting that the AI ​​big model can respond to the call request within the maximum waiting time, it can also synchronously predict how long the call request can wait for the AI ​​big model to start processing. This is equivalent to determining the time unit in the ring buffer in which the call request can be processed, and then placing the call request in the request queue corresponding to the time unit.

[0058] In a specific example, assuming that a call request initiated at 12:05:23 is predicted to wait for 3 seconds to be processed by the AI ​​large model, we can first determine that the time unit to which the request time point belongs is the 24th time unit in the ring buffer, and accordingly determine that the call request can be executed at the specific system time corresponding to the 26th time unit, so we can directly assign the call request to the request queue Queue

[26] corresponding to the 26th time unit in the ring buffer.

[0059] This request queue is like a "to-be-processed task list." All accepted requests are arranged in the order they arrive, waiting for the corresponding time unit to be triggered by the system clock. For example, a call request that is estimated to consume 200 tokens is added to Queue

[30] corresponding to the 30th time unit in the ring buffer after passing the prediction of S130. Then, when the system time reaches the 30th time unit in the ring buffer, the call request in Queue

[30] is processed.

[0060] On the contrary, after passing the prediction of S130, the call request cannot be processed by the AI ​​large model within the maximum waiting time, and a timeout interrupt feedback message will be generated immediately. This fast failure mechanism is similar to the "elevator full load prompt" - when it is detected that the elevator is overloaded, it directly prompts passengers to wait for the next one, rather than forcing them to enter and causing abnormal operation. For example, during peak hours, a complex code generation request (for example, consuming 2000 tokens) may be predicted to be unable to be processed within the maximum waiting time, such as 10. At this time, a feedback message similar to "The current service is busy, please try again later (error code: 503)" can be returned immediately to avoid unnecessary waiting for users. At the same time, the call request will be discarded directly to avoid meaningless consumption of computing and storage resources on the server.

[0061] Understandably, this dual-path processing design offers significant technical advantages: precise screening of requests to ensure they are completed within the promised timeframe and maintain service quality; rapid rejection of overloaded requests to avoid system overload and crashes; and a clear feedback mechanism to enhance user experience and facilitate timely adjustment of request strategies.

[0062] S150. When the current system time hits any time unit in the ring buffer, each request in the request queue corresponding to the hit time unit is provided to the AI ​​big model for model calculation.

[0063] In an embodiment of the present invention, a ring buffer is constructed and each time unit corresponds to a request queue. The system time is divided into multiple time segments according to the standard duration, and a mapping relationship is established with the time units of the ring buffer. When the system is running, the system time continues to advance. Once the current system time matches a time unit in the ring buffer, that is, the time unit is "hit", the corresponding operation will be triggered. This operation is to transmit all requests in the request queue corresponding to the hit time unit to the AI ​​big model, and let the AI ​​big model perform model calculations on these requests.

[0064] Specifically, assuming the standard duration is 1 minute, divide this 1 minute into 60 time units, each of which is 1 second. The ring buffer contains 60 request queues corresponding to these 60 time units, labeled Queue[1]-Queue

[60] . During operation, the system continuously records the call requests for the AI ​​large model received in different time periods and places these requests into the corresponding request queues by estimating the request response time.

[0065] For example, the time period of 8:53:00-8:53:01 corresponds to time unit 1 in the ring buffer, which corresponds to request queue Queue[1]. When the system time reaches 8:53:00, Queue[1] is hit. At this point, all requests currently stored in Queue[1] can be extracted and provided to the AI ​​big model for model calculation. The AI ​​big model will then perform corresponding model calculations for these requests and provide corresponding feedback on the model processing results.

[0066] The technical solution of the embodiment of the present invention obtains the standard throughput value of the AI ​​large model within the standard time length and constructs a circular buffer, splits the standard time length into multiple time units, establishes a mapping relationship between time segments and buffers, and allocates an independent request queue for each time unit, thereby building a refined time management and capacity planning system, effectively solving the resource waste problem caused by traditional coarse-grained scheduling. In the request processing stage, the system makes intelligent predictions based on real-time throughput data and queue status, accurately judges the feasibility of request processing and implements a dual-path response strategy - adding processable requests to the matching queue or immediately returning a timeout response, thereby significantly improving decision-making accuracy and avoiding invalid waiting. Finally, through the time-driven mechanism, the processing of the corresponding queue request is automatically triggered when the current system time hits the buffer time unit, realizing accurate scheduling of requests. This solution deeply integrates circular buffer management with time wheel scheduling technology to form a complete intelligent flow control closed loop, which optimizes resource utilization while improving the success rate of interface response, and provides highly reliable technical guarantees for AI large model services.

[0067] Optionally, based on the above embodiments, predicting whether the AI ​​large model can respond to the call request within the maximum waiting time based on the completed throughput value, standard throughput value, and the number of requests in each request queue in the ring buffer during the time period of the call request may include:

[0068] Get the request time of the call request and the completed throughput value of the AI ​​large model in the time segment of the request time;

[0069] According to the request time point and the maximum waiting time, a target time unit set is determined in the ring buffer, and the request queues corresponding to each time unit in the target time unit set are obtained as candidate queues;

[0070] If, based on the completed throughput value, the standard throughput value, and the number of requests in each request queue in the ring buffer, it is determined that the number of requests in the target candidate queue does not reach the upper limit of the number of requests in the target candidate queue, then it is determined that the AI ​​large model can respond to the call request within the maximum waiting time;

[0071] Accordingly, the call request is added to the matching request queue in the ring buffer, which may include:

[0072] Add the call request to the target candidate queue in the ring buffer.

[0073] Generally speaking, when a user sends a call request to an AI large model (for example, a "Generate Product Description" call initiated by a user at 12:05:23), the call request's timestamp (12:05:23) is first accurately recorded, and then the standard time segment to which it belongs is determined (if the standard duration is 1 minute, it belongs to the time segment 12:05:00-12:05:59). Next, the AI ​​large model's throughput completed within this time segment is counted in real time, such as the number of calls executed (for example, 38) and the total number of tokens consumed (for example, 24k tokens). For example, if the standard throughput value for this time segment is capped at 50 requests or 30k tokens, the current completed values ​​indicate that 76% of the calls and 80% of the token computing resources have been utilized within this time segment. This real-time data provides key information for subsequent request scheduling decisions.

[0074] Generally speaking, if you want to determine the target time unit, you can first perform a time-space coordinate conversion. Specifically, you can parse the exact timestamp of the call request (for example, 12:05:23.450) and map it to the corresponding position of the ring buffer through a modulo operation. Taking the configuration of 1 second as the time unit granularity and 60 seconds as the standard duration as an example, the time point 12:05:23.450 will be mapped to the 23rd time unit position in the ring buffer. At the same time, the range of subsequent time units that need to be examined will be calculated based on the maximum waiting time (such as 5 seconds), thus determining a continuous time window.

[0075] Among them, when determining the target time unit set, the cyclic characteristics of the ring buffer can be considered. Taking the above example, the request time point is mapped to the 23rd time unit position. The maximum waiting time of 5 seconds corresponds to the need to examine a total of 5 time units from 23 to 27 in the ring buffer. Then, a candidate set containing these 5 time units (i.e., the target time unit set) can be generated as the time position for possible processing of the call request. This target time unit set represents all possible time periods that can process the request under the maximum waiting time.

[0076] In another specific example, still taking a circular buffer containing 60 time units as an example, if the request time point is mapped to the 58th time unit position and the maximum waiting time is 5 seconds, the target time unit set can be cyclically mapped to the 58th-60th time units plus the 1st to 2nd time units.

[0077] Finally, the request storage status of the request queues corresponding to the target time units in the target time unit set needs to be checked for a detailed capacity assessment. Specifically, the call request is checked to see if it can be stored in any of these request queues. If so, it indicates that the call request can be processed within the maximum waiting time. Otherwise, it cannot be processed within the maximum waiting time.

[0078] Capacity assessment metrics can include the number of requests currently in each queue or the number of tokens consumed by each request in each queue. This refined assessment allows us to identify the queue most suitable for handling the current request, ensuring that processing is completed within the maximum waiting time while maintaining system load balancing. This mechanism effectively balances user experience and system performance, enabling intelligent request scheduling.

[0079] Generally speaking, when a large AI model call request is received, current resource usage and future short-term load are evaluated in real time. For example, if the standard processing capacity is 50k tokens and 30k tokens have been consumed within a one-minute period, only 20k tokens will be available within that minute. Furthermore, based on the user-set maximum wait time of 5 seconds, the system calculates the total computing resource demand for pending requests in each request queue within the time window defined by the call request (e.g., 10k tokens).

[0080] Furthermore, based on the above embodiments, determining the target time unit set in the ring buffer according to the request time point and the maximum waiting time may include:

[0081] Determine a start time unit in the ring buffer according to the request time point, and determine a plurality of delay time units in the ring buffer according to the maximum waiting time and the start time unit;

[0082] The start time unit and the plurality of delay time units are combined to obtain the target time unit set.

[0083] In an embodiment of the present invention, the delay time unit can be understood as: starting from the request start time point, the subsequent processing time window is calculated according to the maximum waiting time. Specifically, when a call request is received, a time unit range is delineated in the ring buffer according to the maximum waiting time (such as 5 seconds), and each time unit within this range may be a potential processing period for the call request. For example, if the call request arrives at 12:00:23 and the maximum waiting time is 5 seconds, the delay time unit determined in the ring buffer corresponds to the time unit position of 12:00:24-12:00:27.

[0084] Specifically, based on the arrival time of the call request (e.g., 12:00:23), a modular operation is performed to locate the starting time unit (e.g., time unit 23) in the ring buffer. Then, based on the maximum waiting time set by the user (e.g., 5 seconds), consecutive delay time units (e.g., time units 24 to 27) are calculated backward in the ring buffer. These delay time units represent potential time windows in which the call request can be processed within the maximum waiting time. Finally, the starting time unit is combined with all delay time units to form a complete target time unit set (time units 23-27). By loading the real-time status of each request queue in the target time unit set (including indicators such as the request depth in the request queue and the predicted total token consumption for each request in the request queue), a processing boundary in the spatiotemporal dimension is established for subsequent intelligent scheduling decisions. This ensures that the call request can be processed within the time window defined by the maximum waiting time while maintaining the cyclic characteristics of the ring buffer (e.g., the next time unit after time unit 59 is time unit 1), thereby achieving coordinated management of service quality commitments and system resource optimization.

[0085] Furthermore, based on the above embodiments, the types of standard throughput may include: number of model calls and number of tokens consumed;

[0086] Accordingly, when the AI ​​large model cannot respond to the call request within the maximum waiting time for any type of standard throughput verification, a timeout interrupt feedback information is generated as a response to the call request.

[0087] Specifically, you can simultaneously track the number of API calls (e.g., a maximum of 100 calls per minute) and the total token consumption (e.g., a maximum of 50k tokens per minute) within the current time period. When any of these indicators reaches a threshold, current limiting protection is immediately triggered. For example, during the period of 12:05:00-12:05:59, if it is detected that 98 calls have been processed (close to the 100-time limit) or 48k tokens have been consumed (close to the 50k limit), for newly arrived call requests: if they are normal requests (requiring 0.5k tokens) and the number of calls is within the limit, they will still be accepted. However, large token consumption requests (requiring 5k tokens) will be immediately rejected due to insufficient tokens even if the number of calls is sufficient, and the response "Current service is busy, please try again later (error code: 503)" will be returned.

[0088] This design provides dual safeguards by setting a call count warning line and a token consumption threshold. This prevents system overload caused by high-frequency calls and avoids computing resources being exhausted by large requests, achieving the optimal balance between service stability and resource utilization efficiency.

[0089] Example 2

[0090] Figure 2 This is a flowchart of another AI large model interface response processing provided in Example 2 of the present invention. This embodiment is optimized based on the above embodiments. In this embodiment, the operation of "when a call request to the AI ​​large model is detected, based on the completed throughput value of the AI ​​large model in the time period to which the call request belongs, the standard throughput value, and the number of requests in each request queue in the ring buffer, predicting whether the AI ​​large model can respond to the call request within the maximum waiting time" is specifically refined.

[0091] Correspondingly, such as Figure 2 As shown, the method may specifically include:

[0092] S210. Obtain the standard throughput value of the AI ​​large model within the standard duration, split the standard duration into multiple time units, and construct a ring buffer corresponding to each time unit in the standard duration.

[0093] S220 , after dividing the system time into multiple time segments according to the standard duration, a mapping relationship between each time segment and the ring buffer is established, and a request queue is allocated to each time unit of the ring buffer.

[0094] S230. When a call request for the AI ​​big model is detected, the request time point of the call request is obtained, and the completed throughput value of the AI ​​big model in the time segment to which the request time point belongs is obtained.

[0095] As mentioned above, whenever a call request to the AI ​​big model is detected, the specific time segment to which the call request belongs can be obtained based on the request time point of the call request. Then, based on the real-time operating status information of the AI ​​big model, the number of completed calls or the number of tokens consumed in the time segment can be obtained.

[0096] S240 : Determine a target time unit set in the ring buffer according to the request time point and the maximum waiting time, and obtain request queues corresponding to each time unit in the target time unit set as candidate queues.

[0097] In this embodiment, after determining the target time unit set that matches the request time point, the request queues corresponding to each target time unit in the target time unit set can be obtained as candidate queues.

[0098] S250. If, based on the completed throughput value, the standard throughput value, and the number of requests in each request queue in the ring buffer, it is determined that the number of requests in the target candidate queue does not reach the upper limit of the number of requests in the target candidate queue, it is determined that the AI ​​large model can respond to the call request within the maximum waiting time.

[0099] In an optional implementation of this embodiment, if it is determined based on the completed throughput value, the standard throughput value, and the number of requests in each request queue in the ring buffer that the number of requests in the target candidate queue does not reach the upper limit of the number of requests in the target candidate queue, then determining whether the AI ​​large model can respond to the call request within the maximum waiting time may specifically include:

[0100] S2501. Traverse the request queues of each time unit in the target time unit set in a time extension order, and obtain the first request queue that is not marked as having reached the queue upper limit as the current request queue.

[0101] In this embodiment, two states can be set for the request queue, one is that the number of requests in the request queue has not reached the queue upper limit, and the other is that the number of requests in the request queue has reached the queue upper limit. Specifically, the request queue that has reached the queue upper limit of the request queue will be marked as having reached the queue upper limit.

[0102] It is understandable that when we arrange the call request into various alternative request queues, we often hope to respond to the call request as early as possible. At this time, we can obtain a request queue that may not be filled in turn in the order of time extension to check whether the call request can be filled into the request queue.

[0103] Furthermore, the request queues verified to be full can be marked as request queues that have reached the queue upper limit in order of time extension. These marked request queues will not be tested again.

[0104] S2502: Detect whether the time segment to which the current time unit corresponding to the current request queue belongs is the same as the time segment to which the calling request belongs.

[0105] It is understandable that due to the cyclic nature of the ring buffer, the target time units in the target time unit set may belong to the same time segment or two adjacent time segments. When the system time enters a new time segment, the AI ​​large model resources with a new standard throughput value can be generated and re-available for user use.

[0106] Furthermore, the calculation methods used for the two different situations mentioned above are different and need to be discussed separately. In S2503-S2509 of this optional implementation method, taking the case where the time segment of the current time unit is the same as the time segment of the call request as an example, the method for determining whether the AI ​​large model can respond to the call request within the maximum waiting time is introduced.

[0107] S2503: If they are the same, calculate a first remaining throughput value according to the standard throughput value and the completed throughput value.

[0108] Specifically, the standard throughput value is the AI ​​big model computing resources allocated within each standard duration, for example, 1000 AI big model calls per minute. The completed throughput value is the AI ​​big model resource consumption completed during the time period of each standard duration.

[0109] In a specific example, a user sends a call request at 19:15:08 on April 18, 2025. The call request belongs to the time period (19:15:00-19:15:59). During the time period (19:15:00-19:15:07), the completed throughput value of the AI ​​large model is known, for example, 300 model calls.

[0110] Furthermore, the difference between the standard throughput value and the completed throughput value can be calculated as the first remaining throughput value, that is, the AI ​​large model resources that can be used in the remaining future time period within this time segment, that is, the remaining time (19:15:08-19:15:59) in the previous example, for example, the remaining available model call times or the remaining available token consumption number.

[0111] S2504 . Obtain each first processed time unit that is located before the current time unit in the target time unit set in a time extension order.

[0112] It can be understood that although the calculated first remaining throughput value is the remaining resources of the AI ​​big model, the call requests that have been arranged in the request queue are requests that are expected to consume AI big model resources for processing. Although the requests in these request queues have not been processed by the AI ​​big model, they are also expected to be processed (because the call requests that cannot be processed are directly discarded). The AI ​​big model resources required to be consumed by these call requests need to be deducted as well.

[0113] Since these call requests must be stored in the request queue of future time units that have not passed before the current time unit, the current time unit can be used as the starting point to traverse forward in the target time unit set all the time units that have completed the call request allocation, that is, the first processed time unit.

[0114] S2505: Obtain the total number of stored requests in the request queue corresponding to each first processed time unit, and calculate a second remaining throughput value based on the first remaining throughput value and the total number of stored requests.

[0115] Specifically, if the standard throughput type is the number of tokens consumed, the total number of tokens required for each of the stored requests can be estimated based on the total number of stored requests in each request queue. Then, the first remaining throughput value can be subtracted from the total number of tokens required to be consumed that has just been calculated to obtain the second remaining throughput value.

[0116] The second remaining throughput value can be understood as the number of tokens that can be consumed in the remaining future time units within the time segment to which the request time point belongs, starting from the current time unit.

[0117] In another example, if the type of the standard throughput is the number of model calls, the first remaining throughput value minus the total number of stored requests can be directly calculated as the second remaining throughput value.

[0118] For example, if the call request belongs to the time range (19:15:00-19:15:59), and the current time unit is the 12th time unit corresponding to (19:15:12-19:15:13), it can be calculated that in the remaining time range (19:15:12-19:15:59), the AI ​​large model can still use computing resources of the second remaining throughput value.

[0119] S2506: Calculate the upper limit of the number of requests that match the current request queue according to the second remaining throughput value and the position of the current time unit in the corresponding time segment.

[0120] Specifically, the current request queue corresponds to the current time unit and is used to store various call requests that need to use AI large model resources for model calculations under the current time unit.

[0121] In an optional implementation of this embodiment, calculating the upper limit of the number of requests matching the current request queue based on the second remaining throughput value and the position of the current time unit in the corresponding time segment may include:

[0122] According to the position of the current time unit in the time segment to which it belongs, obtaining at least one future time unit located after the current time unit in the time segment to which the current time unit belongs, and calculating the future duration composed of each future time unit;

[0123] The upper limit of the number of requests that match the current request queue is calculated based on the proportional relationship between the current time unit and the future duration, the second remaining throughput, and the number of historical call requests under different historical time units in each historical time segment.

[0124] The future time unit may be understood as all time units starting from the next time unit after the current time unit and ending at the last time unit in the time segment to which the current time unit belongs.

[0125] For example, assuming that the standard duration of 1 minute is divided into 60 time units, and the current time unit is time unit 12 corresponding to (19:15:12-19:15:13), then the future time units corresponding to the current time unit are time unit 13 through time unit 60. Accordingly, the future duration corresponding to each of these future time units is: total number of future time units * unit duration of a time unit, for example, 48 * 1 second = 48 seconds.

[0126] The number of historical call requests can be understood as the total number of call requests sent from each client. These historical call requests may be responded to by the AI ​​large model, or they may be directly abandoned due to long waiting time.

[0127] The number of historical call requests in different historical time units may be the number of historical call requests in each of the multiple historical time units that are closest in time to the time unit to which the current time unit belongs, for example, the number of historical call requests per second in the last five minutes.

[0128] Alternatively, the number of historical call requests under different historical time units may also be the number of historical call requests under each historical time unit in multiple historical time segments that have a certain time correlation with the time segment to which the current time unit belongs. For example, the time segment to which the current time unit belongs is the minute from 15:00:00 on September 19, 2025 to 15:00:59 on September 19, 2025. Furthermore, the minute from 15:00:00 on September 18, 2025 to 15:00:59 on September 18, 2025, and the minute from 15:00:00 on September 17, 2025 to 15:00:59 on September 17, 2025 can both be regarded as historical time segments associated with them. Alternatively, the time segment to which the current time unit belongs is the minute from 15:00:00 to 15:00:59 on a certain Saturday. Furthermore, the minute from 15:00:00 to 15:00:59 on two neighboring Saturdays can both be regarded as historical time segments associated with them.

[0129] Based on the number of historical call requests obtained for each historical time unit, we can simply estimate the number of call requests that may be generated within the time segment consisting of the unit duration of the current time unit and the future duration (for example, 1 second + 48 seconds = 49 seconds). Based on the estimated number of call requests and the proportional relationship between the current time unit and the future duration, we can further estimate how many resources in the second remaining throughput can be allocated to the current time unit for use. For example, the number of model calls that can be executed in the current time unit, or the number of tokens that can be consumed in the current time unit, etc.

[0130] Based on the above calculation results, the maximum number of call requests that can be processed in the current time unit can be further estimated. Then, the maximum number of call requests can be used as the upper limit of the number of requests matching the current request queue.

[0131] S2507. If the number of requests in the current request queue is less than the upper limit of the number of requests in the current request queue, it is determined that the AI ​​large model can respond to the call request within the maximum waiting time.

[0132] S2508. If the number of requests in the current request queue is greater than or equal to the upper limit of the number of requests, the current request queue is marked as having reached the queue upper limit, and a check is made to see whether the current request queue is the last request queue in the target time unit set. If so, execute S2509; otherwise, return to execute S2501.

[0133] S2509: Determine that the AI ​​large model cannot respond to the call request within the maximum waiting time.

[0134] If the last request queue in the target time unit set is traversed and still cannot accommodate the call request, it means that the call request cannot be responded to within the maximum waiting time.

[0135] It is understandable that this composite judgment mechanism achieves multiple optimizations in system stability, resource utilization, and user experience through multi-dimensional collaborative verification. First, by dynamically tracking the throughput difference, the resource consumption trend is accurately controlled to avoid chain reactions caused by local overloads; secondly, relying on the refined management of the ring buffer, the spatiotemporal optimization of request scheduling is achieved, which not only prevents the accumulation of requests in specific periods of time, but also ensures the balanced distribution of computing resources. Compared with the traditional single-threshold judgment, this mechanism effectively solves the one-sidedness of resource evaluation. It can not only identify the resource consumption of large requests, but also prevent the concurrent pressure of high-frequency calls, comprehensively guaranteeing service reliability and response quality, and ultimately provide users with a better service experience while ensuring stable system throughput.

[0136] S260: If yes, add the calling request to the matching request queue in the ring buffer; otherwise, directly generate timeout interrupt feedback information as a response result to the calling request.

[0137] S270. When the current system time hits any time unit in the ring buffer, each request in the request queue corresponding to the hit time unit is provided to the AI ​​big model for model calculation.

[0138] The technical solution of the embodiments of the present invention obtains the standard throughput value of a large AI model within a standard duration and constructs a ring buffer. This standard duration is then divided into multiple time units, a mapping relationship between time segments and buffers is established, and an independent request queue is assigned to each time unit. This creates a refined time management and capacity planning system, effectively addressing the resource waste problem caused by traditional coarse-grained scheduling. During the request processing phase, the system dynamically determines candidate processing queues by tracking request arrival time and current throughput status in real time, combining a preset maximum wait time. It then intelligently determines the feasibility of request processing based on real-time resource headroom and queue load. This mechanism achieves the dual advantages of millisecond-level precise scheduling and intelligent overload protection, ensuring system stability while maximizing throughput, avoiding resource waste and providing users with timely responses or rapid feedback, significantly improving overall service quality. Finally, a time-driven mechanism automatically triggers the processing of requests in the corresponding queue when the current system time hits a buffer time unit, achieving precise request scheduling. This solution deeply integrates ring buffer management with time-wheel scheduling technology to form a complete intelligent flow control closed loop, improving interface response success rate while optimizing resource utilization, providing highly reliable technical support for large AI model services.

[0139] Based on the above embodiments, after detecting whether the time segment to which the current time unit corresponding to the current request queue belongs is the same as the time segment to which the call request belongs, the following steps may also be included:

[0140] If the time segment of the current time unit corresponding to the current request queue is different from the time segment of the calling request, then obtain the second processed time units that are located before the current time unit and have the same time segment as the current time unit in the order of time extension;

[0141] Calculating a target remaining throughput value based on the standard throughput value and a total number of requests stored in the request queue corresponding to each second processed time unit;

[0142] In this optional embodiment, if the time segment to which the current time unit belongs is different from the time segment to which the call request belongs, it means that the current time unit is located in the time segment after the time segment to which the call request belongs. For example, if the request time point of the call request is 15:19:56 on April 19, 2025, which is the 56th time unit in the ring buffer, and the current time unit is the 3rd time unit in the ring buffer, it means that the current time unit corresponds to the time interval of 15:20:02-15:20:03 on April 19, 2025. Furthermore, the time segment to which the call request belongs and the time segment to which the current time unit belongs are located within two different 1-minute intervals.

[0143] Accordingly, since a new standard throughput value will be allocated within the time segment to which the current time unit belongs, for example, a new 1,000 model calls, and each time unit within this new time segment belongs to a future time unit, the completed throughput value within this new time segment is 0.

[0144] Furthermore, as long as the total amount of stored requests in the request queue corresponding to each second processed time unit before the current time unit in this new time segment is first obtained, and the total amount of AI model resources allocated to this new time segment is calculated based on the total amount of stored requests, the difference between the standard throughput value and the total amount of allocated AI model resources can be calculated as the target remaining throughput value.

[0145] Specifically, the total amount of allocated AI model resources can be the number of allocated call requests, or the estimated number of consumed tokens for all allocated call requests.

[0146] Calculate the upper limit of the number of requests that match the current request queue based on the target remaining throughput value and the position of the current time unit in the corresponding time segment;

[0147] If the number of requests in the current request queue is less than the upper limit of the current request queue, it is determined that the AI ​​large model can respond to the call request within the maximum waiting time;

[0148] If the number of requests in the current request queue is greater than or equal to the upper limit of the request number, the current request queue is marked as having reached the queue upper limit, and the operation of traversing the request queues of each time unit in the target time unit set in the order of time extension is returned.

[0149] Figure 3 The following diagram shows a typical application scenario applicable to embodiments of the present invention. In this specific application scenario, to achieve efficient response processing for AI large model interface requests, each embodiment of the present invention designs a complete set of interface response timeout processing solutions. Through intelligent scheduling and resource allocation of call requests, it ensures that the AI ​​large model can maintain stable service quality under high load conditions, providing key technical support for the large-scale deployment of AI large model services.

[0150] Correspondingly, such as Figure 3 As shown, during the initialization phase, benchmark testing is performed to obtain the standard throughput value of the large AI model. The standard duration is then divided into multiple time units and a corresponding ring buffer is constructed. Each time unit is assigned an independent request queue, and a precise mapping relationship between system time and the buffer is established. Next, interface call requests are continuously monitored. When a call request is detected, the precise timestamp is recorded, the time segment to which the call belongs is determined, and the current completed throughput value is obtained. Based on the request time point and the maximum waiting time, the target time unit set and its corresponding candidate queue are determined in the ring buffer.

[0151] Next, we enter the intelligent decision-making phase, traversing the alternative queues in chronological order to check whether each alternative queue has reached the dynamically calculated upper limit of the number of requests. For an incomplete queue, we first verify whether its time segment is the same as the calling request. If so, we calculate the first remaining throughput (standard value minus completed amount), and then count the total number of all pending requests before this time unit to obtain the second remaining throughput. If it is a cross-segment queue, the target remaining amount calculation method is used. Based on the remaining throughput and the time unit position, the real-time capacity upper limit of the current queue is calculated. If the current load of the queue is lower than the upper limit, the request is accepted. Otherwise, the queue is marked as full and the next queue is checked.

[0152] In the final execution phase, when the system time reaches the time unit corresponding to any queue, the processing flow for all requests in that queue is automatically triggered, batching the requests to the AI ​​model for computation and returning a response. If all alternative queues are unavailable, a timeout interrupt feedback message is immediately generated and returned to the caller. This entire process achieves precise control of request scheduling and efficient utilization of system resources through the coordinated mechanism of time-driven and dynamic load assessment.

[0153] Furthermore, through the clever coordination of the above steps, efficient response processing of AI large model interface requests can be achieved, achieving the following effective results:

[0154] (1) Through the time unit division of the ring buffer and real-time throughput monitoring, intelligent allocation of requests is achieved to ensure that the system can still maintain load balancing in each time unit in high concurrency scenarios, avoiding local overload or idle resources.

[0155] (2) Dynamically predict the feasibility of request processing based on the maximum waiting time, and immediately return timeout feedback for requests that cannot be responded to in time, reducing invalid waiting and improving user experience and system response efficiency.

[0156] (3) Dynamically adjust the capacity limit of each time unit based on historical data and real-time queue status to maximize the computing power of the AI ​​large model and avoid wasting tokens or calls.

[0157] (4) Through the time-driven mechanism and dual residual throughput calculation (first residual and second residual), accurate scheduling of cross-segment requests is achieved to ensure system stability and prevent crashes caused by sudden traffic.

[0158] (5) A complete control closed loop is formed from initialization, request allocation to final execution, which automatically adapts to load fluctuations and supports the collaborative management of dual indicators (number of model calls / token consumption), providing a scalable traffic control solution for AI services.

[0159] Example 3

[0160] Figure 4 This is a schematic diagram of the structure of an interface response processing device for an AI large model provided in the third embodiment of the present invention. Figure 4 As shown, the device includes:

[0161] Throughput benchmark acquisition module 410 is used to obtain the standard throughput value of the AI ​​large model within the standard duration, split the standard duration into multiple time units, and construct a ring buffer corresponding to each time unit in the standard duration;

[0162] The time partition mapping module 420 is used to divide the system time into multiple time segments according to the standard time length, establish a mapping relationship between each time segment and the ring buffer, and allocate a request queue for each time unit in the ring buffer;

[0163] The request response prediction module 430 is configured to, upon detecting a call request to the AI ​​large model, predict whether the AI ​​large model can respond to the call request within the maximum waiting time based on the completed throughput value of the AI ​​large model, the standard throughput value, and the number of requests in each request queue in the ring buffer during the time period of the call request;

[0164] The request scheduling decision module 440 is configured to, if yes, add the call request to the matching request queue in the ring buffer; otherwise, directly generate timeout interrupt feedback information as a response result to the call request;

[0165] The time-triggered execution module 450 is used to provide each request in the request queue corresponding to the hit time unit to the AI ​​big model for model calculation when the current system time hits any time unit in the ring buffer.

[0166] The technical solution of the embodiment of the present invention obtains the standard throughput value of the AI ​​large model within the standard time length and constructs a circular buffer, splits the standard time length into multiple time units, establishes a mapping relationship between time segments and buffers, and allocates an independent request queue for each time unit, thereby building a refined time management and capacity planning system, effectively solving the resource waste problem caused by traditional coarse-grained scheduling. In the request processing stage, the system makes intelligent predictions based on real-time throughput data and queue status, accurately judges the feasibility of request processing and implements a dual-path response strategy - adding processable requests to the matching queue or immediately returning a timeout response, thereby significantly improving decision-making accuracy and avoiding invalid waiting. Finally, through the time-driven mechanism, the processing of the corresponding queue request is automatically triggered when the current system time hits the buffer time unit, realizing accurate scheduling of requests. This solution deeply integrates circular buffer management with time wheel scheduling technology to form a complete intelligent flow control closed loop, which optimizes resource utilization while improving the success rate of interface response, and provides highly reliable technical guarantees for AI large model services.

[0167] Based on the above embodiments, the request response prediction module 430 may specifically include:

[0168] A request time information acquisition unit is used to obtain the request time point of the call request and obtain the completed throughput value of the AI ​​large model in the time segment to which the request time point belongs;

[0169] a candidate queue determination unit, configured to determine a target time unit set in the ring buffer according to the request time point and the maximum waiting time, and obtain a request queue corresponding to each time unit in the target time unit set as a candidate queue;

[0170] A request processing feasibility determination unit, configured to determine whether the AI ​​large model can respond to the call request within the maximum waiting time if it is determined, based on the completed throughput value, the standard throughput value, and the number of requests in each request queue in the ring buffer, that the number of requests in the target candidate queue does not reach the upper limit of the number of requests in the target candidate queue;

[0171] Accordingly, the call request is added to the matching request queue in the ring buffer, which can be used specifically for:

[0172] Add the call request to the target candidate queue in the ring buffer.

[0173] Based on the above embodiments, the candidate queue determination unit may be further configured to:

[0174] Determine a start time unit in the ring buffer according to the request time point, and determine a plurality of delay time units in the ring buffer according to the maximum waiting time and the start time unit;

[0175] The start time unit and the plurality of delay time units are combined to obtain the target time unit set.

[0176] Based on the above embodiments, the request processing feasibility determination unit may further include:

[0177] The candidate queue traversal subunit is used to traverse the request queues of each time unit in the target time unit set in the order of time extension, and obtain the first request queue that is not marked as having reached the queue limit as the current request queue;

[0178] Time segment consistency detection subunit: used to detect whether the time segment of the current time unit corresponding to the current request queue is the same as the time segment of the calling request;

[0179] A first remaining throughput calculation subunit is configured to calculate a first remaining throughput value according to the standard throughput value and the completed throughput value if the values ​​are the same;

[0180] Historical request statistics subunit: used to obtain the first processed time units before the current time unit in the target time unit set in time extension order;

[0181] The second remaining throughput calculation subunit is configured to obtain the total amount of stored requests in the request queue corresponding to each first processed time unit, and calculate a second remaining throughput value based on the first remaining throughput value and the total amount of stored requests;

[0182] Dynamic capacity upper limit calculation subunit: used to calculate the upper limit of the number of requests matching the current request queue according to the second remaining throughput value and the position of the current time unit in the corresponding time segment;

[0183] Request admission determination subunit: used to determine whether the AI ​​large model can respond to the call request within the maximum waiting time if the number of requests in the current request queue is less than the upper limit of the request number of the current request queue;

[0184] The queue status update subunit is used to mark the current request queue as having reached the queue limit if the number of requests in the current request queue is greater than or equal to the request limit, and return to execute the operation of traversing the request queues of each time unit in the target time unit set in the order of time extension;

[0185] Request rejection processing subunit: used to determine that the AI ​​large model cannot respond to the call request within the maximum waiting time if it is determined that there is no qualified target alternative queue in the target time unit set.

[0186] Based on the above embodiments, the request processing feasibility determination unit further includes:

[0187] Cross-segment historical request acquisition sub-unit: used to acquire, in the order of time extension, all second processed time units that are located before the current time unit and have the same time segment as the current time unit, when the time segment of the current time unit corresponding to the current request queue is different from the time segment of the calling request;

[0188] a target remaining throughput calculation subunit, configured to calculate a target remaining throughput value based on a standard throughput value and a total number of requests stored in a request queue corresponding to each second processed time unit;

[0189] Cross-segment capacity upper limit calculation subunit: used to calculate the upper limit of the number of requests that match the current request queue based on the target remaining throughput value and the position of the current time unit in the time segment to which it belongs;

[0190] Cross-segment request admission determination subunit: used to determine whether the AI ​​large model can respond to the call request within the maximum waiting time if the number of requests in the current request queue is less than the upper limit of the current request queue;

[0191] Cross-segment queue status update subunit: used to mark the current request queue as having reached the queue limit if the number of requests in the current request queue is greater than or equal to the request limit, and return to execute the operation of traversing the request queues of each time unit in the target time unit set in time extension order.

[0192] Based on the above embodiments, the dynamic capacity upper limit calculation subunit can be further used to:

[0193] According to the position of the current time unit in the time segment to which it belongs, obtaining at least one future time unit located after the current time unit in the time segment to which the current time unit belongs, and calculating the future duration composed of each future time unit;

[0194] The upper limit of the number of requests that match the current request queue is calculated based on the proportional relationship between the current time unit and the future duration, the second remaining throughput, and the number of historical call requests under different historical time units in each historical time segment.

[0195] Based on the above embodiments, the types of standard throughput may include: number of model calls and number of tokens consumed;

[0196] Accordingly, the request scheduling decision module 440 may be further configured to:

[0197] When the AI ​​large model cannot respond to the call request within the maximum waiting time for any type of standard throughput verification, a timeout interrupt feedback information is generated as a response to the call request.

[0198] The interface response processing device of the AI ​​large model provided in an embodiment of the present invention can execute an interface response processing method of the AI ​​large model provided in any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0199] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0200] Example 4

[0201] Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0202] like Figure 5As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0203] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0204] The processor 11 may be a variety of general and / or specialized processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, for example, executing an interface response processing method for an AI large model as described in any one of the embodiments of the present invention, that is:

[0205] Obtain the standard throughput value of the large AI model within the standard duration, split the standard duration into multiple time units, and construct a ring buffer corresponding to each time unit in the standard duration;

[0206] After dividing the system time into multiple time segments according to the standard duration, a mapping relationship between each time segment and the ring buffer is established, and a request queue is allocated for each time unit of the ring buffer;

[0207] When a call request to the AI ​​large model is detected, the AI ​​large model is predicted to be able to respond to the call request within the maximum waiting time based on the completed throughput value, standard throughput value, and the number of requests in each request queue in the ring buffer during the time period of the call request.

[0208] If yes, the call request is added to the matching request queue in the ring buffer; otherwise, a timeout interrupt feedback message is directly generated as a response result to the call request;

[0209] When the current system time hits any time unit in the ring buffer, each request in the request queue corresponding to the hit time unit is provided to the AI ​​big model for model calculation.

[0210] In some embodiments, an interface response processing method for an AI large model may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the interface response processing method for an AI large model described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute an interface response processing method for an AI large model as described in any one of the embodiments of the present invention in any other appropriate manner (for example, by means of firmware).

[0211] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0212] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0213] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0214] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0215] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0216] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0217] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0218] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. An interface response processing method for a large AI model, characterized in that: include: Obtain the standard throughput value of the large AI model within the standard duration, split the standard duration into multiple time units, and construct a ring buffer corresponding to each time unit in the standard duration; After dividing the system time into multiple time segments according to the standard duration, a mapping relationship between each time segment and the ring buffer is established, and a request queue is allocated for each time unit of the ring buffer; When a call request to the AI ​​large model is detected, the AI ​​large model is predicted to be able to respond to the call request within the maximum waiting time based on the completed throughput value, standard throughput value, and the number of requests in each request queue in the ring buffer during the time period of the call request. If yes, the call request is added to the matching request queue in the ring buffer; otherwise, a timeout interrupt feedback message is directly generated as a response result to the call request; When the current system time hits any time unit in the ring buffer, each request in the request queue corresponding to the hit time unit is provided to the AI ​​big model for model calculation.

2. The method according to claim 1, characterized in that Based on the completed throughput value of the AI ​​large model in the time segment of the call request, the standard throughput value, and the number of requests in each request queue in the ring buffer, it is predicted whether the AI ​​large model can respond to the call request within the maximum waiting time, including: Get the request time of the call request and the completed throughput value of the AI ​​large model in the time segment of the request time; According to the request time point and the maximum waiting time, a target time unit set is determined in the ring buffer, and the request queues corresponding to each time unit in the target time unit set are obtained as candidate queues; If, based on the completed throughput value, the standard throughput value, and the number of requests in each request queue in the ring buffer, it is determined that the number of requests in the target candidate queue does not reach the upper limit of the number of requests in the target candidate queue, then it is determined that the AI ​​large model can respond to the call request within the maximum waiting time; Accordingly, the call request is added to the matching request queue in the ring buffer, including: Add the call request to the target candidate queue in the ring buffer.

3. The method according to claim 2, characterized in that Based on the request time point and the maximum waiting time, a target time unit set is determined in the ring buffer, including: Determine a start time unit in the ring buffer according to the request time point, and determine a plurality of delay time units in the ring buffer according to the maximum waiting time and the start time unit; The start time unit and the plurality of delay time units are combined to obtain the target time unit set.

4. The method according to claim 2, characterized in that If, based on the completed throughput value, the standard throughput value, and the number of requests in each request queue in the ring buffer, it is determined that the number of requests in the target candidate queue does not reach the upper limit of the number of requests in the target candidate queue, then it is determined that the AI ​​large model can respond to the call request within the maximum waiting time, including: Traverse the request queues of each time unit in the target time unit set in order of time extension, and obtain the first request queue that is not marked as having reached the queue limit as the current request queue; Check whether the time segment of the current time unit corresponding to the current request queue is the same as the time segment of the calling request; If they are the same, a first remaining throughput value is calculated based on the standard throughput value and the completed throughput value; Obtaining, in a time extension order, the first processed time units preceding the current time unit in the target time unit set; Obtaining a total number of stored requests in the request queue corresponding to each first processed time unit, and calculating a second remaining throughput value based on the first remaining throughput value and the total number of stored requests; Calculate the upper limit of the number of requests that match the current request queue based on the second remaining throughput value and the position of the current time unit in the corresponding time segment; If the number of requests in the current request queue is less than the upper limit of the current request queue, it is determined that the AI ​​large model can respond to the call request within the maximum waiting time; If the number of requests in the current request queue is greater than or equal to the upper limit of the request number, the current request queue is marked as having reached the queue limit, and the operation of traversing the request queues of each time unit in the target time unit set in the order of time extension is returned; If it is determined that there is no qualified target candidate queue in the target time unit set, it is determined that the AI ​​large model cannot respond to the call request within the maximum waiting time.

5. The method according to claim 4, characterized in that After detecting whether the time segment of the current time unit corresponding to the current request queue is the same as the time segment of the calling request, the method further includes: If they are different, then obtaining, in order of time extension, all second processed time units that are located before the current time unit and belong to the same time segment as the current time unit; Calculating a target remaining throughput value based on the standard throughput value and a total number of requests stored in the request queue corresponding to each second processed time unit; Calculate the upper limit of the number of requests that match the current request queue based on the target remaining throughput value and the position of the current time unit in the corresponding time segment; If the number of requests in the current request queue is less than the upper limit of the current request queue, it is determined that the AI ​​large model can respond to the call request within the maximum waiting time; If the number of requests in the current request queue is greater than or equal to the upper limit of the request number, the current request queue is marked as having reached the queue upper limit, and the operation of traversing the request queues of each time unit in the target time unit set in the order of time extension is returned.

6. The method according to claim 4, characterized in that Based on the second remaining throughput value and the position of the current time unit in the corresponding time segment, the upper limit of the number of requests matching the current request queue is calculated, including: According to the position of the current time unit in the time segment to which it belongs, obtaining at least one future time unit located after the current time unit in the time segment to which the current time unit belongs, and calculating the future duration composed of each future time unit; The upper limit of the number of requests that match the current request queue is calculated based on the proportional relationship between the current time unit and the future duration, the second remaining throughput, and the number of historical call requests under different historical time units in each historical time segment.

7. The method according to any one of claims 1 to 6, characterized in that The types of standard throughput include: the number of model calls and the number of minimum semantic unit tokens consumed; Accordingly, when the AI ​​large model cannot respond to the call request within the maximum waiting time for any type of standard throughput verification, a timeout interrupt feedback information is generated as a response to the call request.

8. An interface response processing device for an AI large model, characterized in that: include: The throughput benchmark acquisition module is used to obtain the standard throughput value of the AI ​​large model within the standard duration, split the standard duration into multiple time units, and construct a ring buffer corresponding to each time unit in the standard duration; The time partition mapping module is used to divide the system time into multiple time segments according to the standard time length, establish a mapping relationship between each time segment and the ring buffer, and allocate a request queue for each time unit in the ring buffer; The request response prediction module is used to predict whether the AI ​​large model can respond to the call request within the maximum waiting time when a call request to the AI ​​large model is detected, based on the completed throughput value of the AI ​​large model in the time segment to which the call request belongs, the standard throughput value, and the number of requests in each request queue in the ring buffer; The request scheduling decision module is used to add the call request to the matching request queue in the ring buffer if yes; otherwise, directly generate timeout interrupt feedback information as a response result to the call request; The time-triggered execution module is used to provide each request in the request queue corresponding to the hit time unit to the AI ​​big model for model calculation when the current system time hits any time unit in the ring buffer.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the interface response processing method of an AI large model as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement an interface response processing method for an AI large model according to any one of claims 1 to 7 when executed.

Images