Request scheduling method and apparatus, device, and storage medium

Abstract

This application discloses a request scheduling method, apparatus, device, and storage medium, relating to the field of large model technology. The method includes: receiving batch tasks submitted by a user for large model services, and pre-connecting the batch tasks with the large model to obtain current model call requests; adding the current model call requests to a model call thread pool according to a first-in-first-out (FIFO) principle; determining the current scheduling state of the large model based on the current number of requests and the current lexical usage during the service call process; and, if the current scheduling state is schedulable, invoking the current model call requests in the model call thread pool through the large model. Compared to existing methods that control the number of requests by limiting the rate to a fixed QPS threshold, this application can manage and schedule resources through the number of requests and the lexical usage, thereby improving the utilization efficiency of large model resources.

Description

Technical Field

[0001] This application relates to the field of large model technology, and in particular to a request scheduling method, apparatus, device and storage medium. Background Technology

[0002] Currently, in scenarios involving the batch invocation of large model services (such as intelligent customer service, content generation, data analysis, etc.), in order to meet the enterprise's demand for large-scale, high-efficiency model services and improve business processing speed and user experience, the batch request processing method is generally used to invoke large model services.

[0003] When handling large batch requests for models, a simple counter-based rate limiting strategy is typically used to control the number of requests, i.e., queries per second (QPS), to prevent system overload. For example, a fixed QPS threshold is set, and subsequent requests are rejected when the number of requests reaches that threshold. However, this approach ignores the core resources of large model services, leading to either idle resources or resource exhaustion during sudden surges in requests, making it difficult to fully utilize large model resources. For instance, in high-concurrency scenarios, even if the QPS does not reach the threshold, the high resource consumption of a single request can quickly deplete model resources, preventing subsequent legitimate requests from being processed in a timely manner and impacting user experience. Summary of the Invention

[0004] The main purpose of this application is to provide a request scheduling method, apparatus, device and storage medium, which aims to solve the technical problem that traditional QPS threshold rate limiting methods cannot fully utilize large model resources.

[0005] To achieve the above objectives, this application proposes a request scheduling method, the method comprising:

[0006] Receive batch tasks submitted by users for large model services, and pre-connect the batch tasks with the large model to obtain the current model call request; The current model call request is added to the model call thread pool according to the first-in-first-out principle; The current scheduling status of the large model is determined based on the current number of requests and the current usage of terms during the service invocation process. When the current scheduling state is schedulable, the current model call request of the model call thread pool is invoked through the large model.

[0007] In one embodiment, the step of receiving batch tasks submitted by the user for large model services and pre-connecting the batch tasks with the large model to obtain the current model invocation request includes: Receive batch tasks submitted by users for large model services, and extract business prompt words from the batch tasks; Write the business prompts into the temporary elastic search index associated with the batch task; Write the batch tasks into a message queue to obtain a task queue; Based on the elastic search temporary index, the task queue is polled at a preset period, and a model call request is initiated on the polling results through a long event connection sent by the server.

[0008] In one embodiment, the step of polling the task queue at a preset period based on the elastic search temporary index and initiating a model invocation request for the polling results through a long event connection sent by the server includes: The task queue is polled at a preset period to determine the tasks to be executed, and the index data corresponding to the tasks to be executed is obtained from the elastic search temporary index. The index data is assembled to obtain a model call request object; The server establishes a long-lived connection for sending events to the model by using a thread pool to invoke the request object. The server sends a long-lived event connection to initiate a corresponding model call request to the large model.

[0009] In one embodiment, the step of determining the current scheduling state of the large model based on the current number of requests and the current token usage during the service invocation process includes: The number of requests made by the large model during the service call process is counted using a sliding window to obtain the current number of requests; The input and output terms of the large model during the service call process are summed to obtain the current term usage. The current number of requests is compared with a first upper limit to obtain a first comparison result, and the current word usage is compared with a second upper limit to obtain a second comparison result; Based on the first comparison result and the second comparison result, the current scheduling state of the large model is determined.

[0010] In one embodiment, the step of invoking the current model invocation request of the model invocation thread pool through the large model when the current scheduling state is a schedulable state includes: When the current scheduling state is schedulable, the token consumption and query rate per second corresponding to the current model call request are estimated, wherein the schedulable state is determined when the number of current requests is less than the first upper limit and the current token usage is less than the second upper limit. Based on the word consumption and the query rate per second, the model lock of the large model is preempted; After successfully preempting, the current model call request in the model call thread pool is invoked through the large model.

[0011] In one embodiment, after the step of preempting the model lock of the large model based on the word consumption and the query rate per second, the method further includes: In the event of a preemption failure, the current model call request is added to the waiting queue, and the waiting time of the current model call request in the waiting queue is recorded. If the waiting time reaches a preset time, a wake-up command is generated; According to the wake-up command, return to the step of counting the requests of the large model during the service call process through a sliding window to obtain the current number of requests; Once the preemption is successful, the large model invokes the current model in the waiting queue.

[0012] In one embodiment, after the step of invoking the current model invocation request of the model invocation thread pool through the large model when the current scheduling state is a schedulable state, the method further includes: Receive the model call results returned by the large model through an asynchronous response listener; If the model call result is successful, obtain the response result of the large model and send the response result to the business logic thread pool for processing; If the model call result is a call failure, the task status of the task to be executed is updated to obtain a re-call status; Based on the recall status, return to the step of writing the business prompt words into the elastic search temporary index associated with the batch task, until the model call result is a successful call.

[0013] Furthermore, to achieve the above objectives, this application also proposes a request scheduling device, the device comprising: The request module is used to receive batch tasks submitted by users for large model services, and pre-connect the batch tasks with the large model to obtain the current model call request; The thread module is used to add the current model call request to the model call thread pool according to the first-in-first-out principle; The status module is used to determine the current scheduling status of the large model based on the current number of requests and the current usage of lexical terms during the service call process. The calling module is used to invoke the current model invocation request of the model invocation thread pool through the large model when the current scheduling state is schedulable.

[0014] In addition, to achieve the above objectives, this application also proposes a request scheduling device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the request scheduling method as described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the request scheduling method described above.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: The request scheduling method of this application includes: receiving batch tasks submitted by users for large model services, and pre-connecting the batch tasks with the large model to obtain current model call requests; adding the current model call requests to the model call thread pool according to the first-in-first-out principle; determining the current scheduling state of the large model based on the current number of requests and the current word usage during the service call process; and, if the current scheduling state is schedulable, calling the current model call requests in the model call thread pool through the large model.

[0017] This application first receives batch tasks submitted by users for large model services and pre-connects them to the large models. Then, it adds model call requests to a thread pool according to the first-in-first-out principle. Simultaneously, it determines the scheduling state based on the current number of requests to the large model and the amount of lexical usage, and performs model calls when the model is schedulable. Compared to existing methods that control the number of requests by limiting the rate to a fixed QPS threshold, this application manages and schedules resources based on the number of requests and the amount of lexical usage, thereby improving the utilization efficiency of large model resources. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1This is a flowchart illustrating the scheduling method provided in Embodiment 1 of this application. Figure 2 This is an overall architecture diagram of request scheduling provided in Embodiment 1 of this application; Figure 3 This is a flowchart illustrating the second embodiment of the scheduling method requested in this application; Figure 4 This is a flowchart of preprocessing batch tasks provided in Embodiment 2 of this application; Figure 5 This is a flowchart illustrating the scheduling method provided in Embodiment 3 of this application; Figure 6 A flowchart of the rate limiting stage provided in Embodiment 3 of this application; Figure 7 This is a block diagram of the module structure of the scheduling device requested in an embodiment of this application; Figure 8 This is a schematic diagram of the hardware operating environment involved in the request scheduling device in the embodiments of this application.

[0021] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0023] The main solution of this application is: Currently, in scenarios involving batch calls to large model services (such as intelligent customer service, content generation, data analysis, etc.), in order to meet the enterprise's demand for large-scale, high-efficiency model services and improve business processing speed and user experience, batch request processing is generally used to call large model services.

[0024] When handling large batch requests for models, a simple counter-based rate limiting strategy is typically used to control the number of requests, i.e., queries per second (QPS), to prevent system overload. For example, a fixed QPS threshold is set, and subsequent requests are rejected when the number of requests reaches that threshold. However, this approach ignores the core resources of large model services, leading to either idle resources or resource exhaustion during sudden surges in requests, making it difficult to fully utilize large model resources. For instance, in high-concurrency scenarios, even if the QPS does not reach the threshold, the high resource consumption of a single request can quickly deplete model resources, preventing subsequent legitimate requests from being processed in a timely manner and impacting user experience.

[0025] To address the aforementioned issues, this application provides a request scheduling method. First, it receives batch tasks submitted by users for large model services and pre-connects them to the large model. Then, it adds model call requests to a thread pool according to a first-in, first-out (FIFO) principle. Simultaneously, it determines the scheduling state based on the current number of requests to the large model and the amount of lexical usage. Model calls are then performed when the model is schedulable. Compared to existing methods that control the number of requests by limiting the rate to a fixed QPS threshold, this application manages and schedules resources based on the number of requests and the amount of lexical usage, thereby improving the utilization efficiency of large model resources.

[0026] It should be noted that the executing entity of this application embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a personal computer, a server, etc., or an electronic device capable of implementing the above functions, a request scheduling device executing the request scheduling method of this application, etc., and this embodiment does not limit it. The following uses a request scheduling device (hereinafter referred to as the device) as an example to describe this embodiment and the following embodiments.

[0027] Based on this, this application proposes a request scheduling method according to a first embodiment, referring to... Figure 1 , Figure 1 This is a flowchart illustrating a first embodiment of the request scheduling method of this application. In this embodiment, the request scheduling method may include steps S10 to S40: Step S10: Receive the batch tasks submitted by the user for the large model service, and pre-connect the batch tasks with the large model to obtain the current model call request.

[0028] It should be noted that batch tasks can refer to multiple requests submitted by a user at once that require processing by a large model, occurring in scenarios involving the batch invocation of large model services (such as intelligent customer service handling thousands of user inquiries simultaneously, content generation platforms generating marketing copy in batches, and data analysis systems processing multi-dimensional reports in parallel). For example, an e-commerce platform might need to generate 100,000 product descriptions in batches during a promotional period; these tasks would be submitted to the device as a set.

[0029] It should also be noted that the current model call request may be the request parameters obtained after establishing an initial connection with the large model service and encapsulating them before the formal processing of the batch task.

[0030] In practical use, such as Figure 2 As shown, Figure 2 This is an overall architecture diagram of request scheduling provided in Embodiment 1 of this application. After the user submits a batch of tasks for the large model service to the device, the device writes the batch of tasks into the elastic search temporary index, and then retrieves data from the elastic search temporary index sequentially through the execution thread pool. Before the model is called, the tasks are pre-processed; for example, the pre-connection interface of the large model service can be called to initialize the model parameter cache and verify the service availability, and then the task information is parsed and encapsulated into a model call request object; the model call request object is pre-connected with the large model to obtain the current model call request.

[0031] Step S20: Add the current model call request to the model call thread pool according to the first-in-first-out principle.

[0032] Understandably, the model call thread pool can be a resource-pooled thread used to manage concurrent calls to large model services, enabling saturated throughput of large models while ensuring system stability. The model call thread pool continuously acquires tasks to be processed, ensuring no queuing for data processing tasks and maximizing batch processing efficiency to its theoretical limit.

[0033] In practical use, such as Figure 2 As shown, the device can store the current model call requests in the model call thread pool queue in the order of arrival, and wait for resource allocation.

[0034] Step S30: Determine the current scheduling status of the large model based on the current number of requests and the current usage of terms during the service invocation process.

[0035] It is also understandable that the current request count (i.e., QPS) can be the total number of service requests that the large model is processing at the current moment. For example, in an intelligent customer service scenario, if the device receives 500 user inquiry requests per second, but due to limitations in computing resources, the large model can only process 100 requests simultaneously, then the current request count can refer to the 100 requests that are being processed.

[0036] It should be understood that the current token usage can be the total number of tokens actually consumed by the large model when processing the current request (i.e., the core resource consumption unit of the large model), used to reflect the actual consumption of computing resources by the large model. For example, in a content generation scenario, generating a 500-word marketing copy may require 2000 tokens, while generating a 50-word short copy only requires 100 tokens.

[0037] In practical use, such as Figure 2 As shown, the device can monitor the current number of requests and token consumption (i.e., the current number of tokens used) of the large model in real time, comprehensively evaluate the resource usage of the large model, determine the current scheduling status, and perform rate limiting on the large model.

[0038] Furthermore, in order to obtain the current scheduling state of the aforementioned large model, in this embodiment, the step of determining the current scheduling state of the large model based on the current number of requests and the current token usage during the service invocation process includes: Step S31: Count the requests made by the large model during the service call process using a sliding window to obtain the current number of requests.

[0039] It should be noted that the sliding window can be a time window used to dynamically count the number of requests. By setting a fixed duration (such as 1 second or 1 minute) of scrolling interval, the total number of requests within the window can be calculated in real time.

[0040] Step S32: Accumulate the input and output tokens of the large model during the service call process to obtain the current token usage.

[0041] It should also be noted that input tokens and output tokens refer to the input tokens (i.e., the length of the text submitted by the user) consumed by the large model when processing the request, and the generated output tokens (i.e., the length of the text returned by the model), respectively. For example, in a content generation scenario, if the user inputs "generate a 100-word product introduction" (approximately 20 input tokens), and the model returns "this product has high performance, is easy to operate, etc." (approximately 80 output tokens), then the total number of tokens used in this request is 100.

[0042] Step S33: Compare the current number of requests with the first upper limit value to obtain the first comparison result, and compare the current word usage with the second upper limit value to obtain the second comparison result.

[0043] Step S34: Determine the current scheduling state of the large model based on the first comparison result and the second comparison result.

[0044] Understandably, the first and second upper limits correspond to safety thresholds for the number of requests and the amount of tokens used, respectively, to determine whether the large model is overloaded. For example, the first upper limit can be set to a QPS threshold of 1000 times / second (i.e., the upper limit for the number of requests), and the second upper limit can be set to 500,000 tokens per second (i.e., the upper limit for the amount of tokens used). When the number of current requests counted by the sliding window exceeds 1000, or the current amount of tokens used exceeds 500,000, the large model will trigger a rate-limiting mechanism.

[0045] Furthermore, different rate limiting mechanisms can be set up according to the usage scenario to adapt to the differentiated rate limiting strategies of extreme computing and intelligent computing. For example, for extreme computing scenarios (such as model call scenarios requiring extremely fast response, such as high concurrency and short requests), the upper limit threshold for the number of requests can be adjusted to reduce the upper limit threshold for token consumption, focusing on ensuring throughput stability. For intelligent computing scenarios (such as model call scenarios with high token consumption, such as complex inference and long text), the upper limit threshold for the number of requests can be adjusted to increase the upper limit threshold for the number of requests, while reducing the upper limit threshold for token consumption, focusing on the efficient utilization of token resources.

[0046] In practical use, during the aforementioned rate limiting process, the device performs multi-dimensional budget management on the number of requests (QPS) and the amount of input / output tokens. First, using a sliding window, the total usage of QPS and tokens is calculated at different time intervals (e.g., seconds, minutes). Based on the upper limit thresholds configured for each dimension, rate limiting is triggered whenever the total consumption reaches the upper limit. If any total consumption does not reach the upper limit, the large model invocation phase proceeds directly. Thus, by employing a dual control mechanism of request count (QPS) and input / output tokens, large model invocations are always run at the edge of available resources, ensuring that all available tokens for large models are fully utilized and avoiding resource idleness and sudden exhaustion.

[0047] Step S40: If the current scheduling state is schedulable, invoke the current model invocation request of the model invocation thread pool through the large model.

[0048] It should be understood that the schedulable state can be a state in which the large model service can safely handle new requests, based on an assessment of the current resource usage (i.e., the current number of requests and the current usage of lexical terms).

[0049] For example, when the number of requests processed by the large model per second does not reach the QPS threshold (e.g., 1000 requests / second), and the remaining token quota can support the consumption of new requests (e.g., 10000 tokens remaining, while new requests require an average of 1000 tokens), the large model is marked as schedulable, allowing requests to be retrieved from the thread pool for invocation; conversely, if the number of requests is close to the threshold or the token balance is insufficient, it is switched to an unschedulable state, pausing the processing of new requests to avoid resource overload.

[0050] Furthermore, such as Figure 2 As shown, in this embodiment, after the step of invoking the current model invocation request of the model invocation thread pool through the large model when the current scheduling state is a schedulable state, the method further includes: Step S51: Receive the model call result returned by the large model through the asynchronous response listener.

[0051] Understandably, the model call result can be the feedback information returned by the large model after performing inference on the input task (such as user questions or text generation instructions).

[0052] It's also understandable that asynchronous response listeners can be components used for non-blocking reception of large model return results. They can monitor the completion status of model inference tasks (such as success or failure) to avoid blocking business threads while waiting for results. For example, in a batch content generation scenario, after a user submits 1000 text generation requests, the device captures the inference results of each request in real time through an asynchronous response listener.

[0053] Step S52: If the model call result is successful, obtain the response result of the large model and send the response result to the business logic thread pool for processing.

[0054] It should be noted that the business logic thread pool can be a collection of threads specifically for handling business-related tasks, responsible for executing the business logic output by the model.

[0055] Step S53: If the model call result is a call failure, update the task status of the task to be executed to obtain a re-call status.

[0056] Step S54: Based on the recall status, return to the step of writing the business prompt words into the elastic search temporary index associated with the batch task, until the model call result is a successful call.

[0057] It should also be noted that the task status can be an identifier describing the current stage of the task to be executed. For example, the initial status might be "Pending," updated to "Recall" after a model call fails, and updated to "Completed" upon success. The elastic search temporary index can be a temporary data storage structure used for quickly writing to and retrieving intermediate states of batch tasks. For example, in a batch task copywriting generation scenario, the device can write user-inputted prompts to the temporary index for the model to read when calling it.

[0058] In this embodiment, as Figure 2As shown, in the model invocation and business logic thread pool, after the model invocation message is processed, the device receives the model invocation result returned by the large model through an asynchronous response listener. When the model invocation is successful, the device can encapsulate the response result into a result object and submit it to an independent business logic asynchronous thread pool for processing. If the model invocation fails (e.g., timeout, network interruption), the task to be executed is returned to the task queue, and the status of the task to be executed is updated to the retry status and rewritten into the elastic search temporary index, and then it re-enters the queue to wait for the retry phase. Through this failure queuing mechanism combined with the elastic search temporary index, it is ensured that batch tasks can automatically recover from transient failures without manual intervention, improving the stability of batch processing and preventing scheduling blockage.

[0059] This application provides a request scheduling method, the method comprising: receiving a batch task submitted by a user for a large model service, and pre-connecting the batch task with the large model to obtain a current model call request; adding the current model call request to a model call thread pool according to a first-in-first-out principle; determining the current scheduling state of the large model based on the current number of requests and the current word usage during the service call process; and, if the current scheduling state is schedulable, calling the current model call request in the model call thread pool through the large model.

[0060] This embodiment first receives batch tasks submitted by users for large model services and pre-connects them to the large model. Then, it adds model call requests to the thread pool according to the first-in-first-out principle. Simultaneously, it determines the scheduling state based on the current number of requests to the large model and the amount of lexical usage, and performs model calls when the model is schedulable. Compared to existing methods that control the number of requests by limiting the rate to a fixed QPS threshold, this application can manage and schedule resources based on the number of requests and the amount of lexical usage, thereby improving the utilization efficiency of large model resources.

[0061] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to the above embodiment can be referred to the above description, and will not be repeated hereafter. On this basis, a second embodiment of the dialogue method of this application is proposed, please refer to... Figure 3 , Figure 3 This is a flowchart illustrating a second embodiment of the request scheduling method in this application. To obtain the aforementioned current model invocation request, as shown... Figure 3 As shown, in this embodiment, the step of receiving batch tasks submitted by the user for large model services and pre-connecting the batch tasks with the large model to obtain the current model invocation request includes: Step S11: Receive batch tasks submitted by users for large model services, and extract business prompt words from the batch tasks.

[0062] It should be noted that business prompts can be extracted from batch tasks and are core input information used to guide the generation of results by large models. They can include business-related constraints such as task type, style requirements, and output format.

[0063] Step S12: Write the business prompt words into the elastic search temporary index associated with the batch task.

[0064] It should also be noted that the elastic search temporary index can be a constructed temporary data storage structure used to quickly write business prompts associated with batch tasks.

[0065] Step S13: Write the batch tasks into the message queue to obtain the task queue.

[0066] Step S14: Based on the elastic search temporary index, poll the task queue at a preset period, and send a long event connection through the server to initiate a model call request for the polling results.

[0067] Understandably, the preset periodic polling could be a fixed time interval (e.g., every 5 seconds) to query the status of the task queue in order to trigger subsequent model calls.

[0068] It is also understandable that Server-Sent Events (SSE) long-lived connections can be based on a one-way HTTP communication protocol, allowing the server to proactively push real-time data to the client. For example, a device can push tasks to a large model in real time via an SSE long-lived connection without requiring frequent client polling.

[0069] In practical use, refer to Figure 4 , Figure 4 This is a flowchart illustrating the preprocessing of batch tasks provided in Embodiment 2 of this application. The device receives batch tasks submitted by the user for large model services and then initiates an initialization process for these batch tasks. First, the business prompts for the batch tasks are loaded, and each business prompt is validated, standardized in format, and has its metadata completed. Then, the preprocessed business prompts are written in batches to the elastic search temporary index. Subsequently, the task ID or complete message body of the batch tasks is pushed into a message queue to build a task queue, completing the enqueue operation. Finally, tasks are periodically pulled from the head of the queue at fixed time intervals to start executing the tasks through the execution thread pool. Thus, the elastic search temporary index can improve task processing efficiency and avoid task loss or duplicate processing.

[0070] Furthermore, in order to initiate a model invocation request, in this embodiment, the step of polling the task queue at a preset period based on the elastic search temporary index and initiating a model invocation request for the polling results through a long event connection sent by the server includes: Step S141: Poll the task queue at a preset period to determine the tasks to be executed, and obtain the index data corresponding to the tasks to be executed from the elastic search temporary index.

[0071] It should be noted that the index data can be business data associated with the task to be executed, such as task ID, business prompts, input parameters, and other structured information stored in the temporary index of Elastic Search.

[0072] Step S142: Assemble the index data to obtain a model call request object.

[0073] Step S143: Build a long-lived connection for the server to send events to the model by using the execution thread pool to call the request object.

[0074] Step S144: Send an event long connection through the server to initiate a corresponding model call request to the large model.

[0075] It should also be noted that the model call request object can be a standardized request structure that encapsulates index data, used to initiate a call to the large model service.

[0076] In this embodiment, as Figure 4 As shown, the device sequentially retrieves data, polls the task queue, and determines the tasks to be executed. Then, based on the task information, it retrieves complete index data from the temporary elastic search index and assembles it into a model call request object containing information such as model name, parameter configuration, and callback address. Next, a worker thread in the execution thread pool establishes a long-lived server-side event connection for this model call request object, sending the model call request to the large model server through this connection. This long-lived server-side event connection enables sequential task scheduling, ensuring the thread pool operates at full capacity continuously, improving concurrency and real-time performance.

[0077] Based on the first and / or second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to that in embodiments one and two above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 5 , Figure 5 This is a flowchart illustrating a third embodiment of the request scheduling method in this application. To invoke the current model invocation request, such as... Figure 5 As shown, in this embodiment, the step of invoking the current model invocation request of the model invocation thread pool through the large model when the current scheduling state is a schedulable state includes: Step S41: If the current scheduling state is schedulable, estimate the token consumption and query rate per second corresponding to the current model call request, wherein the schedulable state is determined when the current number of requests is less than the first upper limit and the current token usage is less than the second upper limit.

[0078] It should be noted that the schedulable state can be a system state that allows model calls to be initiated. The determination conditions are that the current number of requests is less than the first upper limit (e.g., concurrent requests ≤ 100) and the current word usage is less than the second upper limit (e.g., consumed words ≤ 100,000). When these two conditions are met, the device considers that the large model resources have not reached the bottleneck and can safely process new requests.

[0079] It should also be noted that the lexical consumption can be the total number of lexical units required by a large model to process the current model call request, based on the input (such as text length and complexity). The query rate per second can be the number of queries that the current model call request may trigger per unit of time.

[0080] Step S42: Based on the word consumption and the query rate per second, preempt the model lock of the large model.

[0081] Step S43: After successful preemption, invoke the current model invocation request of the model invocation thread pool through the large model.

[0082] Understandably, a model lock can be a distributed resource coordination mechanism used to control concurrent access permissions to large model services, preventing multiple requests from simultaneously occupying the model and causing conflicts or performance degradation. After successful preemption, the current model call request holding the lock can exclusively access the large model resource, ensuring the task completes smoothly.

[0083] In this embodiment, reference Figure 6 , Figure 6 This is a flowchart of the rate limiting stage provided in Embodiment 3 of this application. The pre-processing for model calls mainly involves rate limiting. Upon arrival of a current model call request, the model lock is first preempted based on a distributed lock. At this time, the device can read the current number of requests and the current token usage of the large model in real time and compare them with preset upper limit thresholds. If neither dimension reaches the upper limit, the large model is considered to have sufficient resources (i.e., the current scheduling state is schedulable). In this case, the model lock preemption is successful, and the request is directly allowed to enter the model call stage. Simultaneously, the token consumption and query rate per second of the current model call request are pre-deducted to prevent instantaneous over-limit. Thus, model lock preemption ensures priority execution of requests and improves resource utilization.

[0084] Furthermore, in this embodiment, after the step of preempting the model lock of the large model based on the word consumption and the query rate per second, the method further includes: if the preemption fails, adding the current model call request to the waiting queue and recording the waiting time of the current model call request in the waiting queue; if the waiting time reaches a preset time, generating a wake-up command; according to the wake-up command, returning to the step of counting the requests of the large model in the process of calling the service through a sliding window to obtain the current number of requests; until the preemption is successful, calling the current model call request in the waiting queue through the large model.

[0085] It should be noted that a waiting queue can be a first-in, first-out (FIFO) data structure used to temporarily store model call requests that cannot be processed immediately due to preemption failure.

[0086] It should also be noted that the waiting duration can be the time interval from when the current model call request is added to the waiting queue to the current moment, used to determine whether timeout processing needs to be triggered. The preset duration can be a pre-set waiting timeout threshold, used to control the maximum tolerable time for a request in the waiting queue. The wake-up command can be a signal that triggers the device to reassess the resource status and attempt to process the timeout request.

[0087] In this embodiment, as Figure 6 As shown, if resources are exhausted, preemption fails. In this case, no error is returned; instead, the current model call request is added to a waiting queue, awaiting the next round of preemption and a response. The waiting time of the current model call request in the waiting queue is recorded. Based on the waiting time, current model call requests in the queue are periodically woken up for retrying until resources become available. Upon successful preemption, the model call is executed directly. Thus, by using a waiting queue and recording the waiting time, request pausing and timeout management are achieved, preventing request loss due to resource contention.

[0088] It should be noted that the above examples are for understanding this application only and do not constitute a limitation on the scheduling method requested in this application. Any simple modifications based on this technical concept are within the scope of protection of this application. All actions involving the acquisition of signals, information, or data in this application are performed in accordance with the relevant data protection regulations of the country where the application is located and with authorization from the owner of the relevant device.

[0089] This application also provides a request scheduling device, please refer to... Figure 7 , Figure 7 This is a block diagram of the module structure of the request scheduling device according to an embodiment of this application; in this embodiment, the request scheduling device includes: The request module 701 is used to receive batch tasks submitted by the user for large model services, and pre-connect the batch tasks with the large model to obtain the current model call request; Thread module 702 is used to add the current model call request to the model call thread pool according to the first-in-first-out principle; The status module 703 is used to determine the current scheduling status of the large model based on the current number of requests and the current usage of lexical terms during the service call process of the large model. The calling module 704 is used to invoke the current model invocation request of the model invocation thread pool through the large model when the current scheduling state is a schedulable state.

[0090] This embodiment first receives batch tasks submitted by users for large model services and pre-connects them to the large model. Then, it adds model call requests to a thread pool according to the first-in-first-out principle. Simultaneously, it determines the scheduling state based on the current number of requests to the large model and the amount of lexical usage. Model calls are then performed when the model is schedulable. Compared to existing methods that control the number of requests by limiting the rate to a fixed QPS threshold, this embodiment manages and schedules resources based on the number of requests and the amount of lexical usage, thereby improving the utilization efficiency of large model resources.

[0091] In one implementation, the request module 701 is further configured to receive batch tasks submitted by the user for large model services, and extract business prompt words from the batch tasks; write the business prompt words into a temporary elastic search index associated with the batch tasks; write the batch tasks into a message queue to obtain a task queue; poll the task queue according to a preset period based on the temporary elastic search index, and initiate a model call request for the polling results through a long event connection sent by the server.

[0092] In one implementation, the request module 701 is further configured to poll the task queue at a preset period to determine the task to be executed, and obtain the index data corresponding to the task to be executed from the elastic search temporary index; assemble the index data to obtain a model call request object; build a server sending event long connection for the model call request object through the execution thread pool; and initiate a corresponding model call request to the large model through the server sending event long connection.

[0093] In one implementation, the state module 703 is further configured to count the requests made by the large model during the service invocation process using a sliding window to obtain the current request count; accumulate the input and output terms of the large model during the service invocation process to obtain the current term usage; compare the current request count with a first upper limit value to obtain a first comparison result, and compare the current term usage with a second upper limit value to obtain a second comparison result; and determine the current scheduling state of the large model based on the first comparison result and the second comparison result.

[0094] In one implementation, the calling module 704 is further configured to, when the current scheduling state is schedulable, estimate the token consumption and query rate per second corresponding to the current model call request, wherein the schedulable state is determined when the current number of requests is less than the first upper limit and the current token usage is less than the second upper limit; based on the token consumption and the query rate per second, preempt the model lock of the large model; after successful preemption, invoke the current model call request of the model call thread pool through the large model.

[0095] In one implementation, the calling module 704 is further configured to, in the event of a preemption failure, add the current model call request to a waiting queue and record the waiting time of the current model call request in the waiting queue; if the waiting time reaches a preset time, generate a wake-up command; according to the wake-up command, return to the step of counting the requests of the large model in the service call process through a sliding window to obtain the current request count; until the preemption is successful, invoke the current model call request in the waiting queue through the large model.

[0096] In one implementation, the calling module 704 is further configured to receive the model calling result returned by the large model through an asynchronous response listener; if the model calling result is a successful call, obtain the response result of the large model and send the response result to the business logic thread pool for processing; if the model calling result is a failed call, update the task status of the task to be executed to obtain a re-call status; based on the re-call status, return to the step of writing the business prompt words into the elastic search temporary index associated with the batch task, until the model calling result is a successful call.

[0097] Other embodiments or specific implementations of the scheduling device requested in this application can be found in the above-described method embodiments, and will not be repeated here.

[0098] The request scheduling apparatus provided in this application, employing the request scheduling method in the above embodiments, can solve the technical problem that traditional QPS threshold rate limiting methods cannot fully utilize large model resources. Compared with the prior art, the beneficial effects of the request scheduling apparatus provided in this application are the same as those of the request scheduling method provided in the above embodiments, and other technical features in the request scheduling apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0099] This application provides a request scheduling device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the request scheduling methods in the above embodiments.

[0100] The following is for reference. Figure 8 , Figure 8 This is a schematic diagram of the hardware operating environment involved in the request scheduling device in the embodiments of this application, showing a structural schematic diagram suitable for implementing the request scheduling device in the embodiments of this application. The request scheduling device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable media players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The request scheduling device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0101] like Figure 8As shown, the request scheduling device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in the read-only memory 1002 or a program loaded from the storage device 1003 into the random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the request scheduling device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the request dispatch device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows request dispatch devices with various systems, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.

[0102] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0103] The request scheduling device provided in this application, employing the request scheduling method in the above embodiments, can solve the technical problem that traditional QPS threshold rate limiting methods cannot fully utilize large model resources. Compared with the prior art, the beneficial effects of the request scheduling device provided in this application are the same as those of the request scheduling method provided in the above embodiments, and other technical features in this request scheduling device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0104] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0105] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0106] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the request scheduling method in the above embodiments.

[0107] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0108] The aforementioned computer-readable storage medium may be included in the request scheduling device; or it may exist independently and not assembled into the request scheduling device.

[0109] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a request scheduling device, cause the request scheduling device to: receive batch tasks submitted by a user for large model services, and pre-connect the batch tasks with the large model to obtain a current model invocation request; add the current model invocation request to the model invocation thread pool according to the first-in-first-out principle; determine the current scheduling state of the large model based on the current number of requests and the current word usage during the service invocation process; and, if the current scheduling state is schedulable, invoke the current model invocation request in the model invocation thread pool through the large model.

[0110] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0111] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation that may be implemented in systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0112] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0113] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described request scheduling method. This solves the technical problem that traditional QPS threshold-based rate limiting methods struggle to fully utilize large model resources. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the request scheduling method provided in the above embodiments, and will not be elaborated upon here.

[0114] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the request scheduling method described above.

[0115] The computer program product provided in this application can solve the technical problem that traditional QPS threshold rate limiting methods cannot fully utilize large model resources. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the request scheduling method provided in the above embodiments, and will not be repeated here.

[0116] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the scope of protection of this application.

Claims

1. A request scheduling method, characterized in that, The method includes: Receive batch tasks submitted by users for large model services, and pre-connect the batch tasks with the large model to obtain the current model call request; The current model call request is added to the model call thread pool according to the first-in-first-out principle; The current scheduling status of the large model is determined based on the current number of requests and the current usage of terms during the service invocation process. When the current scheduling state is schedulable, the current model call request of the model call thread pool is invoked through the large model.

2. The method as described in claim 1, characterized in that, The step of receiving batch tasks submitted by users for large model services and pre-connecting the batch tasks with the large model to obtain the current model invocation request includes: Receive batch tasks submitted by users for large model services, and extract business prompt words from the batch tasks; Write the business prompts into the temporary elastic search index associated with the batch task; Write the batch tasks into a message queue to obtain a task queue; Based on the elastic search temporary index, the task queue is polled at a preset period, and a model call request is initiated on the polling results through a long event connection sent by the server.

3. The method as described in claim 2, characterized in that, The steps of polling the task queue at a preset period based on the elastic search temporary index and initiating a model invocation request for the polling results through a long event connection sent by the server include: The task queue is polled at a preset period to determine the tasks to be executed, and the index data corresponding to the tasks to be executed is obtained from the elastic search temporary index. The index data is assembled to obtain a model call request object; The server establishes a long-lived connection for sending events to the model by using a thread pool to invoke the request object. The server sends a long-lived event connection to initiate a corresponding model call request to the large model.

4. The method as described in claim 1, characterized in that, The step of determining the current scheduling state of the large model based on the current number of requests and the current usage of terms during the service invocation process includes: The number of requests made by the large model during the service call process is counted using a sliding window to obtain the current number of requests; The input and output terms of the large model during the service call process are summed to obtain the current term usage. The current number of requests is compared with a first upper limit to obtain a first comparison result, and the current word usage is compared with a second upper limit to obtain a second comparison result; Based on the first comparison result and the second comparison result, the current scheduling state of the large model is determined.

5. The method as described in claim 4, characterized in that, The step of invoking the current model invocation request of the model invocation thread pool through the large model when the current scheduling state is schedulable includes: When the current scheduling state is schedulable, the token consumption and query rate per second corresponding to the current model call request are estimated, wherein the schedulable state is determined when the number of current requests is less than the first upper limit and the current token usage is less than the second upper limit. Based on the word consumption and the query rate per second, the model lock of the large model is preempted; After successfully preempting, the current model call request in the model call thread pool is invoked through the large model.

6. The method as described in claim 5, characterized in that, After the step of preempting the model lock of the large model based on the word consumption and the query rate per second, the method further includes: In the event of a preemption failure, the current model call request is added to the waiting queue, and the waiting time of the current model call request in the waiting queue is recorded. If the waiting time reaches a preset time, a wake-up command is generated; According to the wake-up command, return to the step of counting the requests of the large model during the service call process through a sliding window to obtain the current number of requests; Once the preemption is successful, the large model invokes the current model in the waiting queue.

7. The method according to any one of claims 3 to 6, characterized in that, After the step of invoking the current model invocation request of the model invocation thread pool through the large model when the current scheduling state is schedulable, the method further includes: Receive the model call results returned by the large model through an asynchronous response listener; If the model call result is successful, obtain the response result of the large model and send the response result to the business logic thread pool for processing; If the model call result is a call failure, the task status of the task to be executed is updated to obtain a re-call status; Based on the recall status, return to the step of writing the business prompt words into the elastic search temporary index associated with the batch task, until the model call result is a successful call.

8. A request scheduling device, characterized in that, The device includes: The request module is used to receive batch tasks submitted by users for large model services, and pre-connect the batch tasks with the large model to obtain the current model call request; The thread module is used to add the current model call request to the model call thread pool according to the first-in-first-out principle; The status module is used to determine the current scheduling status of the large model based on the current number of requests and the current usage of lexical terms during the service call process. The calling module is used to invoke the current model invocation request of the model invocation thread pool through the large model when the current scheduling state is schedulable.

9. A request scheduling device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the request scheduling method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the request scheduling method as described in any one of claims 1 to 7.

Images