Large model API parallel scheduling method and system based on matrix operation

Abstract

The invention provides a large model API parallel scheduling method and system based on matrix operation, and relates to the technical field of computer scheduling and optimization. The method comprises the steps that API calling tasks to be processed and features of the API calling tasks are expressed as a task feature matrix; determining a feature weight vector corresponding to the task feature matrix, a current system constraint vector and a running state; calculating a basic score vector of the target task according to a feature weight vector corresponding to the task feature matrix; determining a constraint conformity vector according to the current system constraint vector and the operation state; according to the basic score vector and the constraint conformity vector, an effective score vector is obtained through element-level multiplication calculation; selecting one or more tasks for scheduling execution according to the effective score vector; and updating the task feature matrix, the system constraint and the operation state after the task is executed. The problems that in the prior art, a parallel API calling management method is not intelligent enough in scheduling and poor in adaptability, and the overall performance is difficult to optimize are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer scheduling and optimization, and in particular to a large model API parallel scheduling method and system based on matrix operations. Background Art

[0002] In recent years, artificial intelligence technologies, represented by large language models (LLMs), have developed rapidly. Their powerful natural language processing capabilities are widely used in various scenarios, including question answering, translation, and content generation. To improve processing efficiency, application systems often need to initiate multiple requests to the large model API simultaneously, known as parallel calls. However, when making large-scale parallel API calls directly, the following problems often arise:

[0003] (1) API rate limiting: API service providers usually set limits on requests per minute (RPM), tokens per minute (TPM), or concurrent connections to protect their service resources. Large-scale parallel calls can easily trigger these limits, resulting in request rejection (e.g., returning a 429TooManyRequests error).

[0004] (2) Server-side overload: Even if the explicit rate limit is not reached, a large number of concurrent requests in a short period of time may temporarily overload the API server, resulting in response delays, timeouts, or returning internal server errors (such as 5×× errors);

[0005] (3) Client resource exhaustion: Initiating a large number of parallel requests will consume the caller's (client's) CPU, memory, network connection, thread / coroutine and other resources, which may cause client performance degradation or even crash.

[0006] (4) Network fluctuations: Unstable network connections may also cause some requests to fail or time out.

[0007] Existing technologies often use relatively simple strategies to manage parallel API calls, such as fixed-size concurrent pools, simple request queues, or basic retry logic. These methods struggle to dynamically adapt to the real-time state of the API server (such as temporary overloads), resource changes within the client itself, and the priority differences between different tasks. Especially when multiple constraints, such as the request rate (RPM), token consumption rate (TPM), and maximum number of concurrent connections, need to be met simultaneously, simple scheduling strategies can easily lead to resource waste (too conservative) or frequent limit triggering (too aggressive), affecting overall efficiency and stability. Summary of the Invention

[0008] In order to overcome the shortcomings of the existing technology, the purpose of the present invention is to provide a large-model API parallel scheduling method and system based on matrix operations. The present invention solves the problems in the existing technology of parallel API call management methods that are not intelligent enough, have poor adaptability, and are difficult to optimize overall performance when facing multiple dynamic constraints and task heterogeneity.

[0009] To achieve the above object, the present invention provides the following solutions:

[0010] A large model API parallel scheduling method based on matrix operations, comprising:

[0011] Represent the API call tasks to be processed and their features as a task feature matrix;

[0012] Determining a feature weight vector corresponding to the task feature matrix and a current system constraint vector and an operating state;

[0013] Calculate the basic score vector of the target task according to the feature weight vector corresponding to the task feature matrix;

[0014] Determining a constraint compliance vector based on the current system constraint vector and the operating state;

[0015] According to the basic score vector and the constraint compliance vector, the effective score vector is obtained by element-wise multiplication.

[0016] Select one or more tasks for scheduling execution according to the valid score vector;

[0017] After the task is executed, the task feature matrix, system constraints and operating status are updated.

[0018] Preferably, the task feature matrix includes:

[0019] Task categories and task characteristic dimensions;

[0020] The task feature dimensions include:

[0021] Task priority, estimated token consumption, estimated time, and current number of retries.

[0022] Preferably, the system constraints include:

[0023] API service rate limits (RPM, TPM), maximum concurrency limits, and client available resource limits.

[0024] Preferably, the calculation formula for the task priority is:

[0025] p i =p max -(p max / N)(i-1);

[0026] Among them, p max is the maximum task priority, N is the total number of tasks, and i is the i-th task.

[0027] Preferably, the calculation formula for the estimated token consumption is:

[0028] t i =t input +t maxoutput ;

[0029] Among them, t input is the number of tokens of the original input data of the API, t maxoutput The maximum output tokens value.

[0030] Preferably, the calculation formula for the estimated time consumption is:

[0031] c i =c latency +c factor ×t i ;

[0032] Among them, c latency is the time constant, c factor The average additional time required for the API server to process each token.

[0033] Preferably, the calculation expression of the effective score vector is:

[0034] S effective =S base ⊙C;

[0035] Among them, S base is the basic score vector, and C is the constraint compliance vector.

[0036] A large model API parallel scheduling system based on matrix operations, including:

[0037] A first matrix determination module is used to represent the API call task to be processed and its features as a task feature matrix;

[0038] A second matrix determination module is used to determine the feature weight vector corresponding to the task feature matrix and the current system constraint vector and operating state;

[0039] A first score calculation module is used to calculate a basic score vector of the target task according to the feature weight vector corresponding to the task feature matrix;

[0040] A third matrix determination module is used to determine a constraint compliance vector according to the current system constraint vector and the operating state;

[0041] A second score calculation module is used to obtain an effective score vector by element-wise multiplication based on the basic score vector and the constraint compliance vector;

[0042] A scheduling module, configured to select one or more tasks for scheduling execution according to the valid score vector;

[0043] The updating module is used to update the task feature matrix, system constraints and operating status after the task is executed.

[0044] The present invention discloses the following technical effects:

[0045] The present invention provides a large-model API parallel scheduling method and system based on matrix operations, the method comprising: representing the API call task to be processed and its features as a task feature matrix; determining the feature weight vector and the current system constraint vector and operating status corresponding to the task feature matrix; calculating the basic score vector of the target task according to the feature weight vector corresponding to the task feature matrix; determining the constraint compliance vector according to the current system constraint vector and operating status; calculating the effective score vector by element-wise multiplication based on the basic score vector and the constraint compliance vector; selecting one or more tasks for scheduling execution according to the effective score vector; the present invention updates the task feature matrix, system constraints and operating status after the task is executed. Multi-dimensional decision-making basis formalization: unifying complex task features and system constraints into a matrix framework, so that scheduling decisions can comprehensively consider multiple dimensions such as priority, cost, resource consumption, retry status, constraint compliance, etc., making the decision-making process more structured and explainable; scheduling strategy flexibility and scalability: by adjusting the feature weight vector V, the scheduling strategy can be easily changed (for example, giving priority to high-priority tasks, or giving priority to low-resource-consuming tasks, or giving priority to retrying failed tasks, etc.). It is also relatively easy to add new task features or system constraints, just by expanding the matrix dimensions and weight vectors; Dynamic adaptability is enhanced: the weight vector V and the system constraint vector L can be dynamically updated, so that the scheduling strategy can adapt to the real-time performance changes of the API server and the client's own resource status; Optimization potential: The matrix-based representation provides a basis for applying more complex optimization algorithms (such as prediction-based scheduling, reinforcement learning adjustment of weights V, etc.), which may achieve better overall performance (throughput, latency, success rate) than simple heuristic rules. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 A flowchart of a large model API parallel scheduling method based on matrix operations provided by an embodiment of the present invention;

[0048] Figure 2 A schematic diagram of the API parallel call process provided by an embodiment of the present invention;

[0049] Figure 3 A schematic diagram of a block translation workflow for an extremely long text provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0051] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] like Figure 1 As shown, the present invention provides a large model API parallel scheduling method based on matrix operations, including:

[0053] Step 100: Represent the API call task to be processed and its features as a task feature matrix;

[0054] Step 200: Determine the feature weight vector corresponding to the task feature matrix and the current system constraint vector and operating state;

[0055] Step 300: Calculate a basic score vector for the target task based on the feature weight vector corresponding to the task feature matrix;

[0056] Step 400: Determine a constraint compliance vector according to the current system constraint vector and operating status;

[0057] Step 500: Obtain an effective score vector by element-wise multiplication based on the basic score vector and the constraint compliance vector;

[0058] Step 600: Select one or more tasks for scheduling and execution according to the valid score vector;

[0059] Step 700: After the task is executed, the task feature matrix, system constraints and operating status are updated.

[0060] Specifically, construct a task feature matrix F: a batch (N) of API call tasks to be processed is represented as an N×M matrix F, where each row represents a task and each column represents a feature dimension of the task. Feature dimensions include: task priority p i , estimated token consumption t i , estimated time c i , Current retry count r i .

[0061] Task priority p i : It is used to quantify the business importance or expected execution priority of an API call task T relative to other tasks to be scheduled. The calculation formula is:

[0062] p i =p max -(p max / N)(i-1)

[0063] Where p max The maximum task priority, usually 10;

[0064] N is the total number of calls to the API to execute the task;

[0065] i is the i-th task. In the API parallel call process, the task ranking is determined based on the fixed node, such as Figure 2 shown. Figure 2 In the example, the priority value of task1 is p1=10, and the priority value of task1 is p i =10-(10 / N)(i-1).

[0066] Furthermore, the estimated token consumption t i : Used to quantitatively estimate the total number of tokens expected to be consumed when an API call (especially for large language model services) is successfully executed. This value is mainly calculated based on the input tokens, specifically:

[0067] Step 1: Use the official tokenizer library (Tokenizer) that strictly corresponds to the target API model to obtain the number of tokens t of the original input data that needs to be sent to the API for calling task T input .

[0068] Step 2: Get the maximum output tokens value t of the large model that strictly corresponds to the API model maxoutput .

[0069] Step 3: Calculate the estimated token consumption using the formula:

[0070] t i =t input +tmaxoutput ;

[0071] Estimated time c i : It is used to estimate the wall clock time (WC Time) that an API call task T will take from the time its request is issued to the time the complete response from the API server is received. The calculation formula is:

[0072] c i =c latency +c factor ×t i ;

[0073] Where c latency It is a fixed time value, representing a fixed time overhead that is not related to the token processing volume, such as the minimum network round trip time, request establishment and server basic processing time, etc. It can be taken as 0.5s

[0074] c facor This represents the average additional time required for the API server to process each token. This is related to the complexity of the model used. For models with small parameters or fast processing speed, it can be set in the range of 0.008 to 0.01s / token; for models with medium processing speed, it can be set in the range of 0.02 to 0.045s / token; for full-version general or deep thinking models with slow processing speed or particularly complex calculations, it can be set in the range of 0.04 to 0.12s / token.

[0075] Current retry count r i : Used to accurately measure the cumulative number of times a specific API call task has been instructed or retried by the scheduling system since its initial submission due to recoverable failures (for example, a temporary loss of network connection, or receiving an API response status code such as HTTP 429 or HTTP 503 that clearly indicates a later retry). The initial value is 0, and the subsequent execution is +1.

[0076] Maintenance system constraint vector L: represents the constraints faced by the system at the current moment, L = [MaxConcurrency, RemainingRPM, RemainingTPM]

[0077] MaxConcurrency: Maximum concurrency, a constant value that defines the maximum number of API call tasks that this scheduling system allows to be executed concurrently at the same time.

[0078] RemainingRPM: The number of requests remaining per minute. This is a constant value, typically less than the RPM limit specified by the API service provider. This value indicates how many new requests the system can initiate to the target API within the current one-minute window, ensuring it does not exceed the RPM limit.

[0079] RemainingTPM: The number of tokens remaining per minute. This is a constant value, typically less than the TPM limit specified by the API service provider. It indicates the estimated number of tokens that can be consumed by the system through new API calls within the current one-minute window.

[0080] Maintain a running state vector / matrix (R): This represents the currently running tasks and their resource usage. For example, a K×P matrix R (K is the current concurrency number) records the ID of the running task, the elapsed time, the consumed tokens, etc., or an N×1 state vector indicates whether the task is running.

[0081] Define the feature weight vector V: an M×1 vector that represents the importance weights of different task features in the scheduling decision.

[0082] V=[w p , w c , w t , w r ]

[0083] Priority weight w p : Baseline weight, directly reflecting the priority of API calls

[0084] Estimated time weight w c : Imposing minimal penalty on the maximum time consumption, providing a mild preference for short tasks;

[0085] Estimated token weight w t : impose significant penalties on maximum token consumption;

[0086] Retry weight w t : impose a medium penalty on the maximum number of retries;

[0087] In the present invention, the characteristic weight vector is given as: [1, -0.00006, -0.015, -0.5].

[0088] Furthermore, if you want to control costs more strictly, you can increase the estimated token weight w t If you find that long tasks are always not executed, you can reduce the estimated time weight w cIf the system tends to repeatedly try failed tasks (circular thinking), you can increase the retry weight w t ;

[0089] Calculate the basic score vector S base :Calculate a basic priority score for each task to be scheduled by performing operations (matrix multiplication) on the task feature matrix F and the feature weight vector V. This step combines multi-dimensional features into a single score; the matrix operation for calculating the basic priority score is matrix multiplication S base =F*V.

[0090] Furthermore, for each task i to be scheduled, based on its characteristics and the current system constraint vector L and the running state R, it is determined whether scheduling the task will violate the constraints.

[0091] Construct the constraint compliance vector C: an N×1 vector, c i =1 means that task i can be scheduled currently (all constraints are satisfied), c i = 0 means at least one constraint is violated. If only the concurrent number constraint check, the request number per minute constraint check, and the token number per minute constraint check meet the requirements, c i =1, otherwise c i =0.

[0092] Concurrency constraint check: Check whether the total number of concurrently executed API call tasks in the current system has reached the preset upper limit;

[0093] Requests per minute constraint check: Checks whether the system has a remaining quota of API requests within the current one-minute window.

[0094] Token count constraint check per minute: Checks whether the estimated number of tokens consumed by scheduling the execution of this particular candidate task i exceeds the remaining token amount in the current one-minute time window.

[0095] Furthermore, other constraint checks can be added, such as client resource (CPU, memory) constraint checks, concurrency limits for specific API endpoints, and the maximum number of retries allowed for a single task.

[0096] Calculate the effective score vector S effective : Through the basic score vector S base The element-wise multiplication (Hadamard product operation) of the constraint compliance vector C is performed, and the scores of tasks that do not meet the constraints are set to zero.

[0097] S effective =S base ⊙C;

[0098] The purpose of this vector is to ensure that a task's effective score reflects its base score only if it satisfies all constraints (the corresponding C vector element is 1). If a task fails to satisfy any constraint (the corresponding C vector element is 0), its effective score is set to 0 (or a specific value indicating invalidity), and it is excluded from the current scheduling decision, even if its base score is high. This vector is the direct basis for the scheduler to ultimately select the next task (or batch of tasks) to execute.

[0099] From S effective The task with the highest score (or several tasks with the highest scores, if multiple tasks can be started simultaneously) is selected as the next task to be scheduled.

[0100] Schedule selected tasks and execute API calls using mechanisms such as asynchronous I / O.

[0101] After the task is executed (success, failure, completion), the corresponding number of retries r in the task feature matrix F is updated i .

[0102] The system constraint vector L and operating status R are updated in real time or periodically.

[0103] Furthermore, the feature weight vector V can be dynamically and adaptively adjusted based on historical API call success rates, average response times, or the frequency of specific errors (such as 429). For example, if 429 errors are frequently encountered, the weight of the "low token consumption" feature can be temporarily increased, or the weight of the "high priority" feature can be temporarily decreased.

[0104] The calculation of the constraint compliance vector C can include more complex logic, such as predictive judgment: if the scheduled task i, within its estimated execution time c i Will it cause the subsequent RPM / TPM consumption to exceed the limit?

[0105] Furthermore, Figure 3 As shown, the present invention also provides another embodiment, which builds a workflow to achieve block translation of very long texts;

[0106] Specifically, the chunking standard is limited to the maximum input tokens of the DeepSeek model, 63K, and the estimated time is set to 120s and the maximum number of retries is set to 5.

[0107] When translating a 250KB long text, the text chunking node divided the original text into five chunks, with chunks 1 to 4 being 60KB long and chunk 5 being 10KB long. These chunks were then fed into six DeepSeek model API call subtasks, with the execution matrix parameters shown in Table 1.

[0108] Table 1

[0109]

[0110] From the above table, we can see that the priority execution order of each task is: Task 1, Task 2, Task 5, Task 6, Task 3, Task 4.

[0111] The task execution process is as follows: Schedule T1 for execution. Update the running status R (the concurrency count becomes 2). After T1 begins executing, decrement the RemainingRPM in L. If T1 completes, update its status in F, update R (the concurrency count returns to 1), and increment the RPM / TPM consumption record for the completed task. Then repeat S2-S5 to select the next task (T2, score 2.75).

[0112] This embodiment also provides a large model API parallel scheduling system based on matrix operations, including:

[0113] A first matrix determination module is used to represent the API call task to be processed and its features as a task feature matrix;

[0114] A second matrix determination module is used to determine the feature weight vector corresponding to the task feature matrix and the current system constraint vector and operating state;

[0115] A first score calculation module is used to calculate a basic score vector of the target task according to the feature weight vector corresponding to the task feature matrix;

[0116] A third matrix determination module is used to determine a constraint compliance vector according to the current system constraint vector and the operating state;

[0117] A second score calculation module is used to obtain an effective score vector by element-wise multiplication based on the basic score vector and the constraint compliance vector;

[0118] A scheduling module, configured to select one or more tasks for scheduling execution according to the valid score vector;

[0119] The updating module is used to update the task feature matrix, system constraints and operating status after the task is executed.

[0120] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0121] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A large model API parallel scheduling method based on matrix operations, characterized in that: include: Represent the API call tasks to be processed and their features as a task feature matrix; Determining a feature weight vector corresponding to the task feature matrix and a current system constraint vector and an operating state; Calculate the basic score vector of the target task according to the feature weight vector corresponding to the task feature matrix; Determining a constraint compliance vector based on the current system constraint vector and the operating state; According to the basic score vector and the constraint compliance vector, the effective score vector is obtained by element-wise multiplication. Select one or more tasks for scheduling execution according to the valid score vector; After the task is executed, the task feature matrix, system constraints and operating status are updated.

2. A large model API parallel scheduling method based on matrix operations according to claim 1, characterized in that: The task feature matrix includes: Task categories and task characteristic dimensions; The task feature dimensions include: Task priority, estimated token consumption, estimated time, and current number of retries.

3. A large model API parallel scheduling method based on matrix operations according to claim 1, characterized in that: The system constraints include: API service rate limits (RPM, TPM), maximum concurrency limits, and client available resource limits.

4. A large model API parallel scheduling method based on matrix operations according to claim 2, characterized in that: The calculation formula for the task priority is: p i =p max -(p max / N)(i-1); Among them, p max is the maximum task priority, N is the total number of tasks, and i is the i-th task.

5. The large model API parallel scheduling method based on matrix operation according to claim 2 is characterized in that: The calculation formula for the estimated token consumption is: t i =t input +t maxoutput ; Among them, t input is the number of tokens of the original input data of the API, t maxoutput The maximum output tokens value of the large model API.

6. A large model API parallel scheduling method based on matrix operations according to claim 2, characterized in that: The calculation formula for the estimated time consumption is: c i =c latency +c factor ×t i ; Among them, c latency is the time constant, c factor The average additional time required for the API server to process each token.

7. The large model API parallel scheduling method based on matrix operation according to claim 2 is characterized in that: The calculation expression of the effective score vector is: S effective =S base ⊙C; Among them, S base is the basic score vector, and C is the constraint compliance vector.

8. A large model API parallel scheduling system based on matrix operations, characterized in that: include: A first matrix determination module is used to represent the API call task to be processed and its features as a task feature matrix; A second matrix determination module is used to determine the feature weight vector corresponding to the task feature matrix and the current system constraint vector and operating state; A first score calculation module is used to calculate a basic score vector of the target task according to the feature weight vector corresponding to the task feature matrix; A third matrix determination module is used to determine a constraint compliance vector according to the current system constraint vector and the operating state; A second score calculation module is used to obtain an effective score vector by element-wise multiplication based on the basic score vector and the constraint compliance vector; A scheduling module, configured to select one or more tasks for scheduling execution according to the valid score vector; The updating module is used to update the task feature matrix, system constraints and operating status after the task is executed.

Images