A GPU computing power active scheduling method and system of a multi-tenant AI platform

By generating structured events when the AI ​​Agent tool is invoked, the waiting window is estimated in real time and the MIG partition is proactively reduced to release computing power to the elastic shared pool, which is then redistributed according to priority and expanded in advance before the tool responds. This solves the problems of signal lag and resource waste in multi-tenant GPU scheduling, and achieves efficient computing power utilization and zero-wait startup.

CN123114686APending Publication Date: 2026-11-17SHANGHAI EDICH TECHNOLOGY SERVICES CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611413820.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-11
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing multi-tenant GPU scheduling technology suffers from signal lag, amplified lag loss due to continuous exceeding of limits, and lack of predictive expansion and return mechanisms in AI Agent tool invocation scenarios, resulting in wasted computing resources and increased latency.

Method used

By generating structured events when the AI ​​Agent issues tool call commands, the waiting window is estimated in real time and the MIG partition is proactively reduced to release computing power to the elastic shared pool, which is then redistributed according to priority and expanded in advance before the tool responds, thus achieving proactive scheduling of computing power.

Benefits of technology

Eliminate the loss of computing power during the tool call waiting period, improve GPU utilization, ensure zero-wait startup of AI Agent generation, and improve platform service capacity and fairness.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123114686A_ABST
    Figure CN123114686A_ABST
Patent Text Reader

Abstract

A GPU computing power active scheduling method and system of a multi-tenant AI platform The present application relates to the technical field of intelligent computing power, and particularly relates to a GPU computing power active scheduling method. When an AI Agent issues a tool calling instruction, an execution engine generates a structured event and sends it to a scheduler in real time; the scheduler queries historical delay distribution according to the tool type, estimates the waiting window length of the tool calling, and obtains an effective release window; within the effective release window, the MIG partition of a tenant is reduced from the current specification to the minimum maintenance specification, and the released computing power is injected into an elastic shared pool; the computing power in the elastic shared pool is redistributed according to the real-time priority queue of other tenants; it is judged whether the tool calling response arrives before the end of the estimated window, and if it does not arrive in advance, the scheduler actively triggers MIG expansion before the tool calling response arrives, and the computing power of the tenant is restored to the original specification. The present application eliminates the computing power vacancy loss of the AI Agent tool calling waiting period in the multi-tenant AI platform, improves the overall utilization rate of the GPU, and guarantees zero waiting start in the AI Agent generation stage.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of intelligent computing power technology, specifically to a method and system for actively scheduling GPU computing power. Background Technology

[0002] With the rapid development of Large Language Model (LLM) technology, AI Agent systems have been widely applied to various multi-tenant cloud service platforms. When performing complex tasks, AI Agents typically need to call external tools multiple times (such as database queries, search engines, API calls, etc.), forming a cyclical execution pattern of "inference-tool call-inference". In this pattern, after the AI ​​Agent issues a tool call command, the GPU enters a waiting state until the tool responds before it can continue generating data. This waiting window typically lasts from 200 milliseconds to 5 seconds, during which the GPU's computing power is completely idle, resulting in significant waste of computing resources.

[0003] In multi-tenant environments, to improve GPU resource utilization efficiency, the industry commonly adopts MIG (Multi-Instance GPU) technology, which divides physical GPUs into multiple independent partitions and allocates them to different tenants. In existing technologies, the dynamic adjustment of MIG partitions mainly relies on passive scheduling methods based on latency metrics. For example, the paper "Predictable LLM Serving on GPU Clusters" (arXiv ID: 2508.20274, authors: Erfan Darzi et al.), published on arXiv in August 2025, proposes a latency-aware controller for multi-tenant GPU clusters. The controller continuously collects system metrics such as P99 / P999 tail latency and GPU utilization for each tenant every 1-5 seconds. When a tenant's P99 latency exceeds the preset threshold for multiple consecutive sampling windows, it triggers actions such as MIG partition specification adjustment (e.g., upgrading from 1g.10gb to 2g.20gb), tenant migration (e.g., migrating the tenant to a GPU location with better PCIe topology), or interfering neighbor rate limiting (e.g., applying MPS quotas or IO rate limits to interfering neighbor tenants). After the action is executed, the controller enters a cooldown period (DWELL_TIME), during which no new reconfiguration operation is triggered to prevent frequent jitter.

[0004] However, the aforementioned prior art has the following technical problems: First, the scheduling signal originates from external latency metrics, resulting in a structural lag. Current technology uses latency observations as the sole trigger signal, meaning that GPU computing power must already be wasting resources and latency must have increased before the controller receives input data. For AI Agent tool invocation scenarios, there is a non-eliminable signal lag of 1–5 seconds between the agent issuing the tool invocation command, the GPU entering a completely idle state, and the latency metric increasing and being detected. During this period, computing power waste has already occurred and is irreversible.

[0005] Second, the triggering mechanism requires multiple consecutive sampling windows to exceed the limit, further amplifying the lag loss. Existing technologies use a persistent condition of "multiple consecutive (e.g., Y times) exceedances" to prevent misjudgments. While this is a reasonable anti-jitter design in ordinary inference scenarios, for AI Agent tool invocation scenarios, the waiting time itself has a clear time boundary (automatically ending after the tool responds), making "continuous exceedances" unnecessary. This condition causes the entire triggering delay to be further amplified into multiple sampling periods, artificially extending the computational power loss window that could have been avoided.

[0006] Third, it lacks a predictive scaling and return mechanism, requiring additional waiting time for the AI ​​Agent to resume the generation phase. Current technology cannot predict when a tool call will end, therefore there is no "pre-scaling" design. When the tool call response returns and the AI ​​Agent needs to immediately enter the generation phase, the MIG scaling operation has not yet started, and the AI ​​Agent must wait for the GPU to be in place before continuing execution. This waiting delay is directly added to the user-perceived end-to-end response time and cannot be eliminated within the existing technological framework. Summary of the Invention

[0007] To address the aforementioned problems, the present invention aims to provide a method for proactive scheduling of GPU computing power in a multi-tenant AI platform.

[0008] Another objective of this invention is to provide an active GPU computing power scheduling system for a multi-tenant AI platform.

[0009] A method for proactively scheduling GPU computing power in a multi-tenant AI platform includes: Step S1: When the AI ​​Agent issues a tool invocation command, the execution engine generates a structured event and sends it to the scheduler in real time. Step S2: The scheduler queries the historical delay distribution based on the tool type contained in the structured event, estimates the waiting window duration T for tool calls, and obtains the effective release window T_release based on the waiting window duration T. Step S3: Within the effective release window T_release, the scheduler issues an instruction to the GPU driver layer to reduce the tenant's MIG partition from the current size to the minimum maintenance size and inject the released computing power into the elastic shared pool. Step S4: The computing power in the elastic shared pool is redistributed according to the real-time priority queues of other tenants; Step S5: Determine whether the tool call response arrives before the estimated window ends. If it does not arrive in advance, the scheduler actively triggers MIG expansion before the tool call response arrives, restoring the tenant's computing power to its original specifications.

[0010] The GPU computing power active scheduling method for a multi-tenant AI platform described in this invention includes a structured event containing a tenant ID, an agent instance ID, a tool type, and / or an estimated waiting time. The tool type includes database queries, external APIs, or search engines.

[0011] In the GPU computing power active scheduling method of the multi-tenant AI platform described in this invention, the effective release window T_release is 80% of the waiting window duration T.

[0012] The GPU computing power active scheduling method for the multi-tenant AI platform described in this invention refers to a minimum maintenance specification that is the minimum MIG partition specification that only retains the context state and does not perform computation.

[0013] The GPU computing power active scheduling method for a multi-tenant AI platform described in this invention uses a weighted fair scheduling algorithm for the reallocation in step S4, and writes the allocation record into the tenant service ledger.

[0014] In the GPU computing power active scheduling method of the multi-tenant AI platform described in this invention, in step S5, before the tool call response arrives, the scheduler triggers MIG expansion by a time amount Δt, and the time amount Δt is determined based on the sum of the estimated network transmission delay and the MIG expansion time. In step S5, if the tool call response arrives ahead of schedule, the system will immediately interrupt the borrowing after detecting the response signal, prioritize returning the computing power of this tenant, and other tenant tasks used during the borrowing period will be put into the pause queue.

[0015] The active GPU computing power scheduling method for the multi-tenant AI platform described in this invention also includes a passive scheduling path based on latency indicators. When an event fails to be captured or an anomaly occurs, the system switches to the passive scheduling path for computing power scheduling.

[0016] A GPU computing power proactive scheduling system for a multi-tenant AI platform, comprising the following methods for proactively scheduling GPU computing power in the multi-tenant AI platform: The event capture module is used by the execution engine to generate structured events when the AI ​​Agent issues tool call instructions and send them to the scheduler in real time; The window estimation module, in which the scheduler queries the historical delay distribution based on the tool type contained in the structured event, estimates the waiting window duration T for tool calls, and obtains the effective release window T_release based on the waiting window duration T; The active reduction module is used to, within the effective release window T_release, instruct the scheduler to issue an instruction to the GPU driver layer to reduce the tenant's MIG partition from the current size to the minimum maintenance size. The computing power reallocation module is used to inject the released computing power into the elastic sharing pool and reallocate the computing power in the elastic sharing pool according to the real-time priority queue of other tenants. The early return module is used to proactively trigger MIG expansion by the scheduler before the tool call response arrives, restoring the tenant's computing power to its original specifications.

[0017] The GPU computing power active scheduling system of the multi-tenant AI platform described in this invention also includes an exception fallback module, which is used to immediately interrupt the borrowing when the tool call response arrives early, prioritize the return of the computing power of this tenant, and put other tenant tasks used during the borrowing period into the pause queue.

[0018] A computer-readable storage medium storing a computer program that executes the GPU computing power active scheduling method of the multi-tenant AI platform.

[0019] Beneficial effects: This invention constructs a complete active scheduling link of "event capture → window estimation → active reduction → elastic reuse → early return", which eliminates the computing power idle loss during the waiting period of AI Agent tool calls in multi-tenant AI platforms without increasing hardware resources, improves the overall utilization of GPU, and ensures zero-wait startup of AI Agent generation stage. Attached Figure Description

[0020] Figure 1 This is a flowchart of a GPU computing power active scheduling method for a multi-tenant AI platform according to the present invention; Figure 2 This is a comparison diagram of the technical effects of the prior art and the present invention; Figure 3 This is a flowchart illustrating a specific embodiment of the present invention; Figure 4 This is a block diagram of a GPU computing power active scheduling system for a multi-tenant AI platform according to the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0023] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.

[0024] Reference Figure 1 A method for proactively scheduling GPU computing power in a multi-tenant AI platform, comprising: Step S1: When the AI ​​Agent issues a tool invocation command, the execution engine generates a structured event and sends it to the scheduler in real time. Step S2: The scheduler queries the historical delay distribution based on the tool type contained in the structured event, estimates the waiting window duration T for tool calls, and obtains the effective release window T_release based on the waiting window duration T. Step S3: Within the effective release window T_release, the scheduler issues an instruction to the GPU driver layer to reduce the tenant's MIG partition from the current size to the minimum maintenance size and inject the released computing power into the elastic shared pool. Step S4: Reallocate the computing power in the elastic shared pool according to the real-time priority queues of other tenants; Step S5: Determine whether the tool call response arrives before the estimated window ends. If it does not arrive in advance, the scheduler will actively trigger MIG expansion before the tool call response arrives, restoring the tenant's computing power to its original specifications.

[0025] The purpose of this invention is to address three technical problems existing in current multi-tenant GPU scheduling technologies for AI Agent tool invocation scenarios, and to provide a proactive GPU computing power scheduling method for multi-tenant AI platforms. To address the structural lag in scheduling signals, this invention uses tool invocation semantic events generated by the AgentLoop execution engine as scheduling trigger signals, replacing the passive sampling mechanism based on external latency indicators in existing technologies. This advances the signal response time from "after the GPU is idle and latency has increased" to "the instant the tool invocation command is issued," fundamentally eliminating the 1-5 second structural signal lag. To address the amplification of lag loss due to Y consecutive exceedance conditions, this invention uses a single semantic event as the trigger condition, eliminating the need to wait for any continuous sampling windows. This eliminates the additional triggering delay introduced by persistence conditions (persistence Y) in existing technologies and fully covers tool invocation scenarios with durations shorter than Y sampling windows. To address the lack of a predictive expansion and return mechanism, this invention estimates the tool invocation waiting window based on historical latency distribution and triggers MIG expansion and return at a preset time before the window ends, ensuring the GPU is ready when the AI ​​Agent is restored and generating, reducing the recovery waiting time to near zero.

[0026] This invention constructs a complete active scheduling chain of "event capture → window estimation → active reduction → elastic reuse → early return", which eliminates the computing power idle loss during the waiting period of AI Agent tool calls in multi-tenant AI platforms without increasing hardware resources, improves the overall utilization of GPU, and ensures zero-wait startup of AI Agent generation stage.

[0027] The GPU computing power active scheduling method of the multi-tenant AI platform of the present invention includes structured events containing tenant ID, agent instance ID, tool type and / or estimated waiting time, and tool type includes database query or external API or search engine.

[0028] The present invention eliminates the loss of computing power idleness during the tool invocation waiting period in the stage signal capture stage of step S1. When the AI ​​Agent issues a tool invocation command, the platform's Agent loop execution engine generates a structured event, in which the estimated waiting time is based on historical data. The GPU computing power during the tool invocation waiting period can be fully captured and transferred to other tenants for use, instead of being idle.

[0029] In one specific embodiment, the effective release window T_release is 80% of the waiting window duration T.

[0030] Step S2 of this invention is the waiting window estimation stage, which provides an accurate time base for proactive reduction and early return. After receiving a structured event, the scheduler estimates the current waiting window duration T based on the historical latency distribution of tool types such as database queries, external APIs, and search engines. After setting a safety margin, an effective release window is obtained, in which a 20% margin is reserved for return preparation. By predicting the tool call duration through historical data, an accurate time window is provided for subsequent computing power release and return operations, avoiding blind operations. By setting an effective release window, the computing power can be maximized for other tenants while ensuring sufficient time for return preparation.

[0031] Step S3 of this invention is the active reduction of the MIG partition, achieving active release of computing power and zero idle time. Within the effective release window T_release, the scheduler issues an instruction to the GPU driver to reduce the tenant's MIC partition from its current size to the minimum maintenance size, and the released computing power is injected into the elastic shared pool. The minimum maintenance size is the smallest MIG partition size that retains only the context state and does not perform computations. This fundamentally solves the problem of completely idle GPU computing power during tool call waiting periods, transforming the previously wasted computing power into resources that can be effectively utilized by other tenants.

[0032] In a preferred embodiment, the reallocation in step S4 adopts a weighted fair scheduling algorithm, and the allocation record is written into the tenant service ledger.

[0033] Step S4 is the computing power reallocation stage, realizing dynamic sharing and fair scheduling of computing power across tenants. Computing power in the elastic shared pool is allocated according to the real-time priority queues of other tenants, using weighted fair scheduling to ensure that high-priority tenants receive power first. Allocation records are written to the tenant service ledger for subsequent fairness calculations. Released computing power is reallocated according to priority and fairness principles, ensuring that high-priority tenants receive power first, and the borrowing status is recorded in the service ledger to prevent long-term unfairness. This invention improves the platform's service capacity through computing power time reuse without adding any hardware, which is equivalent to increasing the platform's hardware computing power output. The service ledger mechanism ensures long-term fairness and prevents one tenant from harming the rights of other tenants due to frequent borrowing.

[0034] In a preferred embodiment, in step S5, before the tool call response arrives, the scheduler triggers MIG expansion by a time amount Δt, where the time amount Δt is determined based on the sum of the estimated network transmission delay and the MIG expansion time.

[0035] In step S5, if the tool call response arrives ahead of schedule, the system will immediately interrupt the borrowing upon detecting the response signal, prioritize returning the computing power of this tenant, and put other tenant tasks used during the borrowing period into the pause queue.

[0036] Step S5 comprises the early return phase and the exception fallback phase. The early return phase eliminates GPU waiting latency during the AI ​​Agent recovery and generation phase. Based on historical latency distribution, the waiting window is estimated, and MIG expansion is triggered at a preset time before the window ends, allowing the expansion operation to execute in parallel with the tool call tail segment. The exception fallback phase provides a first layer of fallback outside the active path. When the tool response arrives early, the scheduled timer is immediately canceled, and MIG expansion and return are triggered to prevent the AI ​​Agent from waiting due to insufficient computing power. Compared to existing technologies, this application significantly improves performance under normal scenarios and maintains performance parity with existing technologies under abnormal scenarios, with no risk of performance degradation.

[0037] Reference Figure 2 By comparing the GPU computing power utilization of the present invention with that of the prior art during the tool call waiting period, the advantages of the present invention are intuitively demonstrated.

[0038] The problem with existing technology is: In Phase ①, the GPU computing power is idle. During the tool call waiting period (t0–t1) of tenant A, its GPU computing power is completely idle and is not used by any tenant.

[0039] There is a recovery delay in phase ②. When the tool response returns (at time t1), tenant A cannot immediately resume generation because it needs to wait for MIG expansion, resulting in a significant recovery delay.

[0040] At time t0, tenant A issues a tool call command. At time t1, the tool call response returns. The period from t0 to t1 is the tool call waiting period. In the prior art, the computing power is idle. In this invention, the computing power is borrowed by tenant B. After time t1, the prior art has a recovery delay. This invention has zero-wait recovery.

[0041] In the third stage of this invention, tenant B obtains borrowed computing power. During the tool call waiting period (t0–t1) of tenant A, the released computing power is injected into the elastic shared pool for tenant B to use, thereby eliminating the idle computing power. Before the tool response returns (at time t1), the system has completed the MIG expansion in advance, so tenant A can resume generation with zero waiting time.

[0042] One specific embodiment, referring to Figure 3 A represents the tool call event, and P represents the delayed observation. The event type is determined by whether to enter the tool call event (A) or the delayed observation (P).

[0043] The steps of the active path from A1 to A8 are as follows: Step A1: Capture tool invocation events; Step A2: Estimate the waiting window; Step A3: Shrink the MIG partition; Step A4: Release computing power to the elastic shared pool; Step A5: Set the return timer; Step A6 determines whether the response was delivered ahead of schedule. If yes, execute A7a to cancel the timer and immediately expand the capacity. If no, execute A7b to trigger the timer and expand the capacity and return the data. Step A8 updates the tenant service ledger, and the process ends.

[0044] The steps of the passive fallback path from P1 to P6 are as follows: Step P1: Update window W and calculate P99; During the cooldown period of step P2, if yes, proceed to start; otherwise, proceed to step P3 to determine if p99 exceeds the threshold. If yes, proceed to step P4 to upgrade MIG isolation; otherwise, proceed to step P5 to relax isolation, restart the instance, and then proceed to step P6 to write to the scheduling log and end.

[0045] Step A1 of this invention involves monitoring tool invocation events generated by the AgentLoop execution engine and capturing structured event signals at the moment the AI ​​Agent issues a tool invocation command. These structured event signals include tenant ID, tool type, and timestamp. Existing technologies use external latency metrics as scheduling signals, requiring the GPU to be idle and latency to be elevated before detection. This invention switches the signal source from "external latency metrics" to "internal execution semantic events." The AI ​​Agent issuing the tool invocation command occurs at a precisely known moment at the AgentLoop level. This moment is no more than the time it takes for the GPU to enter an idle state (typically less than 100ms), significantly earlier than any external metric sampling period (1–5 seconds). The response time has been moved from "after the GPU has been idle for a few seconds" to "when the GPU is about to be idle", with a signal lead time of 1-5 seconds. The GPU computing power during the tool call waiting period (usually 200ms-5 seconds) can be fully captured and transferred to other tenants for use, instead of being idle. In multi-agent concurrent scenarios, if the platform runs N agents at the same time, and each agent has an average tool call waiting period of T seconds per round, the additional effective computing power that can be released per hour is approximately N×T×number of rounds per hour. The efficiency improvement is positively correlated with the scale of concurrency.

[0046] Step A1 of this invention replaces the persistent judgment condition of "p99 exceeding the preset threshold τ for Y consecutive sampling windows" in the prior art with an event-driven triggering mechanism. This eliminates the trigger lag caused by "Y consecutive exceedances." The prior art sets the persistent condition Y to prevent accidental triggering due to momentary jitter, which is necessary in ordinary inference scenarios with random fluctuations. However, the GPU idle window invoked by the AI ​​Agent tool is not random jitter, but a deterministic interval defined by a clear semantic event (tool invocation instruction), with both its start and end points having clear logical meanings. This invention uses semantic events as triggering conditions, eliminating the possibility of misjudgment. Therefore, it does not require persistent window protection and can respond immediately upon the occurrence of the event. This invention completely eliminates the waiting loss of Y sampling windows in the prior art. Based on a sampling period of 1 to 5 seconds and Y equal to 2 to 3, the prior art has a minimum delay of 2 to 15 seconds for each trigger, while the delay of this invention is less than 100ms. For tool calls with a duration shorter than Y sampling windows, such as fast database queries of 200 to 500ms, the prior art cannot be optimized at all because the triggering conditions can never be met. This invention can fully cover such short window scenarios.

[0047] The steps A3 (reducing the MIG partition) and A4 (releasing computing power to the elastic shared pool and writing it into the tenant service ledger) of the present invention can improve the overall GPU utilization of multi-tenants.

[0048] Existing technologies keep MIG partitions static during tenant holding, preventing other tenants from borrowing them even if the tenant's GPU utilization is zero. This invention, upon detecting a tool call event, proactively reduces the tenant's MIG partition to the minimum size required to maintain the context, injects the released computing power into an elastic shared pool, and allocates it to other tenants according to their real-time priority queues by the scheduler. Simultaneously, the borrowing record is written to the tenant service ledger for long-term fairness calculations, preventing tenants who frequently borrow from accumulating service debt. This invention improves overall GPU utilization: In multi-tenant scenarios, if each tenant agent spends an average of 30% of their time waiting for tool calls, this invention can convert this previously idle computing power into effective computation, theoretically increasing the platform's effective GPU utilization by approximately 20-30 percentage points. Without adding any hardware, it increases platform service capacity through computing power time reuse, equivalent to increasing the platform's unit hardware computing power output, i.e., productivity. The service ledger mechanism ensures long-term fairness, preventing one tenant from harming the rights of other tenants due to frequent borrowing.

[0049] The steps A6 of this invention, which set the scheduled return timer Schedule(RestoreMIG, T_release), and A7a and A7b, which trigger the MIG expansion return in advance, can eliminate the GPU waiting delay during the AI ​​Agent recovery and generation phase.

[0050] Existing technologies lack a pre-emptive return mechanism. MIG expansion only begins after the tool call response returns, and the expansion operation itself requires a certain amount of time; for example, MIG reconfiguration typically takes hundreds of milliseconds. During this period, although the AI ​​Agent has received the tool results, it cannot immediately enter the generation stage, resulting in additional waiting. This invention estimates the waiting window T for the current tool call based on the historical latency distribution (P50 quantile) and sets the effective release time T_release = T × 0.8. MIG expansion is triggered in advance at the T_release time, allowing the expansion operation to be executed in parallel with the tool call tail segment. By the time the tool response arrives, the GPU is already expanded and ready. This invention reduces the waiting time for the AIAgent to enter the generation stage from "MIG expansion time (hundreds of milliseconds)" to near zero. This optimization is directly reflected in the end-to-end response time perceived by the user, representing a tangible improvement in user experience. The 20% safety margin design ensures early return while providing a buffer for estimation errors, ensuring the solution remains stable and reliable even in real-world environments where tool response times fluctuate.

[0051] The present invention's step A8, which updates the tenant service ledger to the end and simultaneously retains the original passive paths P1 to P6 as a double fallback, is exceptionally robust and does not introduce new service risks.

[0052] This invention retains two fallback layers in addition to the active path. The first layer is an immediate return mechanism when the tool response arrives early, preventing the AI ​​Agent from waiting due to insufficient computing power. The second layer retains the passive path for exceeding latency limits in existing technologies, covering edge scenarios that the active path fails to capture (such as AgentLoop event loss, significant tool response timeouts, etc.). These two fallback layers ensure that the service quality of this invention is no lower than that of existing technologies under abnormal conditions. Compared to existing technologies, this invention significantly improves performance in normal scenarios and maintains performance parity with existing technologies in abnormal scenarios, without the risk of performance degradation. The dual-path design of the active path and the passive fallback makes this invention reliable for engineering implementation, rather than only effective under ideal conditions.

[0053] Reference Figure 4 A GPU computing power proactive scheduling system for a multi-tenant AI platform, comprising a method for proactively scheduling GPU computing power in a multi-tenant AI platform, including: Event capture module 1 is used to generate structured events when the AI ​​Agent issues a tool call command and send them to the scheduler in real time; Window estimation module 2: The scheduler queries the historical delay distribution based on the tool type contained in the structured event, estimates the waiting window duration T for tool calls, and obtains the effective release window T_release based on the waiting window duration T; Active reduction module 3 is used to send instructions from the scheduler to the GPU driver layer within the effective release window T_release to reduce the tenant's MIG partition from the current size to the minimum maintenance size. The computing power reallocation module 4 is used to inject the released computing power into the elastic sharing pool and reallocate the computing power in the elastic sharing pool according to the real-time priority queue of other tenants. The early return module 5 is used to proactively trigger MIG expansion before the tool call response arrives, restoring the tenant's computing power to its original specifications.

[0054] This invention provides an agent-aware GPU computing power proactive scheduling system for multi-tenant AI agent platforms, which solves three technical problems in existing technologies: passive occupation of GPU computing power, scheduling lag, and recovery delay in AI agent tool invocation scenarios. It enables proactive release, elastic reuse, and zero-wait return of GPU computing power in multi-tenant environments.

[0055] The GPU computing power active scheduling system of the multi-tenant AI platform of the present invention also includes an anomaly fallback module 6, which is used to immediately interrupt the lending when the system detects the response signal when the tool call response arrives early, prioritize the return of the computing power of this tenant, and put other tenant tasks used during the lending period into the pause queue.

[0056] A computer-readable storage medium storing a computer program for actively scheduling GPU computing power in a multi-tenant AI platform.

[0057] For those skilled in the art, various changes and modifications will undoubtedly be apparent after reading the above description. Therefore, the appended claims should be construed as covering all changes and modifications that encompass the true intent and scope of the invention. Any and all equivalent scope and content within the scope of the claims should be considered to remain within the intent and scope of the invention.

Claims

1. A GPU computing power active scheduling method of a multi-tenant AI platform, characterized by, include: Step S1: When the AI ​​Agent issues a tool invocation command, the execution engine generates a structured event and sends it to the scheduler in real time. Step S2: The scheduler queries the historical delay distribution based on the tool type contained in the structured event, estimates the waiting window duration T for tool calls, and obtains the effective release window T_release based on the waiting window duration T. Step S3: Within the effective release window T_release, the scheduler issues an instruction to the GPU driver layer to reduce the tenant's MIG partition from the current size to the minimum maintenance size and inject the released computing power into the elastic shared pool. Step S4: The computing power in the elastic shared pool is redistributed according to the real-time priority queues of other tenants; Step S5: Determine whether the tool call response arrives before the estimated window ends. If it does not arrive in advance, the scheduler actively triggers MIG expansion before the tool call response arrives, restoring the tenant's computing power to its original specifications.

2. The GPU computing power active scheduling method of the multi-tenant AI platform according to claim 1, wherein, The structured event includes tenant ID, agent instance ID, tool type, and / or estimated wait time, where the tool type includes database query, external API, or search engine.

3. The GPU computing power active scheduling method of the multi-tenant AI platform according to claim 1, wherein, In step S2, the effective release window T_release is 80% of the waiting window duration T.

4. The GPU computing power active scheduling method of the multi-tenant AI platform according to claim 1, wherein, The minimum maintenance specification is the minimum MIG partition specification that retains only the context state and does not perform calculations.

5. The GPU computing power active scheduling method of the multi-tenant AI platform according to claim 1, wherein, The reallocation in step S4 adopts a weighted fair scheduling algorithm, and the allocation record is written into the tenant service ledger.

6. The GPU computing power active scheduling method of the multi-tenant AI platform according to claim 1, wherein, In step S5, before the tool call response arrives, the scheduler triggers MIG expansion by a time amount Δt, where the time amount Δt is determined based on the sum of the estimated network transmission delay and the MIG expansion time. In step S5, if the tool call response arrives ahead of schedule, the system will immediately interrupt the borrowing after detecting the response signal, prioritize returning the computing power of this tenant, and other tenant tasks used during the borrowing period will be put into the pause queue.

7. The GPU computing power active scheduling method of the multi-tenant AI platform according to claim 1, wherein, It also includes a passive scheduling path based on latency metrics, which switches to computing power scheduling when an event fails to be captured or an anomaly occurs. 8.A GPU computing power active scheduling system of a multi-tenant AI platform, characterized in that, A method for actively scheduling GPU computing power for a multi-tenant AI platform according to any one of claims 1-7, comprising: The event capture module is used by the execution engine to generate structured events when the AI ​​Agent issues tool call instructions and send them to the scheduler in real time; The window estimation module, in which the scheduler queries the historical delay distribution based on the tool type contained in the structured event, estimates the waiting window duration T for tool calls, and obtains the effective release window T_release based on the waiting window duration T; The active reduction module is used to, within the effective release window T_release, instruct the scheduler to issue an instruction to the GPU driver layer to reduce the tenant's MIG partition from the current size to the minimum maintenance size. The computing power reallocation module is used to inject the released computing power into the elastic sharing pool and reallocate the computing power in the elastic sharing pool according to the real-time priority queue of other tenants. An early return module is configured to proactively trigger MIG scaling by the scheduler to restore the computing power of the tenant to the original specification before the tool call response arrives. 9.The GPU computing power active scheduling system of the multi-tenant AI platform of claim 8, wherein, An exception bottoming module is also included, which is configured to interrupt the loan immediately after the system detects the response signal when the tool call response arrives early, and preferentially return the computing power of the tenant. Other tenant tasks used during the loan period are placed in a suspension queue.

10. A computer readable storage medium storing a computer program, characterized in that, The GPU computing power active scheduling method of the multi-tenant AI platform according to any one of claims 1 to 7 is executed.