Agent-schedule based gpu scheduling system

Abstract

This invention relates to the fields of cloud computing, artificial intelligence infrastructure, and container resource scheduling technology, specifically an Agent-Schedule-based GPU scheduling system. The system includes: a Scheduler Center for receiving resource request and release requests from service callers, maintaining the real-time status of all hosts, GPUs, and containers in the cluster, generating and issuing container operation instructions, and processing execution callbacks from Agent nodes; Agent Resource Nodes for registering their local GPU hardware information, periodically reporting container running status and resource usage, receiving and executing container operation instructions, and sending callbacks of execution results; and a database for persistently storing resource configuration, running status, and instruction flow records, providing data support for global lock concurrency control, instruction execution closed-loop, and container state machine management. This solves the problems of slow resource request response, low GPU memory utilization, and poor scheduling execution reliability in existing technologies.

Description

Technical Field

[0001] This invention relates to the fields of cloud computing, artificial intelligence infrastructure, and container resource scheduling technology, specifically a GPU scheduling system based on Agent-Schedule. Background Technology

[0002] Container resource scheduling technology is a core component of cloud computing and artificial intelligence infrastructure. With the rapid development of AI model training, inference, and graphics processing, GPU computing power has become a critical resource supporting these tasks, and multi-GPU server clusters are widely used in various computing-intensive scenarios. To meet the needs of businesses for temporary GPU computing power requests, existing technologies primarily employ three methods for resource allocation: Kubernetes GPU scheduling, traditional resource management platforms, or manual operation and maintenance scripts. Kubernetes typically uses node or entire GPU scheduling as the primary granularity; traditional resource management platforms focus on approval process management and resource ledger recording; and manual operation and maintenance scripts involve operations personnel manually performing operations such as server selection, GPU card allocation, Docker image pulling, and port mapping.

[0003] However, the aforementioned existing technologies are difficult to balance resource response speed, memory utilization and operational automation in application scenarios with multiple users, high-frequency requests, slow image startup, large differences in GPU memory, and the need for container preheating. Specifically, existing technologies suffer from several drawbacks. First, the resource request chain is lengthy, with image retrieval, container creation, and service startup only occurring after a user request arrives, failing to meet the demand for available containers within seconds. Second, the scheduling granularity is insufficient, scheduling is only performed on a host or entire GPU basis, failing to incorporate GPU memory, estimated memory usage of the image, container status, and port mapping for fine-grained decision-making. Third, the communication loop between the scheduling center and execution nodes is incomplete, relying on single HTTP requests or manual polling to ensure reliable instruction delivery and timely perception of execution results. Fourth, online status assessment is delayed, lacking short-cycle heartbeats and expiration caching mechanisms, making it prone to incorrect resource allocation based on expired status. Fifth, the distribution of idle containers is uneven, maintaining only the total number of idle containers without considering load differences between GPUs, resulting in some GPUs experiencing memory shortages while others remain idle, reducing resource utilization and subsequent allocation success rates. Sixth, the lack of effective locking and memory pre-allocation mechanisms during concurrent resource requests easily leads to duplicate allocations or over-allocation of memory, impacting system stability and reliability. Summary of the Invention

[0004] The present invention aims to at least partially solve one of the technical problems in the related art.

[0005] Therefore, the purpose of this invention is to propose a GPU scheduling system based on Agent-Schedule to solve problems such as slow resource request response, low GPU memory utilization, and poor scheduling execution reliability in the prior art.

[0006] To achieve the above objectives, a first aspect of this invention proposes a GPU scheduling system based on Agent-Schedule, comprising: a Schedule scheduling center, multiple Agent resource nodes, and a database; wherein, the Schedule scheduling center is deployed on an independent server and is used to uniformly receive resource request and release requests from business callers, maintain the real-time status of all cluster hosts, GPUs, and containers, generate and issue container operation instructions, process the instruction execution callback results of Agent nodes, and complete intelligent allocation and load balancing of GPU resources through a pre-created idle container pool mechanism, a concurrent control mechanism for pre-occupied video memory and global locks, a GPU rebalancing algorithm that migrates only READY idle containers, and a container lifecycle state machine management mechanism; Agent resource nodes are deployed independently on each GPU server. They are used to register local GPU hardware information with the Schedule scheduling center, periodically report container running status and resource usage, receive and execute container operation instructions issued by the Schedule, and send instruction execution results back to the Schedule. Through a long polling instruction channel and callback closed-loop mechanism, and a short-cycle heartbeat and expired cache status maintenance mechanism, they achieve reliable communication and status synchronization with the Schedule scheduling center. The database stores six core data models: image, host, GPU card, container, instruction, and lock. It is used to persist resource configuration, running status, and instruction flow records, providing data support for the global lock concurrency control mechanism, instruction execution closed-loop mechanism, and container state machine management mechanism.

[0007] Furthermore, in one embodiment of the present invention, the Schedule scheduling center includes a resource interface module, a resource service implementation module, an automatic scaling processing module, an instruction holding module, a connection holding module, and a callback processing module. The resource interface module provides an open application programming interface to receive resource application requests, release requests, and query requests from the business caller. The resource service implementation module executes core resource allocation logic, selecting the optimal GPU resources and containers based on the real-time cluster status. The automatic scaling processing module handles automatic scaling events, maintains the idle container pool holding amount corresponding to each image, and triggers the GPU rebalancing process. The instruction holding module maintains an independent queue of pending instructions based on host identifiers, storing all unexecuted container operation instructions. The connection holding module maintains a long-polling connection queue of currently online Agent resource nodes based on host identifiers, establishing a correspondence between instructions and execution nodes. The callback processing module receives instruction execution results reported by the Agent resource nodes and synchronously updates the instruction status, container status, and GPU memory usage information in the database.

[0008] Furthermore, in one embodiment of the present invention, the Agent resource node includes a proxy connection module, a host registration module, a heartbeat reporting module, an instruction distribution module, and a container command execution component. The proxy connection module is used to establish and maintain a long polling connection with the Schedule scheduling center to receive encrypted container operation instructions. The host registration module is used to register its local Internet Protocol address, GPU hardware information, and a list of currently running containers with the Schedule scheduling center when the Agent resource node starts or detects a change in the Schedule scheduling center service instance. The heartbeat reporting module is used to periodically report the real-time running status and resource usage data of hosts and containers to the Schedule scheduling center. The instruction distribution module is used to parse the received instructions and call the corresponding execution component according to the instruction type. The container command execution component is used to perform container creation, startup, shutdown, deletion, preheating, and health check operations.

[0009] Furthermore, in one embodiment of the present invention, the pre-created idle container pool mechanism specifically comprises: The system configures an independent idle container availability parameter for each image to define the total number of idle containers that the image should maintain. The idle containers include those in the creation and ready states. The automatic scaling up / down processing module periodically receives automatic scaling up / down events, and performs throttling for 60 seconds for non-forced events. When the number of ready idle containers for an image is less than the available idle container availability, or when the total number of ready and creation idle containers is less than the available idle container availability, a scaling up process is triggered to create new idle containers on the GPU with sufficient remaining video memory. When the number of ready idle containers for an image continuously exceeds the available idle container availability and exceeds a preset cooldown period, a scaling down process is triggered to select and release excess idle containers according to a preset strategy.

[0010] Furthermore, in one embodiment of the present invention, the memory pre-allocation and global lock concurrency control mechanism specifically comprises: when the resource request is received, the resource service implementation module first acquires a global distributed lock to ensure that only one request processes the resource allocation of the image at a time; queries available free containers on the online host, and if there are ready free containers, prioritizes the container with the most available free containers on the same GPU for allocation; if there are no available free containers, selects the GPU with the largest remaining memory in the cluster, and pre-allocates the memory resources on that GPU equal to the expected memory usage of the image; if the pre-allocation is successful, a container record is created and a creation and startup command is issued; if the pre-allocation fails, a resource insufficiency prompt is returned.

[0011] Further, in one embodiment of the present invention, the GPU rebalancing algorithm for migrating only ready-to-use idle containers specifically comprises: after the automatic scaling up and down processing module completes the scaling up and down of the idle container pool, when the number of online hosts in the cluster is not less than 2, starting the GPU rebalancing process; collecting the used video memory, available video memory, and ready-to-use idle container list of all online GPUs, selecting the GPU with the largest used video memory as the source GPU, and selecting the GPU with the largest available video memory as the target GPU; selecting the ready-to-use idle container with the smallest video memory usage from the source GPU as a migration candidate; when the available video memory of the target GPU is greater than the video memory usage of the candidate container, and the available video memory of the target GPU after migration is not less than the available video memory of the source GPU after migration, performing the migration operation; the migration operation is completed by first creating a new ready-to-use idle container on the target GPU, and then releasing the old idle container on the source GPU, without migrating any service containers in use.

[0012] Furthermore, in one embodiment of the present invention, the container lifecycle state machine management mechanism specifically includes seven states: creating, ready, task starting, in use, recycling, recycled, and creation failed. When creating an idle container, the state is in the process of creating; after the Agent resource node successfully creates the container and sends a callback, the state is updated to ready. When allocating an idle container, the state is updated to task starting; after successful business preheating, the state is updated to in use. When the user releases the resource, the state is updated to recycling; after the Agent resource node successfully stops the container, the state is updated to recycled and the corresponding GPU memory is released. When container creation or startup fails, the state is updated to creation failed, and the pre-occupied GPU memory and port resources are automatically released.

[0013] Further, in one embodiment of the present invention, the long polling instruction channel and callback closed-loop mechanism are specifically as follows: the Agent resource node initiates a 10-second timeout long polling request to the Schedule scheduling center through the long polling interface; after the Schedule scheduling center generates an instruction, it first writes the instruction information into the instruction table of the database, and then puts the instruction into the instruction queue of the instruction holding module corresponding to the host identifier; the instruction executor retrieves the instruction from the instruction queue, returns the encrypted instruction content through the long polling connection corresponding to the Agent resource node, and if the connection times out or is disconnected, the instruction is put back to the head of the queue to wait for the next connection; after the Agent resource node executes the instruction, it reports the execution result to the Schedule scheduling center through the callback interface, and the Schedule scheduling center updates the confirmation status, container status and GPU memory information of the instruction table according to the execution result.

[0014] Furthermore, in one embodiment of the present invention, the short-cycle heartbeat and expired cache status maintenance mechanism specifically comprises: the heartbeat reporting module of the Agent resource node reports the container running status and resource usage data to the Schedule scheduling center every 2 seconds; the Schedule scheduling center stores the host status and container status in the host status storage area and the container status storage area respectively, both of which use a 20-second expiration cache; during resource allocation, only hosts that have reported heartbeats within 20 seconds are considered online hosts, and all offline nodes are filtered out; when the Agent resource node detects that the identifier of the currently connected Schedule scheduling center service instance has changed, it automatically triggers the host registration module to re-register and reports all GPU information and container status of the local machine.

[0015] Furthermore, in one embodiment of the present invention, an exception handling mechanism is also included, specifically: when the long polling connection of the Agent resource node is disconnected, the unsuccessfully issued instructions are retained in the instruction queue of the corresponding host identifier, and are automatically reissued after the Agent resource node reconnects; the Schedule scheduling center sets differentiated timeout times for different types of instructions, with a timeout of 120 seconds for creating and starting container-type instructions and a timeout of 60 seconds for stopping container-type instructions; when an instruction execution times out or fails, a retry process is automatically executed, with a maximum of 3 retries, and retry intervals of 5 seconds, 10 seconds, and 20 seconds; if the maximum number of retries is reached and the instruction still fails, the instruction is marked as a final failure, the corresponding pre-occupied GPU memory, ports, and other resources are released, and the container status is updated to the corresponding failure status.

[0016] Therefore, this application has the following beneficial effects: This application embodiment utilizes a Schedule scheduling center to uniformly receive resource request and release requests from business callers, maintain the real-time running status of all cluster hosts, GPUs, and containers, generate and issue container operation instructions, and process the instruction execution callback results of Agent nodes, achieving unified management and intelligent scheduling decisions for global resources. Agent resource nodes can register their local GPU hardware information with the Schedule scheduling center, periodically report container running status and resource usage, receive and execute issued container operation instructions, and receive callback execution results, achieving precise execution and status synchronization of local resources. A pre-created idle container pool mechanism allows configuring the number of idle containers based on image dimensions, automatically executing scaling processes in the background, and completing container creation, environment initialization, and business warm-up in advance, shifting container creation costs to idle periods and significantly shortening user resource request waiting time. A memory pre-allocation and global lock concurrency control mechanism allows pre-allocation of corresponding GPU memory during resource allocation, combined with a global distributed lock for serialization protection, prioritizing allocation to GPUs with the most idle containers or the largest remaining memory, effectively avoiding... Overselling of GPU memory and conflicts with concurrent resources; a persistent connection between the scheduling center and Agent nodes can be established through a long polling instruction channel and a callback closed-loop mechanism. Instructions are persisted before being issued, and encrypted transmission and callback confirmation mechanisms are used to ensure reliable instruction issuance and timely feedback of execution results, forming a complete scheduling execution closed loop; a short-cycle heartbeat and expired cache status maintenance mechanism can collect host and container status at high frequency, use short-expiration cache to filter offline nodes, automatically trigger re-registration and status recovery when service instances switch, and update the global resource view in real time to avoid incorrect scheduling based on expired status; a GPU rebalancing algorithm that migrates only ready idle containers can automatically detect uneven load among multiple GPUs, select migration objects according to benefit-oriented conditions, and use an indirect migration method of target new source release to optimize the distribution of idle containers, improve the overall utilization of GPU memory, and not affect running business containers; a container full lifecycle state machine management mechanism can finely manage the seven states of containers from creation to recycling, automatically handle the transition logic between each state, and automatically roll back GPU memory and port resources in case of failure to ensure consistency between resource usage and system status. This solves the problems of slow resource request response, low GPU memory utilization, and poor scheduling execution reliability in existing technologies.

[0017] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0018] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a schematic diagram of the structure of the Agent-Schedule-based GPU scheduling system provided according to an embodiment of this application; Figure 2 This is a schematic diagram of the Agent resource node registration and heartbeat status reporting process according to an embodiment of this application; Figure 3 This is a schematic diagram of an automatic scaling up / down of an idle container pool and GPU load rebalancing process according to an embodiment of this application; Figure 4 This is a schematic diagram of the GPU resource request and allocation execution process according to an embodiment of this application; Figure 5 This is a schematic diagram of a closed loop for long polling instruction transmission and execution result callback according to an embodiment of this application; Figure 6 This is a schematic diagram of the container's full lifecycle state transition according to an embodiment of this application. Detailed Implementation

[0019] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0020] The following describes an embodiment of the Agent-Schedule-based GPU scheduling system of the present invention with reference to the accompanying drawings.

[0021] Figure 1 This is a schematic diagram of the structure of the Agent-Schedule-based GPU scheduling system provided in the embodiments of this application.

[0022] This application provides an Agent-Schedule-based GPU scheduling system, including: a Schedule scheduling center, multiple Agent resource nodes, and a database.

[0023] The Schedule center, deployed on a separate server, is used to uniformly receive resource request and release requests from business callers, maintain the real-time status of all cluster hosts, GPUs, and containers, generate and issue container operation instructions, process the instruction execution callback results of Agent nodes, and achieve intelligent allocation and load balancing of GPU resources through a pre-created idle container pool mechanism, a GPU memory pre-allocation and global lock concurrency control mechanism, a GPU rebalancing algorithm that migrates only READY idle containers, and a container lifecycle state machine management mechanism. Agent resource nodes are deployed independently on each GPU server and are used to inject data into the Schedule center. The system registers local GPU hardware information, periodically reports container running status and resource usage, receives and executes container operation instructions issued by the Schedule, and sends instruction execution results back to the Schedule. It also achieves reliable communication and status synchronization with the Schedule scheduling center through a long polling instruction channel and callback closed-loop mechanism, a short-cycle heartbeat and expired cache status maintenance mechanism. The database stores six core data models: image, host, GPU card, container, instruction, and lock. It is used to persist resource configuration, running status and instruction flow records, and provide data support for the global lock concurrency control mechanism, instruction execution closed-loop mechanism and container state machine management mechanism.

[0024] Specifically, the Schedule scheduling center can uniformly receive resource request and release requests from business callers, maintain the real-time running status of all cluster hosts, GPUs, and containers, generate and issue container operation instructions, and process the instruction execution callback results of Agent nodes, achieving unified management and intelligent scheduling decisions for global resources. Agent resource nodes can register their local GPU hardware information with the Schedule scheduling center, periodically report container running status and resource usage, receive and execute issued container operation instructions, and receive callback execution results, achieving precise execution and status synchronization of local resources. The pre-created idle container pool mechanism can configure the number of idle containers based on image dimensions, automatically executing scaling processes in the background, completing container creation, environment initialization, and business warm-up in advance, shifting container creation costs to idle periods and significantly shortening user resource request waiting time. The memory pre-allocation and global lock concurrency control mechanism can pre-allocate the corresponding GPU memory during resource allocation, using a global distributed lock for serialization protection, prioritizing allocation to the GPU with the most idle containers or the largest remaining memory, effectively avoiding memory concurrency issues. Overselling and concurrent resource conflicts are addressed. A persistent connection between the scheduling center and Agent nodes is established through a long-polling command channel and callback closed-loop mechanism. Commands are persisted before being issued, and encrypted transmission and callback confirmation mechanisms ensure reliable command issuance and timely feedback of execution results, forming a complete scheduling execution closed loop. A short-cycle heartbeat and expired cache status maintenance mechanism enables high-frequency collection of host and container status. Short-expired caches filter offline nodes, and automatic re-registration and status recovery are triggered when service instances switch, updating the global resource view in real time to avoid erroneous scheduling based on expired states. A GPU rebalancing algorithm that migrates only ready-to-use idle containers automatically detects uneven load distribution among multiple GPUs, selects migration targets based on benefit-oriented conditions, and uses an indirect migration method of target-created source release to optimize the distribution of idle containers, improve overall GPU memory utilization, and does not affect running business containers. A container lifecycle state machine management mechanism enables fine-grained management of the seven states of containers from creation to recycling, automatically handling the transition logic between states, and automatically rolling back memory and port resources in case of failure, ensuring consistency between resource usage and system state. This solves the problems of slow resource request response, low GPU memory utilization, and poor scheduling execution reliability in existing technologies.

[0025] In one embodiment of the present invention, the Schedule scheduling center includes: a resource interface module, a resource service implementation module, an automatic scaling processing module, an instruction holding module, a connection holding module, and a callback processing module.

[0026] The system comprises the following modules: a resource interface module, a resource service implementation module, and a callback module. The resource interface module provides an open application programming interface (API) to receive resource requests, releases, and queries from business users. The resource service implementation module executes the core resource allocation logic, selecting the optimal GPU resources and containers based on the real-time cluster status. The auto-scaling module handles auto-scaling events, maintains the available container pool for each image, and triggers the GPU rebalancing process. The instruction holding module maintains an independent queue of pending instructions based on host identifiers, storing all unexecuted container operation instructions. The connection holding module maintains a long-polling connection queue of currently online Agent resource nodes based on host identifiers, establishing a mapping between instructions and execution nodes. The callback processing module receives instruction execution results reported by Agent resource nodes and synchronously updates the instruction status, container status, and GPU memory usage information in the database.

[0027] Specifically, the resource interface module provides standardized OpenAPI interfaces, covering three core types of interfaces: resource application, resource release, and resource query, adapting to the access needs of various business callers; the resource service implementation module reads host running information, GPU hardware parameters, container running information, and image configuration information from the database in real time, and combines multiple fixed parameters such as the remaining GPU memory value, the current running status of the container, and the preset memory usage value of the image to perform fine-grained resource allocation decisions; the automatic scaling handling module receives periodically triggered and user-mandated automatic scaling events, performs 60-second throttling on non-mandatory events to avoid high-frequency repeated scaling operations in a short period of time, and dynamically adjusts the idle container availability based on the idle container availability configured for each image. The system counts containers and initiates the GPU rebalancing process. The instruction holding module first writes each instruction to be issued to the database for persistent storage, and then stores it in an independent instruction queue with the corresponding host identifier to prevent instruction loss and ensure that instructions are traceable and retryable. The connection holding module maintains an independent long polling connection for each online Agent resource node, uses a delayed result mechanism to keep the connection active, and matches the instruction queue with online connections in real time to ensure that instructions can be issued to the corresponding Agent resource nodes in a timely manner. After receiving the instruction execution results returned by the Agent resource nodes, the callback processing module distinguishes the instruction type and executes the corresponding processing logic, and synchronously updates the instruction confirmation status, container running status and GPU memory usage data in the database to ensure the consistency and accuracy of the cluster resource status.

[0028] In one embodiment of the present invention, the Agent resource node includes a proxy connection module, a host registration module, a heartbeat reporting module, an instruction distribution module, and a container command execution component.

[0029] The system comprises the following modules: Agent Connection Module (ACM) establishes and maintains a long polling connection with the Schedule scheduling center to receive encrypted container operation commands; Host Registration Module registers the host's Internet Protocol address, GPU hardware information, and a list of currently running containers with the Schedule scheduling center when the Agent resource node starts or detects a change in the Schedule scheduling center service instance; Heartbeat Reporting Module periodically reports the real-time running status and resource usage data of hosts and containers to the Schedule scheduling center; Command Distribution Module parses received commands and calls the corresponding execution component based on the command type; and Container Command Execution Module performs container creation, startup, shutdown, deletion, preheating, and health check operations.

[0030] Specifically, the proxy connection module sets a ten-second timeout threshold for long polling requests, and transmits command data in an encrypted manner throughout to prevent command content leakage. It can automatically re-initiate a connection request after a connection is lost. The host registration module submits registration information including the local network address, GPU device number, memory specifications, and basic information of all locally deployed containers. After registration, the node identity is bound. The heartbeat reporting module reports every two seconds, and the reported data includes host load, real-time GPU memory usage, container process ID, and running status, providing a basis for the scheduling center to determine the node's online status. The command distribution module can accurately distinguish between different command categories for container creation, start-up, and destruction, and accurately match the corresponding execution channels to avoid mismatched command execution. The container command execution component relies on the local operating environment to complete the entire container operation and maintenance process. Containers can only be used in actual business after preheating, and regular container health checks are performed to promptly identify abnormally running containers and ensure the stable and orderly operation of local GPU containers.

[0031] In one embodiment of the present invention, the pre-creation of an idle container pool mechanism specifically includes: The system configures an independent idle container availability parameter for each image, defining the total number of idle containers that the image should maintain. Idle containers include those in the creation and ready states. The automatic scaling up / down processing module periodically receives automatic scaling up / down events, and performs throttling for 60 seconds for non-forced events. When the number of ready idle containers for an image is less than the idle container availability, or the total number of ready and creation idle containers is less than the idle container availability, the scaling up process is triggered, creating new idle containers on GPUs with sufficient remaining video memory. When the number of ready idle containers for an image continuously exceeds the idle container availability and exceeds a preset cooldown period, the scaling down process is triggered, selecting and releasing excess idle containers according to a preset strategy.

[0032] Specifically, the system pre-defines dedicated idle container values ​​for images in different usage scenarios to adapt to the resource call frequency of different business scenarios. Preferably, the idle container capacity under different business scenarios can be configured with the following typical values: For large model inference images with long cold start times and high single-request value, the idle container capacity is preferably configured to 1~2 to ensure rapid response to sudden requests while controlling GPU memory usage; for online inference service images with frequent calls, dense request arrivals, and relatively controllable single-container resource usage, the idle container capacity is preferably configured to 2~5; for lightweight utility, testing, or low-frequency call images, the idle container capacity can be configured to 0~1; for popular base images shared by different tenants, the idle container capacity can be appropriately increased, for example, configured to 3~5, to reduce repeated cold starts. The automatic scaling module periodically checks and counts the actual number of containers in the creation and ready states under various images, strictly implementing throttling restrictions on non-actively initiated scaling events to avoid frequent scheduling adjustments. The expansion process is triggered when one of the following conditions is met: the number of ready idle containers for a certain image is less than the available idle containers; the total number of ready and creating idle containers for a certain image is less than the available idle containers; or idle containers are occupied after a user resource request, causing the idle pool to fall below the target available capacity. After expansion is triggered, new idle containers are created on GPUs with sufficient remaining video memory. During the creation of new idle containers, the entire process of image retrieval, runtime environment configuration, and business function warm-up is completed simultaneously. The shrinkage process is triggered when all of the following conditions are met: the number of ready idle containers for a certain image is consistently greater than the available idle containers; the above-mentioned excess state persists for more than a preset cooldown period; there is currently no need to migrate any containers that are in use, and releasing the idle container will not affect the minimum available capacity. After shrinkage is triggered, excess idle containers are selected and released according to a preset strategy. Setting a preset cooling period can smoothly handle short-term peak resource usage and prevent erroneous scaling-down operations caused by instantaneous traffic changes. When performing scaling-down operations, the idle time of containers is used as the filtering criterion, and ready containers that have been idle for a long time are cleaned up first. This reduces the space occupied by idle resources while ensuring that daily business can access container resources normally. When scaling down, one or more of the following combined strategies can be used to select ready-to-release free containers: prioritize free containers that have not been allocated or used recently; prioritize containers with a large number of free containers on the GPU to improve overall distribution balance; prioritize free containers with small memory usage to reduce the risk of high costs associated with recreating them after release; prioritize free containers located on non-hotspot hosts or non-priority reserved GPUs; if the system is performing GPU rebalancing, prioritize releasing old free containers from the source GPU that have already been rebuilt on the target GPU.

[0033] In one embodiment of the present invention, the concurrent control mechanism for video memory pre-occupancy and global lock is specifically as follows: When a resource request is received, the resource service implementation module first acquires a global distributed lock to ensure that only one request can process the resource allocation for the image at a time. It then queries the available free containers on the online hosts. If there are ready free containers, it prioritizes the container with the most available free containers on the same GPU for allocation. If there are no available free containers, it selects the GPU with the largest remaining video memory in the cluster and pre-allocates video memory resources on that GPU equal to the expected video memory usage of the image. If the pre-allocation is successful, a container record is created and a creation and startup command is issued. If the pre-allocation fails, a resource insufficiency prompt is returned.

[0034] Specifically, a globally distributed lock locks the resource allocation permissions for a single image, restricting multiple requests from simultaneously intervening in the resource allocation of the same image, thus preventing allocation errors caused by concurrent contention from the source. The allocation is completed using the container with the most free containers within the same GPU, fully utilizing the existing container resources within a single device and reducing the interaction time caused by cross-server scheduling. The memory pre-allocation process strictly allocates the occupied space according to the fixed memory value required for image operation, accurately locking the hardware resource usage quota and completely avoiding excessive memory usage caused by multiple tasks running in parallel. After the memory pre-allocation process is verified, the system synchronously records the container ownership information and binds the running hardware information to form a complete data record, and synchronously pushes the corresponding execution operation instructions. Once the remaining hardware memory cannot match the image's running standards, the system directly outputs a fixed resource shortage feedback message, directly terminating the current resource allocation process and reducing the consumption of system resources by ineffective scheduling processes.

[0035] In one embodiment of the present invention, the GPU rebalancing algorithm for migrate-ready idle containers is specifically as follows: After the automatic scaling up and down of the idle container pool, the GPU rebalancing process is initiated when there are at least two online hosts in the cluster. This involves collecting the used memory, available memory, and ready idle containers of all online GPUs, selecting the GPU with the largest used memory as the source GPU, and the GPU with the largest available memory as the target GPU. From the source GPU, the ready idle container with the smallest memory usage is selected as a migration candidate. If the available memory of the target GPU is greater than the memory usage of the candidate container, and the available memory of the target GPU after migration is not less than the available memory of the source GPU after migration, the migration operation is performed. The migration operation is completed by first creating a new ready idle container on the target GPU and then releasing the old idle container on the source GPU, without migrating any in-use service containers.

[0036] Specifically, the GPU rebalancing process is set with pre-trigger conditions, requiring the scaling up / down operation to be completed and the number of online hosts in the cluster to be two or more before it can start, avoiding invalid migration operations on a single node cluster. The system batch collects two types of fixed video memory data and a list of ready-to-use idle containers from all online GPUs, excluding data from containers in use, to reduce system data processing pressure. The GPU with the highest video memory consumption is used as the migration source, and the GPU with the largest remaining video memory capacity is used as the migration destination, accurately locating the hardware devices with unbalanced loads. The system prioritizes selecting the idle containers with the smallest video memory usage within the source GPU as migration targets, reducing the resource fluctuation range of a single migration. The system sets dual judgment conditions: the first condition is that the target GPU's video memory capacity can fully accommodate the candidate containers, and the second condition is that the video memory load of the two GPUs tends to be balanced after the migration, preventing invalid reverse migrations. An asynchronous migration mode of creating and then destroying is adopted, ensuring that the running business containers are not modified or interrupted throughout the process, guaranteeing that online business operations are not affected by load balancing operations and stabilizing the overall video memory allocation structure of the cluster.

[0037] In one embodiment of the present invention, the container lifecycle state machine management mechanism is as follows: the container state includes seven types: creating, ready, task starting, in use, recycling, recycled, and creation failed; when creating an idle container, the state is creating, and after the Agent resource node successfully creates and calls back, it is updated to ready; when allocating an idle container, the state is updated to task starting, and after the business preheating is successful, it is updated to in use; when the user releases resources, the state is updated to recycling, and after the Agent resource node successfully stops the container, it is updated to recycled and the corresponding GPU memory is released; when container creation or startup fails, the state is updated to creation failed, and the pre-occupied GPU memory and port resources are automatically released.

[0038] Specifically, the system defines seven non-overlapping standard operating states for containers, forming a closed-loop state transition logic and clarifying the working attributes of containers at each stage. After a container is initialized and generated, it is fixedly marked as being in the "creating" state. It waits for the node to return a creation success callback information and then changes to the "ready" state, ensuring that idle containers are fully warmed up before participating in resource allocation. After the system completes the container allocation instruction, it switches the container to the "task starting" state. After the business program is warmed up, it officially enters the "in use" state, strictly distinguishing between the scheduling phase and the business operation phase. After the user initiates a resource release request, the container enters the "reclaiming" state. After the node completes the container stop operation, it completely releases the GPU memory resources and marks the container as reclaimed. If any abnormal error occurs in any creation or startup process, the container directly jumps to the "creation failed" state, and the system actively clears the occupied GPU memory resources and port resources to avoid invalid resources occupying GPU hardware space for a long time.

[0039] In one embodiment of the present invention, the long polling instruction channel and the callback closed-loop mechanism are specifically as follows: Agent resource nodes initiate long polling requests with a 10-second timeout to the Schedule scheduling center via a long polling interface. After generating instructions, the Schedule scheduling center first writes the instruction information into the instruction table in the database, and then puts the instructions into the instruction queue of the instruction holding module corresponding to the host identifier. The instruction executor retrieves the instructions from the instruction queue and returns the encrypted instruction content through the long polling connection of the corresponding Agent resource node. If the connection times out or is disconnected, the instruction is put back to the head of the queue to wait for the next connection. After the Agent resource node executes the instructions, it reports the execution result to the Schedule scheduling center through a callback interface. The Schedule scheduling center updates the confirmation status, container status, and GPU memory information of the instruction table according to the execution result.

[0040] Specifically, Agent resource nodes establish long-polling communication links with a fixed ten-second timeout period to continuously maintain smooth communication with the scheduling center. After the scheduling center generates any operation command, it prioritizes completing the persistent database write operation before sorting and storing the commands in the queue, ensuring the traceability of all commands. The system extracts command content sequentially from the head of the queue and pushes it to the corresponding node through encrypted transmission to prevent security risks caused by plaintext transmission of commands. When the communication link experiences a timeout interruption or active disconnection, the command is not directly discarded but is uniformly rolled back to the head of the queue to await reconnection and resending, ensuring that the command is not lost. After the node completes all command actions, it sends back the execution result data through an independent callback interface. The scheduling center synchronously modifies the command confirmation flag, container running parameters, and GPU memory usage values ​​in the database, forming a complete closed-loop link of issuance, execution, callback, and update.

[0041] In one embodiment of the present invention, the short-cycle heartbeat and expired cache state maintenance mechanism is specifically as follows: The Agent resource node's heartbeat reporting module reports container running status and resource usage data to the Schedule scheduling center every 2 seconds. The Schedule scheduling center stores host status and container status in the host status storage area and container status storage area respectively, both of which use a 20-second expiration cache. During resource allocation, only hosts that have reported heartbeats within 20 seconds are considered online hosts, filtering out all offline nodes. When the Agent resource node detects a change in the identifier of the currently connected Schedule scheduling center service instance, it automatically triggers the host registration module to re-register and reports all GPU information and container status of the local machine.

[0042] Specifically, the heartbeat reporting module strictly uploads node data at a fixed two-second interval. The uploaded content includes host operating load, GPU memory usage, and container running status. The scheduling center separately divides the storage areas to store host data and container data, respectively. All data adopts a fixed expiration time of twenty seconds to eliminate outdated and invalid data. Before the system performs resource allocation calculations, it strictly filters valid hosts within the heartbeat time limit and completely eliminates disconnected or downtimed offline nodes to avoid making incorrect scheduling judgments based on expired data. Nodes monitor the binding scheduling center instance number in real time. Once a service instance switch change is detected, the node automatically starts the re-registration process, fully reports the local GPU hardware parameters, device number, and all current container running information, and quickly synchronizes the latest cluster resource view.

[0043] In one embodiment of the present invention, an exception handling mechanism is further included, specifically: When the Agent resource node's long polling connection is disconnected, unsuccessfully issued instructions are retained in the instruction queue of the corresponding host identifier and automatically reissued after the Agent resource node reconnects. The Schedule center sets differentiated timeout times for different types of instructions: 120 seconds for creating and starting container-type instructions, and 60 seconds for stopping container-type instructions. When an instruction times out or fails, a retry process is automatically executed, with a maximum of 3 retries at intervals of 5, 10, and 20 seconds. If the instruction still fails after reaching the maximum number of retries, it is marked as a final failure, the corresponding pre-allocated GPU memory, ports, and other resources are released, and the container status is updated to the corresponding failure status.

[0044] Specifically, the system is equipped with a comprehensive exception handling mechanism throughout the entire process. When the communication link is abnormally disconnected, the pending instructions are continuously stored in the dedicated host queue without being cleared. For instructions that have been retrieved from the queue but have not yet returned, they are put back to the head of the queue or the pending position when the connection fails, to avoid instruction loss. After the node restores the connection, the system automatically completes the resending of all pending instructions, ensuring uninterrupted instruction execution. Different timeout durations are set for two different types of commands: container creation / startup and container shutdown. The maximum execution time for container creation / startup commands is set to 120 seconds, while the maximum execution time for container shutdown / destruction commands is set to 60 seconds, adapting to the different execution times of different operations. If no Agent callback result is received within the timeout period, the corresponding command table record is marked as timed out or failed, and a retry process is triggered. If the command involves memory pre-allocation, port occupancy, or container state changes, a resource rollback operation is performed after exceeding the maximum number of retries. When a Schedule service instance switches, the Agent detects a change in the identifier of the currently connected service instance during periodic communication. This automatically triggers the host registration module to re-register and re-report all local GPU information, container list, and status information. The new Schedule instance restores the host status, GPU status, and container status based on the data re-reported by the Agent. Simultaneously, after the Agent re-establishes a long polling connection, the connection queue is re-registered in the new Schedule instance. Previously unconfirmed instructions continue to be dequeued and issued according to the host identifier, thus restoring the correspondence between the instruction queue and the connection queue.

[0045] After an instruction times out or fails to execute, retrying is performed sequentially at intervals of 5 seconds, 10 seconds, and 20 seconds, with a maximum of 3 retries. If execution still fails after reaching the maximum number of retries, the instruction is marked as a final failure, and differentiated follow-up processing is performed based on the instruction type: for creation or startup instructions, pre-allocated GPU memory and port resources are released, and the container status is updated to a creation failure status; for stop or recycling instructions, the container status is updated to a pending manual handling or recycling exception status, and detailed failure reasons are recorded for subsequent compensation or manual intervention. This comprehensive approach ensures the stable and reliable operation of the entire GPU scheduling system, effectively improving the allocation and overall utilization efficiency of cluster GPU resources.

[0046] The Agent-Schedule-based GPU scheduling system proposed in this application can uniformly receive resource request and release requests from business callers through the Schedule scheduling center, maintain the real-time running status of all cluster hosts, GPUs, and containers, generate and issue container operation instructions, and process the instruction execution callback results of Agent nodes, thereby achieving unified management and intelligent scheduling decisions for global resources. Agent resource nodes can register their local GPU hardware information with the Schedule scheduling center, periodically report container running status and resource usage, receive and execute issued container operation instructions, and receive callback execution results, achieving precise execution and status synchronization of local resources. The pre-created idle container pool mechanism can configure the number of idle containers based on image dimensions, automatically execute scaling processes in the background, and complete container creation, environment initialization, and business warm-up in advance, shifting container creation costs to idle periods and significantly shortening user resource request waiting time. The memory pre-allocation and global lock concurrency control mechanism can pre-allocate the corresponding GPU memory during resource allocation, and use a global distributed lock to achieve serialization protection, prioritizing the GPU with the most idle containers or the most remaining memory. Large GPUs are allocated to effectively avoid over-sale of VRAM and concurrent resource conflicts; a persistent connection between the scheduling center and Agent nodes is established through a long polling instruction channel and callback closed-loop mechanism. Instructions are persisted before being issued, and encrypted transmission and callback confirmation mechanisms ensure reliable instruction issuance and timely feedback of execution results, forming a complete scheduling execution closed loop; a short-cycle heartbeat and expired cache status maintenance mechanism can collect host and container status frequently, use short-expiration cache to filter offline nodes, automatically trigger re-registration and status recovery when service instances switch, and update the global resource view in real time to avoid base Error scheduling is performed in expired states; a GPU rebalancing algorithm that migrates only idle containers in the ready state can automatically detect uneven load distribution among multiple GPUs, select migration targets based on benefit-oriented conditions, and adopt an indirect migration method of target-created source release to optimize the distribution of idle containers, improve the overall utilization of GPU memory, and not affect running business containers; a container lifecycle state machine management mechanism can finely manage the seven states of containers from creation to recycling, automatically handle the transition logic between each state, and automatically roll back memory and port resources in case of failure, ensuring the consistency of resource usage and system state. Thus, it solves the problems of slow resource request response, low GPU memory utilization, and poor scheduling execution reliability in existing technologies.

[0047] The following will illustrate an Agent-Schedule-based GPU scheduling system through a specific embodiment, including: After system deployment, the Agent resource nodes deployed on each GPU server sequentially start up and go online. Upon startup, each node proactively submits its local network address, GPU hardware parameters, device number, and information about its existing containers to the Schedule center, completing node authentication and registration. During normal system operation, the heartbeat reporting module within each Agent resource node continuously reports host load, real-time GPU memory usage, and the running status data of all local containers to the Schedule center at a fixed two-second interval. The Schedule center uses a twenty-second expiration cache mechanism to filter valid online nodes and remove offline devices that have not reported data for an extended period. If an Agent resource node detects a change in the service instance it is connecting to, it automatically re-initiates the registration process, synchronously updating all its hardware and resource information. This entire node registration, online, and real-time status monitoring process is as follows: Figure 2 As shown.

[0048] With nodes running stably online, the scheduling center, relying on the preset idle container availability parameters for each image, automatically manages the number of idle containers and optimizes cluster load in the background. The entire automatic scaling up / down and GPU load rebalancing process is as follows: Figure 3 As shown, the system periodically checks and counts the actual number of containers in the creation and ready states. If the number does not reach the set retention standard, it automatically performs expansion operations, prioritizing the use of GPU devices with sufficient video memory to build idle containers. During the building process, image retrieval, environment configuration, and business warm-up are completed simultaneously. Once the number of idle containers exceeds the set standard and meets the preset cooling period, the system filters containers according to their idle duration and performs shrinkage cleanup. At the same time, it imposes a 60-second throttling limit on non-forced expansion and shrinkage events to avoid frequent scheduling and increase the system's operating burden. When the number of online hosts in the cluster reaches two or more, the system automatically starts the GPU rebalancing process. It summarizes the used video memory, available video memory, and ready idle containers of all online GPUs, selects the device with the highest video memory usage as the source GPU, selects the device with the largest remaining video memory as the target GPU, and selects the ready idle container with the smallest video memory usage in the source GPU as the migration target. After meeting the dual video memory judgment conditions, the allocation is completed by first creating a ready idle container on the target GPU and then releasing the original idle container on the source GPU. No migration operation is performed on the service containers that are in use throughout the process, and the cluster hardware load is balanced without affecting the operation of services.

[0049] When a business application generates a need for model inference, computing power, or other uses and initiates a formal resource request, the system formally enters the resource allocation and execution phase. The entire resource request and allocation process is as follows: Figure 4As shown. The resource service implementation module in the scheduling center first acquires the global distributed lock of the corresponding image to lock the resource allocation authority of a single image, avoiding scheduling chaos caused by multiple groups of application requests concurrently competing for resources; then it traverses all online server nodes, searches for ready idle containers that have completed preheating within the cluster, and prioritizes the idle container with the most idle containers within the same GPU device to directly complete the allocation and invocation; if there are no ready idle containers that can be used directly in the current cluster, the system automatically filters out the GPU hardware device with the largest remaining video memory capacity within the cluster, accurately pre-allocates hardware resources with the same video memory value as the target image's running, and generates the corresponding container creation and startup instructions after the video memory pre-allocation verification is successful; if the remaining video memory of the hardware cannot meet the usage requirements, the system directly sends out a resource shortage prompt message.

[0050] After the resource scheduling instruction is generated, the instruction issuance, transmission, and result feedback phase officially begins. The complete closed-loop process of instruction transmission and interaction is as follows: Figure 5 As shown, the scheduling center first records all pending operation instructions into the database for persistent storage, and then arranges them in an orderly manner into corresponding independent instruction queues according to their host identifiers. Each online Agent resource node maintains a long polling communication link with a 10-second timeout standard, continuously listening for the encrypted instructions issued by the scheduling center. When network communication is normal, it quickly receives the instruction content and calls the corresponding local functional components to complete various operations such as container creation, startup, and shutdown. If network fluctuations or temporary disconnection of the communication link occur, the unfinished instructions will remain in the original instruction queue, waiting for the node to re-establish the communication connection before being automatically resent, ensuring that no instructions are lost. After the Agent resource node completes all the corresponding operation actions of the instructions locally, it sends the accurate execution results back to the scheduling center through a dedicated callback interface. The scheduling center updates the instruction confirmation status, container running status, and real-time GPU memory usage data in the database based on the returned results, realizing a complete closed loop of instruction issuance, local execution, result return, and data synchronization.

[0051] Throughout the entire process of container deployment and subsequent resource recycling, all containers will strictly follow established rules to complete the orderly switching of their operating states. The complete state transition logic of a container is as follows: Figure 6As shown. When a container is first built but not yet used, it is in a "creating" state. Once the Agent resource node reports successful build information, the container's state officially changes to "ready." When a container is selected and invoked by a business application, it enters the "task initiation" state until all internal business processes are fully warmed up, at which point it officially switches to the "in use" state to undertake business computing tasks. When the corresponding business task ends and the business initiates a resource release request, the container immediately enters the "reclaiming" state. After the node completes the container shutdown operation and releases hardware resources, the container's state is updated to "reclaimed." If a runtime failure or abnormal parameter occurs during container build, creation, or startup, the container will directly jump to the "creation failed" state, and the system will automatically release the previously reserved GPU memory and port resources, preventing invalid hardware resources from being occupied for a long time.

[0052] In addition, the system is equipped with a comprehensive exception handling mechanism. Different timeout durations are set for two different types of instructions: container creation and startup, and container shutdown. After an instruction execution times out or fails, it will be retried at fixed intervals. If the maximum number of retries is reached and the instruction still cannot be executed normally, the system will actively terminate the scheduling process and release all reserved resources. This ensures the stable and reliable operation of the entire GPU scheduling system and effectively improves the allocation efficiency and overall utilization efficiency of cluster GPU resources.

[0053] In summary, in this embodiment of the invention, the entire system first completes the registration and real-time operation status monitoring of each node, then uses preset parameters to achieve autonomous control of the number of idle containers and load balancing of the cluster GPUs. Upon receiving a request for business resource usage, it uses a global lock and memory pre-allocation mechanism to carry out resource allocation in an orderly manner. It uses a dedicated communication method to complete the secure issuance of instructions, local execution, and synchronous update of running data. At the same time, it strictly follows the established standards to complete the orderly switching of the container's state throughout its entire lifecycle. Furthermore, it uses a comprehensive exception handling strategy to deal with various operational problems. The overall operation process is smooth and can quickly respond to various computing power service usage needs, reasonably balance the cluster hardware usage pressure, and effectively improve the GPU cluster resource scheduling efficiency and overall resource utilization.

[0054] In the description of this specification, the references to the terms "an embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0055] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of those features. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0056] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0057] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0058] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0059] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A GPU scheduling system based on Agent-Schedule, characterized in that, include: The system comprises a schedule center, multiple agent resource nodes, and a database; among which... The Schedule scheduling center is deployed on an independent server and is used to uniformly receive resource application and release requests from business callers, maintain the real-time status of all cluster hosts, GPUs and containers, generate and issue container operation instructions, process the instruction execution callback results of Agent nodes, and complete the intelligent allocation and load balancing of GPU resources through a pre-created idle container pool mechanism, a GPU rebalancing algorithm that migrates only READY idle containers, and a container lifecycle state machine management mechanism. The Agent resource nodes are deployed independently on each GPU server. They are used to register local GPU hardware information with the Schedule scheduling center, periodically report container running status and resource usage, receive and execute container operation instructions issued by the Schedule, and send instruction execution results back to the Schedule. They also achieve reliable communication and status synchronization with the Schedule scheduling center through long polling instruction channels and callback closed-loop mechanisms, short-cycle heartbeat and expired cache status maintenance mechanisms. The database stores six core data models: image, host, GPU card, container, instruction, and lock. These models are used to persist resource configuration, running status, and instruction flow records, providing data support for the global lock concurrency control mechanism, instruction execution closed-loop mechanism, and container state machine management mechanism.

2. The GPU scheduling system based on Agent-Schedule according to claim 1, characterized in that, The Schedule scheduling center includes a resource interface module, a resource service implementation module, an automatic scaling processing module, an instruction holding module, a connection holding module, and a callback processing module. The resource interface module is used to provide an open application programming interface to the outside world and to receive resource application, release and query requests from the business caller; The resource service implementation module is used to execute the core resource allocation logic and select the optimal GPU resources and containers based on the real-time status of the cluster. The automatic scaling processing module is used to handle automatic scaling events, maintain the amount of idle container pools corresponding to each image, and trigger the GPU rebalancing process. The instruction holding module is used to maintain an independent queue of instructions to be issued according to the host identifier, and to store all container operation instructions that have not been completed. The connection holding module is used to maintain a long polling connection queue of the currently online Agent resource nodes according to the host identifier, and to establish a correspondence between instructions and execution nodes; The callback processing module is used to receive the instruction execution results reported by the Agent resource node and synchronously update the instruction status, container status and GPU memory usage information in the database.

3. The GPU scheduling system based on Agent-Schedule according to claim 1, characterized in that, The Agent resource node includes a proxy connection module, a host registration module, a heartbeat reporting module, an instruction distribution module, and a container command execution component, wherein... The proxy connection module is used to establish and maintain a long polling connection with the Schedule scheduling center and receive encrypted container operation instructions. The host registration module is used to register the local Internet Protocol address, GPU hardware information and currently running container list with the Schedule scheduling center when the Agent resource node starts or detects a change in the Schedule scheduling center service instance; The heartbeat reporting module is used to periodically report the real-time running status and resource usage data of the host and container to the Schedule scheduling center; The instruction distribution module is used to parse the received instructions and call the corresponding execution component according to the instruction type; The container command execution component is used to perform container creation, startup, shutdown, deletion, preheating, and health check operations.

4. The GPU scheduling system based on Agent-Schedule according to claim 1, characterized in that, The pre-created idle container pool mechanism is specifically as follows: The system configures an independent idle container availability parameter for each image, which defines the total number of idle containers that the image should maintain. The idle containers include those in the creation and those in the ready state. The automatic scaling up / down processing module periodically receives automatic scaling up / down events, and performs throttling processing for 60 seconds for non-forced events; When the number of ready free containers in a certain image is less than the available free containers, or the total number of ready and creating free containers is less than the available free containers, the expansion process is triggered to create new free containers on the GPU with sufficient remaining video memory. When the number of ready idle containers in a certain image continuously exceeds the total number of idle containers and exceeds a preset cooling cycle, a scaling-down process is triggered, and excess idle containers are selected and released according to a preset strategy.

5. The GPU scheduling system based on Agent-Schedule according to claim 1, characterized in that, The specific mechanism for concurrent control of video memory pre-occupancy and global lock is as follows: When the resource request is received, the resource service implementation module first acquires a global distributed lock to ensure that only one request can process the resource allocation of the image at a time. Query the available free containers on the online host. If there are ready free containers, prioritize the container with the largest number of available free containers on the same GPU for allocation. If no available free container exists, select the GPU card with the largest remaining video memory in the cluster and pre-allocate video memory resources on that GPU equal to the video memory usage expected by the image. If the pre-allocation is successful, a container record is created and a creation and startup command is issued. If the pre-allocation fails, an insufficient resource message is returned.

6. The GPU scheduling system based on Agent-Schedule according to claim 1, characterized in that, The GPU rebalancing algorithm that only migrates idle containers in the ready state is as follows: After the automatic scaling up and down processing module completes the scaling up and down of the idle container pool, it starts the GPU rebalancing process when the number of online hosts in the cluster is not less than 2. Collect the used video memory, available video memory, and ready-to-use idle container list of all online GPUs, select the GPU with the largest used video memory as the source GPU, and select the GPU with the largest available video memory as the target GPU; Select the ready-state free container with the smallest video memory usage from the source GPU as the migration candidate; When the available video memory of the target GPU is greater than the video memory usage of the candidate container, and the available video memory of the target GPU after migration is not less than the available video memory of the source GPU after migration, the migration operation is performed. The migration operation is completed by first creating a new ready-to-use free container on the target GPU, and then releasing the old free container on the source GPU, without migrating any service containers in use.

7. The GPU scheduling system based on Agent-Schedule according to claim 1, characterized in that, The container lifecycle state machine management mechanism is specifically as follows: The container status includes seven types: creating, ready, task starting, in use, being recycled, recycled, and creation failed. When the idle container is created, its status is "creating". After the Agent resource node is successfully created and a callback is executed, its status is updated to "ready". When allocating the idle container, the status is updated to "task in progress"; after successful service preheating, it is updated to "in use". When the user releases the resource, the status is updated to "reclaiming". After the Agent resource node successfully stops the container, the status is updated to "reclaimed" and the corresponding GPU memory is released. When container creation or startup fails, the status is updated to "creation failed," and pre-allocated GPU memory and port resources are automatically released.

8. The GPU scheduling system based on Agent-Schedule according to claim 1, characterized in that, The long polling command channel and callback closed-loop mechanism are specifically as follows: The Agent resource node initiates a long polling request with a 10-second timeout to the Schedule scheduling center through the long polling interface. After the Schedule scheduling center generates an instruction, it first writes the instruction information into the instruction table of the database, and then puts the instruction into the instruction queue of the instruction holding module corresponding to the host identifier. The instruction executor retrieves an instruction from the instruction queue and returns the encrypted instruction content through a long polling connection to the corresponding Agent resource node. If the connection times out or is disconnected, the instruction is placed back at the head of the queue to wait for the next connection. After the Agent resource node executes the instruction, it reports the execution result to the Schedule scheduling center through a callback interface. The Schedule scheduling center updates the confirmation status, container status, and GPU memory information of the instruction table based on the execution result.

9. The GPU scheduling system based on Agent-Schedule according to claim 1, characterized in that, The short-cycle heartbeat and expired cache status maintenance mechanism is as follows: The heartbeat reporting module of the Agent resource node reports the container running status and resource usage data to the Schedule scheduling center every 2 seconds. The Schedule center stores the host state and container state in the host state storage area and the container state storage area respectively, and both use a 20-second expiration cache. When allocating resources, only hosts that report a heartbeat within 20 seconds are considered online hosts, and all offline nodes are filtered out. When the Agent resource node detects that the identifier of the currently connected Schedule service instance has changed, it automatically triggers the host registration module to re-register and reports all GPU information and container status of the local machine.

10. The GPU scheduling system based on Agent-Schedule according to claim 1, characterized in that, It also includes an exception handling mechanism, specifically: When the long polling connection of the Agent resource node is disconnected, the unsuccessfully sent instructions are retained in the instruction queue of the corresponding host identifier and will be automatically resent after the Agent resource node reconnects. The Schedule scheduling center sets differentiated timeout times for different types of instructions. The timeout time for creating and starting container-type instructions is 120 seconds, and the timeout time for stopping container-type instructions is 60 seconds. When an instruction execution times out or fails, a retry process is automatically executed, with a maximum of 3 retries. The retry interval is an incremental interval of 5 seconds, 10 seconds, and 20 seconds. If the maximum number of retries is reached and the failure still occurs, the instruction is marked as a final failure, the corresponding pre-occupied GPU memory, ports and other resources are released, and the container status is updated to the corresponding failure status.

Images